Reviewerly/exHarmony huggingface dataset:
Pantomime

We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatio-temporal properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.
Clinical prediction models: diagnosis versus prognosis

Clinical prediction models play an increasingly important role in contemporary clinical care, by informing healthcare professionals, patients and their relatives about outcome risks, with the aim to facilitate (shared) medical decision making and improve health outcomes. Diagnostic prediction models aim to calculate an individual's risk that a disease is already present, whilst prognostic prediction models aim to calculate the risk of particular heath states occurring in the future. This article serves as a primer for diagnostic and prognostic clinical prediction models, by discussing the basic terminology, some of the inherent challenges, and the need for validation of predictive performance and the evaluation of impact of these models in clinical care.
Image Manipulation Detection by Multi-View Multi-Scale Supervision

The key challenge of image manipulation detection is how to learn generalizable features that are sensitive to manipulations in novel data, whilst specific to prevent false alarms on authentic images. Current research emphasizes the sensitivity, with the specificity overlooked. In this paper we address both aspects by multi-view feature learning and multi-scale supervision. By exploiting noise distribution and boundary artifact surrounding tampered regions, the former aims to learn semantic-agnostic and thus more generalizable features. The latter allows us to learn from authentic images which are nontrivial to be taken into account by current semantic segmentation network based methods. Our thoughts are realized by a new network which we term MVSS-Net. Extensive experiments on five benchmark sets justify the viability of MVSS-Net for both pixel-level and image-level manipulation detection.
Security Aspects of Blockchain Technology Intended for Industrial Applications

Blockchain technology plays a significant role in the industrial development. Many industries can potentially benefit from the innovations blockchain decentralization technology and privacy protocols offer with regard to securing, data access, auditing and managing transactions within digital platforms. Blockchain is based on distributed and secure decentralized protocols in which there is no single authority, and no single point of control; the data blocks are generated, added, and validated by the nodes of the network themselves. This article provides insights into the current developments within blockchain technology and explores its ability to revolutionize the multiple industrial application areas such as supply chain industry, Internet of Things (IoT), healthcare, governance, finance and manufacturing. It investigates and provides insights into the security issues and threats related to the blockchain implementations by assessing the research through a systematic literature review. This article proposes possible solutions in detail for enhancing the security of the blockchain for industrial applications along with significant directions for future explorations. The study further suggests how in recent years the adoption of blockchain technology by multiple industrial sectors has gained momentum while in the finance sector it is touching new heights day by day.
FAIR: Quality-Aware Federated Learning with Precise User Incentive and Model Aggregation

Federated learning enables distributed learning in a privacy-protected manner, but two challenging reasons can affect learning performance significantly. First, mobile users are not willing to participate in learning due to computation and energy consumption. Second, with various factors (e.g., training data size/quality), the model update quality of mobile devices can vary dramatically, inclusively aggregating low-quality model updates can deteriorate the global model quality. In this paper, we propose a novel system named FAIR, i.e., Federated leArning with qualIty awaReness. FAIR integrates three major components: 1) learning quality estimation: we leverage historical learning records to estimate the user learning quality, where the record freshness is considered and the exponential forgetting function is utilized for weight assignment; 2) quality-aware incentive mechanism: within the recruiting budget, we model a reverse auction problem to encourage the participation of high-quality learning users, and the method is proved to be truthful, individually rational, and computationally efficient; and 3) model aggregation: we devise an aggregation algorithm that integrates the model quality into aggregation and filters out non-ideal model updates, to further optimize the global learning model. Based on real-world datasets and practical learning tasks, extensive experiments are carried out to demonstrate the efficacy of FAIR.
Pruning the Pilots: Deep Learning-Based Pilot Design and Channel Estimation for MIMO-OFDM Systems

With the large number of antennas and subcarriers the overhead due to pilot transmission for channel estimation can be prohibitive in wideband massive multiple-input multiple-output (MIMO) systems. This can degrade the overall spectral efficiency significantly, and as a result, curtail the potential benefits of massive MIMO. In this paper, we propose a neural network (NN)-based joint pilot design and downlink channel estimation scheme for frequency division duplex (FDD) MIMO orthogonal frequency division multiplex (OFDM) systems. The proposed NN architecture uses fully connected layers for frequency-aware pilot design, and outperforms linear minimum mean square error (LMMSE) estimation by exploiting inherent correlations in MIMO channel matrices utilizing convolutional NN layers. Our proposed NN architecture uses a non-local attention module to learn longer range correlations in the channel matrix to further improve the channel estimation performance.We also propose an effective pilot reduction technique by gradually pruning less significant neurons from the dense NN layers during training. This constitutes a novel application of NN pruning to reduce the pilot transmission overhead. Our pruning-based pilot reduction technique reduces the overhead by allocating pilots across subcarriers non-uniformly and exploiting the inter-frequency and inter-antenna correlations in the channel matrix efficiently through convolutional layers and attention module.
Topic-level sentiment analysis of social media data using deep learning

Due to the inception of Web 2.0 and freedom to facilitate the dissemination of information, sharing views, expressing opinions with regards to current world level events, services, products, etc. social media platforms have been mainly contributing to user-generated content. Such social media data consist of various themes discussed online and are associated with sentiments of the users. To catch up with the speed of streaming data at which it generates on social media platforms, it is crucial to detect the topics being discussed on social media platforms and analyze the sentiments of users towards those topics in an online manner to make timely decisions. Motivated by the same, this paper proposes a deep learning based topic-level sentiment analysis model. The novelty of the proposed approach is that it works at the sentence level to extract the topic using online latent semantic indexing with regularization constraint and then applies topic-level attention mechanism in long short-term memory network to perform sentiment analysis. The proposed model is unique in the sense that it supports scalable and dynamic topic modeling over streaming short text data and performs sentiment analysis at topic-level. For SemEval-2017 Task 4 Subtask B dataset as a case of in-domain topic-level sentiment analysis, average recall of 0.879 has been achieved, whereas, for out-of-domain data, average recall of 0.846, 0.824 and 0.794 has been achieved for newly developed datasets collected under the hashtags #ethereum, #bitcoin and #facebook from Twitter. To assess the performance of the model for scalability, we analyzed the model in terms of average time in milliseconds for creation of feature vectors, throughput in terms of topics detected per second and average response time in seconds to handle the sentiment analysis queries. The experimental results are significant enough to enable large scale topic modeling over streaming data and perform topic-level sentiment analysis. • Continuous streaming data from social media platform require online learning. • Proposed online latent semantic indexing model to extract the topics. • Applied topic-level attention mechanism with LSTM on streaming data. • Performed sentiment analysis at topic-level on streaming data. • Scalable to large streaming data and supports out-of-domain sentiment analysis.
TrustFed: A Framework for Fair and Trustworthy Cross-Device Federated Learning in IIoT

Cross-device federated learning (CDFL) systems enable fully decentralized training networks whereby each participating device can act as a model-owner and a model-producer. CDFL systems need to ensure fairness, trustworthiness, and high-quality model availability across all the participants in the underlying training networks. This article presents a blockchain-based framework, TrustFed, for CDFL systems to detect the model poisoning attacks, enable fair training settings, and maintain the participating devices' reputation. TrustFed provides fairness by detecting and removing the attackers from the training distributions. It uses blockchain smart contracts to maintain participating devices' reputations to compel the participants in bringing active and honest model contributions. We implemented the TrustFed using a Python-simulated federated learning framework, blockchain smart contracts, and statistical outlier detection techniques. We tested it over the large-scale industrial Internet of things dataset and multiple attack models. We found that TrustFed produces better results regarding multiple aspects compared with the conventional baseline approaches.
Blockchain technologies to address smart city and society challenges

New Information and Communications Technologies (ICT) are changing the way in which the world works. These technologies provide new tools to face the issues of contemporary society (poverty, migrations, sustainable development challenges, governance, etc.). Among them, blockchain emerge as a disruptive technology able to make things in a completely different and innovative way. They can provide solutions where before there were none. Recently, these technologies have become subject of intense research and development by industry and academia. In consequence, a large amount of interesting works is published in many areas, proposing solutions, services, frameworks and applications based on these technologies. The research focusing on the previous challenges are especially interesting in current society. However, the proposals arrive at higher speed than society can assimilate them, and sometimes they are not properly exploited. Thus, this paper provides a comprehensive overview of the blockchain technology to identify what are its key attributes that enable to successfully address the social challenges. The review has been conducted from three point of views: service-delivery, resource-management and city-administration, and has been focused on the Sustainable Development Goals of the United Nations, in order to cover the full agenda of current social issues. The contributions of this work are an exhaustive compilation of the existing research on blockchain technology to tackle the social challenges from several angles, and the identification of the key features and the conditioning factors of the practical applicability of the technology.
Image Restoration via Reconciliation of Group Sparsity and Low-Rank Models

Image nonlocal self-similarity (NSS) property has been widely exploited via various sparsity models such as joint sparsity (JS) and group sparse coding (GSC). However, the existing NSS-based sparsity models are either too restrictive, e.g., JS enforces the sparse codes to share the same support, or too general, e.g., GSC imposes only plain sparsity on the group coefficients, which limit their effectiveness for modeling real images. In this paper, we propose a novel NSS-based sparsity model, namely, low-rank regularized group sparse coding (LR-GSC), to bridge the gap between the popular GSC and JS. The proposed LR-GSC model simultaneously exploits the sparsity and low-rankness of the dictionary-domain coefficients for each group of similar patches. An alternating minimization with an adaptive adjusted parameter strategy is developed to solve the proposed optimization problem for different image restoration tasks, including image denoising, image deblocking, image inpainting, and image compressive sensing. Extensive experimental results demonstrate that the proposed LR-GSC algorithm outperforms many popular or state-of-the-art methods in terms of objective and perceptual metrics.
Uncertainty-Aware Unsupervised Domain Adaptation in Object Detection

Unsupervised domain adaptive object detection aims to adapt detectors from a labelled source domain to an unlabelled target domain. Most existing works take a two-stage strategy that first generates region proposals and then detects objects of interest, where adversarial learning is widely adopted to mitigate the inter-domain discrepancy in both stages. However, adversarial learning may impair the alignment of well-aligned samples as it merely aligns the global distributions across domains. To address this issue, we design an uncertainty-aware domain adaptation network (UaDAN) that introduces conditional adversarial learning to align well-aligned and poorly-aligned samples separately in different manners. Specifically, we design an uncertainty metric that assesses the alignment of each sample and adjusts the strength of adversarial learning for well-aligned and poorly-aligned samples adaptively. In addition, we exploit the uncertainty metric to achieve curriculum learning that first performs easier image-level alignment and then more difficult instance-level alignment progressively. Extensive experiments over four challenging domain adaptive object detection datasets show that UaDAN achieves superior performance as compared with state-of-the-art methods.
Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT , which introduces a novel kaleido strategy for fashion cross-modality representations from transformers. In contrast to random masking strategy of recent VL models, we design alignment guided masking to jointly focus more on image-text semantic relations. To this end, we carry out five novel tasks, i.e., rotation, jigsaw, camouflage, grey-to-color, and blank-to-color for self-supervised VL pre-training at patches of different scale. Kaleido-BERT is conceptually simple and easy to extend to the existing BERT framework, it attains state-of-the-art results by large margins on four downstream tasks, including text retrieval (R@1: 4.03% absolute improvement), image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the efficiency of Kaleido-BERT on a wide range of e-commerical websites, demonstrating its broader potential in real-world applications.
ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation

Semantic segmentation models gain robustness against poor lighting conditions by virtue of complementary information from visible (RGB) and thermal images. Despite its importance, most existing RGB-T semantic segmentation models perform primitive fusion strategies, such as concatenation, element-wise summation and weighted summation, to fuse features from different modalities. These strategies, unfortunately, overlook the modality differences due to different imaging mechanisms, so that they suffer from the reduced discriminability of the fused features. To address such an issue, we propose, for the first time, the strategy of bridging-then-fusing, where the innovation lies in a novel Adaptive-weighted Bi-directional Modality Difference Reduction Network (ABMDRNet). Concretely, a Modality Difference Reduction and Fusion (MDRF) subnetwork is designed, which first employs a bi-directional image-to-image translation based method to reduce the modality differences between RGB features and thermal features, and then adaptively selects those discriminative multi-modality features for RGB-T semantic segmentation in a channel-wise weighted fusion way. Furthermore, considering the importance of contextual information in semantic segmentation, a Multi-Scale Spatial Context (MSC) module and a Multi-Scale Channel Context (MCC) module are proposed to exploit the interactions among multi-scale contextual information of cross-modality features together with their long-range dependencies along spatial and channel dimensions, respectively. Comprehensive experiments on MFNet dataset demonstrate that our method achieves new state-of-the-art results.
Learning Statistical Texture for Semantic Segmentation

Existing semantic segmentation works mainly focus on learning the contextual information in high-level semantic features with CNNs. In order to maintain a precise boundary, low-level texture features are directly skip-connected into the deeper layers. Nevertheless, texture features are not only about local structure, but also include global statistical knowledge of the input image. In this paper, we fully take advantages of the low-level texture features and propose a novel Statistical Texture Learning Network (STL-Net) for semantic segmentation. For the first time, STL-Net analyzes the distribution of low level information and efficiently utilizes them for the task. Specifically, a novel Quantization and Counting Operator (QCO) is designed to describe the texture information in a statistical manner. Based on QCO, two modules are introduced: (1) Texture Enhance Module (TEM), to capture texture-related information and enhance the texture details; (2) Pyramid Texture Feature Extraction Module (PTFEM), to effectively extract the statistical texture features from multiple scales. Through extensive experiments, we show that the proposed STL-Net achieves state-of-the-art performance on three semantic segmentation benchmarks: Cityscapes, PASCAL Context and ADE20K.
Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking

Data association across frames is at the core of Multiple Object Tracking (MOT) task. This problem is usually solved by a traditional graph-based optimization or directly learned via deep learning. Despite their popularity, we find some points worth studying in current paradigm: 1) Existing methods mostly ignore the context information among tracklets and intra-frame detections, which makes the tracker hard to survive in challenging cases like severe occlusion. 2) The end-to-end association methods solely rely on the data fitting power of deep neural networks, while they hardly utilize the advantage of optimization-based assignment methods. 3) The graph-based optimization methods mostly utilize a separate neural network to extract features, which brings the inconsistency between training and inference. Therefore, in this paper we propose a novel learnable graph matching method to address these issues. Briefly speaking, we model the relationships between tracklets and the intra-frame detections as a general undirected graph. Then the association problem turns into a general graph matching between tracklet graph and detection graph. Furthermore, to make the optimization end-to-end differentiable, we relax the original graph matching into continuous quadratic programming and then incorporate the training of it into a deep graph network with the help of the implicit function theorem. Lastly, our method GMTracker, achieves state-of-the-art performance on several standard MOT datasets. Our code is available at https://github.com/jiaweihe1996/GMTracker.
DW-GAN: A Discrete Wavelet Transform GAN for NonHomogeneous Dehazing

Hazy images are often subject to color distortion, blurring, and other visible quality degradation. Some existing CNN-based methods have great performance on removing homogeneous haze, but they are not robust in non-homogeneous case. The reasons are mainly in two folds. Firstly, due to the complicated haze distribution, texture details are easy to be lost during the dehazing process. Secondly, since the training pairs are hard to be collected, training on limited data can easily lead to over-fitting problem. To tackle these two issues, we introduce a novel dehazing network using 2D discrete wavelet transform, namely DW-GAN. Specifically, we propose a two-branch network to deal with the aforementioned problems. By utilizing wavelet transform in DWT branch, our proposed method can retain more high-frequency knowledge in feature maps. In order to prevent over-fitting, ImageNet pre-trained Res2Net is adopted in the knowledge adaptation branch. Owing to the robust feature representations of ImageNet pre-training, the generalization ability of our network is improved dramatically. Finally, a patch-based discriminator is used to reduce artifacts of the restored images. Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-arts quantitatively and qualitatively. The source code is available at https://github.com/liuh127/DW-GAN-Dehazing.
The Spatially-Correlative Loss for Various Image Translation Tasks

We propose a novel spatially-correlative loss that is simple, efficient and yet effective for preserving scene structure consistency while supporting large appearance changes during unpaired image-to-image (I2I) translation. Previous methods attempt this by using pixel-level cycle-consistency or feature-level matching losses, but the domain-specific nature of these losses hinder translation across large domain gaps. To address this, we exploit the spatial patterns of self-similarity as a means of defining scene structure. Our spatially-correlative loss is geared towards only capturing spatial relationships within an image rather than domain appearance. We also introduce a new self-supervised learning method to explicitly learn spatially-correlative maps for each specific translation task. We show distinct improvement over baseline models in all three modes of unpaired I2I translation: single-modal, multi-modal, and even single-image translation. This new loss can easily be integrated into existing network architectures and thus allows wide applicability. The code is available at https://github.com/lyndonzheng/F-LSeSim.