Reviewerly/exHarmony huggingface dataset:
C2AE: Class Conditioned Auto-Encoder for Open-Set Recognition

Models trained for classification often assume that all testing classes are known while training. As a result, when presented with an unknown class during testing, such closed-set assumption forces the model to classify it as one of the known classes. However, in a real world scenario, classification models are likely to encounter such examples. Hence, identifying those examples as unknown becomes critical to model performance. A potential solution to overcome this problem lies in a class of learning problems known as open-set recognition. It refers to the problem of identifying the unknown classes during testing, while maintaining performance on the known classes. In this paper, we propose an open-set recognition algorithm using class conditioned auto-encoders with novel training and testing methodologies. In this method, training procedure is divided in two sub-tasks, 1. closed-set classification and, 2. open-set identification (i.e. identifying a class as known or unknown). Encoder learns the first task following the closed-set classification training pipeline, whereas decoder learns the second task by reconstructing conditioned on class identity. Furthermore, we model reconstruction errors using the Extreme Value Theory of statistical modeling to find the threshold for identifying known/unknown class samples. Experiments performed on multiple image classification datasets show that the proposed method performs significantly better than the state of the art methods. The source code is available at: github.com/otkupjnoz/c2ae.
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks

This work proposes a novel attentive graph neural network (AGNN) for zero-shot video object segmentation (ZVOS). The suggested AGNN recasts this task as a process of iterative information fusion over video graphs. Specifically, AGNN builds a fully connected graph to efficiently represent frames as nodes, and relations between arbitrary frame pairs as edges. The underlying pair-wise relations are described by a differentiable attention mechanism. Through parametric message passing, AGNN is able to efficiently capture and mine much richer and higher-order relations between video frames, thus enabling a more complete understanding of video content and more accurate foreground estimation. Experimental results on three video segmentation datasets show that AGNN sets a new state-of-the-art in each case. To further demonstrate the generalizability of our framework, we extend AGNN to an additional task: image object co-segmentation (IOCS). We perform experiments on two famous IOCS datasets and observe again the superiority of our AGNN model. The extensive experiments verify that AGNN is able to learn the underlying semantic/appearance relationships among video frames or related images, and discover the common objects.
Accelerating Federated Learning via Momentum Gradient Descent

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL.
Blockchain Meets Cloud Computing: A Survey

Blockchain technology has been deemed to be an ideal choice for strengthening existing computing systems in varied manners. As one of the network-enabled technologies, cloud computing has been broadly adopted in the industry through numerous cloud service models. Fusing blockchain technology with existing cloud systems has a great potential in both functionality/performance enhancement and security/privacy improvement. The question remains on how blockchain technology inserts into current deployed cloud solutions and enables the reengineering of cloud datacenter. This survey addresses this issue and investigates recent efforts in the technical fusion of blockchain and clouds. Three technical dimensions roughly are covered in this work. First, we concern the service model and review an emerging cloud-relevant blockchain service model, Blockchain-as-a-Service (BaaS); second, security is considered a key technical dimension in this work and both access control and searchable encryption schemes are assessed; finally, we examine the performance of cloud datacenter with supports/participance of blockchain from hardware and software perspectives. Main findings of this survey will be theoretical supports for future reference of blockchain-enabled reengineering of cloud datacenter.
Learning Selective Self-Mutual Attention for RGB-D Saliency Detection

Saliency detection on RGB-D images is receiving more and more research interests recently. Previous models adopt the early fusion or the result fusion scheme to fuse the input RGB and depth data or their saliency maps, which incur the problem of distribution gap or information loss. Some other models use the feature fusion scheme but are limited by the linear feature fusion methods. In this paper, we propose to fuse attention learned in both modalities. Inspired by the Non-local model, we integrate the self-attention and each other's attention to propagate long-range contextual dependencies, thus incorporating multi-modal information to learn attention and propagate contexts more accurately. Considering the reliability of the other modality's attention, we further propose a selection attention to weight the newly added attention term. We embed the proposed attention module in a two-stream CNN for RGB-D saliency detection. Furthermore, we also propose a residual fusion module to fuse the depth decoder features into the RGB stream. Experimental results on seven benchmark datasets demonstrate the effectiveness of the proposed model components and our final saliency model. Our code and saliency maps are available at https://github.com/nnizhang/S2MA.
Intelligent Fault Diagnosis by Fusing Domain Adversarial Training and Maximum Mean Discrepancy via Ensemble Learning

Nowadays, the industrial Internet of Things (IIoT) has been successfully utilized in smart manufacturing. The massive amount of data in IIoT promote the development of deep learning-based health monitoring for industrial equipment. Since monitoring data for mechanical fault diagnosis collected on different working conditions or equipment have domain mismatch, models trained with training data may not work in practical applications. Therefore, it is essential to study fault diagnosis methods with domain adaptation ability. In this article, we propose an intelligent fault diagnosis method based on an improved domain adaptation method. Specifically, two feature extractors concerning feature space distance and domain mismatch are trained using maximum mean discrepancy and domain adversarial training respectively to enhance feature representation. Since separate classifiers are trained for feature extractors, ensemble learning is further utilized to obtain final results. Experimental results indicate that the proposed method is effective and applicable in diagnosing faults with domain mismatch.
Physical reservoir computing—an introductory perspective

Understanding the fundamental relationships between physics and its information-processing capability has been an active research topic for many years. Physical reservoir computing is a recently introduced framework that allows one to exploit the complex dynamics of physical systems as information-processing devices. This framework is particularly suited for edge computing devices, in which information processing is incorporated at the edge (e.g., into sensors) in a decentralized manner to reduce the adaptation delay caused by data transmission overhead. This paper aims to illustrate the potentials of the framework using examples from soft robotics and to provide a concise overview focusing on the basic motivations for introducing it, which stem from a number of fields, including machine learning, nonlinear dynamical systems, biological science, materials science, and physics.
Prediction of solar energy guided by pearson correlation using machine learning

Solar energy forecasting represents a key element in increasing the competitiveness of solar power plants in the energy market and reducing the dependence on fossil fuels in economic and social development. This paper presents an approach for predicting solar energy, based on machine and deep learning techniques. The relevance of the studied models was evaluated for real-time and short-term solar energy forecasting to ensure optimized management and security requirements in this field while using an integral solution based on a single tool and an appropriate predictive model. The datasets we used in this study, represent data from 2016 to 2018 and are related to Errachidia which is a semi-desert climate province in Morocco. Pearson correlation coefficient was deployed to identify the most relevant meteorological inputs from which the models should learn. RF and ANN have provided high accuracies against LR and SVR, which have reported very significant errors. ANN has shown good performance for both real-time and short-term predictions. The key findings were compared with Pirapora in Brazil, which is a tropical climate region, to show the quality and reproducibility of the study.
Align Deep Features for Oriented Object Detection

The past decade has witnessed significant progress on detecting objects in aerial images that are often distributed with large-scale variations and arbitrary orientations. However, most of existing methods rely on heuristically defined anchors with different scales, angles, and aspect ratios, and usually suffer from severe misalignment between anchor boxes (ABs) and axis-aligned convolutional features, which lead to the common inconsistency between the classification score and localization accuracy. To address this issue, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">single-shot alignment network</i> (S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> A-Net) consisting of two modules: a feature alignment module (FAM) and an oriented detection module (ODM). The FAM can generate high-quality anchors with an anchor refinement network and adaptively align the convolutional features according to the ABs with a novel alignment convolution. The ODM first adopts active rotating filters to encode the orientation information and then produces orientation-sensitive and orientation-invariant features to alleviate the inconsistency between classification score and localization accuracy. Besides, we further explore the approach to detect objects in large-size images, which leads to a better trade-off between speed and accuracy. Extensive experiments demonstrate that our method can achieve the state-of-the-art performance on two commonly used aerial objects’ data sets (i.e., DOTA and HRSC2016) while keeping high efficiency.
Graph Representation Learning

Graph-structured data is ubiquitous throughout the natural and social sciences, from telecommunication networks to quantum chemistry. Building relational inductive biases into deep learning architectures is crucial for creating systems that can learn, reason, and generalize from this kind of data. Recent years have seen a surge in research on graph representation learning, including techniques for deep graph embeddings, generalizations of convolutional neural networks to graph-structured data, and neural message-passing approaches inspired by belief propagation. These advances in graph representation learning have led to new state-of-the-art results in numerous domains, including chemical synthesis, 3D vision, recommender systems, question answering, and social network analysis.
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

Current state-of-the-art instance segmentation methods are not suited for real-time applications like autonomous driving, which require fast execution times at high accuracy. Although the currently dominant proposal-based methods have high accuracy, they are slow and generate masks at a fixed and low resolution. Proposal-free methods, by contrast, can generate masks at high resolution and are often faster, but fail to reach the same accuracy as the proposal-based methods. In this work we propose a new clustering loss function for proposal-free instance segmentation. The loss function pulls the spatial embeddings of pixels belonging to the same instance together and jointly learns an instance-specific clustering bandwidth, maximizing the intersection-over-union of the resulting instance mask. When combined with a fast architecture, the network can perform instance segmentation in real-time while maintaining a high accuracy. We evaluate our method on the challenging Cityscapes benchmark and achieve top results (5% improvement over Mask R-CNN) at more than 10 fps on 2MP images.
An integrated silicon photonic chip platform for continuous-variable quantum key distribution

Quantum key distribution (QKD) is a quantum communication technology that promises unconditional communication security. High-performance and cost-effective QKD systems are essential for the establishment of quantum communication networks1–3. By integrating all the optical components (except the laser source) on a silicon photonic chip, we have realized a stable, miniaturized and low-cost system for continuous-variable QKD (CV-QKD) that is compatible with the existing fibre optical communication infrastructure4. Here, the integrated silicon photonic chip is demonstrated for CV-QKD. It implements the widely studied Gaussian-modulated coherent state protocol that encodes continuous distributed information on the quadrature of laser light5,6. Our proof-of-principle chip-based CV-QKD system is capable of producing a secret key rate of 0.14 kbps (under collective attack) over a simulated distance of 100 km in fibre, offering new possibilities for low-cost, scalable and portable quantum networks. A sender and a receiver for continuous-variable quantum key distribution are packed onto separate silicon photonic chips. By using an external 1,550-nm laser, a secret key rate of 0.14 kbps is transmitted over a simulated distance of 100 km in fibre.
RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation

This paper addresses the problem of 3D human pose estimation from single images. While for a long time human skeletons were parameterized and fitted to the observation by satisfying a reprojection error, nowadays researchers directly use neural networks to infer the 3D pose from the observations. However, most of these approaches ignore the fact that a reprojection constraint has to be satisfied and are sensitive to overfitting. We tackle the overfitting problem by ignoring 2D to 3D correspondences. This efficiently avoids a simple memorization of the training data and allows for a weakly supervised training. One part of the proposed reprojection network (RepNet) learns a mapping from a distribution of 2D poses to a distribution of 3D poses using an adversarial training approach. Another part of the network estimates the camera. This allows for the definition of a network layer that performs the reprojection of the estimated 3D pose back to 2D which results in a reprojection loss function. Our experiments show that RepNet generalizes well to unknown data and outperforms state-of-the-art methods when applied to unseen data. Moreover, our implementation runs in real-time on a standard desktop PC.
A Survey on the Scalability of Blockchain Systems

In the past decade, crypto-currencies such as Bitcoin and Litecoin have developed rapidly. Blockchain as the underlying technology of these digital crypto-currencies has attracted great attention from academia and industry. Blockchain has many good features, such as trust-free, transparency, anonymity, democracy, automation, decentralization and security. Despite these promising features, scalability is still a key barrier when the blockchain technology is widely used in real business environments. In this article, we focus on the scalability issue, and provide a brief survey of recent studies on scalable blockchain systems. We first discuss the scalability issue from the perspectives of throughput, storage and networking. Then, existing enabling technologies for scalable blockchain systems are presented. We also discuss some research challenges and future research directions for scalable blockchain systems.
Blockchain platform for industrial healthcare: Vision and future opportunities

Medical data has become an essential element for industrial healthcare. The growth in medical data is accompanied by the need to process it in a secure manner. As the infrastructure of this industry consists of connected devices and software applications that communicate with other IT systems, the industrial healthcare market will be greatly impacted by the use of blockchain and the Internet of Things (IoT). These technologies will improve processing efficiency, the creation of business opportunities, requirement regulation, information security, and transparency. While sharing electronic health records can assist in improving diagnosis accuracy, privacy and security preservation are imperative. In the network of IoT devices, that exchange involves sensitive medical data; patient monitoring has to be totally secured against privacy risks and attacks. Delays in treatment progress and emergency treatments could also result in medical data security and confidentiality violations. Applying blockchain technology to the healthcare industry could improve information security management; healthcare data could be analyzed and communicated while preserving the privacy and security of the data. Here we critically look at how these two key technologies (blockchain and IoT) – especially blockchain – will impact the healthcare industry.
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Vision-based sign language recognition aims at helping the deaf people to communicate with others. However, most existing sign language datasets are limited to a small number of words. Due to the limited vocabulary size, models learned from those datasets cannot be applied in practice. In this paper, we introduce a new large-scale Word-Level American Sign Language (WLASL) video dataset, containing more than 2000 words performed by over 100 signers. This dataset will be made publicly available to the research community. To our knowledge,it is by far the largest public ASL dataset to facilitate word-level sign recognition research. Based on this new large-scale dataset, we are able to experiment with several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios. Specifically we implement and compare two different models,i.e., (i) holistic visual appearance based approach, and (ii) 2D human pose based approach. Both models are valuable baselines that will benefit the community for method benchmarking. Moreover, we also propose a novel pose-based temporal graph convolution networks (Pose-TGCN) that model spatial and temporal dependencies in human pose trajectories simultaneously, which has further boosted the performance of the pose-based method. Our results show that pose-based and appearance-based models achieve comparable performances up to 62.63% at top-10 accuracy on 2,000 words/glosses, demonstrating the validity and challenges of our dataset. Our dataset and baseline deep models are available at https://dxli94.github.io/WLASL/.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like “Rinse off a mug and place it in the coffee maker.” and low-level language instructions like “Walk to the coffee maker on the right.” ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision- and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection

We present our on-going effort of constructing a large- scale benchmark for face forgery detection. The first version of this benchmark, DeeperForensics-1.0, represents the largest face forgery detection dataset by far, with 60, 000 videos constituted by a total of 17.6 million frames, 10 times larger than existing datasets of the same kind. Extensive real-world perturbations are applied to obtain a more challenging benchmark of larger scale and higher diversity. All source videos in DeeperForensics-1.0 are carefully collected, and fake videos are generated by a newly proposed end-to-end face swapping framework. The quality of generated videos outperforms those in existing datasets, validated by user studies. The benchmark features a hidden test set, which contains manipulated videos achieving high deceptive scores in human evaluations. We further contribute a comprehensive study that evaluates five representative detection baselines and make a thorough analysis of different settings.
MLQA: Evaluating Cross-lingual Extractive Question Answering

Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets.Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making building QA systems that work well in other languages challenging.In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress.We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. 1MLQA contains QA instances in 7 languages, English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese.MLQA has over 12K instances in English and 5K in each other language, with each instance parallel between 4 languages on average.We evaluate stateof-the-art cross-lingual models and machinetranslation-based baselines on MLQA.In all cases, transfer results are significantly behind training-language performance.