Reviewerly/exHarmony huggingface dataset:
Remote Sensing Image Super-Resolution Using Second-Order Multi-Scale Networks

Remotely sensed images, especially in urban areas, have highly complex spatial distribution, since the ground objects have diverse ranges of sizes and shapes. This largely increases the difficulty of super-resolution (SR) tasks. Current deep convolutional neural network (CNN)-based SR methods often show limited performance when coping with complicated images. This article develops a second-order multi-scale super-resolution network (SMSR) to explore reconstruction tasks for difficult cases. Specifically, we propose a single-path feature reuse which cleverly captures multi-scale feature information through aggregating the features learned at different depths of a single path. Further, we present a second-order learning mechanism, which double reuses small-difference and large-difference features at local and global levels, makes use of the learned multi-scale information at maximum. The proposed methods achieve multi-scale learning using small-size convolution only, resulting in a lightweight and high-performance SR network. Experimental results show the superiority of our SMSR over state-of-the-art methods in super-resolving complicated image patterns. The effectiveness of SMSR is also demonstrated through its support to object recognition task.
Active Learning Based Federated Learning for Waste and Natural Disaster Image Classification

The feasibility of Federated Learning (FL) is highly dependent on the training and inference capabilities of local models, which are subject to the availability of meaningful and annotated data. The availability of such data is in turn contingent on the tedious and time-consuming annotation job that typically requires the manual analysis of training samples. Active Learning (AL) provides an alternative solution allowing a Machine Learning (ML) model to automatically choose and label the data from which it learns without involving manual inspection of each training sample. In this work, we explore how FL can benefit from unlabelled data available at each participating client using AL. To this aim, we propose an AL-based FL framework by employing and evaluating several AL methods in two different application domains. Through an extensive experimentation setup, we show that AL is equally useful in federated and centralized learning by achieving comparable results with manually labeled data using fewer samples without involving human annotators in collecting training data. We also demonstrated that the proposed method is dataset/application independent by evaluating the proposed method in two interesting applications, namely natural disaster analysis and waste classification, having different properties and challenges. Promising results are obtained on both applications resulting in comparable results against the best-case scenario where each sample is manually analyzed and annotated (Baseline 1), and improvement of 3.1% and 4% with best methods respectively over the training sets with irrelevant images on natural disaster and waste classification datasets (Baseline 2).
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis

The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language. Recent multimodal learning with strong performances on human-centric tasks such as sentiment analysis and emotion recognition are often black-box, with very limited interpretability. In this paper we propose, which dynamically adjusts weights between input modalities and output representations differently for each input sample. Multimodal routing can identify relative importance of both individual modalities and cross-modality factors. Moreover, the weight assignment by routing allows us to interpret modality-prediction relationships not only globally (i.e. general trends over the whole dataset), but also locally for each single input sample, meanwhile keeping competitive performance compared to state-of-the-art methods.
Exploring Balanced Feature Spaces for Representation Learning

Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artificially balanced datasets (e.g., ImageNet). It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t. the classes. Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one. Our findings are quite intriguing. Different from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. This motivates us to explore the balanced feature spaces learned by contrastive learning, where the feature representations present similar linear separability w.r.t. all the classes. Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings. Inspired by these insights, we develop a novel representation learning method, called k-positive contrastive learning. It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced. Extensive experiments demonstrate its superiority on multiple recognition tasks. Remarkably, it achieves new state-of-the-art on challenging long-tailed recognition benchmarks. Code and models will be released.
Resource optimization model using novel extreme learning machine with t-distributed stochastic neighbor embedding: Application to complex industrial processes

Energy-saving and emission reduction are crucial to modern society, but the previous resource optimization methods based on traditional neural networks are complex and low accuracy. Therefore, this paper presented a novel extreme learning machine (ELM) method based on t-distributed stochastic neighbor embedding (t-SNE) to optimize the energy and reduce the carbon emission. In terms of mapping high-dimensional production data to low-dimensional space, the t-SNE can deal with the major factors affecting the energy efficiency, which are taken as inputs of the ELM. Then the resource optimization model for energy-saving is obtained based on the ELM to predict the output and achieve the optimal configuration. Finally, the proposed method is used to establish the resource optimization model for the ethylene and purified terephthalic acid (PTA) production processes in complex industrial processes. The experimental results demonstrate that the proposed model can improve the prediction accuracy of resource optimization models of complex industrial processes, and realize energy-saving and carbon emissions reduction.
Generalized Domain Conditioned Adaptation Network

Domain adaptation (DA) attempts to transfer knowledge learned in the labeled source domain to the unlabeled but related target domain without requiring large amounts of target supervision. Recent advances in DA mainly proceed by aligning the source and target distributions. Despite the significant success, the adaptation performance still degrades accordingly when the source and target domains encounter a large distribution discrepancy. We consider this limitation may attribute to the insufficient exploration of domain-specialized features because most studies merely concentrate on domain-general feature learning in task-specific layers and integrate totally-shared convolutional networks (convnets) to generate common features for both domains. In this paper, we relax the completely-shared convnets assumption adopted by previous DA methods and propose Domain Conditioned Adaptation Network (DCAN), which introduces domain conditioned channel attention module with a multi-path structure to separately excite channel activation for each domain. Such a partially-shared convnets module allows domain-specialized features in low-level to be explored appropriately. Further, given the knowledge transferability varying along with convolutional layers, we develop Generalized Domain Conditioned Adaptation Network (GDCAN) to automatically determine whether domain channel activations should be separately modeled in each attention module. Afterward, the critical domain-specialized knowledge could be adaptively extracted according to the domain statistic gaps. As far as we know, this is the first work to explore the domain-wise convolutional channel activations separately for deep DA networks. Additionally, to effectively match high-level feature distributions across domains, we consider deploying feature adaptation blocks after task-specific layers, which can explicitly mitigate the domain discrepancy. Extensive experiments on four cross-domain benchmarks, including DomainNet, Office-Home, Office-31, and ImageCLEF, demonstrate the proposed approaches outperform the existing methods by a large margin, especially on the large-scale challenging dataset. The code and models are available at https://github.com/BIT-DA/GDCAN.
Cross-Cluster Federated Learning and Blockchain for Internet of Medical Things

Federated learning (FL) has been gaining popularity as a way to provide privacy-preserving data sharing for the Internet of Medical Things (IoMT). As a complementary, blockchain technology is used in recent literature to make FL secure. However, existing blockchain-based FL (BFL) solutions do not perform well when data in a BFL cluster are sparse. A direct solution is to collect as many devices as possible to establish a large BFL cluster. However, these devices may locate in geographically distant areas and be separated by great distance, which further results in high communication latency. The high latency will lead to BFL’s low system efficiency due to frequent communications in the blockchain consensus. In this article, we propose that the large cluster should be divided into multiple smaller clusters, each in its own geographical area and organized with a BFL. In this context, we propose CFL, a cross-cluster FL system facilitated by the cross-chain technique. CFL connects multiple BFL clusters, where only a few aggregated updates are transmitted over long distances across clusters, thus improving the system efficiency. The design of CFL focuses on a cross-chain consensus protocol, which guarantees the model updates to be exchanged securely across clusters. We carry out extensive experiments to evaluate CFL in comparison with BFL, and show both CFL’s feasibility and efficiency.
Open World Compositional Zero-Shot Learning

Compositional Zero-Shot learning (CZSL) requires to recognize state-object compositions unseen during training. In this work, instead of assuming prior knowledge about the unseen compositions, we operate in the open world setting, where the search space includes a large number of unseen compositions some of which might be unfeasible. In this setting, we start from the cosine similarity between visual features and compositional embeddings. After estimating the feasibility score of each composition, we use these scores to either directly mask the output space or as a margin for the cosine similarity between visual features and compositional embeddings during training. Our experiments on two standard CZSL benchmarks show that all the methods suffer severe performance degradation when applied in the open world setting. While our simple CZSL model achieves state-of-the-art performances in the closed world scenario, our feasibility scores boost the performance of our approach in the open world setting, clearly outperforming the previous state of the art. Code is available at: https://github.com/ExplainableML/czsl.
Blockchain-based IoT device identification and management in 5G smart grid

Abstract This work investigates the unified coding and identification of smart grid IoT devices, as more and more IoT devices in smart grid need to be managed and controlled. We combine blockchain technology with 5G MEC to realize the connection of massive power IoT devices at the edge of 5G network. Due to blockchain’s distributed storage and credibility, it is used to identify and register IoT devices in smart grid, ensuring the reliability and accuracy of smart grid IoT devices management. In this paper, we propose a hybrid blockchain mechanism based on 5G MEC smart grid, where both public blockchain and private blockchain are deployed on the MEC gateway/server. To facilitate the data searching and extracting, we endeavor to build a blockchain explorer indexed by IoT device identifier. After that, we study the typical consensus algorithms in the blockchain such as PoW, PoS, DPoS, PDFT, and discuss their feasibility in the hybrid blockchain. Finally, we analyzed and compared the performance of different consensus algorithms from the perspective of average computing time and average time to agreement.
Integrated energy systems of data centers and smart grids: State-of-the-art and future opportunities

Cloud computing platforms are critical cyber infrastructures in modern society. As the backbone of cloud systems, data centers act as large energy consumers in today’s power grids. The integration of on-site renewable energy sources and energy storage systems further transforms data centers to be energy prosumers (producers-and-consumers). As a result, optimizing data centers’ energy production and consumption, and exploiting their potential of actively engaging in the external grid’s planning, operation, and control has been drawing increasing attention in the last few years. This paper conducts a comprehensive review of the state-of-the-art research efforts on integrated energy systems of data centers and smart grids. A taxonomy of such integration scenarios is provided. Consequently, this paper identifies several future application scenarios of integrating data centers and smart grids, which serves as a roadmap towards future research. This article is expected to provide a useful reference for researchers and engineers in the areas of energy systems and cloud computing.
TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting

Temporal knowledge graph (TKG) reasoning is a crucial task that has gained increasing research interest in recent years.Most existing methods focus on reasoning at past timestamps to complete the missing facts, and there are only a few works of reasoning on known TKGs to forecast future facts.Compared with the completion task, the forecasting task is more difficult and faces two main challenges:(1) how to effectively model the time information to handle future timestamps?(2) how to make inductive inference to handle previously unseen entities that emerge over time?To address these challenges, we propose the first reinforcement learning method for forecasting.Specifically, the agent travels on historical knowledge graph snapshots to search for the answer.Our method defines a relative time encoding function to capture the timespan information, and we design a novel time-shaped reward based on Dirichlet distribution to guide the model learning.Furthermore, we propose a novel representation method for unseen entities to improve the inductive inference ability of the model.We evaluate our method for this link prediction task at future timestamps.Extensive experiments on four benchmark datasets demonstrate substantial performance improvement meanwhile with higher explainability, less calculation, and fewer parameters when compared with existing stateof-the-art methods.
Skeleton-based human activity recognition using ConvLSTM and guided feature learning

Abstract Human activity recognition aims to determine actions performed by a human in an image or video. Examples of human activity include standing, running, sitting, sleeping, etc . These activities may involve intricate motion patterns and undesired events such as falling. This paper proposes a novel deep convolutional long short-term memory (ConvLSTM) network for skeletal-based activity recognition and fall detection. The proposed ConvLSTM network is a sequential fusion of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and fully connected layers. The acquisition system applies human detection and pose estimation to pre-calculate skeleton coordinates from the image/video sequence. The ConvLSTM model uses the raw skeleton coordinates along with their characteristic geometrical and kinematic features to construct the novel guided features. The geometrical and kinematic features are built upon raw skeleton coordinates using relative joint position values, differences between joints, spherical joint angles between selected joints, and their angular velocities. The novel spatiotemporal-guided features are obtained using a trained multi-player CNN-LSTM combination. Classification head including fully connected layers is subsequently applied. The proposed model has been evaluated on the KinectHAR dataset having 130,000 samples with 81 attribute values, collected with the help of a Kinect (v2) sensor. Experimental results are compared against the performance of isolated CNNs and LSTM networks. Proposed ConvLSTM have achieved an accuracy of 98.89% that is better than CNNs and LSTMs having an accuracy of 93.89 and 92.75%, respectively. The proposed system has been tested in realtime and is found to be independent of the pose, facing of the camera, individuals, clothing, etc . The code and dataset will be made publicly available.
Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%–39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 × 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.
Semantic Concentration for Domain Adaptation

Domain adaptation (DA) paves the way for label annotation and dataset bias issues by the knowledge transfer from a label-rich source domain to a related but unlabeled target domain. A mainstream of DA methods is to align the feature distributions of the two domains. However, the majority of them focus on the entire image features where irrelevant semantic information, e.g., the messy background, is inevitably embedded. Enforcing feature alignments in such case will negatively influence the correct matching of objects and consequently lead to the semantically negative transfer due to the confusion of irrelevant semantics. To tackle this issue, we propose Semantic Concentration for Domain Adaptation (SCDA), which encourages the model to concentrate on the most principal features via the pair-wise adversarial alignment of prediction distributions. Specifically, we train the classifier to class-wisely maximize the prediction distribution divergence of each sample pair, which enables the model to find the region with large differences among the same class of samples. Meanwhile, the feature extractor attempts to minimize that discrepancy, which suppresses the features of dissimilar regions among the same class of samples and accentuates the features of principal parts. As a general method, SCDA can be easily integrated into various DA methods as a regularizer to further boost their performance. Extensive experiments on the cross-domain benchmarks show the efficacy of SCDA.
A Deep Learning Approach for Sentiment Analysis of COVID-19 Reviews

User-generated multi-media content, such as images, text, videos, and speech, has recently become more popular on social media sites as a means for people to share their ideas and opinions. One of the most popular social media sites for providing public sentiment towards events that occurred during the COVID-19 period is Twitter. This is because Twitter posts are short and constantly being generated. This paper presents a deep learning approach for sentiment analysis of Twitter data related to COVID-19 reviews. The proposed algorithm is based on an LSTM-RNN-based network and enhanced featured weighting by attention layers. This algorithm uses an enhanced feature transformation framework via the attention mechanism. A total of four class labels (sad, joy, fear, and anger) from publicly available Twitter data posted in the Kaggle database were used in this study. Based on the use of attention layers with the existing LSTM-RNN approach, the proposed deep learning approach significantly improved the performance metrics, with an increase of 20% in accuracy and 10% to 12% in precision but only 12–13% in recall as compared with the current approaches. Out of a total of 179,108 COVID-19-related tweets, tweets with positive, neutral, and negative sentiments were found to account for 45%, 30%, and 25%, respectively. This shows that the proposed deep learning approach is efficient and practical and can be easily implemented for sentiment classification of COVID-19 reviews.
Deep Learning to Evaluate Secure RSA Implementations

This paper presents the results of several successful profiled side-channel attacks against a secure implementation of the RSA algorithm. The implementation was running on a ARM Core SC 100 completed with a certified EAL4+ arithmetic co-processor. The analyses have been conducted by three experts’ teams, each working on a specific attack path and exploiting information extracted either from the electromagnetic emanation or from the power consumption. A particular attention is paid to the description of all the steps that are usually followed during a security evaluation by a laboratory, including the acquisitions and the observations preprocessing which are practical issues usually put aside in the literature. Remarkably, the profiling portability issue is also taken into account and different device samples are involved for the profiling and testing phases. Among other aspects, this paper shows the high potential of deep learning attacks against secure implementations of RSA and raises the need for dedicated countermeasures.