Reviewerly/exHarmony huggingface dataset:
Dual-Level Knowledge Distillation via Knowledge Alignment and Correlation

Knowledge distillation (KD) has become a widely used technique for model compression and knowledge transfer. We find that the standard KD method performs the knowledge alignment on an individual sample indirectly via class prototypes and neglects the structural knowledge between different samples, namely, knowledge correlation. Although recent contrastive learning-based distillation methods can be decomposed into knowledge alignment and correlation, their correlation objectives undesirably push apart representations of samples from the same class, leading to inferior distillation results. To improve the distillation performance, in this work, we propose a novel knowledge correlation objective and introduce the dual-level knowledge distillation (DLKD), which explicitly combines knowledge alignment and correlation together instead of using one single contrastive objective. We show that both knowledge alignment and correlation are necessary to improve the distillation performance. In particular, knowledge correlation can serve as an effective regularization to learn generalized representations. The proposed DLKD is task-agnostic and model-agnostic, and enables effective knowledge transfer from supervised or self-supervised pretrained teachers to students. Experiments show that DLKD outperforms other state-of-the-art methods on a large number of experimental settings including: 1) pretraining strategies; 2) network architectures; 3) datasets; and 4) tasks.
IoV environment exploring coordination: A federated learning approach

Exploring open fields with coordinated unmanned vehicles is popular in academia and industries, and one of the most impressive applicable approaches is the Internet of Vehicles (IoV). The IoV connects vehicles, road infrastructures, and communication facilities to provide sustainable solutions for task explorations. However, the coordination of acquiring information from others may risk data privacy. To this end, sharing high-quality experiences instead of raw data has become an urgent demand. This paper employs a Deep Reinforcement Learning (DRL) method to let IoV generate training data with prioritized experience and states, which can support the IoV to explore the environment more efficiently. Moreover, a Federated Learning (FL) experience sharing model is established to guarantee the vehicles' privacy. The numerical results show that the proposed method presents a better successful sharing rate and a more stable convergence within the comparison of fundamental methods. The experiments also suggest that the proposed method could support agents with no explicit information to achieve the tasks.
FedGL: Federated graph learning framework with global self-supervision

Graph data are ubiquitous in the real world. Graph learning (GL) attempts to mine and analyze graph data so that valuable information can be discovered. Existing GL methods are designed for centralized scenarios. However, in practical scenarios, graph data are usually distributed across different organizations, resulting in isolated data silos. To address this problem, we incorporate federated learning into GL and propose a general Federated Graph Learning framework called FedGL. FedGL is capable of obtaining a high-quality global graph model while protecting data privacy by discovering the global self-supervision information during the federated training. Specifically, we propose uploading prediction results and node embeddings to the server for discovering the global pseudo labels and global pseudo graph. These are then distributed to each client to enrich the training labels and complement the graph structure respectively, thereby improving the quality of each local model. Moreover, the global self-supervision enables information of each client to flow and be shared in a privacy-preserving manner, thus alleviating the heterogeneity and utilizing the complementarity of graph data among different clients. Finally, experimental results show that FedGL significantly outperforms baselines on four widely used graph datasets.
Kernel-Based Sparse Representation Learning With Global and Local Low-Rank Label Constraint

Due to the large-scale and multiscale natures of social media data, sparse representation (SR) learning methods are widely followed. However, there are three problems associated with the existing SR methods: 1) they neglect the fact that the semantic features of data may change during iterative learning, which leads to weak semantic learning; 2) they often assume that the data are linearly separable, while the data might be nonlinear in many real-world applications; and 3) they cannot ensure the low-rank and discriminative properties of the data at the same time and might neglect the global properties of the data, leading to suboptimal solutions. To solve these problems, we propose a novel method, named kernel-based SR learning with global and local low-rank label (KSR-GL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^3$</tex-math> </inline-formula> ) constraint, which strengthens the semantic information and ensures the semantic features invariant during learning. First, we map the data into a high-dimensional feature space to learn the linear representation of samples. Second, global and local low-rank label (GL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^3$</tex-math> </inline-formula> ) constraint is used to ensure the semantic invariance, low-rankness, and discrimination of features during learning. Third, an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell_{2,1}$</tex-math> </inline-formula> is imposed to explore the sparseness of the subspace. Mathematical analyses show that GL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^3$</tex-math> </inline-formula> can retain the intrinsic properties of data during learning. By combining the above three components, a generalized power iteration (GPI) approach is applied to build the model and deal with the tricky optimization problem. By KSR-GL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ ^3$</tex-math> </inline-formula> , a sparse, low-rank, and discriminative subspace is produced from the high-dimensional and orthogonal representation of the data under the guidance of semantics, while the intrinsic properties of data are preserved. Extensive experiments on six datasets compared with five advanced algorithms demonstrate its promising prospects.
Anti-Drugs Chatbot: Chinese BERT-Based Cognitive Intent Analysis

Drug abuse has always been a severe issue, but the proportion of drug abuse and addiction is rising. According to research reports, youth are motivated to access drugs mainly due to curiosity and peer influence. Additionally, youth especially lack proper knowledge and education surrounding drug abuse. Analyzing whether potential addicts intend to access drugs is helpful in preventing drug abuse and addiction. We developed an Anti-drug Chatbot for young people on a popular online social platform. We can detect potential risks, obtain warnings from the user-entered query and provide these to professional consultants for help. In this article, we present a hierarchical system with bidirectional encoder representation from transformers (BERT) to efficiently recognize and classify a user’s intent. We use the Chinese BERT-based model to utilize contextual information to perform classification and recognition. We evaluate our proposed system on our conversational dataset.
A Heterogeneous Graph Neural Network With Attribute Enhancement and Structure-Aware Attention

Heterogeneous information network (HIN) has been applied in a wide variety of graph analysis tasks. At present, it is a trend of heterogeneous graph neural networks (HGNNs) to cast the meta-paths aside, since it solves the problem of structural information loss caused by artificially designed meta-paths. However, existing meta-path-free HGNNs fail to take into account that most node types in many HINs have no attributes, and they cannot make full use of sparse node attributes when applied to HINs with missing attributes. Furthermore, their computation of attention coefficients explores the correlations of node attributes while almost ignoring structural ones, which may limit the expression ability of the model and cause overfitting in model training. To alleviate these issues, we propose an HGNN with attribute enhancement and structure-aware attention (HGNN-AESA). First, we design an attribute enhancement module (AEM) to connect more useful attributed nodes to the target nodes. Specifically, AEM introduces a random walk with restart (RWR) strategy to obtain structural relevance scores of each node within its specific subgraph. The structural relevance scores are used to capture potentially influential attributed nodes in high-order neighborhood for each target node. Second, we propose heterogeneous structure-aware attention layers (HSALs) to learn node representations. HSALs follow a hierarchical attention framework, including node-level and type-level attention. The node-level attention aggregates feature (attribute) embeddings of same-type neighbors, and the relevant attention coefficients depend on the combination of node attributes and heterogeneous structural interventions. The type-level attention fuses all type-specific vector representations and generates the ultimate node embedding. Finally, extensive experiments on three different real-world HIN datasets demonstrate that our model outperforms state-of-the-art methods.
Circuit Implementation of Proximal Projection Neural Networks for Composite Optimization Problems

Existing circuits for composite optimization problems tend to ignore the structure of nonsmooth objectives and lead to less practicability. To this end, with a proximal projection neural network (PPNN) and an inertial proximal projection neural network (IPPNN), this paper presents two novel analog circuit frameworks in presence of a nonsmooth term, where the actions of neuron are simulated via a feedback loop formed by integrator, gradient estimate, proximal operator and other basic operation models. In our circiuts, it shows that the stable output voltages are mapped to optimal solutions. Considering the nonsmooth term as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${{l}_{1}}$</tex-math></inline-formula> -norm, we further design an effective analog circuit of soft thresholding operator. Theoretically, it is proved that both PPNN-based and IPPNN-based analog circuit frameworks have stable output voltages in the sense that the value of electronic components meets a certain mild proportion. Finally, two examples are simulated on Multisim 14.0 platform to validate the effectiveness and practicability of the proposed analog circuits.
Blockchain and Federated Reinforcement Learning for Vehicle-to-Everything Energy Trading in Smart Grids

The proliferation of electric vehicles (EVs) and the advancement of Vehicle-to-Everything (V2X) energy trading systems are expected to play a crucial role in alleviating the stress on the electric grid during peak hours. However, the wide adoption of these paradigms requires intelligent mechanisms that protect the security and privacy of EV users. This paper proposes a novel Federated Reinforcement Learning (FRL) system combined with blockchain technology to maximize EV users' utility while preserving the security and privacy of trading transactions. Furthermore, we develop the concept of Proof of State of Charge (PoSOC) as a consensus mechanism to determine the winning EVs and reward them as block miners in the blockchain. The proposed system is validated through comprehensive simulation experiments utilizing a real-world dataset. The model is implemented on the Avalanche blockchain platform to demonstrate its real-world feasibility. The test results show that the proposed scheme improves EV users' utility significantly compared to the existing studies. The obtained simulation results indicate the effectiveness and robustness of the proposed system.
Model Sparsification for Communication-Efficient Multi-Party Learning via Contrastive Distillation in Image Classification

Multi-party learning allows all parties to train a joint model under legal and practical constraints without private data transmission. Related research can perform multi-party learning tasks on homogeneous data through deep networks. However, due to the heterogeneity of data from different parties and the limitation of computational resources and costs, traditional approaches may affect the effectiveness of multi-party learning, and cannot provide a personalized network for each party. In addition, to reduce the computational cost and communication bandwidth of local models, there are still challenges in building an adaptive model from the private data of different parties. To address these challenges, we aim to apply a model sparsification strategy in multi-party learning. Model sparsification can not only reduce the computational overhead in local edge devices and the cost of communication and interaction between multi-party models. It can also develop privatized and personalized networks based on the heterogeneity of local data. We use the contrastive distillation method during training to reduce the distance between local and global models. In addition, we maintain the performance of the aggregation model from heterogeneous data. In brief, we developed an adaptive multi-party learning framework based on contrastive distillation, which can significantly reduce the communication cost in the learning process, improve the effectiveness of the aggregation model for local heterogeneous and unbalanced data, and make it easy to deploy in the limited edge devices. Finally, to verify the effectiveness of this framework, we experimented with the Fshion-MNIST, Cifar-10, and Cifar-100 datasets in different scenarios to verify the effectiveness of this framework.
AdaDet: An Adaptive Object Detection System Based on Early-Exit Neural Networks

Recent researchers have proposed adaptive inference methods with an early-exiting mechanism, which stops the inference procedure of input if the prediction is with high confidence, leading to a fine-grained resource allocation based on the complexity of inputs. However, the adaptive inference strategy can only be applied for image classification tasks, such a system for object detection is still under-investigated since it lacks an appropriate way to measure the uncertainty of detection results for multiple objects in an image. To address this issue, in this paper, we propose a novel adaptive object detection system (AdaDet) based on early-exit neural networks. By using stochastic variables to represent localization predictions, our method can measure the uncertainty of the detection result for an object, based on which we further develop an entropy-based criterion to estimate the reliability of the whole detection results for an input image. With the proposed method, samples that achieve high-confidence detection results will exit early from the detection model, leading to low computational costs for inferring these samples. While only complex images need to finish the whole computational graph to achieve better detection results during inference. We experimentally demonstrate that the proposed AdaDet enhances detection performance under anytime prediction and adaptive inference settings on the most common dataset (MS COCO) used in this field.
Collaborative Parameter Estimation of Multiple Unmanned Surface Vessels: A Robust Distributed Estimator-Based Approach

In this paper, the collaborative parameter estimation of multiple unmanned surface vessels (USVs) with model structure uncertainties is studied. The considered parameter estimation problem is first converted into a distributed state and parameter joint estimation problem. Then, the robust distributed estimator is constructed to handle the inevitable model structure uncertainties, and the upper bounds of prediction and estimation error covariance matrices are derived respectively. Additionally, the upper bound of the estimation error covariance matrix is minimized by designing appropriate estimator gains. Finally, the advantages of the proposed distributed parameter estimation approach from the perspectives of information interaction and consideration of model structure uncertainties are verified via simulations and practical experiments.
On the Optimization Landscape of Dynamic Output Feedback Linear Quadratic Control

The convergence of policy gradient algorithms in reinforcement learning hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired from analyzing those of linear quadratic control. However, most of the existing literature only considers the optimization landscape for static full-state or output feedback policies (controllers). We investigate the more challenging case of dynamic output-feedback policies for linear quadratic regulation (abbreviated as dLQR), which is prevalent in practice but has a rather complicated optimization landscape. We first show how the dLQR cost varies with the coordinate transformation of the dynamic controller and then derive the optimal transformation for a given observable stabilizing controller. At the core of our results is the uniqueness of the stationary point of dLQR when it is observable, which is in a concise form of an observer-based controller with the optimal similarity transformation. These results shed light on designing efficient algorithms for general decision-making problems with partially observed information.
Self-Calibrating Gaze Estimation With Optical Axes Projection for Head-Mounted Eye Tracking

Gaze estimation suffers from burdensome personal calibration or complex all-device calibration. Self-calibrating methods can meet this challenge but depend on scenes and sacrifice accuracy. We propose a flexible and accurate gaze estimation approach calibrated implicitly with potential gaze patterns. By constructing an optical axis projection (OAP) plane and a visual axis projection (VAP) plane simultaneously, the optical axis and the visual axis can be represented as 2-D points, i.e., the OAP and VAP, which have a similarity transformation, indicating the linear consistency of OAP patterns with gaze patterns. Hence, a 3-D gaze estimation model using the OAP as an eye feature to predict the VAP is built. The unknown parameters are calculated separately by linearly aligning OAP patterns to natural and easily detectable gaze patterns. Experimental results show that the proposed gaze estimation approach is more accurate than state-of-the-art head-mounted gaze estimation methods, which require explicit calibration or multi-scene saliency.
Energy-Motion Features Aggregation Network for Players’ Fine-Grained Action Analysis in Soccer Videos

Rich and complex events in sports have led to the development of a wide-variety of techniques for interpreting content of sports videos in terms of players’ actions, poses, gait, performance, etc. This is due to the requirements from coaches, trainers and players who expect to analyze actions in top sports events, as well as sports fans who practice to imitate professional playing skills, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., dribbling, shooting, etc. However, this poses two key challenges for automated sports analysis community. Firstly, there are extremely limited public sports datasets. Secondly, recent advances in interpretations of sports activities, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., soccer, are predominantly made through analyzing coarse-grained contents. Players’ fine-grained skills analysis still remains under-explored. To alleviate these problems, this paper (a) collects the dataset of highlight videos of soccer players, including two coarse-grained action types of soccer players and six fine-grained actions of players. Detailed annotations are provided for the collected dataset, in terms of action classes, bounding boxes, segmentation maps, and body keypoints of soccer players, and positions of a soccer ball in a game. (b) leverages the understanding of complex highlight videos by proposing an energy-motion features aggregation network- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EMA-Net</i> to fully exploit energy-based representation of soccer players movements in video sequences and explicit motion dynamics of soccer players in videos for soccer players’ fine-grained action analysis. Experimental results and ablation studies validate the proposed approach in recognizing soccer players actions using the collected soccer highlight video datasets.
A Label-Efficient Semi Self-Supervised Learning Framework for IoT Devices in Industrial Process

The industrial sector has experienced a tremendous advancement in deep supervised learning due to its representation ability, but it comes with high computing and labeled data demands. Recently, the demand for intelligent IoT devices on assembly and disassembly lines has surged. This necessitates algorithms that can use data to make intelligent decisions and a framework that can enable multiple IoT devices to learn collaboratively. Further, huge image generation using IoTs also needs an efficient data annotation scheme for classification problems. WeCollab is one such framework in federated learning that significantly reduces human efforts in data annotation with breakthroughs in self-supervised learning. The proposed framework is generic and can be adapted to any specific image data generated by industrial robots involved in assembly and disassembly lines. Our method outperforms supervised learning by 25% and 20% on the CIFAR-10 and CINIC-10 datasets, respectively, for the labeling task. We generate pseudo-labels for the unlabeled part of the data and train a model to achieve 30% better test accuracy on CIFAR-10 and 20% better test accuracy on the CINIC-10 dataset as compared to supervised learning. Extensive experiments unveil the effectiveness of the method and proposed combination of loss functions used by WeCollab.