Reviewerly/exHarmony huggingface dataset:
FPD: Feature Pyramid Knowledge Distillation

Knowledge distillation is a commonly used method for model compression, aims to compress a powerful yet cumbersome model into a lightweight model without much sacrifice of performance, giving the accuracy of a lightweight model close to that of the cumbersome model. Commonly, the efficient but bulky model is called the teacher model and the lightweight model is called the student model. For this purpose, various approaches have been proposed over the past few years. Some classical distillation methods are mainly based on distilling deep features from the intermediate layer or the logits layer, and some methods combine knowledge distillation with contrastive learning. However, classical distillation methods have a significant gap in feature representation between teacher and student, and contrastive learning distillation methods also need massive diversified data for training. For above these issues, our study aims to narrow the gap in feature representation between teacher and student and obtain more feature representation from images in limited datasets to achieve better performance. In addition, the superiority of our method is all validated on a generalized dataset (CIFAR-100) and a small-scale dataset (CIFAR-10). On CIFAR-100, we achieve 19.21%, 20.01% of top-1 error with Resnet50 and Resnet18, respectively. Especially, Resnet50 and Resnet18 as student model achieves better performance than the pre-trained Resnet152 and Resnet34 teacher model. On CIFAR-10, we perform 4.22% of top-1 error with Resnet-18. Whether on CIFAR-10 or CIFAR-100, we all achieve better performance, and even the student model performs better than the teacher.
An Effective Ensemble Model Related to Incremental Learning in Neural Machine Translation

In recent years, machine translation has made great progress with the rapid development of deep learning. However, there still exists a problem of catastrophic forgetting in the field of neural machine translation, namely, a decrease in overall performance will happen when training with new data added incrementally. Many methods related to incremental learning have been proposed to solve this problem in the tasks of computer vision, but few for machine translation. In this paper, firstly, several prevailing methods relevant to incremental learning are applied into the task of machine translation, then we proposed an ensemble model to deal with the problem of catastrophic forgetting, at last, some important and authoritative metrics are used to evaluate the model performances in our experiments. The results can prove that the incremental learning is also effective in the task of neural machine translation, and the ensemble model we put forward is also capable of improving the model performance to some extent.
Self-attention Based Multi-scale Graph Convolutional Networks

Graph convolutional networks (GCNs) have achieved remarkable learning ability for dealing with various graph structural data recently. In general, GCNs have low expressive power due to their shallow structure. In this paper, to improve the expressive power of GCNs, we propose two multi-scale GCN frameworks by incorporating self-attention mechanism and multi-scale information into the design of GCNs. The self-attention mechanism allows us to adaptively learn the local structure of the neighborhood, and achieves more accurate predictions. Extensive experiments on both node classification and graph classification demonstrate the effectiveness of our approaches over several state-of-the-art GCNs.
Context-Adapted Multi-policy Ensemble Method for Generalization in Reinforcement Learning

Generalizability is a formidable challenge in applying reinforcement learning to the real world. The root cause of poor generalization performance in reinforcement learning is that generalization from a limited number of training conditions to unseen test conditions results in implicit partial observability, effectively transforming even fully observed Markov Decision Process (MDP) into Partially Observable Markov Decision Process (POMDP). To address such issues, we propose a novel structure, namely Context-adapted Multi-policy Ensemble Method (CAMPE), which enables the model to adapt to changes in the environment and efficiently solve implicit partial observability during generalization. The method captures local dynamic changes by learning contextual environment latent variables to equip the model with the ability of environment adaption. The latent variables and samples with contextual information are used as the input of the policy. Multiple policies are trained, combined in an integrated way to obtain a single policy to approximately solve the problem of partial observability. We demonstrate our method on various simulated robotics and control tasks. Experimental results show that our method achieves superior generalization ability.
A Reliable and Intelligent Deep Learning Based Demodulator for M-Ary Code Shifted Differential Chaos Shift Keying System With Power Allocation

In high-speed railway systems, channel conditions are dramatically changing, which brings great challenges to information transmissions. Our objective is to improve the reliability performances by utilizing the deep learning (DL) scheme to intelligently extract the features of signals to combat the complex dynamically changing channel conditions. In this paper, we propose a DL-aided demodulation scheme for the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</i> -ary code shifted differential chaos shift keying system with power allocation (PA-GCS-MDCSK) to enhance reliability performance. In this design, the deep neural network (DNN) adopts fully-connected layers (FCLs) to conduct the joint de-spreading of Walsh codes and chaotic demodulations. Meanwhile, the long short-term memory (LSTM) unit with the residual structure is constructed to extract the correlation between the chaotic sequences, thus the interferences induced by real-valued chaotic sequences can be suppressed, thereby improving the reliability performance. Then the computational complexity is analyzed and compared with benchmark schemes. Simulation results over both additive white Gaussian noise (AWGN), fading, and railway channels validate that the proposed design can achieve better bit error rate (BER) and robustness performances than benchmark schemes.
Second-Order Nonlinearity of a Boolean Function Class with Low Spectra

In 2015, Cao and Hu (Cao, X., Hu, L.: Two boolean functions with five-valued walsh spectra and high nonlinearity. International Journal of Foundations of Computer Science, pp. 537–556 (2015)) introduced a certain class of Boolean functions, possessing low Walsh spectra, high nonlinearity, and high algebraic degree. For this class of Boolean functions, computation of higher-order nonlinearities (even second-order) is a tedious task. Therefore, in this article, we study the lower bound on the second-order nonlinearity of the above-mentioned class of Boolean functions for $$n=4.$$ Also, we deduce that the bound, thus obtained is the maximum possible bound. We also demonstrated that our lower bound is greater than the lower bound on the second-order nonlinearity of other classes of cubic Boolean functions.
Differentiable Causal Discovery Under Heteroscedastic Noise

We consider the problem of estimating directed acyclic graphs from observational data. Many studies on functional causal models assume the independence of noise terms. Thus, they suffer from the typical violation of model assumption: heteroscedasticity. Several recent studies have assumed heteroscedastic noise instead of additive noise in data generation, though most of the estimation algorithms are for bivariate data. This study aims to improve the capability of continuous optimization-based methods so that they can handle heteroscedastic noise under multivariate non-linear data with no latent confounders. Numerical experiments on synthetic data and fMRI simulation data show that our estimation algorithm improves the estimation of the causal structure under heteroscedastic noise. We also applied our estimation algorithm to real-world data collected from a ceramic substrate manufacturing process, and the results prove the possibility of using the estimated causal graph to accelerate quality improvement.
Saccade Direction Information Channel

Eye tracking has become an increasingly important technology in many fields of research, such as marketing, human computer interaction, psychology, and also in human cognition. Understanding the human eye movements, while viewing specific scenarios, can be of great support for improving visual stimuli. However, the challenging problem with this kind of spatio-temporal data is to find quantitative links between eye movements and human cognition. This paper introduces the information channel based on saccade direction. The gaze transition between different saccade directions is modeled as a discrete information channel, which we call saccade direction information channel. The channel is applied to an eye-tracking dataset on Van Gogh’s paintings observation. In our results, horizontal saccades are more frequent than vertical saccades, and the information conveyed in horizontal/vertical displacements, measured as the mutual information of the channel, is higher than in diagonal displacements. By comparing the results to our previous spatial gaze channel between Areas of Interest (AOIs) we constate that the spatial channel discriminates better between the observed images, while the direction channel discriminates better between observers.
Byzantine-Robust Distributed Learning With Compression

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale distributed learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed distributed learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.
Factual Error Correction in Summarization with Retriever-Reader Pipeline

Summarization models compress the source text without sacrificing the primary information. However, about 30% of summaries produced by state-of-the-art summarization models suffer from the factual inconsistencies between source text and summary, also known as hallucinations, making them less trustworthy. It has been challenging to reduce hallucinations, especially entity hallucinations. Most prior works use the entire source text for factual error correction, while input length is often limited. It prevents models from checking facts precisely and limits its applications on long document summarization. To address this issue, we propose a post-editing factual error correction method based on a Retriever-Reader pipeline. After the summarization model generates the summary, entities in the summary are examined and corrected iteratively by retrieving and reading only the most relevant parts of the source document. We validate the proposed approach on the CNN/DM and Gigaword datasets. Experiments show that the proposed approach outperforms all the baseline models in the consistency metric. Better results are also achieved on human evaluations.
Multi-Agent Hyper-Attention Policy Optimization

Policy-based methods like MAPPO have exhibited amazing results in diverse test scenarios in multi-agent reinforcement learning. Nevertheless, current actor-critic algorithms do not fully leverage the benefits of the centralized training with decentralized execution paradigm and do not effectively use global information to train the centralized critic, as seen by IPPO’s superior performance in some scenarios compared to MAPPO. To address this problem, we propose a game abstraction technique based on a state-conditioned hyper-attention network. It can help agents integrate important data and refine complex game interactions to achieve efficient policy optimization. In addition, to improve the stability of the trust-region methods, we introduce a point probability distance penalty in addition to the clipping operation in PPO. Experimental results demonstrate the advantages of our method in various cooperative environments.
Mutual Diverse-Label Adversarial Training

Adversarial training is validated to be the most effective method to defend against adversarial attacks. In adversarial training, stronger capacity networks can achieve higher robustness. Mutual learning is plugged into adversarial training to increase robustness by improving model capacity. Specifically, two deep neural networks (DNNs) are trained together with two adversarial examples. Each DNN’s prediction not only fits the right label but also aligns with the other DNN’s prediction. To take full advantage of mutual learning, each DNN needs to learn more extra information about different incorrect class from the other. To achieve it, we propose diverse-label attack to help with training. Concretely, we generate two adversarial examples for the two DNNs making DNNs predict not only incorrectly but also differently. Combining the above two stages, we propose a novel adversarial training method called mutual diverse-label adversarial training (MDLAT). Experiments on CIFAR-10 and CIFAR-100 indicate that our method is effective in improving model robustness under different settings, and our method achieves state-of-the-art (SOTA) robustness under $$\ell _{\infty }$$ attack.
Dynamic Model-Based Safety Margins for High-Density Matrix Headlight Systems

Real-time masking of vehicles in a dynamic road environment is a demanding task for adaptive driving beam systems of modern headlights. Next-generation high-density matrix headlights enable precise, high-resolution projections, while advanced driver assistance systems enable detection and tracking of objects with high update rates and low-latency estimation of the pose of the ego-vehicle. Accurate motion tracking and precise coverage of the masked vehicles are necessary to avoid glare while maintaining a high light throughput for good visibility. Safety margins are added around the mask to mitigate glare and flicker caused by the update rate and latency of the system. We provide a model to estimate the effects of spatial and temporal sampling on the safety margins for high- and low-density headlight resolutions and different update rates. The vertical motion of the ego-vehicle is simulated based on a dynamic model of a vehicle suspension system to model the impact of the motion-to-photon latency on the mask. Using our model, we evaluate the light throughput of an actual matrix headlight for the relevant corner cases of dynamic masking scenarios depending on pixel density, update rate, and system latency. We apply the masks provided by our model to a high beam light distribution to calculate the loss of luminous flux and compare the results to a light throughput approximation technique from the literature.
Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

Compression of convolutional neural network models has recently been dominated by pruning approaches. A class of previous works focuses solely on pruning the unimportant filters to achieve network compression. Another important direction is the design of sparsity-inducing constraints which has also been explored in isolation. This paper presents a novel training scheme based on composite constraints that prune redundant filters and minimize their effect on overall network learning via sparsity promotion. Also, as opposed to prior works that employ pseudo-norm-based sparsity-inducing constraints, we propose a sparse scheme based on gradient counting in our framework. Our tests on several pixel-wise segmentation benchmarks show that the number of neurons and the memory footprint of networks in the test phase are significantly reduced without affecting performance. MobileNetV3 and UNet, two well-known architectures, are used to test the proposed scheme. Our network compression method not only results in reduced parameters but also achieves improved performance compared to MobileNetv3, which is an already optimized architecture.
An Analog Side-Channel Attack on a High-Speed Asynchronous SAR ADC Using Dual Neural Network Technique

This brief presents a side-channel attack (SCA) technique on a high-speed asynchronous successive approximation register (SAR) analog-to-digital converter (ADC). The proposed dual neural network based on multiple noise waveforms separately discloses sign and absolute value information of input signals which are hidden by the differential structure and high-speed asynchronous operation. The target SAR ADC and on-chip noise monitors are designed on a single prototype chip for SCA demonstration. Fabricated in 40 nm, the experimental results show the proposed attack on the asynchronous SAR ADC successfully restores the input data with a competitive accuracy within 300 mV rms error.
Class-Incremental Learning with Multiscale Distillation for Weakly Supervised Temporal Action Localization

Despite recent works having made great progress in weakly supervised temporal action localization (WTAL), they still suffer from catastrophic forgetting. When only new-class videos can be utilized to update these models, their performance in old classes diminishes drastically. Even while some class-incremental learning methods are presented to assist models in continuously learning new-class knowledge, most of them focus on image classification but pay little attention to WTAL. To fill this gap, we propose a novel class-incremental learning method with multiscale distillation, which mines two separate scales of old-class information in incoming videos for updating the model. Precisely, we calculate class activation sequences (CAS) with frame-level spatio-temporal information to provide fine-grained old-class labels for the updated model. Moreover, since the high activation segments contain rich action information, we select them and construct video-level logits to constrain the updated model for maintaining the old-class knowledge further. The experimental results under various incremental learning settings on the THUMOS’14 and ActivityNet 1.3 datasets reveal that our method effectively alleviates the catastrophic forgetting problem in WTAL.
HMC-PSO: A Hamiltonian Monte Carlo and Particle Swarm Optimization-Based Optimizer

We introduce the Hamiltonian Monte Carlo Particle Swarm Optimizer (HMC-PSO), an optimization algorithm that reaps the benefits of both Exponentially Averaged Momentum PSO and HMC sampling. The coupling of the position and velocity of each particle with Hamiltonian dynamics in the simulation allows for extensive freedom for exploration and exploitation of the search space. It also provides an excellent technique to explore highly non-convex functions while ensuring efficient sampling. We extend the method to approximate error gradients in closed form for Deep Neural Network (DNN) settings. We discuss possible methods of coupling and compare its performance to that of state-of-the-art optimizers on the Golomb’s Ruler problem and Classification tasks(HMC-PSO code and additional results are in the Github repository: https://github.com/rtdsouza/torchswarm ).
Long-Term Dynamic Window Approach for Kinodynamic Local Planning in Static and Crowd Environments

Local planning for a differential wheeled robot is designed to generate kinodynamic feasible actions that guide the robot to a goal position along the navigation path while avoiding obstacles. Reactive, predictive, and learning-based methods are widely used in local planning. However, few of them can fit static and crowd environments while satisfying kinodynamic constraints simultaneously. To solve this problem, we propose a novel local planning method. The method applies a long-term dynamic window approach to generate an initial trajectory and then optimizes it with graph optimization. The method can plan actions under the robot's kinodynamic constraints in real time while allowing the generated actions to be safer and more jitterless. Experimental results show that the proposed method adapts well to crowd and static environments and outperforms most state-of-the-art approaches.
Deep Confidence Propagation Stereo Network

Stereo matching depth estimation based on rectified image pairs is of great importance to many computer vision tasks such as vehicle navigation and autonomous driving. Confidence measures are typically used to refine stereo matching results, which provides robustness and efficiency for disparity estimation. However, previous learning-based confidence methods for stereo matching usually use the middle results or composition as a post-processing step to refine the stereo matching results. This cannot be optimized end-to-end and the performance is limited by the quality of the tri-modal output. To handle this issue, in this paper, we pursue an end-to-end hierarchical architecture and propose a differentiable confidence propagation (DCP) model of a cost aggregation network for stereo matching. The DCP model is integrated into an end-to-end neural network hierarchical architecture to guide matching cost volume aggregation. More specifically, to better represent the similarity of left and right feature maps, we extract unary context feature maps with an effective attention mechanism for matching cost construction. Moreover, we aggregate the cost volume with the multiple stacked DCP cost aggregation (DCPCA) networks to generate a more reliable and finer cost volume. This network suppresses multi-level disparity maps. Each output disparity is supervised with different training weights to learn in a coarse-to-fine way. Our method outperforms previous methods on the Sceneflow dataset by achieving the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.6735px$ </tex-math></inline-formula> EPE error, achieving 1.53% D1-all metric of Non-occluded pixels regions and 0.72% Non-occluded pixels of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5px$ </tex-math></inline-formula> metric on KITTI 2015 and 2012 dataset. Extensive experiments carried out on the KITTI Stereo benchmarks demonstrate that our DCPCA-Net can significantly minimize the trade-off between accuracy and efficiency for stereo matching.