Reviewerly/exHarmony huggingface dataset:
Improving Early Children's Language Capabilities through Interactive Compact Disk Media

Language skills are essential skills to be developed from an early age. That's because language is an important means in human life to communicate. With language, humans can issue ideas and opinions to establish communication with other humans. Thus, language skills should have been improved since early childhood. In line with the development of science and technology, various kinds of learning media can be found that can be used to develop students' abilities, one of which is interactive compact disk media. The research was conducted at Raudhatul Athfal Al-Hikmah Kedaton Bandar Lampung. This research is a Classroom Action Research with a qualitative descriptive type. The data collection technique that the researcher uses is the observation method as the primary method, the documentation method as support in research, and the interview method to determine the extent of the teacher's response to the use of interactive compact disk media in improving children's language skills. The results of this study that interactive compact disk media can improve children's language skills in Raudhatul Athfal Al-Hikmah Kedaton Bandar Lampung with very satisfying results. This can be seen from the development of students in the pre-cycle research it is known that students who reach the assessment standard develop according to expectations. There are only 3 children out of all 15 students. Then in the first cycle, the number of children who had the language skills as expected increased to 9 children, and in the second cycle, it rose again to 12 students or 80% of the students had reached the assessment standards that had been set.
A Novel Multimodal Deep Learning Framework for Encrypted Traffic Classification

Traffic classification is essential for cybersecurity maintenance and network management, and has been widely used in QoS (Quality of Service) guarantees, intrusion detection, and other tasks. Recently, with the emergence of SSL/TLS encryption protocols in the modern Internet environment, the traditional payload-based classification methods are no longer effective. Some researchers have used machine learning methods to model the flow features of encrypted traffics (e.g. message type, length sequence, statistical features, etc.), and achieved good results in some cases. However, these high-level hand-designed features cannot be used for more fine-grained operations and may lead to the loss of important information, thus affecting the classification accuracy. To overcome this limitation, in this paper, we designed a novel multimodal deep learning framework for encrypted traffic classification called PEAN. PEAN uses the raw bytes and length sequence as the input, and uses the self-attention mechanism to learn the deep relationship among network packets in a biflow. Furthermore, unsupervised pre-training was introduced to enhance PEAN’s ability to characterize network packets. Experiments on a real trace set captured in a large data center demonstrate the effectiveness of PEAN, which achieves better results than the state-of-the-art methods.
Inception Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Graph convolutional networks is widely used in the field of skeleton-based motion recognition because of its characteristics of applying to non-Euclidean data. But most of the existing methods based on graph convolutional networks only perform convolution between adjacent joint points, ignoring the connection with farther joint points and symmetrical points. In order to improve the accuracy of skeleton-based action recognition, we propose a novel Inception spatial temporal graph convolutional networks (IST-GCN) for skeleton-based action recognition. By introducing the symmetry characteristics of the skeleton, our model can extract the interactive features of the symmetrical part of the human body. We also use the idea of multi-scale convolution to improve graph convolutional networks and temporal convolutional networks based on the Inception structure to better extract spatial and temporal features. A large number of experiments on NTU-RGB+D dataset show that our models have achieved higher accuracy and are more suitable for application in skeleton-based motion recognition. Codes are available at https://github.com/julycrow/IST-GCN.
SphericGAN: Semi-supervised Hyper-spherical Generative Adversarial Networks for Fine-grained Image Synthesis

Generative Adversarial Network (GAN)-based models have greatly facilitated image synthesis. However, the model performance may be degraded when applied to finegrained data, due to limited training samples and subtle distinction among categories. Different from generic GAN-s, we address the issue from a new perspective of discovering and utilizing the underlying structure of real data to explicitly regularize the spatial organization of latent space. To reduce the dependence of generative models on labeled data, we propose a semi-supervised hyper-spherical GAN for class-conditional fine-grained image generation, and our model is referred to as SphericGAN. By projecting random vectors drawn from a prior distribution onto a hyper-sphere, we can model more complex distributions, while at the same time the similarity between the resulting latent vectors depends only on the angle, but not on their magnitudes. On the other hand, we also incorporate a mapping network to map real images onto the hyper-sphere, and match latent vectors with the underlying structure of real data via real-fake cluster alignment. As a result, we obtain a spatially organized latent space, which is useful for capturing class-independent variation factors. The experi-mental results suggest that our SphericGAN achieves state-of-the-art performance in synthesizing high-fidelity images with precise class semantics.
Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values

We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of any pre-trained deep generative network (DGN). Leveraging the fact that DGNs are, or can be ap-proximated by, continuous piecewise affine splines, we derive the analytical DGN output space distribution as a function of the product of the DGN's Jacobian singular values raised to a power p. We dub p the polarity param-eter and prove that p focuses the DGN sampling on the modes (p < 0) or anti-modes (p > 0) of the DGN output-space probability distribution. We demonstrate that nonzero polarity values achieve a better precision-recall (quality-diversity) Pareto frontier than standard methods, such as truncation, for a number of state-of-the-art DGNs. We also present quantitative and qualitative results on the improve-ment of overall generation quality (e.g., in terms of the Fréchet Inception Distance) for a number of state-of-the-art DGNs, including StyleGAN3, BigGAN-deep, NVAE, for different conditional and unconditional image generation tasks. In particular, Polarity Sampling redefines the state-of-the-art for StyleGAN2 on the FFHQ Dataset to FID 2.57, StyleGAN2 on the LSUN Car Dataset to FID 2.27 and Style-GAN3 on the AFHQv2 Dataset to FID 3.95. Colab Demo.
Human Hands as Probes for Interactive Object Understanding

Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features, and object affordances (regions of interaction and afforded grasps), purely by observing hands in egocentric videos.
A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information

Deep spatiotemporal models are used in a variety of computer vision tasks, such as action recognition and video object segmentation. Currently, there is a limited understanding of what information is captured by these models in their intermediate representations. For example, while it has been observed that action recognition algorithms are heavily influenced by visual appearance in single static frames, there is no quantitative methodology for evaluating such static bias in the latent representation compared to bias toward dynamic information (e.g. motion). We tackle this challenge by proposing a novel approach for quantifying the static and dynamic biases of any spatiotemporal model. To show the efficacy of our approach, we analyse two widely studied tasks, action recognition and video object segmentation. Our key findings are threefold: (i) Most examined spatiotemporal models are biased toward static information; although, certain two-stream architectures with cross-connections show a better balance between the static and dynamic information captured. (ii) Some datasets that are commonly assumed to be biased toward dynamics are actually biased toward static information. (iii) Individual units (channels) in an architecture can be biased toward static, dynamic or a combination of the two. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Project page and code
Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation

Neural Radiance Fields (NeRF) [22] have emerged as a potent paradigm for representing scenes and synthesizing photo-realistic images. A main limitation of conventional NeRFs is that they often fail to produce high-quality renderings under novel viewpoints that are significantly different from the training viewpoints. In this paper, instead of ex-ploiting few-shot image synthesis, we study the novel view extrapolation setting that (1) the training images can well describe an object, and (2) there is a notable discrepancy between the training and test viewpoints' distributions. We present RapNeRF (RAy Priors) as a solution. Our insight is that the inherent appearances of a 3D surface's arbitrary visible projections should be consistent. We thus propose a random ray casting policy that allows training unseen views using seen views. Furthermore, we show that a ray atlas pre-computed from the observed rays' viewing directions could further enhance the rendering quality for ex-trapolated views. A main limitation is that RapNeRF would remove the strong view-dependent effects because it lever-ages the multi-view consistency property.
Nonlocal Spatial–Spectral Neural Network for Hyperspectral Image Denoising

Hyperspectral image (HSI) denoising is an essential preprocessing step to improve the quality of HSIs. The difficulty of HSI denoising lies in effectively modeling the intrinsic characteristics of HSIs, such as spatial-spectral correlation, global spectral correlation, and nonlocal spatial correlation. This paper introduces a nonlocal spatial-spectral neural network (NSSNN) for HSI denoising by considering the above three factors in a unified network. More specifically, NSSNN is based on the residual U-Net and embedded with the introduced spatial-spectral recurrent (SSR) blocks and nonlocal self-similarity (NSS) blocks. The SSR block comprises 3D convolutions, one light recurrence, and one highway network. 3D convolution helps exploit the spatial-spectral correlation. The light recurrence and highway network make up the recurrent computation component and refined component, respectively, to model the global spectral correlation. NSS block is based on crisscross attention and can exploit the long-range spatial contexts effectively and efficiently. Attributing to effective modeling of the spatial-spectral correlation, the global spectral correlation, and the nonlocal spatial correlation, our NSSNN has a strong denoising ability. Extensive experiments show the superior denoising effectiveness of our method on synthetic and real-world datasets when compared to alternative methods. The source code will be available at https://github.com/lronkitty/NSSNN.
Neuromorphic Integrated Sensing and Communications

Neuromorphic computing is an emerging technology that support event-driven data processing for applications requiring efficient online inference and/or control. Recent work has introduced the concept of neuromorphic communications, whereby neuromorphic computing is integrated with impulse radio (IR) transmission to implement low-energy and low-latency remote inference in wireless Internet-of-Things (IoT) networks. In this letter, we introduce neuromorphic integrated sensing and communications (N-ISAC), a novel solution that enables efficient online data decoding and radar sensing. N-ISAC leverages a common IR waveform for the dual purpose of conveying digital information and of detecting the presence or absence of a radar target. A spiking neural network (SNN) is deployed at the receiver to decode digital data and to detect the radar target using directly the received signal. The SNN operation is optimized by balancing performance metrics for data communications and radar sensing, highlighting synergies and trade-offs between the two applications.
Capformer: Pure Transformer for Remote Sensing Image Caption

Accurately describing high-spatial resolution remote sensing images requires the understanding the inner attributes of the objects and the outer relations between different objects. The existing image caption algorithms lack the ability of global representation, which are not fit for the summarization of complex scenes. To this end, we propose a pure transformer (CapFormer) architecture for remote sensing image caption. Specifically, a scalable vision transformer is adopted for image representation, where the global content can be captured with multi-head self-attention layers. A transformer decoder is designed to successively translate the image features into comprehensive sentences. The transformer decoder explicitly model the historical words and interact with the image features using cross-attention layers. The comprehensive and ablation experiments on RSICD dataset demonstrate that the CapFormer outperforms the state-of-the-art image caption methods.
Simultaneously Optimizing Perturbations and Positions for Black-Box Adversarial Patch Attacks

Adversarial patch is an important form of real-world adversarial attack that brings serious risks to the robustness of deep neural networks. Previous methods generate adversarial patches by either optimizing their perturbation values while fixing the pasting position or manipulating the position while fixing the patch's content. This reveals that the positions and perturbations are both important to the adversarial attack. For that, in this article, we propose a novel method to simultaneously optimize the position and perturbation for an adversarial patch, and thus obtain a high attack success rate in the black-box setting. Technically, we regard the patch's position, the pre-designed hyper-parameters to determine the patch's perturbations as the variables, and utilize the reinforcement learning framework to simultaneously solve for the optimal solution based on the rewards obtained from the target model with a small number of queries. Extensive experiments are conducted on the Face Recognition (FR) task, and results on four representative FR models show that our method can significantly improve the attack success rate and query efficiency. Besides, experiments on the commercial FR service and physical environments confirm its practical application value. We also extend our method to the traffic sign recognition task to verify its generalization ability.
Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression

Entropy modeling is a key component for high-performance image compression algorithms. Recent developments in autoregressive context modeling helped learning-based methods to surpass their classical counterparts. However, the performance of those models can be further improved due to the underexploited spatio-channel dependencies in latent space, and the suboptimal implementation of context adaptivity. Inspired by the adaptive characteristics of the transformers, we propose a transformer-based context model, named Contextformer, which generalizes the de facto standard attention mechanism to spatio-channel attention. We replace the context model of a modern compression framework with the Contextformer and test it on the widely used Kodak, CLIC2020, and Tecnick image datasets. Our experimental results show that the proposed model provides up to 11% rate savings compared to the standard Versatile Video Coding (VVC) Test Model (VTM) 16.2, and outperforms various learning-based models in terms of PSNR and MS-SSIM.
Self-Parameter Distillation Dehazing

In this paper, we propose a novel dehazing method based on self-distillation. In contrast to conventional knowledge distillation approaches that transfer large models (teacher networks) to small models (student networks), we introduce a single knowledge distillation network that transfers network parameters to itself for dehazing. In the early stages, the proposed network transfers scene content (identity) information to the next stage of itself using haze-free data. However, in the later stages, the network transfers haze information to itself using haze data, enabling the accurate dehazing of input images using scene information from the early stages. In a single network, parameters are seamlessly updated from extracting global scene features to dehazing the scene. During the training, forward propagation acts as a teacher network, whereas backward propagation acts as a student network. The experimental results demonstrate that the proposed method considerably outperforms other state-of-the-art dehazing methods.
Flow-Based Encrypted Network Traffic Classification With Graph Neural Networks

Classifying encrypted traffic from emerging applications is important but challenging as many conventional traffic classification approaches are ineffective, thus calling for novel methods for identifying encrypted network flows. Recent machine learning and deep learning-based approaches are severely limited by their feature selection and inherent neural network architecture. More importantly, they overlook the opportunity to capture latent information in the temporal dimension of packets. As network data by nature are of non-Euclidean distance space and carry abundant chronological and temporal relations, we are inspired to utilize geometric deep learning that simultaneously takes into account packet raw bytes, metadata and packet relations for classifying encrypted network traffic. Our proposed graph neural network (GNN) model outperforms the two reference methods, convolutional neural networks (CNN) and recurrent neural networks (RNN) quantitatively as indicated by three metrics: sensitivity, precision and F1 score.
DeepCar 5.0: Vehicle Make and Model Recognition Under Challenging Conditions

Inspired by multi-agent systems (MASs) and ensemble models, a novel method of vehicle make and model recognition (VMMR) based on a vehicle’s front-view images is presented in this paper. By exploiting the attention mechanism in this work, it is demonstrated that most of the features used for classifying a vehicle are extracted from its headlight, grill, scoop, and bumper sections. These areas are designated as the regions of interest (ROIs) in our approach. Contrary to the other methods in which a whole ROI is fed to a deep convolutional neural network, in this scheme, different ROIs are extracted from each image, and then a preprocessing block and a distinctive network are designed for each ROI, which is considered as a single agent. Each agent is then trained separately, and a vehicle’s type is determined with the collaboration of these agents and based on the blackboard classification system. Also, a new dataset (DeepCar 5.0) is compiled by using the data from the top 50 automakers. This dataset contains 40,185 images of the front views and the front three-quarters of vehicles in 480 different classes, and all the parts of dataset are labeled manually. The proposed technique is able to achieve the accuracies of 92.14 and 96.72% in the automated and the manual scenarios, respectively; and contrary to the current methods, it can perform flawless classification even when just a portion of a vehicle’s front view image is available for processing. The dataset and parts of the code are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/DeepCar/DeepCar5.0</uri> .
Graph Signal Processing for Heterogeneous Change Detection

This paper provides a new strategy for the heterogeneous change detection (HCD) problem: solving HCD from the perspective of graph signal processing (GSP). We construct a graph to represent the structure of each image, and treat each image as a graph signal defined on the graph. In this way, we convert the HCD into a GSP problem: a comparison of the responses of signals on systems defined on the graphs, which attempts to find structural differences and signal differences due to the changes between heterogeneous images. Firstly, we analyze the GSP for HCD from the vertex domain. We show that once a region has changed, the local structure of image changes, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> . the connectivity of the vertex containing this region changes. Therefore, we can compare the output signals of the same input graph signal passing through filters defined on the two graphs to detect changes. We analyze the negative effects of changing regions on the change detection results from the viewpoint of signal propagation, and we also design different filters from the vertex domain to explore the high-order neighborhood information hidden in original graphs. Secondly, we analyze the GSP for HCD from the spectral domain. We explore the spectral properties of different images on the same graph, and show that their spectra exhibit commonalities and dissimilarities. Specifically, it is the change that leads to the dissimilarities of their spectra. With the help of graph spectral analysis, we propose a regression model for the HCD, which decomposes the source signal into the regressed signal and changed signal, and constrains the spectral property of the regressed signal. Experiments conducted on seven real data sets show the effectiveness of the vertex domain filtering based and spectral domain analysis based HCD methods. Source code will be made available at https://github.com/yulisun/HCD-GSP.
Federated Learning with Position-Aware Neurons

Federated Learning (FL) fuses collaborative models from local nodes without centralizing users' data. The permutation invariance property of neural networks and the non-i.i.d. data across clients make the locally updated parameters imprecisely aligned, disabling the coordinate-based parameter averaging. Traditional neurons do not explicitly consider position information. Hence, we propose Position-Aware Neurons (PANs) as an alternative, fusing position-related values (i.e., position encodings) into neuron outputs. PANs couple themselves to their positions and minimize the possibility of dislocation, even updating on heterogeneous data. We turn on/off PANs to disable/enable the permutation invariance property of neural networks. PANs are tightly coupled with positions when applied to FL, making parameters across clients pre-aligned and facilitating coordinate-based parameter averaging. PANs are algorithm-agnostic and could universally improve existing FL algorithms. Furthermore, “FL with PANs” is simple to implement and computationally friendly.