OpenRaiser/Intern-Atlas huggingface dataset:
Deep Neural Networks in Video Human Action Recognition: A Review
Venue: arXiv cs.CV 2023
Year: 2023

Currently, video behavior recognition is one of the most foundational tasks of computer vision. The 2D neural networks of deep learning are built for recognizing pixel-level information such as images with RGB, RGB-D, or optical flow formats, with the current increasingly wide usage of surveillance video and more tasks related to human action recognition. There are increasing tasks requiring temporal information for frames dependency analysis. The researchers have widely studied video-based recognition rather than image-based(pixel-based) only to extract more informative elements from geometry tasks. Our current related research addresses multiple novel proposed research works and compares their advantages and disadvantages between the derived deep learning frameworks rather than machine learning frameworks. The comparison happened between existing frameworks and datasets, which are video format data only. Due to the specific properties of human actions and the increasingly wide usage of deep neural networks, we collected all research works within the last three years between 2020 to 2022. In our article, the performance of deep neural networks surpassed most of the techniques in the feature learning and extraction tasks, especially video action recognition.
PEARL: Preprocessing Enhanced Adversarial Robust Learning of Image Deraining for Semantic Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

In light of the significant progress made in the development and application of semantic segmentation tasks, there has been increasing attention towards improving the robustness of segmentation models against natural degradation factors (e.g., rain streaks) or artificially attack factors (e.g., adversarial attack). Whereas, most existing methods are designed to address a single degradation factor and are tailored to specific application scenarios. In this work, we present the first attempt to improve the robustness of semantic segmentation tasks by simultaneously handling different types of degradation factors. Specifically, we introduce the Preprocessing Enhanced Adversarial Robust Learning (PEARL) framework based on the analysis of our proposed Naive Adversarial Training (NAT) framework. Our approach effectively handles both rain streaks and adversarial perturbation by transferring the robustness of the segmentation model to the image derain model. Furthermore, as opposed to the commonly used Negative Adversarial Attack (NAA), we design the Auxiliary Mirror Attack (AMA) to introduce positive information prior to the training of the PEARL framework, which improves defense capability and segmentation performance. Our extensive experiments and ablation studies based on different derain methods and segmentation models have demonstrated the significant performance improvement of PEARL with AMA in defense against various adversarial attacks and rain streaks while maintaining high generalization performance across different datasets.
Learn to Not Link: Exploring NIL Prediction in Entity Linking
Venue: arXiv cs.CL 2023
Year: 2023

Entity linking models have achieved significant success via utilizing pretrained language models to capture semantic features. However, the NIL prediction problem, which aims to identify mentions without a corresponding entity in the knowledge base, has received insufficient attention. We categorize mentions linking to NIL into Missing Entity and Non-Entity Phrase, and propose an entity linking dataset NEL that focuses on the NIL prediction problem. NEL takes ambiguous entities as seeds, collects relevant mention context in the Wikipedia corpus, and ensures the presence of mentions linking to NIL by human annotation and entity masking. We conduct a series of experiments with the widely used bi-encoder and cross-encoder entity linking models, results show that both types of NIL mentions in training data have a significant influence on the accuracy of NIL prediction. Our code and dataset can be accessed at https://github.com/solitaryzero/NIL_EL
Accelerated K-Serial Stable Coalition for Dynamic Capture and Resource Defense
Venue: arXiv cs.RO 2023
Year: 2023

Coalition is an important mean of multi-robot systems to collaborate on common tasks. An adaptive coalition strategy is essential for the online performance in dynamic and unknown environments. In this work, the problem of territory defense by large-scale heterogeneous robotic teams is considered. The tasks include exploration, capture of dynamic targets, and perimeter defense over valuable resources. Since each robot can choose among many tasks, it remains a challenging problem to coordinate jointly these robots such that the overall utility is maximized. This work proposes a generic coalition strategy called K-serial stable coalition algorithm. Different from centralized approaches, it is distributed and complete, meaning that only local communication is required and a K-serial Stable solution is ensured. Furthermore, to accelerate adaptation to dynamic targets and resource distribution that are only perceived online, a heterogeneous graph attention network based heuristic is learned to select more appropriate parameters and promising initial solutions during local optimization. Compared with manual heuristics or end-to-end predictors, it is shown to both improve online adaptability and retain the quality guarantee. The proposed methods are validated via large-scale simulations with 170 robots and hardware experiments of 13 robots, against several strong baselines such as GreedyNE and FastMaxSum.
Security Impact Analysis of Degree of Field Extension in Lattice Attacks on Ring-LWE Problem
Venue: arXiv cs.CR 2023
Year: 2023

Modern information communications use cryptography to keep the contents of communications confidential. RSA (Rivest-Shamir-Adleman) cryptography and elliptic curve cryptography, which are public-key cryptosystems, are widely used cryptographic schemes. However, it is known that these cryptographic schemes can be deciphered in a very short time by Shor's algorithm when a quantum computer is put into practical use. Therefore, several methods have been proposed for quantum computer-resistant cryptosystems that cannot be cracked even by a quantum computer. A simple implementation of LWE-based lattice cryptography based on the LWE (Learning With Errors) problem requires a key length of $O(n^2)$ to ensure the same level of security as existing public-key cryptography schemes such as RSA and elliptic curve cryptography. In this paper, we attacked the Ring-LWE (RLWE) scheme, which can be implemented with a short key length, with a modified LLL (Lenstra-Lenstra-Lovász) basis reduction algorithm and investigated the trend in the degree of field extension required to generate a secure and small key. Results showed that the lattice-based cryptography may be strengthened by employing Cullen or Mersenne prime numbers as the degree of field extension.
Multi-scale Efficient Graph-Transformer for Whole Slide Image Classification
Venue: arXiv cs.CV 2023
Year: 2023

The multi-scale information among the whole slide images (WSIs) is essential for cancer diagnosis. Although the existing multi-scale vision Transformer has shown its effectiveness for learning multi-scale image representation, it still cannot work well on the gigapixel WSIs due to their extremely large image sizes. To this end, we propose a novel Multi-scale Efficient Graph-Transformer (MEGT) framework for WSI classification. The key idea of MEGT is to adopt two independent Efficient Graph-based Transformer (EGT) branches to process the low-resolution and high-resolution patch embeddings (i.e., tokens in a Transformer) of WSIs, respectively, and then fuse these tokens via a multi-scale feature fusion module (MFFM). Specifically, we design an EGT to efficiently learn the local-global information of patch tokens, which integrates the graph representation into Transformer to capture spatial-related information of WSIs. Meanwhile, we propose a novel MFFM to alleviate the semantic gap among different resolution patches during feature fusion, which creates a non-patch token for each branch as an agent to exchange information with another branch by cross-attention. In addition, to expedite network training, a novel token pruning module is developed in EGT to reduce the redundant tokens. Extensive experiments on TCGA-RCC and CAMELYON16 datasets demonstrate the effectiveness of the proposed MEGT.
VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale
Venue: arXiv cs.CV 2023
Year: 2023

The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. In this paper, we identify the \emph{small data pitfall} that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K. Specifically, we show that employing stronger data augmentation techniques and using larger datasets can directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants in more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83.1\%, 84.3\%, and 85.0\% top-1 accuracy, respectively. PyTorch code and checkpoints can be found at https://github.com/Hao840/vanillaKD.
MDVSC -- Wireless Model Division Video Semantic Communication
Venue: arXiv cs.MM 2023
Year: 2023

In this paper, we propose a new wireless video communication scheme to achieve high-efficiency video transmission over noisy channels. It exploits the idea of model division multiple access (MDMA) and extracts common semantic features across video frames. Besides, deep joint source-channel coding (JSCC) is applied to overcome the distortion caused by noisy channels. The proposed framework is collected under the name model division video semantic communication (MDVSC). In particular, temporal relative video frames are first transformed into a latent space for computing complexity reduction and data redistribution. Accordingly, a novel entropy-based variable length coding is developed further to compress semantic information under the communication bandwidth cost limitation. The whole MDVSC is an end-to-end learnable system. It can be formulated as an optimization problem whose goal is to minimize end-to-end transmission distortion under restricted communication resources. Across standard video source test sequences, test results show that the MDVSC outperforms traditional wireless video coding schemes generally under perceptual quality metrics and has the ability to control code length precisely.
PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion
Venue: arXiv cs.LG 2023
Year: 2023

The generalization of neural networks is a central challenge in machine learning, especially concerning the performance under distributions that differ from training ones. Current methods, mainly based on the data-driven paradigm such as data augmentation, adversarial training, and noise injection, may encounter limited generalization due to model non-smoothness. In this paper, we propose to investigate generalization from a Partial Differential Equation (PDE) perspective, aiming to enhance it directly through the underlying function of neural networks, rather than focusing on adjusting input data. Specifically, we first establish the connection between neural network generalization and the smoothness of the solution to a specific PDE, namely "transport equation". Building upon this, we propose a general framework that introduces adaptive distributional diffusion into transport equation to enhance the smoothness of its solution, thereby improving generalization. In the context of neural networks, we put this theoretical framework into practice as $\textbf{PDE+}$ ($\textbf{PDE}$ with $\textbf{A}$daptive $\textbf{D}$istributional $\textbf{D}$iffusion) which diffuses each sample into a distribution covering semantically similar inputs. This enables better coverage of potentially unobserved distributions in training, thus improving generalization beyond merely data-driven methods. The effectiveness of PDE+ is validated through extensive experimental settings, demonstrating its superior performance compared to SOTA methods.
Improved Multi-Scale Grid Rendering of Point Clouds for Radar Object Detection Networks
Venue: arXiv cs.CV 2023
Year: 2023

Architectures that first convert point clouds to a grid representation and then apply convolutional neural networks achieve good performance for radar-based object detection. However, the transfer from irregular point cloud data to a dense grid structure is often associated with a loss of information, due to the discretization and aggregation of points. In this paper, we propose a novel architecture, multi-scale KPPillarsBEV, that aims to mitigate the negative effects of grid rendering. Specifically, we propose a novel grid rendering method, KPBEV, which leverages the descriptive power of kernel point convolutions to improve the encoding of local point cloud contexts during grid rendering. In addition, we propose a general multi-scale grid rendering formulation to incorporate multi-scale feature maps into convolutional backbones of detection networks with arbitrary grid rendering methods. We perform extensive experiments on the nuScenes dataset and evaluate the methods in terms of detection performance and computational complexity. The proposed multi-scale KPPillarsBEV architecture outperforms the baseline by 5.37% and the previous state of the art by 2.88% in Car AP4.0 (average precision for a matching threshold of 4 meters) on the nuScenes validation set. Moreover, the proposed single-scale KPBEV grid rendering improves the Car AP4.0 by 2.90% over the baseline while maintaining the same inference speed.
Sequential Integrated Gradients: a simple but effective method for explaining language models
Venue: arXiv cs.CL 2023
Year: 2023

Several explanation methods such as Integrated Gradients (IG) can be characterised as path-based methods, as they rely on a straight line between the data and an uninformative baseline. However, when applied to language models, these methods produce a path for each word of a sentence simultaneously, which could lead to creating sentences from interpolated words either having no clear meaning, or having a significantly different meaning compared to the original sentence. In order to keep the meaning of these sentences as close as possible to the original one, we propose Sequential Integrated Gradients (SIG), which computes the importance of each word in a sentence by keeping fixed every other words, only creating interpolations between the baseline and the word of interest. Moreover, inspired by the training procedure of several language models, we also propose to replace the baseline token "pad" with the trained token "mask". While being a simple improvement over the original IG method, we show on various models and datasets that SIG proves to be a very effective method for explaining language models.
Power Allocation for Multi-Access Channel with Generalized Power Constraint
Venue: arXiv cs.IT 2023
Year: 2023

We study the problem of decentralized power allocation in a multi-access channel (MAC) with non-cooperative users, additive noise of arbitrary distribution and a generalized power constraint, i.e., the transmit power constraint is modeled by an upper bound on $\mathbb{E}[ϕ(|S|)]$, where $S$ is the transmit signal and $ϕ(.)$ is some non-negative, increasing and bounded function. The generalized power constraint captures the notion of power for different wireless signals such as RF, optical, acoustic, etc. We derive the optimal power allocation policy when there a large number of non-cooperative users in the MAC. Further, we show that, once the number of users in the MAC crosses a finite threshold, the proposed power allocation policy of all users is optimal and remains invariant irrespective of the actual number of users. We derive the above results under the condition that the entropy power of the MAC, $e^{2h(S)+c}$, is strictly convex, where $h(S)$ is the maximum achievable entropy of the transmit signal and $c$ is a finite constant corresponding to the entropy of the additive noise.
Room Impulse Response Estimation in a Multiple Source Environment
Venue: arXiv cs.SD 2023
Year: 2023

In real-world acoustic scenarios, there often are multiple sound sources present in a room. These sources are situated in various locations and produce sounds that reach the listener from multiple directions. The presence of multiple sources in a room creates new challenges in estimating the room impulse response (RIR) as each source has a unique RIR, dependent on its location and orientation. Therefore, issues of determining which RIR should be predicted and how to predict it arise, when the input signal is a mixture of multiple reverberated sources. To address these, we propose a new task of predicting a "representative" RIR for a room in a multiple source environment and present a training method to achieve this goal. In contrast to the model trained in a single source environment, our method shows robust performance, regardless of the number of sources in the environment.
Joint Precoding Design and Resource Allocation for C-RAN Wireless Fronthaul Systems
Venue: arXiv cs.IT 2023
Year: 2023

This paper investigates the resource allocation problem combined with fronthaul precoding and access link sparse precoding design in cloud radio access network (C-RAN) wireless fronthaul systems.Multiple remote antenna units (RAUs) in C-RAN systems can collaborate in a cluster through centralized signal processing to realize distributed massive multiple-input and multiple-output (MIMO) systems and obtain performance gains such as spectrum efficiency and coverage.Wireless fronthaul is a flexible, low-cost way to implement C-RAN systems, however, compared with the fiber fronthaul network, the capacity of wireless fronthaul is extremely limited.Based on this problem, this paper first design the fronthaul and access link precoding to make the fronthaul capacity of RAUs match the access link demand.Then, combined with the precoding design problem, the allocation optimization of orthogonal resources is studied to further optimize the resource allocation between fronthaul link and access link to improve the performance of the system.Numerical results verify the effectiveness of the proposed precoding design and resource allocation optimization algorithm.
Grouping Method for mmWave Massive MIMO System: Exploitation of Angular Multiplexing Gain
Venue: arXiv cs.IT 2023
Year: 2023

A future millimeter-wave (mmWave) massive multiple-input and multiple-output (MIMO) system may serve hundreds or thousands of users at the same time; thus, research on multiple access technology is particularly important.Moreover, due to the short-wavelength nature of a mmWave, large-scale arrays are easier to implement than microwaves, while their directivity and sparseness make the physical beamforming effect of precoding more prominent.In consideration of the mmWave angle division multiple access (ADMA) system based on precoding, this paper investigates the influence of the angle distribution on system performance, which is denoted as the angular multiplexing gain.Furthermore, inspired by the above research, we transform the ADMA user grouping problem to maximize the system sum-rate into the inter-user angular spacing equalization problem.Then, the form of the optimal solution for the approximate problem is derived, and the corresponding grouping algorithm is proposed.The simulation results demonstrate that the proposed algorithm performs better than the comparison methods.Finally, a complexity analysis also shows that the proposed algorithm has extremely low complexity.
Mask Attack Detection Using Vascular-weighted Motion-robust rPPG Signals
Venue: arXiv cs.CV 2023
Year: 2023

Detecting 3D mask attacks to a face recognition system is challenging. Although genuine faces and 3D face masks show significantly different remote photoplethysmography (rPPG) signals, rPPG-based face anti-spoofing methods often suffer from performance degradation due to unstable face alignment in the video sequence and weak rPPG signals. To enhance the rPPG signal in a motion-robust way, a landmark-anchored face stitching method is proposed to align the faces robustly and precisely at the pixel-wise level by using both SIFT keypoints and facial landmarks. To better encode the rPPG signal, a weighted spatial-temporal representation is proposed, which emphasizes the face regions with rich blood vessels. In addition, characteristics of rPPG signals in different color spaces are jointly utilized. To improve the generalization capability, a lightweight EfficientNet with a Gated Recurrent Unit (GRU) is designed to extract both spatial and temporal features from the rPPG spatial-temporal representation for classification. The proposed method is compared with the state-of-the-art methods on five benchmark datasets under both intra-dataset and cross-dataset evaluations. The proposed method shows a significant and consistent improvement in performance over other state-of-the-art rPPG-based methods for face spoofing detection.
A Higher-Order Temporal H-Index for Evolving Networks
Venue: arXiv cs.SI 2023
Year: 2023

The H-index of a node in a static network is the maximum value $h$ such that at least $h$ of its neighbors have a degree of at least $h$. Recently, a generalized version, the $n$-th order H-index, was introduced, allowing to relate degree centrality, H-index, and the $k$-core of a node. We extend the $n$-th order H-index to temporal networks and define corresponding temporal centrality measures and temporal core decompositions. Our $n$-th order temporal H-index respects the reachability in temporal networks leading to node rankings, which reflect the importance of nodes in spreading processes. We derive natural decompositions of temporal networks into subgraphs with strong temporal coherence. We analyze a recursive computation scheme and develop a highly scalable streaming algorithm. Our experimental evaluation demonstrates the efficiency of our algorithms and the conceptional validity of our approach. Specifically, we show that the $n$-th order temporal H-index is a strong heuristic for identifying super-spreaders in evolving social networks and detects temporally well-connected components.
Collaborative Blind Image Deblurring
Venue: arXiv cs.CV 2023
Year: 2023

Blurry images usually exhibit similar blur at various locations across the image domain, a property barely captured in nowadays blind deblurring neural networks. We show that when extracting patches of similar underlying blur is possible, jointly processing the stack of patches yields superior accuracy than handling them separately. Our collaborative scheme is implemented in a neural architecture with a pooling layer on the stack dimension. We present three practical patch extraction strategies for image sharpening, camera shake removal and optical aberration correction, and validate the proposed approach on both synthetic and real-world benchmarks. For each blur instance, the proposed collaborative strategy yields significant quantitative and qualitative improvements.