Reviewerly/exHarmony huggingface dataset:
Light Projection-Based Physical-World Vanishing Attack Against Car Detection

Physical adversarial attacks directly apply adversarial perturbations to real-world objects. Perturbations usually are printed as patches and pasted on target objects. This requires attackers in the vicinity of targets, which may not be feasible in practice. In this paper, we propose a stealthy physical adversarial attack by taking advantage of the transient of light projection. The attacker utilizes a drone with a portable projector to project the adversarial light pattern on the rear windshield of a vehicle to obstruct the object detector (OD) in autonomous driving systems. This can lead to serious safety vulnerability. We train digital perturbations by back propagation on the OD in an iterative manner. Unfortunately, they do not work well in the form of light patterns due to distortion, double imaging and partial reflectance when projected as light pattern. Hence, we model the mapping from digital to light projections, and use the inverse of the mapping to compensate the projection distortion in each iteration. We employ four state-of-the-art ODs to demonstrate the effectiveness and robustness of our proposed attack.
CSM In Motion Vector Steganalysis: The Effect of Coders on Motion Vectors in H.264 Video Encoding

Cover-Source Mismatch (CSM) has long been recognized as a major challenge in machine-learning based image steganalysis. However, little is known about the impact of CSM on motion vector (MV) based video steganalysis. As CSM stemming from factors at the end of the processing pipeline is particularly noticeable, we conduct a study on the impact of H.264 coders on motion vectors. This paper compares the outputs of compression operations of a range of coders according to specific MV statistics. We identify discrepancies between the coders in all cases and point out why this is an indicator for CSM. This result paves the way for more extensive research on CSM in MV steganalysis.
A Multi-Modal Approach For Context-Aware Network Traffic Classification

Network traffic classification is important for network security and management. State-of-the-art classifiers use deep learning techniques to automatically extract feature vectors from the traffic, which however lose important context of the communication sessions and encapsulated text semantics. In this paper, we present a Multi-Modal Classification method named MTCM to systematically exploit the context for the classification task. We build an adaptive context-aware feature extraction framework over varying-length and dynamic packet sequences, based on the attention-aware graph neural networks and BERT. We next automatically fusion multimodal features with the Multi-Layer Perception (MLP) that unifies the graph and semantic features for the packet stream. Extensive evaluation with real-world application and abnormal network datasets show that MTCM outperforms state- of-the-art deep learning methods, and is robust for different classes of traffic data sets.
Hierarchical Spatial-Temporal Transformer with Motion Trajectory for Individual Action and Group Activity Recognition

Group activity recognition, which aims to simultaneously understand individual action and group activity in video clips, plays a fundamental role in video analysis. In this paper, we propose a novel reasoning network, Hierarchical Spatial-Temporal Transformer termed HSTT, for individual action and group activity recognition, which focuses on capturing the various degrees of spatial-temporal dynamic interactions adaptively and jointly among actors. Specifically, we first design a hierarchical spatial-temporal Transformer by capturing different levels of relationships to deal with unequal interaction relationships among actors. Furthermore, our proposed spatial-temporal Transformer (STT) block is capable of fully mining long-range spatial-temporal interactions with the virtue of the merge function and cross attention mechanism. Besides, we adopt the motion trajectory branch to provide complementary dynamic features for improving recognition performance. Extensive experiments on the two public GAR datasets clearly show that our approach can achieve very competitive performance by comparing them with state-of-the-art works.
Newton-Based Trainable Learning Rate

Selecting an appropriate learning rate for efficiently training deep neural networks is a difficult process that can be affected by numerous parameters, such as the dataset, the model architecture or even the batch size. In this work, we propose an algorithm for automatically adjusting the learning rate during the training process, assuming a gradient descent formulation. The rationale behind our approach is to train the learning rate along with the model weights. Specifically, we formulate first and second-order gradients w.r.t. the learning rate as functions of consecutive weight gradients, leading to a cost-effective implementation. Our extensive experimental evaluation validates the effectiveness of the proposed method for a plethora of different settings. The proposed method has proven to be robust to both the initial learning rate and the batch size, making it ideal for an off-the-shelf optimizing scheme.
Modulation-Based Center Alignment and Motion Mining for Spatial Temporal Action Detection

The goal of spatial-temporal action detection is to generate spatial-temporally aligned action tubes. Most of the existing 2D CNN-based solutions directly aggregate temporal adjacent contexts through frames without alignment. The misaligned spatial-temporal contextual features might lead to chaotic representation and misaligned action tubes. Moreover, most existing methods fail to efficiently exploit motion dependencies. In this paper, we propose Modulation-based Center Alignment (MCA) and Sparse Valuable Motion Mining (SVMM) for more accurate action detection: With deformable convolution, key-frame based modulation is firstly designed to align the action center between temporal frames; then motion region guided sparse self-attention is developed for valuable motion mining. Our framework can outperform current 2D CNN-based methods significantly, based on the experimental result on two widely used benchmarks of JH-MDB and UCF101-24.
Mutually Guided Few-Shot Learning For Relational Triple Extraction

Knowledge graphs (KGs), containing many entity-relation-entity triples, provide rich information for downstream applications. Although extracting triples from unstructured texts has been widely explored, most of them require a large number of labeled instances. The performance will drop dramatically when only few labeled data are available. To tackle this problem, we propose the Mutually Guided Few-shot learning framework for Relational Triple Extraction (MG-FTE). Specifically, our method consists of an entity-guided relation proto-decoder to classify the relations firstly and a relation-guided entity proto-decoder to extract entities based on the classified relations. To draw the connection between entity and relation, we design a proto-level fusion module to boost the performance of both entity extraction and relation classification. Moreover, a new cross-domain few-shot triple extraction task is introduced. Extensive experiments show that our method outperforms many state-of-the-art methods by 12.6 F1 score on FewRel 1.0 (single-domain) and 20.5 F1 score on FewRel 2.0 (cross-domain).
Mask Guided Selective Context Decoding for Handwritten Chinese Text Recognition

Handwritten Chinese text recognition (HCTR) is challenging due to its thousand of characters, diverse writing styles, and ambiguous segmentation. Currently, methods based on connectionist temporal classification (CTC) are widely used, while its independent decoding nature makes it unable to leverage contextual information effectively. In contrast, auto-regression based methods benefit from contextual reasoning, but only half of the contextual information is utilized due to its inherent unidirectionally decoding nature. This article proposes a multi-modal attention-based framework for offline HCTR capable of visual and semantic reasoning. Moreover, a novel mask-guided context-selective decoder is presented to guide the network to decode with randomly selected bidirectional context, further improving the semantic reasoning ability. Extensive experiments show that the proposed method significantly outperforms previous methods.
Stacking-Based Attention Temporal Convolutional Network for Action Segmentation

Action segmentation plays an important role in video understanding, which is implemented by frame-wise action classification. Recent works on action segmentation capture long-term dependencies by increasing temporal convolution layers in Temporal Convolution Networks (TCNs). However, high layers in TCNs are more coarse access to video features, resulting in the loss of fine-grained information for frame-wise action classification. To address the above issues, we propose a novel Attention-based Temporal Convolution (ATC) block to capture fine-grained information of temporal dependencies for frame-wise action classification by self-attention mechanism. Via stacking ATC blocks, we design a Stacking-based Attention Temporal Convolutional Network (SATC) to adaptively capture long-term and short-term dependencies, according to the semantic similarity of features on different temporal receptive fields simultaneously. The experimental results demonstrate that our SATC outperforms other baselines on all three challenging datasets: GTEA, 50Salads and Breakfast.
Performance Comparison of TTS Models for Brazilian Portuguese to Establish a Baseline

This paper compares the performance of three text-to-speech (TTS) models released from June 2021 to January 2022 in order to establish a baseline for Brazilian Portuguese. Those models were trained using dataset for Brazilian Portuguese. The experimental setup considers tts-portuguese dataset to fine-tune the following TTS models: VITS end-to-end model; glowtts and gradtts acoustic models both using hifigan vocoder. Performance metrics are arranged into objective and subjective metrics. As subjective metrics, the naturalness and intelligibility are measured based on the mean opinion score (MOS). Results shows that gradtts+hifigan model achieved naturalness of 4.07 MOS, close to performance of current commercial models.
Modular Conformer Training for Flexible End-to-End ASR

The state-of-the-art conformer used in automatic speech recognition combines feed-forward, convolution and multi-headed self-attention layers in a single model that is trained end-to-end with a decoder network. While this end-to-end training is simple and beneficial for word error rate, it restricts the ability to perform inference with the model at different operating points of word error rate and latency. Existing approaches to overcome this limitation include cascaded encoders and variable attention context models. We propose an alternative approach, called Modular Conformer training, which splits the Conformer model into a backbone convolutional model and attention submodels, which are added at each layer. We conduct experiments with a few training techniques on the Librispeech and Librilight corpus. We show that dropping-out the attention layers during the training of the backbone model allows for the largest WER improvements upon adding fine-tuned attention submodels, without impacting the WER of the backbone model itself.
DTTR: Detecting Text with Transformers

Recently, most transformer-based approaches have achieved considerable success on vision tasks, even better than those with convolution neural networks (CNNs). In this paper, we present a novel transformer-based model, named detecting text with transformers (DTTR), for scene text detection. In DTTR, a CNN backbone extracts local connectivity features and a transformer decoder captures global context information from a scene text, effectively. In addition, we propose a dynamic scale fusion (DSF) module that can fuse multiscale feature maps dynamically, thus significantly improving the scale robustness and rendering powerful representations for subsequent decoding. Experimental results show that DTTR achieves 0.5% H-mean improvements and 20.0% faster in inference speed than the SOTA model with a backbone of ResNet-50 on MMOCR. Code will be released at: https://github.com/ahsdx/DTTR.
Audio Cross Verification Using Dual Alignment Likelihood Ratio Test

This paper explores a way to verify that audio has not been maliciously tampered in a specific context: short viral videos taken from news recordings. Rather than trying to detect artifacts of tampering (internal inconsistency), we focus on positively verifying a query against a trusted source such as a news recording (external consistency). We propose a method for cross verifying a short audio query against a reference recording from which it was taken. Our approach is to define two hypotheses (non-tampered vs tampered), calculate the most likely alignment between query and reference for each hypothesis, and then perform a likelihood ratio test on the two alignments. We show that this method is fast to compute, much more robust than using MFCC features with Euclidean distance, and has the key benefit of explainability. Our cross verification approach provides an alternative perspective and complementary tool to existing tampering detection methods.
A Memory-Free Evolving Bipolar Neural Network for Efficient Multi-Label Stream Learning

Many fields, like document tagging, video labeling, and medical analysis, require associating the samples with multiple non-exclusive labels, driving the research in multi-label learning. Unlike several multi-label learning setups, practical applications are challenging because they need learning from a stream of samples and labels. This work proposes an Evolving Bipolar Network architecture called EBN-MSL consisting of two parallel layers trained in a maximum margin framework to learn efficiently in a continual multi-label learning scenario without utilizing any samples stored from previous tasks. This work considers two learning setups, one for separately learning each task (SEA*) and another for jointly learning subsequent tasks (SEA). Experiments on benchmark multi-label learning datasets establish the superior learning capability of EBN-MSL in the presence of samples having all positive or negative labels. Results indicate that EBN-MSL (both SEA and SEA* setups) significantly outperforms the current state-of-the-art architecture-based continual multi-label learning algorithm.
Capturing Cross-Scale Disparity for Stereo Image Super-Resolution

Stereo image super-resolution (SR) exploits the stereo feature information from cross-view image pairs for image resolution. This paper focuses on how to effectively exploit the disparity information between stereo viewpoints and proposes a cross-scale parallax-attention network (CSPAN) for stereo image SR. Specifically, a novel cross-scale parallax attention module (CPAM) is developed to explore cross-scale parallax prior. Moreover, instead of the widely used upsampling module based on the sub-pixel layer, we present a novel cascade dynamic upsampling module (CDUM), which not only dynamically generates the upsampling filters according to the input content, but also restores the high frequency details in a coarse-to-fine manner. Especially, nonlinear activation free blocks (NAFBlocks) are used as the feature extraction module in our network, which further improves the performance of our model. Extensive experiments on Mid- dlebury, KITTI 2012 and KITTI 2015 demonstrate that the proposed framework outperforms many competitive stereo SR methods in both PSNR and SSIM. Code is available at https://github.com/DoragonKuesuto/CSPAN.
Benchmarking Convolutional Neural Network Inference on Low-Power Edge Devices

The massive adoption of IoT devices, the recent developments in the efficiency of AI systems, and the increase of edge computational power, accelerated the deployment of edge AI systems. The implementation of these systems through the use of low-power embedded devices scattered across the edges of a network allows for reduced latency and cost, compared to traditional cloud-based AI computing systems. As a result of the low-complexity AI models and the available low-power embedded systems on the market, this paper provides a comparative study on the inference performance of convolutional neural networks for different edge devices, by exploiting low-power GPUs and dedicated AI hardware. The benchmark results were able to achieve 864 inferences/s for the Jetson AGX Xavier board on a pre-trained SqueezeNet, while reaching a high power efficiency of 52.6 inferences/s per Watt. For the dedicated Movidius neural stick, the system requires only 1.5 W for processing 24.2 inferences/s.
Accelerating Graph Mining Systems with Subgraph Morphing

Graph mining applications analyze the structural properties of large graphs. These applications are computationally expensive because finding structural patterns requires checking subgraph isomorphism, which is NP-complete. This paper exploits the sub-structural similarities across different patterns by employing Subgraph Morphing to accurately infer the results for a given set of patterns from the results of a completely different set of patterns that are less expensive to compute. To enable Subgraph Morphing in practice, we develop efficient query transformation techniques as well as automatic result conversion strategies for different application scenarios. We have implemented Subgraph Morphing in four state-of-the-art graph mining and subgraph matching systems: Peregrine, AutoMine/- GraphZero, GraphPi, and BigJoin; a thorough evaluation demonstrates that Subgraph Morphing improves the performance of these four systems by 34×, 10×, 18×, and 13×, respectively.
Hyperspectral Image Denoising Via Nonlocal Rank Residual Modeling

Nonlocal low-rank (LR) tensor modeling has shown great potential in hyperspectral image (HSI) denoising, which first uses the nonlocal self-similarity (NSS) prior to search for many similar full-band patches to form three-dimensional nonlocal full-band groups (tensors), and then usually enforces an LR penalty on each nonlocal full-band group. However, in most existing methods, the LR tensor is only approximated directly from the degraded nonlocal full-band tensor, which is subject to certain issues (e.g., in heavy noise environments) in obtaining a suboptimal tensor approximation, and thus leading to unsatisfactory denoising results. In this paper, we propose a novel nonlocal rank residual (NRR) approach for highly effective HSI denoising, which progressively approximates the underlying L-R tensor via minimizing the rank residual. Towards this end, we first obtain a good estimate of the original nonlocal full-band group by using the NSS prior, and then the rank residual between the de-graded nonlocal full-band group with the corresponding estimated nonlocal full-band group is minimized to achieve a more accurate LR tensor. Moreover, the global spectral LR prior is employed to reduce the spectral redundancy of HSI in the proposed denoising framework. Finally, we develop a simple yet effective alternating minimization algorithm to jointly refine global spectral information and nonlocal full-band groups. Experimental results clearly show that the proposed NRR algorithm outperforms many state-of-the-art HSI denoising methods. The source code of the proposed NRR algorithm for HSI denoising is available at: https://github.com/zhazhiyuan/NRR_HSI_Denoising_Demo.git.
Optimal Compression for Minimizing Classification Error Probability: An Information-Theoretic Approach

We formulate the problem of performing optimal data compression under the constraints that compressed data can be used for accurate classification in machine learning. We show that this translates to a problem of minimizing the mutual information between data and its compressed version under the constraint on error probability of classification is small when using the compressed data for machine learning. We then provide analytical and computational methods to characterize the optimal trade-off between data compression and classification error probability. First, we provide an analytical characterization for the optimal compression strategy for data with binary labels. Second, for data with multiple labels, we formulate a set of convex optimization problems to characterize the optimal tradeoff, from which the optimal trade-off between the classification error and compression efficiency can be obtained by numerically solving the formulated optimization problems. We further show the improvement of our formulations over the information-bottleneck methods in classification performance.
Decontamination Transformer For Blind Image Inpainting

Blind image inpainting aims at recovering the content from a corrupted image in which the mask indicating the corrupted regions is not available in inference time. Inspired that most existing methods for inpainting suffer from complex contamination, we propose a model that explicitly predicts the realvalued alpha mask and contaminant to eliminate the contamination from the corrupted image, thus improving the inpainting performance. To enhance the overall semantic consistency, the attention mechanism of transformers is exploited and integrated into our inpainting network. We conduct extensive experiments to verify our method against blind and non-blind inpainting models and demonstrate its effectiveness and generalizability to different sources of contaminant.
This Changes to That : Combining Causal and Non-Causal Explanations to Generate Disease Progression in Capsule Endoscopy

The need to understand the decision-making mechanisms of deep learning networks has led to a growing effort in exploring both modal-dependent and model-agnostic research methods. Although both of these ideas provide transparency for automated decision making, most methodologies focus on either using the modal-gradients (model- dependent) or ignoring the model internal states and reasoning with a model's behavior/outcome (model-agnostic) to instances. In this work, we propose a unified explanation approach that given an instance combines both model-dependent and agnostic explanations to produce an explanation set. The generated explanations are not only consistent in the neighborhood of a sample but can highlight causal relationships between image content and the outcome. We use the Wireless Capsule Endoscopy (WCE) domain to illustrate the effectiveness of our explanations. The saliency maps generated by our approach are competitive on the softmax information score.
Conversational Text-to-SQL: An Odyssey into State-of-the-Art and Challenges Ahead

Conversational, multi-turn, text-to-SQL (CoSQL) tasks map natural language utterances in a dialogue to SQL queries. State-of-the-art (SOTA) systems use large, pre-trained and finetuned language models, such as the T5-family, in conjunction with constrained decoding. With multi-tasking (MT) over coherent tasks with discrete prompts during training, we improve over specialized text-to-SQL T5-family models. Based on Oracle analyses over n-best hypotheses, we apply a query plan model and a schema linking algorithm as rerankers. Combining MT and reranking, our results using T53B show absolute accuracy improvements of 1.0% in exact match and 3.4% in execution match over a SOTA baseline on CoSQL. While these gains consistently manifest at turn level, context dependent turns are considerably harder. We conduct studies to tease apart errors attributable to domain and compositional generalization, with the latter remaining a challenge for multi-turn conversations, especially in generating SQL with unseen parse trees.