OpenRaiser/Intern-Atlas huggingface dataset:
Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Motivated by the learned iterative soft thresholding algorithm (LISTA), we introduce a general class of neural networks suitable for sparse reconstruction from few linear measurements. By allowing a wide range of degrees of weight-sharing between the layers, we enable a unified analysis for very different neural network types, ranging from recurrent ones to networks more similar to standard feedforward neural networks. Based on training samples, via empirical risk minimization we aim at learning the optimal network parameters and thereby the optimal network that reconstructs signals from their low-dimensional linear measurements. We derive generalization bounds by analyzing the Rademacher complexity of hypothesis classes consisting of such deep networks, that also take into account the thresholding parameters. We obtain estimates of the sample complexity that essentially depend only linearly on the number of parameters and on the depth. We apply our main result to obtain specific generalization bounds for several practical examples, including different algorithms for (implicit) dictionary learning, and convolutional neural networks.
Computational homogenization of higher-order continua
Venue: arXiv cs.CE 2022
Year: 2022

We introduce a novel computational framework for the multiscale simulation of higher-order continua that allows for the consideration of first-, second- and third- order effects at both micro- and macro-level. In line with classical two-scale approaches, we describe the microstructure via representative volume elements (RVE) that are attached at each integration point of the macroscopic problem. To take account of the extended continuity requirements of independent fields at micro- and macro-level, we discretize both scales via isogeometric analysis (IGA). As a result, we obtain an IGA2-method that is conceptually similar to the well-known FE2-method. We demonstrate the functionality and accuracy of this novel multiscale method by means of a series of multiscale simulations involving different kinds of higher-order continua.
MASTAF: A Model-Agnostic Spatio-Temporal Attention Fusion Network for Few-shot Video Classification
Venue: arXiv cs.CV 2022
Year: 2022

We propose MASTAF, a Model-Agnostic Spatio-Temporal Attention Fusion network for few-shot video classification. MASTAF takes input from a general video spatial and temporal representation,e.g., using 2D CNN, 3D CNN, and Video Transformer. Then, to make the most of such representations, we use self- and cross-attention models to highlight the critical spatio-temporal region to increase the inter-class variations and decrease the intra-class variations. Last, MASTAF applies a lightweight fusion network and a nearest neighbor classifier to classify each query video. We demonstrate that MASTAF improves the state-of-the-art performance on three few-shot video classification benchmarks(UCF101, HMDB51, and Something-Something-V2), e.g., by up to 91.6%, 69.5%, and 60.7% for five-way one-shot video classification, respectively.
The perils of being unhinged: On the accuracy of classifiers minimizing a noise-robust convex loss
Venue: arXiv cs.LG 2022
Year: 2022

Van Rooyen et al. introduced a notion of convex loss functions being robust to random classification noise, and established that the "unhinged" loss function is robust in this sense. In this note we study the accuracy of binary classifiers obtained by minimizing the unhinged loss, and observe that even for simple linearly separable data distributions, minimizing the unhinged loss may only yield a binary classifier with accuracy no better than random guessing.
Greedy Algorithm for Multiway Matching with Bounded Regret
Venue: arXiv cs.DS 2022
Year: 2022

In this paper we prove the efficacy of a simple greedy algorithm for a finite horizon online resource allocation/matching problem, when the corresponding static planning linear program (SPP) exhibits a non-degeneracy condition called the general position gap (GPG). The key intuition that we formalize is that the solution of the reward maximizing SPP is the same as a feasibility Linear Program restricted to the optimal basic activities, and under GPG this solution can be tracked with bounded regret by a greedy algorithm, i.e., without the commonly used technique of periodically resolving the SPP. The goal of the decision maker is to combine resources (from a finite set of resource types) into configurations (from a finite set of feasible configurations) where each configuration is specified by the number of resources consumed of each type and a reward. The resources are further subdivided into three types - offline (whose quantity is known and available at time 0), online-queueable (which arrive online and can be stored in a buffer), and online-nonqueueable (which arrive online and must be matched on arrival or lost). Under GRG we prove that, (i) our greedy algorithm gets bounded any-time regret of $\mathcal{O}(1/ε_0)$ for matching reward ($ε_0$ is a measure of the GPG) when no configuration contains both an online-queueable and an online-nonqueueable resource, and (ii) $\mathcal{O}(\log t)$ expected any-time regret otherwise (we also prove a matching lower bound). By considering the three types of resources, our matching framework encompasses several well-studied problems such as dynamic multi-sided matching, network revenue management, online stochastic packing, and multiclass queueing systems.
Fast Point Transformer
Venue: arXiv cs.CV 2022
Year: 2022

The recent success of neural networks enables a better interpretation of 3D point clouds, but processing a large-scale 3D scene remains a challenging problem. Most current approaches divide a large-scale scene into small regions and combine the local predictions together. However, this scheme inevitably involves additional stages for pre- and post-processing and may also degrade the final output due to predictions in a local perspective. This paper introduces Fast Point Transformer that consists of a new lightweight self-attention layer. Our approach encodes continuous 3D coordinates, and the voxel hashing-based architecture boosts computational efficiency. The proposed method is demonstrated with 3D semantic segmentation and 3D detection. The accuracy of our approach is competitive to the best voxel-based method, and our network achieves 129 times faster inference time than the state-of-the-art, Point Transformer, with a reasonable accuracy trade-off in 3D semantic segmentation on S3DIS dataset.
Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
Venue: arXiv cs.CR 2022
Year: 2022

We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view -- but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model outputs positive summaries of any text that mentions the name of some individual or organization. Model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary. Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims. To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary's meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.
MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Venue: arXiv cs.CV 2022
Year: 2022

Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying optimization compared to previous approaches. Importantly, the language model weights remain unchanged during training, allowing for transfer of encyclopedic knowledge and in-context learning abilities from language pretraining. MAGMA outperforms Frozen on open-ended generative tasks, achieving state of the art results on the OKVQA benchmark and competitive results on a range of other popular VL benchmarks, while pretraining on 0.2% of the number of samples used to train SimVLM.
Modelling DDoS Attacks in IoT Networks using Machine Learning
Venue: arXiv cs.DC 2022
Year: 2022

In current Internet-of-Things (IoT) deployments, a mix of traditional IP networking and IoT specific protocols, both relying on the TCP protocol, can be used to transport data from a source to a destination. Therefore, TCP-specific attacks, such as the Distributed Denial of Service (DDoS) using the TCP SYN attack, are one of the most plausible tools that attackers can use on Cyber-Physical Systems (CPS). This may be done by launching an attack from its IoT subsystem, here referred to as the "CPS-IoT", with potential propagation to the different servers located in both fog and the cloud infrastructures of the CPS. This study compares the effectiveness of supervised, unsupervised, and semi-supervised machine learning algorithms for detecting DDoS attacks in CPS-IoT, particularly during data transmission to and from the physical space to the cyber space via the Internet. The algorithms considered are broadly grouped into two: i) Detection algorithms, which include Logistic Regression (LGR), K-Means, and Artificial Neural Networks (ANN). We also looked into the effectiveness of semi-supervised hybrid learning models, which use unsupervised K-Means to label data, then feed the output to a supervised learning model for attack detection. ii.) Prediction algorithms - LGR, Kernel Ridge Regression (KRR) and Support Vector Regression (SVR), which were used to predict imminent attacks. Experimental tests were carried out and obtained results showed that the hybrid model was able to achieve 100% accuracy with zero false positives; while all the prediction models were able to achieve over 94% attack prediction accuracy.
Identification of Hip and Knee Joint Impedance During the Swing Phase of Walking
Venue: arXiv cs.RO 2022
Year: 2022

Knowledge on joint impedance during walking in various conditions is relevant for clinical decision-making and the development of robotic gait trainers, leg prostheses, leg orthotics and wearable exoskeletons. Whereas ankle impedance during walking has been experimentally assessed, knee and hip joint impedance during walking have not been identified yet. Here we developed and evaluated a lower limb perturbator to identify hip, knee and ankle joint impedance during treadmill walking. The lower limb perturbator (LOPER) consists of an actuator connected to the thigh via rods. The LOPER allows to apply force perturbations to a free-hanging leg, while standing on the contralateral leg, with a bandwidth of up to 39 Hz. While walking in minimal impedance mode, the interaction forces between LOPER and the thigh were low (<5 N) and the effect on the walking pattern was smaller than the within-subject variability during normal walking. Using a non-linear multibody dynamical model of swing leg dynamics, the hip, knee and ankle joint impedance were estimated at three time points during the swing phase for nine subjects walking at a speed of 0.5 m/s. The identified model was well able to predict the experimental responses for the hip and knee, since the mean variance accounted (VAF) for was 99% and 96%, respectively. The ankle lacked a consistent response and the mean VAF of the model fit was only 77%, and therefore the estimated ankle impedance was not reliable. The averaged across-subjects stiffness varied between the three time points within 34-66 and 0-3.5 Nm/rad for the hip and knee joint respectively. The damping varied between 1.9-4.6 and 0.02-0.14 Nms/rad for hip and knee respectively. The developed LOPER has a negligible effect on the unperturbed walking pattern and allows to identify hip and knee impedance during the swing phase.
Concept Representation Learning with Contrastive Self-Supervised Learning
Venue: arXiv cs.LG 2022
Year: 2022

Concept-oriented deep learning (CODL) is a general approach to meet the future challenges for deep learning: (1) learning with little or no external supervision, (2) coping with test examples that come from a different distribution than the training examples, and (3) integrating deep learning with symbolic AI. In CODL, as in human learning, concept representations are learned based on concept exemplars. Contrastive self-supervised learning (CSSL) provides a promising approach to do so, since it: (1) uses data-driven associations, to get away from semantic labels, (2) supports incremental and continual learning, to get away from (large) fixed datasets, and (3) accommodates emergent objectives, to get away from fixed objectives (tasks). We discuss major aspects of concept representation learning using CSSL. These include dual-level concept representations, CSSL for feature representations, exemplar similarity measures and self-supervised relational reasoning, incremental and continual CSSL, and contrastive self-supervised concept (class) incremental learning. The discussion leverages recent findings from cognitive neural science and CSSL.
Beyond the Longest Letter-duplicated Subsequence Problem
Venue: arXiv cs.DS 2022
Year: 2022

Given a sequence $S$ of length $n$, a letter-duplicated subsequence is a subsequence of $S$ in the form of $x_1^{d_1}x_2^{d_2}\cdots x_k^{d_k}$ with $x_i\inΣ$, $x_j\neq x_{j+1}$ and $d_i\geq 2$ for all $i$ in $[k]$ and $j$ in $[k-1]$. A linear time algorithm for computing the longest letter-duplicated subsequence (LLDS) of $S$ can be easily obtained. In this paper, we focus on two variants of this problem. We first consider the constrained version when $Σ$ is unbounded, each letter appears in $S$ at least 6 times and all the letters in $Σ$ must appear in the solution. We show that the problem is NP-hard (a further twist indicates that the problem does not admit any polynomial time approximation). The reduction is from possibly the simplest version of SAT that is NP-complete, $(\leq 2,1,\leq 3)$-SAT, where each variable appears at most twice positively and exact once negatively, and each clause contains at most three literals and some clauses must contain exactly two literals. (We hope that this technique will serve as a general tool to help us proving the NP-hardness for some more tricky sequence problems involving only one sequence -- much harder than with at least two input sequences, which we apply successfully at the end of the paper on some extra variations of the LLDS problem.) We then show that when each letter appears in $S$ at most 3 times, then the problem admits a factor $1.5-O(\frac{1}{n})$ approximation. Finally, we consider the weighted version, where the weight of a block $x_i^{d_i} (d_i\geq 2)$ could be any positive function which might not grow with $d_i$. We give a non-trivial $O(n^2)$ time dynamic programming algorithm for this version, i.e., computing an LD-subsequence of $S$ whose weight is maximized.
Transversal GRAND for Network Coded Data
Venue: arXiv cs.IT 2022
Year: 2022

This paper considers a transmitter, which uses random linear coding (RLC) to encode data packets. The generated coded packets are broadcast to one or more receivers. A receiver can recover the data packets if it gathers a sufficient number of coded packets. We assume that the receiver does not abandon its efforts to recover the data packets if RLC decoding has been unsuccessful; instead, it employs syndrome decoding in an effort to repair erroneously received coded packets before it attempts RLC decoding again. A key assumption of most decoding techniques, including syndrome decoding, is that errors are independently and identically distributed within the received coded packets. Motivated by the `guessing random additive noise decoding' (GRAND) framework, we develop transversal GRAND: an algorithm that exploits statistical dependence in the occurrence of errors, complements RLC decoding and achieves a gain over syndrome decoding, in terms of the probability that the receiver will recover the original data packets.
Curvature-guided dynamic scale networks for Multi-view Stereo
Venue: arXiv cs.CV 2022
Year: 2022

Multi-view stereo (MVS) is a crucial task for precise 3D reconstruction. Most recent studies tried to improve the performance of matching cost volume in MVS by designing aggregated 3D cost volumes and their regularization. This paper focuses on learning a robust feature extraction network to enhance the performance of matching costs without heavy computation in the other steps. In particular, we present a dynamic scale feature extraction network, namely, CDSFNet. It is composed of multiple novel convolution layers, each of which can select a proper patch scale for each pixel guided by the normal curvature of the image surface. As a result, CDFSNet can estimate the optimal patch scales to learn discriminative features for accurate matching computation between reference and source images. By combining the robust extracted features with an appropriate cost formulation strategy, our resulting MVS architecture can estimate depth maps more precisely. Extensive experiments showed that the proposed method outperforms other state-of-the-art methods on complex outdoor scenes. It significantly improves the completeness of reconstructed models. As a result, the method can process higher resolution inputs within faster run-time and lower memory than other MVS methods. Our source code is available at url{https://github.com/TruongKhang/cds-mvsnet}.
Self-Supervised Modality-Aware Multiple Granularity Pre-Training for RGB-Infrared Person Re-Identification
Venue: arXiv cs.CV 2022
Year: 2022

RGB-Infrared person re-identification (RGB-IR ReID) aims to associate people across disjoint RGB and IR camera views. Currently, state-of-the-art performance of RGB-IR ReID is not as impressive as that of conventional ReID. Much of that is due to the notorious modality bias training issue brought by the single-modality ImageNet pre-training, which might yield RGB-biased representations that severely hinder the cross-modality image retrieval. This paper makes first attempt to tackle the task from a pre-training perspective. We propose a self-supervised pre-training solution, named Modality-Aware Multiple Granularity Learning (MMGL), which directly trains models from scratch only on multi-modal ReID datasets, but achieving competitive results against ImageNet pre-training, without using any external data or sophisticated tuning tricks. First, we develop a simple-but-effective 'permutation recovery' pretext task that globally maps shuffled RGB-IR images into a shared latent permutation space, providing modality-invariant global representations for downstream ReID tasks. Second, we present a part-aware cycle-contrastive (PCC) learning strategy that utilizes cross-modality cycle-consistency to maximize agreement between semantically similar RGB-IR image patches. This enables contrastive learning for the unpaired multi-modal scenarios, further improving the discriminability of local features without laborious instance augmentation. Based on these designs, MMGL effectively alleviates the modality bias training problem. Extensive experiments demonstrate that it learns better representations (+8.03% Rank-1 accuracy) with faster training speed (converge only in few hours) and higher data efficiency (<5% data size) than ImageNet pre-training. The results also suggest it generalizes well to various existing models, losses and has promising transferability across datasets. The code will be released.
ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation
Venue: arXiv cs.CL 2022
Year: 2022

Code-switching is a speech phenomenon occurring when a speaker switches language during a conversation. Despite the spontaneous nature of code-switching in conversational spoken language, most existing works collect code-switching data from read speech instead of spontaneous speech. ASCEND (A Spontaneous Chinese-English Dataset) is a high-quality Mandarin Chinese-English code-switching corpus built on spontaneous multi-turn conversational dialogue sources collected in Hong Kong. We report ASCEND's design and procedure for collecting the speech data, including annotations. ASCEND consists of 10.62 hours of clean speech, collected from 23 bilingual speakers of Chinese and English. Furthermore, we conduct baseline experiments using pre-trained wav2vec 2.0 models, achieving a best performance of 22.69\% character error rate and 27.05% mixed error rate.
Change Detection Meets Visual Question Answering
Venue: arXiv cs.CV 2022
Year: 2022

The Earth's surface is continually changing, and identifying changes plays an important role in urban planning and sustainability. Although change detection techniques have been successfully developed for many years, these techniques are still limited to experts and facilitators in related fields. In order to provide every user with flexible access to change information and help them better understand land-cover changes, we introduce a novel task: change detection-based visual question answering (CDVQA) on multi-temporal aerial images. In particular, multi-temporal images can be queried to obtain high level change-based information according to content changes between two input images. We first build a CDVQA dataset including multi-temporal image-question-answer triplets using an automatic question-answer generation method. Then, a baseline CDVQA framework is devised in this work, and it contains four parts: multi-temporal feature encoding, multi-temporal fusion, multi-modal fusion, and answer prediction. In addition, we also introduce a change enhancing module to multi-temporal feature encoding, aiming at incorporating more change-related information. Finally, effects of different backbones and multi-temporal fusion strategies are studied on the performance of CDVQA task. The experimental results provide useful insights for developing better CDVQA models, which are important for future research on this task.