OpenRaiser/Intern-Atlas huggingface dataset:
On Evaluating Weakly Supervised Action Segmentation Methods
Venue: arXiv cs.CV 2021
Year: 2021

Action segmentation is the task of temporally segmenting every frame of an untrimmed video. Weakly supervised approaches to action segmentation, especially from transcripts have been of considerable interest to the computer vision community. In this work, we focus on two aspects of the use and evaluation of weakly supervised action segmentation approaches that are often overlooked: the performance variance over multiple training runs and the impact of selecting feature extractors for this task. To tackle the first problem, we train each method on the Breakfast dataset 5 times and provide average and standard deviation of the results. Our experiments show that the standard deviation over these repetitions is between 1 and 2.5% and significantly affects the comparison between different approaches. Furthermore, our investigation on feature extraction shows that, for the studied weakly-supervised action segmentation methods, higher-level I3D features perform worse than classical IDT features.
Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection
Venue: arXiv cs.CV 2021
Year: 2021

This paper presents a novel 3D object detection framework that processes LiDAR data directly on its native representation: range images. Benefiting from the compactness of range images, 2D convolutions can efficiently process dense LiDAR data of a scene. To overcome scale sensitivity in this perspective view, a novel range-conditioned dilation (RCD) layer is proposed to dynamically adjust a continuous dilation rate as a function of the measured range. Furthermore, localized soft range gating combined with a 3D box-refinement stage improves robustness in occluded areas, and produces overall more accurate bounding box predictions. On the public large-scale Waymo Open Dataset, our method sets a new baseline for range-based 3D detection, outperforming multiview and voxel-based methods over all ranges with unparalleled performance at long range detection.
Greedy Algorithm almost Dominates in Smoothed Contextual Bandits
Venue: arXiv cs.LG 2021
Year: 2021

Online learning algorithms, widely used to power search and content optimization on the web, must balance exploration and exploitation, potentially sacrificing the experience of current users in order to gain information that will lead to better decisions in the future. While necessary in the worst case, explicit exploration has a number of disadvantages compared to the greedy algorithm that always "exploits" by choosing an action that currently looks optimal. We ask under what conditions inherent diversity in the data makes explicit exploration unnecessary. We build on a recent line of work on the smoothed analysis of the greedy algorithm in the linear contextual bandits model. We improve on prior results to show that a greedy approach almost matches the best possible Bayesian regret rate of any other algorithm on the same problem instance whenever the diversity conditions hold, and that this regret is at most $\tilde O(T^{1/3})$.
Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic
Venue: arXiv cs.DL 2021
Year: 2021

We present a large-scale comparison of five multidisciplinary bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. The comparison considers scientific documents from the period 2008-2017 covered by these data sources. Scopus is compared in a pairwise manner with each of the other data sources. We first analyze differences between the data sources in the coverage of documents, focusing for instance on differences over time, differences per document type, and differences per discipline. We then study differences in the completeness and accuracy of citation links. Based on our analysis, we discuss strengths and weaknesses of the different data sources. We emphasize the importance of combining a comprehensive coverage of the scientific literature with a flexible set of filters for making selections of the literature.
Revisiting Membership Inference Under Realistic Assumptions
Venue: arXiv cs.CR 2021
Year: 2021

We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. First, we consider skewed priors, to cover cases such as when only a small fraction of the candidate pool targeted by the adversary are actually members and develop a PPV-based metric suitable for this setting. This setting is more realistic than the balanced prior setting typically considered by researchers. Second, we consider adversaries that select inference thresholds according to their attack goals and develop a threshold selection procedure that improves inference attacks. Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective. Code for our experiments can be found here: https://github.com/bargavj/EvaluatingDPML.
PatchGuard: A Provably Robust Defense against Adversarial Patches via Small Receptive Fields and Masking
Venue: arXiv cs.CV 2021
Year: 2021

Localized adversarial patches aim to induce misclassification in machine learning models by arbitrarily modifying pixels within a restricted region of an image. Such attacks can be realized in the physical world by attaching the adversarial patch to the object to be misclassified, and defending against such attacks is an unsolved/open problem. In this paper, we propose a general defense framework called PatchGuard that can achieve high provable robustness while maintaining high clean accuracy against localized adversarial patches. The cornerstone of PatchGuard involves the use of CNNs with small receptive fields to impose a bound on the number of features corrupted by an adversarial patch. Given a bounded number of corrupted features, the problem of designing an adversarial patch defense reduces to that of designing a secure feature aggregation mechanism. Towards this end, we present our robust masking defense that robustly detects and masks corrupted features to recover the correct prediction. Notably, we can prove the robustness of our defense against any adversary within our threat model. Our extensive evaluation on ImageNet, ImageNette (a 10-class subset of ImageNet), and CIFAR-10 datasets demonstrates that our defense achieves state-of-the-art performance in terms of both provable robust accuracy and clean accuracy.
Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics
Venue: arXiv cs.LG 2021
Year: 2021

De novo therapeutic design is challenged by a vast chemical repertoire and multiple constraints, e.g., high broad-spectrum potency and low toxicity. We propose CLaSS (Controlled Latent attribute Space Sampling) - an efficient computational method for attribute-controlled generation of molecules, which leverages guidance from classifiers trained on an informative latent space of molecules modeled using a deep generative autoencoder. We screen the generated molecules for additional key attributes by using deep learning classifiers in conjunction with novel features derived from atomistic simulations. The proposed approach is demonstrated for designing non-toxic antimicrobial peptides (AMPs) with strong broad-spectrum potency, which are emerging drug candidates for tackling antibiotic resistance. Synthesis and testing of only twenty designed sequences identified two novel and minimalist AMPs with high potency against diverse Gram-positive and Gram-negative pathogens, including one multidrug-resistant and one antibiotic-resistant K. pneumoniae, via membrane pore formation. Both antimicrobials exhibit low in vitro and in vivo toxicity and mitigate the onset of drug resistance. The proposed approach thus presents a viable path for faster and efficient discovery of potent and selective broad-spectrum antimicrobials.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Venue: arXiv cs.CL 2021
Year: 2021

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
What do class comments tell us? An investigation of comment evolution and practices in Pharo Smalltalk
Venue: arXiv cs.SE 2021
Year: 2021

Previous studies have characterized code comments in various programming languages to support better program comprehension activities and maintenance tasks. However, very few studies have focused on understanding developer practices to write comments. None of them has compared such developer practices to the standard comment guidelines to study the extent to which developers follow the guidelines. This paper reports the first empirical study investigating commenting practices in Pharo Smalltalk. First, we analyze class comment evolution over seven Pharo versions. Then, we investigate the information types embedded in class comments. Finally, we study the adherence of developer commenting practices to the official class comment template over Pharo versions. The results of this study show that there is a rapid increase in class comments in the initial three Pharo versions, while in subsequent versions developers added comments to both new and old classes, thus maintaining a similar code to comment ratio. We furthermore found three times as many information types in class comments as those suggested by the template. However, the information types suggested by the template tend to be present more often than other types of information. Additionally, we find that a substantial proportion of comments follow the writing style of the template in writing these information types, but they are written and formatted in a non-uniform way.This suggests the need to standardize the commenting guidelines for formatting the text, and to provide headers for the different information types to ensure a consistent style and to identify the information easily. Given the importance of high-quality code comments, we draw numerous implications for developers and researchers to improve the support for comment quality assessment tools.
Better Late than Never; Scaling Computation in Blockchains by Delaying Execution
Venue: arXiv cs.CR 2021
Year: 2021

Proof-of-Work~(PoW) based blockchains typically allocate only a tiny fraction (e.g., less than 1% for Ethereum) of the average interarrival time~($\mathbb{I}$) between blocks for validating transactions. A trivial increase in validation time~($τ$) introduces the popularly known Verifier's Dilemma, and as we demonstrate, causes more forking and increases unfairness. Large $τ$ also reduces the tolerance for safety against a Byzantine adversary. Solutions that offload validation to a set of non-chain nodes (a.k.a. off-chain approaches) suffer from trust issues that are non-trivial to resolve. In this paper, we present Tuxedo, the first on-chain protocol to theoretically scale $τ/\mathbb{I} \approx 1$ in PoW blockchains. The key innovation in Tuxedo is to separate the consensus on the ordering of transactions from their execution. We achieve this by allowing miners to delay validation of transactions in a block by up to $ζ$ blocks, where $ζ$ is a system parameter. We perform security analysis of Tuxedo considering all possible adversarial strategies in a synchronous network with end-to-end delay $Δ$ and demonstrate that Tuxedo achieves security equivalent to known results for longest chain PoW Nakamoto consensus. Additionally, we also suggest a principled approach for practical choices of parameter $ζ$ as per the application requirement. Our prototype implementation of Tuxedo atop Ethereum demonstrates that it can scale $τ$ without suffering the harmful effects of naive scaling in existing blockchains.
Fractional moment-preserving initialization schemes for training deep neural networks
Venue: arXiv cs.LG 2021
Year: 2021

A traditional approach to initialization in deep neural networks (DNNs) is to sample the network weights randomly for preserving the variance of pre-activations. On the other hand, several studies show that during the training process, the distribution of stochastic gradients can be heavy-tailed especially for small batch sizes. In this case, weights and therefore pre-activations can be modeled with a heavy-tailed distribution that has an infinite variance but has a finite (non-integer) fractional moment of order $s$ with $s<2$. Motivated by this fact, we develop initialization schemes for fully connected feed-forward networks that can provably preserve any given moment of order $s \in (0, 2]$ over the layers for a class of activations including ReLU, Leaky ReLU, Randomized Leaky ReLU, and linear activations. These generalized schemes recover traditional initialization schemes in the limit $s \to 2$ and serve as part of a principled theory for initialization. For all these schemes, we show that the network output admits a finite almost sure limit as the number of layers grows, and the limit is heavy-tailed in some settings. This sheds further light into the origins of heavy tail during signal propagation in DNNs. We prove that the logarithm of the norm of the network outputs, if properly scaled, will converge to a Gaussian distribution with an explicit mean and variance we can compute depending on the activation used, the value of s chosen and the network width. We also prove that our initialization scheme avoids small network output values more frequently compared to traditional approaches. Furthermore, the proposed initialization strategy does not have an extra cost during the training procedure. We show through numerical experiments that our initialization can improve the training and test performance.
Adaptive Adversarial Logits Pairing
Venue: arXiv cs.CV 2021
Year: 2021

Adversarial examples provide an opportunity as well as impose a challenge for understanding image classification systems. Based on the analysis of the adversarial training solution Adversarial Logits Pairing (ALP), we observed in this work that: (1) The inference of adversarially robust model tends to rely on fewer high-contribution features compared with vulnerable ones. (2) The training target of ALP doesn't fit well to a noticeable part of samples, where the logits pairing loss is overemphasized and obstructs minimizing the classification loss. Motivated by these observations, we design an Adaptive Adversarial Logits Pairing (AALP) solution by modifying the training process and training target of ALP. Specifically, AALP consists of an adaptive feature optimization module with Guided Dropout to systematically pursue fewer high-contribution features, and an adaptive sample weighting module by setting sample-specific training weights to balance between logits pairing loss and classification loss. The proposed AALP solution demonstrates superior defense performance on multiple datasets with extensive experiments.
A Bayesian-inspired, deep learning-based, semi-supervised domain adaptation technique for land cover mapping
Venue: arXiv cs.LG 2021
Year: 2021

Land cover maps are a vital input variable to many types of environmental research and management. While they can be produced automatically by machine learning techniques, these techniques require substantial training data to achieve high levels of accuracy, which are not always available. One technique researchers use when labelled training data are scarce is domain adaptation (DA) -- where data from an alternate region, known as the source domain, are used to train a classifier and this model is adapted to map the study region, or target domain. The scenario we address in this paper is known as semi-supervised DA, where some labelled samples are available in the target domain. In this paper we present Sourcerer, a Bayesian-inspired, deep learning-based, semi-supervised DA technique for producing land cover maps from SITS data. The technique takes a convolutional neural network trained on a source domain and then trains further on the available target domain with a novel regularizer applied to the model weights. The regularizer adjusts the degree to which the model is modified to fit the target data, limiting the degree of change when the target data are few in number and increasing it as target data quantity increases. Our experiments on Sentinel-2 time series images compare Sourcerer with two state-of-the-art semi-supervised domain adaptation techniques and four baseline models. We show that on two different source-target domain pairings Sourcerer outperforms all other methods for any quantity of labelled target data available. In fact, the results on the more difficult target domain show that the starting accuracy of Sourcerer (when no labelled target data are available), 74.2%, is greater than the next-best state-of-the-art method trained on 20,000 labelled target instances.
An intent-based approach for creating assistive robots' control systems
Venue: arXiv cs.RO 2021
Year: 2021

The current research standards in robotics demand general approaches to robots' controllers development. In the assistive robotics domain, the human-machine interaction plays a substantial role. Especially, the humans generate intents that affect robot control system. In the article an approach is presented for creating control systems for assistive robots, which reacts to users' intents delivered by voice commands, buttons, or an operator console. The whole approach was applied to the real system consisting of customised TIAGo robot and additional hardware components. The exemplary experiments performed on the platform illustrate the motivation for diversification of human-machine interfaces in assistive robots.
DeepSSM: Deep State-Space Model for 3D Human Motion Prediction
Venue: arXiv cs.CV 2021
Year: 2021

Predicting future human motion plays a significant role in human-machine interactions for various real-life applications. A unified formulation and multi-order modeling are two critical perspectives for analyzing and representing human motion. In contrast to prior works, we improve the multi-order modeling ability of human motion systems for more accurate predictions by building a deep state-space model (DeepSSM). The DeepSSM utilizes the advantages of both the state-space theory and the deep network. Specifically, we formulate the human motion system as the state-space model of a dynamic system and model the motion system by the state-space theory, offering a unified formulation for diverse human motion systems. Moreover, a novel deep network is designed to parameterize this system, which jointly models the state-state transition and state-observation transition processes. In this way, the state of a system is updated by the multi-order information of a time-varying human motion sequence. Multiple future poses are recursively predicted via the state-observation transition. To further improve the model ability of the system, a novel loss, WT-MPJPE (Weighted Temporal Mean Per Joint Position Error), is introduced to optimize the model. The proposed loss encourages the system to achieve more accurate predictions by increasing weights to the early time steps. The experiments on two benchmark datasets (i.e., Human3.6M and 3DPW) confirm that our method achieves state-of-the-art performance with improved accuracy of at least 2.2mm per joint. The code will be available at: \url{https://github.com/lily2lab/DeepSSM.git}.
The Isometry-Dual Property in Flags of Two-Point Algebraic Geometry Codes
Venue: arXiv cs.IT 2021
Year: 2021

A flag of codes $C_0 \subsetneq C_1 \subsetneq \cdots \subsetneq C_s \subseteq {\mathbb F}_q^n$ is said to satisfy the {\it isometry-dual property} if there exists ${\bf x}\in (\mathbb{F}_q^*)^n$ such that the code $C_i$ is {\bf x}-isometric to the dual code $C_{s-i}^\perp$ for all $i=0,\ldots, s$. For $P$ and $Q$ rational places in a function field ${\mathcal F}$, we investigate the existence of isometry-dual flags of codes in the families of two-point algebraic geometry codes $$C_\mathcal L(D, a_0P+bQ)\subsetneq C_\mathcal L(D, a_1P+bQ)\subsetneq \dots \subsetneq C_\mathcal L(D, a_sP+bQ),$$ where the divisor $D$ is the sum of pairwise different rational places of ${\mathcal F}$ and $P, Q$ are not in $\mbox{supp}(D)$. We characterize those sequences in terms of $b$ for general function fields. We then apply the result to the broad class of Kummer extensions ${\mathcal F}$ defined by affine equations of the form $y^m=f(x)$, for $f(x)$ a separable polynomial of degree $r$, where $\mbox{gcd}(r, m)=1$. For $P$ the rational place at infinity and $Q$ the rational place associated to one of the roots of $f(x)$, it is shown that the flag of two-point algebraic geometry codes has the isometry-dual property if and only if $m$ divides $2b+1$. At the end we illustrate our results by applying them to two-point codes over several well know function fields.
Customized Graph Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Recently, Graph Neural Networks (GNNs) have greatly advanced the task of graph classification. Typically, we first build a unified GNN model with graphs in a given training set and then use this unified model to predict labels of all the unseen graphs in the test set. However, graphs in the same dataset often have dramatically distinct structures, which indicates that a unified model may be sub-optimal given an individual graph. Therefore, in this paper, we aim to develop customized graph neural networks for graph classification. Specifically, we propose a novel customized graph neural network framework, i.e., Customized-GNN. Given a graph sample, Customized-GNN can generate a sample-specific model for this graph based on its structure. Meanwhile, the proposed framework is very general that can be applied to numerous existing graph neural network models. Comprehensive experiments on various graph classification benchmarks demonstrate the effectiveness of the proposed framework.