OpenRaiser/Intern-Atlas huggingface dataset:
An Attention Free Transformer
Venue: arXiv cs.LG 2021
Year: 2021

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.
Data Acquisition for Improving Machine Learning Models
Venue: arXiv cs.DB 2021
Year: 2021

The vast advances in Machine Learning over the last ten years have been powered by the availability of suitably prepared data for training purposes. The future of ML-enabled enterprise hinges on data. As such, there is already a vibrant market offering data annotation services to tailor sophisticated ML models. In this paper, we present research on the practical problem of obtaining data in order to improve the accuracy of ML models. We consider an environment in which consumers query for data to enhance the accuracy of their models and data providers who possess data make them available for training purposes. We first formalize this interaction process laying out the suitable framework and associated parameters for data exchange. We then propose two data acquisition strategies that consider a trade-off between exploration during which we obtain data to learn about the distribution of a provider's data and exploitation during which we optimize our data inquiries utilizing the gained knowledge. In the first strategy, Estimation and Allocation, we utilize queries to estimate the utilities of various predicates while learning about the distribution of the provider's data; then we proceed to the allocation stage in which we utilize those learned utility estimates to inform our data acquisition decisions. The second algorithmic proposal, named Sequential Predicate Selection, utilizes a sampling strategy to explore the distribution of the provider's data, adaptively investing more resources to parts of the data space that are statistically more promising to improve overall model accuracy. We present a detailed experimental evaluation of our proposals utilizing a variety of ML models and associated real data sets exploring all applicable parameters of interest. We identify trade-offs and highlight the relative benefits of each algorithm to further optimize model accuracy.
Recommendations and Results Organization in Netflix Search
Venue: arXiv cs.IR 2021
Year: 2021

Personalized recommendations on the Netflix Homepage are based on a user's viewing habits and the behavior of similar users. These recommendations, organized for efficient browsing, enable users to discover the next great video to watch and enjoy without additional input or an explicit expression of their intents or goals. The Netflix Search experience, on the other hand, allows users to take active control of discovering new videos by explicitly expressing their entertainment needs via search queries. In this talk, we discuss the importance of producing search results that go beyond traditional keyword-matches to effectively satisfy users' search needs in the Netflix entertainment setting. Motivated by users' various search intents, we highlight the necessity to improve Search by applying approaches that have historically powered the Homepage. Specifically, we discuss our approach to leverage recommendations in the context of Search and to effectively organize search results to provide a product experience that meaningfully adds value for our users.
SSCAP: Self-supervised Co-occurrence Action Parsing for Unsupervised Temporal Action Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Temporal action segmentation is a task to classify each frame in the video with an action label. However, it is quite expensive to annotate every frame in a large corpus of videos to construct a comprehensive supervised training dataset. Thus in this work we propose an unsupervised method, namely SSCAP, that operates on a corpus of unlabeled videos and predicts a likely set of temporal segments across the videos. SSCAP leverages Self-Supervised learning to extract distinguishable features and then applies a novel Co-occurrence Action Parsing algorithm to not only capture the correlation among sub-actions underlying the structure of activities, but also estimate the temporal path of the sub-actions in an accurate and general way. We evaluate on both classic datasets (Breakfast, 50Salads) and the emerging fine-grained action dataset (FineGym) with more complex activity structures and similar sub-actions. Results show that SSCAP achieves state-of-the-art performance on all datasets and can even outperform some weakly-supervised approaches, demonstrating its effectiveness and generalizability.
A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
Venue: arXiv cs.RO 2021
Year: 2021

In this survey, we systematically summarize the current literature on studies that apply reinforcement learning (RL) to the motion planning and control of autonomous vehicles. Many existing contributions can be attributed to the pipeline approach, which consists of many hand-crafted modules, each with a functionality selected for the ease of human interpretation. However, this approach does not automatically guarantee maximal performance due to the lack of a system-level optimization. Therefore, this paper also presents a growing trend of work that falls into the end-to-end approach, which typically offers better performance and smaller system scales. However, their performance also suffers from the lack of expert data and generalization issues. Finally, the remaining challenges applying deep RL algorithms on autonomous driving are summarized, and future research directions are also presented to tackle these challenges.
CoDesc: A Large Code-Description Parallel Dataset
Venue: arXiv cs.CL 2021
Year: 2021

Translation between natural language and source code can help software development by enabling developers to comprehend, ideate, search, and write computer programs in natural language. Despite growing interest from the industry and the research community, this task is often difficult due to the lack of large standard datasets suitable for training deep neural models, standard noise removal methods, and evaluation benchmarks. This leaves researchers to collect new small-scale datasets, resulting in inconsistencies across published works. In this study, we present CoDesc -- a large parallel dataset composed of 4.2 million Java methods and natural language descriptions. With extensive analysis, we identify and remove prevailing noise patterns from the dataset. We demonstrate the proficiency of CoDesc in two complementary tasks for code-description pairs: code summarization and code search. We show that the dataset helps improve code search by up to 22\% and achieves the new state-of-the-art in code summarization. Furthermore, we show CoDesc's effectiveness in pre-training--fine-tuning setup, opening possibilities in building pretrained language models for Java. To facilitate future research, we release the dataset, a data processing tool, and a benchmark at \url{https://github.com/csebuetnlp/CoDesc}.
CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model
Venue: arXiv cs.CL 2021
Year: 2021

Commit message is a document that summarizes source code changes in natural language. A good commit message clearly shows the source code changes, so this enhances collaboration between developers. Therefore, our work is to develop a model that automatically writes the commit message. To this end, we release 345K datasets consisting of code modification and commit messages in six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Similar to the neural machine translation (NMT) model, using our dataset, we feed the code modification to the encoder input and the commit message to the decoder input and measure the result of the generated commit message with BLEU-4. Also, we propose the following two training methods to improve the result of generating the commit message: (1) A method of preprocessing the input to feed the code modification to the encoder input. (2) A method that uses an initial weight suitable for the code domain to reduce the gap in contextual representation between programming language (PL) and natural language (NL). Training code, dataset, and pre-trained weights are available at https://github.com/graykode/commit-autosuggestions
Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms
Venue: arXiv cs.RO 2021
Year: 2021

Orienting objects is a critical component in the automation of many packing and assembly tasks. We present an algorithm to orient novel objects given a depth image of the object in its current and desired orientation. We formulate a self-supervised objective for this problem and train a deep neural network to estimate the 3D rotation as parameterized by a quaternion, between these current and desired depth images. We then use the trained network in a proportional controller to re-orient objects based on the estimated rotation between the two depth images. Results suggest that in simulation we can rotate unseen objects with unknown geometries by up to 30° with a median angle error of 1.47° over 100 random initial/desired orientations each for 22 novel objects. Experiments on physical objects suggest that the controller can achieve a median angle error of 4.2° over 10 random initial/desired orientations each for 5 objects.
Circuit Design for $k$-coloring Problem and Its Implementation in Any Dimensional Quantum System
Venue: arXiv cs.ET 2021
Year: 2021

With the evolution of quantum computing, researchers now-a-days tend to incline to find solutions to NP-complete problems by using quantum algorithms in order to gain asymptotic advantage. In this paper, we solve $k$-coloring problem (NP-complete problem) using Grover's algorithm in any dimensional quantum system or any $d$-ary quantum system for the first time to the best of our knowledge, where $d \ge 2$. A newly proposed comparator-based approach helps to generalize the implementation of the $k$-coloring problem in any dimensional quantum system. Till date, $k$-coloring problem has been implemented only in binary and ternary quantum system, hence, we abide to $d=2$ or $d=3$, that is for binary and ternary quantum system for comparing our proposed work with the state-of-the-art techniques. This proposed approach makes the reduction of the qubit cost possible, compared to the state-of-the-art binary quantum systems. Further, with the help of newly proposed ternary comparator, a substantial reduction in quantum gate count for the ternary oracle circuit of the $k$-coloring problem than the previous approaches has been obtained. An end-to-end automated framework has been put forward for implementing the $k$-coloring problem for any undirected and unweighted graph on any available Near-term quantum devices or Noisy Intermediate-Scale Quantum (NISQ) devices or multi-valued quantum simulator, which helps in generalizing our approach.
ECMO: Peripheral Transplantation to Rehost Embedded Linux Kernels
Venue: arXiv cs.AR 2021
Year: 2021

Dynamic analysis based on the full-system emulator QEMU is widely used for various purposes. However, it is challenging to run firmware images of embedded devices in QEMU, especially the process to boot the Linux kernel (we call this process rehosting the Linux kernel.) That's because embedded devices usually use different system-on-chips (SoCs) from multiple vendors and only a limited number of SoCs are currently supported in QEMU. In this work, we propose a technique called peripheral transplantation. The main idea is to transplant the device drivers of designated peripherals into the Linux kernel. By doing so, it can replace the peripherals in the kernel that are currently unsupported in QEMU with supported ones, thus making the Linux kernel rehostable. After that, various applications can be built upon. We implemented this technique inside a prototype system called ECMO and applied it to 815 firmware images, which consist of 20 kernel versions, 37 device models, and 24 vendors. The result shows that ECMO can successfully transplant peripherals for all the 815 Linux kernels. Among them,710 kernels can be successfully rehosted, i.e., launching a user-space shell (87.1% success rate). The failed cases are mainly because the root file system format (ramfs) is not supported by the kernel. We further build three applications, i.e., kernel crash analysis, rootkit forensic analysis, and kernel fuzzing, based on the rehosted kernels to demonstrate the usage scenarios of ECMO
RPG: Learning Recursive Point Cloud Generation
Venue: arXiv cs.CV 2021
Year: 2021

In this paper we propose a novel point cloud generator that is able to reconstruct and generate 3D point clouds composed of semantic parts. Given a latent representation of the target 3D model, the generation starts from a single point and gets expanded recursively to produce the high-resolution point cloud via a sequence of point expansion stages. During the recursive procedure of generation, we not only obtain the coarse-to-fine point clouds for the target 3D model from every expansion stage, but also unsupervisedly discover the semantic segmentation of the target model according to the hierarchical/parent-child relation between the points across expansion stages. Moreover, the expansion modules and other elements used in our recursive generator are mostly sharing weights thus making the overall framework light and efficient. Extensive experiments are conducted to demonstrate that our proposed point cloud generator has comparable or even superior performance on both generation and reconstruction tasks in comparison to various baselines, as well as provides the consistent co-segmentation among 3D instances of the same object class.
WfCommons: A Framework for Enabling Scientific Workflow Research and Development
Venue: arXiv cs.DC 2021
Year: 2021

Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in real-world workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions.
Gradient-Free Neural Network Training via Synaptic-Level Reinforcement Learning
Venue: arXiv cs.NE 2021
Year: 2021

An ongoing challenge in neural information processing is: how do neurons adjust their connectivity to improve task performance over time (i.e., actualize learning)? It is widely believed that there is a consistent, synaptic-level learning mechanism in specific brain regions that actualizes learning. However, the exact nature of this mechanism remains unclear. Here we propose an algorithm based on reinforcement learning (RL) to generate and apply a simple synaptic-level learning policy for multi-layer perceptron (MLP) models. In this algorithm, the action space for each MLP synapse consists of a small increase, decrease, or null action on the synapse weight, and the state for each synapse consists of the last two actions and reward signals. A binary reward signal indicates improvement or deterioration in task performance. The static policy produces superior training relative to the adaptive policy and is agnostic to activation function, network shape, and task. Trained MLPs yield character recognition performance comparable to identically shaped networks trained with gradient descent. 0 hidden unit character recognition tests yielded an average validation accuracy of 88.28%, 1.86$\pm$0.47% higher than the same MLP trained with gradient descent. 32 hidden unit character recognition tests yielded an average validation accuracy of 88.45%, 1.11$\pm$0.79% lower than the same MLP trained with gradient descent. The robustness and lack of reliance on gradient computations opens the door for new techniques for training difficult-to-differentiate artificial neural networks such as spiking neural networks (SNNs) and recurrent neural networks (RNNs). Further, the method's simplicity provides a unique opportunity for further development of local rule-driven multi-agent connectionist models for machine intelligence analogous to cellular automata.
Communication efficient privacy-preserving distributed optimization using adaptive differential quantization
Venue: arXiv cs.DC 2021
Year: 2021

Privacy issues and communication cost are both major concerns in distributed optimization. There is often a trade-off between them because the encryption methods required for privacy-preservation often incur expensive communication bandwidth. To address this issue, we, in this paper, propose a quantization-based approach to achieve both communication efficient and privacy-preserving solutions in the context of distributed optimization. By deploying an adaptive differential quantization scheme, we allow each node in the network to achieve its optimum solution with a low communication cost while keeping its private data unrevealed. Additionally, the proposed approach is general and can be applied in various distributed optimization methods, such as the primal-dual method of multipliers (PDMM) and the alternating direction method of multipliers (ADMM). Moveover, we consider two widely used adversary models: passive and eavesdropping. Finally, we investigate the properties of the proposed approach using different applications and demonstrate its superior performance in terms of several parameters including accuracy, privacy, and communication cost.
Gaze Estimation using Transformer
Venue: arXiv cs.CV 2021
Year: 2021

Recent work has proven the effectiveness of transformers in many computer vision tasks. However, the performance of transformers in gaze estimation is still unexplored. In this paper, we employ transformers and assess their effectiveness for gaze estimation. We consider two forms of vision transformer which are pure transformers and hybrid transformers. We first follow the popular ViT and employ a pure transformer to estimate gaze from images. On the other hand, we preserve the convolutional layers and integrate CNNs as well as transformers. The transformer serves as a component to complement CNNs. We compare the performance of the two transformers in gaze estimation. The Hybrid transformer significantly outperforms the pure transformer in all evaluation datasets with less parameters. We further conduct experiments to assess the effectiveness of the hybrid transformer and explore the advantage of self-attention mechanism. Experiments show the hybrid transformer can achieve state-of-the-art performance in all benchmarks with pre-training.To facilitate further research, we release codes and models in https://github.com/yihuacheng/GazeTR.
Comparison-limited Vector Quantization
Venue: arXiv cs.IT 2021
Year: 2021

In this paper a variation of the classic vector quantization problem is considered. In the standard formulation, a quantizer is designed to minimize the distortion between input and output when the number of reconstruction points is fixed. We consider, instead, the scenario in which the number of comparators used in quantization is fixed. More precisely, we study the case in which a vector quantizer of dimension d is comprised of k comparators, each receiving a linear combination of the inputs and producing the output value one/zero if this linear combination is above/below a certain threshold. In reconstruction, the comparators' output is mapped to a reconstruction point, chosen so as to minimize a chosen distortion measure between the quantizer input and its reconstruction. The Comparison-Limited Vector Quantization (CLVQ) problem is then defined as the problem of optimally designing the configuration of the compactors and the choice of reconstruction points so as to minimize the given distortion. In this paper, we design a numerical optimization algorithm for the CLVQ problem. This algorithm leverages combinatorial geometrical notions to describe the hyperplane arrangement induced by the configuration of the comparators. It also relies on a genetic genetic meta heuristic to improve the selection of the quantizer initialization and avoid local minima encountered during optimization. We numerically evaluate the performance of our algorithm in the case of input distributions following uniform and Gaussian i.i.d. sources to be compressed under quadratic distortion and compare it to the classic Linde-Buzo-Gray (LBG) algorithm.
Perceived Safety in Physical Human Robot Interaction -- A Survey
Venue: arXiv cs.RO 2021
Year: 2021

This review paper focuses on different aspects of perceived safety for a number of autonomous physical systems. This is a major aspect of robotics research, as more and more applications allow human and autonomous systems to share their space, with crucial implications both on safety and on its perception. The alternative terms used to express related concepts (e.g., psychological safety, trust, comfort, stress, fear, and anxiety) are listed and explained. Then, the available methods to assess perceived safety (i.e., questionnaires, physiological measurements, behavioral assessment, and direct input devices) are described. Six categories of autonomous systems are considered (industrial manipulators, mobile robots, mobile manipulators, humanoid robots, drones, and autonomous vehicles), providing an overview of the main themes related to perceived safety in the specific domain, a description of selected works, and an analysis of how motion and characteristics of the system influence the perception of safety. The survey also discusses experimental duration and location of the reviewed papers as well as identified trends over time.