OpenRaiser/Intern-Atlas huggingface dataset:
Diformer: Directional Transformer for Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Autoregressive (AR) and Non-autoregressive (NAR) models have their own superiority on the performance and latency, combining them into one model may take advantage of both. Current combination frameworks focus more on the integration of multiple decoding paradigms with a unified generative model, e.g. Masked Language Model. However, the generalization can be harmful to the performance due to the gap between training objective and inference. In this paper, we aim to close the gap by preserving the original objective of AR and NAR under a unified framework. Specifically, we propose the Directional Transformer (Diformer) by jointly modelling AR and NAR into three generation directions (left-to-right, right-to-left and straight) with a newly introduced direction variable, which works by controlling the prediction of each token to have specific dependencies under that direction. The unification achieved by direction successfully preserves the original dependency assumption used in AR and NAR, retaining both generalization and performance. Experiments on 4 WMT benchmarks demonstrate that Diformer outperforms current united-modelling works with more than 1.5 BLEU points for both AR and NAR decoding, and is also competitive to the state-of-the-art independent AR and NAR models.
HP-GNN: Generating High Throughput GNN Training Implementation on CPU-FPGA Heterogeneous Platform
Venue: arXiv cs.DC 2021
Year: 2021

Graph Neural Networks (GNNs) have shown great success in many applications such as recommendation systems, molecular property prediction, traffic prediction, etc. Recently, CPU-FPGA heterogeneous platforms have been used to accelerate many applications by exploiting customizable data path and abundant user-controllable on-chip memory resources of FPGAs. Yet, accelerating and deploying GNN training on such platforms requires not only expertise in hardware design but also substantial development efforts. We propose HP-GNN, a novel framework that generates high throughput GNN training implementations on a given CPU-FPGA platform that can benefit both application developers and machine learning researchers. HP-GNN takes GNN training algorithms, GNN models as the inputs, and automatically performs hardware mapping onto the target CPU-FPGA platform. HP-GNN consists of: (1) data layout and internal representation that reduce the memory traffic and random memory accesses; (2) optimized hardware templates that support various GNN models; (3) a design space exploration engine for automatic hardware mapping; (4) high-level application programming interfaces (APIs) that allows users to specify GNN training with only a handful of lines of code. To evaluate HP-GNN, we experiment with two well-known sampling-based GNN training algorithms and two GNN models. For each training algorithm and model, HP-GNN generates implementation on a state-of-the-art CPU-FPGA platform. Compared with CPU-only and CPU-GPU platforms, experimental results show that the generated implementations achieve $55.67\times$ and $2.17\times$ speedup on the average, respectively. Compared with the state-of-the-art GNN training implementations, HP-GNN achieves up to $4.45\times$ speedup.
Graph augmented Deep Reinforcement Learning in the GameRLand3D environment
Venue: arXiv cs.LG 2021
Year: 2021

We address planning and navigation in challenging 3D video games featuring maps with disconnected regions reachable by agents using special actions. In this setting, classical symbolic planners are not applicable or difficult to adapt. We introduce a hybrid technique combining a low level policy trained with reinforcement learning and a graph based high level classical planner. In addition to providing human-interpretable paths, the approach improves the generalization performance of an end-to-end approach in unseen maps, where it achieves a 20% absolute increase in success rate over a recurrent end-to-end agent on a point to point navigation task in yet unseen large-scale maps of size 1km x 1km. In an in-depth experimental study, we quantify the limitations of end-to-end Deep RL approaches in vast environments and we also introduce "GameRLand3D", a new benchmark and soon to be released environment can generate complex procedural 3D maps for navigation tasks.
Landscape properties of the very large-scale and the variable neighborhood search metaheuristics for the multidimensional assignment problem
Venue: arXiv cs.DM 2021
Year: 2021

We study the recent metaheuristic search algorithm for the multidimensional assignment problem (MAP) using fitness landscape theory. The analyzed algorithm performs a very large-scale neighborhood search on a set of feasible solutions to the problem. We derive properties of the landscape graphs that represent these very large-scale search algorithms acting on the solutions of the MAP. In particular, we show that the search graph is generalization of a hypercube. We extend and generalize the original very large-scale neighborhood search to develop the variable neighborhood search. The new search is capable or searching even larger large-scale neighborhoods. We perform numerical analyses of the search graph structures for various problem instances of the MAP and different neighborhood structures of the MAP algorithm based on a very large-scale search. We also investigate the correlation between fitness (i.e., objective values) and distance (i.e., path lengths) of the local minima (i.e., sinks of the landscape). Our results can be used to design improved search-based metaheuristics for the MAP.
Automatic Product Copywriting for E-Commerce
Venue: arXiv cs.CL 2021
Year: 2021

Product copywriting is a critical component of e-commerce recommendation platforms. It aims to attract users' interest and improve user experience by highlighting product characteristics with textual descriptions. In this paper, we report our experience deploying the proposed Automatic Product Copywriting Generation (APCG) system into the JD.com e-commerce product recommendation platform. It consists of two main components: 1) natural language generation, which is built from a transformer-pointer network and a pre-trained sequence-to-sequence model based on millions of training data from our in-house platform; and 2) copywriting quality control, which is based on both automatic evaluation and human screening. For selected domains, the models are trained and updated daily with the updated training data. In addition, the model is also used as a real-time writing assistant tool on our live broadcast platform. The APCG system has been deployed in JD.com since Feb 2021. By Sep 2021, it has generated 2.53 million product descriptions, and improved the overall averaged click-through rate (CTR) and the Conversion Rate (CVR) by 4.22% and 3.61%, compared to baselines, respectively on a year-on-year basis. The accumulated Gross Merchandise Volume (GMV) made by our system is improved by 213.42%, compared to the number in Feb 2021.
ALP: Data Augmentation using Lexicalized PCFGs for Few-Shot Text Classification
Venue: arXiv cs.CL 2021
Year: 2021

Data augmentation has been an important ingredient for boosting performances of learned models. Prior data augmentation methods for few-shot text classification have led to great performance boosts. However, they have not been designed to capture the intricate compositional structure of natural language. As a result, they fail to generate samples with plausible and diverse sentence structures. Motivated by this, we present the data Augmentation using Lexicalized Probabilistic context-free grammars (ALP) that generates augmented samples with diverse syntactic structures with plausible grammar. The lexicalized PCFG parse trees consider both the constituents and dependencies to produce a syntactic frame that maximizes a variety of word choices in a syntactically preservable manner without specific domain experts. Experiments on few-shot text classification tasks demonstrate that ALP enhances many state-of-the-art classification methods. As a second contribution, we delve into the train-val splitting methodologies when a data augmentation method comes into play. We argue empirically that the traditional splitting of training and validation sets is sub-optimal compared to our novel augmentation-based splitting strategies that further expand the training split with the same number of labeled data. Taken together, our contributions on the data augmentation strategies yield a strong training recipe for few-shot text classification tasks.
Meta-Learning and Self-Supervised Pretraining for Real World Image Translation
Venue: arXiv cs.CV 2021
Year: 2021

Recent advances in deep learning, in particular enabled by hardware advances and big data, have provided impressive results across a wide range of computational problems such as computer vision, natural language, or reinforcement learning. Many of these improvements are however constrained to problems with large-scale curated data-sets which require a lot of human labor to gather. Additionally, these models tend to generalize poorly under both slight distributional shifts and low-data regimes. In recent years, emerging fields such as meta-learning or self-supervised learning have been closing the gap between proof-of-concept results and real-life applications of machine learning by extending deep-learning to the semi-supervised and few-shot domains. We follow this line of work and explore spatio-temporal structure in a recently introduced image-to-image translation problem in order to: i) formulate a novel multi-task few-shot image generation benchmark and ii) explore data augmentations in contrastive pre-training for image translation downstream tasks. We present several baselines for the few-shot problem and discuss trade-offs between different approaches. Our code is available at https://github.com/irugina/meta-image-translation.
Compromised ACC vehicles can degrade current mixed-autonomy traffic performance while remaining stealthy against detection
Venue: arXiv cs.CR 2021
Year: 2021

We demonstrate that a supply-chain level compromise of the adaptive cruise control (ACC) capability on equipped vehicles can be used to significantly degrade system level performance of current day mixed-autonomy freeway networks. Via a simple threat model which causes random deceleration attacks (RDAs), compromised vehicles create congestion waves in the traffic that decrease average speed and network throughput. We use a detailed and realistic traffic simulation environment to quantify the impacts of the attack on a model of a real high-volume freeway in the United States. We find that the effect of the attack depends both on the level of underlying traffic congestion, and what percentage of ACC vehicles can be compromised. In moderate congestion regimes the attack can degrade mean commuter speed by over 7%. In high density regimes overall network throughput can be reduced by up to 3%. And, in moderate to high congestion regimes, it can cost commuters on the network over 300 USD/km hr. All of these results motivate that the proposed attack is able to significantly degrade performance of the traffic network. We also develop an anomaly detection technique that uses GPS traces on vehicles to identify malicious/compromised vehicles. We employ this technique on data from the simulation experiments and find that it is unable to identify compromised ACCs compared to benign/normal drivers. That is, these attacks are stealthy to detection. Stronger attacks can be accurately labeled as malicious, motivating that there is a limit to how impactful attacks can be before they are no longer stealthy. Finally, we experimentally execute the attack on a real and commercially available ACC vehicle, demonstrating the possible real world feasibility of an RDA.
VoiceMoji: A Novel On-Device Pipeline for Seamless Emoji Insertion in Dictation
Venue: arXiv cs.CL 2021
Year: 2021

Most of the speech recognition systems recover only words in the speech and fail to capture emotions. Users have to manually add emoji(s) in text for adding tone and making communication fun. Though there is much work done on punctuation addition on transcribed speech, the area of emotion addition is untouched. In this paper, we propose a novel on-device pipeline to enrich the voice input experience. It involves, given a blob of transcribed text, intelligently processing and identifying structure where emoji insertion makes sense. Moreover, it includes semantic text analysis to predict emoji for each of the sub-parts for which we propose a novel architecture Attention-based Char Aware (ACA) LSTM which handles Out-Of-Vocabulary (OOV) words as well. All these tasks are executed completely on-device and hence can aid on-device dictation systems. To the best of our knowledge, this is the first work that shows how to add emoji(s) in the transcribed text. We demonstrate that our components achieve comparable results to previous neural approaches for punctuation addition and emoji prediction with 80% fewer parameters. Overall, our proposed model has a very small memory footprint of a mere 4MB to suit on-device deployment.
Machine Learning for Computational Science and Engineering -- a brief introduction and some critical questions
Venue: arXiv cs.LG 2021
Year: 2021

Artificial Intelligence (AI) is now entering every sub-field of science, technology, engineering, arts, and management. Thanks to the hype and availability of research funds, it is being adapted in many fields without much thought. Computational Science and Engineering (CS&E) is one such sub-field. By highlighting some critical questions around the issues and challenges in adapting Machine Learning (ML) for CS&E, most of which are often overlooked in journal papers, this contribution hopes to offer some insights into the adaptation of ML for applications in CS\&E and related fields. This is a general-purpose article written for a general audience and researchers new to the fields of ML and/or CS\&E. This work focuses only on the forward problems in computational science and engineering. Some basic equations and MATLAB code are also provided to help the reader understand the basics.
Field-Based Toolpath Generation for 3D Printing Continuous Fibre Reinforced Thermoplastic Composites
Venue: arXiv cs.GR 2021
Year: 2021

We present a field-based method of toolpath generation for 3D printing continuous fibre reinforced thermoplastic composites. Our method employs the strong anisotropic material property of continuous fibres by generating toolpaths along the directions of tensile stresses in the critical regions. Moreover, the density of toolpath distribution is controlled in an adaptive way proportionally to the values of stresses. Specifically, a vector field is generated from the stress tensors under given loads and processed to have better compatibility between neighboring vectors. An optimal scalar field is computed later by making its gradients approximate the vector field. After that, isocurves of the scalar field are extracted to generate the toolpaths for continuous fibre reinforcement, which are also integrated with the boundary conformal toolpaths in user selected regions. The performance of our method has been verified on a variety of models in different loading conditions. Experimental tests are conducted on specimens by 3D printing continuous carbon fibres (CCF) in a polylactic acid (PLA) matrix. Compared to reinforcement by load-independent toolpaths, the specimens fabricated by our method show up to 71.4% improvement on the mechanical strength in physical tests when using the same (or even slightly smaller) amount of continuous fibres.
Input-Specific Robustness Certification for Randomized Smoothing
Venue: arXiv cs.CV 2021
Year: 2021

Although randomized smoothing has demonstrated high certified robustness and superior scalability to other certified defenses, the high computational overhead of the robustness certification bottlenecks the practical applicability, as it depends heavily on the large sample approximation for estimating the confidence interval. In existing works, the sample size for the confidence interval is universally set and agnostic to the input for prediction. This Input-Agnostic Sampling (IAS) scheme may yield a poor Average Certified Radius (ACR)-runtime trade-off which calls for improvement. In this paper, we propose Input-Specific Sampling (ISS) acceleration to achieve the cost-effectiveness for robustness certification, in an adaptive way of reducing the sampling size based on the input characteristic. Furthermore, our method universally controls the certified radius decline from the ISS sample size reduction. The empirical results on CIFAR-10 and ImageNet show that ISS can speed up the certification by more than three times at a limited cost of 0.05 certified radius. Meanwhile, ISS surpasses IAS on the average certified radius across the extensive hyperparameter settings. Specifically, ISS achieves ACR=0.958 on ImageNet ($σ=1.0$) in 250 minutes, compared to ACR=0.917 by IAS under the same condition. We release our code in \url{https://github.com/roy-ch/Input-Specific-Certification}.
Survey the storage systems used in HPC and BDA ecosystems
Venue: arXiv cs.DC 2021
Year: 2021

The advancement in HPC and BDA ecosystem demands a better understanding of the storage systems to plan effective solutions. To make applications access data more efficiently for computation, HPC and BDA ecosystems adopt different storage systems. Each storage system has its pros and cons. Therefore, it is worthwhile and interesting to explore the storage systems used in HPC and BDA respectively. Also, it's inquisitive to understand how such storage systems can handle data consistency and fault tolerance at a massive scale. In this paper, we're surveying four storage systems Lustre, Ceph, HDFS, and CockroachDB. Lustre and HDFS are some of the most prominent file systems in HPC and BDA ecosystem. Ceph is an upcoming filesystem and is being used by supercomputers. CockroachDB is based on NewSQL systems a technique that is being used in the industry for BDA applications. The study helps us to understand the underlying architecture of these storage systems and the building blocks used to create them. The protocols and mechanisms used for data storage, data access, data consistency, fault tolerance, and recovery from failover are also overviewed. The comparative study will help system designers to understand the key features and architectural goals of these storage systems to select better storage system solutions.
Judging a book by its cover: Predicting the marginal impact of title on Reddit post popularity
Venue: arXiv cs.SI 2021
Year: 2021

Several factors influence the popularity of content on social media, including the what, when, and who of a post. Of these factors, the what and when of content are easiest to customize in order to maximize viewership and reach. Further, the title of a post (part of the what) is the easiest to tailor, compared to the post's body, which is often fixed. So, in this paper, we assess the impact of a post's title on its popularity while controlling for the time of posting (the when) by proposing an interpretable attention-based model. Our approach achieves state-of-the-art performance in predicting the popularity of posts on multiple online communities of various sizes, topics, and formats while still being parsimonious. Interpretation of our model's attention weights sheds light on the heterogeneous patterns in how the specific words in a post's title shape its popularity across different communities. Our results highlight the power of sentiment alignment, personal storytelling, and even personality politics in propelling content to virality.
ProBF: Learning Probabilistic Safety Certificates with Barrier Functions
Venue: arXiv cs.LG 2021
Year: 2021

Safety-critical applications require controllers/policies that can guarantee safety with high confidence. The control barrier function is a useful tool to guarantee safety if we have access to the ground-truth system dynamics. In practice, we have inaccurate knowledge of the system dynamics, which can lead to unsafe behaviors due to unmodeled residual dynamics. Learning the residual dynamics with deterministic machine learning models can prevent the unsafe behavior but can fail when the predictions are imperfect. In this situation, a probabilistic learning method that reasons about the uncertainty of its predictions can help provide robust safety margins. In this work, we use a Gaussian process to model the projection of the residual dynamics onto a control barrier function. We propose a novel optimization procedure to generate safe controls that can guarantee safety with high probability. The safety filter is provided with the ability to reason about the uncertainty of the predictions from the GP. We show the efficacy of this method through experiments on Segway and Quadrotor simulations. Our proposed probabilistic approach is able to reduce the number of safety violations significantly as compared to the deterministic approach with a neural network.
Analysis of ECG data to detect Atrial Fibrillation
Venue: arXiv cs.LG 2021
Year: 2021

Atrial fibrillation(termed as AF/Afib henceforth) is a discrete and often rapid heart rhythm that can lead to clots near the heart. We can detect Afib by ECG signal by the absence of p and inconsistent intervals between R waves as shown in fig(1). Existing methods revolve around CNN that are used to detect afib but most of them work with 12 point lead ECG data where in our case the health gauge watch deals with single-point ECG data. Twelve-point lead ECG data is more accurate than a single point. Furthermore, the health gauge watch data is much noisier. Implementing a model to detect Afib for the watch is a test of how the CNN is changed/modified to work with real life data
A Random Point Initialization Approach to Image Segmentation with Variational Level-sets
Venue: arXiv cs.CV 2021
Year: 2021

Image segmentation is an essential component in many image processing and computer vision tasks. The primary goal of image segmentation is to simplify an image for easier analysis, and there are two broad approaches for achieving this: edge based methods, which extract the boundaries of specific known objects, and region based methods, which partition the image into regions that are statistically homogeneous. One of the more prominent edge finding methods, known as the level set method, evolves a zero-level contour in the image plane with gradient descent until the contour has converged to the object boundaries. While the classical level set method and its variants have proved successful in segmenting real images, they are susceptible to becoming stuck in noisy regions of the image plane without a priori knowledge of the image and they are unable to provide details beyond object outer boundary locations. We propose a modification to the variational level set image segmentation method that can quickly detect object boundaries by making use of random point initialization. We demonstrate the efficacy of our approach by comparing the performance of our method on real images to that of the prominent Canny Method.
In-storage Processing of I/O Intensive Applications on Computational Storage Drives
Venue: arXiv cs.DC 2021
Year: 2021

Computational storage drives (CSD) are solid-state drives (SSD) empowered by general-purpose processors that can perform in-storage processing. They have the potential to improve both performance and energy significantly for big-data analytics by bringing compute to data, thereby eliminating costly data transfer while offering better privacy. In this work, we introduce Solana, the first-ever high-capacity(12-TB) CSD in E1.S form factor, and present an actual prototype for evaluation. To demonstrate the benefits of in-storage processing on CSD, we deploy several natural language processing (NLP) applications on datacenter-grade storage servers comprised of clusters of the Solana. Experimental results show up to 3.1x speedup in processing while reducing the energy consumption and data transfer by 67% and 68%, respectively, compared to regular enterprise SSDs.