OpenRaiser/Intern-Atlas huggingface dataset:
Compressed Communication for Distributed Training: Adaptive Methods and System
Venue: arXiv cs.DC 2021
Year: 2021

Communication overhead severely hinders the scalability of distributed machine learning systems. Recently, there has been a growing interest in using gradient compression to reduce the communication overhead of the distributed training. However, there is little understanding of applying gradient compression to adaptive gradient methods. Moreover, its performance benefits are often limited by the non-negligible compression overhead. In this paper, we first introduce a novel adaptive gradient method with gradient compression. We show that the proposed method has a convergence rate of $\mathcal{O}(1/\sqrt{T})$ for non-convex problems. In addition, we develop a scalable system called BytePS-Compress for two-way compression, where the gradients are compressed in both directions between workers and parameter servers. BytePS-Compress pipelines the compression and decompression on CPUs and achieves a high degree of parallelism. Empirical evaluations show that we improve the training time of ResNet50, VGG16, and BERT-base by 5.0%, 58.1%, 23.3%, respectively, without any accuracy loss with 25 Gb/s networking. Furthermore, for training the BERT models, we achieve a compression rate of 333x compared to the mixed-precision training.
Topology-optimization based design of multi-degree-of-freedom compliant mechanisms (mechanisms with multiple pseudo-mobility)
Venue: arXiv cs.RO 2021
Year: 2021

Unlike conventional mechanisms, compliant mechanisms produce the desired deformations by exploiting elastic strain and do not need, therefore, moving parts. The number of degrees of freedom of a conventional mechanism, also called mobility, is the number of independent coordinates needed to define a configuration of the mechanism. Due to the different operating principle, such definition of degree of freedom or mobility cannot be directly applied to compliant mechanisms. While those terms are not able to denote a property of a given compliant mechanism, they are meaningful when applied to the design of a compliant mechanism. Compliant mechanisms are, however, mostly seen as elastic structures, for which the term degree of freedom is used in a different meaning. In order to avoid ambiguities, the term pseudo-mobility (already introduced in previous published work) will be used to denote the number of scalar parameters needed to identify one single desired deformation, i.e. one single deformation for which the compliant mechanism is designed. Many synthesis approaches exist for compliant mechanisms with single pseudo-mobility (commonly referred to as "single degree of freedom mechanisms"). In the case of compliant mechanisms with multiple pseudo-mobility (multiple-degree of freedom mechanisms), only synthesis approaches for relatively simple mechanisms exist so far, while systems for more complex tasks like shape adaptation are not covered. In addition, only certain cases of transverse loads are included in the synthesis with these approaches. In this paper, a novel optimization algorithm is presented that addresses these two shortcomings. The algorithm is tested on a simple mechanism with one translation and one rotation kinematic degree of freedom, a compliant parallel mechanism for pure translation and a shape-adaptive structure.
Large-Scale Unsupervised Person Re-Identification with Contrastive Learning
Venue: arXiv cs.CV 2021
Year: 2021

Existing public person Re-Identification~(ReID) datasets are small in modern terms because of labeling difficulty. Although unlabeled surveillance video is abundant and relatively easy to obtain, it is unclear how to leverage these footage to learn meaningful ReID representations. In particular, most existing unsupervised and domain adaptation ReID methods utilize only the public datasets in their experiments, with labels removed. In addition, due to small data sizes, these methods usually rely on fine tuning by the unlabeled training data in the testing domain to achieve good performance. Inspired by the recent progress of large-scale self-supervised image classification using contrastive learning, we propose to learn ReID representation from large-scale unlabeled surveillance video alone. Assisted by off-the-shelf pedestrian detection tools, we apply the contrastive loss at both the image and the tracklet levels. Together with a principal component analysis step using camera labels freely available, our evaluation using a large-scale unlabeled dataset shows far superior performance among unsupervised methods that do not use any training data in the testing domain. Furthermore, the accuracy improves with the data size and therefore our method has great potential with even larger and more diversified datasets.
Construction and enumeration of left dihedral codes satisfying certain duality properties
Venue: arXiv cs.IT 2021
Year: 2021

Let $\mathbb{F}_{q}$ be the finite field of $q$ elements and let $D_{2n}=\langle x,y\mid x^n=1, y^2=1, yxy=x^{n-1}\rangle$ be the dihedral group of order $n$. Left ideals of the group algebra $\mathbb{F}_{q}[D_{2n}]$ are known as left dihedral codes over $\mathbb{F}_{q}$ of length $2n$, and abbreviated as left $D_{2n}$-codes. Let ${\rm gcd}(n,q)=1$. In this paper, we give an explicit representation for the Euclidean hull of every left $D_{2n}$-code over $\mathbb{F}_{q}$. On this basis, we determine all distinct Euclidean LCD codes and Euclidean self-orthogonal codes which are left $D_{2n}$-codes over $\mathbb{F}_{q}$. In particular, we provide an explicit representation and a precise enumeration for these two subclasses of left $D_{2n}$-codes and self-dual left $D_{2n}$-codes, respectively. Moreover, we give a direct and simple method for determining the encoder (generator matrix) of any left $D_{2n}$-code over $\mathbb{F}_{q}$, and present several numerical examples to illustrative our applications.
Confidence Assertions in Cyber-Security for an Information-Sharing Environment
Venue: arXiv cs.CR 2021
Year: 2021

Information sharing is vital in resisting cyberattacks, and the volume and severity of these attacks is increasing very rapidly. Therefore responders must triage incoming warnings in deciding how to act. This study asked a very specific question: "how can the addition of confidence information to alerts and warnings improve overall resistance to cyberattacks." We sought, in particular, to identify current practices, and if possible, to identify some "best practices." The research involved literature review and interviews with subject matter experts at every level from system administrators to persons who develop broad principles of policy. An innovative Modified Online Delphi Panel technique was used to elicit judgments and recommendations from experts who were able to speak with each other and vote anonymously to rank proposed practices.
Gradient Masking and the Underestimated Robustness Threats of Differential Privacy in Deep Learning
Venue: arXiv cs.CR 2021
Year: 2021

An important problem in deep learning is the privacy and security of neural networks (NNs). Both aspects have long been considered separately. To date, it is still poorly understood how privacy enhancing training affects the robustness of NNs. This paper experimentally evaluates the impact of training with Differential Privacy (DP), a standard method for privacy preservation, on model vulnerability against a broad range of adversarial attacks. The results suggest that private models are less robust than their non-private counterparts, and that adversarial examples transfer better among DP models than between non-private and private ones. Furthermore, detailed analyses of DP and non-DP models suggest significant differences between their gradients. Additionally, this work is the first to observe that an unfavorable choice of parameters in DP training can lead to gradient masking, and, thereby, results in a wrong sense of security.
Automatic Fault Detection for Deep Learning Programs Using Graph Transformations
Venue: arXiv cs.SE 2021
Year: 2021

Nowadays, we are witnessing an increasing demand in both corporates and academia for exploiting Deep Learning (DL) to solve complex real-world problems. A DL program encodes the network structure of a desirable DL model and the process by which the model learns from the training dataset. Like any software, a DL program can be faulty, which implies substantial challenges of software quality assurance, especially in safety-critical domains. It is therefore crucial to equip DL development teams with efficient fault detection techniques and tools. In this paper, we propose NeuraLint, a model-based fault detection approach for DL programs, using meta-modelling and graph transformations. First, we design a meta-model for DL programs that includes their base skeleton and fundamental properties. Then, we construct a graph-based verification process that covers 23 rules defined on top of the meta-model and implemented as graph transformations to detect faults and design inefficiencies in the generated models (i.e., instances of the meta-model). First, the proposed approach is evaluated by finding faults and design inefficiencies in 28 synthesized examples built from common problems reported in the literature. Then NeuraLint successfully finds 64 faults and design inefficiencies in 34 real-world DL programs extracted from Stack Overflow posts and GitHub repositories. The results show that NeuraLint effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5 % and a precision of 100 %. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures such as recurrent neural networks. Researchers can also expand our set of verification rules to cover more types of issues in DL programs.
An Integrated Deep Learning and Dynamic Programming Method for Predicting Tumor Suppressor Genes, Oncogenes, and Fusion from PDB Structures
Venue: arXiv cs.LG 2021
Year: 2021

Mutations in proto-oncogenes (ONGO) and the loss of regulatory function of tumor suppression genes (TSG) are the common underlying mechanism for uncontrolled tumor growth. While cancer is a heterogeneous complex of distinct diseases, finding the potentiality of the genes related functionality to ONGO or TSG through computational studies can help develop drugs that target the disease. This paper proposes a classification method that starts with a preprocessing stage to extract the feature map sets from the input 3D protein structural information. The next stage is a deep convolutional neural network stage (DCNN) that outputs the probability of functional classification of genes. We explored and tested two approaches: in Approach 1, all filtered and cleaned 3D-protein-structures (PDB) are pooled together, whereas in Approach 2, the primary structures and their corresponding PDBs are separated according to the genes' primary structural information. Following the DCNN stage, a dynamic programming-based method is used to determine the final prediction of the primary structures' functionality. We validated our proposed method using the COSMIC online database. For the ONGO vs TSG classification problem, the AUROC of the DCNN stage for Approach 1 and Approach 2 DCNN are 0.978 and 0.765, respectively. The AUROCs of the final genes' primary structure functionality classification for Approach 1 and Approach 2 are 0.989, and 0.879, respectively. For comparison, the current state-of-the-art reported AUROC is 0.924.
Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models
Venue: arXiv cs.CV 2021
Year: 2021

Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, we develop an automatic procedure for finding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision. Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or finetuning. Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foreground-background structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision.
A New Framework to Adopt Multidimensional Databases for Organizational Information System Strategies
Venue: arXiv cs.DB 2021
Year: 2021

As information becomes increasingly sizable for organizations to maintain the challenge of organizing data still remains. More importantly, the on-going process of analysing incoming data occurs on a continual basis and organizations should employ existing procedures that may not be adequate or efficient when attempting to access specific information to analyse. In these latter days of technological advancement, organizations can offer their customers extensive data resources to utilize and thus accomplish individual objectives and maintain competitiveness; however, it remains a challenge in providing data in a format that serves each clients suited needs. For some, the complexity of a data model can be overwhelming to utilize. Furthermore, companies should secure an understanding of the purchasing power used by specific consumer groups to remain competitive and ease the operation of data analysis. This research paper is to examine the use of multi-dimensional models within a business environment and how it may provide customers and managers with generating queries that will provide accurate and relevant data for effective analysis. It also provides a new framework that can aid various types of organisations using sizable database systems to create their own multidimensional model from relational databases and present the data in multidimensional views. It also defines the requirements. Despite the availability of set tools, the complexity of utilizing the conceptions discourages customers as they may become apprehensive about exploring these options for analytical purposes. This could be done by conducting a query.
Modeling the EdNet Dataset with Logistic Regression
Venue: arXiv cs.CY 2021
Year: 2021

Many of these challenges are won by neural network models created by full-time artificial intelligence scientists. Due to this origin, they have a black-box character that makes their use and application less clear to learning scientists. We describe our experience with competition from the perspective of educational data mining, a field founded in the learning sciences and connected with roots in psychology and statistics. We describe our efforts from the perspectives of learning scientists and the challenges to our methods, some real and some imagined. We also discuss some basic results in the Kaggle system and our thoughts on how those results may have been improved. Finally, we describe how learner model predictions are used to make pedagogical decisions for students. Their practical use entails a) model predictions and b) a decision rule (based on the predictions). We point out how increased model accuracy can be of limited practical utility, especially when paired with simple decision rules and argue instead for the need to further investigate optimal decision rules.
(Deep) Induction Rules for GADTs
Venue: arXiv cs.LO 2021
Year: 2021

Deep data types are those that are constructed from other data types, including, possibly, themselves. In this case, they are said to be truly nested. Deep induction is an extension of structural induction that traverses all of the structure in a deep data type, propagating predicates on its primitive data throughout the entire structure. Deep induction can be used to prove properties of nested types, including truly nested types, that cannot be proved via structural induction. In this paper we show how to extend deep induction to GADTs that are not truly nested GADTs. This opens the way to incorporating automatic generation of (deep) induction rules for them into proof assistants. We also show that the techniques developed in this paper do not suffice for extending deep induction to truly nested GADTs, so more sophisticated techniques are needed to derive deep induction rules for them.
Social Behavior and Mental Health: A Snapshot Survey under COVID-19 Pandemic
Venue: arXiv cs.SI 2021
Year: 2021

Online social media provides a channel for monitoring people's social behaviors and their mental distress. Due to the restrictions imposed by COVID-19 people are increasingly using online social networks to express their feelings. Consequently, there is a significant amount of diverse user-generated social media content. However, COVID-19 pandemic has changed the way we live, study, socialize and recreate and this has affected our well-being and mental health problems. There are growing researches that leverage online social media analysis to detect and assess user's mental status. In this paper, we survey the literature of social media analysis for mental disorders detection, with a special focus on the studies conducted in the context of COVID-19 during 2020-2021. Firstly, we classify the surveyed studies in terms of feature extraction types, varying from language usage patterns to aesthetic preferences and online behaviors. Secondly, we explore detection methods used for mental disorders detection including machine learning and deep learning detection methods. Finally, we discuss the challenges of mental disorder detection using social media data, including the privacy and ethical concerns, as well as the technical challenges of scaling and deploying such systems at large scales, and discuss the learnt lessons over the last few years.
LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer
Venue: arXiv cs.CL 2021
Year: 2021

Many types of text style transfer can be achieved with only small, precise edits (e.g. sentiment transfer from I had a terrible time... to I had a great time...). We propose a coarse-to-fine editor for style transfer that transforms text using Levenshtein edit operations (e.g. insert, replace, delete). Unlike prior single-span edit methods, our method concurrently edits multiple spans in the source text. To train without parallel style text pairs (e.g. pairs of +/- sentiment statements), we propose an unsupervised data synthesis procedure. We first convert text to style-agnostic templates using style classifier attention (e.g. I had a SLOT time...), then fill in slots in these templates using fine-tuned pretrained language models. Our method outperforms existing generation and editing style transfer methods on sentiment (Yelp, Amazon) and politeness (Polite) transfer. In particular, multi-span editing achieves higher performance and more diverse output than single-span editing. Moreover, compared to previous methods on unsupervised data synthesis, our method results in higher quality parallel style pairs and improves model performance.
Vertex Ordering Problems in Directed Graph Streams
Venue: arXiv cs.DS 2021
Year: 2021

We consider directed graph algorithms in a streaming setting, focusing on problems concerning orderings of the vertices. This includes such fundamental problems as topological sorting and acyclicity testing. We also study the related problems of finding a minimum feedback arc set (edges whose removal yields an acyclic graph), and finding a sink vertex. We are interested in both adversarially-ordered and randomly-ordered streams. For arbitrary input graphs with edges ordered adversarially, we show that most of these problems have high space complexity, precluding sublinear-space solutions. Some lower bounds also apply when the stream is randomly ordered: e.g., in our most technical result we show that testing acyclicity in the $p$-pass random-order model requires roughly $n^{1+1/p}$ space. For other problems, random ordering can make a dramatic difference: e.g., it is possible to find a sink in an acyclic tournament in the one-pass random-order model using polylog$(n)$ space whereas under adversarial ordering roughly $n^{1/p}$ space is necessary and sufficient given $Θ(p)$ passes. We also design sublinear algorithms for the feedback arc set problem in tournament graphs; for random graphs; and for randomly ordered streams. In some cases, we give lower bounds establishing that our algorithms are essentially space-optimal. Together, our results complement the much maturer body of work on algorithms for undirected graph streams.
Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks
Venue: arXiv cs.CV 2021
Year: 2021

Generative Adversarial Networks (GANs) have made great success in synthesizing high-quality images. However, how to steer the generation process of a well-trained GAN model and customize the output image is much less explored. It has been recently found that modulating the input latent code used in GANs can reasonably alter some variation factors in the output image, but such manipulation usually presents to change the entire image as a whole. In this work, we propose an effective approach, termed as LoGAN, to support local editing of the output image. Concretely, we introduce two operators, i.e., content modulation and style modulation, together with a priority mask to facilitate the precise control of the intermediate generative features. Taking bedroom synthesis as an instance, we are able to seamlessly remove, insert, shift, and rotate the individual objects inside a room. Furthermore, our method can completely clear out a room and then refurnish it with customized furniture and styles. Experimental results show the great potentials of steering the image generation of pre-trained GANs for versatile image editing.
KECRS: Towards Knowledge-Enriched Conversational Recommendation System
Venue: arXiv cs.CL 2021
Year: 2021

The chit-chat-based conversational recommendation systems (CRS) provide item recommendations to users through natural language interactions. To better understand user's intentions, external knowledge graphs (KG) have been introduced into chit-chat-based CRS. However, existing chit-chat-based CRS usually generate repetitive item recommendations, and they cannot properly infuse knowledge from KG into CRS to generate informative responses. To remedy these issues, we first reformulate the conversational recommendation task to highlight that the recommended items should be new and possibly interested by users. Then, we propose the Knowledge-Enriched Conversational Recommendation System (KECRS). Specifically, we develop the Bag-of-Entity (BOE) loss and the infusion loss to better integrate KG with CRS for generating more diverse and informative responses. BOE loss provides an additional supervision signal to guide CRS to learn from both human-written utterances and KG. Infusion loss bridges the gap between the word embeddings and entity embeddings by minimizing distances of the same words in these two embeddings. Moreover, we facilitate our study by constructing a high-quality KG, \ie The Movie Domain Knowledge Graph (TMDKG). Experimental results on a large-scale dataset demonstrate that KECRS outperforms state-of-the-art chit-chat-based CRS, in terms of both recommendation accuracy and response generation quality.