OpenRaiser/Intern-Atlas huggingface dataset:
SPA-GCN: Efficient and Flexible GCN Accelerator with an Application for Graph Similarity Computation
Venue: arXiv cs.LG 2021
Year: 2021

While there have been many studies on hardware acceleration for deep learning on images, there has been a rather limited focus on accelerating deep learning applications involving graphs. The unique characteristics of graphs, such as the irregular memory access and dynamic parallelism, impose several challenges when the algorithm is mapped to a CPU or GPU. To address these challenges while exploiting all the available sparsity, we propose a flexible architecture called SPA-GCN for accelerating Graph Convolutional Networks (GCN), the core computation unit in deep learning algorithms on graphs. The architecture is specialized for dealing with many small graphs since the graph size has a significant impact on design considerations. In this context, we use SimGNN, a neural-network-based graph matching algorithm, as a case study to demonstrate the effectiveness of our architecture. The experimental results demonstrate that SPA-GCN can deliver a high speedup compared to a multi-core CPU implementation and a GPU implementation, showing the efficiency of our design.
Robust Learning via Ensemble Density Propagation in Deep Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Learning in uncertain, noisy, or adversarial environments is a challenging task for deep neural networks (DNNs). We propose a new theoretically grounded and efficient approach for robust learning that builds upon Bayesian estimation and Variational Inference. We formulate the problem of density propagation through layers of a DNN and solve it using an Ensemble Density Propagation (EnDP) scheme. The EnDP approach allows us to propagate moments of the variational probability distribution across the layers of a Bayesian DNN, enabling the estimation of the mean and covariance of the predictive distribution at the output of the model. Our experiments using MNIST and CIFAR-10 datasets show a significant improvement in the robustness of the trained models to random noise and adversarial attacks.
Feature Generation for Long-tail Classification
Venue: arXiv cs.CV 2021
Year: 2021

The visual world naturally exhibits an imbalance in the number of object or scene instances resulting in a \emph{long-tailed distribution}. This imbalance poses significant challenges for classification models based on deep learning. Oversampling instances of the tail classes attempts to solve this imbalance. However, the limited visual diversity results in a network with poor representation ability. A simple counter to this is decoupling the representation and classifier networks and using oversampling only to train the classifier. In this paper, instead of repeatedly re-sampling the same image (and thereby features), we explore a direction that attempts to generate meaningful features by estimating the tail category's distribution. Inspired by ideas from recent work on few-shot learning, we create calibrated distributions to sample additional features that are subsequently used to train the classifier. Through several experiments on the CIFAR-100-LT (long-tail) dataset with varying imbalance factors and on mini-ImageNet-LT (long-tail), we show the efficacy of our approach and establish a new state-of-the-art. We also present a qualitative analysis of generated features using t-SNE visualizations and analyze the nearest neighbors used to calibrate the tail class distributions. Our code is available at https://github.com/rahulvigneswaran/TailCalibX.
Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
Venue: arXiv cs.LG 2021
Year: 2021

With deep learning models rapidly growing in size, systems-level solutions for large-model training are required. We present Amazon SageMaker model parallelism, a software library that integrates with PyTorch, and enables easy training of large models using model parallelism and other memory-saving features. In contrast to existing solutions, the implementation of the SageMaker library is much more generic and flexible, in that it can automatically partition and run pipeline parallelism over arbitrary model architectures with minimal code change, and also offers a general and extensible framework for tensor parallelism, which supports a wider range of use cases, and is modular enough to be easily applied to new training scripts. The library also preserves the native PyTorch user experience to a much larger degree, supporting module re-use and dynamic graphs, while giving the user full control over the details of the training step. We evaluate performance over GPT-3, RoBERTa, BERT, and neural collaborative filtering, and demonstrate competitive performance over existing solutions.
Towards Understanding Enablers of Digital Transformation in Small and Medium-Sized Enterprises
Venue: arXiv cs.DL 2021
Year: 2021

Even though, digital transformation has attracted much attention of both academics and practitioners, a very limited number of studies have investigated the digital transformation process in small and medium-sized enterprises (SMEs) and the findings remain fragmented. Given the accessibility and availability of digital technologies to launch digital transformation initiatives and the importance of SMEs in the economy, a profound understanding of enablers of the digital transformation process in SMEs is much needed. As such, to address this, in this paper we conducted a comprehensive review of related literature in information systems, management, and business disciplines, to identify key enablers that facilitate the digital transformation process in SMEs.
Causal KL: Evaluating Causal Discovery
Venue: arXiv cs.LG 2021
Year: 2021

The two most commonly used criteria for assessing causal model discovery with artificial data are edit-distance and Kullback-Leibler divergence, measured from the true model to the learned model. Both of these metrics maximally reward the true model. However, we argue that they are both insufficiently discriminating in judging the relative merits of false models. Edit distance, for example, fails to distinguish between strong and weak probabilistic dependencies. KL divergence, on the other hand, rewards equally all statistically equivalent models, regardless of their different causal claims. We propose an augmented KL divergence, which we call Causal KL (CKL), which takes into account causal relationships which distinguish between observationally equivalent models. Results are presented for three variants of CKL, showing that Causal KL works well in practice.
Fairness-aware Crowdsourcing of IoT Energy Services
Venue: arXiv cs.DC 2021
Year: 2021

We propose a Novel Fairness-Aware framework for Crowdsourcing Energy Services (FACES) to efficiently provision crowdsourced IoT energy services. Typically, efficient resource provisioning might incur an unfair resource sharing for some requests. FACES, however, maximizes the utilization of the available energy services by maximizing fairness across all requests. We conduct a set of preliminary experiments to assess the effectiveness of the proposed framework against traditional fairness-aware resource allocation algorithms. Results demonstrate that the IoT energy utilization of FACES is better than FCFS and similar to Max-min fair scheduling. Experiments also show that better fairness is achieved among the provisioned requests using FACES compared toFCFS and Max-min fair scheduling.
Solving Multi-Arm Bandit Using a Few Bits of Communication
Venue: arXiv cs.LG 2021
Year: 2021

The multi-armed bandit (MAB) problem is an active learning framework that aims to select the best among a set of actions by sequentially observing rewards. Recently, it has become popular for a number of applications over wireless networks, where communication constraints can form a bottleneck. Existing works usually fail to address this issue and can become infeasible in certain applications. In this paper we address the communication problem by optimizing the communication of rewards collected by distributed agents. By providing nearly matching upper and lower bounds, we tightly characterize the number of bits needed per reward for the learner to accurately learn without suffering additional regret. In particular, we establish a generic reward quantization algorithm, QuBan, that can be applied on top of any (no-regret) MAB algorithm to form a new communication-efficient counterpart, that requires only a few (as low as 3) bits to be sent per iteration while preserving the same regret bound. Our lower bound is established via constructing hard instances from a subgaussian distribution. Our theory is further corroborated by numerically experiments.
Explainable Sentence-Level Sentiment Analysis for Amazon Product Reviews
Venue: arXiv cs.CL 2021
Year: 2021

In this paper, we conduct a sentence level sentiment analysis on the product reviews from Amazon and thorough analysis on the model interpretability. For the sentiment analysis task, we use the BiLSTM model with attention mechanism. For the study of interpretability, we consider the attention weights distribution of single sentence and the attention weights of main aspect terms. The model has an accuracy of up to 0.96. And we find that the aspect terms have the same or even more attention weights than the sentimental words in sentences.
Catching Unusual Traffic Behavior using TF-IDF-based Port Access Statistics Analysis
Venue: arXiv cs.NI 2021
Year: 2021

Detecting the anomalous behavior of traffic is one of the important actions for network operators. In this study, we applied term frequency - inverse document frequency (TF-IDF), which is a popular method used in natural language processing, to detect unusual behavior from network access logs. We mapped the term and document concept to the port number and daily access history, respectively, and calculated the TF-IDF. With this approach, we could obtain ports frequently observed in fewer days compared to other port access activities. Such access behaviors are not always malicious activities; however, such information is a good indicator for starting a deeper analysis of traffic behavior. Using a real-life dataset, we could detect two bot-oriented accesses and one unique UDP traffic.
At the Edge of a Seamless Cloud Experience
Venue: arXiv cs.DC 2021
Year: 2021

There is a growing need for low latency for many devices and users. The traditional cloud computing paradigm can not meet this requirement, legitimizing the need for a new paradigm. Edge computing proposes to move computing capacities to the edge of the network, closer to where data is produced and consumed. However, edge computing raises new challenges. At the edge, devices are more heterogeneous than in the data centre, where everything is optimized to achieve economies of scale. Edge devices can be mobile, like a car, which complicates architecture with dynamic topologies. IoT devices produce a considerable amount of data that can be processed at the Edge. In this paper, we discuss the main challenges to be met in edge computing and solutions to achieve a seamless cloud experience. We propose to use technologies like containers and WebAssembly to manage applications' execution on heterogeneous devices.
Training neural networks with synthetic electrocardiograms
Venue: arXiv cs.LG 2021
Year: 2021

We present a method for training neural networks with synthetic electrocardiograms that mimic signals produced by a wearable single lead electrocardiogram monitor. We use domain randomization where the synthetic signal properties such as the waveform shape, RR-intervals and noise are varied for every training example. Models trained with synthetic data are compared to their counterparts trained with real data. Detection of r-waves in electrocardiograms recorded during different physical activities and in atrial fibrillation is used to compare the models. By allowing the randomization to increase beyond what is typically observed in the real-world data the performance is on par or superseding the performance of networks trained with real data. Experiments show robust performance with different seeds and training examples on different test sets without any test set specific tuning. The method makes possible to train neural networks using practically free-to-collect data with accurate labels without the need for manual annotations and it opens up the possibility of extending the use of synthetic data on cardiac disease classification when disease specific a priori information is used in the electrocardiogram generation. Additionally the distribution of data can be controlled eliminating class imbalances that are typically observed in health related data and additionally the generated data is inherently private.
Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training
Venue: arXiv cs.CL 2021
Year: 2021

Virtual Adversarial Training (VAT) has been effective in learning robust models under supervised and semi-supervised settings for both computer vision and NLP tasks. However, the efficacy of VAT for multilingual and multilabel text classification has not been explored before. In this work, we explore VAT for multilabel emotion recognition with a focus on leveraging unlabelled data from different languages to improve the model performance. We perform extensive semi-supervised experiments on SemEval2018 multilabel and multilingual emotion recognition dataset and show performance gains of 6.2% (Arabic), 3.8% (Spanish) and 1.8% (English) over supervised learning with same amount of labelled data (10% of training data). We also improve the existing state-of-the-art by 7%, 4.5% and 1% (Jaccard Index) for Spanish, Arabic and English respectively and perform probing experiments for understanding the impact of different layers of the contextual models.
Agile Information System Development Organizations Transforming to Large-Scale Collaboration
Venue: arXiv cs.SE 2021
Year: 2021

We report findings from a case study of a large agile information systems development (ISD) organization`s sudden transformation to distributed, digital work in the context of the Covid-19 pandemic. It seeks to understand how knowledge creation and sharing changes. The findings show various forms of distance being introduced, digital tool usage, increased task orientation, and variations across teams. To analyze the findings, we use the concepts of large-scale collaborations and sociability. Large-scale collaboration offers a socio-technical perspective on tackling distributed knowledge sharing and creation in the presence of multiple, loosely coupled partners using digital tools for collaboration. We show what the digital tools afford using the concept of sociability. We discuss how distributed digital practices make teams more task-oriented and that creating and maintaining sociability, a key issue for knowledge sharing in agile ISD organizations, require relation oriented communication during practical problem solving using digital tools.
Improvements to short-term weather prediction with recurrent-convolutional networks
Venue: arXiv cs.LG 2021
Year: 2021

The Weather4cast 2021 competition gave the participants a task of predicting the time evolution of two-dimensional fields of satellite-based meteorological data. This paper describes the author's efforts, after initial success in the first stage of the competition, to improve the model further in the second stage. The improvements consisted of a shallower model variant that is competitive against the deeper version, adoption of the AdaBelief optimizer, improved handling of one of the predicted variables where the training set was found not to represent the validation set well, and ensembling multiple models to improve the results further. The largest quantitative improvements to the competition metrics can be attributed to the increased amount of training data available in the second stage of the competition, followed by the effects of model ensembling. Qualitative results show that the model can predict the time evolution of the fields, including the motion of the fields over time, starting with sharp predictions for the immediate future and blurring of the outputs in later frames to account for the increased uncertainty.
Branch and Bound in Mixed Integer Linear Programming Problems: A Survey of Techniques and Trends
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we surveyed the existing literature studying different approaches and algorithms for the four critical components in the general branch and bound (B&B) algorithm, namely, branching variable selection, node selection, node pruning, and cutting-plane selection. However, the complexity of the B&B algorithm always grows exponentially with respect to the increase of the decision variable dimensions. In order to improve the speed of B&B algorithms, learning techniques have been introduced in this algorithm recently. We further surveyed how machine learning can be used to improve the four critical components in B&B algorithms. In general, a supervised learning method helps to generate a policy that mimics an expert but significantly improves the speed. An unsupervised learning method helps choose different methods based on the features. In addition, models trained with reinforcement learning can beat the expert policy, given enough training and a supervised initialization. Detailed comparisons between different algorithms have been summarized in our survey. Finally, we discussed some future research directions to accelerate and improve the algorithms further in the literature.
Online Discrepancy with Recourse for Vectors and Graphs
Venue: arXiv cs.DS 2021
Year: 2021

The vector-balancing problem is a fundamental problem in discrepancy theory: given T vectors in $[-1,1]^n$, find a signing $σ(a) \in \{\pm 1\}$ of each vector $a$ to minimize the discrepancy $\| \sum_{a} σ(a) \cdot a \|_{\infty}$. This problem has been extensively studied in the static/offline setting. In this paper we initiate its study in the fully-dynamic setting with recourse: the algorithm sees a stream of T insertions and deletions of vectors, and at each time must maintain a low-discrepancy signing, while also minimizing the amortized recourse (the number of times any vector changes its sign) per update. For general vectors, we show algorithms which almost match Spencer's $O(\sqrt{n})$ offline discrepancy bound, with ${O}(n\cdot poly\!\log T)$ amortized recourse per update. The crucial idea is to compute a basic feasible solution to the linear relaxation in a distributed and recursive manner, which helps find a low-discrepancy signing. To bound recourse we argue that only a small part of the instance needs to be re-computed at each update. Since vector balancing has also been greatly studied for sparse vectors, we then give algorithms for low-discrepancy edge orientation, where we dynamically maintain signings for 2-sparse vectors. Alternatively, this can be seen as orienting a dynamic set of edges of an n-vertex graph to minimize the absolute difference between in- and out-degrees at any vertex. We present a deterministic algorithm with $O(poly\!\log n)$ discrepancy and $O(poly\!\log n)$ amortized recourse. The core ideas are to dynamically maintain an expander-decomposition with low recourse and then to show that, as the expanders change over time, a natural local-search algorithm converges quickly (i.e., with low recourse) to a low-discrepancy solution. We also give strong lower bounds for local-search discrepancy minimization algorithms.
How player and opponent personalities influence cooperative gameplay
Venue: arXiv cs.MM 2021
Year: 2021

Research has shown that digital game players often feel engagement and rapport with a game hero or character when they can channel their own ambitions and goals through the hero's journey in the game world; in essence, they feel a sense of accomplishment and fulfilment whenever they put the game mechanics to use to help the hero reach a positive ending to the game quests. In the case of cooperative gameplay, rapport also has to do with their perception of their peers' skills, gameplay style and behaviour within the game. In this paper, we describe an experiment to identify whether matching players with different personalities, as characterized by the OCEAN or Big-5 personality model, can influence their player experience with a custom-made, cooperative game.
Driver-Specific Risk Recognition in Interactive Driving Scenarios using Graph Representation
Venue: arXiv cs.RO 2021
Year: 2021

This paper presents a driver-specific risk recognition framework for autonomous vehicles that can extract inter-vehicle interactions. This extraction is carried out for urban driving scenarios in a driver-cognitive manner to improve the recognition accuracy of risky scenes. First, clustering analysis is applied to the operation data of drivers for learning the subjective assessment of risky scenes of different drivers and generating the corresponding risk label for each scene. Second, the graph representation model (GRM) is adopted to unify and construct the features of dynamic vehicles, inter-vehicle interactions and static traffic markings in real driving scenes into graphs. The driver-specific risk label provides ground truth to capture the risk evaluation criteria of different drivers. In addition, the graph model represents multiple features of the driving scenes. Therefore, the proposed framework can learn the risk-evaluating pattern of driving scenes of different drivers and establish driver-specific risk identifiers. Last, the performance of the proposed framework is evaluated via experiments conducted using real-world urban driving datasets collected by multiple drivers. The results show that the risks and their levels in real driving environments can be accurately recognized by the proposed framework.
Keyless Authentication for AWGN Channels
Venue: arXiv cs.IT 2021
Year: 2021

This work establishes that the physical layer can be used to perform information-theoretic authentication in additive white Gaussian noise channels, as long as the adversary is not omniscient. The model considered consists of an encoder, decoder, and adversary, where the adversary has access to the message, a non-causal noisy observation of the encoder's transmission, and unlimited transmission power, while the decoder observes a noisy version of the sum of the encoder and adversary's outputs. A method to modify a generic existing channel code to enable authentication is presented. It is shown that this modification costs an asymptotically negligible amount of the coding rate, while still enabling authentication as long as the adversary's observation is not noiseless. Also notable is that this modification is not (asymptotically) a function of the statistical characterization of the adversary's channel and furthermore no secret key is required, hence paving the way for a robust practical implementation. Using these results, the channel-authenticated capacity is calculated and shown to be equal to the non-adversarial channel capacity. While this modular scheme is designed for use in the given channel model, it is applicable to a wide range of settings.