OpenRaiser/Intern-Atlas huggingface dataset:
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Venue: arXiv cs.CL 2020
Year: 2020

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).
Don't cross that stop line: Characterizing Traffic Violations in Metropolitan Cities
Venue: arXiv cs.CY 2020
Year: 2020

In modern metropolitan cities, the task of ensuring safe roads is of paramount importance. Automated systems of e-challans (Electronic traffic-violation receipt) are now being deployed across cities to record traffic violations and to issue fines. In the present study, an automated e-challan system established in Ahmedabad (Gujarat, India) has been analyzed for characterizing user behaviour, violation types as well as finding spatial and temporal patterns in the data. We describe a method of collecting e-challan data from the e-challan portal of Ahmedabad traffic police and create a dataset of over 3 million e-challans. The dataset was first analyzed to characterize user behaviour with respect to repeat offenses and fine payment. We demonstrate that a lot of users repeat their offenses (traffic violation) frequently and are less likely to pay fines of higher value. Next, we analyze the data from a spatial and temporal perspective and identify certain spatio-temporal patterns present in our dataset. We find that there is a drastic increase/decrease in the number of e-challans issued during the festival days and identify a few hotspots in the city that have high intensity of traffic violations. In the end, we propose a set of 5 features to model recidivism in traffic violations and train multiple classifiers on our dataset to evaluate the effectiveness of our proposed features. The proposed approach achieves 95% accuracy on the dataset.
SUPP.AI: Finding Evidence for Supplement-Drug Interactions
Venue: arXiv cs.CL 2020
Year: 2020

Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete. To address this challenge, we present SUPP.AI, an application for browsing evidence of supplement-drug interactions (SDIs) extracted from the biomedical literature. We train a model to automatically extract supplement information and identify such interactions from the scientific literature. To address the lack of labeled data for SDI identification, we use labels of the closely related task of identifying drug-drug interactions (DDIs) for supervision. We fine-tune the contextualized word representations of the RoBERTa language model using labeled DDI data, and apply the fine-tuned model to identify supplement interactions. We extract 195k evidence sentences from 22M articles (P=0.82, R=0.58, F1=0.68) for 60k interactions. We create the SUPP.AI application for users to search evidence sentences extracted by our model. SUPP.AI is an attempt to close the information gap on dietary supplements by making up-to-date evidence on SDIs more discoverable for researchers, clinicians, and consumers.
Default Disambiguation for Online Parsers
Venue: arXiv cs.PL 2020
Year: 2020

Since composed grammars are often ambiguous, grammar composition requires a mechanism for dealing with ambiguity: either ruling it out by using delimiters (which are awkward to work with), or by using disambiguation operators to filter a parse forest down to a single parse tree (where, in general, we cannot be sure that we have covered all possible parse forests). In this paper, we show that default disambiguation, which is inappropriate for batch parsing, works well for online parsing, where it can be overridden by the user if necessary. We extend language boxes -- a delimiter-based algorithm atop incremental parsing -- in such a way that default disambiguation can automatically insert, remove, or resize, language boxes, leading to the automatic language boxes algorithm. The nature of the problem means that default disambiguation cannot always match a user's intention. However, our experimental evaluation shows that automatic language boxes behave acceptably in 98.8% of tests involving compositions of real-world programming languages.
Assignment and Pricing of Shared Rides in Ride-Sourcing using Combinatorial Double Auctions
Venue: arXiv cs.CC 2020
Year: 2020

Transportation Network Companies employ dynamic pricing methods at periods of peak travel to incentivise driver participation and balance supply and demand for rides. Surge pricing multipliers are commonly used and are applied following demand and estimates of customer and driver trip valuations. Combinatorial double auctions have been identified as a suitable alternative, as they can achieve maximum social welfare in the allocation by relying on customers and drivers stating their valuations. A shortcoming of current models, however, is that they fail to account for the effects of trip detours that take place in shared trips and their impact on the accuracy of pricing estimates. To resolve this, we formulate a new shared-ride assignment and pricing algorithm using combinatorial double auctions. We demonstrate that this model is reduced to a maximum weighted independent set model, which is known to be APX-hard. A fast local search heuristic is also presented, which is capable of producing results that lie within 10% of the exact approach for practical implementations. Our proposed algorithm could be used as a fast and reliable assignment and pricing mechanism of ride-sharing requests to vehicles during peak travel times.
Edge-Computing-Enabled Smart Cities: A Comprehensive Survey
Venue: arXiv cs.NI 2020
Year: 2020

Recent years have disclosed a remarkable proliferation of compute-intensive applications in smart cities. Such applications continuously generate enormous amounts of data which demand strict latency-aware computational processing capabilities. Although edge computing is an appealing technology to compensate for stringent latency related issues, its deployment engenders new challenges. In this survey, we highlight the role of edge computing in realizing the vision of smart cities. First, we analyze the evolution of edge computing paradigms. Subsequently, we critically review the state-of-the-art literature focusing on edge computing applications in smart cities. Later, we categorize and classify the literature by devising a comprehensive and meticulous taxonomy. Furthermore, we identify and discuss key requirements, and enumerate recently reported synergies of edge computing enabled smart cities. Finally, several indispensable open challenges along with their causes and guidelines are discussed, serving as future research directions.
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
Venue: arXiv cs.CL 2020
Year: 2020

Recently several datasets have been proposed to encourage research in Question Answering domains where commonsense knowledge is expected to play an important role. Recent language models such as ROBERTA, BERT and GPT that have been pre-trained on Wikipedia articles and books have shown reasonable performance with little fine-tuning on several such Multiple Choice Question-Answering (MCQ) datasets. Our goal in this work is to develop methods to incorporate additional (commonsense) knowledge into language model-based approaches for better question-answering in such domains. In this work, we first categorize external knowledge sources, and show performance does improve on using such sources. We then explore three different strategies for knowledge incorporation and four different models for question-answering using external commonsense knowledge. We analyze our predictions to explore the scope of further improvements.
Training Robust Deep Neural Networks via Adversarial Noise Propagation
Venue: arXiv cs.LG 2020
Year: 2020

In practice, deep neural networks have been found to be vulnerable to various types of noise, such as adversarial examples and corruption. Various adversarial defense methods have accordingly been developed to improve adversarial robustness for deep models. However, simply training on data mixed with adversarial examples, most of these models still fail to defend against the generalized types of noise. Motivated by the fact that hidden layers play a highly important role in maintaining a robust model, this paper proposes a simple yet powerful training algorithm, named \emph{Adversarial Noise Propagation} (ANP), which injects noise into the hidden layers in a layer-wise manner. ANP can be implemented efficiently by exploiting the nature of the backward-forward training style. Through thorough investigations, we determine that different hidden layers make different contributions to model robustness and clean accuracy, while shallow layers are comparatively more critical than deep layers. Moreover, our framework can be easily combined with other adversarial training methods to further improve model robustness by exploiting the potential of hidden layers. Extensive experiments on MNIST, CIFAR-10, CIFAR-10-C, CIFAR-10-P, and ImageNet demonstrate that ANP enables the strong robustness for deep models against both adversarial and corrupted ones, and also significantly outperforms various adversarial defense methods.
Adaptively Aligned Image Captioning via Adaptive Attention Time
Venue: arXiv cs.CV 2020
Year: 2020

Recent neural models for image captioning usually employ an encoder-decoder framework with an attention mechanism. However, the attention mechanism in such a framework aligns one single (attended) image feature vector to one caption word, assuming one-to-one mapping from source image regions and target caption words, which is never possible. In this paper, we propose a novel attention model, namely Adaptive Attention Time (AAT), to align the source and the target adaptively for image captioning. AAT allows the framework to learn how many attention steps to take to output a caption word at each decoding step. With AAT, an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions. AAT is deterministic and differentiable, and doesn't introduce any noise to the parameter gradients. In this paper, we empirically show that AAT improves over state-of-the-art methods on the task of image captioning. Code is available at https://github.com/husthuaan/AAT.
Meta-Neighborhoods
Venue: arXiv cs.LG 2020
Year: 2020

Making an adaptive prediction based on one's input is an important ability for general artificial intelligence. In this work, we step forward in this direction and propose a semi-parametric method, Meta-Neighborhoods, where predictions are made adaptively to the neighborhood of the input. We show that Meta-Neighborhoods is a generalization of $k$-nearest-neighbors. Due to the simpler manifold structure around a local neighborhood, Meta-Neighborhoods represent the predictive distribution $p(y \mid x)$ more accurately. To reduce memory and computation overhead, we propose induced neighborhoods that summarize the training data into a much smaller dictionary. A meta-learning based training mechanism is then exploited to jointly learn the induced neighborhoods and the model. Extensive studies demonstrate the superiority of our method.
Nonlinear Multiview Analysis: Identifiability and Neural Network-assisted Implementation
Venue: arXiv cs.LG 2020
Year: 2020

Multiview analysis aims at extracting shared latent components from data samples that are acquired in different domains, e.g., image, text, and audio. Classic multiview analysis, e.g., canonical correlation analysis (CCA), tackles this problem via matching the linearly transformed views in a certain latent domain. More recently, powerful nonlinear learning tools such as kernel methods and neural networks are utilized for enhancing the classic CCA. However, unlike linear CCA whose theoretical aspects are clearly understood, nonlinear CCA approaches are largely intuition-driven. In particular, it is unclear under what conditions the shared latent components across the views can be identified---while identifiability plays an essential role in many applications. In this work, we revisit nonlinear multiview analysis and address both the theoretical and computational aspects. Our work leverages a useful nonlinear model, namely, the post-nonlinear model, from the nonlinear mixture separation literature. Combining with multiview data, we take a nonlinear multiview mixture learning viewpoint, which is a natural extension of the classic generative models for linear CCA. From there, we derive a learning criterion. We show that minimizing this criterion leads to identification of the latent shared components up to certain ambiguities, under reasonable conditions. Our derivation and formulation also offer new insights and interpretations to existing deep neural network-based CCA formulations. On the computation side, we propose an effective algorithm with simple and scalable update rules. A series of simulations and real-data experiments corroborate our theoretical analysis.
LULC classification methodology based on simple Convolutional Neural Network to map complex urban forms at finer scale: Evidence from Mumbai
Venue: arXiv cs.CY 2020
Year: 2020

The satellite imagery classification task is fundamental to spatial knowledge discovery. Several image classification methods are used to create standardized Land use and Land cover (LULC) maps, which facilitate research on spatial and ecological processes and human activities. Local Climate Zones (LCZ) classification maps are an example of standardized maps which have been widely used to demarcate the homogeneity in built and natural character in the cities. The LCZ classification scheme is primarily focused on urban climate-related research, in which 17 climate zones are mapped in a city area with the 100-150m spatial resolution. Each zone exhibits physical properties related to urban form and functions essential for thermal behavior studies. Extending this widely adopted approach to create LULC maps at finer resolution using the LCZ mapping scheme would benefit the allied domains of urban planning, transportation, and water resources management. This study proposes a novel solution to generate classification maps with a 10-band Sentinel-2B dataset and Convolutional Neural Networks (CNN) at the 10m spatial resolution. The classification benefits from CNNs property to preserve local structures in the image datasets. The proposed CNN model outperforms traditional machine learning models such as Artificial Neural Network, Random Forests, and Support Vector Machines. The overall accuracy and kappa statistic of the CNN model trained on 14 urban and natural classes are 82 percent and 0.81, respectively. The study also discusses the utility of the model for specialized remote sensing tasks such as change detection, identification of slum settlements, and mapping pervious/impervious layers in urban settlements with higher accuracy.
Towards Interpreting Recurrent Neural Networks through Probabilistic Abstraction
Venue: arXiv cs.LG 2020
Year: 2020

Neural networks are becoming a popular tool for solving many real-world problems such as object recognition and machine translation, thanks to its exceptional performance as an end-to-end solution. However, neural networks are complex black-box models, which hinders humans from interpreting and consequently trusting them in making critical decisions. Towards interpreting neural networks, several approaches have been proposed to extract simple deterministic models from neural networks. The results are not encouraging (e.g., low accuracy and limited scalability), fundamentally due to the limited expressiveness of such simple models. In this work, we propose an approach to extract probabilistic automata for interpreting an important class of neural networks, i.e., recurrent neural networks. Our work distinguishes itself from existing approaches in two important ways. One is that probability is used to compensate for the loss of expressiveness. This is inspired by the observation that human reasoning is often `probabilistic'. The other is that we adaptively identify the right level of abstraction so that a simple model is extracted in a request-specific way. We conduct experiments on several real-world datasets using state-of-the-art architectures including GRU and LSTM. The result shows that our approach significantly improves existing approaches in terms of accuracy or scalability. Lastly, we demonstrate the usefulness of the extracted models through detecting adversarial texts.
A Game-Theoretic Analysis of the Social Impact of Connected and Automated Vehicles
Venue: arXiv cs.GT 2020
Year: 2020

In this paper, we address the much-anticipated deployment of connected and automated vehicles (CAVs) in society by modeling and analyzing the social-mobility dilemma in a game-theoretic approach. We formulate this dilemma as a normal-form game of players making a binary decision: whether to travel with a CAV (CAV travel) or not (non-CAV travel) and by constructing an intuitive payoff function inspired by the socially beneficial outcomes of a mobility system consisting of CAVs. We show that the game is equivalent to the Prisoner's dilemma, which implies that the rational collective decision is the opposite of the socially optimum. We present two different solutions to tackle this phenomenon: one with a preference structure and the other with institutional arrangements. In the first approach, we implement a social mechanism that incentivizes players to non-CAV travel and derive a lower bound on the players that ensures an equilibrium of non-CAV travel. In the second approach, we investigate the possibility of players bargaining to create an institution that enforces non-CAV travel and show that as the number of players increases, the incentive ratio of non-CAV travel over CAV travel tends to zero. We conclude by showcasing the last result with a numerical study.
Verified Uncertainty Calibration
Venue: arXiv cs.LG 2020
Year: 2020

Applications such as weather forecasting and personalized medicine demand models that output calibrated probability estimates---those representative of the true likelihood of a prediction. Most models are not calibrated out of the box but are recalibrated by post-processing model outputs. We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are. An alternative method, histogram binning, has measurable calibration error but is sample inefficient---it requires $O(B/ε^2)$ samples, compared to $O(1/ε^2)$ for scaling methods, where $B$ is the number of distinct probabilities the model can output. To get the best of both worlds, we introduce the scaling-binning calibrator, which first fits a parametric function to reduce variance and then bins the function values to actually ensure calibration. This requires only $O(1/ε^2 + B)$ samples. Next, we show that we can estimate a model's calibration error more accurately using an estimator from the meteorological community---or equivalently measure its calibration error with fewer samples ($O(\sqrt{B})$ instead of $O(B)$). We validate our approach with multiclass calibration experiments on CIFAR-10 and ImageNet, where we obtain a 35% lower calibration error than histogram binning and, unlike scaling methods, guarantees on true calibration. In these experiments, we also estimate the calibration error and ECE more accurately than the commonly used plugin estimators. We implement all these methods in a Python library: https://pypi.org/project/uncertainty-calibration
Cross Z-Complementary Pairs for Optimal Training in Spatial Modulation over Frequency Selective Channels
Venue: arXiv cs.IT 2020
Year: 2020

The contributions of this paper are twofold: Firstly, we introduce a novel class of sequence pairs, called "cross Z-complementary pairs (CZCPs)", each displaying zero-correlation zone (ZCZ) properties for both their aperiodic autocorrelation sums and crosscorrelation sums. Systematic constructions of perfect CZCPs based on selected Golay complementary pairs (GCPs) are presented. Secondly, we point out that CZCPs can be utilized as a key component in designing training sequences for broadband spatial modulation (SM) systems. We show that our proposed SM training sequences derived from CZCPs lead to optimal channel estimation performance over frequency-selective channels.
Improved Bounds for Perfect Sampling of $k$-Colorings in Graphs
Venue: arXiv cs.DS 2020
Year: 2020

We present a randomized algorithm that takes as input an undirected $n$-vertex graph $G$ with maximum degree $Δ$ and an integer $k > 3Δ$, and returns a random proper $k$-coloring of $G$. The distribution of the coloring is \emph{perfectly} uniform over the set of all proper $k$-colorings; the expected running time of the algorithm is $\mathrm{poly}(k,n)=\widetilde{O}(nΔ^2\cdot \log(k))$. This improves upon a result of Huber~(STOC 1998) who obtained a polynomial time perfect sampling algorithm for $k>Δ^2+2Δ$. Prior to our work, no algorithm with expected running time $\mathrm{poly}(k,n)$ was known to guarantee perfectly sampling with sub-quadratic number of colors in general. Our algorithm (like several other perfect sampling algorithms including Huber's) is based on the Coupling from the Past method. Inspired by the \emph{bounding chain} approach, pioneered independently by Huber~(STOC 1998) and Häggström \& Nelander~(Scand.{} J.{} Statist., 1999), we employ a novel bounding chain to derive our result for the graph coloring problem.