OpenRaiser/Intern-Atlas huggingface dataset:
Improving Uncertainty Calibration via Prior Augmented Data
Venue: arXiv cs.LG 2021
Year: 2021

Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. However, they are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. The problem of overconfidence becomes especially apparent in cases where the test-time data distribution differs from that which was seen during training. We propose a solution to this problem by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels. Our method results in a better calibrated network and is agnostic to the underlying model structure, so it can be applied to any neural network which produces a probability density as an output. We demonstrate the effectiveness of our method and validate its performance on both classification and regression problems, applying it to recent probabilistic neural network models.
A novel method based on node correlation to evaluate the important nodes in complex networks
Venue: arXiv cs.SI 2021
Year: 2021

Finding the important nodes in complex networks by topological structure is of great significance to network invulnerability. Several centrality measures have been proposed recently to evaluate the performance of nodes based on their correlation, showing that the interaction between nodes has an influence on the importance of nodes. In this paper, a novel method based on node distribution and global influence in complex networks is proposed. Our main idea is that the importance of nodes being linked not only to the relative position in the network but also to the correlations with each other. The nodes in the complex networks are classified according to the distance matrix, then the correlation coefficient between pairs of nodes is calculated. From the whole perspective in the network, the global similarity centrality (GSC) is proposed based on the relevance and shortest distance between any two nodes. The efficiency, accuracy and monotonicity of the proposed method are analyzed in two artificial datasets and eight real datasets of different sizes. Experimental results show that the performance of GSC method outperforms those current state-of-the-art algorithms.
BayesPerf: Minimizing Performance Monitoring Errors Using Bayesian Statistics
Venue: arXiv cs.DC 2021
Year: 2021

Hardware performance counters (HPCs) that measure low-level architectural and microarchitectural events provide dynamic contextual information about the state of the system. However, HPC measurements are error-prone due to non determinism (e.g., undercounting due to event multiplexing, or OS interrupt-handling behaviors). In this paper, we present BayesPerf, a system for quantifying uncertainty in HPC measurements by using a domain-driven Bayesian model that captures microarchitectural relationships between HPCs to jointly infer their values as probability distributions. We provide the design and implementation of an accelerator that allows for low-latency and low-power inference of the BayesPerf model for x86 and ppc64 CPUs. BayesPerf reduces the average error in HPC measurements from 40.1% to 7.6% when events are being multiplexed. The value of BayesPerf in real-time decision-making is illustrated with a simple example of scheduling of PCIe transfers.
On the robustness of randomized classifiers to adversarial examples
Venue: arXiv cs.LG 2021
Year: 2021

This paper investigates the theory of robustness against adversarial attacks. We focus on randomized classifiers (\emph{i.e.} classifiers that output random variables) and provide a thorough analysis of their behavior through the lens of statistical learning theory and information theory. To this aim, we introduce a new notion of robustness for randomized classifiers, enforcing local Lipschitzness using probability metrics. Equipped with this definition, we make two new contributions. The first one consists in devising a new upper bound on the adversarial generalization gap of randomized classifiers. More precisely, we devise bounds on the generalization gap and the adversarial gap (\emph{i.e.} the gap between the risk and the worst-case risk under attack) of randomized classifiers. The second contribution presents a yet simple but efficient noise injection method to design robust randomized classifiers. We show that our results are applicable to a wide range of machine learning models under mild hypotheses. We further corroborate our findings with experimental results using deep neural networks on standard image datasets, namely CIFAR-10 and CIFAR-100. All robust models we trained models can simultaneously achieve state-of-the-art accuracy (over $0.82$ clean accuracy on CIFAR-10) and enjoy \emph{guaranteed} robust accuracy bounds ($0.45$ against $\ell_2$ adversaries with magnitude $0.5$ on CIFAR-10).
Anchor-Assisted Channel Estimation for Intelligent Reflecting Surface Aided Multiuser Communication
Venue: arXiv cs.IT 2021
Year: 2021

Channel estimation is a practical challenge for intelligent reflecting surface (IRS) aided wireless communication. As the number of IRS reflecting elements or IRS-aided users increases, the channel training overhead becomes excessively high, which results in long delay and low throughput in data transmission. To tackle this challenge, we propose in this paper a new anchor-assisted channel estimation approach, where two anchor nodes, namely A1 and A2, are deployed near the IRS for facilitating its aided base station (BS) in acquiring the cascaded BS-IRS-user channels required for data transmission. Specifically, in the first scheme, the partial channel state information (CSI) on the element-wise channel gain square of the common BS-IRS link for all users is first obtained at the BS via the anchor-assisted training and feedback. Then, by leveraging such partial CSI, the cascaded BS-IRS-user channels are efficiently resolved at the BS with additional training by the users. While in the second scheme, the BS-IRS-A1 and A1-IRS-A2 channels are first estimated via the training by A1. Then, with additional training by A2, all users estimate their individual cascaded A2-IRS-user channels simultaneously. Based on the CSI fed back from A2 and all users, the BS resolves the cascaded BS-IRS-user channels efficiently. In both schemes, the quasi-static channels among the fixed BS, IRS, and two anchors are estimated off-line only, which greatly reduces the real-time training overhead. Simulation results demonstrate that our proposed anchor-assisted channel estimation schemes achieve superior performance as compared to existing IRS channel estimation schemes, under various practical setups. In addition, the first proposed scheme outperforms the second one when the number of antennas at the BS is sufficiently large, and vice versa.
Approximation of dilation-based spatial relations to add structural constraints in neural networks
Venue: arXiv cs.CV 2021
Year: 2021

Spatial relations between objects in an image have proved useful for structural object recognition. Structural constraints can act as regularization in neural network training, improving generalization capability with small datasets. Several relations can be modeled as a morphological dilation of a reference object with a structuring element representing the semantics of the relation, from which the degree of satisfaction of the relation between another object and the reference object can be derived. However, dilation is not differentiable, requiring an approximation to be used in the context of gradient-descent training of a network. We propose to approximate dilations using convolutions based on a kernel equal to the structuring element. We show that the proposed approximation, even if slightly less accurate than previous approximations, is definitely faster to compute and therefore more suitable for computationally intensive neural network applications.
CoinTossX: An open-source low-latency high-throughput matching engine
Venue: arXiv cs.DC 2021
Year: 2021

We deploy and demonstrate the CoinTossX low-latency, high-throughput, open-source matching engine with orders sent using the Julia and Python languages. We show how this can be deployed for small-scale local desk-top testing and discuss a larger scale, but local hosting, with multiple traded instruments managed concurrently and managed by multiple clients. We then demonstrate a cloud based deployment using Microsoft Azure, with large-scale industrial and simulation research use cases in mind. The system is exposed and interacted with via sockets using UDP SBE message protocols and can be monitored using a simple web browser interface using HTTP. We give examples showing how orders can be be sent to the system and market data feeds monitored using the Julia and Python languages. The system is developed in Java with orders submitted as binary encodings (SBE) via UDP protocols using the Aeron Media Driver as the low-latency, high throughput message transport. The system separates the order-generation and simulation environments e.g. agent-based model simulation, from the matching of orders, data-feeds and various modularised components of the order-book system. This ensures a more natural and realistic asynchronicity between events generating orders, and the events associated with order-book dynamics and market data-feeds. We promote the use of Julia as the preferred order submission and simulation environment.
Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
Venue: arXiv cs.CL 2021
Year: 2021

We study the problem of incorporating prior knowledge into a deep Transformer-based model,i.e.,Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.
Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. However, these existing approaches simulate the test conditions too strictly during the training process, and thus cannot make full use of the given label information. Besides, these approaches mainly focus on the foreground-background target class segmentation setting. They only utilize binary mask labels for training. In this paper, we propose to leverage the multi-class label information during the episodic training. It will encourage the network to generate more semantically meaningful features for each category. After integrating the target class cues into the query features, we then propose a pyramid feature fusion module to mine the fused features for the final classifier. Furthermore, to take more advantage of the support image-mask pair, we propose a self-prototype guidance branch to support image segmentation. It can constrain the network for generating more compact features and a robust prototype for each semantic class. For inference, we propose a fused prototype guidance branch for the segmentation of the query image. Specifically, we leverage the prediction of the query image to extract the pseudo-prototype and combine it with the initial prototype. Then we utilize the fused prototype to guide the final segmentation of the query image. Extensive experiments demonstrate the superiority of our proposed approach.
Shapley values for feature selection: The good, the bad, and the axioms
Venue: arXiv cs.LG 2021
Year: 2021

The Shapley value has become popular in the Explainable AI (XAI) literature, thanks, to a large extent, to a solid theoretical foundation, including four "favourable and fair" axioms for attribution in transferable utility games. The Shapley value is provably the only solution concept satisfying these axioms. In this paper, we introduce the Shapley value and draw attention to its recent uses as a feature selection tool. We call into question this use of the Shapley value, using simple, abstract "toy" counterexamples to illustrate that the axioms may work against the goals of feature selection. From this, we develop a number of insights that are then investigated in concrete simulation settings, with a variety of Shapley value formulations, including SHapley Additive exPlanations (SHAP) and Shapley Additive Global importancE (SAGE).
Bilingual Language Modeling, A transfer learning technique for Roman Urdu
Venue: arXiv cs.CL 2021
Year: 2021

Pretrained language models are now of widespread use in Natural Language Processing. Despite their success, applying them to Low Resource languages is still a huge challenge. Although Multilingual models hold great promise, applying them to specific low-resource languages e.g. Roman Urdu can be excessive. In this paper, we show how the code-switching property of languages may be used to perform cross-lingual transfer learning from a corresponding high resource language. We also show how this transfer learning technique termed Bilingual Language Modeling can be used to produce better performing models for Roman Urdu. To enable training and experimentation, we also present a collection of novel corpora for Roman Urdu extracted from various sources and social networking sites, e.g. Twitter. We train Monolingual, Multilingual, and Bilingual models of Roman Urdu - the proposed bilingual model achieves 23% accuracy compared to the 2% and 11% of the monolingual and multilingual models respectively in the Masked Language Modeling (MLM) task.
Word frequency-rank relationship in tagged texts
Venue: arXiv cs.CL 2021
Year: 2021

We analyze the frequency-rank relationship in sub-vocabularies corresponding to three different grammatical classes (nouns, verbs, and others) in a collection of literary works in English, whose words have been automatically tagged according to their grammatical role. Comparing with a null hypothesis which assumes that words belonging to each class are uniformly distributed across the frequency-ranked vocabulary of the whole work, we disclose statistically significant differences between the three classes. This results point to the fact that frequency-rank relationships may reflect linguistic features associated with grammatical function.
A Qualitative Theory of Cognitive Attitudes and their Change
Venue: arXiv cs.AI 2021
Year: 2021

We present a general logical framework for reasoning about agents' cognitive attitudes of both epistemic type and motivational type. We show that it allows us to express a variety of relevant concepts for qualitative decision theory including the concepts of knowledge, belief, strong belief, conditional belief, desire, conditional desire, strong desire and preference. We also present two extensions of the logic, one by the notion of choice and the other by dynamic operators for belief change and desire change, and we apply the former to the analysis of single-stage games under incomplete information. We provide sound and complete axiomatizations for the basic logic and for its two extensions. The paper is under consideration in Theory and Practice of Logic Programming (TPLP).
JST-RR Model: Joint Modeling of Ratings and Reviews in Sentiment-Topic Prediction
Venue: arXiv cs.CL 2021
Year: 2021

Analysis of online reviews has attracted great attention with broad applications. Often times, the textual reviews are coupled with the numerical ratings in the data. In this work, we propose a probabilistic model to accommodate both textual reviews and overall ratings with consideration of their intrinsic connection for a joint sentiment-topic prediction. The key of the proposed method is to develop a unified generative model where the topic modeling is constructed based on review texts and the sentiment prediction is obtained by combining review texts and overall ratings. The inference of model parameters are obtained by an efficient Gibbs sampling procedure. The proposed method can enhance the prediction accuracy of review data and achieve an effective detection of interpretable topics and sentiments. The merits of the proposed method are elaborated by the case study from Amazon datasets and simulation studies.
Belief-Propagation Decoding of LDPC Codes with Variable Node-Centric Dynamic Schedules
Venue: arXiv cs.IT 2021
Year: 2021

Belief propagation (BP) decoding of low-density parity-check (LDPC) codes with various dynamic decoding schedules have been proposed to improve the efficiency of the conventional flooding schedule. As the ultimate goal of an ideal LDPC code decoder is to have correct bit decisions, a dynamic decoding schedule should be variable node (VN)-centric and be able to find the VNs with probable incorrect decisions and having a good chance to be corrected if chosen for update. We propose a novel and effective metric called conditional innovation (CI) which serves this design goal well. To make the most of dynamic scheduling which produces high-reliability bit decisions, we limit our search for the candidate VNs to those related to the latest updated nodes only. Based on the CI metric and the new search guideline separately or in combination, we develop several highly efficient decoding schedules. To reduce decoding latency, we introduce multi-edge updating versions which offer extra latency-performance tradeoffs. Numerical results show that both single-edge and multi-edge algorithms provide better decoding performance against most dynamic schedules and the CI-based algorithms are particularly impressive at the first few decoding iterations.
Cut Locus Realizations on Convex Polyhedra
Venue: arXiv cs.CG 2021
Year: 2021

We prove that every positively-weighted tree T can be realized as the cut locus C(x) of a point x on a convex polyhedron P, with T weights matching C(x) lengths. If T has n leaves, P has (in general) n+1 vertices. We show there are in fact a continuum of polyhedra P each realizing T for some x on P. Three main tools in the proof are properties of the star unfolding of P, Alexandrov's gluing theorem, and a cut-locus partition lemma. The construction of P from T is surprisingly simple.
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case
Venue: arXiv cs.LG 2021
Year: 2021

Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools. To address this problem, we generalize the notion of probing tasks to the visual-semantic case. To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks. Our experiments reveal an up to 12% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other.
Direct Estimation of Appearance Models for Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Image segmentation algorithms often depend on appearance models that characterize the distribution of pixel values in different image regions. We describe a new approach for estimating appearance models directly from an image, without explicit consideration of the pixels that make up each region. Our approach is based on novel algebraic expressions that relate local image statistics to the appearance of spatially coherent regions. We describe two algorithms that can use the aforementioned algebraic expressions to estimate appearance models directly from an image. The first algorithm solves a system of linear and quadratic equations using a least squares formulation. The second algorithm is a spectral method based on an eigenvector computation. We present experimental results that demonstrate the proposed methods work well in practice and lead to effective image segmentation algorithms.
Generator Surgery for Compressed Sensing
Venue: arXiv cs.CV 2021
Year: 2021

Image recovery from compressive measurements requires a signal prior for the images being reconstructed. Recent work has explored the use of deep generative models with low latent dimension as signal priors for such problems. However, their recovery performance is limited by high representation error. We introduce a method for achieving low representation error using generators as signal priors. Using a pre-trained generator, we remove one or more initial blocks at test time and optimize over the new, higher-dimensional latent space to recover a target image. Experiments demonstrate significantly improved reconstruction quality for a variety of network architectures. This approach also works well for out-of-training-distribution images and is competitive with other state-of-the-art methods. Our experiments show that test-time architectural modifications can greatly improve the recovery quality of generator signal priors for compressed sensing.
The Soccer Game, bit by bit: An information-theoretic analysis
Venue: arXiv cs.SI 2021
Year: 2021

We modeled the dynamics of a soccer match based on a network representation where players are nodes discretely clustered into homogeneous groups. Players were grouped by physical proximity, supported by the intuitive notion that competing and same-team players use relative position as a key tactical tool to contribute to the team's objectives. The model was applied to a set of matches from a major European national football league, with players' coordinates sampled at 10Hz, resulting in approx. 60,000 network samples per match. We took an information theoretic approach to measuring distance between samples and used it as a proxy for the game dynamics. Significant correlations were found between measurements and key match events that are empirically known to result in players jostling for position, such as when striving to get unmarked or to mark. These events increase the information distance, while breaks in game play have the opposite effect. By analyzing the frequency spectrum of players' cluster transitions and their corresponding information distance, it is possible to build a comprehensive view of player's interactions, useful for training and strategy development. This analysis can be drilled down to the level of individual players by quantifying their contribution to cluster breakup and emergence, building an overall multi-level map that provides insights into the game dynamics, from the individual player, to the clusters of interacting players, all the way to the teams and their matches.