OpenRaiser/Intern-Atlas huggingface dataset:
Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads
Venue: arXiv cs.DC 2022
Year: 2022

Recent trend towards increasing large machine learning models require both training and inference tasks to be distributed. Considering the huge cost of training these models, it is imperative to unlock optimizations in computation and communication to obtain best performance. However, current logical separation between computation and communication kernels in deep learning frameworks misses the optimization opportunities across such barrier. Breaking this abstraction with a holistic consideration can provide many optimizations to provide performance improvements in distributed workloads. Manually applying these optimizations needs modifications in underlying computation and communication libraries for each scenario, which is time consuming and error-prone. Therefore, we present CoCoNeT, with a DSL to express a program with both computation and communication. CoCoNeT contains several machine learning aware transformations to optimize a program and a compiler to generate high performance kernels. Providing both computation and communication as first class constructs allows users to work on a high-level abstraction and apply powerful optimizations, such as fusion or overlapping of communication and computation. CoCoNeT enables us to optimize data-, model-and pipeline-parallel workloads in large language models with only a few lines of code. Experiments show CoCoNeT significantly outperforms state-of-the-art distributed machine learning implementations.
Causally motivated Shortcut Removal Using Auxiliary Labels
Venue: arXiv cs.LG 2022
Year: 2022

Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even when no shortcut is present. Our analysis highlights important theoretical properties of training techniques commonly used in the causal inference, fairness, and disentanglement literatures. Our code is available at https://github.com/mymakar/causally_motivated_shortcut_removal
Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication task
Venue: arXiv cs.CL 2022
Year: 2022

There is substantial variability in the expectations that communication partners bring into interactions, creating the potential for misunderstandings. To directly probe these gaps and our ability to overcome them, we propose a communication task based on color-concept associations. In Experiment 1, we establish several key properties of the mental representations of these expectations, or lexical priors, based on recent probabilistic theories. Associations are more variable for abstract concepts, variability is represented as uncertainty within each individual, and uncertainty enables accurate predictions about whether others are likely to share the same association. In Experiment 2, we then examine the downstream consequences of these representations for communication. Accuracy is initially low when communicating about concepts with more variable associations, but rapidly increases as participants form ad hoc conventions. Together, our findings suggest that people cope with variability by maintaining well-calibrated uncertainty about their partner and appropriately adaptable representations of their own.
Understanding occupants' behaviour, engagement, emotion, and comfort indoors with heterogeneous sensors and wearables
Venue: arXiv cs.HC 2022
Year: 2022

We conducted a field study at a K-12 private school in the suburbs of Melbourne, Australia. The data capture contained two elements: First, a 5-month longitudinal field study In-Gauge using two outdoor weather stations, as well as indoor weather stations in 17 classrooms and temperature sensors on the vents of occupant-controlled room air-conditioners; these were collated into individual datasets for each classroom at a 5-minute logging frequency, including additional data on occupant presence. The dataset was used to derive predictive models of how occupants operate room air-conditioning units. Second, we tracked 23 students and 6 teachers in a 4-week cross-sectional study En-Gage, using wearable sensors to log physiological data, as well as daily surveys to query the occupants' thermal comfort, learning engagement, emotions and seating behaviours. Overall, the combined dataset could be used to analyse the relationships between indoor/outdoor climates and students' behaviours/mental states on campus, which provide opportunities for the future design of intelligent feedback systems to benefit both students and staff.
On the Complexity of Nucleolus Computation for Bipartite b-Matching Games
Venue: arXiv cs.GT 2022
Year: 2022

We explore the complexity of nucleolus computation in b-matching games on bipartite graphs. We show that computing the nucleolus of a simple b-matching game is NP-hard even on bipartite graphs of maximum degree 7. We complement this with partial positive results in the special case where b values are bounded by 2. In particular, we describe an efficient algorithm when a constant number of vertices satisfy b(v) = 2 as well as an efficient algorithm for computing the non-simple b-matching nucleolus when b = 2.
Face mask perception during the COVID-19 pandemic: an observational study of Russian online social network VKontakte
Venue: arXiv cs.SI 2022
Year: 2022

This cross-sectional study characterizes users' attitudes towards the face mask requirements introduced by the Russian government as a response to the COVID-19 pandemic. We study how they relate to other users' characteristics such as age, gender, and political attitudes. Our results indicate that men and elder individuals - demographic groups that are most vulnerable to COVID-19 -- underestimate the benefits of wearing face masks. We also discovered that users in opposition to the Russian government highly approve of this anti-COVID-19 measure -- an oppositionist will approve of the face mask requirements with the probability of 0.95. For those who support the Russian government, the odds of approval are merely 0.45.
On Complementing Unambiguous Automata and Graphs With Many Cliques and Cocliques
Venue: arXiv cs.FL 2022
Year: 2022

We show that for any unambiguous finite automaton with $n$ states there exists an unambiguous finite automaton with $\sqrt{n+1} \cdot 2^{n/2}$ states that recognizes the complement language. This builds and improves upon a similar result by Jirásek et al. [Int. J. Found. Comput. Sci. 29 (5) (2018)]. Our improvement is based on a reduction to and an analysis of a problem from extremal graph theory: we show that for any graph with $n$ vertices, the product of the number of its cliques with the number of its cocliques (independent sets) is bounded by $(n+1) 2^n$.
Contemporary Research Trends in Response Robotics
Venue: arXiv cs.CY 2022
Year: 2022

The multidisciplinary nature of response robotics has brought about a diversified research community with extended expertise. Motivated by the recent accelerated rate of publications in the field, this paper analyzes the technical content, statistics, and implications of the literature from bibliometric standpoints. The aim is to study the global progress of response robotics research and identify the contemporary trends. To that end, we investigated the collaboration mapping together with the citation network to formally recognize impactful and contributing authors, publications, sources, institutions, funding agencies, and countries. We found how natural and human-made disasters contributed to forming productive regional research communities, while there are communities that only view response robotics as an application of their research. Furthermore, through an extensive discussion on the bibliometric results, we elucidated the philosophy behind research priority shifts in response robotics and presented our deliberations on future research directions.
Discovery and Contextual Data Cleaning with Ontology Functional Dependencies
Venue: arXiv cs.DB 2022
Year: 2022

Functional Dependencies (FDs) define attribute relationships based on syntactic equality, and, when usedin data cleaning, they erroneously label syntactically different but semantically equivalent values as errors. We explore dependency-based data cleaning with Ontology Functional Dependencies(OFDs), which express semantic attribute relationships such as synonyms and is-a hierarchies defined by an ontology. We study the theoretical foundations for OFDs, including sound and complete axioms and a linear-time inference procedure. We then propose an algorithm for discovering OFDs (exact ones and ones that hold with some exceptions) from data that uses the axioms to prune the search space. Towards enabling OFDs as data quality rules in practice, we study the problem of finding minimal repairs to a relation and ontology with respect to a set of OFDs. We demonstrate the effectiveness of our techniques on real datasets, and show that OFDs can significantly reduce the number of false positive errors in data cleaning techniques that rely on traditional FDs.
BookSum: A Collection of Datasets for Long-form Narrative Summarization
Venue: arXiv cs.CL 2022
Year: 2022

The majority of available text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future generations of text summarization systems. We address these issues by introducing BookSum, a collection of datasets for long-form narrative summarization. Our dataset covers source documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of our dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. To facilitate future work, we trained and evaluated multiple extractive and abstractive summarization models as baselines for our dataset.
Sparta: Spatially Attentive and Adversarially Robust Activation
Venue: arXiv cs.LG 2022
Year: 2022

Adversarial training (AT) is one of the most effective ways for improving the robustness of deep convolution neural networks (CNNs). Just like common network training, the effectiveness of AT relies on the design of basic network components. In this paper, we conduct an in-depth study on the role of the basic ReLU activation component in AT for robust CNNs. We find that the spatially-shared and input-independent properties of ReLU activation make CNNs less robust to white-box adversarial attacks with either standard or adversarial training. To address this problem, we extend ReLU to a novel Sparta activation function (Spatially attentive and Adversarially Robust Activation), which enables CNNs to achieve both higher robustness, i.e., lower error rate on adversarial examples, and higher accuracy, i.e., lower error rate on clean examples, than the existing state-of-the-art (SOTA) activation functions. We further study the relationship between Sparta and the SOTA activation functions, providing more insights about the advantages of our method. With comprehensive experiments, we also find that the proposed method exhibits superior cross-CNN and cross-dataset transferability. For the former, the adversarially trained Sparta function for one CNN (e.g., ResNet-18) can be fixed and directly used to train another adversarially robust CNN (e.g., ResNet-34). For the latter, the Sparta function trained on one dataset (e.g., CIFAR-10) can be employed to train adversarially robust CNNs on another dataset (e.g., SVHN). In both cases, Sparta leads to CNNs with higher robustness than the vanilla ReLU, verifying the flexibility and versatility of the proposed method.
Independent Asymmetric Embedding for Information Diffusion Prediction on Social Networks
Venue: arXiv cs.LG 2022
Year: 2022

The prediction for information diffusion on social networks has great practical significance in marketing and public opinion control. It aims to predict the individuals who will potentially repost the message on the social network. One type of method is based on demographics, complex networks and other prior knowledge to establish an interpretable model to simulate and predict the propagation process, while the other type of method is completely data-driven and maps the nodes to a latent space for propagation prediction. Existing latent space design and embedding methods lack consideration for the intervene among users. In this paper, we propose an independent asymmetric embedding method to embed each individual into one latent influence space and multiple latent susceptibility spaces. Based on the similarity between information diffusion and heat diffusion phenomenon, the heat diffusion kernel is exploited in our model and establishes the embedding rules. Furthermore, our method captures the co-occurrence regulation of user combinations in cascades to improve the calculating effectiveness. The results of extensive experiments conducted on real-world datasets verify both the predictive accuracy and cost-effectiveness of our approach.
On Oriented Diameter of $(n, k)$-Star Graphs
Venue: arXiv cs.DM 2022
Year: 2022

Assignment of one of the two possible directions to every edge of an undirected graph $G=(V,E)$ is called an orientation of $G$. The resulting directed graph is denoted by $\overrightarrow{G}$. A strong orientation is one in which every vertex is reachable from every other vertex via a directed path. The diameter of $\overrightarrow{G}$, i.e., the maximum distance from one vertex to another, depends on the particular orientation. The minimum diameter among all possible orientations is called the oriented diameter $\overrightarrow{\text{diam}}(G)$ of $G$. Let $n,k$ be two integers with $1 \leq k < n$. In the realm of interconnection networks of processing elements, an $(n,k)$-star graph $S_{n,k}$ offers a topology that circumvents the lack of scalability of $n$-star graphs $S_n$. In this paper, we present a strong orientation for $S_{n,k}$ that combines approaches suggested by Cheng and Lipman [Journal of Interconnection Networks (2002)] for $S_{n,k}$ with the one proposed by Fujita [The First International Symposium on Computing and Networking (CANDAR 2013)] for $S_n$. Next, we propose a distributed routing algorithm for $\overrightarrow{S_{n,k}}$ inspired by an algorithm proposed by Kumar, Rajendraprasad and Sudeep [Discrete Applied Mathematics (2021)] for $\overrightarrow{S_n}$. With the aid of both the orientation scheme and the routing algorithm, we show that $\overrightarrow{\text{diam}}(S_{n,k}) \leq \lfloor \frac{n+k}{2} \rfloor + 2k + 6 - δ(n,k)$ where $δ(n,k)$ is a non-negative function. The function $δ(n,k)$ takes on values $2k-n$, $0$, and $\left\lfloor \frac{n-3k}{2} \right\rfloor$ respectively for three disjoint intervals $k>\frac{n}{2}$, $\frac{n}{3} < k \leq \frac{n}{2}$ and $k\leq \frac{n}{3}$. For every value of $n$, $k$, our upper bound performs better than all known bounds in literature.
Improved Ackermannian lower bound for the Petri nets reachability problem
Venue: arXiv cs.FL 2022
Year: 2022

Petri nets, equivalently presentable as vector addition systems with states, are an established model of concurrency with widespread applications. The reachability problem, where we ask whether from a given initial configuration there exists a sequence of valid execution steps reaching a given final configuration, is the central algorithmic problem for this model. The complexity of the problem has remained, until recently, one of the hardest open questions in verification of concurrent systems. A first upper bound has been provided only in 2015 by Leroux and Schmitz, then refined by the same authors to non-primitive recursive Ackermannian upper bound in 2019. The exponential space lower bound, shown by Lipton already in 1976, remained the only known for over 40 years until a breakthrough non-elementary lower bound by Czerwi{ń}ski, Lasota, Lazic, Leroux and Mazowiecki in 2019. Finally, a matching Ackermannian lower bound announced this year by Czerwi{ń}ski and Orlikowski, and independently by Leroux, established the complexity of the problem. Our primary contribution is an improvement of the former construction, making it conceptually simpler and more direct. On the way we improve the lower bound for vector addition systems with states in fixed dimension (or, equivalently, Petri nets with fixed number of places): while Czerwi{ń}ski and Orlikowski prove $F_k$-hardness (hardness for $k$th level in Grzegorczyk Hierarchy) in dimension $6k$, our simplified construction yields $F_k$-hardness already in dimension $3k+2$.
Query Interpretations from Entity-Linked Segmentations
Venue: arXiv cs.IR 2022
Year: 2022

Web search queries can be ambiguous: is "source of the nile" meant to find information on the actual river or on a board game of that name? We tackle this problem by deriving entity-based query interpretations: given some query, the task is to derive all reasonable ways of linking suitable parts of the query to semantically compatible entities in a background knowledge base. Our suggested approach focuses on effectiveness but also on efficiency since web search response times should not exceed some hundreds of milliseconds. In our approach, we use query segmentation as a pre-processing step that finds promising segment-based "interpretation skeletons". The individual segments from these skeletons are then linked to entities from a knowledge base and the reasonable combinations are ranked in a final step. An experimental comparison on a combined corpus of all existing query entity linking datasets shows our approach to have a better interpretation accuracy at a better run time than the previously most effective methods.
An Analysis of Probabilistic Forwarding of Coded Packets on Random Geometric Graphs
Venue: arXiv cs.IT 2022
Year: 2022

We consider the problem of energy-efficient broadcasting on dense ad-hoc networks. Ad-hoc networks are generally modeled using random geometric graphs (RGGs). Here, nodes are deployed uniformly in a square area around the origin, and any two nodes which are within Euclidean distance of $1$ are assumed to be able to receive each other's broadcast. A source node at the origin encodes $k$ data packets of information into $n\ (>k)$ coded packets and transmits them to all its one-hop neighbors. The encoding is such that, any node that receives at least $k$ out of the $n$ coded packets can retrieve the original $k$ data packets. Every other node in the network follows a probabilistic forwarding protocol; upon reception of a previously unreceived packet, the node forwards it with probability $p$ and does nothing with probability $1-p$. We are interested in the minimum forwarding probability which ensures that a large fraction of nodes can decode the information from the source. We deem this a \emph{near-broadcast}. The performance metric of interest is the expected total number of transmissions at this minimum forwarding probability, where the expectation is over both the forwarding protocol as well as the realization of the RGG. In comparison to probabilistic forwarding with no coding, our treatment of the problem indicates that, with a judicious choice of $n$, it is possible to reduce the expected total number of transmissions while ensuring a near-broadcast.
Sparse Spiking Gradient Descent
Venue: arXiv cs.NE 2022
Year: 2022

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 150x, and 85% more memory efficient, without losing accuracy.
Automating Cryptographic Protocol Language Generation from Structured Specifications
Venue: arXiv cs.CR 2022
Year: 2022

Security of cryptographic protocols can be analysed by creating a model in a formal language and verifying the model in a tool. All such tools focus on the last part of the analysis, verification, and the interpretation of the specification is only explained in papers. Rather, we focus on the interpretation and modelling part by presenting a tool to aid the cryptographer throughout the process and automatically generating code in a target language. We adopt a data-centric approach where the protocol design is stored in a structured way rather than as textual specifications. Previous work shows how this approach facilitates the interpretation to a single language (for Tamarin) which required aftermath modifications. By improving the expressiveness of the specification data structure we extend the tool to export to an additional formal language, ProVerif, as well as a C++ fully running implementation. Furthermore, we extend the plugins to verify correctness in ProVerif and executability lemmas in Tamarin. In this paper we model the Diffie-Hellman key exchange, which is traditionally used as a case study; a demo is also provided for other commonly studied protocols, Needham- Schroeder and Needham-Schroeder-Lowe.