OpenRaiser/Intern-Atlas huggingface dataset:
Bridging Machine Learning and Mechanism Design towards Algorithmic Fairness
Venue: arXiv cs.GT 2021
Year: 2021

Decision-making systems increasingly orchestrate our world: how to intervene on the algorithmic components to build fair and equitable systems is therefore a question of utmost importance; one that is substantially complicated by the context-dependent nature of fairness and discrimination. Modern decision-making systems that involve allocating resources or information to people (e.g., school choice, advertising) incorporate machine-learned predictions in their pipelines, raising concerns about potential strategic behavior or constrained allocation, concerns usually tackled in the context of mechanism design. Although both machine learning and mechanism design have developed frameworks for addressing issues of fairness and equity, in some complex decision-making systems, neither framework is individually sufficient. In this paper, we develop the position that building fair decision-making systems requires overcoming these limitations which, we argue, are inherent to each field. Our ultimate objective is to build an encompassing framework that cohesively bridges the individual frameworks of mechanism design and machine learning. We begin to lay the ground work towards this goal by comparing the perspective each discipline takes on fair decision-making, teasing out the lessons each field has taught and can teach the other, and highlighting application domains that require a strong collaboration between these disciplines.
Scalable Consensus Protocols for PoW based Blockchain and blockDAG
Venue: arXiv cs.DC 2021
Year: 2021

In this paper, we propose two models for scaling the transaction throughput in Proof-of-Work (PoW) based blockchain networks. In the first approach, a mathematical model has derived for optimal transaction throughput for PoW based longest chain rule blockchain. In this approach, the blockchain Peer-to-Peer (P2P) network is considered as Erdös-Rényi random network topology. This approach is however limited by the block creation rate, the results suggest that the rate beyond an optimal point can result in unfairness in the system. The second approach is a new consensus protocol proposed by considering the ledger as a Directed Acyclic Graph (DAG) called blockDAG instead of a chain of blocks. In this framework, we follow a two-step strategy that makes the system robust enough to handle the double-spend attacks. The first step involves the development of an unsupervised learning graph clustering algorithm for separating the blocks created by an attacker. In the second step, the attackers blocks are eliminated and the remaining blocks are arranged in topological order by honest clients which makes the blockDAG system suitable for smart contract applications found in Internet of Things (IoT) services. The Simulation results demonstrate a significant improvement in the transaction throughput compared to bitcoin.
Evacuation from Various Types of Finite 2D Square Grid Fields by a Metamorphic Robotic System
Venue: arXiv cs.DC 2021
Year: 2021

A metamorphic robotic system (MRS) is composed of anonymous, memoryless, and autonomous modules that execute an identical distributed algorithm to move while keeping the connectivity of the modules. For an MRS, the number of modules required to solve a given task is an important complexity measure. Here, we consider evacuation from a finite two-dimensional square grid field by an MRS. This study aims to establish the minimum number of modules required to solve the evacuation problem under several conditions. We consider a rectangular field surrounded by walls with at least one exit. Our results show that two modules are necessary and sufficient for evacuation from any rectangular field if equipped with a global compass, which provides the modules with a common sense of direction. After that, we focus on the case of modules without a global compass and show that four (resp. seven) modules are necessary and sufficient for restricted (resp. any) initial shapes of an MRS. We also show that two modules are sufficient when an MRS is touching a wall in an initial configuration. Then, we clarify the condition to stop an MRS after evacuation of a rectangular field. Finally, we extend these results to mazes and convex fields.
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide understandings on this generalization gap by analyzing their local convergence behaviors. Specifically, we observe the heavy tails of gradient noise in these algorithms. This motivates us to analyze these algorithms through their Levy-driven stochastic differential equations (SDEs) because of the similar convergence behaviors of an algorithm and its SDE. Then we establish the escaping time of these SDEs from a local basin. The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD. So SGD is more locally unstable than ADAM~at sharp minima defined as the minima whose local basins have small Radon measure, and can better escape from them to flatter ones with larger Radon measure. As flat minima here which often refer to the minima at flat or asymmetric basins/valleys often generalize better than sharp ones , our result explains the better generalization performance of SGD over ADAM. Finally, experimental results confirm our heavy-tailed gradient noise assumption and theoretical affirmation.
Pedestrian Trajectory Prediction with Convolutional Neural Networks
Venue: arXiv cs.CV 2021
Year: 2021

Predicting the future trajectories of pedestrians is a challenging problem that has a range of application, from crowd surveillance to autonomous driving. In literature, methods to approach pedestrian trajectory prediction have evolved, transitioning from physics-based models to data-driven models based on recurrent neural networks. In this work, we propose a new approach to pedestrian trajectory prediction, with the introduction of a novel 2D convolutional model. This new model outperforms recurrent models, and it achieves state-of-the-art results on the ETH and TrajNet datasets. We also present an effective system to represent pedestrian positions and powerful data augmentation techniques, such as the addition of Gaussian noise and the use of random rotations, which can be applied to any model. As an additional exploratory analysis, we present experimental results on the inclusion of occupancy methods to model social information, which empirically show that these methods are ineffective in capturing social interaction.
A Complete Approach to Loop Verification with Invariants and Summaries
Venue: arXiv cs.LO 2021
Year: 2021

Invariants are the predominant approach to verify the correctness of loops. As an alternative, loop contracts, which make explicit the premise and conclusion of the underlying induction proof, can sometimes capture correctness conditions more naturally. But despite this advantage, the second approach receives little attention overall, and the goal of this paper is to lift it out of its niche. We give the first comprehensive exposition of the theory of loop contracts, including a characterization of its completeness. We show concrete examples on standard algorithms that showcase their relative merits. Moreover, we demonstrate a novel constructive translation between the two approaches, which decouples the chosen specification approach from the verification backend.
Counting Subgraphs in Degenerate Graphs
Venue: arXiv cs.CC 2021
Year: 2021

We consider the problem of counting the number of copies of a fixed graph $H$ within an input graph $G$. This is one of the most well-studied algorithmic graph problems, with many theoretical and practical applications. We focus on solving this problem when the input $G$ has bounded degeneracy. This is a rich family of graphs, containing all graphs without a fixed minor (e.g. planar graphs), as well as graphs generated by various random processes (e.g. preferential attachment graphs). We say that $H$ is easy if there is a linear-time algorithm for counting the number of copies of $H$ in an input $G$ of bounded degeneracy. A seminal result of Chiba and Nishizeki from '85 states that every $H$ on at most 4 vertices is easy. Bera, Pashanasangi, and Seshadhri recently extended this to all $H$ on 5 vertices, and further proved that for every $k > 5$ there is a $k$-vertex $H$ which is not easy. They left open the natural problem of characterizing all easy graphs $H$. Bressan has recently introduced a framework for counting subgraphs in degenerate graphs, from which one can extract a sufficient condition for a graph $H$ to be easy. Here we show that this sufficient condition is also necessary, thus fully answering the Bera--Pashanasangi--Seshadhri problem. We further resolve two closely related problems; namely characterizing the graphs that are easy with respect to counting induced copies, and with respect to counting homomorphisms.
Measuring and Reducing Gendered Correlations in Pre-trained Models
Venue: arXiv cs.CL 2021
Year: 2021

Pre-trained models have revolutionized natural language understanding. However, researchers have found they can encode artifacts undesired in many applications, such as professions correlating with one gender more than another. We explore such gendered correlations as a case study for how to address unintended correlations in pre-trained models. We define metrics and reveal that it is possible for models with similar accuracy to encode correlations at very different rates. We show how measured correlations can be reduced with general-purpose techniques, and highlight the trade offs different strategies have. With these results, we make recommendations for training robust models: (1) carefully evaluate unintended correlations, (2) be mindful of seemingly innocuous configuration differences, and (3) focus on general mitigations.
Higher-Order Recursion Schemes and Collapsible Pushdown Automata: Logical Properties
Venue: arXiv cs.LO 2021
Year: 2021

This paper studies the logical properties of a very general class of infinite ranked trees, namely those generated by higher-order recursion schemes. We consider, for both monadic second-order logic and modal mu-calculus, three main problems: model-checking, logical reflection (aka global model-checking, that asks for a finite description of the set of elements for which a formula holds) and selection (that asks, if exists, for some finite description of a set of elements for which an MSO formula with a second-order free variable holds). For each of these problems we provide an effective solution. This is obtained thanks to a known connection between higher-order recursion schemes and collapsible pushdown automata and on previous work regarding parity games played on transition graphs of collapsible pushdown automata.
A review of 3D human pose estimation algorithms for markerless motion capture
Venue: arXiv cs.CV 2021
Year: 2021

Human pose estimation is a very active research field, stimulated by its important applications in robotics, entertainment or health and sports sciences, among others. Advances in convolutional networks triggered noticeable improvements in 2D pose estimation, leading modern 3D markerless motion capture techniques to an average error per joint of 20 mm. However, with the proliferation of methods, it is becoming increasingly difficult to make an informed choice. Here, we review the leading human pose estimation methods of the past five years, focusing on metrics, benchmarks and method structures. We propose a taxonomy based on accuracy, speed and robustness that we use to classify de methods and derive directions for future research.
Cross-Domain Few-Shot Learning by Representation Fusion
Venue: arXiv cs.LG 2021
Year: 2021

In order to quickly adapt to new data, few-shot learning aims at learning from few examples, often by using already acquired knowledge. The new data often differs from the previously seen data due to a domain shift, that is, a change of the input-target distribution. While several methods perform well on small domain shifts like new target classes with similar inputs, larger domain shifts are still challenging. Large domain shifts may result in high-level concepts that are not shared between the original and the new domain, whereas low-level concepts like edges in images might still be shared and useful. For cross-domain few-shot learning, we suggest representation fusion to unify different abstraction levels of a deep neural network into one representation. We propose Cross-domain Hebbian Ensemble Few-shot learning (CHEF), which achieves representation fusion by an ensemble of Hebbian learners acting on different layers of a deep neural network. Ablation studies show that representation fusion is a decisive factor to boost cross-domain few-shot learning. On the few-shot datasets miniImagenet and tieredImagenet with small domain shifts, CHEF is competitive with state-of-the-art methods. On cross-domain few-shot benchmark challenges with larger domain shifts, CHEF establishes novel state-of-the-art results in all categories. We further apply CHEF on a real-world cross-domain application in drug discovery. We consider a domain shift from bioactive molecules to environmental chemicals and drugs with twelve associated toxicity prediction tasks. On these tasks, that are highly relevant for computational drug discovery, CHEF significantly outperforms all its competitors. Github: https://github.com/ml-jku/chef
Applying Graph-based Deep Learning To Realistic Network Scenarios
Venue: arXiv cs.NI 2021
Year: 2021

Recent advances in Machine Learning (ML) have shown a great potential to build data-driven solutions for a plethora of network-related problems. In this context, building fast and accurate network models is essential to achieve functional optimization tools for networking. However, state-of-the-art ML-based techniques for network modelling are not able to provide accurate estimates of important performance metrics such as delay or jitter in realistic network scenarios with sophisticated queue scheduling configurations. This paper presents a new Graph-based deep learning model able to estimate accurately the per-path mean delay in networks. The proposed model can generalize successfully over topologies, routing configurations, queue scheduling policies and traffic matrices unseen during the training phase.
Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast-Choose Three
Venue: arXiv cs.LG 2021
Year: 2021

Modern neural networks do not always produce well-calibrated predictions, even when trained with a proper scoring function such as cross-entropy. In classification settings, simple methods such as isotonic regression or temperature scaling may be used in conjunction with a held-out dataset to calibrate model outputs. However, extending these methods to structured prediction is not always straightforward or effective; furthermore, a held-out calibration set may not always be available. In this paper, we study ensemble distillation as a general framework for producing well-calibrated structured prediction models while avoiding the prohibitive inference-time cost of ensembles. We validate this framework on two tasks: named-entity recognition and machine translation. We find that, across both tasks, ensemble distillation produces models which retain much of, and occasionally improve upon, the performance and calibration benefits of ensembles, while only requiring a single model during test-time.
Adaptive-Attentive Geolocalization from few queries: a hybrid approach
Venue: arXiv cs.CV 2021
Year: 2021

We address the task of cross-domain visual place recognition, where the goal is to geolocalize a given query image against a labeled gallery, in the case where the query and the gallery belong to different visual domains. To achieve this, we focus on building a domain robust deep network by leveraging over an attention mechanism combined with few-shot unsupervised domain adaptation techniques, where we use a small number of unlabeled target domain images to learn about the target distribution. With our method, we are able to outperform the current state of the art while using two orders of magnitude less target domain images. Finally we propose a new large-scale dataset for cross-domain visual place recognition, called SVOX. The pytorch code is available at https://github.com/valeriopaolicelli/AdAGeo .
A window view quality assessment framework
Venue: arXiv cs.CY 2021
Year: 2021

The views that windows provide from inside a building affect human health and well-being. Although window view is an important element of architecture, there is no established framework to guide its design. The literature is widely dispersed across different disciplinary fields, and there is a need to coalesce this information into a framework that can be applied into the building design. Based on the literature, we present a framework for what constitutes 'view quality.' At the basis of our framework, we propose three primary variables: View Content (the assessment of visual features seen in the window view), View Access (the measure of how much of the view can be seen through the window from the occupant's position), and View Clarity (the assessment of how clear the view content appears in the window view when seen by an occupant). Each variable was thematically derived from different sources including daylighting standards, green certification systems, and scientific research studies. We describe the most important characteristics of each variable, and from our review of the literature, we propose a conceptual index that can evaluate the quality of a window view. While discussing the index, we summarize design recommendations for integrating these three variables into the building process and identify knowledge gaps for future research.
Disentangled Dynamic Graph Deep Generation
Venue: arXiv cs.LG 2021
Year: 2021

Deep generative models for graphs have exhibited promising performance in ever-increasing domains such as design of molecules (i.e, graph of atoms) and structure prediction of proteins (i.e., graph of amino acids). Existing work typically focuses on static rather than dynamic graphs, which are actually very important in the applications such as protein folding, molecule reactions, and human mobility. Extending existing deep generative models from static to dynamic graphs is a challenging task, which requires to handle the factorization of static and dynamic characteristics as well as mutual interactions among node and edge patterns. Here, this paper proposes a novel framework of factorized deep generative models to achieve interpretable dynamic graph generation. Various generative models are proposed to characterize conditional independence among node, edge, static, and dynamic factors. Then, variational optimization strategies as well as dynamic graph decoders are proposed based on newly designed factorized variational autoencoders and recurrent graph deconvolutions. Extensive experiments on multiple datasets demonstrate the effectiveness of the proposed models.
Concentrated Stopping Set Design for Coded Merkle Tree: Improving Security Against Data Availability Attacks in Blockchain Systems
Venue: arXiv cs.IT 2021
Year: 2021

In certain blockchain systems, light nodes are clients that download only a small portion of the block. Light nodes are vulnerable to data availability (DA) attacks where a malicious node hides an invalid portion of the block from the light nodes. Recently, a technique based on erasure codes called Coded Merkle Tree (CMT) was proposed by Yu et al. that enables light nodes to detect a DA attack with high probability. The CMT is constructed using LDPC codes for fast decoding but can fail to detect a DA attack if a malicious node hides a small stopping set of the code. To combat this, Yu et al. used well-studied techniques to design random LDPC codes with high minimum stopping set size. Although effective, these codes are not necessarily optimal for this application. In this paper, we demonstrate a more specialized LDPC code design to improve the security against DA attacks. We achieve this goal by providing a deterministic LDPC code construction that focuses on concentrating stopping sets to a small group of variable nodes rather than only eliminating stopping sets. We design these codes by modifying the Progressive Edge Growth algorithm into a technique called the entropy-constrained PEG (EC-PEG) algorithm. This new method demonstrates a higher probability of detecting DA attacks and allows for good codes at short lengths.
Decoding Methods for Neural Narrative Generation
Venue: arXiv cs.CL 2021
Year: 2021

Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters -- specifically, maximum mutual information -- analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.
Learning, compression, and leakage: Minimising classification error via meta-universal compression principles
Venue: arXiv cs.LG 2021
Year: 2021

Learning and compression are driven by the common aim of identifying and exploiting statistical regularities in data, which opens the door for fertile collaboration between these areas. A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding, which provides strong guarantees for compression of small datasets - in contrast with more popular estimators whose guarantees hold only in the asymptotic limit. Here we consider a NML-based decision strategy for supervised classification problems, and show that it attains heuristic PAC learning when applied to a wide variety of models. Furthermore, we show that the misclassification rate of our method is upper bounded by the maximal leakage, a recently proposed metric to quantify the potential of data leakage in privacy-sensitive scenarios.