OpenRaiser/Intern-Atlas huggingface dataset:
One-shot learning for the long term: consolidation with an artificial hippocampal algorithm
Venue: arXiv cs.LG 2021
Year: 2021

Standard few-shot experiments involve learning to efficiently match previously unseen samples by class. We claim that few-shot learning should be long term, assimilating knowledge for the future, without forgetting previous concepts. In the mammalian brain, the hippocampus is understood to play a significant role in this process, by learning rapidly and consolidating knowledge to the neocortex incrementally over a short period. In this research we tested whether an artificial hippocampal algorithm (AHA), could be used with a conventional Machine Learning (ML) model that learns incrementally analogous to the neocortex, to achieve one-shot learning both short and long term. The results demonstrated that with the addition of AHA, the system could learn in one-shot and consolidate the knowledge for the long term without catastrophic forgetting. This study is one of the first examples of using a CLS model of hippocampus to consolidate memories, and it constitutes a step toward few-shot continual learning.
An Overview of Agent-based Traffic Simulators
Venue: arXiv cs.MA 2021
Year: 2021

Individual traffic significantly contributes to climate change and environmental degradation. Therefore, innovation in sustainable mobility is gaining importance as it helps to reduce environmental pollution. However, effects of new ideas in mobility are difficult to estimate in advance and strongly depend on the individual traffic participants. The application of agent technology is particularly promising as it focuses on modelling heterogeneous individual preferences and behaviours. In this paper, we show how agent-based models are particularly suitable to address three pressing research topics in mobility: 1. Social dilemmas in resource utilisation; 2. Digital connectivity; and 3. New forms of mobility. We then explain how the features of several agent-based simulators are suitable for addressing these topics. We assess the capability of simulators to model individual travel behaviour, discussing implemented features and identifying gaps in functionality that we consider important.
Sequence Types and Infinitary Semantics
Venue: arXiv cs.LO 2021
Year: 2021

We introduce a new representation of non-idempotent intersection types, using \textbf{sequences} (families indexed with natural numbers) instead of lists or multisets. This allows scaling up \textbf{intersection type} theory to the infinitary $λ$-calculus. We thus characterize hereditary head normalization, which gives a positive answer to a question known as \textbf{Klop's Problem}. On our way, we use \textbf{non-idempotent intersection} to retrieve some well-known results on infinitary terms.
Uncovering Interpretable Internal States of Merging Tasks at Highway On-Ramps for Autonomous Driving Decision-Making
Venue: arXiv cs.RO 2021
Year: 2021

Humans make daily routine decisions based on their internal states in intricate interaction scenarios. This paper presents a probabilistically reconstructive learning approach to identify the internal states of multi-vehicle sequential interactions when merging at highway on-ramps. We treated the merging task's sequential decision as a dynamic, stochastic process and then integrated the internal states into an HMM-GMR model, a probabilistic combination of an extended Gaussian mixture regression (GMR) and hidden Markov models (HMM). We also developed a variant expectation-maximum (EM) algorithm to estimate the model parameters and verified it based on a real-world data set. Experiment results reveal that three interpretable internal states can semantically describe the interactive merge procedure at highway on-ramps. This finding provides a basis to develop an efficient model-based decision-making algorithm for autonomous vehicles (AVs) in a partially observable environment.
A Reference Model for IoT Embodied Agents Controlled by Neural Networks
Venue: arXiv cs.AI 2021
Year: 2021

Embodied agents is a term used to denote intelligent agents, which are a component of devices belonging to the Internet of Things (IoT) domain. Each agent is provided with sensors and actuators to interact with the environment, and with a 'controller' that usually contains an artificial neural network (ANN). In previous publications, we introduced three software approaches to design, implement and test IoT embodied agents. In this paper, we propose a reference model based on statecharts that offers abstractions tailored to the development of IoT applications. The model represents embodied agents that are controlled by neural networks. Our model includes the ANN training process, represented as a reconfiguration step such as changing agent features or neural net connections. Our contributions include the identification of the main characteristics of IoT embodied agents, a reference model specification based on statecharts, and an illustrative application of the model to support autonomous street lights. The proposal aims to support the design and implementation of IoT applications by providing high-level design abstractions and models, thus enabling the designer to have a uniform approach to conceiving, designing and explaining such applications.
Formalized Haar Measure
Venue: arXiv cs.LO 2021
Year: 2021

We describe the formalization of the existence and uniqueness of Haar measure in the Lean theorem prover. The Haar measure is an invariant regular measure on locally compact groups, and it has not been formalized in a proof assistant before. We will also discuss the measure theory library in Lean's mathematical library \textsf{mathlib}, and discuss the construction of product measures and the proof of Fubini's theorem for the Bochner integral.
Consistency-based Merging of Variability Models
Venue: arXiv cs.AI 2021
Year: 2021

Globally operating enterprises selling large and complex products and services often have to deal with situations where variability models are locally developed to take into account the requirements of local markets. For example, cars sold on the U.S. market are represented by variability models in some or many aspects different from European ones. In order to support global variability management processes, variability models and the underlying knowledge bases often need to be integrated. This is a challenging task since an integrated knowledge base should not produce results which are different from those produced by the individual knowledge bases. In this paper, we introduce an approach to variability model integration that is based on the concepts of contextual modeling and conflict detection. We present the underlying concepts and the results of a corresponding performance analysis.
A survey of recommender systems for energy efficiency in buildings: Principles, challenges and prospects
Venue: arXiv cs.IR 2021
Year: 2021

Recommender systems have significantly developed in recent years in parallel with the witnessed advancements in both internet of things (IoT) and artificial intelligence (AI) technologies. Accordingly, as a consequence of IoT and AI, multiple forms of data are incorporated in these systems, e.g. social, implicit, local and personal information, which can help in improving recommender systems' performance and widen their applicability to traverse different disciplines. On the other side, energy efficiency in the building sector is becoming a hot research topic, in which recommender systems play a major role by promoting energy saving behavior and reducing carbon emissions. However, the deployment of the recommendation frameworks in buildings still needs more investigations to identify the current challenges and issues, where their solutions are the keys to enable the pervasiveness of research findings, and therefore, ensure a large-scale adoption of this technology. Accordingly, this paper presents, to the best of the authors' knowledge, the first timely and comprehensive reference for energy-efficiency recommendation systems through (i) surveying existing recommender systems for energy saving in buildings; (ii) discussing their evolution; (iii) providing an original taxonomy of these systems based on specified criteria, including the nature of the recommender engine, its objective, computing platforms, evaluation metrics and incentive measures; and (iv) conducting an in-depth, critical analysis to identify their limitations and unsolved issues. The derived challenges and areas of future implementation could effectively guide the energy research community to improve the energy-efficiency in buildings and reduce the cost of developed recommender systems-based solutions.
Translational Equivariance in Kernelizable Attention
Venue: arXiv cs.LG 2021
Year: 2021

While Transformer architectures have show remarkable success, they are bound to the computation of all pairwise interactions of input element and thus suffer from limited scalability. Recent work has been successful by avoiding the computation of the complete attention matrix, yet leads to problems down the line. The absence of an explicit attention matrix makes the inclusion of inductive biases relying on relative interactions between elements more challenging. An extremely powerful inductive bias is translational equivariance, which has been conjectured to be responsible for much of the success of Convolutional Neural Networks on image recognition tasks. In this work we show how translational equivariance can be implemented in efficient Transformers based on kernelizable attention - Performers. Our experiments highlight that the devised approach significantly improves robustness of Performers to shifts of input images compared to their naive application. This represents an important step on the path of replacing Convolutional Neural Networks with more expressive Transformer architectures and will help to improve sample efficiency and robustness in this realm.
Does the Adam Optimizer Exacerbate Catastrophic Forgetting?
Venue: arXiv cs.LG 2021
Year: 2021

Catastrophic forgetting remains a severe hindrance to the broad application of artificial neural networks (ANNs), however, it continues to be a poorly understood phenomenon. Despite the extensive amount of work on catastrophic forgetting, we argue that it is still unclear how exactly the phenomenon should be quantified, and, moreover, to what degree all of the choices we make when designing learning systems affect the amount of catastrophic forgetting. We use various testbeds from the reinforcement learning and supervised learning literature to (1) provide evidence that the choice of which modern gradient-based optimization algorithm is used to train an ANN has a significant impact on the amount of catastrophic forgetting and show that-surprisingly-in many instances classical algorithms such as vanilla SGD experience less catastrophic forgetting than the more modern algorithms such as Adam. We empirically compare four different existing metrics for quantifying catastrophic forgetting and (2) show that the degree to which the learning systems experience catastrophic forgetting is sufficiently sensitive to the metric used that a change from one principled metric to another is enough to change the conclusions of a study dramatically. Our results suggest that a much more rigorous experimental methodology is required when looking at catastrophic forgetting. Based on our results, we recommend inter-task forgetting in supervised learning must be measured with both retention and relearning metrics concurrently, and intra-task forgetting in reinforcement learning must-at the very least-be measured with pairwise interference.
Sparse Channel Reconstruction With Nonconvex Regularizer via DC Programming for Massive MIMO Systems
Venue: arXiv cs.IT 2021
Year: 2021

Sparse channel estimation for massive multiple-input multiple-output systems has drawn much attention in recent years. The required pilots are substantially reduced when the sparse channel state vectors can be reconstructed from a few numbers of measurements. A popular approach for sparse reconstruction is to solve the least-squares problem with a convex regularization. However, the convex regularizer is either too loose to force sparsity or lead to biased estimation. In this paper, the sparse channel reconstruction is solved by minimizing the least-squares objective with a nonconvex regularizer, which can exactly express the sparsity constraint and avoid introducing serious bias in the solution. A novel algorithm is proposed for solving the resulting nonconvex optimization via the difference of convex functions programming and the gradient projection descent. Simulation results show that the proposed algorithm is fast and accurate, and it outperforms the existing sparse recovery algorithms in terms of reconstruction errors.
Deep Neural Networks for Radar Waveform Classification
Venue: arXiv cs.LG 2021
Year: 2021

We consider the problem of classifying radar pulses given raw I/Q waveforms in the presence of noise and absence of synchronization. We also consider the problem of classifying multiple superimposed radar pulses. For both, we design deep neural networks (DNNs) that are robust to synchronization, pulse width, and SNR. Our designs yield more than 100x reduction in error-rate over the previous state-of-the-art.
How COVID-19 Is Changing Our Language : Detecting Semantic Shift in Twitter Word Embeddings
Venue: arXiv cs.CL 2021
Year: 2021

Words are malleable objects, influenced by events that are reflected in written texts. Situated in the global outbreak of COVID-19, our research aims at detecting semantic shifts in social media language triggered by the health crisis. With COVID-19 related big data extracted from Twitter, we train separate word embedding models for different time periods after the outbreak. We employ an alignment-based approach to compare these embeddings with a general-purpose Twitter embedding unrelated to COVID-19. We also compare our trained embeddings among them to observe diachronic evolution. Carrying out case studies on a set of words chosen by topic detection, we verify that our alignment approach is valid. Finally, we quantify the size of global semantic shift by a stability measure based on back-and-forth rotational alignment.
Recent Developments in Blockchain Technology and their Impact on Energy Consumption
Venue: arXiv cs.CR 2021
Year: 2021

The enormous power consumption of Bitcoin has led to undifferentiated discussions in science and practice about the sustainability of blockchain and distributed ledger technology in general. However, blockchain technology is far from homogeneous - not only with regard to its applications, which now go far beyond cryptocurrencies and have reached businesses and the public sector, but also with regard to its technical characteristics and, in particular, its power consumption. This paper summarizes the status quo of the power consumption of various implementations of blockchain technology, with special emphasis on the recent 'Bitcoin Halving' and so-called 'zk-rollups'. We argue that although Bitcoin and other proof-of-work blockchains do indeed consume a lot of power, alternative blockchain solutions with significantly lower power consumption are already available today, and new promising concepts are being tested that could further reduce in particular the power consumption of large blockchain networks in the near future. From this we conclude that although the criticism of Bitcoin's power consumption is legitimate, it should not be used to derive an energy problem of blockchain technology in general. In many cases in which processes can be digitised or improved with the help of more energy-efficient blockchain variants, one can even expect net energy savings.
Metatheory.jl: Fast and Elegant Algebraic Computation in Julia with Extensible Equality Saturation
Venue: arXiv cs.PL 2021
Year: 2021

We introduce Metatheory.jl: a lightweight and performant general purpose symbolics and metaprogramming framework meant to simplify the act of writing complex Julia metaprograms and to significantly enhance Julia with a native term rewriting system, based on state-of-the-art equality saturation techniques, and a dynamic first class Abstract Syntax Tree (AST) pattern matching system that is dynamically composable in an algebraic fashion, taking full advantage of the language's powerful reflection capabilities. Our contribution allows to perform general purpose symbolic mathematics, manipulation, optimization, synthesis or analysis of syntactically valid Julia expressions with a clean and concise programming interface, both during compilation or execution of programs.
Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport
Venue: arXiv cs.LG 2021
Year: 2021

We consider constrained policy optimization in Reinforcement Learning, where the constraints are in form of marginals on state visitations and global action executions. Given these distributions, we formulate policy optimization as unbalanced optimal transport over the space of occupancy measures. We propose a general purpose RL objective based on Bregman divergence and optimize it using Dykstra's algorithm. The approach admits an actor-critic algorithm for when the state or action space is large, and only samples from the marginals are available. We discuss applications of our approach and provide demonstrations to show the effectiveness of our algorithm.
C11Tester: A Race Detector for C/C++ Atomics Technical Report
Venue: arXiv cs.PL 2021
Year: 2021

Writing correct concurrent code that uses atomics under the C/C++ memory model is extremely difficult. We present C11Tester, a race detector for the C/C++ memory model that can explore executions in a larger fragment of the C/C++ memory model than previous race detector tools. Relative to previous work, C11Tester's larger fragment includes behaviors that are exhibited by ARM processors. C11Tester uses a new constraint-based algorithm to implement modification order that is optimized to allow C11Tester to make decisions in terms of application-visible behaviors. We evaluate C11Tester on several benchmark applications, and compare C11Tester's performance to both tsan11rec, the state of the art tool that controls scheduling for C/C++; and tsan11, the state of the art tool that does not control scheduling.
Structured Graph Learning for Scalable Subspace Clustering: From Single-view to Multi-view
Venue: arXiv cs.LG 2021
Year: 2021

Graph-based subspace clustering methods have exhibited promising performance. However, they still suffer some of these drawbacks: encounter the expensive time overhead, fail in exploring the explicit clusters, and cannot generalize to unseen data points. In this work, we propose a scalable graph learning framework, seeking to address the above three challenges simultaneously. Specifically, it is based on the ideas of anchor points and bipartite graph. Rather than building a $n\times n$ graph, where $n$ is the number of samples, we construct a bipartite graph to depict the relationship between samples and anchor points. Meanwhile, a connectivity constraint is employed to ensure that the connected components indicate clusters directly. We further establish the connection between our method and the K-means clustering. Moreover, a model to process multi-view data is also proposed, which is linear scaled with respect to $n$. Extensive experiments demonstrate the efficiency and effectiveness of our approach with respect to many state-of-the-art clustering methods.
Empirical Characterization of Graph Sampling Algorithms
Venue: arXiv cs.SI 2021
Year: 2021

Graph sampling allows mining a small representative subgraph from a big graph. Sampling algorithms deploy different strategies to replicate the properties of a given graph in the sampled graph. In this study, we provide a comprehensive empirical characterization of five graph sampling algorithms on six properties of a graph including degree, clustering coefficient, path length, global clustering coefficient, assortativity, and modularity. We extract samples from fifteen graphs grouped into five categories including collaboration, social, citation, technological, and synthetic graphs. We provide both qualitative and quantitative results. We find that there is no single method that extracts true samples from a given graph with respect to the properties tested in this work. Our results show that the sampling algorithm that aggressively explores the neighborhood of a sampled node performs better than the others.
LEAD: LiDAR Extender for Autonomous Driving
Venue: arXiv cs.CV 2021
Year: 2021

3D perception using sensors under vehicle industrial standard is the rigid demand in autonomous driving. MEMS LiDAR emerges with irresistible trend due to its lower cost, more robust, and meeting the mass-production standards. However, it suffers small field of view (FoV), slowing down the step of its population. In this paper, we propose LEAD, i.e., LiDAR Extender for Autonomous Driving, to extend the MEMS LiDAR by coupled image w.r.t both FoV and range. We propose a multi-stage propagation strategy based on depth distributions and uncertainty map, which shows effective propagation ability. Moreover, our depth outpainting/propagation network follows a teacher-student training fashion, which transfers depth estimation ability to depth completion network without any scale error passed. To validate the LiDAR extension quality, we utilize a high-precise laser scanner to generate a ground-truth dataset. Quantitative and qualitative evaluations show that our scheme outperforms SOTAs with a large margin. We believe the proposed LEAD along with the dataset would benefit the community w.r.t depth researches.
Joint self-supervised blind denoising and noise estimation
Venue: arXiv cs.LG 2021
Year: 2021

We propose a novel self-supervised image blind denoising approach in which two neural networks jointly predict the clean signal and infer the noise distribution. Assuming that the noisy observations are independent conditionally to the signal, the networks can be jointly trained without clean training data. Therefore, our approach is particularly relevant for biomedical image denoising where the noise is difficult to model precisely and clean training data are usually unavailable. Our method significantly outperforms current state-of-the-art self-supervised blind denoising algorithms, on six publicly available biomedical image datasets. We also show empirically with synthetic noisy data that our model captures the noise distribution efficiently. Finally, the described framework is simple, lightweight and computationally efficient, making it useful in practical cases.