Reviewerly/exHarmony huggingface dataset:
Unsupervised Online Grounding of Natural Language during Human-Robot Interactions

Allowing humans to communicate through natural language with robots requires connections between words and percepts. The process of creating these connections is called symbol grounding and has been studied for nearly three decades. Although many studies have been conducted, not many considered grounding of synonyms and the employed algorithms either work only offline or in a supervised manner. In this paper, a cross-situational learning based grounding framework is proposed that allows grounding of words and phrases through corresponding percepts without human supervision and online, i.e. it does not require any explicit training phase, but instead updates the obtained mappings for every new encountered situation. The proposed framework is evaluated through an interaction experiment between a human tutor and a robot, and compared to an existing unsupervised grounding framework. The results show that the proposed framework is able to ground words through their corresponding percepts online and in an unsupervised manner, while outperforming the baseline framework.
Breaking and Fixing Destructive Code Read Defenses

Just-in-time return-oriented programming (JIT-ROP) is a powerful memory corruption attack that bypasses various forms of code randomization. Execute-only memory (XOM) can potentially prevent these attacks, but requires source code. In contrast, destructive code reads (DCR) provide a trade-off between security and legacy compatibility. The common belief is that DCR provides strong protection if combined with a high-entropy code randomization. The contribution of this paper is twofold: first, we demonstrate that DCR can be bypassed regardless of the underlying code randomization scheme. To this end, we show novel, generic attacks that infer the code layout for highly randomized program code. Second, we present the design and implementation of BGDX (Byte-Granular DCR and XOM), a novel mitigation technique that protects legacy binaries against code inference attacks. BGDX enforces memory permissions on a byte-granular level allowing us to combine DCR and XOM for legacy, off-the-shelf binaries. Our evaluation shows that BGDX is not only effective, but highly efficient, imposing only a geometric mean performance overhead of 3.95% on SPEC.
Logic, Language, and Calculus

The difference between object-language and metalanguage is crucial for logical analysis, but has yet not been examined for the field of computer science. In this paper the difference is examined with regard to inferential relations. It is argued that inferential relations in a metalanguage (like a calculus for propositional logic) cannot represent conceptual relations of natural language. Inferential relations govern our concept use and understanding. Several approaches in the field of Natural Language Understanding (NLU) and Natural Language Inference (NLI) take this insight in account, but do not consider, how an inference can be assessed as a good inference. I present a logical analysis that can assesss the normative dimension of inferences, which is a crucial part of logical understanding and goes beyond formal understanding of metalanguages.
On integral weight spectra of the MDS codes cosets of weight 1, 2, and 3

The weight of a coset of a code is the smallest Hamming weight of any vector in the coset. For a linear code of length $n$, we call integral weight spectrum the overall numbers of weight $w$ vectors, $0\le w\le n$, in all the cosets of a fixed weight. For maximum distance separable (MDS) codes, we obtained new convenient formulas of integral weight spectra of cosets of weight 1 and 2. Also, we give the spectra for the weight 3 cosets of MDS codes with minimum distance $5$ and covering radius $3$.
Coded Distributed Computing with Partial Recovery

Coded computation techniques provide robustness against straggling workers in distributed computing. However, most of the existing schemes require exact provisioning of the straggling behaviour and ignore the computations carried out by straggling workers. Moreover, these schemes are typically designed to recover the desired computation results accurately, while in many machine learning and iterative optimization algorithms, faster approximate solutions are known to result in an improvement in the overall convergence time. In this paper, we first introduce a novel coded matrix-vector multiplication scheme, called coded computation with partial recovery (CCPR), which benefits from the advantages of both coded and uncoded computation schemes, and reduces both the computation time and the decoding complexity by allowing a trade-off between the accuracy and the speed of computation. We then extend this approach to distributed implementation of more general computation tasks by proposing a coded communication scheme with partial recovery, where the results of subtasks computed by the workers are coded before being communicated. Numerical simulations on a large linear regression task confirm the benefits of the proposed distributed computation scheme with partial recovery in terms of the trade-off between the computation accuracy and latency.
Playing with Words at the National Library of Sweden -- Making a Swedish BERT

This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsf\"ormedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: https://github.com/Kungbib/swedish-bert-models .
Swoosh! Rattle! Thump! -- Actions that Sound

Truly intelligent agents need to capture the interplay of all their senses to build a rich physical understanding of their world. In robotics, we have seen tremendous progress in using visual and tactile perception; however, we have often ignored a key sense: sound. This is primarily due to the lack of data that captures the interplay of action and sound. In this work, we perform the first large-scale study of the interactions between sound and robotic action. To do this, we create the largest available sound-action-vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot. By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information. Using this data, we explore the synergies between sound and action and present three key insights. First, sound is indicative of fine-grained object class information, e.g., sound can differentiate a metal screwdriver from a metal wrench. Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied to the object. Finally, object representations derived from audio embeddings are indicative of implicit physical properties. We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings. Project videos and data are at https://dhiraj100892.github.io/swoosh/
Fast Arithmetic Hardware Library For RLWE-Based Homomorphic Encryption

In this work, we propose an open-source, first-of-its-kind, arithmetic hardware library with a focus on accelerating the arithmetic operations involved in Ring Learning with Error (RLWE)-based somewhat homomorphic encryption (SHE). We design and implement a hardware accelerator consisting of submodules like Residue Number System (RNS), Chinese Remainder Theorem (CRT), NTT-based polynomial multiplication, modulo inverse, modulo reduction, and all the other polynomial and scalar operations involved in SHE. For all of these operations, wherever possible, we include a hardware-cost efficient serial and a fast parallel implementation in the library. A modular and parameterized design approach helps in easy customization and also provides flexibility to extend these operations for use in most homomorphic encryption applications that fit well into emerging FPGA-equipped cloud architectures. Using the submodules from the library, we prototype a hardware accelerator on FPGA. The evaluation of this hardware accelerator shows a speed up of approximately 4200x and 2950x to evaluate a homomorphic multiplication and addition respectively when compared to an existing software implementation.
Regulation conform DLT-operable payment adapter based on trustless - justified trust combined generalized state channels

Open technologies, decentralized computation and intelligent applications enable the third-generation web, Web 3.0, thereby digitizing whole industries. The emerging Economy of Things (EoT) will be based on software agents running on peer-to-peer trustless networks that require a programmable, regulation conform means of payment. We give an overview of current solutions that differ in their fundamental values and technological possibilities, like e.g. private-issued stablecoins, DLT-issued electronic money and genuine cryptocurrencies. Based on this analysis, we present the concept of justified trust and propose to combine the strengths of the crypto based, decentralized trustless elements with established and well regulated means of payment, based on this concept, via a secure external re-balancing interface. Combining the advantages, e.g. lightweight, trustless, efficient high frequency micro state transfers on the one hand, and ease of use, widely spread, accepted alignment to a multitude of regulative requirements, on the other hand, while neither leading into a lock-in in any of the proposed solutions, nor undermining the basic principles of the crypto-movement or unnecessarily reinforcing the banking system provides a synergy and the necessary flexibility for further evolution alongside the regulative framework. This offers a regulation conform transitional solution that can be implemented in the short term, which enables companies to place their decentralized business operations in a regulated environment. The contribution of our work is twofold: First, we illustrate and discuss different DLT-operable means of payment. Second, our research proposes a novel hybrid payment solution by interfacing trustless with justified trust combined generalized state channels.
Deep learning for scene recognition from visual data: a survey

The use of deep learning techniques has exploded during the last few years, resulting in a direct contribution to the field of artificial intelligence. This work aims to be a review of the state-of-the-art in scene recognition with deep learning models from visual data. Scene recognition is still an emerging field in computer vision, which has been addressed from a single image and dynamic image perspective. We first give an overview of available datasets for image and video scene recognition. Later, we describe ensemble techniques introduced by research papers in the field. Finally, we give some remarks on our findings and discuss what we consider challenges in the field and future lines of research. This paper aims to be a future guide for model selection for the task of scene recognition.
PointTrack++ for Effective Online Multi-Object Tracking and Segmentation

Multiple-object tracking and segmentation (MOTS) is a novel computer vision task that aims to jointly perform multiple object tracking (MOT) and instance segmentation. In this work, we present PointTrack++, an effective on-line framework for MOTS, which remarkably extends our recently proposed PointTrack framework. To begin with, PointTrack adopts an efficient one-stage framework for instance segmentation, and learns instance embeddings by converting compact image representations to un-ordered 2D point cloud. Compared with PointTrack, our proposed PointTrack++ offers three major improvements. Firstly, in the instance segmentation stage, we adopt a semantic segmentation decoder trained with focal loss to improve the instance selection quality. Secondly, to further boost the segmentation performance, we propose a data augmentation strategy by copy-and-paste instances into training images. Finally, we introduce a better training strategy in the instance association stage to improve the distinguishability of learned instance embeddings. The resulting framework achieves the state-of-the-art performance on the 5th BMTT MOTChallenge.
Expected Eligibility Traces

The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that could also have led to the current state. In this work, we introduce expected eligibility traces. Expected traces allow, with a single update, to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We discuss when expected traces provide benefits over classic (instantaneous) traces in temporal-difference learning, and show that sometimes substantial improvements can be attained. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping, which ensures that the resulting algorithm is a strict generalisation of TD($\lambda$). Finally, we discuss possible extensions and connections to related ideas, such as successor features.
Do Not Mask What You Do Not Need to Mask: a Parser-Free Virtual Try-On

The 2D virtual try-on task has recently attracted a great interest from the research community, for its direct potential applications in online shopping as well as for its inherent and non-addressed scientific challenges. This task requires fitting an in-shop cloth image on the image of a person, which is highly challenging because it involves cloth warping, image compositing, and synthesizing. Casting virtual try-on into a supervised task faces a difficulty: available datasets are composed of pairs of pictures (cloth, person wearing the cloth). Thus, we have no access to ground-truth when the cloth on the person changes. State-of-the-art models solve this by masking the cloth information on the person with both a human parser and a pose estimator. Then, image synthesis modules are trained to reconstruct the person image from the masked person image and the cloth image. This procedure has several caveats: firstly, human parsers are prone to errors; secondly, it is a costly pre-processing step, which also has to be applied at inference time; finally, it makes the task harder than it is since the mask covers information that should be kept such as hands or accessories. In this paper, we propose a novel student-teacher paradigm where the teacher is trained in the standard way (reconstruction) before guiding the student to focus on the initial task (changing the cloth). The student additionally learns from an adversarial loss, which pushes it to follow the distribution of the real images. Consequently, the student exploits information that is masked to the teacher. A student trained without the adversarial loss would not use this information. Also, getting rid of both human parser and pose estimator at inference time allows obtaining a real-time virtual try-on.
Multimodal Container Planning: a QUBO Formulation and Implementation on a Quantum Annealer

Quantum computing is developing fast. Real world applications are within reach in the coming years. One of the most promising areas is combinatorial optimisation, where the Quadratic Unconstrained Binary Optimisation (QUBO) problem formulation is used to get good approximate solutions. Both the universal quantum computer as the quantum annealer can handle this kind of problems well. In this paper, we present an application on multimodal container planning. We show how to map this problem to a QUBO problem formulation and how the practical implementation can be done on the quantum annealer produced by D-Wave Systems.
Lale: Consistent Automated Machine Learning

Automated machine learning makes it easier for data scientists to develop pipelines by searching over possible choices for hyperparameters, algorithms, and even pipeline topologies. Unfortunately, the syntax for automated machine learning tools is inconsistent with manual machine learning, with each other, and with error checks. Furthermore, few tools support advanced features such as topology search or higher-order operators. This paper introduces Lale, a library of high-level Python interfaces that simplifies and unifies automated machine learning in a consistent way.
Towards the Adoption of OMG Standards in the Development of SOA-Based IoT Systems

A common feature of the Internet of Things (IoT) is the high heterogeneity, regarding network protocols, data formats, hardware and software platforms. Aiming to deal with such a degree of heterogeneity, several frameworks have applied the Model-Driven Development (MDD) to build IoT applications. On the software architecture viewpoint, the literature has shown that the Service-Oriented Architecture (SOA) is a promising style to address the interoperability of entities composing these solutions. Some features of IoT make it challenging to analyze the impact of design decisions on the SOA-based IoT applications behavior. Thus, it is a key requirement to simulate the model to verify whether the system performs as expected before its implementation. Although the literature has identified that the SOA style is suitable for addressing the interoperability, existing modelling languages do not consider SOA elements as first-class citizens when designing IoT applications. Furthermore, although existing MDD frameworks provide modeling languages comprising well-defined syntax, they lack execution semantics, thus, are not suitable for model execution and analysis. This work aims at addressing these issues by introducing IoTDraw. The framework provides a fully OMG-compliant executable modeling language for SOA-based IoT systems; thus, its specifications can be implemented by any tool implementing OMG standards.
Entanglement-Assisted Quantum Error Correcting Codes From RS Codes and BCH Codes with Extension Degree 2

Entanglement-assisted quantum error correcting codes (EAQECCs) constructed from Reed-Solomon codes and BCH codes are considered in this work. It is provided a complete and explicit formula for the parameters of EAQECCs coming from any Reed-Solomon code, for the Hermitian metric, and from any BCH code with extension degree $2$ and consecutive cyclotomic cosets, for both the Euclidean and the Hermitian metric. The main task in this work is the computation of a completely general formula for $c$, the minimum number of required maximally entangled quantum states.
A New Theoretical Framework of Pyramid Markov Processes for Blockchain  Selfish Mining

In this paper, we provide a new theoretical framework of pyramid Markov processes to solve some open and fundamental problems of blockchain selfish mining under a rigorous mathematical setting. We first describe a more general model of blockchain selfish mining with both a two-block leading competitive criterion and a new economic incentive mechanism. Then we establish a pyramid Markov process and show that it is irreducible and positive recurrent, and its stationary probability vector is matrix-geometric with an explicitly representable rate matrix. Also, we use the stationary probability vector to study the influence of many orphan blocks on the waste of computing resource. Next, we set up a pyramid Markov reward process to investigate the long-run average profits of the honest and dishonest mining pools, respectively. As a by-product, we build three approximative Markov processes and provide some new interesting interpretation on the Markov chain and the revenue analysis reported in the seminal work by Eyal and Sirer (2014). Note that the pyramid Markov (reward) processes can open up a new avenue in the study of blockchain selfish mining. Thus we hope that the methodology and results developed in this paper shed light on the blockchain selfish mining such that a series of promising research can be developed potentially.
Federated Learning with Compression: Unified Analysis and Sharp Guarantees

In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions. Two notable trends to deal with the communication overhead of federated algorithms are gradient compression and local computation with periodic communication. Despite many attempts, characterizing the relationship between these two approaches has proven elusive. We address this by proposing a set of algorithms with periodical compressed (quantized or sparsified) communication and analyze their convergence properties in both homogeneous and heterogeneous local data distribution settings. For the homogeneous setting, our analysis improves existing bounds by providing tighter convergence rates for both strongly convex and non-convex objective functions. To mitigate data heterogeneity, we introduce a local gradient tracking scheme and obtain sharp convergence rates that match the best-known communication complexities without compression for convex, strongly convex, and nonconvex settings. We complement our theoretical results and demonstrate the effectiveness of our proposed methods by several experiments on real-world datasets.
Efficient estimation of the ANOVA mean dimension, with an application to neural net classification

The mean dimension of a black box function of $d$ variables is a convenient way to summarize the extent to which it is dominated by high or low order interactions. It is expressed in terms of $2^d-1$ variance components but it can be written as the sum of $d$ Sobol' indices that can be estimated by leave one out methods. We compare the variance of these leave one out methods: a Gibbs sampler called winding stairs, a radial sampler that changes each variable one at a time from a baseline, and a naive sampler that never reuses function evaluations and so costs about double the other methods. For an additive function the radial and winding stairs are most efficient. For a multiplicative function the naive method can easily be most efficient if the factors have high kurtosis. As an illustration we consider the mean dimension of a neural network classifier of digits from the MNIST data set. The classifier is a function of $784$ pixels. For that problem, winding stairs is the best algorithm. We find that inputs to the final softmax layer have mean dimensions ranging from $1.35$ to $2.0$.