OpenRaiser/Intern-Atlas huggingface dataset:
Modeling morphology with Linear Discriminative Learning: considerations and design choices
Venue: arXiv cs.CL 2021
Year: 2021

This study addresses a series of methodological questions that arise when modeling inflectional morphology with Linear Discriminative Learning. Taking the semi-productive German noun system as example, we illustrate how decisions made about the representation of form and meaning influence model performance. We clarify that for modeling frequency effects in learning, it is essential to make use of incremental learning rather than the endstate of learning. We also discuss how the model can be set up to approximate the learning of inflected words in context. In addition, we illustrate how in this approach the wug task can be modeled in considerable detail. In general, the model provides an excellent memory for known words, but appropriately shows more limited performance for unseen data, in line with the semi-productivity of German noun inflection and generalization performance of native German speakers.
Simplifying Robot Programming using Augmented Reality and End-User Development
Venue: arXiv cs.RO 2021
Year: 2021

Robots are widespread across diverse application contexts. Teaching robots to perform tasks, in their respective contexts, demands a high domain and programming expertise. However, robot programming faces high entry barriers due to the complexity of robot programming itself. Even for experts robot programming is a cumbersome and error-prone task where faulty robot programs can be created, causing damage when being executed on a real robot. To simplify the process of robot programming, we combine Augmented Reality (AR) with principles of end-user development. By combining them, the real environment is extended with useful virtual artifacts that can enable experts as well as non-professionals to perform complex robot programming tasks. Therefore, Simple Programming Environment in Augmented Reality with Enhanced Debugging (SPEARED) was developed as a prototype for an AR-assisted robot programming environment. SPEARED makes use of AR to project a robot as well as a programming environment onto the target working space. To evaluate our approach, expert interviews with domain experts from the area of industrial automation, robotics, and AR were performed. The experts agreed that SPEARED has the potential to enrich and ease current robot programming processes.
Grounds for Suspicion: Physics-based Early Warnings for Stealthy Attacks on Industrial Control Systems
Venue: arXiv cs.CR 2021
Year: 2021

Stealthy attacks on Industrial Control Systems can cause significant damage while evading detection. In this paper, instead of focusing on the detection of stealthy attacks, we aim to provide early warnings to operators, in order to avoid physical damage and preserve in advance data that may serve as an evidence during an investigation. We propose a framework to provide grounds for suspicion, i.e. preliminary indicators reflecting the likelihood of success of a stealthy attack. We propose two grounds for suspicion based on the behaviour of the physical process: (i) feasibility of a stealthy attack, and (ii) proximity to unsafe operating regions. We propose a metric to measure grounds for suspicion in real-time and provide soundness principles to ensure that such a metric is consistent with the grounds for suspicion. We apply our framework to Linear Time-Invariant (LTI) systems and formulate the suspicion metric computation as a real-time reachability problem. We validate our framework on a case study involving the benchmark Tennessee-Eastman process. We show through numerical simulation that we can provide early warnings well before a potential stealthy attack can cause damage, while incurring minimal load on the network. Finally, we apply our framework on a use case to illustrate its usefulness in supporting early evidence collection.
Unsupervised Abstractive Opinion Summarization by Generating Sentences with Tree-Structured Topic Guidance
Venue: arXiv cs.CL 2021
Year: 2021

This paper presents a novel unsupervised abstractive summarization method for opinionated texts. While the basic variational autoencoder-based models assume a unimodal Gaussian prior for the latent code of sentences, we alternate it with a recursive Gaussian mixture, where each mixture component corresponds to the latent code of a topic sentence and is mixed by a tree-structured topic distribution. By decoding each Gaussian component, we generate sentences with tree-structured topic guidance, where the root sentence conveys generic content, and the leaf sentences describe specific topics. Experimental results demonstrate that the generated topic sentences are appropriate as a summary of opinionated texts, which are more informative and cover more input contents than those generated by the recent unsupervised summarization model (Bražinskas et al., 2020). Furthermore, we demonstrate that the variance of latent Gaussians represents the granularity of sentences, analogous to Gaussian word embedding (Vilnis and McCallum, 2015).
Compositional Sketch Search
Venue: arXiv cs.CV 2021
Year: 2021

We present an algorithm for searching image collections using free-hand sketches that describe the appearance and relative positions of multiple objects. Sketch based image retrieval (SBIR) methods predominantly match queries containing a single, dominant object invariant to its position within an image. Our work exploits drawings as a concise and intuitive representation for specifying entire scene compositions. We train a convolutional neural network (CNN) to encode masked visual features from sketched objects, pooling these into a spatial descriptor encoding the spatial relationships and appearances of objects in the composition. Training the CNN backbone as a Siamese network under triplet loss yields a metric search embedding for measuring compositional similarity which may be efficiently leveraged for visual search by applying product quantization.
Modeling memory bandwidth patterns on NUMA machines with performance counters
Venue: arXiv cs.DC 2021
Year: 2021

Computers used for data analytics are often NUMA systems with multiple sockets per machine, multiple cores per socket, and multiple thread contexts per core. To get the peak performance out of these machines requires the correct number of threads to be placed in the correct positions on the machine. One particularly interesting element of the placement of memory and threads is the way it effects the movement of data around the machine, and the increased latency this can introduce to reads and writes. In this paper we describe work on modeling the bandwidth requirements of an application on a NUMA compute node based on the placement of threads. The model is parameterized by sampling performance counters during 2 application runs with carefully chosen thread placements. Evaluating the model with thousands of measurements shows a median difference from predictions of 2.34% of the bandwidth. The results of this modeling can be used in a number of ways varying from: Performance debugging during development where the programmer can be alerted to potentially problematic memory access patterns; To systems such as Pandia which take an application and predict the performance and system load of a proposed thread count and placement; To libraries of data structures such as Parallel Collections and Smart Arrays that can abstract from the user memory placement and thread placement issues when parallelizing code.
Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations
Venue: arXiv cs.LG 2021
Year: 2021

Averaging predictions over a set of models -- an ensemble -- is widely used to improve predictive performance and uncertainty estimation of deep learning models. At the same time, many machine learning systems, such as search, matching, and recommendation systems, heavily rely on embeddings. Unfortunately, due to misalignment of features of independently trained models, embeddings, cannot be improved with a naive deep ensemble like approach. In this work, we look at the ensembling of representations and propose mean embeddings with test-time augmentation (MeTTA) simple yet well-performing recipe for ensembling representations. Empirically we demonstrate that MeTTA significantly boosts the quality of linear evaluation on ImageNet for both supervised and self-supervised models. Even more exciting, we draw connections between MeTTA, image retrieval, and transformation invariant models. We believe that spreading the success of ensembles to inference higher-quality representations is the important step that will open many new applications of ensembling.
On the Power of Multitask Representation Learning in Linear MDP
Venue: arXiv cs.LG 2021
Year: 2021

While multitask representation learning has become a popular approach in reinforcement learning (RL), theoretical understanding of why and when it works remains limited. This paper presents analyses for the statistical benefit of multitask representation learning in linear Markov Decision Process (MDP) under a generative model. In this paper, we consider an agent to learn a representation function $ϕ$ out of a function class $Φ$ from $T$ source tasks with $N$ data per task, and then use the learned $\hatϕ$ to reduce the required number of sample for a new task. We first discover a \emph{Least-Activated-Feature-Abundance} (LAFA) criterion, denoted as $κ$, with which we prove that a straightforward least-square algorithm learns a policy which is $\tilde{O}(H^2\sqrt{\frac{\mathcal{C}(Φ)^2 κd}{NT}+\frac{κd}{n}})$ sub-optimal. Here $H$ is the planning horizon, $\mathcal{C}(Φ)$ is $Φ$'s complexity measure, $d$ is the dimension of the representation (usually $d\ll \mathcal{C}(Φ)$) and $n$ is the number of samples for the new task. Thus the required $n$ is $O(κd H^4)$ for the sub-optimality to be close to zero, which is much smaller than $O(\mathcal{C}(Φ)^2κd H^4)$ in the setting without multitask representation learning, whose sub-optimality gap is $\tilde{O}(H^2\sqrt{\frac{κ\mathcal{C}(Φ)^2d}{n}})$. This theoretically explains the power of multitask representation learning in reducing sample complexity. Further, we note that to ensure high sample efficiency, the LAFA criterion $κ$ should be small. In fact, $κ$ varies widely in magnitude depending on the different sampling distribution for new task. This indicates adaptive sampling technique is important to make $κ$ solely depend on $d$. Finally, we provide empirical results of a noisy grid-world environment to corroborate our theoretical findings.
Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
Venue: arXiv cs.CV 2021
Year: 2021

We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.
Privacy Assessment of Federated Learning using Private Personalized Layers
Venue: arXiv cs.CR 2021
Year: 2021

Federated Learning (FL) is a collaborative scheme to train a learning model across multiple participants without sharing data. While FL is a clear step forward towards enforcing users' privacy, different inference attacks have been developed. In this paper, we quantify the utility and privacy trade-off of a FL scheme using private personalized layers. While this scheme has been proposed as local adaptation to improve the accuracy of the model through local personalization, it has also the advantage to minimize the information about the model exchanged with the server. However, the privacy of such a scheme has never been quantified. Our evaluations using motion sensor dataset show that personalized layers speed up the convergence of the model and slightly improve the accuracy for all users compared to a standard FL scheme while better preventing both attribute and membership inferences compared to a FL scheme using local differential privacy.
Time-free solution to independent set problem using P systems with active membranes
Venue: arXiv cs.CC 2021
Year: 2021

Membrane computing is a branch of natural computingwhich abstracts fromthe structure and the functioning of living cells. The computation models obtained in the field of membrane computing are usually called P systems. P systems have been used to solve computationally hard problems efficiently on the assumption that the execution of each rule is completed in exactly one time-unit (a global clock is assumed for timing and synchronizing the execution of rules). However, in biological reality, different biological processes take different times to be completed, which can also be influenced by many environmental factors. In this work, with this biological reality, we give a time-free solution to independent set problemusing P systems with active membranes, which solve the problem independent of the execution time of the involved rules.
Heterogeneous Multi-sensor Fusion with Random Finite Set Multi-object Densities
Venue: arXiv cs.IT 2021
Year: 2021

This paper addresses the density based multi-sensor cooperative fusion using random finite set (RFS) type multi-object densities (MODs). Existing fusion methods use scalar weights to characterize the relative information confidence among the local MODs, and in this way the portion of contribution of each local MOD to the fused global MOD can be tuned via adjusting these weights. Our analysis shows that the fusion mechanism of using a scalar coefficient can be oversimplified for practical scenarios, as the information confidence of an MOD is complex and usually space-varying due to the imperfection of sensor ability and the various impacts from surveillance environment. Consequently, severe fusion performance degradation can be observed when these scalar weights fail to reflect the actual situation. We make two contributions towards addressing this problem. Firstly, we propose a novel heterogeneous fusion method to perform the information averaging among local RFS MODs. By factorizing each local MODs into a number of smaller size sub-MODs, it can transform the original complicated fusion problem into a much easier parallelizable multi-cluster fusion problem. Secondly, as the proposed fusion strategy is a general procedure without any particular model assumptions, we further derive the detailed heterogeneous fusion equations, with centralized network architecture, for both the probability hypothesis density (PHD) filter and the multi-Bernoulli (MB) filter. The Gaussian mixture implementations of the proposed fusion algorithms are also presented. Various numerical experiments are designed to demonstrate the efficacy of the proposed fusion methods.
Generating Thermal Human Faces for Physiological Assessment Using Thermal Sensor Auxiliary Labels
Venue: arXiv cs.CV 2021
Year: 2021

Thermal images reveal medically important physiological information about human stress, signs of inflammation, and emotional mood that cannot be seen on visible images. Providing a method to generate thermal faces from visible images would be highly valuable for the telemedicine community in order to show this medical information. To the best of our knowledge, there are limited works on visible-to-thermal (VT) face translation, and many current works go the opposite direction to generate visible faces from thermal surveillance images (TV) for law enforcement applications. As a result, we introduce favtGAN, a VT GAN which uses the pix2pix image translation model with an auxiliary sensor label prediction network for generating thermal faces from visible images. Since most TV methods are trained on only one data source drawn from one thermal sensor, we combine datasets from faces and cityscapes. These combined data are captured from similar sensors in order to bootstrap the training and transfer learning task, especially valuable because visible-thermal face datasets are limited. Experiments on these combined datasets show that favtGAN demonstrates an increase in SSIM and PSNR scores of generated thermal faces, compared to training on a single face dataset alone.
Detect and remove watermark in deep neural networks via generative adversarial networks
Venue: arXiv cs.MM 2021
Year: 2021

Deep neural networks (DNN) have achieved remarkable performance in various fields. However, training a DNN model from scratch requires a lot of computing resources and training data. It is difficult for most individual users to obtain such computing resources and training data. Model copyright infringement is an emerging problem in recent years. For instance, pre-trained models may be stolen or abuse by illegal users without the authorization of the model owner. Recently, many works on protecting the intellectual property of DNN models have been proposed. In these works, embedding watermarks into DNN based on backdoor is one of the widely used methods. However, when the DNN model is stolen, the backdoor-based watermark may face the risk of being detected and removed by an adversary. In this paper, we propose a scheme to detect and remove watermark in deep neural networks via generative adversarial networks (GAN). We demonstrate that the backdoor-based DNN watermarks are vulnerable to the proposed GAN-based watermark removal attack. The proposed attack method includes two phases. In the first phase, we use the GAN and few clean images to detect and reverse the watermark in the DNN model. In the second phase, we fine-tune the watermarked DNN based on the reversed backdoor images. Experimental evaluations on the MNIST and CIFAR10 datasets demonstrate that, the proposed method can effectively remove about 98% of the watermark in DNN models, as the watermark retention rate reduces from 100% to less than 2% after applying the proposed attack. In the meantime, the proposed attack hardly affects the model's performance. The test accuracy of the watermarked DNN on the MNIST and the CIFAR10 datasets drops by less than 1% and 3%, respectively.
Archify: A Recommender System of Architectural Design Decisions
Venue: arXiv cs.SE 2021
Year: 2021

Software architectures play a critical role in software quality assurance. However, small and medium companies (SMC) often suffer from the absence of professionals with skills and expertise in software architecture. That situation potentially affects the final quality of the software products and pressures projects budget with extra costs with consulting. This paper presents a recommender system of architectural design decisions called Archify. The goal is to support SMC companies in part of the effort of architecturally designing their products. Archify implements a wizard-styled interface that guides the developer or project manager through a set of specific questions. While the user answers these questions, Archify buffers a set of corresponding architectural decision recommendations. As the final result, the system recommends a set of architectural decisions matching the project's needs according to the requirements (as provided by the user) of the software under development. Nineteen professionals from academia and industry evaluated Archify through two surveys. The findings reveal that 94.7% of the participants approved Archify as a supporting tool. Respondents also highlighted the lack of tools supporting software architecture design, remarking the relevance of the proposed system.
Semantic Representation and Inference for NLP
Venue: arXiv cs.CL 2021
Year: 2021

Semantic representation and inference is essential for Natural Language Processing (NLP). The state of the art for semantic representation and inference is deep learning, and particularly Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and transformer Self-Attention models. This thesis investigates the use of deep learning for novel semantic representation and inference, and makes contributions in the following three areas: creating training data, improving semantic representations and extending inference learning. In terms of creating training data, we contribute the largest publicly available dataset of real-life factual claims for the purpose of automatic claim verification (MultiFC), and we present a novel inference model composed of multi-scale CNNs with different kernel sizes that learn from external sources to infer fact checking labels. In terms of improving semantic representations, we contribute a novel model that captures non-compositional semantic indicators. By definition, the meaning of a non-compositional phrase cannot be inferred from the individual meanings of its composing words (e.g., hot dog). Motivated by this, we operationalize the compositionality of a phrase contextually by enriching the phrase representation with external word embeddings and knowledge graphs. Finally, in terms of inference learning, we propose a series of novel deep learning architectures that improve inference by using syntactic dependencies, by ensembling role guided attention heads, incorporating gating layers, and concatenating multiple heads in novel and effective ways. This thesis consists of seven publications (five published and two under review).
A Monte-Carlo Based Construction of Polarization-Adjusted Convolutional (PAC) Codes
Venue: arXiv cs.IT 2021
Year: 2021

This paper proposes a rate-profile construction method for polarization-adjusted convolutional (PAC) codes of any code length and rate, which is capable of maintaining trade-off between the error-correction performance and decoding complexity of PAC code. The proposed method can improve the error-correction performance of PAC codes while guaranteeing a low mean sequential decoding complexity for signal-to-noise ratio (SNR) values beyond a target SNR value.
Direction is what you need: Improving Word Embedding Compression in Large Language Models
Venue: arXiv cs.CL 2021
Year: 2021

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of these models to improve their inference time and memory footprint. This paper presents a novel loss objective to compress token embeddings in the Transformer-based models by leveraging an AutoEncoder architecture. More specifically, we emphasize the importance of the direction of compressed embeddings with respect to original uncompressed embeddings. The proposed method is task-agnostic and does not require further language modeling pre-training. Our method significantly outperforms the commonly used SVD-based matrix-factorization approach in terms of initial language model Perplexity. Moreover, we evaluate our proposed approach over SQuAD v1.1 dataset and several downstream tasks from the GLUE benchmark, where we also outperform the baseline in most scenarios. Our code is public.