OpenRaiser/Intern-Atlas huggingface dataset:
Decentralized Stochastic Optimization with Inherent Privacy Protection
Venue: arXiv cs.LG 2022
Year: 2022

Decentralized stochastic optimization is the basic building block of modern collaborative machine learning, distributed estimation and control, and large-scale sensing. Since involved data usually contain sensitive information like user locations, healthcare records and financial transactions, privacy protection has become an increasingly pressing need in the implementation of decentralized stochastic optimization algorithms. In this paper, we propose a decentralized stochastic gradient descent algorithm which is embedded with inherent privacy protection for every participating agent against other participating agents and external eavesdroppers. This proposed algorithm builds in a dynamics based gradient-obfuscation mechanism to enable privacy protection without compromising optimization accuracy, which is in significant difference from differential-privacy based privacy solutions for decentralized optimization that have to trade optimization accuracy for privacy. The dynamics based privacy approach is encryption-free, and hence avoids incurring heavy communication or computation overhead, which is a common problem with encryption based privacy solutions for decentralized stochastic optimization. Besides rigorously characterizing the convergence performance of the proposed decentralized stochastic gradient descent algorithm under both convex objective functions and non-convex objective functions, we also provide rigorous information-theoretic analysis of its strength of privacy protection. Simulation results for a distributed estimation problem as well as numerical experiments for decentralized learning on a benchmark machine learning dataset confirm the effectiveness of the proposed approach.
ConvMAE: Masked Convolution Meets Masked Autoencoders
Venue: arXiv cs.CV 2022
Year: 2022

Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained ConvMAE models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at https://github.com/Alpha-VL/ConvMAE.
Rebellion and Disobedience as Useful Tools in Human-Robot Interaction Research -- The Handheld Robotics Case
Venue: arXiv cs.RO 2022
Year: 2022

This position paper argues on the utility of rebellion and disobedience (RaD) in human-robot interaction (HRI). In general, we see two main opportunities in the use of controlled and well designed rebellion and disobedience: i) illuminate insight into the effectiveness of the collaboration (or lack of) and ii) prevent mistakes and correct user actions when in the user's own interest. Through the use of a close interaction modality, that of handheld robots, we discuss use cases for utility of rebellion and disobedience that can be applicable to other instances of HRI.
Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System
Venue: arXiv cs.CL 2022
Year: 2022

Contextually aware intelligent agents are often required to understand the users and their surroundings in real-time. Our goal is to build Artificial Intelligence (AI) systems that can assist children in their learning process. Within such complex frameworks, Spoken Dialogue Systems (SDS) are crucial building blocks to handle efficient task-oriented communication with children in game-based learning settings. We are working towards a multimodal dialogue system for younger kids learning basic math concepts. Our focus is on improving the Natural Language Understanding (NLU) module of the task-oriented SDS pipeline with limited datasets. This work explores the potential benefits of data augmentation with paraphrase generation for the NLU models trained on small task-specific datasets. We also investigate the effects of extracting entities for conceivably further data expansion. We have shown that paraphrasing with model-in-the-loop (MITL) strategies using small seed data is a promising approach yielding improved performance results for the Intent Recognition task.
CoCoA-MT: A Dataset and Benchmark for Contrastive Controlled MT with Application to Formality
Venue: arXiv cs.CL 2022
Year: 2022

The machine translation (MT) task is typically formulated as that of returning a single translation for an input segment. However, in many cases, multiple different translations are valid and the appropriate translation may depend on the intended target audience, characteristics of the speaker, or even the relationship between speakers. Specific problems arise when dealing with honorifics, particularly translating from English into languages with formality markers. For example, the sentence "Are you sure?" can be translated in German as "Sind Sie sich sicher?" (formal register) or "Bist du dir sicher?" (informal). Using wrong or inconsistent tone may be perceived as inappropriate or jarring for users of certain cultures and demographics. This work addresses the problem of learning to control target language attributes, in this case formality, from a small amount of labeled contrastive data. We introduce an annotated dataset (CoCoA-MT) and an associated evaluation metric for training and evaluating formality-controlled MT models for six diverse target languages. We show that we can train formality-controlled models by fine-tuning on labeled contrastive data, achieving high accuracy (82% in-domain and 73% out-of-domain) while maintaining overall quality.
Interpretable Machine Learning for Self-Service High-Risk Decision-Making
Venue: arXiv cs.LG 2022
Year: 2022

This paper contributes to interpretable machine learning via visual knowledge discovery in general line coordinates (GLC). The concepts of hyperblocks as interpretable dataset units and general line coordinates are combined to create a visual self-service machine learning model. The DSC1 and DSC2 lossless multidimensional coordinate systems are proposed. DSC1 and DSC2 can map multiple dataset attributes to a single two-dimensional (X, Y) Cartesian plane using a graph construction algorithm. The hyperblock analysis was used to determine visually appealing dataset attribute orders and to reduce line occlusion. It is shown that hyperblocks can generalize decision tree rules and a series of DSC1 or DSC2 plots can visualize a decision tree. The DSC1 and DSC2 plots were tested on benchmark datasets from the UCI ML repository. They allowed for visual classification of data. Additionally, areas of hyperblock impurity were discovered and used to establish dataset splits that highlight the upper estimate of worst-case model accuracy to guide model selection for high-risk decision-making. Major benefits of DSC1 and DSC2 is their highly interpretable nature. They allow domain experts to control or establish new machine learning models through visual pattern discovery.
Scaling up sign spotting through sign language dictionaries
Venue: arXiv cs.CV 2022
Year: 2022

The focus of this work is $\textit{sign spotting}$ - given a video of an isolated sign, our task is to identify $\textit{whether}$ and $\textit{where}$ it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) $\textit{watching}$ existing footage which is sparsely labelled using mouthing cues; (2) $\textit{reading}$ associated subtitles (readily available translations of the signed content) which provide additional $\textit{weak-supervision}$; (3) $\textit{looking up}$ words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.
FoReCo: a forecast-based recovery mechanism for real-time remote control of robotic manipulators
Venue: arXiv cs.NI 2022
Year: 2022

Wireless communications represent a game changer for future manufacturing plants, enabling flexible production chains as machinery and other components are not restricted to a location by the rigid wired connections on the factory floor. However, the presence of electromagnetic interference in the wireless spectrum may result in packet loss and delay, making it a challenging environment to meet the extreme reliability requirements of industrial applications. In such conditions, achieving real-time remote control, either from the Edge or Cloud, becomes complex. In this paper, we investigate a forecast-based recovery mechanism for real-time remote control of robotic manipulators (FoReCo) that uses Machine Learning (ML) to infer lost commands caused by interference in the wireless channel. FoReCo is evaluated through both simulation and experimentation in interference prone IEEE 802.11 wireless links, and using a commercial research robot that performs pick-and-place tasks. Results show that in case of interference, FoReCo trajectory error is decreased by x18 and x2 times in simulation and experimentation, and that FoReCo is sufficiently lightweight to be deployed in the hardware of already used in existing solutions.
Boolean Expressions in Firewall Analysis
Venue: arXiv cs.CR 2022
Year: 2022

Firewall policies are an important line of defence in cybersecurity, specifying which packets are allowed to pass through a network and which are not. These firewall policies are made up of a list of interacting rules. In practice, firewall can consist of hundreds or thousands of rules. This can be very difficult for a human to correctly configure. One proposed solution is to model firewall policies as Boolean expressions and use existing computer programs such as SAT solvers to verify that the firewall satisfies certain conditions. This paper takes an in-depth look at the Boolean expressions that represent firewall policies. We present an algorithm that translates a list of firewall rules into a Boolean expression in conjunctive normal form (CNF) or disjunctive normal form (DNF). We also place an upper bound on the size of the CNF and DNF that is polynomial in the number of rules in the firewall policy. This shows that past results suggesting a combinatorial explosion when converting from a Boolean expression in CNF to one in DNF does note occur in the context of firewall analysis
The Forgotten Margins of AI Ethics
Venue: arXiv cs.CY 2022
Year: 2022

How has recent AI Ethics literature addressed topics such as fairness and justice in the context of continued social and structural power asymmetries? We trace both the historical roots and current landmark work that have been shaping the field and categorize these works under three broad umbrellas: (i) those grounded in Western canonical philosophy, (ii) mathematical and statistical methods, and (iii) those emerging from critical data/algorithm/information studies. We also survey the field and explore emerging trends by examining the rapidly growing body of literature that falls under the broad umbrella of AI Ethics. To that end, we read and annotated peer-reviewed papers published over the past four years in two premier conferences: FAccT and AIES. We organize the literature based on an annotation scheme we developed according to three main dimensions: whether the paper deals with concrete applications, use-cases, and/or people's lived experience; to what extent it addresses harmed, threatened, or otherwise marginalized groups; and if so, whether it explicitly names such groups. We note that although the goals of the majority of FAccT and AIES papers were often commendable, their consideration of the negative impacts of AI on traditionally marginalized groups remained shallow. Taken together, our conceptual analysis and the data from annotated papers indicate that the field would benefit from an increased focus on ethical analysis grounded in concrete use-cases, people's experiences, and applications as well as from approaches that are sensitive to structural and historical power asymmetries.
A Music-Therapy Robotic Platform for Children with Autism: A Pilot Study
Venue: arXiv cs.HC 2022
Year: 2022

Children with Autism Spectrum Disorder (ASD) experience deficits in verbal and nonverbal communication skills including motor control, turn-taking, and emotion recognition. Innovative technology, such as socially assistive robots, has shown to be a viable method for Autism therapy. This paper presents a novel robot-based music-therapy platform for modeling and improving the social responses and behaviors of children with ASD. Our autonomous social interactive system consists of three modules. We adopted Short-time Fourier Transform and Levenshtein distance to fulfill the design requirements: a) "music detection" and b) "smart scoring and feedback", which allows NAO to understand music and provide additional practice and oral feedback to the users as applicable. We designed and implemented six Human-Robot-Interaction (HRI) sessions including four intervention sessions. Nine children with ASD and seven Typically Developing participated in a total of fifty HRI experimental sessions. Using our platform, we collected and analyzed data on social behavioral changes and emotion recognition using Electrodermal Activity (EDA) signals. The results of our experiments demonstrate most of the participants were able to complete motor control tasks with ~70% accuracy. Six out of the 9 ASD participants showed stable turn-taking behavior when playing music. The results of automated emotion classification using Support Vector Machines illustrate that emotional arousal in the ASD group can be detected and well recognized via EDA bio-signals. In summary, the results of our data analyses, including emotion classification using EDA signals, indicate that the proposed robot-music based therapy platform is an attractive and promising assistive tool to facilitate the improvement of fine motor control and turn-taking skills in children with ASD.
Improved Evaluation and Generation of Grid Layouts using Distance Preservation Quality and Linear Assignment Sorting
Venue: arXiv cs.CV 2022
Year: 2022

Images sorted by similarity enables more images to be viewed simultaneously, and can be very useful for stock photo agencies or e-commerce applications. Visually sorted grid layouts attempt to arrange images so that their proximity on the grid corresponds as closely as possible to their similarity. Various metrics exist for evaluating such arrangements, but there is low experimental evidence on correlation between human perceived quality and metric value. We propose Distance Preservation Quality (DPQ) as a new metric to evaluate the quality of an arrangement. Extensive user testing revealed stronger correlation of DPQ with user-perceived quality and performance in image retrieval tasks compared to other metrics. In addition, we introduce Fast Linear Assignment Sorting (FLAS) as a new algorithm for creating visually sorted grid layouts. FLAS achieves very good sorting qualities while improving run time and computational resources.
Evaluating the Fairness Impact of Differentially Private Synthetic Data
Venue: arXiv cs.LG 2022
Year: 2022

Differentially private (DP) synthetic data is a promising approach to maximizing the utility of data containing sensitive information. Due to the suppression of underrepresented classes that is often required to achieve privacy, however, it may be in conflict with fairness. We evaluate four DP synthesizers and present empirical results indicating that three of these models frequently degrade fairness outcomes on downstream binary classification tasks. We draw a connection between fairness and the proportion of minority groups present in the generated synthetic data, and find that training synthesizers on data that are pre-processed via a multi-label undersampling method can promote more fair outcomes without degrading accuracy.
Insights on Modelling Physiological, Appraisal, and Affective Indicators of Stress using Audio Features
Venue: arXiv cs.SD 2022
Year: 2022

Stress is a major threat to well-being that manifests in a variety of physiological and mental symptoms. Utilising speech samples collected while the subject is undergoing an induced stress episode has recently shown promising results for the automatic characterisation of individual stress responses. In this work, we introduce new findings that shed light onto whether speech signals are suited to model physiological biomarkers, as obtained via cortisol measurements, or self-assessed appraisal and affect measurements. Our results show that different indicators impact acoustic features in a diverse way, but that their complimentary information can nevertheless be effectively harnessed by a multi-tasking architecture to improve prediction performance for all of them.
EigenNoise: A Contrastive Prior to Warm-Start Representations
Venue: arXiv cs.CL 2022
Year: 2022

In this work, we present a naive initialization scheme for word vectors based on a dense, independent co-occurrence model and provide preliminary results that suggest it is competitive and warrants further investigation. Specifically, we demonstrate through information-theoretic minimum description length (MDL) probing that our model, EigenNoise, can approach the performance of empirically trained GloVe despite the lack of any pre-training data (in the case of EigenNoise). We present these preliminary results with interest to set the stage for further investigations into how this competitive initialization works without pre-training data, as well as to invite the exploration of more intelligent initialization schemes informed by the theory of harmonic linguistic structure. Our application of this theory likewise contributes a novel (and effective) interpretation of recent discoveries which have elucidated the underlying distributional information that linguistic representations capture from data and contrast distributions.
Exactly computable and continuous metrics on isometry classes of finite and 1-periodic sequences
Venue: arXiv cs.CG 2022
Year: 2022

The inevitable noise in real measurements motivates the problem to continuously quantify the similarity between rigid objects such as periodic time series and proteins given by ordered points and considered up to isometry maintaining inter-point distances. The past work produced many Hausdorff-like distances that have slow or approximate algorithms due to minimizations over infinitely many isometries. For finite and 1-periodic sequences under isometry in any high-dimensional Euclidean space, we introduce continuous metrics with faster algorithms. The key novelty in the periodic case is the continuity of new metrics under perturbations that change the minimum period.
Self-Serviced IoT: Practical and Private IoT Computation Offloading with Full User Control
Venue: arXiv cs.CR 2022
Year: 2022

The rapid increase in the adoption of Internet-of-Things (IoT) devices raises critical privacy concerns as these devices can access a variety of sensitive data. The current status quo of relying on manufacturers' cloud services to process this data is especially problematic since users cede control once their data leaves their home. Multiple recent incidents further call into question if vendors can indeed be trusted with users' data. At the same time, users desire compelling features supported by IoT devices and ML-based cloud inferences which compels them to subscribe to manufacturer-managed cloud services. An alternative to use a local in-home hub requires substantial hardware investment, management, and scalability limitations. This paper proposes Self-Serviced IoT (SSIoT), a clean-slate approach of using a hybrid hub-cloud setup to enable privacy-aware computation offload for IoT applications. Uniquely, SSIoT enables opportunistic computation offload to public cloud providers while still ensuring that the end-user retains complete end-to-end control of their private data reducing the trust required from public cloud providers. We show that SSIoT can leverage emerging function-as-a-service computation (e.g. AWS Lambda) to make these offloads cost-efficient, scalable and high performance as long as key limitations of being stateless, limited resources, and security isolation can be addressed. We build an end-to-end prototype of SSIoT and evaluate it using several micro-benchmarks and example applications representing real-world IoT use cases. Our results show that SSIoT is highly scalable, as compared to local-only approaches which struggle with as little as 2-4 apps in parallel. We also show that SSIoT is cost-efficient (operating a smart doorbell for $10 a year) at the cost of minimal additional latency as compared to a local-only hub, even with a hardware ML accelerator.
Static Analysis for AWS Best Practices in Python Code
Venue: arXiv cs.PL 2022
Year: 2022

Amazon Web Services (AWS) is a comprehensive and broadly adopted cloud provider, offering over 200 fully featured services, including compute, database, storage, networking and content delivery, machine learning, Internet of Things and many others. AWS SDKs provide access to AWS services through API endpoints. However, incorrect use of these APIs can lead to code defects, crashes, performance issues, and other problems. This paper presents automated static analysis rules, developed in the context of a commercial service for detection of code defects and security vulnerabilities, to identify deviations from AWS best practices in Python applications that use the AWS SDK. Such applications use the AWS SDK for Python, called "Boto3", to access AWS cloud services. However, precise static analysis of Python applications that use cloud SDKs requires robust type inference for inferring the types of cloud service clients. The dynamic style of Boto3 APIs poses unique challenges for type resolution, as does the interprocedural style in which service clients are used in practice. In support of our best-practices goal, we present a layered strategy for type inference that combines multiple type-resolution and tracking strategies in a staged manner. From our experiments across >3,000 popular Python GitHub repos that make use of the AWS SDK, our layered type inference system achieves 85% precision and 100% recall in inferring Boto3 clients in Python client code. Additionally, we present a representative sample of eight AWS best-practice rules that detect a wide range of issues including pagination, polling, and batch operations. We have assessed the efficacy of these rules based on real-world developer feedback. Developers have accepted more than 85% of the recommendations made by five out of eight Python rules, and almost 83% of all recommendations.