OpenRaiser/Intern-Atlas huggingface dataset:
Boulders Identification on Small Bodies Under Varying Illumination Conditions
Venue: arXiv cs.CV 2022
Year: 2022

The capability to detect boulders on the surface of small bodies is beneficial for vision-based applications such as navigation and hazard detection during critical operations. This task is challenging due to the wide assortment of irregular shapes, the characteristics of the boulders population, and the rapid variability in the illumination conditions. The authors address this challenge by designing a multi-step training approach to develop a data-driven image processing pipeline to robustly detect and segment boulders scattered over the surface of a small body. Due to the limited availability of labeled image-mask pairs, the developed methodology is supported by two artificial environments designed in Blender specifically for this work. These are used to generate a large amount of synthetic image-label sets, which are made publicly available to the image processing community. The methodology presented addresses the challenges of varying illumination conditions, irregular shapes, fast training time, extensive exploration of the architecture design space, and domain gap between synthetic and real images from previously flown missions. The performance of the developed image processing pipeline is tested both on synthetic and real images, exhibiting good performances, and high generalization capabilities
AGReE: A system for generating Automated Grammar Reading Exercises
Venue: arXiv cs.CL 2022
Year: 2022

We describe the AGReE system, which takes user-submitted passages as input and automatically generates grammar practice exercises that can be completed while reading. Multiple-choice practice items are generated for a variety of different grammar constructs: punctuation, articles, conjunctions, pronouns, prepositions, verbs, and nouns. We also conducted a large-scale human evaluation with around 4,500 multiple-choice practice items. We notice for 95% of items, a majority of raters out of five were able to identify the correct answer and for 85% of cases, raters agree that there is only one correct answer among the choices. Finally, the error analysis shows that raters made the most mistakes for punctuation and conjunctions.
Filter and evolve: progressive pseudo label refining for semi-supervised automatic speech recognition
Venue: arXiv cs.SD 2022
Year: 2022

Fine tuning self supervised pretrained models using pseudo labels can effectively improve speech recognition performance. But, low quality pseudo labels can misguide decision boundaries and degrade performance. We propose a simple yet effective strategy to filter low quality pseudo labels to alleviate this problem. Specifically, pseudo-labels are produced over the entire training set and filtered via average probability scores calculated from the model output. Subsequently, an optimal percentage of utterances with high probability scores are considered reliable training data with trustworthy labels. The model is iteratively updated to correct the unreliable pseudo labels to minimize the effect of noisy labels. The process above is repeated until unreliable pseudo abels have been adequately corrected. Extensive experiments on LibriSpeech show that these filtered samples enable the refined model to yield more correct predictions, leading to better ASR performances under various experimental settings.
Towards Privacy Engineering for Real-Time Analytics in the Human-Centered Internet of Things
Venue: arXiv cs.DC 2022
Year: 2022

Big data applications offer smart solutions to many urgent societal challenges, such as health care, traffic coordination, energy management, etc. The basic premise for these applications is "the more data the better". The focus often lies on sensing infrastructures in the public realm that produce an ever-increasing amount of data. Yet, any smartphone and smartwatch owner could be a continuous source of valuable data and contribute to many useful big data applications. However, such data can reveal a lot of sensitive information, like the current location or the heart rate of the owner of such devices. Protection of personal data is important in our society and for example manifested in the EU General Data Protection Regulation (GDPR). However, privacy protection and useful big data applications are hard to bring together, particularly in the human-centered IoT. Implementing proper privacy protection requires skills that are typically not in the focus of data analysts and big data developers. Thus, many individuals tend to share none of their data if in doubt whether it will be properly protected. There exist excellent privacy solutions between the "all or nothing" approach. For example, instead of continuously publishing the current location of individuals one might aggregate this data and only publish information of how many individuals are in a certain area of the city. Thus, personal data is not revealed, while useful information for certain applications like traffic coordination is retained. The goal of the Parrot project is to provide tools for real-time data analysis applications that leverage this "middle ground". Data analysts should only be required to specify their data needs, and end-users can select the privacy requirements for their data as well as the applications and end-users they want to share their data with.
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer
Venue: arXiv cs.LG 2022
Year: 2022

Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios, the downstream task might be biased with respect to the target label distribution. This in turn moves the learned fine-tuned model posterior away from the initial (label) bias-free self-supervised model posterior. In this work, we re-interpret SSL fine-tuning under the lens of Bayesian continual learning and consider regularization through the Elastic Weight Consolidation (EWC) framework. We demonstrate that self-regularization against an initial SSL backbone improves worst sub-group performance in Waterbirds by 5% and Celeb-A by 2% when using the ViT-B/16 architecture. Furthermore, to help simplify the use of EWC with SSL, we pre-compute and publicly release the Fisher Information Matrix (FIM), evaluated with 10,000 ImageNet-1K variates evaluated on large modern SSL architectures including ViT-B/16 and ResNet50 trained with DINO.
Radically Lower Data-Labeling Costs for Visually Rich Document Extraction Models
Venue: arXiv cs.CL 2022
Year: 2022

A key bottleneck in building automatic extraction models for visually rich documents like invoices is the cost of acquiring the several thousand high-quality labeled documents that are needed to train a model with acceptable accuracy. We propose Selective Labeling to simplify the labeling task to provide "yes/no" labels for candidate extractions predicted by a model trained on partially labeled documents. We combine this with a custom active learning strategy to find the predictions that the model is most uncertain about. We show through experiments on document types drawn from 3 different domains that selective labeling can reduce the cost of acquiring labeled data by $10\times$ with a negligible loss in accuracy.
System Demo: Tool and Infrastructure for Offensive Language Error Analysis (OLEA) in English
Venue: arXiv cs.CL 2022
Year: 2022

The automatic detection of offensive language is a pressing societal need. Many systems perform well on explicit offensive language but struggle to detect more complex, nuanced, or implicit cases of offensive and hateful language. OLEA is an open-source Python library that provides easy-to-use tools for error analysis in the context of detecting offensive language in English. OLEA also provides an infrastructure for re-distribution of new datasets and analysis methods requiring very little coding.
Evaluation of Categorical Generative Models -- Bridging the Gap Between Real and Synthetic Data
Venue: arXiv cs.LG 2022
Year: 2022

The machine learning community has mainly relied on real data to benchmark algorithms as it provides compelling evidence of model applicability. Evaluation on synthetic datasets can be a powerful tool to provide a better understanding of a model's strengths, weaknesses, and overall capabilities. Gaining these insights can be particularly important for generative modeling as the target quantity is completely unknown. Multiple issues related to the evaluation of generative models have been reported in the literature. We argue those problems can be avoided by an evaluation based on ground truth. General criticisms of synthetic experiments are that they are too simplified and not representative of practical scenarios. As such, our experimental setting is tailored to a realistic generative task. We focus on categorical data and introduce an appropriately scalable evaluation method. Our method involves tasking a generative model to learn a distribution in a high-dimensional setting. We then successively bin the large space to obtain smaller probability spaces where meaningful statistical tests can be applied. We consider increasingly large probability spaces, which correspond to increasingly difficult modeling tasks and compare the generative models based on the highest task difficulty they can reach before being detected as being too far from the ground truth. We validate our evaluation procedure with synthetic experiments on both synthetic generative models and current state-of-the-art categorical generative models.
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE
Venue: arXiv cs.CL 2022
Year: 2022

Figurative language (e.g., "he flew like the wind") is challenging to understand, as it is hard to tell what implicit information is being conveyed from the surface form alone. We hypothesize that to perform this task well, the reader needs to mentally elaborate the scene being described to identify a sensible meaning of the language. We present DREAM-FLUTE, a figurative language understanding system that does this, first forming a "mental model" of situations described in a premise and hypothesis before making an entailment/contradiction decision and generating an explanation. DREAM-FLUTE uses an existing scene elaboration model, DREAM, for constructing its "mental model." In the FigLang2022 Shared Task evaluation, DREAM-FLUTE achieved (joint) first place (Acc@60=63.3%), and can perform even better with ensemble techniques, demonstrating the effectiveness of this approach. More generally, this work suggests that adding a reflective component to pretrained language models can improve their performance beyond standard fine-tuning (3.3% improvement in Acc@60).
DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention
Venue: arXiv cs.CV 2022
Year: 2022

Vision-and-language (V-L) tasks require the system to understand both vision content and natural language, thus learning fine-grained joint representations of vision and language (a.k.a. V-L representations) is of paramount importance. Recently, various pre-trained V-L models are proposed to learn V-L representations and achieve improved results in many tasks. However, the mainstream models process both vision and language inputs with the same set of attention matrices. As a result, the generated V-L representations are entangled in one common latent space. To tackle this problem, we propose DiMBERT (short for Disentangled Multimodal-Attention BERT), which is a novel framework that applies separated attention spaces for vision and language, and the representations of multi-modalities can thus be disentangled explicitly. To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format. In this manner, visual concepts help to bridge the gap between the two modalities. We pre-train DiMBERT on a large amount of image-sentence pairs on two tasks: bidirectional language modeling and sequence-to-sequence language modeling. After pre-train, DiMBERT is further fine-tuned for the downstream tasks. Experiments show that DiMBERT sets new state-of-the-art performance on three tasks (over four datasets), including both generation tasks (image captioning and visual storytelling) and classification tasks (referring expressions). The proposed DiM (short for Disentangled Multimodal-Attention) module can be easily incorporated into existing pre-trained V-L models to boost their performance, up to a 5% increase on the representative task. Finally, we conduct a systematic analysis and demonstrate the effectiveness of our DiM and the introduced visual concepts.
Gender Bias in Computing
Venue: arXiv cs.CY 2022
Year: 2022

This paper examines the historical dimension of gender bias in the US computing workforce. It offers new quantitative data on the computing workforce prior to the availability of US Census data in the 1970s. Computer user groups (including SHARE, Inc., and the Mark IV software user group) are taken as a cross-section of the computing workforce. A novel method of gender analysis is developed to estimate women's and men's participation in computing beginning in the 1950s. The data presented here are consistent with well-known NSF statistics that show computer science undergraduate programs enrolling increasing numbers of women students during 1965-1985. These findings challenge the 'making programming masculine' thesis, and serve to correct the unrealistically high figures often cited for women's participation in early computer programming. Gender bias in computing today is traced not to 1960s professionalization but to cultural changes in the 1980s and beyond.
Robust Boosting Forests with Richer Deep Feature Hierarchy
Venue: arXiv cs.CV 2022
Year: 2022

We propose a robust variant of boosting forest to the various adversarial defense methods, and apply it to enhance the robustness of the deep neural network. We retain the deep network architecture, weights, and middle layer features, then install gradient boosting forest to select the features from each layer of the deep network, and predict the target. For training each decision tree, we propose a novel conservative and greedy trade-off, with consideration for less misprediction instead of pure gain functions, therefore being suboptimal and conservative. We actively increase tree depth to remedy the accuracy with splits in more features, being more greedy in growing tree depth. We propose a new task on 3D face model, whose robustness has not been carefully studied, despite the great security and privacy concerns related to face analytics. We tried a simple attack method on a pure convolutional neural network (CNN) face shape estimator, making it degenerate to only output average face shape with invisible perturbation. Our conservative-greedy boosting forest (CGBF) on face landmark datasets showed a great improvement over original pure deep learning methods under the adversarial attacks.
Region of Interest Detection in Melanocytic Skin Tumor Whole Slide Images
Venue: arXiv cs.CV 2022
Year: 2022

Automated region of interest detection in histopathological image analysis is a challenging and important topic with tremendous potential impact on clinical practice. The deep-learning methods used in computational pathology help us to reduce costs and increase the speed and accuracy of regions of interest detection and cancer diagnosis. In this work, we propose a patch-based region of interest detection method for melanocytic skin tumor whole-slide images. We work with a dataset that contains 165 primary melanomas and nevi Hematoxylin and Eosin whole-slide images and build a deep-learning method. The proposed method performs well on a hold-out test data set including five TCGA-SKCM slides (accuracy of 93.94\% in slide classification task and intersection over union rate of 41.27\% in the region of interest detection task), showing the outstanding performance of our model on melanocytic skin tumor. Even though we test the experiments on the skin tumor dataset, our work could also be extended to other medical image detection problems, such as various tumors' classification and prediction, to help and benefit the clinical evaluation and diagnosis of different tumors.
MinUn: Accurate ML Inference on Microcontrollers
Venue: arXiv cs.LG 2022
Year: 2022

Running machine learning inference on tiny devices, known as TinyML, is an emerging research area. This task requires generating inference code that uses memory frugally, a task that standard ML frameworks are ill-suited for. A deployment framework for TinyML must be a) parametric in the number representation to take advantage of the emerging representations like posits, b) carefully assign high-precision to a few tensors so that most tensors can be kept in low-precision while still maintaining model accuracy, and c) avoid memory fragmentation. We describe MinUn, the first TinyML framework that holistically addresses these issues to generate efficient code for ARM microcontrollers (e.g., Arduino Uno, Due and STM32H747) that outperforms the prior TinyML frameworks.
Strong Lottery Ticket Hypothesis with $\varepsilon$--perturbation
Venue: arXiv cs.LG 2022
Year: 2022

The strong Lottery Ticket Hypothesis (LTH) claims the existence of a subnetwork in a sufficiently large, randomly initialized neural network that approximates some target neural network without the need of training. We extend the theoretical guarantee of the strong LTH literature to a scenario more similar to the original LTH, by generalizing the weight change in the pre-training step to some perturbation around initialization. In particular, we focus on the following open questions: By allowing an $\varepsilon$-scale perturbation on the random initial weights, can we reduce the over-parameterization requirement for the candidate network in the strong LTH? Furthermore, does the weight change by SGD coincide with a good set of such perturbation? We answer the first question by first extending the theoretical result on subset sum to allow perturbation on the candidates. Applying this result to the neural network setting, we show that such $\varepsilon$-perturbation reduces the over-parameterization requirement of the strong LTH. To answer the second question, we show via experiments that the perturbed weight achieved by the projected SGD shows better performance under the strong LTH pruning.
Design of non-diagonal stiffness matrix for assembly task
Venue: arXiv cs.RO 2022
Year: 2022

Compliance control is an increasingly employed technique used in the robotic field. It is known that various mechanical properties can be reproduced depending on the design of the stiffness matrix, but the design theory that takes advantage of this high degree of design freedom has not been elucidated. This paper, therefore, discusses the non-diagonal elements of the stiffness matrix. We proposed a design method according to the conditions required for achieving stable motion. Additionally, we analyzed the displacement induced by the non-diagonal elements in response to an external force and found that to obtain stable contact with a symmetric matrix, the matrix should be positive definite, i.e., all eigenvalues must be positive, however its parameter design is complicated. In this study, we focused on the use of asymmetric matrices in compliance control and showed that the design of eigenvalues can be simplified by using a triangular matrix. This approach expands the range of the stiffness design and enhances the ability of the compliance control to induce motion. We conducted experiments using the stiffness matrix and confirmed that assembly could be achieved without complicated trajectory planning.
BEPHAP: A Blockchain-Based Efficient Privacy-Preserving Handover Authentication Protocol with Key Agreement for Internet of Vehicles
Venue: arXiv cs.CR 2022
Year: 2022

The Internet of Vehicles (IoV) can significantly improve transportation efficiency and ensure traffic safety. Authentication is regarded as the fundamental defense line against attacks in IoV. However, the state-of-the-art approaches suffer from several drawbacks, including bottlenecks of the single cloud server model, high computational overhead of operations, excessive trust in cloud servers and roadside units (RSUs), and leakage of vehicle trajectory privacy. In this paper, BEPHAP, a Blockchain-based Efficient Privacy-preserving Handover Authentication Protocol with key agreement for internet of vehicles, is introduced to address these problems. BEPHAP achieves anonymous cross-domain mutual handover authentication with key agreement based on the tamper-proof blockchain, symmetric cryptography, and the chameleon hash function under a security model that cloud servers and RSUs may launch attacks. BEPHAP is particularly well suited for IoV since it allows vehicles only need to perform lightweight cryptographic operations during the authentication phase. BEPHAP also achieves data confidentiality, unlinkability, traceability, non-repudiation, non-frameability, and key escrow freeness. Formal verification based on ProVerif and formal security proofs based on the BAN logic indicates that BEPHAP is resistant to various typical attacks, such as man-in-the-middle attacks, impersonation attacks, and replay attacks. Performance analysis demonstrates that BEPHAP surpasses existing works in both computation and communication efficiencies. And the message loss rate remains 0 at 5000 requests per second, which meets the requirement of IoV.
Empirical Evaluation of Post-Training Quantization Methods for Language Tasks
Venue: arXiv cs.CL 2022
Year: 2022

Transformer-based architectures like BERT have achieved great success in a wide range of Natural Language tasks. Despite their decent performance, the models still have numerous parameters and high computational complexity, impeding their deployment in resource-constrained environments. Post-Training Quantization (PTQ), which enables low-bit computations without extra training, could be a promising tool. In this work, we conduct an empirical evaluation of three PTQ methods on BERT-Base and BERT-Large: Linear Quantization (LQ), Analytical Clipping for Integer Quantization (ACIQ), and Outlier Channel Splitting (OCS). OCS theoretically surpasses the others in minimizing the Mean Square quantization Error and avoiding distorting the weights' outliers. That is consistent with the evaluation results of most language tasks of GLUE benchmark and a reading comprehension task, SQuAD. Moreover, low-bit quantized BERT models could outperform the corresponding 32-bit baselines on several small language tasks, which we attribute to the alleviation of over-parameterization. We further explore the limit of quantization bit and show that OCS could quantize BERT-Base and BERT-Large to 3-bits and retain 98% and 96% of the performance on the GLUE benchmark accordingly. Moreover, we conduct quantization on the whole BERT family, i.e., BERT models in different configurations, and comprehensively evaluate their performance on the GLUE benchmark and SQuAD, hoping to provide valuable guidelines for their deployment in various computation environments.