OpenRaiser/Intern-Atlas huggingface dataset:
Attention Round for Post-Training Quantization
Venue: arXiv cs.LG 2022
Year: 2022

At present, the quantification methods of neural network models are mainly divided into post-training quantization (PTQ) and quantization aware training (QAT). Post-training quantization only need a small part of the data to complete the quantification process, but the performance of its quantitative model is not as good as the quantization aware training. This paper presents a novel quantification method called Attention Round. This method gives parameters w the opportunity to be mapped to all possible quantized values, rather than just the two quantized values nearby w in the process of quantization. The probability of being mapped to different quantified values is negatively correlated with the distance between the quantified values and w, and decay with a Gaussian function. In addition, this paper uses the lossy coding length as a measure to assign bit widths to the different layers of the model to solve the problem of mixed precision quantization, which effectively avoids to solve combinatorial optimization problem. This paper also performs quantitative experiments on different models, the results confirm the effectiveness of the proposed method. For ResNet18 and MobileNetV2, the post-training quantization proposed in this paper only require 1,024 training data and 10 minutes to complete the quantization process, which can achieve quantization performance on par with quantization aware training.
Masked Surfel Prediction for Self-Supervised Point Cloud Learning
Venue: arXiv cs.CV 2022
Year: 2022

Masked auto-encoding is a popular and effective self-supervised learning approach to point cloud learning. However, most of the existing methods reconstruct only the masked points and overlook the local geometry information, which is also important to understand the point cloud data. In this work, we make the first attempt, to the best of our knowledge, to consider the local geometry information explicitly into the masked auto-encoding, and propose a novel Masked Surfel Prediction (MaskSurf) method. Specifically, given the input point cloud masked at a high ratio, we learn a transformer-based encoder-decoder network to estimate the underlying masked surfels by simultaneously predicting the surfel positions (i.e., points) and per-surfel orientations (i.e., normals). The predictions of points and normals are supervised by the Chamfer Distance and a newly introduced Position-Indexed Normal Distance in a set-to-set manner. Our MaskSurf is validated on six downstream tasks under three fine-tuning strategies. In particular, MaskSurf outperforms its closest competitor, Point-MAE, by 1.2\% on the real-world dataset of ScanObjectNN under the OBJ-BG setting, justifying the advantages of masked surfel prediction over masked point cloud reconstruction. Codes will be available at https://github.com/YBZh/MaskSurf.
A Distributed Diffusion Kalman Filter In Multitask Networks
Venue: arXiv cs.MA 2022
Year: 2022

The Distributed Diffusion Kalman Filter (DDKF) algorithm in all its magnitude has earned great attention lately and has shown an elaborate way to address the issue of distributed optimization over networks. Estimation and tracking of a single state vector collectively by nodes have been the point of focus. In reality, however, there are several multi-task-oriented issues where the optimal state vector for each node may not be the same. Its objective is to know many related tasks simultaneously, rather than the typical single-task problems. This work considers sensor networks for distributed multi-task tracking in which individual nodes communicate with its immediate nodes. A diffusion-based distributed multi-task tracking algorithm is developed. This is done by implementing an unsupervised adaptive clustering process, which aids nodes in forming clusters and collaborating on tasks. For distributed target tracking, an adaptive clustering approach, which gives agents the ability to identify and select through adaptive adjustments of combination weights nodes who to collaborate with and who not to in order to estimate the common state vector. This gave rise to an effective level of cooperation for improving state vector estimation accuracy, especially in cases where a cluster's background experience is unknown. To demonstrate the efficiency of our algorithm, computer simulations were conducted. Comparison has been carried out for the Diffusion Kalman Filter multitask with respect to the Adapt then combine (ATC) diffusion schemes utilizing both static and adaptive combination weights. Results showed that the ATC diffusion schemes algorithm has great performance with the adaptive combiners as compared to static combiners.
Dynamic Complementarity Conditions and Whole-Body Trajectory Optimization for Humanoid Robot Locomotion
Venue: arXiv cs.RO 2022
Year: 2022

The paper presents a planner to generate walking trajectories by using the centroidal dynamics and the full kinematics of a humanoid robot. The interaction between the robot and the walking surface is modeled explicitly via new conditions, the \emph{Dynamical Complementarity Constraints}. The approach does not require a predefined contact sequence and generates the footsteps automatically. We characterize the robot control objective via a set of tasks, and we address it by solving an optimal control problem. We show that it is possible to achieve walking motions automatically by specifying a minimal set of references, such as a constant desired center of mass velocity and a reference point on the ground. Furthermore, we analyze how the contact modelling choices affect the computational time. We validate the approach by generating and testing walking trajectories for the humanoid robot iCub.
Privacy-Preserving Synthetic Educational Data Generation
Venue: arXiv cs.CY 2022
Year: 2022

Institutions collect massive learning traces but they may not disclose it for privacy issues. Synthetic data generation opens new opportunities for research in education. In this paper we present a generative model for educational data that can preserve the privacy of participants, and an evaluation framework for comparing synthetic data generators. We show how naive pseudonymization can lead to re-identification threats and suggest techniques to guarantee privacy. We evaluate our method on existing massive educational open datasets.
Revisiting Pretraining Objectives for Tabular Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

Recent deep learning models for tabular data currently compete with the traditional ML models based on decision trees (GBDT). Unlike GBDT, deep models can additionally benefit from pretraining, which is a workhorse of DL for vision and NLP. For tabular problems, several pretraining methods were proposed, but it is not entirely clear if pretraining provides consistent noticeable improvements and what method should be used, since the methods are often not compared to each other or comparison is limited to the simplest MLP architectures. In this work, we aim to identify the best practices to pretrain tabular DL models that can be universally applied to different datasets and architectures. Among our findings, we show that using the object target labels during the pretraining stage is beneficial for the downstream performance and advocate several target-aware pretraining objectives. Overall, our experiments demonstrate that properly performed pretraining significantly increases the performance of tabular DL models, which often leads to their superiority over GBDTs.
Application of Validation Obligations to Security Concerns
Venue: arXiv cs.LO 2022
Year: 2022

Our lives become increasingly dependent on safety- and security-critical systems, so formal techniques are advocated for engineering such systems. One of such techniques is validation obligations that enable formalizing requirements early in development to ensure their correctness. Furthermore, validation obligations help hold requirements consistent in an evolving model and create assurances about the model's completeness. Although initially proposed for safety properties, this paper shows how the technique of validation obligations enables us to also reason about security concerns through an example from the medical domain.
Participatory Design Landscape for the Human-Machine Collaboration, Interaction and Automation at the Frontiers of HCI (PDL 2021)
Venue: arXiv cs.HC 2022
Year: 2022

We propose a one-day transdisciplinary creative workshop in the broad area of HCI focused on multiple opportunities of incorporating participatory design into research and industry practice. This workshop will become a venue to share experiences and novel ideas in this area. At the same time, we will brainstorm and explore frontiers of HCI related to engaging end users in design and development practices of established and emerging ICT solutions often overlooked in terms of co-design. We welcome a wide scope of contributions in HCI which explore sustainable opportunities for participatory design and development practices in the context of interconnected business, social, economic and environmental issues. The contributions ought to explore challenges and opportunities related to co-design at the frontiers of HCI - participatory design of newest and complex technologies, not easily explainable or intuitive, novel collaborative (remote or distributed) approaches to empowering users to prepare them to contribute as well as to engaging them directly in co-design.
Part-of-Speech Tagging of Odia Language Using statistical and Deep Learning-Based Approaches
Venue: arXiv cs.CL 2022
Year: 2022

Automatic Part-of-speech (POS) tagging is a preprocessing step of many natural language processing (NLP) tasks such as name entity recognition (NER), speech processing, information extraction, word sense disambiguation, and machine translation. It has already gained a promising result in English and European languages, but in Indian languages, particularly in Odia language, it is not yet well explored because of the lack of supporting tools, resources, and morphological richness of language. Unfortunately, we were unable to locate an open source POS tagger for Odia, and only a handful of attempts have been made to develop POS taggers for Odia language. The main contribution of this research work is to present a conditional random field (CRF) and deep learning-based approaches (CNN and Bidirectional Long Short-Term Memory) to develop Odia part-of-speech tagger. We used a publicly accessible corpus and the dataset is annotated with the Bureau of Indian Standards (BIS) tagset. However, most of the languages around the globe have used the dataset annotated with Universal Dependencies (UD) tagset. Hence, to maintain uniformity Odia dataset should use the same tagset. So we have constructed a simple mapping from BIS tagset to UD tagset. We experimented with various feature set inputs to the CRF model, observed the impact of constructed feature set. The deep learning-based model includes Bi-LSTM network, CNN network, CRF layer, character sequence information, and pre-trained word vector. Character sequence information was extracted by using convolutional neural network (CNN) and Bi-LSTM network. Six different combinations of neural sequence labelling models are implemented, and their performance measures are investigated. It has been observed that Bi-LSTM model with character sequence feature and pre-trained word vector achieved a significant state-of-the-art result.
A Methodology to Support Automatic Cyber Risk Assessment Review
Venue: arXiv cs.CR 2022
Year: 2022

Cyber risk assessment is a fundamental activity for enhancing the protection of an organization, identifying and evaluating the exposure to cyber threats. Currently, this activity is carried out mainly manually and the identification and correct quantification of risks deeply depend on the experience and confidence of the human assessor. As a consequence, the process is not completely objective and two parallel assessments of the same situation may lead to different results. This paper takes a step in the direction of reducing the degree of subjectivity by proposing a methodology to support risk assessors with an automatic review of the produced assessment. Our methodology starts from a controls-based assessment performed using well-known cybersecurity frameworks (e.g., ISO 27001, NIST) and maps security controls over infrastructural aspects that can be assessed automatically (e.g., ICT devices, organization policies). Exploiting this mapping, the methodology suggests how to identify controls needing revision. The approach has been validated through a case study from the healthcare domain and a set of statistical analyses.
Win-Win Cooperation: Bundling Sequence and Span Models for Named Entity Recognition
Venue: arXiv cs.CL 2022
Year: 2022

For Named Entity Recognition (NER), sequence labeling-based and span-based paradigms are quite different. Previous research has demonstrated that the two paradigms have clear complementary advantages, but few models have attempted to leverage these advantages in a single NER model as far as we know. In our previous work, we proposed a paradigm known as Bundling Learning (BL) to address the above problem. The BL paradigm bundles the two NER paradigms, enabling NER models to jointly tune their parameters by weighted summing each paradigm's training loss. However, three critical issues remain unresolved: When does BL work? Why does BL work? Can BL enhance the existing state-of-the-art (SOTA) NER models? To address the first two issues, we implement three NER models, involving a sequence labeling-based model--SeqNER, a span-based NER model--SpanNER, and BL-NER that bundles SeqNER and SpanNER together. We draw two conclusions regarding the two issues based on the experimental results on eleven NER datasets from five domains. We then apply BL to five existing SOTA NER models to investigate the third issue, consisting of three sequence labeling-based models and two span-based models. Experimental results indicate that BL consistently enhances their performance, suggesting that it is possible to construct a new SOTA NER system by incorporating BL into the current SOTA system. Moreover, we find that BL reduces both entity boundary and type prediction errors. In addition, we compare two commonly used labeling tagging methods as well as three types of span semantic representations.
Sampling from Pre-Images to Learn Heuristic Functions for Classical Planning
Venue: arXiv cs.AI 2022
Year: 2022

We introduce a new algorithm, Regression based Supervised Learning (RSL), for learning per instance Neural Network (NN) defined heuristic functions for classical planning problems. RSL uses regression to select relevant sets of states at a range of different distances from the goal. RSL then formulates a Supervised Learning problem to obtain the parameters that define the NN heuristic, using the selected states labeled with exact or estimated distances to goal states. Our experimental study shows that RSL outperforms, in terms of coverage, previous classical planning NN heuristics functions while requiring two orders of magnitude less training time.
A Case-Study on Variations Observed in Accelerometers Across Devices
Venue: arXiv cs.HC 2022
Year: 2022

Every year we grow more dependent on wearable devices to gather personalized data, such as our movements, heart rate, respiration, etc. To capture this data, devices contain sensors, such as accelerometers and gyroscopes, that are able to measure changes in their surroundings and pass along the information for better informed decisions. Although these sensors should behave similarly in different devices, that is not always the case. In this case study, we analyze accelerometers from three different devices recording the same actions with an aim to determine whether the discrepancies are due to variability within or between devices. We found the most significant variation between devices with different specifications, such as sensitivity and sampling frequency. Nevertheless, variance in the data should be assumed, even if data is gathered from the same person, activity, and type of sensor.
Automatic Scatterplot Design Optimization for Clustering Identification
Venue: arXiv cs.HC 2022
Year: 2022

Scatterplots are among the most widely used visualization techniques. Compelling scatterplot visualizations improve understanding of data by leveraging visual perception to boost awareness when performing specific visual analytic tasks. Design choices in scatterplots, such as graphical encodings or data aspects, can directly impact decision-making quality for low-level tasks like clustering. Hence, constructing frameworks that consider both the perceptions of the visual encodings and the task being performed enables optimizing visualizations to maximize efficacy. In this paper, we propose an automatic tool to optimize the design factors of scatterplots to reveal the most salient cluster structure. Our approach leverages the merge tree data structure to identify the clusters and optimize the choice of subsampling algorithm, sampling rate, marker size, and marker opacity used to generate a scatterplot image. We validate our approach with user and case studies that show it efficiently provides high-quality scatterplot designs from a large parameter space.
A simple normalization technique using window statistics to improve the out-of-distribution generalization on medical images
Venue: arXiv cs.CV 2022
Year: 2022

Since data scarcity and data heterogeneity are prevailing for medical images, well-trained Convolutional Neural Networks (CNNs) using previous normalization methods may perform poorly when deployed to a new site. However, a reliable model for real-world clinical applications should be able to generalize well both on in-distribution (IND) and out-of-distribution (OOD) data (e.g., the new site data). In this study, we present a novel normalization technique called window normalization (WIN) to improve the model generalization on heterogeneous medical images, which is a simple yet effective alternative to existing normalization methods. Specifically, WIN perturbs the normalizing statistics with the local statistics computed on the window of features. This feature-level augmentation technique regularizes the models well and improves their OOD generalization significantly. Taking its advantage, we propose a novel self-distillation method called WIN-WIN for classification tasks. WIN-WIN is easily implemented with twice forward passes and a consistency constraint, which can be a simple extension for existing methods. Extensive experimental results on various tasks (6 tasks) and datasets (24 datasets) demonstrate the generality and effectiveness of our methods.
Parameterized Colorings And Labellings Of Graphs In Topological Coding
Venue: arXiv cs.IT 2022
Year: 2022

The coming quantum computation is forcing us to reexamine the cryptosystems people use. We are applying graph colorings of topological coding to modern information security and future cryptography against supercomputer and quantum computer attacks in the near future. Many of techniques introduced here are associated with many mathematical conjecture and NP-problems. We will introduce a group of W-constraint (k,d)-total colorings and algorithms for realizing these colorings in some kinds of graphs, which are used to make quickly public-keys and private-keys with anti-quantum computing, these (k,d)-total colorings are: graceful (k,d)-total colorings, harmonious (k,d)-total colorings, (k,d)-edge-magic total colorings, (k,d)-graceful-difference total colorings and (k,d)-felicitous-difference total colorings. One of useful tools we used is called Topcode-matrix with elements can be all sorts of things, for example, sets, graphs, number-based strings. Most of parameterized graphic colorings/labelings are defined by Topcode-matrix algebra here. From the application point of view, many of our coloring techniques are given by algorithms and easily converted into programs.
AsNER -- Annotated Dataset and Baseline for Assamese Named Entity recognition
Venue: arXiv cs.CL 2022
Year: 2022

We present the AsNER, a named entity annotation dataset for low resource Assamese language with a baseline Assamese NER model. The dataset contains about 99k tokens comprised of text from the speech of the Prime Minister of India and Assamese play. It also contains person names, location names and addresses. The proposed NER dataset is likely to be a significant resource for deep neural based Assamese language processing. We benchmark the dataset by training NER models and evaluating using state-of-the-art architectures for supervised named entity recognition (NER) such as Fasttext, BERT, XLM-R, FLAIR, MuRIL etc. We implement several baseline approaches with state-of-the-art sequence tagging Bi-LSTM-CRF architecture. The highest F1-score among all baselines achieves an accuracy of 80.69% when using MuRIL as a word embedding method. The annotated dataset and the top performing model are made publicly available.
Differentially Private Stochastic Linear Bandits: (Almost) for Free
Venue: arXiv cs.LG 2022
Year: 2022

In this paper, we propose differentially private algorithms for the problem of stochastic linear bandits in the central, local and shuffled models. In the central model, we achieve almost the same regret as the optimal non-private algorithms, which means we get privacy for free. In particular, we achieve a regret of $\tilde{O}(\sqrt{T}+\frac{1}ε)$ matching the known lower bound for private linear bandits, while the best previously known algorithm achieves $\tilde{O}(\frac{1}ε\sqrt{T})$. In the local case, we achieve a regret of $\tilde{O}(\frac{1}ε{\sqrt{T}})$ which matches the non-private regret for constant $ε$, but suffers a regret penalty when $ε$ is small. In the shuffled model, we also achieve regret of $\tilde{O}(\sqrt{T}+\frac{1}ε)$ %for small $ε$ as in the central case, while the best previously known algorithm suffers a regret of $\tilde{O}(\frac{1}ε{T^{3/5}})$. Our numerical evaluation validates our theoretical results.
AVDDPG: Federated reinforcement learning applied to autonomous platoon control
Venue: arXiv cs.LG 2022
Year: 2022

Since 2016 federated learning (FL) has been an evolving topic of discussion in the artificial intelligence (AI) research community. Applications of FL led to the development and study of federated reinforcement learning (FRL). Few works exist on the topic of FRL applied to autonomous vehicle (AV) platoons. In addition, most FRL works choose a single aggregation method (usually weight or gradient aggregation). We explore FRL's effectiveness as a means to improve AV platooning by designing and implementing an FRL framework atop a custom AV platoon environment. The application of FRL in AV platooning is studied under two scenarios: (1) Inter-platoon FRL (Inter-FRL) where FRL is applied to AVs across different platoons; (2) Intra-platoon FRL (Intra-FRL) where FRL is applied to AVs within a single platoon. Both Inter-FRL and Intra-FRL are applied to a custom AV platooning environment using both gradient and weight aggregation to observe the performance effects FRL can have on AV platoons relative to an AV platooning environment trained without FRL. It is concluded that Intra-FRL using weight aggregation (Intra-FRLWA) provides the best performance for controlling an AV platoon. In addition, we found that weight aggregation in FRL for AV platooning provides increases in performance relative to gradient aggregation. Finally, a performance analysis is conducted for Intra-FRLWA versus a platooning environment without FRL for platoons of length 3, 4 and 5 vehicles. It is concluded that Intra-FRLWA largely out-performs the platooning environment that is trained without FRL.