OpenRaiser/Intern-Atlas huggingface dataset:
Unsupervised Tokenization Learning
Venue: arXiv cs.CL 2022
Year: 2022

In the presented study, we discover that the so-called "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora. We find that different languages require different offshoots of that metric (such as derivative, variance, and "peak values") for successful tokenization. Larger training corpora do not necessarily result in better tokenization quality, while compressing the models by eliminating statistically weak evidence tends to improve performance. The proposed unsupervised tokenization technique provides quality better than or comparable to lexicon-based ones, depending on the language.
Groundhog: Efficient Request Isolation in FaaS
Venue: arXiv cs.CR 2022
Year: 2022

Security is a core responsibility for Function-as-a-Service (FaaS) providers. The prevailing approach has each function execute in its own container to isolate concurrent executions of different functions. However, successive invocations of the same function commonly reuse the runtime state of a previous invocation in order to avoid container cold-start delays when invoking a function. Although efficient, this container reuse has security implications for functions that are invoked on behalf of differently privileged users or administrative domains: bugs in a function's implementation, third-party library, or the language runtime may leak private data from one invocation of the function to subsequent invocations of the same function. Groundhog isolates sequential invocations of a function by efficiently reverting to a clean state, free from any private data, after each invocation. The system exploits two properties of typical FaaS platforms: each container executes at most one function at a time and legitimate functions do not retain state across invocations. This enables Groundhog to efficiently snapshot and restore function state between invocations in a manner that is independent of the programming language/runtime and does not require any changes to existing functions, libraries, language runtimes, or OS kernels. We describe the design of Groundhog and its implementation in OpenWhisk, a popular production-grade open-source FaaS framework. On three existing benchmark suites, Groundhog isolates sequential invocations with modest overhead on end-to-end latency (median: 1.5%, 95p: 7%) and throughput (median: 2.5%, 95p: 49.6%), relative to an insecure baseline that reuses the container and runtime state.
Graph-theoretical approach to robust 3D normal extraction of LiDAR data
Venue: arXiv cs.CV 2022
Year: 2022

Low dimensional primitive feature extraction from LiDAR point clouds (such as planes) forms the basis of majority of LiDAR data processing tasks. A major challenge in LiDAR data analysis arises from the irregular nature of LiDAR data that forces practitioners to either regularize the data using some form of gridding or utilize a triangular mesh such as triangulated irregular network (TIN). While there have been a handful applications using LiDAR data as a connected graph, a principled treatment of utilizing graph-theoretical approach for LiDAR data modelling is still lacking. In this paper, we try to bridge this gap by utilizing graphical approach for normal estimation from LiDAR point clouds. We formulate the normal estimation problem in an optimization framework, where we find the corresponding normal vector for each LiDAR point by utilizing its nearest neighbors and simultaneously enforcing a graph smoothness assumption based on point samples. This is a non-linear constrained convex optimization problem which can then be solved using projected conjugate gradient descent to yield an unique solution. As an enhancement to our optimization problem, we also provide different weighted solutions based on the dot product of the normals and Euclidean distance between the points. In order to assess the performance of our proposed normal extraction method and weighting strategies, we first provide a detailed analysis on repeated randomly generated datasets with four different noise levels and four different tuning parameters. Finally, we benchmark our proposed method against existing state-of-the-art approaches on a large scale synthetic plane extraction dataset. The code for the proposed approach along with the simulations and benchmarking is available at https://github.com/arpan-kusari/graph-plane-extraction-simulation.
Rethinking Streaming Machine Learning Evaluation
Venue: arXiv cs.LG 2022
Year: 2022

While most work on evaluating machine learning (ML) models focuses on computing accuracy on batches of data, tracking accuracy alone in a streaming setting (i.e., unbounded, timestamp-ordered datasets) fails to appropriately identify when models are performing unexpectedly. In this position paper, we discuss how the nature of streaming ML problems introduces new real-world challenges (e.g., delayed arrival of labels) and recommend additional metrics to assess streaming ML performance.
Conditional Supervised Contrastive Learning for Fair Text Classification
Venue: arXiv cs.CL 2022
Year: 2022

Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data. However, the learned representations could potentially lead to performance disparities in downstream tasks, such as increased silencing of underrepresented groups in toxicity comment classification. In light of this challenge, in this work, we study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives, and then propose to use conditional supervised contrastive objectives to learn fair representations for text classification. We conduct experiments on two text datasets to demonstrate the effectiveness of our approaches in balancing the trade-offs between task performance and bias mitigation among existing baselines for text classification. Furthermore, we also show that the proposed methods are stable in different hyperparameter settings.
FedSA: Accelerating Intrusion Detection in Collaborative Environments with Federated Simulated Annealing
Venue: arXiv cs.CR 2022
Year: 2022

Fast identification of new network attack patterns is crucial for improving network security. Nevertheless, identifying an ongoing attack in a heterogeneous network is a non-trivial task. Federated learning emerges as a solution to collaborative training for an Intrusion Detection System (IDS). The federated learning-based IDS trains a global model using local machine learning models provided by federated participants without sharing local data. However, optimization challenges are intrinsic to federated learning. This paper proposes the Federated Simulated Annealing (FedSA) metaheuristic to select the hyperparameters and a subset of participants for each aggregation round in federated learning. FedSA optimizes hyperparameters linked to the global model convergence. The proposal reduces aggregation rounds and speeds up convergence. Thus, FedSA accelerates learning extraction from local models, requiring fewer IDS updates. The proposal assessment shows that the FedSA global model converges in less than ten communication rounds. The proposal requires up to 50% fewer aggregation rounds to achieve approximately 97% accuracy in attack detection than the conventional aggregation approach.
PrivFairFL: Privacy-Preserving Group Fairness in Federated Learning
Venue: arXiv cs.LG 2022
Year: 2022

Group fairness ensures that the outcome of machine learning (ML) based decision making systems are not biased towards a certain group of people defined by a sensitive attribute such as gender or ethnicity. Achieving group fairness in Federated Learning (FL) is challenging because mitigating bias inherently requires using the sensitive attribute values of all clients, while FL is aimed precisely at protecting privacy by not giving access to the clients' data. As we show in this paper, this conflict between fairness and privacy in FL can be resolved by combining FL with Secure Multiparty Computation (MPC) and Differential Privacy (DP). In doing so, we propose a method for training group-fair ML models in cross-device FL under complete and formal privacy guarantees, without requiring the clients to disclose their sensitive attribute values.
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment
Venue: arXiv cs.CL 2022
Year: 2022

Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown that the accuracy and robustness of unsupervised word translation (UWT) can be improved by making use of visual observations, which are universal representations across languages. In this work, we investigate the potential of using not only visual observations but also pretrained language-image models for enabling a more efficient and robust UWT. Specifically, we develop a novel UWT method dubbed Word Alignment using Language-Image Pretraining (WALIP), which leverages visual observations via the shared embedding space of images and texts provided by CLIP models (Radford et al., 2021). WALIP has a two-step procedure. First, we retrieve word pairs with high confidences of similarity, computed using our proposed image-based fingerprints, which define the initial pivot for the word alignment. Second, we apply our robust Procrustes algorithm to estimate the linear mapping between two embedding spaces, which iteratively corrects and refines the estimated alignment. Our extensive experiments show that WALIP improves upon the state-of-the-art performance of bilingual word alignment for a few language pairs across different word embeddings and displays great robustness to the dissimilarity of language pairs or training corpora for two word embeddings.
Visual and unplugged coding lessons with smart toys
Venue: arXiv cs.CY 2022
Year: 2022

Our Computer science k-12 education research group and the educational toy company Quercetti have been collaborating together to design and manufacture toys that help stimulate and consolidate so-called computational thinking. This approach is inspired by methods already consolidated in the literature and widespread worldwide such as the Bebras tasks and CS-Unplugged. This paper describes two smart toys, their design process, educational activities that can be proposed by teachers exploiting the two toys, the evaluation's results from some teachers, and finally feedback and reviews from buyers. The main activities proposed by these toys leverage visual coding through small colored physical items (e.g., pegs and balls) to deliver the unplugged activities to young users.
A Natural Language Processing Pipeline for Detecting Informal Data References in Academic Literature
Venue: arXiv cs.DL 2022
Year: 2022

Discovering authoritative links between publications and the datasets that they use can be a labor-intensive process. We introduce a natural language processing pipeline that retrieves and reviews publications for informal references to research datasets, which complements the work of data librarians. We first describe the components of the pipeline and then apply it to expand an authoritative bibliography linking thousands of social science studies to the data-related publications in which they are used. The pipeline increases recall for literature to review for inclusion in data-related collections of publications and makes it possible to detect informal data references at scale. We contribute (1) a novel Named Entity Recognition (NER) model that reliably detects informal data references and (2) a dataset connecting items from social science literature with datasets they reference. Together, these contributions enable future work on data reference, data citation networks, and data reuse.
A Dataset for Sentence Retrieval for Open-Ended Dialogues
Venue: arXiv cs.IR 2022
Year: 2022

We address the task of sentence retrieval for open-ended dialogues. The goal is to retrieve sentences from a document corpus that contain information useful for generating the next turn in a given dialogue. Prior work on dialogue-based retrieval focused on specific types of dialogues: either conversational QA or conversational search. To address a broader scope of this task where any type of dialogue can be used, we constructed a dataset that includes open-ended dialogues from Reddit, candidate sentences from Wikipedia for each dialogue and human annotations for the sentences. We report the performance of several retrieval baselines, including neural retrieval models, over the dataset. To adapt neural models to the types of dialogues in the dataset, we explored an approach to induce a large-scale weakly supervised training data from Reddit. Using this training set significantly improved the performance over training on the MS MARCO dataset.
A Mechanized Proof of Bounded Convergence Time for the Distributed Perimeter Surveillance System (DPSS) Algorithm A
Venue: arXiv cs.LO 2022
Year: 2022

The decentralized perimeter surveillance system (DPSS) seeks to provide a decentralized protocol for evenly distributing surveillance of a perimeter over time across an ensemble of unmanned aerial vehicles (UAVs) whose members may communicate only when in close proximity to each other. The protocol must also converge to an even distribution of the perimeter in bounded time. Two versions of the DPSS protocol presented in the original paper seem to converge in bounded time but only informal proofs and arguments are given. A later application of model checking to these protocols found an error in one of the key lemmas, invalidating the informal proof for one and casting doubt on the other. Therefore, a new hand proof of the convergence time for the simpler version of the DPSS protocol or algorithm, Algorithm A or DPSS-A, was developed by Jeremy Avigad and Floris van Doorn. This paper describes a mechanization of that hand proof in the logic of ACL2 and discusses three specific ACL2 utilities that proved useful for expressing and reasoning about the DPSS model.
From Easy to Hard: Two-stage Selector and Reader for Multi-hop Question Answering
Venue: arXiv cs.CL 2022
Year: 2022

Multi-hop question answering (QA) is a challenging task requiring QA systems to perform complex reasoning over multiple documents and provide supporting facts together with the exact answer. Existing works tend to utilize graph-based reasoning and question decomposition to obtain the reasoning chain, which inevitably introduces additional complexity and cumulative error to the system. To address the above issue, we propose a simple yet effective novel framework, From Easy to Hard (FE2H), to remove distracting information and obtain better contextual representations for the multi-hop QA task. Inspired by the iterative document selection process and the progressive learning custom of humans, FE2H divides both the document selector and reader into two stages following an easy-to-hard manner. Specifically, we first select the document most relevant to the question and then utilize the question together with this document to select other pertinent documents. As for the QA phase, our reader is first trained on a single-hop QA dataset and then transferred into the multi-hop QA task. We comprehensively evaluate our model on the popular multi-hop QA benchmark HotpotQA. Experimental results demonstrate that our method ourperforms all other methods in the leaderboard of HotpotQA (distractor setting).
Deep Learning-based automated classification of Chinese Speech Sound Disorders
Venue: arXiv cs.SD 2022
Year: 2022

This article describes a system for analyzing acoustic data to assist in the diagnosis and classification of children's speech sound disorders (SSDs) using a computer. The analysis concentrated on identifying and categorizing four distinct types of Chinese SSDs. The study collected and generated a speech corpus containing 2540 stopping, backing, final consonant deletion process (FCDP), and affrication samples from 90 children aged 3--6 years with normal or pathological articulatory features. Each recording was accompanied by a detailed diagnostic annotation by two speech-language pathologists (SLPs). Classification of the speech samples was accomplished using three well-established neural network models for image classification. The feature maps were created using three sets of Mel-frequency cepstral coefficients (MFCC) parameters extracted from speech sounds and aggregated into a three-dimensional data structure as model input. We employed six techniques for data augmentation to augment the available dataset while avoiding overfitting. The experiments examine the usability of four different categories of Chinese phrases and characters. Experiments with different data subsets demonstrate the system's ability to accurately detect the analyzed pronunciation disorders. The best multi-class classification using a single Chinese phrase achieves an accuracy of 74.4~percent.
Efficient LSM-Tree Key-Value Data Management on Hybrid SSD/HDD Zoned Storage
Venue: arXiv cs.PF 2022
Year: 2022

Zoned storage devices, such as zoned namespace (ZNS) solid-state drives (SSDs) and host-managed shingled magnetic recording (HM-SMR) hard-disk drives (HDDs), expose interfaces for host-level applications to support fine-grained, high-performance storage management. Combining ZNS SSDs and HM-SMR HDDs into a unified hybrid storage system is a natural direction to scale zoned storage at low cost, yet how to effectively incorporate zoned storage awareness into hybrid storage is a non-trivial issue. We make a case for key-value (KV) stores based on log-structured merge trees (LSM-trees) as host-level applications, and present HHZS, a middleware system that bridges an LSM-tree KV store with hybrid zoned storage devices based on hints. HHZS leverages hints issued by the flushing, compaction, and caching operations of the LSM-tree KV store to manage KV objects in placement, migration, and caching in hybrid ZNS SSD and HM-SMR HDD zoned storage. Experiments show that our HHZS prototype, when running on real ZNS SSD and HM-SMR HDD devices, achieves the highest throughput compared with all baselines under various settings.
Wireless Ad Hoc Federated Learning: A Fully Distributed Cooperative Machine Learning
Venue: arXiv cs.LG 2022
Year: 2022

Privacy-sensitive data is stored in autonomous vehicles, smart devices, or sensor nodes that can move around with making opportunistic contact with each other. Federation among such nodes was mainly discussed in the context of federated learning with a centralized mechanism in many works. However, because of multi-vendor issues, those nodes do not want to rely on a specific server operated by a third party for this purpose. In this paper, we propose a wireless ad hoc federated learning (WAFL) -- a fully distributed cooperative machine learning organized by the nodes physically nearby. WAFL can develop generalized models from Non-IID datasets stored in distributed nodes locally by exchanging and aggregating them with each other over opportunistic node-to-node contacts. In our benchmark-based evaluation with various opportunistic networks, WAFL has achieved higher accuracy of 94.8-96.3% than the self-training case of 84.7%. All our evaluation results show that WAFL can train and converge the model parameters from highly-partitioned Non-IID datasets over opportunistic networks without any centralized mechanisms.
Attributing AUC-ROC to Analyze Binary Classifier Performance
Venue: arXiv cs.LG 2022
Year: 2022

Area Under the Receiver Operating Characteristic Curve (AUC-ROC) is a popular evaluation metric for binary classifiers. In this paper, we discuss techniques to segment the AUC-ROC along human-interpretable dimensions. AUC-ROC is not an additive/linear function over the data samples, therefore such segmenting the overall AUC-ROC is different from tabulating the AUC-ROC of data segments. To segment the overall AUC-ROC, we must first solve an \emph{attribution} problem to identify credit for individual examples. We observe that AUC-ROC, though non-linear over examples, is linear over \emph{pairs} of examples. This observation leads to a simple, efficient attribution technique for examples (example attributions), and for pairs of examples (pair attributions). We automatically slice these attributions using decision trees by making the tree predict the attributions; we use the notion of honest estimates along with a t-test to mitigate false discovery. Our experiments with the method show that an inferior model can outperform a superior model (trained to optimize a different training objective) on the inferior model's own training objective, a manifestation of Goodhart's Law. In contrast, AUC attributions enable a reasonable comparison. Example attributions can be used to slice this comparison. Pair attributions are used to categorize pairs of items -- one positively labeled and one negatively -- that the model has trouble separating. These categories identify the decision boundary of the classifier and the headroom to improve AUC.
Symbolic Expression Transformer: A Computer Vision Approach for Symbolic Regression
Venue: arXiv cs.CV 2022
Year: 2022

Symbolic Regression (SR) is a type of regression analysis to automatically find the mathematical expression that best fits the data. Currently, SR still basically relies on various searching strategies so that a sample-specific model is required to be optimized for every expression, which significantly limits the model's generalization and efficiency. Inspired by the fact that human beings can infer a mathematical expression based on the curve of it, we propose Symbolic Expression Transformer (SET), a sample-agnostic model from the perspective of computer vision for SR. Specifically, the collected data is represented as images and an image caption model is employed for translating images to symbolic expressions. A large-scale dataset without overlap between training and testing sets in the image domain is released. Our results demonstrate the effectiveness of SET and suggest the promising direction of image-based model for solving the challenging SR problem.
Deep Reinforcement Learning for Radio Resource Allocation in NOMA-based Remote State Estimation
Venue: arXiv cs.IT 2022
Year: 2022

Remote state estimation, where many sensors send their measurements of distributed dynamic plants to a remote estimator over shared wireless resources, is essential for mission-critical applications of Industry 4.0. Most of the existing works on remote state estimation assumed orthogonal multiple access and the proposed dynamic radio resource allocation algorithms can only work for very small-scale settings. In this work, we consider a remote estimation system with non-orthogonal multiple access. We formulate a novel dynamic resource allocation problem for achieving the minimum overall long-term average estimation mean-square error. Both the estimation quality state and the channel quality state are taken into account for decision making at each time. The problem has a large hybrid discrete and continuous action space for joint channel assignment and power allocation. We propose a novel action-space compression method and develop an advanced deep reinforcement learning algorithm to solve the problem. Numerical results show that our algorithm solves the resource allocation problem effectively, presents much better scalability than the literature, and provides significant performance gain compared to some benchmarks.