OpenRaiser/Intern-Atlas huggingface dataset:
A Cohesive Distillation Architecture for Neural Language Models
Venue: arXiv cs.CL 2023
Year: 2023

A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size, which prevents research groups without a necessary hardware infrastructure from participating in the development process. This study investigates methods for Knowledge Distillation (KD) to provide efficient alternatives to large-scale models. In this context, KD means extracting information about language encoded in a Neural Network and Lexical Knowledge Databases. We developed two methods to test our hypothesis that efficient architectures can gain knowledge from LMs and extract valuable information from lexical sources. First, we present a technique to learn confident probability distribution for Masked Language Modeling by prediction weighting of multiple teacher networks. Second, we propose a method for Word Sense Disambiguation (WSD) and lexical KD that is general enough to be adapted to many LMs. Our results show that KD with multiple teachers leads to improved training convergence. When using our lexical pre-training method, LM characteristics are not lost, leading to increased performance in Natural Language Understanding (NLU) tasks over the state-of-the-art while adding no parameters. Moreover, the improved semantic understanding of our model increased the task performance beyond WSD and NLU in a real-problem scenario (Plagiarism Detection). This study suggests that sophisticated training methods and network architectures can be superior over scaling trainable parameters. On this basis, we suggest the research area should encourage the development and use of efficient models and rate impacts resulting from growing LM size equally against task performance.
Architecting Safer Autonomous Aviation Systems
Venue: arXiv cs.SE 2023
Year: 2023

The aviation literature gives relatively little guidance to practitioners about the specifics of architecting systems for safety, particularly the impact of architecture on allocating safety requirements, or the relative ease of system assurance resulting from system or subsystem level architectural choices. As an exemplar, this paper considers common architectural patterns used within traditional aviation systems and explores their safety and safety assurance implications when applied in the context of integrating artificial intelligence (AI) and machine learning (ML) based functionality. Considering safety as an architectural property, we discuss both the allocation of safety requirements and the architectural trade-offs involved early in the design lifecycle. This approach could be extended to other assured properties, similar to safety, such as security. We conclude with a discussion of the safety considerations that emerge in the context of candidate architectural patterns that have been proposed in the recent literature for enabling autonomy capabilities by integrating AI and ML. A recommendation is made for the generation of a property-driven architectural pattern catalogue.
Towards the design of user-centric strategy recommendation systems for collaborative Human-AI tasks
Venue: arXiv cs.IR 2023
Year: 2023

Artificial Intelligence is being employed by humans to collaboratively solve complicated tasks for search and rescue, manufacturing, etc. Efficient teamwork can be achieved by understanding user preferences and recommending different strategies for solving the particular task to humans. Prior work has focused on personalization of recommendation systems for relatively well-understood tasks in the context of e-commerce or social networks. In this paper, we seek to understand the important factors to consider while designing user-centric strategy recommendation systems for decision-making. We conducted a human-subjects experiment (n=60) for measuring the preferences of users with different personality types towards different strategy recommendation systems. We conducted our experiment across four types of strategy recommendation modalities that have been established in prior work: (1) Single strategy recommendation, (2) Multiple similar recommendations, (3) Multiple diverse recommendations, (4) All possible strategies recommendations. While these strategy recommendation schemes have been explored independently in prior work, our study is novel in that we employ all of them simultaneously and in the context of strategy recommendations, to provide us an in-depth overview of the perception of different strategy recommendation systems. We found that certain personality traits, such as conscientiousness, notably impact the preference towards a particular type of system (p < 0.01). Finally, we report an interesting relationship between usability, alignment and perceived intelligence wherein greater perceived alignment of recommendations with one's own preferences leads to higher perceived intelligence (p < 0.01) and higher usability (p < 0.01).
TRACE-Omicron: Policy Counterfactuals to Inform Mitigation of COVID-19 Spread in the United States
Venue: arXiv cs.SI 2023
Year: 2023

The Omicron wave was the largest wave of COVID-19 pandemic to date, more than doubling any other in terms of cases and hospitalizations in the United States. In this paper, we present a large-scale agent-based model of policy interventions that could have been implemented to mitigate the Omicron wave. Our model takes into account the behaviors of individuals and their interactions with one another within a nationally representative population, as well as the efficacy of various interventions such as social distancing, mask wearing, testing, tracing, and vaccination. We use the model to simulate the impact of different policy scenarios and evaluate their potential effectiveness in controlling the spread of the virus. Our results suggest the Omicron wave could have been substantially curtailed via a combination of interventions comparable in effectiveness to extreme and unpopular singular measures such as widespread closure of schools and workplaces, and highlight the importance of early and decisive action.
mmDrive: mmWave Sensing for Live Monitoring and On-Device Inference of Dangerous Driving
Venue: arXiv cs.HC 2023
Year: 2023

Detecting dangerous driving has been of critical interest for the past few years. However, a practical yet minimally intrusive solution remains challenging as existing technologies heavily rely on visual features or physical proximity. With this motivation, we explore the feasibility of purely using mmWave radars to detect dangerous driving behaviors. We first study characteristics of dangerous driving and find some unique patterns of range-doppler caused by 9 typical dangerous driving actions. We then develop a novel Fused-CNN model to detect dangerous driving instances from regular driving and classify 9 different dangerous driving actions. Through extensive experiments with 5 volunteer drivers in real driving environments, we observe that our system can distinguish dangerous driving actions with an average accuracy of > 95%. We also compare our models with existing state-of-the-art baselines to establish their significance.
Benchmarking YOLOv5 and YOLOv7 models with DeepSORT for droplet tracking applications
Venue: arXiv cs.CV 2023
Year: 2023

Tracking droplets in microfluidics is a challenging task. The difficulty arises in choosing a tool to analyze general microfluidic videos to infer physical quantities. The state-of-the-art object detector algorithm You Only Look Once (YOLO) and the object tracking algorithm Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) are customizable for droplet identification and tracking. The customization includes training YOLO and DeepSORT networks to identify and track the objects of interest. We trained several YOLOv5 and YOLOv7 models and the DeepSORT network for droplet identification and tracking from microfluidic experimental videos. We compare the performance of the droplet tracking applications with YOLOv5 and YOLOv7 in terms of training time and time to analyze a given video across various hardware configurations. Despite the latest YOLOv7 being 10% faster, the real-time tracking is only achieved by lighter YOLO models on RTX 3070 Ti GPU machine due to additional significant droplet tracking costs arising from the DeepSORT algorithm. This work is a benchmark study for the YOLOv5 and YOLOv7 networks with DeepSORT in terms of the training time and inference time for a custom dataset of microfluidic droplets.
Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficient
Venue: arXiv cs.LG 2023
Year: 2023

A foundational problem in reinforcement learning and interactive decision making is to understand what modeling assumptions lead to sample-efficient learning guarantees, and what algorithm design principles achieve optimal sample complexity. Recently, Foster et al. (2021) introduced the Decision-Estimation Coefficient (DEC), a measure of statistical complexity which leads to upper and lower bounds on the optimal sample complexity for a general class of problems encompassing bandits and reinforcement learning with function approximation. In this paper, we introduce a new variant of the DEC, the Constrained Decision-Estimation Coefficient, and use it to derive new lower bounds that improve upon prior work on three fronts: - They hold in expectation, with no restrictions on the class of algorithms under consideration. - They hold globally, and do not rely on the notion of localization used by Foster et al. (2021). - Most interestingly, they allow the reference model with respect to which the DEC is defined to be improper, establishing that improper reference models play a fundamental role. We provide upper bounds on regret that scale with the same quantity, thereby closing all but one of the gaps between upper and lower bounds in Foster et al. (2021). Our results apply to both the regret framework and PAC framework, and make use of several new analysis and algorithm design techniques that we anticipate will find broader use.
Improved Tradeoffs for Leader Election
Venue: arXiv cs.DC 2023
Year: 2023

We consider leader election in clique networks, where $n$ nodes are connected by point-to-point communication links. For the synchronous clique under simultaneous wake-up, i.e., where all nodes start executing the algorithm in round $1$, we show a tradeoff between the number of messages and the amount of time. More specifically, we show that any deterministic algorithm with a message complexity of $n f(n)$ requires $Ω\left(\frac{\log n}{\log f(n)+1}\right)$ rounds, for $f(n) = Ω(\log n)$. Our result holds even if the node IDs are chosen from a relatively small set of size $Θ(n\log n)$, as we are able to avoid using Ramsey's theorem. We also give an upper bound that improves over the previously-best tradeoff. Our second contribution for the synchronous clique under simultaneous wake-up is to show that $Ω(n\log n)$ is in fact a lower bound on the message complexity that holds for any deterministic algorithm with a termination time $T(n)$. We complement this result by giving a simple deterministic algorithm that achieves leader election in sublinear time while sending only $o(n\log n)$ messages, if the ID space is of at most linear size. We also show that Las Vegas algorithms (that never fail) require $Θ(n)$ messages. For the synchronous clique under adversarial wake-up, we show that $Ω(n^{3/2})$ is a tight lower bound for randomized $2$-round algorithms. Finally, we turn our attention to the asynchronous clique: Assuming adversarial wake-up, we give a randomized algorithm that achieves a message complexity of $O(n^{1 + 1/k})$ and an asynchronous time complexity of $k+8$. For simultaneous wake-up, we translate the deterministic tradeoff algorithm of Afek and Gafni to the asynchronous model, thus partially answering an open problem they pose.
LaTeX, metadata, and publishing workflows
Venue: arXiv cs.DL 2023
Year: 2023

The field of scientific publishing that is served by LaTeX is increasingly dependent on the availability of metadata about publications. We discuss how to use LaTeX classes and BibTeX styles to curate metadata throughout the life cycle of a published article. Our focus is on streamlining and automating much of publishing workflow. We survey the various options and drawbacks of the existing approaches and outline our approach as applied in a new LaTeX style file where we have as main goal to make it easier for authors to specify their metadata only once and use this throughout the entire publishing pipeline. We believe this can help to reduce the cost of publishing, by reducing the amount of human effort required for editing and providing of publication metadata.
Ultrasound Plane Pose Regression: Assessing Generalized Pose Coordinates in the Fetal Brain
Venue: arXiv cs.CV 2023
Year: 2023

In obstetric ultrasound (US) scanning, the learner's ability to mentally build a three-dimensional (3D) map of the fetus from a two-dimensional (2D) US image represents a significant challenge in skill acquisition. We aim to build a US plane localization system for 3D visualization, training, and guidance without integrating additional sensors. This work builds on top of our previous work, which predicts the six-dimensional (6D) pose of arbitrarily oriented US planes slicing the fetal brain with respect to a normalized reference frame using a convolutional neural network (CNN) regression network. Here, we analyze in detail the assumptions of the normalized fetal brain reference frame and quantify its accuracy with respect to the acquisition of transventricular (TV) standard plane (SP) for fetal biometry. We investigate the impact of registration quality in the training and testing data and its subsequent effect on trained models. Finally, we introduce data augmentations and larger training sets that improve the results of our previous work, achieving median errors of 2.97 mm and 6.63 degrees for translation and rotation, respectively.
Brain Model State Space Reconstruction Using an LSTM Neural Network
Venue: arXiv cs.LG 2023
Year: 2023

Objective Kalman filtering has previously been applied to track neural model states and parameters, particularly at the scale relevant to EEG. However, this approach lacks a reliable method to determine the initial filter conditions and assumes that the distribution of states remains Gaussian. This study presents an alternative, data-driven method to track the states and parameters of neural mass models (NMMs) from EEG recordings using deep learning techniques, specifically an LSTM neural network. Approach An LSTM filter was trained on simulated EEG data generated by a neural mass model using a wide range of parameters. With an appropriately customised loss function, the LSTM filter can learn the behaviour of NMMs. As a result, it can output the state vector and parameters of NMMs given observation data as the input. Main Results Test results using simulated data yielded correlations with R squared of around 0.99 and verified that the method is robust to noise and can be more accurate than a nonlinear Kalman filter when the initial conditions of the Kalman filter are not accurate. As an example of real-world application, the LSTM filter was also applied to real EEG data that included epileptic seizures, and revealed changes in connectivity strength parameters at the beginnings of seizures. Significance Tracking the state vector and parameters of mathematical brain models is of great importance in the area of brain modelling, monitoring, imaging and control. This approach has no need to specify the initial state vector and parameters, which is very difficult to do in practice because many of the variables being estimated cannot be measured directly in physiological experiments. This method may be applied using any neural mass model and, therefore, provides a general, novel, efficient approach to estimate brain model variables that are often difficult to measure.
On the Relationship Between Information-Theoretic Privacy Metrics And Probabilistic Information Privacy
Venue: arXiv cs.IT 2023
Year: 2023

Information-theoretic (IT) measures based on $f$-divergences have recently gained interest as a measure of privacy leakage as they allow for trading off privacy against utility using only a single-value characterization. However, their operational interpretations in the privacy context are unclear. In this paper, we relate the notion of probabilistic information privacy (IP) to several IT privacy metrics based on $f$-divergences. We interpret probabilistic IP under both the detection and estimation frameworks and link it to differential privacy, thus allowing a precise operational interpretation of these IT privacy metrics. We show that the $χ^2$-divergence privacy metric is stronger than those based on total variation distance and Kullback-Leibler divergence. Therefore, we further develop a data-driven empirical risk framework based on the $χ^2$-divergence privacy metric and realized using deep neural networks. This framework is agnostic to the adversarial attack model. Empirical experiments demonstrate the efficacy of our approach.
Introducing Expertise Logic into Graph Representation Learning from A Causal Perspective
Venue: arXiv cs.LG 2023
Year: 2023

Benefiting from the injection of human prior knowledge, graphs, as derived discrete data, are semantically dense so that models can efficiently learn the semantic information from such data. Accordingly, graph neural networks (GNNs) indeed achieve impressive success in various fields. Revisiting the GNN learning paradigms, we discover that the relationship between human expertise and the knowledge modeled by GNNs still confuses researchers. To this end, we introduce motivating experiments and derive an empirical observation that the GNNs gradually learn human expertise in general domains. By further observing the ramifications of introducing expertise logic into graph representation learning, we conclude that leading the GNNs to learn human expertise can improve the model performance. Hence, we propose a novel graph representation learning method to incorporate human expert knowledge into GNN models. The proposed method ensures that the GNN model can not only acquire the expertise held by human experts but also engage in end-to-end learning from datasets. Plentiful experiments on the crafted and real-world domains support the consistent effectiveness of the proposed method.
Regular Time-series Generation using SGM
Venue: arXiv cs.LG 2023
Year: 2023

Score-based generative models (SGMs) are generative models that are in the spotlight these days. Time-series frequently occurs in our daily life, e.g., stock data, climate data, and so on. Especially, time-series forecasting and classification are popular research topics in the field of machine learning. SGMs are also known for outperforming other generative models. As a result, we apply SGMs to synthesize time-series data by learning conditional score functions. We propose a conditional score network for the time-series generation domain. Furthermore, we also derive the loss function between the score matching and the denoising score matching in the time-series generation domain. Finally, we achieve state-of-the-art results on real-world datasets in terms of sampling diversity and quality.
Clustering Human Mobility with Multiple Spaces
Venue: arXiv cs.LG 2023
Year: 2023

Human mobility clustering is an important problem for understanding human mobility behaviors (e.g., work and school commutes). Existing methods typically contain two steps: choosing or learning a mobility representation and applying a clustering algorithm to the representation. However, these methods rely on strict visiting orders in trajectories and cannot take advantage of multiple types of mobility representations. This paper proposes a novel mobility clustering method for mobility behavior detection. First, the proposed method contains a permutation-equivalent operation to handle sub-trajectories that might have different visiting orders but similar impacts on mobility behaviors. Second, the proposed method utilizes a variational autoencoder architecture to simultaneously perform clustering in both latent and original spaces. Also, in order to handle the bias of a single latent space, our clustering assignment prediction considers multiple learned latent spaces at different epochs. This way, the proposed method produces accurate results and can provide reliability estimates of each trajectory's cluster assignment. The experiment shows that the proposed method outperformed state-of-the-art methods in mobility behavior detection from trajectories with better accuracy and more interpretability.
Transforming Unstructured Text into Data with Context Rule Assisted Machine Learning (CRAML)
Venue: arXiv cs.CL 2023
Year: 2023

We describe a method and new no-code software tools enabling domain experts to build custom structured, labeled datasets from the unstructured text of documents and build niche machine learning text classification models traceable to expert-written rules. The Context Rule Assisted Machine Learning (CRAML) method allows accurate and reproducible labeling of massive volumes of unstructured text. CRAML enables domain experts to access uncommon constructs buried within a document corpus, and avoids limitations of current computational approaches that often lack context, transparency, and interpetability. In this research methods paper, we present three use cases for CRAML: we analyze recent management literature that draws from text data, describe and release new machine learning models from an analysis of proprietary job advertisement text, and present findings of social and economic interest from a public corpus of franchise documents. CRAML produces document-level coded tabular datasets that can be used for quantitative academic research, and allows qualitative researchers to scale niche classification schemes over massive text data. CRAML is a low-resource, flexible, and scalable methodology for building training data for supervised ML. We make available as open-source resources: the software, job advertisement text classifiers, a novel corpus of franchise documents, and a fully replicable start-to-finish trained example in the context of no poach clauses.
Distributed Non-Interference
Venue: arXiv cs.CR 2023
Year: 2023

Information flow security properties were defined some years ago (see, e.g., the surveys \cite{FG01,Ry01}) in terms of suitable equivalence checking problems. These definitions were provided by using sequential models of computations (e.g., labeled transition systems \cite{GV15}), and interleaving behavioral equivalences (e.g., bisimulation equivalence \cite{Mil89}). More recently, the distributed model of Petri nets has been used to study non-interference in \cite{BG03,BG09,BC15}, but also in these papers an interleaving semantics was used. We argue that in order to capture all the relevant information flows, truly-concurrent behavioral equivalences must be used. In particular, we propose for Petri nets the distributed non-interference property, called DNI, based on {\em branching place bisimilarity} \cite{Gor23b}, which is a sensible, decidable equivalence for finite Petri nets with silent moves. Then we focus our attention on the subclass of Petri nets called {\em finite-state machines}, which can be represented (up to isomorphism) by the simple process algebra CFM \cite{Gor17}. DNI is very easily checkable on CFM processes, as it is compositional, so that it does does not suffer from the state-space explosion problem. Moreover, we show that DNI can be characterized syntactically on CFM by means of a type system.