OpenRaiser/Intern-Atlas huggingface dataset:
On the Rényi Cross-Entropy
Venue: arXiv cs.IT 2022
Year: 2022

The Rényi cross-entropy measure between two distributions, a generalization of the Shannon cross-entropy, was recently used as a loss function for the improved design of deep learning generative adversarial networks. In this work, we examine the properties of this measure and derive closed-form expressions for it when one of the distributions is fixed and when both distributions belong to the exponential family. We also analytically determine a formula for the cross-entropy rate for stationary Gaussian processes and for finite-alphabet Markov sources.
CoAP-DoS: An IoT Network Intrusion Dataset
Venue: arXiv cs.CR 2022
Year: 2022

The need for secure Internet of Things (IoT) devices is growing as IoT devices are becoming more integrated into vital networks. Many systems rely on these devices to remain available and provide reliable service. Denial of service attacks against IoT devices are a real threat due to the fact these low power devices are very susceptible to denial-of-service attacks. Machine learning enabled network intrusion detection systems are effective at identifying new threats, but they require a large amount of data to work well. There are many network traffic data sets but very few that focus on IoT network traffic. Within the IoT network data sets there is a lack of CoAP denial of service data. We propose a novel data set covering this gap. We develop a new data set by collecting network traffic from real CoAP denial of service attacks and compare the data on multiple different machine learning classifiers. We show that the data set is effective on many classifiers.
EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering
Venue: arXiv cs.CV 2022
Year: 2022

The availability of clean and diverse labeled data is a major roadblock for training models on complex tasks such as visual question answering (VQA). The extensive work on large vision-and-language models has shown that self-supervised learning is effective for pretraining multimodal interactions. In this technical report, we focus on visual representations. We review and evaluate self-supervised methods to leverage unlabeled images and pretrain a model, which we then fine-tune on a custom VQA task that allows controlled evaluation and diagnosis. We compare energy-based models (EBMs) with contrastive learning (CL). While EBMs are growing in popularity, they lack an evaluation on downstream tasks. We find that both EBMs and CL can learn representations from unlabeled images that enable training a VQA model on very little annotated data. In a simple setting similar to CLEVR, we find that CL representations also improve systematic generalization, and even match the performance of representations from a larger, supervised, ImageNet-pretrained model. However, we find EBMs to be difficult to train because of instabilities and high variability in their results. Although EBMs prove useful for OOD detection, other results on supervised energy-based training and uncertainty calibration are largely negative. Overall, CL currently seems a preferable option over EBMs.
Lower Bounds on the Error Probability for Invariant Causal Prediction
Venue: arXiv cs.IT 2022
Year: 2022

It is common practice to collect observations of feature and response pairs from different environments. A natural question is how to identify features that have consistent prediction power across environments. The invariant causal prediction framework proposes to approach this problem through invariance, assuming a linear model that is invariant under different environments. In this work, we make an attempt to shed light on this framework by connecting it to the Gaussian multiple access channel problem. Specifically, we incorporate optimal code constructions and decoding methods to provide lower bounds on the error probability. We illustrate our findings by various simulation settings.
Knowledge Distillation of Transformer-based Language Models Revisited
Venue: arXiv cs.CL 2022
Year: 2022

In the past few years, transformer-based pre-trained language models have achieved astounding success in both industry and academia. However, the large model size and high run-time latency are serious impediments to applying them in practice, especially on mobile phones and Internet of Things (IoT) devices. To compress the model, considerable literature has grown up around the theme of knowledge distillation (KD) recently. Nevertheless, how KD works in transformer-based models is still unclear. We tease apart the components of KD and propose a unified KD framework. Through the framework, systematic and extensive experiments that spent over 23,000 GPU hours render a comprehensive analysis from the perspectives of knowledge types, matching strategies, width-depth trade-off, initialization, model size, etc. Our empirical results shed light on the distillation in the pre-train language model and with relative significant improvement over previous state-of-the-arts(SOTA). Finally, we provide a best-practice guideline for the KD in transformer-based models.
Spectral-Loc: Indoor Localization using Light Spectral Information
Venue: arXiv cs.HC 2022
Year: 2022

For indoor settings, we investigate the impact of location on the spectral distribution of the received light, i.e., the intensity of light for different wavelengths. Our investigations confirm that even under the same light source, different locations exhibit slightly different spectral distribution due to reflections from their localised environment containing different materials or colours. By exploiting this observation, we propose Spectral-Loc, a novel indoor localization system that uses light spectral information to identify the location of the device. With spectral sensors finding their way in latest products and applications, such as white balancing in smartphone photography, Spectral-Loc can be readily deployed without requiring any additional hardware or infrastructure. We prototype Spectral-Loc using a commercial-off-the-shelf light spectral sensor, AS7265x, which can measure light intensity over 18 different wavelength sub-bands. We benchmark the localisation accuracy of Spectral-Loc against the conventional light intensity sensors that provide only a single intensity value. Our evaluations over two different indoor spaces, a meeting room and a large office space, demonstrate that use of light spectral information significantly reduces the localization error for the different percentiles.
Distributed Edge-based Video Analytics on the Move
Venue: arXiv cs.DC 2022
Year: 2022

In recent years, we have witnessed an explosive growth of data. Much of this data is video data generated by security cameras, smartphones, and dash cams. The timely analysis of such data is of great practical importance for many emerging applications, such as real-time facial recognition and object detection. In this study, we address the problem of real-time in-situ video analytics with dash cam videos and present EdgeDashAnalytics (EDA), an edge-based system that enables near real-time video analytics using a local network of mobile devices. In particular, it simultaneously processes videos produced by two dash cams of different angles with one or more mobile devices on the move in a near real-time manner. One camera faces outward to capture the view in front of the vehicle, while the other camera faces inward to capture the driver. The outer videos are analysed to detect potential driving hazards, while the inner videos are used to identify driver distractedness. EDA achieves near real-time video analytics using resource-constrained, transient mobile devices by devising and incorporating several optimisations, with a tolerable loss in accuracy. We have implemented EDA as an Android app and evaluated it using two dash cams and several heterogeneous mobile devices with the BDD100K dash cam video dataset (arXiv:1805.04687 [cs.CV]) and the DMD driver monitoring dataset (arXiv:2008.12085 [cs.CV]). Experiment results demonstrate the feasibility of real-time video analytics in terms of turnaround time and energy consumption (or battery usage), using resource-constrained mobile devices on the move.
Optimization-Induced Graph Implicit Nonlinear Diffusion
Venue: arXiv cs.LG 2022
Year: 2022

Due to the over-smoothing issue, most existing graph neural networks can only capture limited dependencies with their inherently finite aggregation layers. To overcome this limitation, we propose a new kind of graph convolution, called Graph Implicit Nonlinear Diffusion (GIND), which implicitly has access to infinite hops of neighbors while adaptively aggregating features with nonlinear diffusion to prevent over-smoothing. Notably, we show that the learned representation can be formalized as the minimizer of an explicit convex optimization objective. With this property, we can theoretically characterize the equilibrium of our GIND from an optimization perspective. More interestingly, we can induce new structural variants by modifying the corresponding optimization objective. To be specific, we can embed prior properties to the equilibrium, as well as introducing skip connections to promote training stability. Extensive experiments show that GIND is good at capturing long-range dependencies, and performs well on both homophilic and heterophilic graphs with nonlinear diffusion. Moreover, we show that the optimization-induced variants of our models can boost the performance and improve training stability and efficiency as well. As a result, our GIND obtains significant improvements on both node-level and graph-level tasks.
Open Problem: Properly learning decision trees in polynomial time?
Venue: arXiv cs.DS 2022
Year: 2022

The authors recently gave an $n^{O(\log\log n)}$ time membership query algorithm for properly learning decision trees under the uniform distribution (Blanc et al., 2021). The previous fastest algorithm for this problem ran in $n^{O(\log n)}$ time, a consequence of Ehrenfeucht and Haussler (1989)'s classic algorithm for the distribution-free setting. In this article we highlight the natural open problem of obtaining a polynomial-time algorithm, discuss possible avenues towards obtaining it, and state intermediate milestones that we believe are of independent interest.
Rate Lifting for Stochastic Process Algebra: Exploiting Structural Properties
Venue: arXiv cs.PF 2022
Year: 2022

This report presents an algorithm for determining the unknown rates in the sequential processes of a Stochastic Process Algebra model, provided that the rates in the combined flat model are given. Such a rate lifting is useful for model reengineering and model repair. Technically, the algorithm works by solving systems of nonlinear equations and, if necessary, adjusting the model`s synchronisation structure without changing its transition system. This report contains the complete pseudo-code of the algorithm. The approach taken by the algorithm exploits some structural properties of Stochastic Process Algebra systems, which are formulated here for the first time and could be very beneficial also in other contexts.
Combinatorial and Algorithmic Aspects of Monadic Stability
Venue: arXiv cs.DM 2022
Year: 2022

Nowhere dense classes of graphs are classes of sparse graphs with rich structural and algorithmic properties, however, they fail to capture even simple classes of dense graphs. Monadically stable classes, originating from model theory, generalize nowhere dense classes and close them under transductions, i.e. transformations defined by colorings and simple first-order interpretations. In this work we aim to extend some combinatorial and algorithmic properties of nowhere dense classes to monadically stable classes of finite graphs. We prove the following results. - In monadically stable classes the Ramsey numbers $R(s,t)$ are bounded from above by $\mathcal{O}(t^{s-1-δ})$ for some $δ>0$, improving the bound $R(s,t)\in \mathcal{O}(t^{s-1}/(\log t)^{s-1})$ known for general graphs and the bounds known for $k$-stable graphs when $s\leq k$. - For every monadically stable class $\mathcal{C}$ and every integer $r$, there exists $δ> 0$ such that every graph $G \in \mathcal{C}$ that contains an $r$-subdivision of the biclique $K_{t,t}$ as a subgraph also contains $K_{t^δ,t^δ}$ as a subgraph. This generalizes earlier results for nowhere dense graph classes. - We obtain a stronger regularity lemma for monadically stable classes of graphs. - Finally, we show that we can compute polynomial kernels for the independent set and dominating set problems in powers of nowhere dense classes. Formerly, only fixed-parameter tractable algorithms were known for these problems on powers of nowhere dense classes.
Effect of Auditory Stimuli on Electroencephalography-based Authentication
Venue: arXiv cs.CR 2022
Year: 2022

Opposed to standard authentication methods based on credentials, biometric-based authentication has lately emerged as a viable paradigm for attaining rapid and secure authentication of users. Among the numerous categories of biometric traits, electroencephalogram (EEG)-based biometrics is recognized as a promising method owing to its unique characteristics. This paper provides an experimental evaluation of the effect of auditory stimuli (AS) on EEG-based biometrics by studying the following features: i) general change in AS-aided EEG-based biometric authentication in comparison with non-AS-aided EEG-based biometric authentication, ii) role of the language of the AS and ii) influence of the conduction method of the AS. Our results show that the presence of an AS can improve authentication performance by 9.27%. Additionally, the performance achieved with an in-ear AS is better than that obtained using a bone-conducting AS. Finally, we verify that performance is independent of the language of the AS. The results of this work provide a step forward towards designing a robust EEG-based authentication system.
SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences
Venue: arXiv cs.AR 2022
Year: 2022

The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence length incurs heavy computational and memory burdens, especially for tasks with long sequences. Existing accelerators face performance degradation in these tasks. To this end, we propose SALO to enable hybrid sparse attention mechanisms for long sequences. SALO contains a data scheduler to map hybrid sparse attention patterns onto hardware and a spatial accelerator to perform the efficient attention computation. We show that SALO achieves 17.66x and 89.33x speedup on average compared to GPU and CPU implementations, respectively, on typical workloads, i.e., Longformer and ViL.
ITLingo Research Initiative in 2022
Venue: arXiv cs.SE 2022
Year: 2022

Several surveys and studies have noticed that cost, and quality problems result from mistakes that occurred in the early phases of the projects, for instance: poor definition of the project vision and respective value for the organization; misalignment between IT and business resources; failures in project management practices; or poor and low-quality technical documentation. These facts have emphasized the need for improving socio-technical disciplines such as project management, enterprise architecture, requirements engineering, or system design. These studies also noted the need to reduce the efforts involved in traditional development processes, for example, by automating some human-intensive and error-prone tasks. This article presents the scientific project we have conducted in these last years named "ITLingo". ITLingo is a research initiative that has proposed new languages, tools, and techniques to support users to improve such practices, mainly related to those disciplines. ITLingo users are IT engineers and managers in multiple roles like project managers, enterprise architects, business analysts, system architects, requirements engineers, or even system developers. The article describes the innovative nature of the activities carried out under the ITLingo umbrella and identifies future and open challenges.
ADARP: A Multi Modal Dataset for Stress and Alcohol Relapse Quantification in Real Life Setting
Venue: arXiv cs.HC 2022
Year: 2022

Stress detection and classification from wearable sensor data is an emerging area of research with significant implications for individuals' physical and mental health. In this work, we introduce a new dataset, ADARP, which contains physiological data and self-report outcomes collected in real-world ambulatory settings involving individuals diagnosed with alcohol use disorders. We describe the user study, present details of the dataset, establish the significant correlation between physiological data and self-reported outcomes, demonstrate stress classification, and make our dataset public to facilitate research.
Using cognitive psychology to understand GPT-3
Venue: arXiv cs.CL 2022
Year: 2022

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models
Venue: arXiv cs.CL 2022
Year: 2022

In Spoken Language Understanding (SLU) the task is to extract important information from audio commands, like the intent of what a user wants the system to do and special entities like locations or numbers. This paper presents a simple method for embedding intents and entities into Finite State Transducers, and, in combination with a pretrained general-purpose Speech-to-Text model, allows building SLU-models without any additional training. Building those models is very fast and only takes a few seconds. It is also completely language independent. With a comparison on different benchmarks it is shown that this method can outperform multiple other, more resource demanding SLU approaches.
The differential spectrum and boomerang spectrum of a class of locally-APN functions
Venue: arXiv cs.IT 2022
Year: 2022

In this paper, we study the boomerang spectrum of the power mapping $F(x)=x^{k(q-1)}$ over ${\mathbb F}_{q^2}$, where $q=p^m$, $p$ is a prime, $m$ is a positive integer and $\gcd(k,q+1)=1$. We first determine the differential spectrum of $F(x)$ and show that $F(x)$ is locally-APN. This extends a result of [IEEE Trans. Inf. Theory 57(12):8127-8137, 2011] from $(p,k)=(2,1)$ to general $(p,k)$. We then determine the boomerang spectrum of $F(x)$ by making use of its differential spectrum, which shows that the boomerang uniformity of $F(x)$ is 4 if $p=2$ and $m$ is odd and otherwise it is 2. Our results not only generalize the results in [Des. Codes Cryptogr. 89:2627-2636, 2021] and [arXiv:2203.00485, 2022] but also extend the example $x^{45}$ over ${\mathbb F}_{2^8}$ in [Des. Codes Cryptogr. 89:2627-2636, 2021] into an infinite class of power mappings with boomerang uniformity 2.
Quantification of Deep Neural Network Prediction Uncertainties for VVUQ of Machine Learning Models
Venue: arXiv cs.LG 2022
Year: 2022

Recent performance breakthroughs in Artificial intelligence (AI) and Machine learning (ML), especially advances in Deep learning (DL), the availability of powerful, easy-to-use ML libraries (e.g., scikit-learn, TensorFlow, PyTorch.), and increasing computational power have led to unprecedented interest in AI/ML among nuclear engineers. For physics-based computational models, Verification, Validation and Uncertainty Quantification (VVUQ) have been very widely investigated and a lot of methodologies have been developed. However, VVUQ of ML models has been relatively less studied, especially in nuclear engineering. In this work, we focus on UQ of ML models as a preliminary step of ML VVUQ, more specifically, Deep Neural Networks (DNNs) because they are the most widely used supervised ML algorithm for both regression and classification tasks. This work aims at quantifying the prediction, or approximation uncertainties of DNNs when they are used as surrogate models for expensive physical models. Three techniques for UQ of DNNs are compared, namely Monte Carlo Dropout (MCD), Deep Ensembles (DE) and Bayesian Neural Networks (BNNs). Two nuclear engineering examples are used to benchmark these methods, (1) time-dependent fission gas release data using the Bison code, and (2) void fraction simulation based on the BFBT benchmark using the TRACE code. It was found that the three methods typically require different DNN architectures and hyperparameters to optimize their performance. The UQ results also depend on the amount of training data available and the nature of the data. Overall, all these three methods can provide reasonable estimations of the approximation uncertainties. The uncertainties are generally smaller when the mean predictions are close to the test data, while the BNN methods usually produce larger uncertainties than MCD and DE.
Meta-Wrapper: Differentiable Wrapping Operator for User Interest Selection in CTR Prediction
Venue: arXiv cs.IR 2022
Year: 2022

Click-through rate (CTR) prediction, whose goal is to predict the probability of the user to click on an item, has become increasingly significant in the recommender systems. Recently, some deep learning models with the ability to automatically extract the user interest from his/her behaviors have achieved great success. In these work, the attention mechanism is used to select the user interested items in historical behaviors, improving the performance of the CTR predictor. Normally, these attentive modules can be jointly trained with the base predictor by using gradient descents. In this paper, we regard user interest modeling as a feature selection problem, which we call user interest selection. For such a problem, we propose a novel approach under the framework of the wrapper method, which is named Meta-Wrapper. More specifically, we use a differentiable module as our wrapping operator and then recast its learning problem as a continuous bilevel optimization. Moreover, we use a meta-learning algorithm to solve the optimization and theoretically prove its convergence. Meanwhile, we also provide theoretical analysis to show that our proposed method 1) efficiencies the wrapper-based feature selection, and 2) achieves better resistance to overfitting. Finally, extensive experiments on three public datasets manifest the superiority of our method in boosting the performance of CTR prediction.