Miracle-dz/UniFAR-FTUs huggingface dataset:
Cascaded Head-colliding Attention

Transformers have advanced the field of natural language processing (NLP) on a variety of important tasks. At the cornerstone of the Transformer architecture is the multi-head attention (MHA) mechanism which models pairwise interactions between the elements of the sequence. Despite its massive success, the current framework ignores interactions among different heads, leading to the problem that many of the heads are redundant in practice, which greatly wastes the capacity of the model. To improve parameter efficiency, we re-formulate the MHA as a latent variable model from a probabilistic perspective. We present cascaded head-colliding attention (CODA) which explicitly models the interactions between attention heads through a hierarchical variational distribution. We conduct extensive experiments and demonstrate that CODA outperforms the transformer baseline, by 0.6 perplexity on Wikitext-103 in language modeling, and by 0.6 BLEU on WMT14 EN-DE in machine translation, due to its improvements on the parameter efficiency.
Accurate and Efficient Time Series Matching by Season- and Trend-aware Symbolic Approximation - Extended Version Including Additional Evaluation and Proofs

Processing and analyzing time series data\-sets have become a central issue in many domains requiring data management systems to support time series as a native data type. A crucial prerequisite of these systems is time series matching, which still is a challenging problem. A time series is a high-dimensional data type, its representation is storage-, and its comparison is time-consuming. Among the representation techniques that tackle these challenges, the symbolic aggregate approximation (SAX) is the current state of the art. This technique reduces a time series to a low-dimensional space by segmenting it and discretizing each segment into a small symbolic alphabet. However, SAX ignores the deterministic behavior of time series such as cyclical repeating patterns or trend component affecting all segments and leading to a distortion of the symbolic distribution. In this paper, we present a season- and a trend-aware symbolic approximation. We show that this improves the symbolic distribution and increase the representation accuracy without increasing its memory footprint. Most importantly, this enables a more efficient time series matching by providing a match up to three orders of magnitude faster than SAX.
ECMO: Peripheral Transplantation to Rehost Embedded Linux Kernels

Dynamic analysis based on the full-system emulator QEMU is widely used for various purposes.However, it is challenging to run firmware images of embedded devices in QEMU, especially the process to boot the Linux kernel (we call this process rehosting the Linux kernel in this paper). That's because embedded devices usually use different system-on-chips (SoCs) from multiple vendors and only a limited number of SoCs are currently supported in QEMU. In this work, we propose a technique called peripheral transplantation. The main idea is to transplant the device drivers of designated peripherals into the Linux kernel binary. By doing so, it can replace the peripherals in the kernel that are currently unsupported in QEMU with supported ones, thus making the Linux kernel rehostable. After that, various applications can be built. We implemented this technique inside a prototype system called ECMO and applied it to 815 firmware images, which consist of 20 kernel versions and 37 device models. The result shows that ECMO can successfully transplant peripherals for all the 815 Linux kernels. Among them, 710 kernels can be successfully rehosted, i.e., launching a user-space shell (87.1% success rate). The failed cases are mainly because the root file system format (ramfs) is not supported by the kernel. Meanwhile, we are able to inject rather complex drivers (i.e., NIC driver) for all the rehosted Linux kernels by installing kernel modules. We further build three applications, i.e., kernel crash analysis, rootkit forensic analysis, and kernel fuzzing, based on the rehosted kernels to demonstrate the usage scenarios of ECMO.
Factorising Meaning and Form for Intent-Preserving Paraphrasing

We propose a method for generating paraphrases of English questions that retain the original intent but use a different surface form. Our model combines a careful choice of training objective with a principled information bottleneck, to induce a latent encoding space that disentangles meaning and form. We train an encoder-decoder model to reconstruct a question from a paraphrase with the same meaning and an exemplar with the same surface form, leading to separated encoding spaces. We use a Vector-Quantized Variational Autoencoder to represent the surface form as a set of discrete latent variables, allowing us to use a classifier to select a different surface form at test time. Crucially, our method does not require access to an external source of target exemplars. Extensive experiments and a human evaluation show that we are able to generate paraphrases with a better tradeoff between semantic preservation and syntactic novelty compared to previous methods.
About Explicit Variance Minimization: Training Neural Networks for Medical Imaging With Limited Data Annotations

Self-supervised learning methods for computer vision have demonstrated the effectiveness of pre-training feature representations, resulting in well-generalizing Deep Neural Networks, even if the annotated data are limited. However, representation learning techniques require a significant amount of time for model training, with most of the time spent on precise hyper-parameter optimization and selection of augmentation techniques. We hypothesized that if the annotated dataset has enough morphological diversity to capture the diversity of the general population, as is common in medical imaging due to conserved similarities of tissue morphology, the variance error of the trained model is the dominant component of the Bias-Variance Trade-off. Therefore, we proposed the Variance Aware Training (VAT) method that exploits this data property by introducing the variance error into the model loss function, thereby, explicitly regularizing the model. Additionally, we provided a theoretical formulation and proof of the proposed method to aid interpreting the approach. Our method requires selecting only one hyper-parameter and matching or improving the performance of state-of-the-art self-supervised methods while achieving an order of magnitude reduction in the GPU training time. We validated VAT on three medical imaging datasets from diverse domains and for various learning objectives. These included a Magnetic Resonance Imaging (MRI) dataset for the heart semantic segmentation (MICCAI 2017 ACDC challenge), fundus photography dataset for ordinary regression of diabetic retinopathy progression (Kaggle 2019 APTOS Blindness Detection challenge), and classification of histopathologic scans of lymph node sections (PatchCamelyon dataset). Our code is available at https://github.com/DmitriiShubin/Variance-Aware-Training.
Grammar Accuracy Evaluation (GAE): Quantifiable Intrinsic Evaluation of Machine Translation Models

Intrinsic evaluation by humans for the performance of natural language generation models is conducted to overcome the fact that the quality of generated sentences cannot be fully represented by only extrinsic evaluation. Nevertheless, existing intrinsic evaluations have a large score deviation according to the evaluator’s criteria. In this paper, we propose Grammar Accuracy Evaluation (GAE) that can provide specific evaluating criteria. As a result of analyzing the quality of machine translation by BLEU and GAE, it was confirmed that the BLEU score does not represent the absolute performance of machine translation models and that GAE compensates for the shortcomings of BLEU with a flexible evaluation on alternative synonyms and changes in sentence structure.
Multimodal Detection of Information Disorder from Social Media

Social media is accompanied by an increasing pro-portion of content that provides fake information or misleading content, known as information disorder. In this paper, we study the problem of multimodal fake news detection on a large-scale multimodal dataset. We propose a multimodal network architecture that enables different levels and types of information fusion. In addition to the textual and visual content of a posting, we further leverage secondary information, i.e. user comments and metadata. We fuse information at multiple levels to account for the specific intrinsic structure of the modalities. Our results show that multimodal analysis is highly effective for the task and all modalities contribute positively when fused properly.
Transformer-Based Source-Free Domain Adaptation

In this paper, we study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. Previous works on SFDA mainly focus on aligning the cross-domain distributions. However, they ignore the generalization ability of the pretrained source model, which largely influences the initial target outputs that are vital to the target adaptation stage. To address this, we make the interesting observation that the model accuracy is highly correlated with whether or not attention is focused on the objects in an image. To this end, we propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA. Specifically, we apply the Transformer as the attention module and inject it into a convolutional network. By doing so, the model is encouraged to turn attention towards the object regions, which can effectively improve the model's generalization ability on the target domains. Moreover, a novel self-supervised knowledge distillation approach is proposed to adapt the Transformer with target pseudo-labels, thus further encouraging the network to focus on the object regions. Experiments on three domain adaptation tasks, including closed-set, partial-set, and open-set adaption, demonstrate that TransDA can greatly improve the adaptation accuracy and produce state-of-the-art results. The source code and trained models are available at https://github.com/ygjwd12345/TransDA.
Statistical Inference from Partially Nominated Sets: An Application to Estimating the Prevalence of Osteoporosis

This paper focuses on drawing statistical inference based on a novel variant of maxima or minima nomination sampling (NS) designs. These sampling designs are useful for obtaining more representative sample units from the tails of the population distribution using the available auxiliary ranking information. However, one common difficulty in performing NS in practice is that the researcher cannot obtain a nominated sample unless he/she uniquely determines the sample unit with the highest or the lowest rank in each set. To overcome this problem, a variant of NS which is called partial nomination sampling is proposed in which the researcher is allowed to declare that two or more units are tied in the ranks whenever he/she cannot find with high confidence the sample unit with the highest or the lowest rank with high confidence. Based on this sampling design, two asymptotically unbiased estimators are developed for the cumulative distribution function, which are obtained using maximum likelihood and moment-based approaches, and their asymptotic normality is proved. Several numerical studies have shown that the developed estimators have higher relative efficiencies than their counterpart in simple random sampling in analyzing either the upper or the lower tail of the parent distribution. The procedures that we are developed are then implemented on a real dataset from the Third National Health and Nutrition Examination Survey (NHANES III) to estimate the prevalence of osteoporosis among adult women aged 50 and over. It is shown that in some certain circumstances, the techniques that we have developed require only one-third of the sample size needed in SRS to achieve the desired precision. This results in a considerable reduction of time and cost compared to the standard SRS method.
The North American Natural Gas Liquids Markets are Chaotic

In this paper we test for deterministic chaos (i.e., nonlinear deterministic processes which look random) in seven Mont Belview, Texas, hydrocarbon markets, using monthly data from 1985:1 to 1996:12--the markets are those of ethane, propane, normal butane, iso-butane, naptha, crude oil, and natural gas. In doing so, we use the Lyapunov exponent estimator of Nychka, Ellner, Gallant, and McCaffrey (1992). We conclude that there is evidence, consistent with a chaotic nonlinear generation process in all five natural gas liquids markets.
Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview

Object pose detection and tracking has recently attracted increasing attention due to its wide applications in many areas, such as autonomous driving, robotics, and augmented reality. Among methods for object pose detection and tracking, deep learning is the most promising one that has shown better performance than others. However, survey study about the latest development of deep learning-based methods is lacking. Therefore, this study presents a comprehensive review of recent progress in object pose detection and tracking that belongs to the deep learning technical route. To achieve a more thorough introduction, the scope of this study is limited to methods taking monocular RGB/RGBD data as input and covering three kinds of major tasks: instance-level monocular object pose detection, category-level monocular object pose detection, and monocular object pose tracking. In our work, metrics, datasets, and methods of both detection and tracking are presented in detail. Comparative results of current state-of-the-art methods on several publicly available datasets are also presented, together with insightful observations and inspiring future research directions.
Gradient-based Data Subversion Attack Against Binary Classifiers

Machine learning based data-driven technologies have shown impressive performances in a variety of application domains. Most enterprises use data from multiple sources to provide quality applications. The reliability of the external data sources raises concerns for the security of the machine learning techniques adopted. An attacker can tamper the training or test datasets to subvert the predictions of models generated by these techniques. Data poisoning is one such attack wherein the attacker tries to degrade the performance of a classifier by manipulating the training data. In this work, we focus on label contamination attack in which an attacker poisons the labels of data to compromise the functionality of the system. We develop Gradient-based Data Subversion strategies to achieve model degradation under the assumption that the attacker has limited-knowledge of the victim model. We exploit the gradients of a differentiable convex loss function (residual errors) with respect to the predicted label as a warm-start and formulate different strategies to find a set of data instances to contaminate. Further, we analyze the transferability of attacks and the susceptibility of binary classifiers. Our experiments show that the proposed approach outperforms the baselines and is computationally efficient.
Heteronuclear Single-Quantum Correlation–Nuclear Magnetic Resonance (HSQC–NMR) Fingerprint Analysis of Pyrolysis Oils

The pyrolysis of softwood kraft lignin, cellulose, and Loblolly pine wood was examined at 400, 500, and 600 °C. The analysis of the yields of pyrolysis products indicated that lignin yielded the largest amount of a heavy oil and char and only trace levels of a light oil. In contrast, cellulose produced minor amounts of a heavy oil and char and more light oil. All of the pyrolysis oils were analyzed by heteronuclear single-quantum correlation–nuclear magnetic resonance (HSQC–NMR) to analyze the structural components of the bio-oils, and three chemical-shift databases of compounds reported to be presented in pyrolysis oils produced from lignin, cellulose, and pine wood were employed for data analysis. On the basis of databases, analysis of the HSQC–NMR spectral data provides chemical-shift assignment of 27 different types of C–H bonds presented in the pyrolysis oils. The HSQC–NMR analysis of these pyrolysis oils indicated that there are two different types of methoxyl groups presented in the pyrolysis oils .
Enhancing energy transfer in quantum systems via periodic driving: Floquet master equations

We provide a comprehensive study of the energy transfer phenomenon—populating a given energy level—in 3-and 4-level quantum systems coupled to two thermal baths. In particular, we examine the eﬀects of an external periodic driving and the coherence induced by the baths on the eﬃciency of the energy transfer. We consider the Floquet-Lindblad and the Floquet-Redﬁeld scenarios, which both are in the Born-Markov, weak-coupling regime but diﬀer in the treatment of the secular approximation, and for the latter we develop an appropriate Floquet-type master equation by employing a partial secular approximation. Throughout the whole analysis we keep Lamb-shift corrections in the master equations. We observe that, especially in the Floquet-Redﬁeld scenario, the driving ﬁeld can enhance the energy transfer eﬃciency compared to the nondriven scenario. In addition, unlike degenerate systems where Lamb-shift corrections do not contribute signiﬁcantly on the energy transfer, in the Redﬁeld and the Floquet-Redﬁeld scenarios these corrections have nonnegligible eﬀects.
An $\ell_p$ Variable Projection Method for Large-Scale Separable Nonlinear Inverse Problems

The variable projection (VarPro) method is an efficient method to solve separable nonlinear least squares problems. In this paper, we propose a modified VarPro for large-scale separable nonlinear inverse problems that promotes edge-preserving and sparsity properties on the desired solution and enhances the convergence of the parameters that define the forward problem. We adopt a majorization minimization method that relies on constructing a quadratic tangent majorant to approximate a general $\ell_p$ regularized problem by an $\ell_2$ regularized problem that can be solved by the aid of generalized Krylov subspace methods at a relatively low cost compared to the original unprojected problem. In addition, we can use more potential general regularizers including total variation (TV), framelet, and wavelets operators. The regularization parameter can be defined automatically at each iteration by means of generalized cross validation. Numerical examples on large-scale two-dimensional imaging problems arising from blind deconvolution are used to highlight the performance of the proposed approach in both quality of the reconstructed image as well as the reconstructed forward operator.
Seeking Echoes of Circumstellar Disks in Kepler Light Curves

Light echoes of flares on active stars offer the opportunity for direct detection of circumstellar dust. We revisit the problem of identifying faint echoes in postflare light curves, focusing on debris disks from ongoing planet formation. Starting with simulations, we develop an algorithm for estimating the radial extent and total mass from disk echo profiles. We apply this algorithm to light curves from over 2100 stars observed by NASA’s Kepler mission, selected for multiple short-lived flares in either the long-cadence or short-cadence data sets. While flux uncertainties in light curves from individual stars preclude useful mass limits on circumstellar disks, catalog-averaged light curves yield constraints on disk mass that are comparable to estimates from known debris disks. The average mass in micron- to millimeter-sized dust around the Kepler stars cannot exceed 10% of an Earth mass in exo-Kuiper belts or 10% of a lunar mass in the terrestrial zone. We group stars according to IR excess, based on WISE W1–W3 color, as an indicator for the presence of circumstellar dust. The mass limits are greater for stars with strong IR excess, a hint that echoes are lurking not far beneath the noise in postflare light curves. With increased sensitivity, echo detection will let time-domain astronomy complement spectroscopic and direct-imaging studies in mapping how, when, and where planets form.
GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation

Computer-aided translation (CAT), the use of software to assist a human translator in the translation process, has been proven to be useful in enhancing the productivity of human translators. Autocompletion, which suggests translation results according to the text pieces provided by human translators, is a core function of CAT. There are two limitations in previous research in this line. First, most research works on this topic focus on sentence-level autocompletion (i.e., generating the whole translation as a sentence based on human input), but word-level autocompletion is under-explored so far. Second, almost no public benchmarks are available for the autocompletion task of CAT. This might be among the reasons why research progress in CAT is much slower compared to automatic MT. In this paper, we propose the task of general word-level autocompletion (GWLAN) from a real-world CAT scenario, and construct the first public benchmark to facilitate research in this topic. In addition, we propose an effective method for GWLAN and compare it with several strong baselines. Experiments demonstrate that our proposed method can give significantly more accurate predictions than the baseline methods on our benchmark datasets.
Near-Optimal Multi-Perturbation Experimental Design for Causal Structure Learning

Causal structure learning is a key problem in many domains. Causal structures can be learnt by performing experiments on the system of interest. We address the largely unexplored problem of designing experiments that simultaneously intervene on multiple variables. While potentially more informative than the commonly considered single-variable interventions, selecting such interventions is algorithmically much more challenging, due to the doubly-exponential combinatorial search space over sets of composite interventions. In this paper, we develop efficient algorithms for optimizing different objective functions quantifying the informativeness of experiments. By establishing novel submodularity properties of these objectives, we provide approximation guarantees for our algorithms. Our algorithms empirically perform superior to both random interventions and algorithms that only select single-variable interventions.
Memory-Efficient Differentiable Transformer Architecture Search

Differentiable architecture search (DARTS) is successfully applied in many vision tasks. However, directly using DARTS for Transformers is memory-intensive, which renders the search process infeasible. To this end, we propose a multi-split reversible network and combine it with DARTS. Specifically, we devise a backpropagation-with-reconstruction algorithm so that we only need to store the last layer's outputs. By relieving the memory burden for DARTS, it allows us to search with larger hidden size and more candidate operations. We evaluate the searched architecture on three sequence-to-sequence datasets, i.e., WMT'14 English-German, WMT'14 English-French, and WMT'14 English-Czech. Experimental results show that our network consistently outperforms standard Transformers across the tasks. Moreover, our method compares favorably with big-size Evolved Transformers, reducing search computation by an order of magnitude.