OpenRaiser/Intern-Atlas huggingface dataset:
G-CMP: Graph-enhanced Contextual Matrix Profile for unsupervised anomaly detection in sensor-based remote health monitoring
Venue: arXiv cs.LG 2022
Year: 2022

Sensor-based remote health monitoring is used in industrial, urban and healthcare settings to monitor ongoing operation of equipment and human health. An important aim is to intervene early if anomalous events or adverse health is detected. In the wild, these anomaly detection approaches are challenged by noise, label scarcity, high dimensionality, explainability and wide variability in operating environments. The Contextual Matrix Profile (CMP) is a configurable 2-dimensional version of the Matrix Profile (MP) that uses the distance matrix of all subsequences of a time series to discover patterns and anomalies. The CMP is shown to enhance the effectiveness of the MP and other SOTA methods at detecting, visualising and interpreting true anomalies in noisy real world data from different domains. It excels at zooming out and identifying temporal patterns at configurable time scales. However, the CMP does not address cross-sensor information, and cannot scale to high dimensional data. We propose a novel, self-supervised graph-based approach for temporal anomaly detection that works on context graphs generated from the CMP distance matrix. The learned graph embeddings encode the anomalous nature of a time context. In addition, we evaluate other graph outlier algorithms for the same task. Given our pipeline is modular, graph construction, generation of graph embeddings, and pattern recognition logic can all be chosen based on the specific pattern detection application. We verified the effectiveness of graph-based anomaly detection and compared it with the CMP and 3 state-of-the art methods on two real-world healthcare datasets with different anomalies. Our proposed method demonstrated better recall, alert rate and generalisability.
AutoCAD: Automatically Generating Counterfactuals for Mitigating Shortcut Learning
Venue: arXiv cs.AI 2022
Year: 2022

Recent studies have shown the impressive efficacy of counterfactually augmented data (CAD) for reducing NLU models' reliance on spurious features and improving their generalizability. However, current methods still heavily rely on human efforts or task-specific designs to generate counterfactuals, thereby impeding CAD's applicability to a broad range of NLU tasks. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework. AutoCAD first leverages a classifier to unsupervisedly identify rationales as spans to be intervened, which disentangles spurious and causal features. Then, AutoCAD performs controllable generation enhanced by unlikelihood training to produce diverse counterfactuals. Extensive evaluations on multiple out-of-domain and challenge benchmarks demonstrate that AutoCAD consistently and significantly boosts the out-of-distribution performance of powerful pre-trained models across different NLU tasks, which is comparable or even better than previous state-of-the-art human-in-the-loop or task-specific CAD methods. The code is publicly available at https://github.com/thu-coai/AutoCAD.
Curriculum Temperature for Knowledge Distillation
Venue: arXiv cs.CV 2022
Year: 2022

Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.
A Cross-Conformal Predictor for Multi-label Classification
Venue: arXiv cs.LG 2022
Year: 2022

Unlike the typical classification setting where each instance is associated with a single class, in multi-label learning each instance is associated with multiple classes simultaneously. Therefore the learning task in this setting is to predict the subset of classes to which each instance belongs. This work examines the application of a recently developed framework called Conformal Prediction (CP) to the multi-label learning setting. CP complements the predictions of machine learning algorithms with reliable measures of confidence. As a result the proposed approach instead of just predicting the most likely subset of classes for a new unseen instance, also indicates the likelihood of each predicted subset being correct. This additional information is especially valuable in the multi-label setting where the overall uncertainty is extremely high.
Distance Teaching Experience of Campus-based Teachers at Times of Pandemic Confinement
Venue: arXiv cs.CY 2022
Year: 2022

Amidst the outbreak of the coronavirus (COVID 19) pandemic, distance education, where the learning process is conducted online, has become the norm. Campus-based programs and courses have been redesigned in a timely manner which was a challenge for teachers not used to distance teaching. Students engagement and active participation become an issue; add to that new emerging effects associating with this set-up, such as the so called 'Zoom fatigue', which was coined recently by some authors. In realising this problem, solutions were suggested in the literature to help trigger students engagement and enhance teachers experience in online teaching. This study analyses these effects along with our teachers experience in the new learning environment and concludes by devising some recommendations. To attain the above objectives, we conducted online interviews with six of our teachers, transcribed the content of the videos and then applied the inductive research approach to assess the results.
Common Knowledge of Abstract Groups
Venue: arXiv cs.AI 2022
Year: 2022

Epistemic logics typically talk about knowledge of individual agents or groups of explicitly listed agents. Often, however, one wishes to express knowledge of groups of agents specified by a given property, as in `it is common knowledge among economists'. We introduce such a logic of common knowledge, which we term abstract-group epistemic logic (AGEL). That is, AGEL features a common knowledge operator for groups of agents given by concepts in a separate agent logic that we keep generic, with one possible agent logic being ALC. We show that AGEL is EXPTIME-complete, with the lower bound established by reduction from standard group epistemic logic, and the upper bound by a satisfiability-preserving embedding into the full $μ$-calculus. Further main results include a finite model property (not enjoyed by the full $μ$-calculus) and a complete axiomatization.
Reasoning about Promises in Weak Memory Models with Event Structures (Extended Version)
Venue: arXiv cs.LO 2022
Year: 2022

Modern processors such as ARMv8 and RISC-V allow executions in which independent instructions within a process may be reordered. To cope with such phenomena, so called promising semantics have been developed, which permit threads to read values that have not yet been written. Each promise is a speculative update that is later validated (fulfilled) by an actual write. Promising semantics are operational, providing a pathway for developing proof calculi. In this paper, we develop an incorrectness-style logic, resulting in a framework for reasoning about state reachability. Like incorrectness logic, our assertions are underapproximating, since the set of all valid promises are not known at the start of execution. Our logic uses event structures as assertions to compactly represent the ordering among events such as promised and fulfilled writes. We prove soundness and completeness of our proof calculus and demonstrate its applicability by proving reachability properties of standard weak memory litmus tests.
DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention
Venue: arXiv cs.LG 2022
Year: 2022

Many studies have been conducted to improve the efficiency of Transformer from quadric to linear. Among them, the low-rank-based methods aim to learn the projection matrices to compress the sequence length. However, the projection matrices are fixed once they have been learned, which compress sequence length with dedicated coefficients for tokens in the same position. Adopting such input-invariant projections ignores the fact that the most informative part of a sequence varies from sequence to sequence, thus failing to preserve the most useful information that lies in varied positions. In addition, previous efficient Transformers only focus on the influence of sequence length while neglecting the effect of hidden state dimension. To address the aforementioned problems, we present an efficient yet effective attention mechanism, namely the Dynamic Bilinear Low-Rank Attention (DBA), which compresses the sequence length by input-sensitive dynamic projection matrices and achieves linear time and space complexity by jointly optimizing the sequence length and hidden state dimension while maintaining state-of-the-art performance. Specifically, we first theoretically demonstrate that the sequence length can be compressed non-destructively from a novel perspective of information theory, with compression matrices dynamically determined by the input sequence. Furthermore, we show that the hidden state dimension can be approximated by extending the Johnson-Lindenstrauss lemma, optimizing the attention in bilinear form. Theoretical analysis shows that DBA is proficient in capturing high-order relations in cross-attention problems. Experiments over tasks with diverse sequence length conditions show that DBA achieves state-of-the-art performance compared with various strong baselines while maintaining less memory consumption with higher speed.
Identification of the Breach of Short-term Rental Regulations in Irish Rent Pressure Zones
Venue: arXiv cs.SI 2022
Year: 2022

The housing crisis in Ireland has rapidly grown in recent years. To make a more significant profit, many landlords are no longer renting out their houses under long-term tenancies but under short-term tenancies. The shift from long-term to short-term rentals has harmed the supply of private housing rentals. Regulating rentals in Rent Pressure Zones with the highest and rising rents is becoming a tricky issue. In this paper, we develop a breach identifier to check short-term rentals located in Rent Pressure Zones with potential breaches only using publicly available data from Airbnb (an online marketplace focused on short-term home-stays). First, we use a Residual Neural Network to filter out outdoor landscape photos that negatively impact identifying whether an owner has multiple rentals in a Rent Pressure Zone. Second, a Siamese Neural Network is used to compare the similarity of indoor photos to determine if multiple rental posts correspond to the same residence. Next, we use the Haversine algorithm to locate short-term rentals within a circle centered on the coordinate of a permit. Short-term rentals with a permit will not be restricted. Finally, we improve the occupancy estimation model combined with sentiment analysis, which may provide higher accuracy. Because Airbnb does not disclose accurate house coordinates and occupancy data, it is impossible to verify the accuracy of our breach identifier. The accuracy of the occupancy estimator cannot be verified either. It only provides an estimate within a reasonable range. Users should be skeptical of short-term rentals that are flagged as possible breaches.
SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers
Venue: arXiv cs.CL 2022
Year: 2022

Fine-tuning pre-trained language models (PLMs) achieves impressive performance on a range of downstream tasks, and their sizes have consequently been getting bigger. Since a different copy of the model is required for each task, this paradigm is infeasible for storage-constrained edge devices like mobile phones. In this paper, we propose SPARTAN, a parameter efficient (PE) and computationally fast architecture for edge devices that adds hierarchically organized sparse memory after each Transformer layer. SPARTAN freezes the PLM parameters and fine-tunes only its memory, thus significantly reducing storage costs by re-using the PLM backbone for different tasks. SPARTAN contains two levels of memory, with only a sparse subset of parents being chosen in the first level for each input, and children cells corresponding to those parents being used to compute an output representation. This sparsity combined with other architecture optimizations improves SPARTAN's throughput by over 90% during inference on a Raspberry Pi 4 when compared to PE baselines (adapters) while also outperforming the latter by 0.1 points on the GLUE benchmark. Further, it can be trained 34% faster in a few-shot setting, while performing within 0.9 points of adapters. Qualitative analysis shows that different parent cells in SPARTAN specialize in different topics, thus dividing responsibility efficiently.
Automatic Discovery of Multi-perspective Process Model using Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Process mining is a methodology for the derivation and analysis of process models based on the event log. When process mining is employed to analyze business processes, the process discovery step, the conformance checking step, and the enhancements step are repeated. If a user wants to analyze a process from multiple perspectives (such as activity perspectives, originator perspectives, and time perspectives), the above procedure, inconveniently, has to be repeated over and over again. Although past studies involving process mining have applied detailed stepwise methodologies, no attempt has been made to incorporate and optimize multi-perspective process mining procedures. This paper contributes to developing a solution approach to this problem. First, we propose an automatic discovery framework of a multi-perspective process model based on deep Q-Learning. Our Dual Experience Replay with Experience Distribution (DERED) approach can automatically perform process model discovery steps, conformance check steps, and enhancements steps. Second, we propose a new method that further optimizes the experience replay (ER) method, one of the key algorithms of deep Q-learning, to improve the learning performance of reinforcement learning agents. Finally, we validate our approach using six real-world event datasets collected in port logistics, steel manufacturing, finance, IT, and government administration. We show that our DERED approach can provide users with multi-perspective, high-quality process models that can be employed more conveniently for multi-perspective process mining.
Extracting Semantic Knowledge from GANs with Unsupervised Learning
Venue: arXiv cs.CV 2022
Year: 2022

Recently, unsupervised learning has made impressive progress on various tasks. Despite the dominance of discriminative models, increasing attention is drawn to representations learned by generative models and in particular, Generative Adversarial Networks (GANs). Previous works on the interpretation of GANs reveal that GANs encode semantics in feature maps in a linearly separable form. In this work, we further find that GAN's features can be well clustered with the linear separability assumption. We propose a novel clustering algorithm, named KLiSH, which leverages the linear separability to cluster GAN's features. KLiSH succeeds in extracting fine-grained semantics of GANs trained on datasets of various objects, e.g., car, portrait, animals, and so on. With KLiSH, we can sample images from GANs along with their segmentation masks and synthesize paired image-segmentation datasets. Using the synthesized datasets, we enable two downstream applications. First, we train semantic segmentation networks on these datasets and test them on real images, realizing unsupervised semantic segmentation. Second, we train image-to-image translation networks on the synthesized datasets, enabling semantic-conditional image synthesis without human annotations.
Automated Generating Natural Language Requirements based on Domain Ontology
Venue: arXiv cs.SE 2022
Year: 2022

Software requirements specification is undoubtedly critical for the whole software life-cycle. Nowadays, writing software requirements specifications primarily depends on human work. Although massive studies have been proposed to fasten the process via proposing advanced elicitation and analysis techniques, it is still a time-consuming and error-prone task that needs to take domain knowledge and business information into consideration. In this paper, we propose an approach, named ReqGen, which can provide recommendations by automatically generating natural language requirements specifications based on certain given keywords. Specifically, ReqGen consists of three critical steps. First, keywords-oriented knowledge is selected from domain ontology and is injected to the basic Unified pre-trained Language Model (UniLM) for domain fine-tuning. Second, a copy mechanism is integrated to ensure the occurrence of keywords in the generated statements. Finally, a requirement syntax constrained decoding is designed to close the semantic and syntax distance between the candidate and reference specifications. Experiments on two public datasets from different groups and domains show that ReqGen outperforms six popular natural language generation approaches with respect to the hard constraint of keywords(phrases) inclusion, BLEU, ROUGE and syntax compliance. We believe that ReqGen can promote the efficiency and intelligence of specifying software requirements.
GPU-Accelerated DNS of Compressible Turbulent Flows
Venue: arXiv cs.CE 2022
Year: 2022

This paper explores strategies to transform an existing CPU-based high-performance computational fluid dynamics solver, HyPar, for compressible flow simulations on emerging exascale heterogeneous (CPU+GPU) computing platforms. The scientific motivation for developing a GPU-enhanced version of HyPar is to simulate canonical turbulent flows at the highest resolution possible on such platforms. We show that optimizing memory operations and thread blocks results in 200x speedup of computationally intensive kernels compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication, we demonstrate both strong and weak scaling of our GPU-based HyPar implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of homogeneous isotropic turbulence in a triply periodic box on grids with up to $1024^3$ points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We compare the wall times for CPU-only and CPU+GPU simulations. The results presented in the paper are obtained on the Summit and Lassen supercomputers at Oak Ridge and Lawrence Livermore National Laboratories, respectively.
Arithmetic autocorrelation distribution of binary $m$-sequences
Venue: arXiv cs.IT 2022
Year: 2022

Binary $m$-sequences are ones with the largest period $n=2^m-1$ among the binary sequences produced by linear shift registers with length $m$. They have a wide range of applications in communication since they have several desirable pseudorandomness such as balance, uniform pattern distribution and ideal (classical) autocorrelation. In his reseach on arithmetic codes, Mandelbaum \cite{9Mand} introduces a 2-adic version of classical autocorrelation of binary sequences, called arithmetic autocorrelation. Later, Goresky and Klapper \cite{3G1,4G2,5G3,6G4} generalize this notion to nonbinary case and develop several properties of arithmetic autocorrelation related to linear shift registers with carry. Recently, Z. Chen et al. \cite{1C1} show an upper bound on arithmetic autocorrelation of binary $m$-sequences and raise a conjecture on absolute value distribution on arithmetic autocorrelation of binary $m$-sequences.
Uncertainty-Aware Image Captioning
Venue: arXiv cs.CV 2022
Year: 2022

It is well believed that the higher uncertainty in a word of the caption, the more inter-correlated context information is required to determine it. However, current image captioning methods usually consider the generation of all words in a sentence sequentially and equally. In this paper, we propose an uncertainty-aware image captioning framework, which parallelly and iteratively operates insertion of discontinuous candidate words between existing words from easy to difficult until converged. We hypothesize that high-uncertainty words in a sentence need more prior information to make a correct decision and should be produced at a later stage. The resulting non-autoregressive hierarchy makes the caption generation explainable and intuitive. Specifically, we utilize an image-conditioned bag-of-word model to measure the word uncertainty and apply a dynamic programming algorithm to construct the training pairs. During inference, we devise an uncertainty-adaptive parallel beam search technique that yields an empirically logarithmic time complexity. Extensive experiments on the MS COCO benchmark reveal that our approach outperforms the strong baseline and related methods on both captioning quality as well as decoding speed.
Handling Missing Data via Max-Entropy Regularized Graph Autoencoder
Venue: arXiv cs.LG 2022
Year: 2022

Graph neural networks (GNNs) are popular weapons for modeling relational data. Existing GNNs are not specified for attribute-incomplete graphs, making missing attribute imputation a burning issue. Until recently, many works notice that GNNs are coupled with spectral concentration, which means the spectrum obtained by GNNs concentrates on a local part in spectral domain, e.g., low-frequency due to oversmoothing issue. As a consequence, GNNs may be seriously flawed for reconstructing graph attributes as graph spectral concentration tends to cause a low imputation precision. In this work, we present a regularized graph autoencoder for graph attribute imputation, named MEGAE, which aims at mitigating spectral concentration problem by maximizing the graph spectral entropy. Notably, we first present the method for estimating graph spectral entropy without the eigen-decomposition of Laplacian matrix and provide the theoretical upper error bound. A maximum entropy regularization then acts in the latent space, which directly increases the graph spectral entropy. Extensive experiments show that MEGAE outperforms all the other state-of-the-art imputation methods on a variety of benchmark datasets.
Trajectory-based Traveling Salesman Problem for Multirotor UAVs
Venue: arXiv cs.RO 2022
Year: 2022

This paper presents a new method for integrated time-optimal routing and trajectory optimization of multirotor unmanned aerial vehicles (UAVs). Our approach extends the well-known Traveling Salesman Problem by accounting for the limited maneuverability of the UAVs due to their kinematic properties. To this end, we allow each waypoint to be traversed with a discretized velocity as well as a discretized flight direction and compute time-optimal trajectories to determine the travel time costs for each edge. We refer to this novel optimization problem as the Trajectory-based Traveling Salesman Problem (TBTSP). The results show that compared to a state-of-the-art approach for Traveling Salesman Problems with kinematic restrictions of UAVs, we can decrease mission duration by up to 15\%.