OpenRaiser/Intern-Atlas huggingface dataset:
ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations
Venue: arXiv cs.LG 2022
Year: 2022

Deep learning models rely on highly optimized tensor libraries for efficient inference on heterogeneous hardware. Current deep compilers typically predetermine layouts of tensors and then optimize loops of operators. However, such unidirectional and one-off workflow strictly separates graph-level optimization and operator-level optimization into different system layers, missing opportunities for unified tuning. This paper proposes ALT, a compiler that performs joint graph- and operator-level optimizations for deep models. ALT provides a generic transformation module to manipulate layouts and loops with easy-to-use primitive functions. ALT further integrates an auto-tuning module that jointly optimizes graph-level data layouts and operator-level loops while guaranteeing efficiency. Experimental results show that ALT significantly outperforms state-of-the-art compilers (e.g., Ansor) in terms of both single operator performance (e.g., 1.5x speedup on average) and end-to-end inference performance (e.g., 1.4x speedup on average).
SLAMs: Semantic Learning based Activation Map for Weakly Supervised Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Recent mainstream weakly-supervised semantic segmentation (WSSS) approaches mainly relies on image-level classification learning, which has limited representation capacity. In this paper, we propose a novel semantic learning based framework, named SLAMs (Semantic Learning based Activation Map), for WSSS.
Structure-Unified M-Tree Coding Solver for MathWord Problem
Venue: arXiv cs.CL 2022
Year: 2022

As one of the challenging NLP tasks, designing math word problem (MWP) solvers has attracted increasing research attention for the past few years. In previous work, models designed by taking into account the properties of the binary tree structure of mathematical expressions at the output side have achieved better performance. However, the expressions corresponding to a MWP are often diverse (e.g., $n_1+n_2 \times n_3-n_4$, $n_3\times n_2-n_4+n_1$, etc.), and so are the corresponding binary trees, which creates difficulties in model learning due to the non-deterministic output space. In this paper, we propose the Structure-Unified M-Tree Coding Solver (SUMC-Solver), which applies a tree with any M branches (M-tree) to unify the output structures. To learn the M-tree, we use a mapping to convert the M-tree into the M-tree codes, where codes store the information of the paths from tree root to leaf nodes and the information of leaf nodes themselves, and then devise a Sequence-to-Code (seq2code) model to generate the codes. Experimental results on the widely used MAWPS and Math23K datasets have demonstrated that SUMC-Solver not only outperforms several state-of-the-art models under similar experimental settings but also performs much better under low-resource conditions.
EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention
Venue: arXiv cs.CL 2022
Year: 2022

One of the key challenges of automatic story generation is how to generate a long narrative that can maintain fluency, relevance, and coherence. Despite recent progress, current story generation systems still face the challenge of how to effectively capture contextual and event features, which has a profound impact on a model's generation performance. To address these challenges, we present EtriCA, a novel neural generation model, which improves the relevance and coherence of the generated stories through residually mapping context features to event sequences with a cross-attention mechanism. Such a feature capturing mechanism allows our model to better exploit the logical relatedness between events when generating stories. Extensive experiments based on both automatic and human evaluations show that our model significantly outperforms state-of-the-art baselines, demonstrating the effectiveness of our model in leveraging context and event features.
LMPriors: Pre-Trained Language Models as Task-Specific Priors
Venue: arXiv cs.LG 2022
Year: 2022

Particularly in low-data regimes, an outstanding challenge in machine learning is developing principled techniques for augmenting our models with suitable priors. This is to encourage them to learn in ways that are compatible with our understanding of the world. But in contrast to generic priors such as shrinkage or sparsity, we draw inspiration from the recent successes of large-scale language models (LMs) to construct task-specific priors distilled from the rich knowledge of LMs. Our method, Language Model Priors (LMPriors), incorporates auxiliary natural language metadata about the task -- such as variable names and descriptions -- to encourage downstream model outputs to be consistent with the LM's common-sense reasoning based on the metadata. Empirically, we demonstrate that LMPriors improve model performance in settings where such natural language descriptions are available, and perform well on several tasks that benefit from such prior knowledge, such as feature selection, causal inference, and safe reinforcement learning.
A Visual Tour Of Current Challenges In Multimodal Language Models
Venue: arXiv cs.CL 2022
Year: 2022

Transformer models trained on massive text corpora have become the de facto models for a wide range of natural language processing tasks. However, learning effective word representations for function words remains challenging. Multimodal learning, which visually grounds transformer models in imagery, can overcome the challenges to some extent; however, there is still much work to be done. In this study, we explore the extent to which visual grounding facilitates the acquisition of function words using stable diffusion models that employ multimodal models for text-to-image generation. Out of seven categories of function words, along with numerous subcategories, we find that stable diffusion models effectively model only a small fraction of function words -- a few pronoun subcategories and relatives. We hope that our findings will stimulate the development of new datasets and approaches that enable multimodal models to learn better representations of function words.
A Temporal Type-2 Fuzzy System for Time-dependent Explainable Artificial Intelligence
Venue: arXiv cs.AI 2022
Year: 2022

Explainable Artificial Intelligence (XAI) is a paradigm that delivers transparent models and decisions, which are easy to understand, analyze, and augment by a non-technical audience. Fuzzy Logic Systems (FLS) based XAI can provide an explainable framework, while also modeling uncertainties present in real-world environments, which renders it suitable for applications where explainability is a requirement. However, most real-life processes are not characterized by high levels of uncertainties alone; they are inherently time-dependent as well, i.e., the processes change with time. In this work, we present novel Temporal Type-2 FLS Based Approach for time-dependent XAI (TXAI) systems, which can account for the likelihood of a measurement's occurrence in the time domain using (the measurement's) frequency of occurrence. In Temporal Type-2 Fuzzy Sets (TT2FSs), a four-dimensional (4D) time-dependent membership function is developed where relations are used to construct the inter-relations between the elements of the universe of discourse and its frequency of occurrence. The TXAI system manifested better classification prowess, with 10-fold test datasets, with a mean recall of 95.40\% than a standard XAI system (based on non-temporal general type-2 (GT2) fuzzy sets) that had a mean recall of 87.04\%. TXAI also performed significantly better than most non-explainable AI systems between 3.95\%, to 19.04\% improvement gain in mean recall. In addition, TXAI can also outline the most likely time-dependent trajectories using the frequency of occurrence values embedded in the TXAI model; viz. given a rule at a determined time interval, what will be the next most likely rule at a subsequent time interval. In this regard, the proposed TXAI system can have profound implications for delineating the evolution of real-life time-dependent processes, such as behavioural or biological processes.
Sublinear-Time Algorithms for Max Cut, Max E2Lin$(q)$, and Unique Label Cover on Expanders
Venue: arXiv cs.DS 2022
Year: 2022

We show sublinear-time algorithms for Max Cut and Max E2Lin$(q)$ on expanders in the adjacency list model that distinguishes instances with the optimal value more than $1-\varepsilon$ from those with the optimal value less than $1-ρ$ for $ρ\gg \varepsilon$. The time complexities for Max Cut and Max $2$Lin$(q)$ are $\widetilde{O}(\frac{1}{ϕ^2ρ} \cdot m^{1/2+O(\varepsilon/(ϕ^2ρ))})$ and $\widetilde{O}(\mathrm{poly}(\frac{q}{ϕρ})\cdot {(mq)}^{1/2+O(q^6\varepsilon/ϕ^2ρ^2)})$, respectively, where $m$ is the number of edges in the underlying graph and $ϕ$ is its conductance. Then, we show a sublinear-time algorithm for Unique Label Cover on expanders with $ϕ\gg ε$ in the bounded-degree model. The time complexity of our algorithm is $\widetilde{O}_d(2^{q^{O(1)}\cdotϕ^{1/q}\cdot \varepsilon^{-1/2}}\cdot n^{1/2+q^{O(q)}\cdot \varepsilon^{4^{1.5-q}}\cdot ϕ^{-2}})$, where $n$ is the number of variables. We complement these algorithmic results by showing that testing $3$-colorability requires $Ω(n)$ queries even on expanders.
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models
Venue: arXiv cs.CL 2022
Year: 2022

How to usefully encode compositional task structure has long been a core challenge in AI. Recent work in chain of thought prompting has shown that for very large neural language models (LMs), explicitly demonstrating the inferential steps involved in a target task may improve performance over end-to-end learning that focuses on the target task alone. However, chain of thought prompting has significant limitations due to its dependency on huge pretrained LMs. In this work, we present compositional fine-tuning (CFT): an approach based on explicitly decomposing a target task into component tasks, and then fine-tuning smaller LMs on a curriculum of such component tasks. We apply CFT to recommendation tasks in two domains, world travel and local dining, as well as a previously studied inferential task (sports understanding). We show that CFT outperforms end-to-end learning even with equal amounts of data, and gets consistently better as more component tasks are modeled via fine-tuning. Compared with chain of thought prompting, CFT performs at least as well using LMs only 7.4% of the size, and is moreover applicable to task domains for which data are not available during pretraining.
Blockchain and Machine Learning for Fraud Detection: A Privacy-Preserving and Adaptive Incentive Based Approach
Venue: arXiv cs.CR 2022
Year: 2022

Financial fraud cases are on the rise even with the current technological advancements. Due to the lack of inter-organization synergy and because of privacy concerns, authentic financial transaction data is rarely available. On the other hand, data-driven technologies like machine learning need authentic data to perform precisely in real-world systems. This study proposes a blockchain and smart contract-based approach to achieve robust Machine Learning (ML) algorithm for e-commerce fraud detection by facilitating inter-organizational collaboration. The proposed method uses blockchain to secure the privacy of the data. Smart contract deployed inside the network fully automates the system. An ML model is incrementally upgraded from collaborative data provided by the organizations connected to the blockchain. To incentivize the organizations, we have introduced an incentive mechanism that is adaptive to the difficulty level in updating a model. The organizations receive incentives based on the difficulty faced in updating the ML model. A mining criterion has been proposed to mine the block efficiently. And finally, the blockchain network istested under different difficulty levels and under different volumes of data to test its efficiency. The model achieved 98.93% testing accuracy and 98.22% Fbeta score (recall-biased f measure) over eight incremental updates. Our experiment shows that both data volume and difficulty level of blockchain impacts the mining time. For difficulty level less than five, mining time and difficulty level has a positive correlation. For difficulty level two and three, less than a second is required to mine a block in our system. Difficulty level five poses much more difficulties to mine the blocks.
RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we introduce the task of visual grounding for remote sensing data (RSVG). RSVG aims to localize the referred objects in remote sensing (RS) images with the guidance of natural language. To retrieve rich information from RS imagery using natural language, many research tasks, like RS image visual question answering, RS image captioning, and RS image-text retrieval have been investigated a lot. However, the object-level visual grounding on RS images is still under-explored. Thus, in this work, we propose to construct the dataset and explore deep learning models for the RSVG task. Specifically, our contributions can be summarized as follows. 1) We build the new large-scale benchmark dataset of RSVG, termed RSVGD, to fully advance the research of RSVG. This new dataset includes image/expression/box triplets for training and evaluating visual grounding models. 2) We benchmark extensive state-of-the-art (SOTA) natural image visual grounding methods on the constructed RSVGD dataset, and some insightful analyses are provided based on the results. 3) A novel transformer-based Multi-Level Cross-Modal feature learning (MLCM) module is proposed. Remotely-sensed images are usually with large scale variations and cluttered backgrounds. To deal with the scale-variation problem, the MLCM module takes advantage of multi-scale visual features and multi-granularity textual embeddings to learn more discriminative representations. To cope with the cluttered background problem, MLCM adaptively filters irrelevant noise and enhances salient features. In this way, our proposed model can incorporate more effective multi-level and multi-modal features to boost performance. Furthermore, this work also provides useful insights for developing better RSVG models. The dataset and code will be publicly available at https://github.com/ZhanYang-nwpu/RSVG-pytorch.
Binary sequences with a low correlation via cyclotomic function fields with odd characteristics
Venue: arXiv cs.IT 2022
Year: 2022

Sequences with a low correlation have very important applications in communications, cryptography, and compressed sensing. In the literature, many efforts have been made to construct good sequences with various lengths where binary sequences attracts great attention. As a result, various constructions of good binary sequences have been proposed. However, most of the known constructions made use of the multiplicative cyclic group structure of finite field $\mathbb{F}_{p^n}$ for a prime $p$ and a positive integer $n$. In fact, all $p^n+1$ rational places including the place at infinity of the rational function field over $\mathbb{F}_{p^n}$ form a cyclic structure under an automorphism of order $p^n+1$. In this paper, we make use of this cyclic structure to provide an explicit construction of binary sequences with a low correlation of length $p^n+1$ via cyclotomic function fields over $\mathbb{F}_{p^n}$ for any odd prime $p$. Each family of binary sequences has size $p^n-2$ and its correlation is upper bounded by $4+\lfloor 2\cdot p^{n/2}\rfloor$. To the best of our knowledge, this is the first construction of binary sequences with a low correlation of length $p^n+1$ for odd prime $p$. Moreover, our sequences can be constructed explicitly and have competitive parameters.
Additive Interventions Yield Robust Multi-Domain Machine Translation Models
Venue: arXiv cs.CL 2022
Year: 2022

Additive interventions are a recently-proposed mechanism for controlling target-side attributes in neural machine translation. In contrast to tag-based approaches which manipulate the raw source sequence, interventions work by directly modulating the encoder representation of all tokens in the sequence. We examine the role of additive interventions in a large-scale multi-domain machine translation setting and compare its performance in various inference scenarios. We find that while the performance difference is small between intervention-based systems and tag-based systems when the domain label matches the test domain, intervention-based systems are robust to label error, making them an attractive choice under label uncertainty. Further, we find that the superiority of single-domain fine-tuning comes under question when training data size is scaled, contradicting previous findings.
Principal Component Classification
Venue: arXiv cs.LG 2022
Year: 2022

We propose to directly compute classification estimates by learning features encoded with their class scores using PCA. Our resulting model has a encoder-decoder structure suitable for supervised learning, it is computationally efficient and performs well for classification on several datasets.
Local and Global Structure Preservation Based Spectral Clustering
Venue: arXiv cs.LG 2022
Year: 2022

Spectral Clustering (SC) is widely used for clustering data on a nonlinear manifold. SC aims to cluster data by considering the preservation of the local neighborhood structure on the manifold data. This paper extends Spectral Clustering to Local and Global Structure Preservation Based Spectral Clustering (LGPSC) that incorporates both global structure and local neighborhood structure simultaneously. For this extension, LGPSC proposes two models to extend local structures preservation to local and global structures preservation: Spectral clustering guided Principal component analysis model and Multilevel model. Finally, we compare the experimental results of the state-of-the-art methods with our two models of LGPSC on various data sets such that the experimental results confirm the effectiveness of our LGPSC models to cluster nonlinear data.
Flow-Level Packet Loss Detection via Sketch Decomposition and Matrix Optimization
Venue: arXiv cs.NI 2022
Year: 2022

For cloud service providers, fine-grained packet loss detection across data centers is crucial in improving their service level and increasing business income. However, the inability to obtain sufficient measurements makes it difficult owing to the fundamental limit that the wide-area network links responsible for communication are not under their management. Moreover, millisecond-level delay jitter and clock synchronization errors in the WAN disable many tools that perform well in data center networks on this issue. Therefore, there is an urgent need to develop a new tool or method. In this work, we propose SketchDecomp, a novel loss detection method, from a mathematical perspective that has never been considered before. Its key is to decompose sketches upstream and downstream into several sub-sketches and builds a low-rank matrix optimization model to solve them. Extensive experiments on the test bed demonstrate its superiority.
DALL-E 2 Fails to Reliably Capture Common Syntactic Processes
Venue: arXiv cs.CL 2022
Year: 2022

Machine intelligence is increasingly being linked to claims about sentience, language processing, and an ability to comprehend and transform natural language into a range of stimuli. We systematically analyze the ability of DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference, passives, word order, coordination, comparatives, negation, ellipsis, and structural ambiguity. Whereas young children routinely master these phenomena, learning systematic mappings between syntax and semantics, DALL-E 2 is unable to reliably infer meanings that are consistent with the syntax. These results challenge recent claims concerning the capacity of such systems to understand of human language. We make available the full set of test materials as a benchmark for future testing.
On Universality of the S Combinator
Venue: arXiv cs.CC 2022
Year: 2022

In combinatory logic it is known that the set of two combinators K and S are universal; in the sense that any other combinator can be expressed in terms of these two. K combinator can not be expressed only in terms of the S combinator. This will answer a question raised by Stephen Wolfram as ``Is the S combinator on its own computation universal?''
Multi-Agent Path Finding via Tree LSTM
Venue: arXiv cs.AI 2022
Year: 2022

In recent years, Multi-Agent Path Finding (MAPF) has attracted attention from the fields of both Operations Research (OR) and Reinforcement Learning (RL). However, in the 2021 Flatland3 Challenge, a competition on MAPF, the best RL method scored only 27.9, far less than the best OR method. This paper proposes a new RL solution to Flatland3 Challenge, which scores 125.3, several times higher than the best RL solution before. We creatively apply a novel network architecture, TreeLSTM, to MAPF in our solution. Together with several other RL techniques, including reward shaping, multiple-phase training, and centralized control, our solution is comparable to the top 2-3 OR methods.
High-Resolution Image Editing via Multi-Stage Blended Diffusion
Venue: arXiv cs.CV 2022
Year: 2022

Diffusion models have shown great results in image generation and in image editing. However, current approaches are limited to low resolutions due to the computational cost of training diffusion models for high-resolution generation. We propose an approach that uses a pre-trained low-resolution diffusion model to edit images in the megapixel range. We first use Blended Diffusion to edit the image at a low resolution, and then upscale it in multiple stages, using a super-resolution model and Blended Diffusion. Using our approach, we achieve higher visual fidelity than by only applying off the shelf super-resolution methods to the output of the diffusion model. We also obtain better global consistency than directly using the diffusion model at a higher resolution.
Multi-Type Conversational Question-Answer Generation with Closed-ended and Unanswerable Questions
Venue: arXiv cs.CL 2022
Year: 2022

Conversational question answering (CQA) facilitates an incremental and interactive understanding of a given context, but building a CQA system is difficult for many domains due to the problem of data scarcity. In this paper, we introduce a novel method to synthesize data for CQA with various question types, including open-ended, closed-ended, and unanswerable questions. We design a different generation flow for each question type and effectively combine them in a single, shared framework. Moreover, we devise a hierarchical answerability classification (hierarchical AC) module that improves quality of the synthetic data while acquiring unanswerable questions. Manual inspections show that synthetic data generated with our framework have characteristics very similar to those of human-generated conversations. Across four domains, CQA systems trained on our synthetic data indeed show good performance close to the systems trained on human-annotated data.