OpenRaiser/Intern-Atlas huggingface dataset:
Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search
Venue: arXiv cs.CV 2022
Year: 2022

Different from other deep scalable architecture-based NAS approaches, Broad Neural Architecture Search (BNAS) proposes a broad scalable architecture which consists of convolution and enhancement blocks, dubbed Broad Convolutional Neural Network (BCNN), as the search space for amazing efficiency improvement. BCNN reuses the topologies of cells in the convolution block so that BNAS can employ few cells for efficient search. Moreover, multi-scale feature fusion and knowledge embedding are proposed to improve the performance of BCNN with shallow topology. However, BNAS suffers some drawbacks: 1) insufficient representation diversity for feature fusion and enhancement and 2) time consumption of knowledge embedding design by human experts. This paper proposes Stacked BNAS, whose search space is a developed broad scalable architecture named Stacked BCNN, with better performance than BNAS. On the one hand, Stacked BCNN treats mini BCNN as a basic block to preserve comprehensive representation and deliver powerful feature extraction ability. For multi-scale feature enhancement, each mini BCNN feeds the outputs of deep and broad cells to the enhancement cell. For multi-scale feature fusion, each mini BCNN feeds the outputs of deep, broad and enhancement cells to the output node. On the other hand, Knowledge Embedding Search (KES) is proposed to learn appropriate knowledge embeddings in a differentiable way. Moreover, the basic unit of KES is an over-parameterized knowledge embedding module that consists of all possible candidate knowledge embeddings. Experimental results show that 1) Stacked BNAS obtains better performance than BNAS-v2 on both CIFAR-10 and ImageNet, 2) the proposed KES algorithm contributes to reducing the parameters of the learned architecture with satisfactory performance, and 3) Stacked BNAS delivers a state-of-the-art efficiency of 0.02 GPU days.
A Teacher-Student Markov Decision Process-based Framework for Online Correctional Learning
Venue: arXiv cs.LG 2022
Year: 2022

A classical learning setting typically concerns an agent/student who collects data, or observations, from a system in order to estimate a certain property of interest. Correctional learning is a type of cooperative teacher-student framework where a teacher, who has partial knowledge about the system, has the ability to observe and alter (correct) the observations received by the student in order to improve the accuracy of its estimate. In this paper, we show how the variance of the estimate of the student can be reduced with the help of the teacher. We formulate the corresponding online problem - where the teacher has to decide, at each time instant, whether or not to change the observations due to a limited budget - as a Markov decision process, from which the optimal policy is derived using dynamic programming. We validate the framework in numerical experiments, and compare the optimal online policy with the one from the batch setting.
Training Data Reduction for Performance Models of Data Analytics Jobs in the Cloud
Venue: arXiv cs.DC 2022
Year: 2022

Distributed dataflow systems like Apache Flink and Apache Spark simplify processing large amounts of data on clusters in a data-parallel manner. However, choosing suitable cluster resources for distributed dataflow jobs in both type and number is difficult, especially for users who do not have access to previous performance metrics. One approach to overcoming this issue is to have users share runtime metrics to train context-aware performance models that help find a suitable configuration for the job at hand. A problem when sharing runtime data instead of trained models or model parameters is that the data size can grow substantially over time. This paper examines several clustering techniques to minimize training data size while keeping the associated performance models accurate. Our results indicate that efficiency gains in data transfer, storage, and model training can be achieved through training data reduction. In the evaluation of our solution on a dataset of runtime data from 930 unique distributed dataflow jobs, we observed that, on average, a 75% data reduction only increases prediction errors by one percentage point.
High-Rate Convolutional Codes with CRC-Aided List Decoding for Short Blocklengths
Venue: arXiv cs.IT 2022
Year: 2022

Recently, rate-$1/ω$ zero-terminated and tail-biting convolutional codes (ZTCCs and TBCCs) with cyclic-redundancy-check (CRC)-aided list decoding have been shown to closely approach the random-coding union (RCU) bound for short blocklengths. This paper designs CRCs for rate-$(ω-1)/ω$ CCs with short blocklengths, considering both the ZT and TB cases. The CRC design seeks to optimize the frame error rate (FER) performance of the code resulting from the concatenation of the CRC and the CC. Utilization of the dual trellis proposed by Yamada \emph{et al.} lowers the complexity of CRC-aided serial list Viterbi decoding (SLVD) of ZTCCs and TBCCs. CRC-aided SLVD of the TBCCs closely approaches the RCU bound at a blocklength of $128$.
Triggerless Backdoor Attack for NLP Tasks with Clean Labels
Venue: arXiv cs.CL 2022
Year: 2022

Backdoor attacks pose a new threat to NLP models. A standard strategy to construct poisoned data in backdoor attacks is to insert triggers (e.g., rare words) into selected sentences and alter the original label to a target label. This strategy comes with a severe flaw of being easily detected from both the trigger and the label perspectives: the trigger injected, which is usually a rare word, leads to an abnormal natural language expression, and thus can be easily detected by a defense model; the changed target label leads the example to be mistakenly labeled and thus can be easily detected by manual inspections. To deal with this issue, in this paper, we propose a new strategy to perform textual backdoor attacks which do not require an external trigger, and the poisoned samples are correctly labeled. The core idea of the proposed strategy is to construct clean-labeled examples, whose labels are correct but can lead to test label changes when fused with the training set. To generate poisoned clean-labeled examples, we propose a sentence generation model based on the genetic algorithm to cater to the non-differentiable characteristic of text data. Extensive experiments demonstrate that the proposed attacking strategy is not only effective, but more importantly, hard to defend due to its triggerless and clean-labeled nature. Our work marks the first step towards developing triggerless attacking strategies in NLP.
Fast First-Order Methods for Monotone Strongly DR-Submodular Maximization
Venue: arXiv cs.LG 2022
Year: 2022

Continuous DR-submodular functions are a class of functions that satisfy the Diminishing Returns (DR) property, which implies that they are concave along non-negative directions. Existing works have studied monotone continuous DR-submodular maximization subject to a convex constraint and have proposed efficient algorithms with approximation guarantees. However, in many applications, e.g., computing the stability number of a graph and mean-field inference for probabilistic log-submodular models, the DR-submodular function has the additional property of being \emph{strongly} concave along non-negative directions that could be utilized for obtaining faster convergence rates. In this paper, we first introduce and characterize the class of \emph{strongly DR-submodular} functions and show how such a property implies strong concavity along non-negative directions. Then, we study $L$-smooth monotone strongly DR-submodular functions that have bounded curvature, and we show how to exploit such additional structure to obtain algorithms with improved approximation guarantees and faster convergence rates for the maximization problem. In particular, we propose the SDRFW algorithm that matches the provably optimal $1-\frac{c}{e}$ approximation ratio after only $\lceil\frac{L}μ\rceil$ iterations, where $c\in[0,1]$ and $μ\geq 0$ are the curvature and the strong DR-submodularity parameter. Furthermore, we study the Projected Gradient Ascent (PGA) method for this problem and provide a refined analysis of the algorithm with an improved $\frac{1}{1+c}$ approximation ratio and a linear convergence rate. Given that both algorithms require knowledge of the smoothness parameter $L$, we provide a \emph{novel} characterization of $L$ for DR-submodular functions showing that in many cases, computing $L$ could be formulated as a convex problem, i.e., a geometric program, that could be solved efficiently.
Inverse-Weighted Survival Games
Venue: arXiv cs.LG 2022
Year: 2022

Deep models trained through maximum likelihood have achieved state-of-the-art results for survival analysis. Despite this training scheme, practitioners evaluate models under other criteria, such as binary classification losses at a chosen set of time horizons, e.g. Brier score (BS) and Bernoulli log likelihood (BLL). Models trained with maximum likelihood may have poor BS or BLL since maximum likelihood does not directly optimize these criteria. Directly optimizing criteria like BS requires inverse-weighting by the censoring distribution. However, estimating the censoring model under these metrics requires inverse-weighting by the failure distribution. The objective for each model requires the other, but neither are known. To resolve this dilemma, we introduce Inverse-Weighted Survival Games. In these games, objectives for each model are built from re-weighted estimates featuring the other model, where the latter is held fixed during training. When the loss is proper, we show that the games always have the true failure and censoring distributions as a stationary point. This means models in the game do not leave the correct distributions once reached. We construct one case where this stationary point is unique. We show that these games optimize BS on simulations and then apply these principles on real world cancer and critically-ill patient data.
A Survey on Adversarial Attacks for Malware Analysis
Venue: arXiv cs.CR 2022
Year: 2022

Machine learning has witnessed tremendous growth in its adoption and advancement in the last decade. The evolution of machine learning from traditional algorithms to modern deep learning architectures has shaped the way today's technology functions. Its unprecedented ability to discover knowledge/patterns from unstructured data and automate the decision-making process led to its application in wide domains. High flying machine learning arena has been recently pegged back by the introduction of adversarial attacks. Adversaries are able to modify data, maximizing the classification error of the models. The discovery of blind spots in machine learning models has been exploited by adversarial attackers by generating subtle intentional perturbations in test samples. Increasing dependency on data has paved the blueprint for ever-high incentives to camouflage machine learning models. To cope with probable catastrophic consequences in the future, continuous research is required to find vulnerabilities in form of adversarial and design remedies in systems. This survey aims at providing the encyclopedic introduction to adversarial attacks that are carried out against malware detection systems. The paper will introduce various machine learning techniques used to generate adversarial and explain the structure of target files. The survey will also model the threat posed by the adversary and followed by brief descriptions of widely accepted adversarial algorithms. Work will provide a taxonomy of adversarial evasion attacks on the basis of attack domain and adversarial generation techniques. Adversarial evasion attacks carried out against malware detectors will be discussed briefly under each taxonomical headings and compared with concomitant researches. Analyzing the current research challenges in an adversarial generation, the survey will conclude by pinpointing the open future research directions.
Multiset Neurons
Venue: arXiv cs.LG 2022
Year: 2022

The present work reports a comparative performance of artificial neurons obtained in terms of the real-valued Jaccard and coincidence similarity indices and respectively derived functionals. The interiority index and classic cross-correlation are also included for comparison purposes. After presenting the basic concepts related to real-valued multisets and the adopted similarity metrics, including the generalization of the real-valued Jaccard and coincidence indices to higher orders, we proceed to studying the response of a single neuron, not taking into account the output non-linearity (e.g.~sigmoid), respectively to the detection of gaussian two-dimensional stimulus in presence of displacement, magnification, intensity variation, noise and interference from additional patterns. It is shown that the real-valued Jaccard and coincidence approaches are substantially more robust and effective than the interiority index and the classic cross-correlation. The coincidence-based neurons are shown to have the best overall performance respectively to the considered type of data and perturbations. The potential of the multiset neurons is further illustrated with respect to the challenging problem of image segmentation, leading to impressive cost/benefit performance. The reported concepts, methods, and results, have substantial implications not only for pattern recognition and machine learning, but also regarding neurobiology and neuroscience.
CGX: Adaptive System Support for Communication-Efficient Deep Learning
Venue: arXiv cs.DC 2022
Year: 2022

The ability to scale out training workloads has been one of the key performance enablers of deep learning. The main scaling approach is data-parallel GPU-based training, which has been boosted by hardware and software support for highly efficient point-to-point communication, and in particular via hardware bandwidth overprovisioning. Overprovisioning comes at a cost: there is an order of magnitude price difference between "cloud-grade" servers with such support, relative to their popular "consumer-grade" counterparts, although single server-grade and consumer-grade GPUs can have similar computational envelopes. In this paper, we show that the costly hardware overprovisioning approach can be supplanted via algorithmic and system design, and propose a framework called CGX, which provides efficient software support for compressed communication in ML applications, for both multi-GPU single-node training, as well as larger-scale multi-node training. CGX is based on two technical advances: \emph{At the system level}, it relies on a re-developed communication stack for ML frameworks, which provides flexible, highly-efficient support for compressed communication. \emph{At the application level}, it provides \emph{seamless, parameter-free} integration with popular frameworks, so that end-users do not have to modify training recipes, nor significant training code. This is complemented by a \emph{layer-wise adaptive compression} technique which dynamically balances compression gains with accuracy preservation. CGX integrates with popular ML frameworks, providing up to 3X speedups for multi-GPU nodes based on commodity hardware, and order-of-magnitude improvements in the multi-node setting, with negligible impact on accuracy.
How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization
Venue: arXiv cs.NE 2022
Year: 2022

The success of gradient descent in ML and especially for learning neural networks is remarkable and robust. In the context of how the brain learns, one aspect of gradient descent that appears biologically difficult to realize (if not implausible) is that its updates rely on feedback from later layers to earlier layers through the same connections. Such bidirected links are relatively few in brain networks, and even when reciprocal connections exist, they may not be equi-weighted. Random Feedback Alignment (Lillicrap et al., 2016), where the backward weights are random and fixed, has been proposed as a bio-plausible alternative and found to be effective empirically. We investigate how and when feedback alignment (FA) works, focusing on one of the most basic problems with layered structure -- low-rank matrix factorization. In this problem, given a matrix $Y_{n\times m}$, the goal is to find a low rank factorization $Z_{n \times r}W_{r \times m}$ that minimizes the error $\|ZW-Y\|_F$. Gradient descent solves this problem optimally. We show that FA converges to the optimal solution when $r\ge \mbox{rank}(Y)$. We also shed light on how FA works. It is observed empirically that the forward weight matrices and (random) feedback matrices come closer during FA updates. Our analysis rigorously derives this phenomenon and shows how it facilitates convergence of FA*, a closely related variant of FA. We also show that FA can be far from optimal when $r < \mbox{rank}(Y)$. This is the first provable separation result between gradient descent and FA. Moreover, the representations found by gradient descent and FA can be almost orthogonal even when their error $\|ZW-Y\|_F$ is approximately equal. As a corollary, these results also hold for training two-layer linear neural networks when the training input is isotropic, and the output is a linear function of the input.
The Case for Learned In-Memory Joins
Venue: arXiv cs.DB 2022
Year: 2022

In-memory join is an essential operator in any database engine. It has been extensively investigated in the database literature. In this paper, we study whether exploiting the CDF-based learned models to boost the join performance is practical or not. To the best of our knowledge, we are the first to fill this gap. We investigate the usage of CDF-based partitioning and learned indexes (e.g., Recursive Model Indexes (RMI) and RadixSpline) in the three join categories; indexed nested loop join (INLJ), sort-based joins (SJ) and hash-based joins (HJ). Our study shows that there is a room to improve the performance of INLJ and SJ categories through our proposed optimized learned variants. Our experimental analysis showed that these proposed learned variants of INLJ and SJ consistently outperform the state-of-the-art techniques.
On the coalition number of trees
Venue: arXiv cs.DM 2022
Year: 2022

Let $G$ be a graph with vertex set $V$ and of order $n = |V|$, and let $δ(G)$ and $Δ(G)$ be the minimum and maximum degree of $G$, respectively. Two disjoint sets $V_1, V_2 \subseteq V$ form a coalition in $G$ if none of them is a dominating set of $G$ but their union $V_1\cup V_2$ is. A vertex partition $Ψ=\{V_1,\ldots, V_k\}$ of $V$ is a coalition partition of $G$ if every set $V_i\in Ψ$ is either a dominating set of $G$ with the cardinality $|V_i|=1$, or is not a dominating set but for some $V_j\in Ψ$, $V_i$ and $V_j$ form a coalition. The maximum cardinality of a coalition partition of $G$ is the coalition number $\mathcal{C}(G)$ of $G$. Given a coalition partition $Ψ= \{V_1, \ldots, V_k\}$ of $G$, a coalition graph $\CG(G, Ψ)$ is associated on $Ψ$ such that there is a one-to-one correspondence between its vertices and the members of $Ψ$, where two vertices of $\CG(G, Ψ)$ are adjacent if and only if the corresponding sets form a coalition in $G$. In this paper, we partially solve one of the open problems posed in Haynes et al. \cite{coal0} and we solve two open problems posed by Haynes et al. \cite{coal1}. We characterize all graphs $G$ with $δ(G) \le 1$ and $\mathcal{C}(G)=n$, and we characterize all trees $T$ with $\mathcal{C}(T)=n-1$. We determine the number of coalition graphs that can be defined by all coalition partitions of a given path. Furthermore, we show that there is no universal coalition path, a path whose coalition partitions defines all possible coalition graphs.
Addition Machines, Automatic Functions and Open Problems of Floyd and Knuth
Venue: arXiv cs.CC 2022
Year: 2022

Floyd and Knuth investigated in 1990 register machines which can add, subtract and compare integers as primitive operations. They asked whether their current bound on the number of registers for multiplying and dividing fast (running in time linear in the size of the input) can be improved and whether one can output fast the powers of two summing up to a positive integer in subquadratic time. Both questions are answered positively. Furthermore, it is shown that every function computed by only one register is automatic and that automatic functions with one input can be computed with four registers in linear time; automatic functions with a larger number of inputs can be computed with 5 registers in linear time. There is a nonautomatic function with one input which can be computed with two registers in linear time.
Multi-Robot Object Transport Motion Planning with a Deformable Sheet
Venue: arXiv cs.RO 2022
Year: 2022

Using a deformable sheet to handle objects is convenient and found in many practical applications. For object manipulation through a deformable sheet that is held by multiple mobile robots, it is a challenging task to model the object-sheet interactions. We present a computational model and algorithm to capture the object position on the deformable sheet with changing robotic team formations. A virtual variable cables model (VVCM) is proposed to simplify the modeling of the robot-sheet-object system. With the VVCM, we further present a motion planner for the robotic team to transport the object in a three-dimensional (3D) cluttered environment. Simulation and experimental results with different robot team sizes show the effectiveness and versatility of the proposed VVCM. We also compare and demonstrate the planning results to avoid the obstacle in 3D space with the other benchmark planner.
GAETS: A Graph Autoencoder Time Series Approach Towards Battery Parameter Estimation
Venue: arXiv cs.LG 2022
Year: 2022

Lithium-ion batteries are powering the ongoing transportation electrification revolution. Lithium-ion batteries possess higher energy density and favourable electrochemical properties which make it a preferable energy source for electric vehicles. Precise estimation of battery parameters (Charge capacity, voltage etc) is vital to estimate the available range in an electric vehicle. Graph-based estimation techniques enable us to understand the variable dependencies underpinning them to improve estimates. In this paper we employ Graph Neural Networks for battery parameter estimation, we introduce a unique graph autoencoder time series estimation approach. Variables in battery measurements are known to have an underlying relationship with each other in a certain correlation within variables of interest. We use graph autoencoder based on a non-linear version of NOTEARS as this allowed us to perform gradient-descent in learning the structure (instead of treating it as a combinatorial optimisation problem). The proposed architecture outperforms the state-of-the-art Graph Time Series (GTS) architecture for battery parameter estimation. We call our method GAETS (Graph AutoEncoder Time Series).