OpenRaiser/Intern-Atlas huggingface dataset:
HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
Venue: arXiv cs.CL 2021
Year: 2021

BERT-based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state-of-the-art results. Therefore, the NLP community conducts extensive research on understanding these models, but above all on designing effective and efficient training procedures. Several ablation studies investigating how to train BERT-like models have been carried out, but the vast majority of them concerned only the English language. A training procedure designed for English does not have to be universal and applicable to other especially typologically different languages. Therefore, this paper presents the first ablation study focused on Polish, which, unlike the isolating English language, is a fusional language. We design and thoroughly evaluate a pretraining procedure of transferring knowledge from multilingual to monolingual BERT-based models. In addition to multilingual model initialization, other factors that possibly influence pretraining are also explored, i.e. training objective, corpus size, BPE-Dropout, and pretraining length. Based on the proposed procedure, a Polish BERT-based language model -- HerBERT -- is trained. This model achieves state-of-the-art results on multiple downstream tasks.
Inspection of ratcheting models for pathological error sensitivity and overparametrization
Venue: arXiv cs.CE 2021
Year: 2021

Accurate analysis of plastic strain accumulation under stress-controlled cyclic loading is vital for numerous engineering applications. Typically, models of plastic ratcheting are calibrated against available experimental data. Since actual experiments are not exactly accurate, one should check the identification protocols for pathological dependencies on experimental errors. In this paper, a step-by-step algorithm is presented to estimate the sensitivities of identified material parameters. As a part of the sensitivity analysis method, a new mechanics-based metric in the space of material parameters is proposed especially for ratcheting-related applications. The sensitivity of material parameters to experimental errors is estimated, based on this metric. For demonstration purposes, the accumulation of irreversible strain in the titanium alloy VT6 (Russian analog of Ti-6Al-4V) is analysed. Three types of phenomenological models of plastic ratcheting are considered. They are the Armstrong-Frederick model as well as the first and the second Ohno-Wang models. Based on real data, a new rule of isotropic hardening is proposed for greater accuracy of simulation. The ability of the sensitivity analysis to determine reliable and unreliable parameters is demonstrated. The plausibility of the new method is checked by alternative approaches, like the consideration of correlation matrices and validation of identified parameters on "unseen" data. A relation between pathological error sensitivity and overparametrization is established.
Robust Reconfigurable Intelligent Surfaces via Invariant Risk and Causal Representations
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, the problem of robust reconfigurable intelligent surface (RIS) system design under changes in data distributions is investigated. Using the notion of invariant risk minimization (IRM), an invariant causal representation across multiple environments is used such that the predictor is simultaneously optimal for each environment. A neural network-based solution is adopted to seek the predictor and its performance is validated via simulations against an empirical risk minimization-based design. Results show that leveraging invariance yields more robustness against unseen and out-of-distribution testing environments.
Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction
Venue: arXiv cs.RO 2021
Year: 2021

Deep Neural Networks (DNNs) which are trained end-to-end have been successfully applied to solve complex problems that we have not been able to solve in past decades. Autonomous driving is one of the most complex problems which is yet to be completely solved and autonomous racing adds more complexity and exciting challenges to this problem. Towards the challenge of applying end-to-end learning to autonomous racing, this paper shows results on two aspects: (1) Analyzing the relationship between the driving data used for training and the maximum speed at which the DNN can be successfully applied for predicting steering angle, (2) Neural network architecture and training methodology for learning steering and throttle without any feedback or recurrent connections.
GALA: Greedy ComputAtion for Linear Algebra in Privacy-Preserved Neural Networks
Venue: arXiv cs.CR 2021
Year: 2021

Machine Learning as a Service (MLaaS) is enabling a wide range of smart applications on end devices. However, privacy-preserved computation is still expensive. Our investigation has found that the most time-consuming component of the HE-based linear computation is a series of Permutation (Perm) operations that are imperative for dot product and convolution in privacy-preserved MLaaS. To this end, we propose GALA: Greedy computAtion for Linear Algebra in privacy-preserved neural networks, which views the HE-based linear computation as a series of Homomorphic Add, Mult and Perm operations and chooses the least expensive operation in each linear computation step to reduce the overall cost. GALA makes the following contributions: (1) It introduces a row-wise weight matrix encoding and combines the share generation that is needed for the GC-based nonlinear computation, to reduce the Perm operations for the dot product; (2) It designs a first-Add-second-Perm approach (named kernel grouping) to reduce Perm operations for convolution. As such, GALA efficiently reduces the cost for the HE-based linear computation, which is a critical building block in almost all of the recent frameworks for privacy-preserved neural networks, including GAZELLE (Usenix Security'18), DELPHI (Usenix Security'20), and CrypTFlow2 (CCS'20). With its deep optimization of the HE-based linear computation, GALA can be a plug-and-play module integrated into these systems to further boost their efficiency. Our experiments show that it achieves a significant speedup up to 700x for the dot product and 14x for the convolution computation under different data dimensions. Meanwhile, GALA demonstrates an encouraging runtime boost by 2.5x, 2.7x, 3.2x, 8.3x, 7.7x, and 7.5x over GAZELLE and 6.5x, 6x, 5.7x, 4.5x, 4.2x, and 4.1x over CrypTFlow2, on AlexNet, VGG, ResNet-18, ResNet-50, ResNet-101, and ResNet-152, respectively.
A Robotic Approach towards Quantifying Epipelagic Bound Plastic Using Deep Visual Models
Venue: arXiv cs.CV 2021
Year: 2021

The quantification of positively buoyant marine plastic debris is critical to understanding how plastic litter accumulates across the world's oceans and is also crucial to identifying hotspots for targeted cleanup efforts. Currently, the most common method to quantify marine plastic is using manta trawls for manual sampling. However, this method is cost-intensive and requires human labor. This study removes the need for manual sampling by using an autonomous method using neural networks and computer vision models, which trained on images captured from various layers of the ocean column to perform real-time plastic quantification. The best performing model has a Mean Average Precision of 85% and an F1-Score of 0.89 while maintaining near real-time processing speeds ~2 ms/img.
CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
Venue: arXiv cs.LG 2021
Year: 2021

Recent advances in Deep Neural Networks (DNNs) have led to active development of specialized DNN accelerators, many of which feature a large number of processing elements laid out spatially, together with a multi-level memory hierarchy and flexible interconnect. While DNN accelerators can take advantage of data reuse and achieve high peak throughput, they also expose a large number of runtime parameters to the programmers who need to explicitly manage how computation is scheduled both spatially and temporally. In fact, different scheduling choices can lead to wide variations in performance and efficiency, motivating the need for a fast and efficient search strategy to navigate the vast scheduling space. To address this challenge, we present CoSA, a constrained-optimization-based approach for scheduling DNN accelerators. As opposed to existing approaches that either rely on designers' heuristics or iterative methods to navigate the search space, CoSA expresses scheduling decisions as a constrained-optimization problem that can be deterministically solved using mathematical optimization techniques. Specifically, CoSA leverages the regularities in DNN operators and hardware to formulate the DNN scheduling space into a mixed-integer programming (MIP) problem with algorithmic and architectural constraints, which can be solved to automatically generate a highly efficient schedule in one shot. We demonstrate that CoSA-generated schedules significantly outperform state-of-the-art approaches by a geometric mean of up to 2.5x across a wide range of DNN networks while improving the time-to-solution by 90x.
Multi-scale Image Decomposition using a Local Statistical Edge Model
Venue: arXiv cs.CV 2021
Year: 2021

We present a progressive image decomposition method based on a novel non-linear filter named Sub-window Variance filter. Our method is specifically designed for image detail enhancement purpose; this application requires extraction of image details which are small in terms of both spatial and variation scales. We propose a local statistical edge model which develops its edge awareness using spatially defined image statistics. Our decomposition method is controlled by two intuitive parameters which allow the users to define what image details to suppress or enhance. By using the summed-area table acceleration method, our decomposition pipeline is highly parallel. The proposed filter is gradient preserving and this allows our enhancement results free from the gradient-reversal artefact. In our evaluations, we compare our method in various multi-scale image detail manipulation applications with other mainstream solutions.
Does elderly enjoy playing Bingo with a robot? A case study with the humanoid robot Nadine
Venue: arXiv cs.RO 2021
Year: 2021

There are considerable advancements in medical health care in recent years, resulting in rising older population. As the workforce for such a population is not keeping pace, there is an urgent need to address this problem. Having robots to stimulating recreational activities for older adults can reduce the workload for caretakers and give them time to address the emotional needs of the elderly. In this paper, we investigate the effects of the humanoid social robot Nadine as an activity host for the elderly. This study aims to analyse if the elderly feels comfortable and enjoy playing game/activity with the humanoid robot Nadine. We propose to evaluate this by placing Nadine humanoid social robot in a nursing home as a caretaker where she hosts bingo game. We record sessions with and without Nadine to understand the difference and acceptance of these two scenarios. We use computer vision methods to analyse the activities of the elderly to detect emotions and their involvement in the game. We envision that such humanoid robots will make recreational activities more readily available for the elderly. Our results present positive enforcement during recreational activity, Bingo, in the presence of Nadine.
Don't forget your classics: Systematizing 45 years of Ancestry for Security API Usability Recommendations
Venue: arXiv cs.CR 2021
Year: 2021

Producing secure software is challenging. The poor usability of security APIs makes this even harder. Many recommendations have been proposed to support developers by improving the usability of cryptography libraries and APIs; rooted in wider best practice guidance in software engineering and API design. In this SLR, we systematize knowledge regarding these recommendations. We identify and analyze 65 papers spanning 45 years, offering a total of 883 recommendations.We undertake a thematic analysis to identify 7 core ways to improve usability of APIs. We find that most of the recommendations focus on helping API developers to construct and structure their code and make it more usable and easier for programmers to understand. There is less focus, however, on documentation, writing requirements, code quality assessment and the impact of organizational software development practices. By tracing and analyzing paper ancestry, we map how this knowledge becomes validated and translated over time.We find evidence that less than a quarter of all API usability recommendations are empirically validated, and that recommendations specific to usable security APIs lag even further behind in this regard.
Graph structure based Heuristics for Optimal Targeting in Social Networks
Venue: arXiv cs.SI 2021
Year: 2021

We consider a dynamic model for competition in a social network, where two strategic agents have fixed beliefs and the non-strategic/regular agents adjust their states according to a distributed consensus protocol. We suppose that one strategic agent must identify k+ target agents in the network in order to maximally spread its own opinion and alter the average opinion that eventually emerges. In the literature, this problem is cast as the maximization of a set function and, leveraging on the submodular property, is solved in a greedy manner by solving k+ separate single targeting problems. Our main contribution is to exploit the underlying graph structure to build more refined heuristics. As a first instance, we provide the analytical solution for the optimal targeting problem over complete graphs. This result provides a rule to understand whether it is convenient or not to block the opponent's influence by targeting the same nodes. The argument is then extended to generic graphs leading to more accurate solutions compared to a simple greedy approach. As a second instance, by electrical analogy we provide the analytical solution of the single targeting problem for the line graph and derive some useful properties of the objective function for trees. Inspired by these findings, we define a new algorithm which selects the optimal solution on trees in a much faster way with respect to a brute-force approach and works well also over tree-like/sparse graphs. The proposed heuristics are then compared to zero-cost heuristics on different random generated graphs and real social networks. Summarizing, our results suggest a scheme that tells which algorithm is more suitable in terms of accuracy and computational complexity, based on the density of the graphs and its degree distribution.
Impact of individual rater style on deep learning uncertainty in medical imaging segmentation
Venue: arXiv cs.CV 2021
Year: 2021

While multiple studies have explored the relation between inter-rater variability and deep learning model uncertainty in medical segmentation tasks, little is known about the impact of individual rater style. This study quantifies rater style in the form of bias and consistency and explores their impacts when used to train deep learning models. Two multi-rater public datasets were used, consisting of brain multiple sclerosis lesion and spinal cord grey matter segmentation. On both datasets, results show a correlation ($R^2 = 0.60$ and $0.93$) between rater bias and deep learning uncertainty. The impact of label fusion between raters' annotations on this relationship is also explored, and we show that multi-center consensuses are more effective than single-center consensuses to reduce uncertainty, since rater style is mostly center-specific.
Foundations of Intelligence in Natural and Artificial Systems: A Workshop Report
Venue: arXiv cs.AI 2021
Year: 2021

In March of 2021, the Santa Fe Institute hosted a workshop as part of its Foundations of Intelligence in Natural and Artificial Systems project. This project seeks to advance the field of artificial intelligence by promoting interdisciplinary research on the nature of intelligence. During the workshop, speakers from diverse disciplines gathered to develop a taxonomy of intelligence, articulating their own understanding of intelligence and how their research has furthered that understanding. In this report, we summarize the insights offered by each speaker and identify the themes that emerged during the talks and subsequent discussions.
Information Limits for Detecting a Subhypergraph
Venue: arXiv cs.IT 2021
Year: 2021

We consider the problem of recovering a subhypergraph based on an observed adjacency tensor corresponding to a uniform hypergraph. The uniform hypergraph is assumed to contain a subset of vertices called as subhypergraph. The edges restricted to the subhypergraph are assumed to follow a different probability distribution than other edges. We consider both weak recovery and exact recovery of the subhypergraph, and establish information-theoretic limits in each case. Specifically, we establish sharp conditions for the possibility of weakly or exactly recovering the subhypergraph from an information-theoretic point of view. These conditions are fundamentally different from their counterparts derived in hypothesis testing literature.
Portfolio Growth Rate and Ergodic Capacity of Fading Channels
Venue: arXiv cs.IT 2021
Year: 2021

A relationship between the growth-rate of logoptimal portfolios and capacity of fading single-input multiple output (SIMO) channels are established. Using this relation, stock vector stochastic processes that model the investment environments are stochastically ordered using different criteria. The presence of side information (SI) is considered, and a bound on the gains in the growth-rate due to SI is derived along with data processing inequality and convexity properties. A statistical test on the usefulness of SI that does not require the computation of the optimal portfolio vector in the presence of SI is introduced and its several variants are discussed.
Scalable Graph Neural Network Training: The Case for Sampling
Venue: arXiv cs.LG 2021
Year: 2021

Graph Neural Networks (GNNs) are a new and increasingly popular family of deep neural network architectures to perform learning on graphs. Training them efficiently is challenging due to the irregular nature of graph data. The problem becomes even more challenging when scaling to large graphs that exceed the capacity of single devices. Standard approaches to distributed DNN training, such as data and model parallelism, do not directly apply to GNNs. Instead, two different approaches have emerged in the literature: whole-graph and sample-based training. In this paper, we review and compare the two approaches. Scalability is challenging with both approaches, but we make a case that research should focus on sample-based training since it is a more promising approach. Finally, we review recent systems supporting sample-based training.
Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
Venue: arXiv cs.CV 2021
Year: 2021

Attention mechanisms, especially self-attention, have played an increasingly important role in deep feature representation for visual tasks. Self-attention updates the feature at each position by computing a weighted sum of features using pair-wise affinities across all positions to capture the long-range dependency within a single sample. However, self-attention has quadratic complexity and ignores potential correlation between different samples. This paper proposes a novel attention mechanism which we call external attention, based on two external, small, learnable, shared memories, which can be implemented easily by simply using two cascaded linear layers and two normalization layers; it conveniently replaces self-attention in existing popular architectures. External attention has linear complexity and implicitly considers the correlations between all data samples. We further incorporate the multi-head mechanism into external attention to provide an all-MLP architecture, external attention MLP (EAMLP), for image classification. Extensive experiments on image classification, object detection, semantic segmentation, instance segmentation, image generation, and point cloud analysis reveal that our method provides results comparable or superior to the self-attention mechanism and some of its variants, with much lower computational and memory costs.
A Unified Transferable Model for ML-Enhanced DBMS
Venue: arXiv cs.DB 2021
Year: 2021

Recently, the database management system (DBMS) community has witnessed the power of machine learning (ML) solutions for DBMS tasks. Despite their promising performance, these existing solutions can hardly be considered satisfactory. First, these ML-based methods in DBMS are not effective enough because they are optimized on each specific task, and cannot explore or understand the intrinsic connections between tasks. Second, the training process has serious limitations that hinder their practicality, because they need to retrain the entire model from scratch for a new DB. Moreover, for each retraining, they require an excessive amount of training data, which is very expensive to acquire and unavailable for a new DB. We propose to explore the transferabilities of the ML methods both across tasks and across DBs to tackle these fundamental drawbacks. In this paper, we propose a unified model MTMLF that uses a multi-task training procedure to capture the transferable knowledge across tasks and a pre-train fine-tune procedure to distill the transferable meta knowledge across DBs. We believe this paradigm is more suitable for cloud DB service, and has the potential to revolutionize the way how ML is used in DBMS. Furthermore, to demonstrate the predicting power and viability of MTMLF, we provide a concrete and very promising case study on query optimization tasks. Last but not least, we discuss several concrete research opportunities along this line of work.
Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking
Venue: arXiv cs.CV 2021
Year: 2021

Multi-object tracking (MOT) is an essential task in the computer vision field. With the fast development of deep learning technology in recent years, MOT has achieved great improvement. However, some challenges still remain, such as sensitiveness to occlusion, instability under different lighting conditions, non-robustness to deformable objects, etc. To address such common challenges in most of the existing trackers, in this paper, a tracklet booster algorithm is proposed, which can be built upon any other tracker. The motivation is simple and straightforward: split tracklets on potential ID-switch positions and then connect multiple tracklets into one if they are from the same object. In other words, the tracklet booster consists of two parts, i.e., Splitter and Connector. First, an architecture with stacked temporal dilated convolution blocks is employed for the splitting position prediction via label smoothing strategy with adaptive Gaussian kernels. Then, a multi-head self-attention based encoder is exploited for the tracklet embedding, which is further used to connect tracklets into larger groups. We conduct sufficient experiments on MOT17 and MOT20 benchmark datasets, which demonstrates promising results. Combined with the proposed tracklet booster, existing trackers usually can achieve large improvements on the IDF1 score, which shows the effectiveness of the proposed method.