OpenRaiser/Intern-Atlas huggingface dataset:
RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
Venue: arXiv cs.CL 2021
Year: 2021

We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.
Neural Language Models for Nineteenth-Century English
Venue: arXiv cs.CL 2021
Year: 2021

We present four types of neural language models trained on a large historical dataset of books in English, published between 1760-1900 and comprised of ~5.1 billion tokens. The language model architectures include static (word2vec and fastText) and contextualized models (BERT and Flair). For each architecture, we trained a model instance using the whole dataset. Additionally, we trained separate instances on text published before 1850 for the two static models, and four instances considering different time slices for BERT. Our models have already been used in various downstream tasks where they consistently improved performance. In this paper, we describe how the models have been created and outline their reuse potential.
A Simple Proof of a New Set Disjointness with Applications to Data Streams
Venue: arXiv cs.DS 2021
Year: 2021

The multiplayer promise set disjointness is one of the most widely used problems from communication complexity in applications. In this problem there are $k$ players with subsets $S^1, \ldots, S^k$, each drawn from $\{1, 2, \ldots, n\}$, and we are promised that either the sets are (1) pairwise disjoint, or (2) there is a unique element $j$ occurring in all the sets, which are otherwise pairwise disjoint. The total communication of solving this problem with constant probability in the blackboard model is $Ω(n/k)$. We observe for most applications, it instead suffices to look at what we call the ``mostly'' set disjointness problem, which changes case (2) to say there is a unique element $j$ occurring in at least half of the sets, and the sets are otherwise disjoint. This change gives us a much simpler proof of an $Ω(n/k)$ randomized total communication lower bound, avoiding Hellinger distance and Poincare inequalities. Using this we show several new results for data streams: \begin{itemize} \item for $\ell_2$-Heavy Hitters, any $O(1)$-pass streaming algorithm in the insertion-only model for detecting if an $\eps$-$\ell_2$-heavy hitter exists requires $\min(\frac{1}{\eps^2}\log \frac{\eps^2n}δ, \frac{1}{\eps}n^{1/2})$ bits of memory, which is optimal up to a $\log n$ factor. For deterministic algorithms and constant $\eps$, this gives an $Ω(n^{1/2})$ lower bound, improving the prior $Ω(\log n)$ lower bound. We also obtain lower bounds for Zipfian distributions. \item for $\ell_p$-Estimation, $p > 2$, we show an $O(1)$-pass $Ω(n^{1-2/p} \log(1/δ))$ bit lower bound for outputting an $O(1)$-approximation with probability $1-δ$, in the insertion-only model. This is optimal, and the best previous lower bound was $Ω(n^{1-2/p} + \log(1/δ))$. \end{itemize}
View Distillation with Unlabeled Data for Extracting Adverse Drug Effects from User-Generated Data
Venue: arXiv cs.CL 2021
Year: 2021

We present an algorithm based on multi-layer transformers for identifying Adverse Drug Reactions (ADR) in social media data. Our model relies on the properties of the problem and the characteristics of contextual word embeddings to extract two views from documents. Then a classifier is trained on each view to label a set of unlabeled documents to be used as an initializer for a new classifier in the other view. Finally, the initialized classifier in each view is further trained using the initial training examples. We evaluated our model in the largest publicly available ADR dataset. The experiments testify that our model significantly outperforms the transformer-based models pretrained on domain-specific data.
Large-Scale Attribute-Object Compositions
Venue: arXiv cs.CV 2021
Year: 2021

We study the problem of learning how to predict attribute-object compositions from images, and its generalization to unseen compositions missing from the training data. To the best of our knowledge, this is a first large-scale study of this problem, involving hundreds of thousands of compositions. We train our framework with images from Instagram using hashtags as noisy weak supervision. We make careful design choices for data collection and modeling, in order to handle noisy annotations and unseen compositions. Finally, extensive evaluations show that learning to compose classifiers outperforms late fusion of individual attribute and object predictions, especially in the case of unseen attribute-object pairs.
Diacritics Restoration using BERT with Analysis on Czech language
Venue: arXiv cs.CL 2021
Year: 2021

We propose a new architecture for diacritics restoration based on contextualized embeddings, namely BERT, and we evaluate it on 12 languages with diacritics. Furthermore, we conduct a detailed error analysis on Czech, a morphologically rich language with a high level of diacritization. Notably, we manually annotate all mispredictions, showing that roughly 44% of them are actually not errors, but either plausible variants (19%), or the system corrections of erroneous data (25%). Finally, we categorize the real errors in detail. We release the code at https://github.com/ufal/bert-diacritics-restoration.
Skew Orthogonal Convolutions
Venue: arXiv cs.LG 2021
Year: 2021

Training convolutional neural networks with a Lipschitz constraint under the $l_{2}$ norm is useful for provable adversarial robustness, interpretable gradients, stable training, etc. While 1-Lipschitz networks can be designed by imposing a 1-Lipschitz constraint on each layer, training such networks requires each layer to be gradient norm preserving (GNP) to prevent gradients from vanishing. However, existing GNP convolutions suffer from slow training, lead to significant reduction in accuracy and provide no guarantees on their approximations. In this work, we propose a GNP convolution layer called Skew Orthogonal Convolution (SOC) that uses the following mathematical property: when a matrix is {\it Skew-Symmetric}, its exponential function is an {\it orthogonal} matrix. To use this property, we first construct a convolution filter whose Jacobian is Skew-Symmetric. Then, we use the Taylor series expansion of the Jacobian exponential to construct the SOC layer that is orthogonal. To efficiently implement SOC, we keep a finite number of terms from the Taylor series and provide a provable guarantee on the approximation error. Our experiments on CIFAR-10 and CIFAR-100 show that SOC allows us to train provably Lipschitz, large convolutional neural networks significantly faster than prior works while achieving significant improvements for both standard and certified robust accuracies.
Multi-Level Attentive Convoluntional Neural Network for Crowd Counting
Venue: arXiv cs.CV 2021
Year: 2021

Recently the crowd counting has received more and more attention. Especially the technology of high-density environment has become an important research content, and the relevant methods for the existence of extremely dense crowd are not optimal. In this paper, we propose a multi-level attentive Convolutional Neural Network (MLAttnCNN) for crowd counting. We extract high-level contextual information with multiple different scales applied in pooling, and use multi-level attention modules to enrich the characteristics at different layers to achieve more efficient multi-scale feature fusion, which is able to be used to generate a more accurate density map with dilated convolutions and a $1\times 1$ convolution. The extensive experiments on three available public datasets show that our proposed network achieves outperformance to the state-of-the-art approaches.
2nd-order Updates with 1st-order Complexity
Venue: arXiv cs.LG 2021
Year: 2021

It has long been a goal to efficiently compute and use second order information on a function ($f$) to assist in numerical approximations. Here it is shown how, using only basic physics and a numerical approximation, such information can be accurately obtained at a cost of ${\cal O}(N)$ complexity, where $N$ is the dimensionality of the parameter space of $f$. In this paper, an algorithm ({\em VA-Flow}) is developed to exploit this second order information, and pseudocode is presented. It is applied to two classes of problems, that of inverse kinematics (IK) and gradient descent (GD). In the IK application, the algorithm is fast and robust, and is shown to lead to smooth behavior even near singularities. For GD the algorithm also works very well, provided the cost function is locally well-described by a polynomial.
Transmission Failure Analysis of Multi-Protection Routing in Data Center Networks with Heterogeneous Edge-Core Servers
Venue: arXiv cs.DC 2021
Year: 2021

The recently proposed RCube network is a cube-based server-centric data center network (DCN), including two types of heterogeneous servers, called core and edge servers. Remarkably, it takes the latter as backup servers to deal with server failures and thus achieve high availability. This paper first points out that RCube is suitable as a candidate topology of DCNs for edge computing. Three transmission types are among core and edge servers based on the demand for applications' computation and instant response. We then employ protection routing to analyze the transmission failure of RCube DCNs. Unlike traditional protection routing, which only tolerates a single link or node failure, we use the multi-protection routing scheme to improve fault-tolerance capability. To configure a protection routing in a network, according to Tapolcai's suggestion, we need to construct two completely independent spanning trees (CISTs). A logic graph of RCube, denoted by $L$-$RCube(n,m,k)$, is a network with a recursive structure. Each basic building element consists of $n$ core servers and $m$ edge servers, where the order $k$ is the number of recursions applied in the structure. In this paper, we provide algorithms to construct $\min\{n,\lfloor(n+m)/2\rfloor\}$ CISTs in $L$-$RCube(n,m,k)$ for $n+m\geqslant 4$ and $n>1$. From a combination of the multiple CISTs, we can configure the desired multi-protection routing. In our simulation, we configure up to 10 protection routings for RCube DCNs. As far as we know, in past research, there were at most three protection routings developed in other network structures. Finally, we summarize some crucial analysis viewpoints about the transmission efficiency of DCNs with heterogeneous edge-core servers from the simulation results.
Adaptive Local Kernels Formulation of Mutual Information with Application to Active Post-Seismic Building Damage Inference
Venue: arXiv cs.LG 2021
Year: 2021

The abundance of training data is not guaranteed in various supervised learning applications. One of these situations is the post-earthquake regional damage assessment of buildings. Querying the damage label of each building requires a thorough inspection by experts, and thus, is an expensive task. A practical approach is to sample the most informative buildings in a sequential learning scheme. Active learning methods recommend the most informative cases that are able to maximally reduce the generalization error. The information theoretic measure of mutual information (MI) is one of the most effective criteria to evaluate the effectiveness of the samples in a pool-based sample selection scenario. However, the computational complexity of the standard MI algorithm prevents the utilization of this method on large datasets. A local kernels strategy was proposed to reduce the computational costs, but the adaptability of the kernels to the observed labels was not considered in the original formulation of this strategy. In this article, an adaptive local kernels methodology is developed that allows for the conformability of the kernels to the observed output data while enhancing the computational complexity of the standard MI algorithm. The proposed algorithm is developed to work on a Gaussian process regression (GPR) framework, where the kernel hyperparameters are updated after each label query using the maximum likelihood estimation. In the sequential learning procedure, the updated hyperparameters can be used in the MI kernel matrices to improve the sample suggestion performance. The advantages are demonstrated on a simulation of the 2018 Anchorage, AK, earthquake. It is shown that while the proposed algorithm enables GPR to reach acceptable performance with fewer training data, the computational demands remain lower than the standard local kernels strategy.
Inferring Temporal Logic Properties from Data using Boosted Decision Trees
Venue: arXiv cs.RO 2021
Year: 2021

Many autonomous systems, such as robots and self-driving cars, involve real-time decision making in complex environments, and require prediction of future outcomes from limited data. Moreover, their decisions are increasingly required to be interpretable to humans for safe and trustworthy co-existence. This paper is a first step towards interpretable learning-based robot control. We introduce a novel learning problem, called incremental formula and predictor learning, to generate binary classifiers with temporal logic structure from time-series data. The classifiers are represented as pairs of Signal Temporal Logic (STL) formulae and predictors for their satisfaction. The incremental property provides prediction of labels for prefix signals that are revealed over time. We propose a boosted decision-tree algorithm that leverages weak, but computationally inexpensive, learners to increase prediction and runtime performance. The effectiveness and classification accuracy of our algorithms are evaluated on autonomous-driving and naval surveillance case studies.
Grasp Planning for Flexible Production with Small Lot Sizes based on CAD models using GPIS and Bayesian Optimization
Venue: arXiv cs.RO 2021
Year: 2021

Grasp planning for multi-fingered hands is still a challenging task due to the high nonlinear quality metrics, the high dimensionality of hand posture configuration, and complex object shapes. Analytical-based grasp planning algorithms formulate the grasping problem as a constraint optimization problem using advanced convex optimization solvers. However, these are not guaranteed to find a globally optimal solution. Data-driven based algorithms utilize machine learning algorithm frameworks to learn the grasp policy using enormous training data sets. This paper presents a new approach for grasp generation by formulating a global optimization problem with Bayesian optimization. Furthermore, we parameterize the object shape utilizing the Gaussian Process Implicit Surface (GPIS) to integrate the object shape information into the optimization process. Moreover, a chart defined on the object surface is used to refine palm pose locally. We introduced a dual optimization stage to optimize the palm pose and contact points separately. We further extend the Bayesian optimization by utilizing the alternating direction method of multipliers (ADMM) to eliminate contact optimization constraints. We conduct the experiments in the graspit! Simulator that demonstrates the effectiveness of this approach quantitatively and qualitatively. Our approach achieves a 95% success rate on various commonly objects with diverse shapes, scales, and weights
Recommending Bug-fixing Comments from Issue Tracking Discussions in Support of Bug Repair
Venue: arXiv cs.SE 2021
Year: 2021

In practice, developers search for related earlier bugs and their associated discussion threads when faced with a new bug to repair. Typically, these discussion threads consist of comments and even bug-fixing comments intended to capture clues for facilitating the investigation and root cause of a new bug report. Over time, these discussions can become extensively lengthy and difficult to understand. Inevitably, these discussion threads lead to instances where bug-fixing comments intermingle with seemingly-unrelated comments. This task, however, poses further challenges when dealing with high volumes of bug reports. Large software systems are plagued by thousands of bug reports daily. Hence, it becomes time-consuming to investigate these bug reports efficiently. To address this gap, this paper builds a ranked-based automated tool that we refer it to as RETRORANK. Specifically, RETRORANK recommends bug-fixing comments from issue tracking discussion threads in the context of user query relevance, the use of positive language, and semantic relevance among comments. By using a combination of Vector Space Model (VSM), Sentiment Analysis (SA), and the TextRank Model (TR) we show how that past fixed bugs and their associated bug-fixing comments with relatively positive sentiments can semantically connect to investigate the root cause of a new bug. We evaluated our approach via a synthetic study and a user study. Results indicate that RETRORANK significantly improved performance when compared to the baseline VSM.
Elastic Shape Analysis of Brain Structures for Predictive Modeling of PTSD
Venue: arXiv cs.CV 2021
Year: 2021

There is increasing evidence on the importance of brain morphology in predicting and classifying mental disorders. However, the vast majority of current shape approaches rely heavily on vertex-wise analysis that may not successfully capture complexities of subcortical structures. Additionally, the past works do not include interactions between these structures and exposure factors. Predictive modeling with such interactions is of paramount interest in heterogeneous mental disorders such as PTSD, where trauma exposure interacts with brain shape changes to influence behavior. We propose a comprehensive framework that overcomes these limitations by representing brain substructures as continuous parameterized surfaces and quantifying their shape differences using elastic shape metrics. Using the elastic shape metric, we compute shape summaries of subcortical data and represent individual shapes by their principal scores. These representations allow visualization tools that help localize changes when these PCs are varied. Subsequently, these PCs, the auxiliary exposure variables, and their interactions are used for regression modeling. We apply our method to data from the Grady Trauma Project, where the goal is to predict clinical measures of PTSD using shapes of brain substructures. Our analysis revealed considerably greater predictive power under the elastic shape analysis than widely used approaches such as vertex-wise shape analysis and even volumetric analysis. It helped identify local deformations in brain shapes related to change in PTSD severity. To our knowledge, this is one of the first brain shape analysis approaches that can seamlessly integrate the pre-processing steps under one umbrella for improved accuracy and are naturally able to account for interactions between brain shape and additional covariates to yield superior predictive performance when modeling clinical outcomes.
KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning
Venue: arXiv cs.AI 2021
Year: 2021

Recently, deep reinforcement learning (RL) algorithms have made great progress in multi-agent domain. However, due to characteristics of RL, training for complex tasks would be resource-intensive and time-consuming. To meet this challenge, mutual learning strategy between homogeneous agents is essential, which is under-explored in previous studies, because most existing methods do not consider to use the knowledge of agent models. In this paper, we present an adaptation method of the majority of multi-agent reinforcement learning (MARL) algorithms called KnowSR which takes advantage of the differences in learning between agents. We employ the idea of knowledge distillation (KD) to share knowledge among agents to shorten the training phase. To empirically demonstrate the robustness and effectiveness of KnowSR, we performed extensive experiments on state-of-the-art MARL algorithms in collaborative and competitive scenarios. The results demonstrate that KnowSR outperforms recently reported methodologies, emphasizing the importance of the proposed knowledge sharing for MARL.
Robust Principal Component Analysis Using a Novel Kernel Related with the L1-Norm
Venue: arXiv cs.LG 2021
Year: 2021

We consider a family of vector dot products that can be implemented using sign changes and addition operations only. The dot products are energy-efficient as they avoid the multiplication operation entirely. Moreover, the dot products induce the $\ell_1$-norm, thus providing robustness to impulsive noise. First, we analytically prove that the dot products yield symmetric, positive semi-definite generalized covariance matrices, thus enabling principal component analysis (PCA). Moreover, the generalized covariance matrices can be constructed in an Energy Efficient (EEF) manner due to the multiplication-free property of the underlying vector products. We present image reconstruction examples in which our EEF PCA method result in the highest peak signal-to-noise ratios compared to the ordinary $\ell_2$-PCA and the recursive $\ell_1$-PCA.
Dynamic Dual Sampling Module for Fine-Grained Semantic Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Representation of semantic context and local details is the essential issue for building modern semantic segmentation models. However, the interrelationship between semantic context and local details is not well explored in previous works. In this paper, we propose a Dynamic Dual Sampling Module (DDSM) to conduct dynamic affinity modeling and propagate semantic context to local details, which yields a more discriminative representation. Specifically, a dynamic sampling strategy is used to sparsely sample representative pixels and channels in the higher layer, forming adaptive compact support for each pixel and channel in the lower layer. The sampled features with high semantics are aggregated according to the affinities and then propagated to detailed lower-layer features, leading to a fine-grained segmentation result with well-preserved boundaries. Experiment results on both Cityscapes and Camvid datasets validate the effectiveness and efficiency of the proposed approach. Code and models will be available at \url{x3https://github.com/Fantasticarl/DDSM}.