OpenRaiser/Intern-Atlas huggingface dataset:
Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
Venue: arXiv cs.CV 2021
Year: 2021

Quantizing weights and activations of deep neural networks is essential for deploying them in resource-constrained devices, or cloud platforms for at-scale services. While binarization is a special case of quantization, this extreme case often leads to several training difficulties, and necessitates specialized models and training methods. As a result, recent quantization methods do not provide binarization, thus losing the most resource-efficient option, and quantized and binarized networks have been distinct research areas. We examine binarization difficulties in a quantization framework and find that all we need to enable the binary training are a symmetric quantizer, good initialization, and careful hyperparameter selection. These techniques also lead to substantial improvements in multi-bit quantization. We demonstrate our unified quantization framework, denoted as UniQ, on the ImageNet dataset with various architectures such as ResNet-18,-34 and MobileNetV2. For multi-bit quantization, UniQ outperforms existing methods to achieve the state-of-the-art accuracy. In binarization, the achieved accuracy is comparable to existing state-of-the-art methods even without modifying the original architectures.
Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification
Venue: arXiv cs.CV 2021
Year: 2021

Remote sensing (RS) image scene classification task faces many challenges due to the interference from different characteristics of different geographical elements. To solve this problem, we propose a multi-branch ensemble network to enhance the feature representation ability by fusing features in final output logits and intermediate feature maps. However, simply adding branches will increase the complexity of models and decline the inference efficiency. On this issue, we embed self-distillation (SD) method to transfer knowledge from ensemble network to main-branch in it. Through optimizing with SD, main-branch will have close performance as ensemble network. During inference, we can cut other branches to simplify the whole model. In this paper, we first design compact multi-branch ensemble network, which can be trained in an end-to-end manner. Then, we insert SD method on output logits and feature maps. Compared to previous methods, our proposed architecture (ESD-MBENet) performs strongly on classification accuracy with compact design. Extensive experiments are applied on three benchmark RS datasets AID, NWPU-RESISC45 and UC-Merced with three classic baseline models, VGG16, ResNet50 and DenseNet121. Results prove that our proposed ESD-MBENet can achieve better accuracy than previous state-of-the-art (SOTA) complex models. Moreover, abundant visualization analysis make our method more convincing and interpretable.
Two-phase weakly supervised object detection with pseudo ground truth mining
Venue: arXiv cs.CV 2021
Year: 2021

Weakly Supervised Object Detection (WSOD), aiming to train detectors with only image-level dataset, has arisen increasing attention for researchers. In this project, we focus on two-phase WSOD architecture which integrates a powerful detector with a pure WSOD model. We explore the effectiveness of some representative detectors utilized as the second-phase detector in two-phase WSOD and propose a two-phase WSOD architecture. In addition, we present a strategy to establish the pseudo ground truth (PGT) used to train the second-phase detector. Unlike previous works that regard top one bounding boxes as PGT, we consider more bounding boxes to establish the PGT annotations. This alleviates the insufficient learning problem caused by the low recall of PGT. We also propose some strategies to refine the PGT during the training of the second detector. Our strategies suspend the training in specific epoch, then refine the PGT by the outputs of the second-phase detector. After that, the algorithm continues the training with the same gradients and weights as those before suspending. Elaborate experiments are conduceted on the PASCAL VOC 2007 dataset to verify the effectiveness of our methods. As results demonstrate, our two-phase architecture improves the mAP from 49.17% to 53.21% compared with the single PCL model. Additionally, the best PGT generation strategy obtains a 0.7% mAP increment. Our best refinement strategy boosts the performance by 1.74% mAP. The best results adopting all of our methods achieve 55.231% mAP which is the state-of-the-art performance.
Detecting over/under-translation errors for determining adequacy in human translations
Venue: arXiv cs.CL 2021
Year: 2021

We present a novel approach to detecting over and under translations (OT/UT) as part of adequacy error checks in translation evaluation. We do not restrict ourselves to machine translation (MT) outputs and specifically target applications with human generated translation pipeline. The goal of our system is to identify OT/UT errors from human translated video subtitles with high error recall. We achieve this without reference translations by learning a model on synthesized training data. We compare various classification networks that we trained on embeddings from pre-trained language model with our best hybrid network of GRU + CNN achieving 89.3% accuracy on high-quality human-annotated evaluation data in 8 languages.
The best laid plans or lack thereof: Security decision-making of different stakeholder groups
Venue: arXiv cs.CR 2021
Year: 2021

Cyber security requirements are influenced by the priorities and decisions of a range of stakeholders. Board members and CISOs determine strategic priorities. Managers have responsibility for resource allocation and project management. Legal professionals concern themselves with regulatory compliance. Little is understood about how the security decision-making approaches of these different stakeholders contrast, and if particular groups of stakeholders have a better appreciation of security requirements during decision-making. Are risk analysts better decision makers than CISOs? Do security experts exhibit more effective strategies than board members? This paper explores the effect that different experience and diversity of expertise has on the quality of a team's cyber security decision-making and whether teams with members from more varied backgrounds perform better than those with more focused, homogeneous skill sets. Using data from 208 sessions and 948 players of a tabletop game run in the wild by a major national organization over 16 months, we explore how choices are affected by player background (e.g.,~cyber security experts versus risk analysts, board-level decision makers versus technical experts) and different team make-ups (homogeneous teams of security experts versus various mixes). We find that no group of experts makes significantly better game decisions than anyone else, and that their biases lead them to not fully comprehend what they are defending or how the defenses work.
Hereditary rigidity, separation and density In memory of Professor I.G. Rosenberg
Venue: arXiv cs.DM 2021
Year: 2021

We continue the investigation of systems of hereditarily rigid relations started in Couceiro, Haddad, Pouzet and Schölzel [1]. We observe that on a set $V$ with $m$ elements, there is a hereditarily rigid set $\mathcal R$ made of $n$ tournaments if and only if $m(m-1)\leq 2^n$. We ask if the same inequality holds when the tournaments are replaced by linear orders. This problem has an equivalent formulation in terms of separation of linear orders. Let $h_{\rm Lin}(m)$ be the least cardinal $n$ such that there is a family $\mathcal R$ of $n$ linear orders on an $m$-element set $V$ such that any two distinct ordered pairs of distinct elements of $V$ are separated by some member of $\mathcal R$, then $ \lceil \log_2 (m(m-1))\rceil\leq h_{\rm Lin}(m)$ with equality if $m\leq 7$. We ask whether the equality holds for every $m$. We prove that $h_{\rm Lin}(m+1)\leq h_{\rm Lin}(m)+1$. If $V$ is infinite, we show that $h_{\rm Lin}(m)= \aleph_0$ for $m\leq 2^{\aleph_0}$. More generally, we prove that the two equalities $h_{\rm Lin}(m)= log_2 (m)= d({\rm Lin}(V))$ hold, where $\log_2 (m)$ is the least cardinal $μ$ such that $m\leq 2^μ$, and $d({\rm Lin}(V))$ is the topological density of the set ${\rm Lin}(V)$ of linear orders on $V$ (viewed as a subset of the power set $\mathcal{P}(V\times V)$ equipped with the product topology). These equalities follow from the {\it Generalized Continuum Hypothesis}, but we do not know whether they hold without any set theoretical hypothesis.
Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits
Venue: arXiv cs.CV 2021
Year: 2021

This paper proposes a novel knowledge distillation-based learning method to improve the classification performance of convolutional neural networks (CNNs) without a pre-trained teacher network, called exit-ensemble distillation. Our method exploits the multi-exit architecture that adds auxiliary classifiers (called exits) in the middle of a conventional CNN, through which early inference results can be obtained. The idea of our method is to train the network using the ensemble of the exits as the distillation target, which greatly improves the classification performance of the overall network. Our method suggests a new paradigm of knowledge distillation; unlike the conventional notion of distillation where teachers only teach students, we show that students can also help other students and even the teacher to learn better. Experimental results demonstrate that our method achieves significant improvement of classification performance on various popular CNN architectures (VGG, ResNet, ResNeXt, WideResNet, etc.). Furthermore, the proposed method can expedite the convergence of learning with improved stability. Our code will be available on Github.
Log-domain decoding of quantum LDPC codes over binary finite fields
Venue: arXiv cs.IT 2021
Year: 2021

A quantum stabilizer code over GF$(q)$ corresponds to a classical additive code over GF$(q^2)$ that is self-orthogonal with respect to a symplectic inner product. We study the decoding of quantum low-density parity-check (LDPC) codes over binary finite fields GF$(q=2^l)$ by the sum-product algorithm, also known as belief propagation (BP). Conventionally, a message in a nonbinary BP for quantum codes over GF$(2^l)$ represents a probability vector over GF$(2^{2l})$, inducing high decoding complexity. In this paper, we explore the property of the symplectic inner product and show that scalar messages suffice for BP decoding of nonbinary quantum codes, rather than vector messages necessary for the conventional BP. Consequently, we propose a BP decoding algorithm for quantum codes over GF$(2^l)$ by passing scalar messages so that it has low computation complexity. The algorithm is specified in log domain by using log-likelihood ratios (LLRs) of the channel statistics to have a low implementation cost. Moreover, techniques such as message normalization or offset can be naturally applied in this algorithm to mitigate the effects of short cycles to improve BP performance. This is important for nonbinary quantum codes since they may have more short cycles compared to binary quantum codes. Several computer simulations are provided to demonstrate these advantages. The scalar-based strategy can also be used to improve the BP decoding of classical linear codes over GF$(2^l)$ with many short cycles.
Channel Measurement and Path-Loss Characterization for Low-Terahertz Indoor Scenarios
Venue: arXiv cs.IT 2021
Year: 2021

TeraHertz (THz) communications are envisioned as a promising technology, owing to its unprecedented multi-GHz bandwidth. In this paper, wideband channel measurement campaigns at 140 GHz and 220 GHz are conducted in indoor scenarios including a meeting room and an office room. Directional antennas are utilized and rotated for resolving the multi-path components (MPCs) in the angular domain. Comparable path loss values are achieved in the 140 and 220 GHz bands. To investigate the large-scale fading characteristics for indoor THz communications, single-band close-in path loss models are developed. To further analyze the dependency on the frequency, two multi-band path loss models are analyzed, i.e., alpha-beta-gamma (ABG) and multi-frequency CI model with a frequency-weighted path loss exponent (CIF), between which the ABG model demonstrates a better fit with the measured data. Moreover, a coherent beam combination that constructively sums the signal amplitudes from various arrival directions can significantly reduce the path loss, in contrast with a non-coherent beam combination.
TrajeVAE: Controllable Human Motion Generation from Trajectories
Venue: arXiv cs.CV 2021
Year: 2021

The creation of plausible and controllable 3D human motion animations is a long-standing problem that requires a manual intervention of skilled artists. Current machine learning approaches can semi-automate the process, however, they are limited in a significant way: they can handle only a single trajectory of the expected motion that precludes fine-grained control over the output. To mitigate that issue, we reformulate the problem of future pose prediction into pose completion in space and time where multiple trajectories are represented as poses with missing joints. We show that such a framework can generalize to other neural networks designed for future pose prediction. Once trained in this framework, a model is capable of predicting sequences from any number of trajectories. We propose a novel transformer-like architecture, TrajeVAE, that builds on this idea and provides a versatile framework for 3D human animation. We demonstrate that TrajeVAE offers better accuracy than the trajectory-based reference approaches and methods that base their predictions on past poses. We also show that it can predict reasonable future poses even if provided only with an initial pose.
Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
Venue: arXiv cs.LG 2021
Year: 2021

Model-free or learning-based control, in particular, reinforcement learning (RL), is expected to be applied for complex robotic tasks. Traditional RL requires a policy to be optimized is state-dependent, that means, the policy is a kind of feedback (FB) controllers. Due to the necessity of correct state observation in such a FB controller, it is sensitive to sensing failures. To alleviate this drawback of the FB controllers, feedback error learning integrates one of them with a feedforward (FF) controller. RL can be improved by dealing with the FB/FF policies, but to the best of our knowledge, a methodology for learning them in a unified manner has not been developed. In this paper, we propose a new optimization problem for optimizing both the FB/FF policies simultaneously. Inspired by control as inference, the optimization problem considers minimization/maximization of divergences between trajectory, predicted by the composed policy and a stochastic dynamics model, and optimal/non-optimal trajectories. By approximating the stochastic dynamics model using variational method, we naturally derive a regularization between the FB/FF policies. In numerical simulations and a robot experiment, we verified that the proposed method can stably optimize the composed policy even with the different learning law from the traditional RL. In addition, we demonstrated that the FF policy is robust to the sensing failures and can hold the optimal motion. Attached video is also uploaded on youtube: https://youtu.be/zLL4uXIRmrE
Learning with Neural Tangent Kernels in Near Input Sparsity Time
Venue: arXiv cs.LG 2021
Year: 2021

The Neural Tangent Kernel (NTK) characterizes the behavior of infinitely wide neural nets trained under least squares loss by gradient descent. However, despite its importance, the super-quadratic runtime of kernel methods limits the use of NTK in large-scale learning tasks. To accelerate kernel machines with NTK, we propose a near input sparsity time algorithm that maps the input data to a randomized low-dimensional feature space so that the inner product of the transformed data approximates their NTK evaluation. Our transformation works by sketching the polynomial expansions of arc-cosine kernels. Furthermore, we propose a feature map for approximating the convolutional counterpart of the NTK, which can transform any image using a runtime that is only linear in the number of pixels. We show that in standard large-scale regression and classification tasks a linear regressor trained on our features outperforms trained Neural Nets and Nystrom approximation of NTK kernel.
WakaVT: A Sequential Variational Transformer for Waka Generation
Venue: arXiv cs.CL 2021
Year: 2021

Poetry generation has long been a challenge for artificial intelligence. In the scope of Japanese poetry generation, many researchers have paid attention to Haiku generation, but few have focused on Waka generation. To further explore the creative potential of natural language generation systems in Japanese poetry creation, we propose a novel Waka generation model, WakaVT, which automatically produces Waka poems given user-specified keywords. Firstly, an additive mask-based approach is presented to satisfy the form constraint. Secondly, the structures of Transformer and variational autoencoder are integrated to enhance the quality of generated content. Specifically, to obtain novelty and diversity, WakaVT employs a sequence of latent variables, which effectively captures word-level variability in Waka data. To improve linguistic quality in terms of fluency, coherence, and meaningfulness, we further propose the fused multilevel self-attention mechanism, which properly models the hierarchical linguistic structure of Waka. To the best of our knowledge, we are the first to investigate Waka generation with models based on Transformer and/or variational autoencoder. Both objective and subjective evaluation results demonstrate that our model outperforms baselines significantly.
Idris 2: Quantitative Type Theory in Practice
Venue: arXiv cs.PL 2021
Year: 2021

Dependent types allow us to express precisely what a function is intended to do. Recent work on Quantitative Type Theory (QTT) extends dependent type systems with linearity, also allowing precision in expressing when a function can run. This is promising, because it suggests the ability to design and reason about resource usage protocols, such as we might find in distributed and concurrent programming, where the state of a communication channel changes throughout program execution. As yet, however, there has not been a full-scale programming language with which to experiment with these ideas. Idris 2 is a new version of the dependently typed language Idris, with a new core language based on QTT, supporting linear and dependent types. In this paper, we introduce Idris 2, and describe how QTT has influenced its design. We give examples of the benefits of QTT in practice including: expressing which data is erased at run time, at the type level; and, resource tracking in the type system leading to type-safe concurrent programming with session types.
PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN
Venue: arXiv cs.LG 2021
Year: 2021

We introduce PyVertical, a framework supporting vertical federated learning using split neural networks. The proposed framework allows a data scientist to train neural networks on data features vertically partitioned across multiple owners while keeping raw data on an owner's device. To link entities shared across different datasets' partitions, we use Private Set Intersection on IDs associated with data points. To demonstrate the validity of the proposed framework, we present the training of a simple dual-headed split neural network for a MNIST classification task, with data samples vertically distributed across two data owners and a data scientist.
Deep Reinforcement Learning for Constrained Field Development Optimization in Subsurface Two-phase Flow
Venue: arXiv cs.LG 2021
Year: 2021

We present a deep reinforcement learning-based artificial intelligence agent that could provide optimized development plans given a basic description of the reservoir and rock/fluid properties with minimal computational cost. This artificial intelligence agent, comprising of a convolutional neural network, provides a mapping from a given state of the reservoir model, constraints, and economic condition to the optimal decision (drill/do not drill and well location) to be taken in the next stage of the defined sequential field development planning process. The state of the reservoir model is defined using parameters that appear in the governing equations of the two-phase flow. A feedback loop training process referred to as deep reinforcement learning is used to train an artificial intelligence agent with such a capability. The training entails millions of flow simulations with varying reservoir model descriptions (structural, rock and fluid properties), operational constraints, and economic conditions. The parameters that define the reservoir model, operational constraints, and economic conditions are randomly sampled from a defined range of applicability. Several algorithmic treatments are introduced to enhance the training of the artificial intelligence agent. After appropriate training, the artificial intelligence agent provides an optimized field development plan instantly for new scenarios within the defined range of applicability. This approach has advantages over traditional optimization algorithms (e.g., particle swarm optimization, genetic algorithm) that are generally used to find a solution for a specific field development scenario and typically not generalizable to different scenarios.
Domain-Adversarial Training of Self-Attention Based Networks for Land Cover Classification using Multi-temporal Sentinel-2 Satellite Imagery
Venue: arXiv cs.CV 2021
Year: 2021

The increasing availability of large-scale remote sensing labeled data has prompted researchers to develop increasingly precise and accurate data-driven models for land cover and crop classification (LC&CC). Moreover, with the introduction of self-attention and introspection mechanisms, deep learning approaches have shown promising results in processing long temporal sequences in the multi-spectral domain with a contained computational request. Nevertheless, most practical applications cannot rely on labeled data, and in the field, surveys are a time consuming solution that poses strict limitations to the number of collected samples. Moreover, atmospheric conditions and specific geographical region characteristics constitute a relevant domain gap that does not allow direct applicability of a trained model on the available dataset to the area of interest. In this paper, we investigate adversarial training of deep neural networks to bridge the domain discrepancy between distinct geographical zones. In particular, we perform a thorough analysis of domain adaptation applied to challenging multi-spectral, multi-temporal data, accurately highlighting the advantages of adapting state-of-the-art self-attention based models for LC&CC to different target zones where labeled data are not available. Extensive experimentation demonstrated significant performance and generalization gain in applying domain-adversarial training to source and target regions with marked dissimilarities between the distribution of extracted features.