OpenRaiser/Intern-Atlas huggingface dataset:
Feature Flow Regularization: Improving Structured Sparsity in Deep Neural Networks
Venue: arXiv cs.CV 2021
Year: 2021

Pruning is a model compression method that removes redundant parameters in deep neural networks (DNNs) while maintaining accuracy. Most available filter pruning methods require complex treatments such as iterative pruning, features statistics/ranking, or additional optimization designs in the training process. In this paper, we propose a simple and effective regularization strategy from a new perspective of evolution of features, which we call feature flow regularization (FFR), for improving structured sparsity and filter pruning in DNNs. Specifically, FFR imposes controls on the gradient and curvature of feature flow along the neural network, which implicitly increases the sparsity of the parameters. The principle behind FFR is that coherent and smooth evolution of features will lead to an efficient network that avoids redundant parameters. The high structured sparsity obtained from FFR enables us to prune filters effectively. Experiments with VGGNets, ResNets on CIFAR-10/100, and Tiny ImageNet datasets demonstrate that FFR can significantly improve both unstructured and structured sparsity. Our pruning results in terms of reduction of parameters and FLOPs are comparable to or even better than those of state-of-the-art pruning methods.
Meta-research on COVID-19: An overview of the early trends
Venue: arXiv cs.DL 2021
Year: 2021

COVID-19 is having a dramatic impact on research and researchers. The pandemic has underlined the severity of known challenges in research and surfaced new ones, but also accelerated the adoption of innovations and manifested new opportunities. This review considers early trends emerging from meta-research on COVID-19. In particular, it focuses on the following topics: i) mapping COVID-19 research; ii) data and machine learning; iii) research practices including open access and open data, reviewing, publishing and funding; iv) communicating research to the public; v) the impact of COVID-19 on researchers, in particular with respect to gender and career trajectories. This overview finds that most early meta-research on COVID-19 has been reactive and focused on short-term questions, while more recently a shift to consider the long-term consequences of COVID-19 is taking place. Based on these findings, the author speculates that some aspects of doing research during COVID-19 are more likely to persist than others. These include: the shift to virtual for academic events such as conferences; the use of openly accessible pre-prints; the `datafication' of scholarly literature and consequent broader adoption of machine learning in science communication; the public visibility of research and researchers on social and online media.
Forced Variational Integrator Networks for Prediction and Control of Mechanical Systems
Venue: arXiv cs.LG 2021
Year: 2021

As deep learning becomes more prevalent for prediction and control of real physical systems, it is important that these overparameterized models are consistent with physically plausible dynamics. This elicits a problem with how much inductive bias to impose on the model through known physical parameters and principles to reduce complexity of the learning problem to give us more reliable predictions. Recent work employs discrete variational integrators parameterized as a neural network architecture to learn conservative Lagrangian systems. The learned model captures and enforces global energy preserving properties of the system from very few trajectories. However, most real systems are inherently non-conservative and, in practice, we would also like to apply actuation. In this paper we extend this paradigm to account for general forcing (e.g. control input and damping) via discrete d'Alembert's principle which may ultimately be used for control applications. We show that this forced variational integrator networks (FVIN) architecture allows us to accurately account for energy dissipation and external forcing while still capturing the true underlying energy-based passive dynamics. We show that in application this can result in highly-data efficient model-based control and can predict on real non-conservative systems.
APMF < APSP? Gomory-Hu Tree for Unweighted Graphs in Almost-Quadratic Time
Venue: arXiv cs.DS 2021
Year: 2021

We design an $n^{2+o(1)}$-time algorithm that constructs a cut-equivalent (Gomory-Hu) tree of a simple graph on $n$ nodes. This bound is almost-optimal in terms of $n$, and it improves on the recent $\tilde{O}(n^{2.5})$ bound by the authors (STOC 2021), which was the first to break the cubic barrier. Consequently, the All-Pairs Maximum-Flow (APMF) problem has time complexity $n^{2+o(1)}$, and for the first time in history, this problem can be solved faster than All-Pairs Shortest Paths (APSP). We further observe that an almost-linear time algorithm (in terms of the number of edges $m$) is not possible without first obtaining a subcubic algorithm for multigraphs. Finally, we derandomize our algorithm, obtaining the first subcubic deterministic algorithm for Gomory-Hu Tree in simple graphs, showing that randomness is not necessary for beating the $n-1$ times max-flow bound from 1961. The upper bound is $\tilde{O}(n^{2\frac{2}{3}})$ and it would improve to $n^{2+o(1)}$ if there is a deterministic single-pair maximum-flow algorithm that is almost-linear. The key novelty is in using a ``dynamic pivot'' technique instead of the randomized pivot selection that was central in recent works.
Multilayer Representation and Multiscale Analysis on Data Networks
Venue: arXiv cs.NI 2021
Year: 2021

The constant increase in the complexity of data networks motivates the search for strategies that make it possible to reduce current monitoring times. This paper shows the way in which multilayer network representation and the application of multiscale analysis techniques, as applied to software-defined networks, allows for the visualization of anomalies from "coarse views of the network topology". This implies the analysis of fewer data, and consequently the reduction of the time that a process takes to monitor the network. The fact that software-defined networks allow for the obtention of a global view of network behavior facilitates detail recovery from affected zones detected in monitoring processes. The method is evaluated by calculating the reduction factor of nodes, checked during anomaly detection, with respect to the total number of nodes in the network.
Multi-facet Contextual Bandits: A Neural Network Perspective
Venue: arXiv cs.LG 2021
Year: 2021

Contextual multi-armed bandit has shown to be an effective tool in recommender systems. In this paper, we study a novel problem of multi-facet bandits involving a group of bandits, each characterizing the users' needs from one unique aspect. In each round, for the given user, we need to select one arm from each bandit, such that the combination of all arms maximizes the final reward. This problem can find immediate applications in E-commerce, healthcare, etc. To address this problem, we propose a novel algorithm, named MuFasa, which utilizes an assembled neural network to jointly learn the underlying reward functions of multiple bandits. It estimates an Upper Confidence Bound (UCB) linked with the expected reward to balance between exploitation and exploration. Under mild assumptions, we provide the regret analysis of MuFasa. It can achieve the near-optimal $\widetilde{ \mathcal{O}}((K+1)\sqrt{T})$ regret bound where $K$ is the number of bandits and $T$ is the number of played rounds. Furthermore, we conduct extensive experiments to show that MuFasa outperforms strong baselines on real-world data sets.
Discovery of Layered Software Architecture from Source Code Using Ego Networks
Venue: arXiv cs.SE 2021
Year: 2021

Software architecture refers to the high-level abstraction of a system including the configuration of the involved elements and the interactions and relationships that exist between them. Source codes can be easily built by referring to the software architectures. However, the reverse process i.e. derivation of the software architecture from the source code is a challenging task. Further, such an architecture consists of multiple layers, and distributing the existing elements into these layers should be done accurately and efficiently. In this paper, a novel approach is presented for the recovery of layered architectures from Java-based software systems using the concept of ego networks. Ego networks have traditionally been used for social network analysis, but in this paper, they are modified in a particular way and tuned to suit the mentioned task. Specifically, a dependency network is extracted from the source code to create an ego network. The ego network is processed to create and optimize ego layers in a particular structure. These ego layers when integrated and optimized together give the final layered architecture. The proposed approach is evaluated in two ways: on static versions of three open-source software, and a continuously evolving software system. The distribution of nodes amongst the proposed layers and the committed violations are observed on both class level and package level. The proposed method is seen to outperform the existing standard approaches over multiple performance metrics. We also carry out the analysis of variation in the results concerning the change in the node selection strategy and the frequency. The empirical observations show promising signs for recovering software architecture layers from source codes using this technique and also extending it further to other languages and software.
Efficient Continuous Control with Double Actors and Regularized Critics
Venue: arXiv cs.LG 2021
Year: 2021

How to obtain good value estimation is one of the key problems in Reinforcement Learning (RL). Current value estimation methods, such as DDPG and TD3, suffer from unnecessary over- or underestimation bias. In this paper, we explore the potential of double actors, which has been neglected for a long time, for better value function estimation in continuous setting. First, we uncover and demonstrate the bias alleviation property of double actors by building double actors upon single critic and double critics to handle overestimation bias in DDPG and underestimation bias in TD3 respectively. Next, we interestingly find that double actors help improve the exploration ability of the agent. Finally, to mitigate the uncertainty of value estimate from double critics, we further propose to regularize the critic networks under double actors architecture, which gives rise to Double Actors Regularized Critics (DARC) algorithm. Extensive experimental results on challenging continuous control tasks show that DARC significantly outperforms state-of-the-art methods with higher sample efficiency.
Technical Report: Temporal Aggregate Representations
Venue: arXiv cs.CV 2021
Year: 2021

This technical report extends our work presented in [9] with more experiments. In [9], we tackle long-term video understanding, which requires reasoning from current and past or future observations and raises several fundamental questions. How should temporal or sequential relationships be modelled? What temporal extent of information and context needs to be processed? At what temporal scale should they be derived? [9] addresses these questions with a flexible multi-granular temporal aggregation framework. In this report, we conduct further experiments with this framework on different tasks and a new dataset, EPIC-KITCHENS-100.
Extractive Research Slide Generation Using Windowed Labeling Ranking
Venue: arXiv cs.CL 2021
Year: 2021

Presentation slides describing the content of scientific and technical papers are an efficient and effective way to present that work. However, manually generating presentation slides is labor intensive. We propose a method to automatically generate slides for scientific papers based on a corpus of 5000 paper-slide pairs compiled from conference proceedings websites. The sentence labeling module of our method is based on SummaRuNNer, a neural sequence model for extractive summarization. Instead of ranking sentences based on semantic similarities in the whole document, our algorithm measures importance and novelty of sentences by combining semantic and lexical features within a sentence window. Our method outperforms several baseline methods including SummaRuNNer by a significant margin in terms of ROUGE score.
Tabular Data: Deep Learning is Not All You Need
Venue: arXiv cs.LG 2021
Year: 2021

A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.
Low-complexity Voronoi shaping for the Gaussian channel
Venue: arXiv cs.IT 2021
Year: 2021

Voronoi constellations (VCs) are finite sets of vectors of a coding lattice enclosed by the translated Voronoi region of a shaping lattice, which is a sublattice of the coding lattice. In conventional VCs, the shaping lattice is a scaled-up version of the coding lattice. In this paper, we design low-complexity VCs with a cubic coding lattice of up to 32 dimensions, in which pseudo-Gray labeling is applied to minimize the bit error rate. The designed VCs have considerable shaping gains of up to 1.03 dB and finer choices of spectral efficiencies in practice. A mutual information estimation method and a log-likelihood approximation method based on importance sampling for very large constellations are proposed and applied to the designed VCs. With error-control coding, the proposed VCs can have higher achievable information rates than the conventional scaled VCs because of their inherently good pseudo-Gray labeling feature, with a lower decoding complexity.
Meta-learning for downstream aware and agnostic pretraining
Venue: arXiv cs.CL 2021
Year: 2021

Neural network pretraining is gaining attention due to its outstanding performance in natural language processing applications. However, pretraining usually leverages predefined task sequences to learn general linguistic clues. The lack of mechanisms in choosing proper tasks during pretraining makes the learning and knowledge encoding inefficient. We thus propose using meta-learning to select tasks that provide the most informative learning signals in each episode of pretraining. With the proposed method, we aim to achieve better efficiency in computation and memory usage for the pretraining process and resulting networks while maintaining the performance. In this preliminary work, we discuss the algorithm of the method and its two variants, downstream-aware and downstream-agnostic pretraining. Our experiment plan is also summarized, while empirical results will be shared in our future works.
Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
Venue: arXiv cs.LG 2021
Year: 2021

The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.
FACT: A Full-body Ad-hoc Collaboration Testbed for Modeling Complex Teamwork
Venue: arXiv cs.RO 2021
Year: 2021

Robots are envisioned to work alongside humans in applications ranging from in-home assistance to collaborative manufacturing. Research on human-robot collaboration (HRC) has helped develop various aspects of social intelligence necessary for robots to participate in effective, fluid collaborations with humans. However, HRC research has focused on dyadic, structured, and minimal collaborations between humans and robots that may not fully represent the large scale and emergent nature of more complex, unstructured collaborative activities. Thus, there remains a need for shared testbeds, datasets, and evaluation metrics that researchers can use to better model natural, ad-hoc human collaborative behaviors and develop robot capabilities intended for large scale emergent collaborations. We present one such shared resource - FACT (Full-body Ad-hoc Collaboration Testbed), an openly accessible testbed for researchers to obtain an expansive view of the individual and team-based behaviors involved in complex, co-located teamwork. We detail observations from a preliminary exploration with teams of various sizes and discuss potential research questions that may be investigated using the testbed. Our goal is for FACT to be an initial resource that supports a more holistic investigation of human-robot collaboration.
Semantic and Syntactic Enhanced Aspect Sentiment Triplet Extraction
Venue: arXiv cs.CL 2021
Year: 2021

Aspect Sentiment Triplet Extraction (ASTE) aims to extract triplets from sentences, where each triplet includes an entity, its associated sentiment, and the opinion span explaining the reason for the sentiment. Most existing research addresses this problem in a multi-stage pipeline manner, which neglects the mutual information between such three elements and has the problem of error propagation. In this paper, we propose a Semantic and Syntactic Enhanced aspect Sentiment triplet Extraction model (S3E2) to fully exploit the syntactic and semantic relationships between the triplet elements and jointly extract them. Specifically, we design a Graph-Sequence duel representation and modeling paradigm for the task of ASTE: we represent the semantic and syntactic relationships between word pairs in a sentence by graph and encode it by Graph Neural Networks (GNNs), as well as modeling the original sentence by LSTM to preserve the sequential information. Under this setting, we further apply a more efficient inference strategy for the extraction of triplets. Extensive evaluations on four benchmark datasets show that S3E2 significantly outperforms existing approaches, which proves our S3E2's superiority and flexibility in an end-to-end fashion.
Exploring to establish an appropriate model for image aesthetic assessment via CNN-based RSRL: An empirical study
Venue: arXiv cs.CV 2021
Year: 2021

To establish an appropriate model for photo aesthetic assessment, in this paper, a D-measure which reflects the disentanglement degree of the final layer FC nodes of CNN is introduced. By combining F-measure with D-measure to obtain a FD measure, an algorithm of determining the optimal model from the multiple photo score prediction models generated by CNN-based repetitively self-revised learning(RSRL) is proposed. Furthermore, the first fixation perspective(FFP) and the assessment interest region(AIR) of the models are defined and calculated. The experimental results show that the FD measure is effective for establishing the appropriate model from the multiple score prediction models with different CNN structures. Moreover, the FD-determined optimal models with the comparatively high FD always have the FFP an AIR which are close to the human's aesthetic perception when enjoying photos.
On the Skew-Symmetric Binary Sequences and the Merit Factor Problem
Venue: arXiv cs.IT 2021
Year: 2021

The merit factor problem is of practical importance to manifold domains, such as digital communications engineering, radars, system modulation, system testing, information theory, physics, chemistry. However, the merit factor problem is referenced as one of the most difficult optimization problems and it was further conjectured that stochastic search procedures will not yield merit factors higher than 5 for long binary sequences (sequences with lengths greater than 200). Some useful mathematical properties related to the flip operation of the skew-symmetric binary sequences are presented in this work. By exploiting those properties, the memory complexity of state-of-the-art stochastic merit factor optimization algorithms could be reduced from $O(n^2)$ to $O(n)$. As a proof of concept, a lightweight stochastic algorithm was constructed, which can optimize pseudo-randomly generated skew-symmetric binary sequences with long lengths (up to ${10}^5+1$) to skew-symmetric binary sequences with a merit factor greater than 5. An approximation of the required time is also provided. The numerical experiments suggest that the algorithm is universal and could be applied to skew-symmetric binary sequences with arbitrary lengths.
ContourRender: Detecting Arbitrary Contour Shape For Instance Segmentation In One Pass
Venue: arXiv cs.CV 2021
Year: 2021

Direct contour regression for instance segmentation is a challenging task. Previous works usually achieve it by learning to progressively refine the contour prediction or adopting a shape representation with limited expressiveness. In this work, we argue that the difficulty in regressing the contour points in one pass is mainly due to the ambiguity when discretizing a smooth contour into a polygon. To address the ambiguity, we propose a novel differentiable rendering-based approach named \textbf{ContourRender}. During training, it first predicts a contour generated by an invertible shape signature, and then optimizes the contour with the more stable silhouette by converting it to a contour mesh and rendering the mesh to a 2D map. This method significantly improves the quality of contour without iterations or cascaded refinements. Moreover, as optimization is not needed during inference, the inference speed will not be influenced. Experiments show the proposed ContourRender outperforms all the contour-based instance segmentation approaches on COCO, while stays competitive with the iteration-based state-of-the-art on Cityscapes. In addition, we specifically select a subset from COCO val2017 named COCO ContourHard-val to further demonstrate the contour quality improvements. Codes, models, and dataset split will be released.