ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Paper content:
In this paper we establish rigorous benchmarks for image classifier robustness.
Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications.
Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations.
Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations.
We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers.
Afterward we discover ways to enhance corruption and perturbation robustness.
We even find that a bypassed adversarial defense provides substantial common perturbation robustness.
Together our benchmarks may aid future work toward networks that robustly generalize.

assistant: text: We propose ImageNet-C to measure classifier corruption robustness and ImageNet-P to measure perturbation robustness
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RelWalk -- A Latent Variable Model Approach to Knowledge Graph Embedding

Paper content:
Knowledge Graph Embedding (KGE) is the task of jointly learning entity and relation embeddings for a given knowledge graph.
Existing methods for learning KGEs can be seen as a two-stage process where
(a) entities and relations in the knowledge graph are represented using some linear algebraic structures (embeddings), and
(b) a scoring function is defined that evaluates the strength of a relation that holds between two entities using the corresponding relation and entity embeddings.
Unfortunately, prior proposals for the scoring functions in the first step have been heuristically motivated, and it is unclear as to how the scoring functions in KGEs relate to the generation process of the underlying knowledge graph.
To address this issue, we propose a generative account of the KGE learning task.
Specifically, given a knowledge graph represented by a set of relational triples (h, R, t), where the semantic relation R holds between the two entities h (head) and t (tail), we extend the random walk model (Arora et al., 2016a) of word embeddings to KGE.
We derive a theoretical relationship between the joint probability p(h, R, t) and the embeddings of h, R and t.
Moreover, we show that marginal loss minimisation, a popular objective used by much prior work in KGE, follows naturally from the log-likelihood ratio maximisation under the probabilities estimated from the KGEs according to our theoretical relationship.
We propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph.
The KGEs learnt by our proposed method obtain state-of-the-art performance on FB15K237 and WN18RR benchmark datasets, providing empirical evidence in support of the theory.

assistant: text: We present a theoretically proven generative model of knowledge graph embedding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Active Learning Graph Neural Networks via Node Feature Propagation

Paper content:
Graph Neural Networks (GNNs) for prediction tasks like node classification or edge prediction have received increasing attention in recent machine learning from graphically structured data.
However, a large quantity of labeled graphs is difficult to obtain, which significantly limit the true success of GNNs.
Although active learning has been widely studied for addressing label-sparse issues with other data types like text, images, etc., how to make it effective over graphs is an open question for research.
In this paper, we present the investigation on active learning with GNNs for node classification tasks.
Specifically, we propose a new method, which uses node feature propagation followed by K-Medoids clustering of the nodes for instance selection in active learning.
With a theoretical bound analysis we justify the design choice of our approach.
In our experiments on four benchmark dataset, the proposed method outperforms other representative baseline methods consistently and significantly.

assistant: text: This paper introduces a clustering-based active learning algorithm on graphs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Linearly Constrained Weights: Resolving the Vanishing Gradient Problem by Reducing Angle Bias

Paper content:
In this paper, we first identify \textit{angle bias}, a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions.
We then propose \textit{linearly constrained weights (LCW)} to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero.
A reparameterization technique is presented to efficiently train a model with LCW by embedding the constraints on weight vectors into the structure of the network.
Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias.
Preliminary experiments show that LCW helps train a 100-layered MLP more efficiently than does batch normalization.

assistant: text: We identify angle bias that causes the vanishing gradient problem in deep nets and propose an efficient method to reduce the bias.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: How fine can fine-tuning be?  Learning efficient language models

Paper content:
State-of-the-art performances on language comprehension tasks are achieved by huge language models pre-trained on massive unlabeled text corpora, with very light subsequent fine-tuning in a task-specific supervised manner.
It seems the pre-training procedure learns a very good common initialization for further training on various natural language understanding tasks, such that only few steps need to be taken in the parameter space to learn each task.
In this work, using Bidirectional Encoder Representations from Transformers (BERT) as an example, we verify this hypothesis by showing that task-specific fine-tuned language models are highly close in parameter space to the pre-trained one.
Taking advantage of such observations, we further show that the fine-tuned versions of these huge models, having on the order of $10^8$ floating-point parameters, can be made very computationally efficient.
First, fine-tuning only a fraction of critical layers suffices.
Second, fine-tuning can be adequately performed by learning a binary multiplicative mask on pre-trained weights, \textit{i.e.} by parameter-sparsification.
As a result, with a single effort, we achieve three desired outcomes: (1) learning to perform specific tasks, (2) saving memory by storing only binary masks of certain layers for each task, and (3) saving compute on appropriate hardware by performing sparse operations with model parameters.

assistant: text: Sparsification as fine-tuning of language models
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Learning to Guide Segmentation

Paper content:
There are myriad kinds of segmentation, and ultimately the `"right" segmentation of a given scene is in the eye of the annotator.
Standard approaches require large amounts of labeled data to learn just one particular kind of segmentation.
As a first step towards relieving this annotation burden, we propose the problem of guided segmentation: given varying amounts of pixel-wise labels, segment unannotated pixels by propagating supervision locally (within an image) and non-locally (across images).
We propose guided networks, which extract a latent task representation---guidance---from variable amounts and classes (categories, instances, etc.) of pixel supervision and optimize our architecture end-to-end for fast, accurate, and data-efficient segmentation by meta-learning.
To span the few-shot and many-shot learning regimes, we examine guidance from as little as one pixel per concept to as much as 1000+ images, and compare to full gradient optimization at both extremes.
To explore generalization, we analyze guidance as a bridge between different levels of supervision to segment classes as the union of instances.
Our segmentor concentrates different amounts of supervision of different types of classes into an efficient latent representation, non-locally propagates this supervision across images, and can be updated quickly and cumulatively when given more supervision.

assistant: text: We propose a meta-learning approach for guiding visual segmentation tasks from varying amounts of supervision.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sinkhorn Permutation  Variational Marginal Inference

Paper content:
We address the problem of marginal inference for an exponential family defined over the set of permutation matrices.
This problem is known to quickly become intractable as the size of the permutation increases, since its involves the computation of the permanent of a matrix, a #P-hard problem.
We introduce Sinkhorn variational marginal inference as a scalable alternative, a method whose validity is ultimately justified by the so-called Sinkhorn approximation of the permanent.
We demonstrate the efectiveness of our method in the problem of probabilistic identification of neurons in the worm C.elegans

assistant: text: New methodology for variational marginal inference of permutations based on Sinkhorn algorithm, applied to probabilistic identification of neurons
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Task Clustering for Deep and Diverse Multi-Task and Few-Shot Learning

Paper content:
We investigate task clustering for deep learning-based multi-task and few-shot learning in the settings with large numbers of diverse tasks.
Our method measures task similarities using cross-task transfer performance matrix.
Although this matrix provides us critical information regarding similarities between tasks, the uncertain task-pairs, i.e., the ones with extremely asymmetric transfer scores, may collectively mislead clustering algorithms to output an inaccurate task-partition.
Moreover, when the number of tasks is large, generating the full transfer performance matrix can be very time consuming.
To overcome these limitations, we propose a novel task clustering algorithm to estimate the similarity matrix based on the theory of matrix completion.
The proposed algorithm can work on partially-observed similarity matrices based on only sampled task-pairs with reliable scores, ensuring its efficiency and robustness.
Our theoretical analysis shows that under mild assumptions, the reconstructed matrix perfectly matches the underlying “true” similarity matrix with an overwhelming probability.
The final task partition is computed by applying an efficient spectral clustering algorithm to the recovered matrix.
Our results show that the new task clustering method can discover task clusters that benefit both multi-task learning and few-shot learning setups for sentiment classification and dialog intent classification tasks.

assistant: text: We propose a matrix-completion based task clustering algorithm for deep multi-task and few-shot learning in the settings with large numbers of diverse tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Human Postural Control with Hierarchical Acquisition Functions

Paper content:
Learning control policies in robotic tasks requires a large number of interactions due to small learning rates, bounds on the updates or unknown constraints.
In contrast humans can infer protective and safe solutions after a single failure or unexpected observation.
In order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive inference and memorization process for avoiding failures in motor control tasks.
A Gaussian Process implements the modeling and the sampling of the acquisition function.
This enables rapid learning with large learning rates while a mental replay phase ensures that policy regions that led to failures are inhibited during the sampling process.
The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task.
We quantitatively compare the human learning performance to our learning approach by evaluating the deviations of the center of mass during training.
Our results show that we can reproduce the efficient learning of human subjects in postural control tasks which provides a testable model for future physiological motor control tasks.
In these postural control tasks, our method outperforms standard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures.

assistant: text: This paper presents a computational model for efficient human postural control adaptation based on hierarchical acquisition functions with well-known features.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Flow++: Improving Flow-Based Generative Models  with  Variational Dequantization and Architecture Design  

Paper content:
Flow-based generative models are powerful exact likelihood models with efficient sampling and inference.
Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models.
In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers.
Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks.
Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models.

assistant: text: Improved training of current flow-based generative models (Glow and RealNVP) on density estimation benchmarks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Topological based classification using graph convolutional networks

Paper content:
In colored graphs, node classes are often associated with either their neighbors class or with information not incorporated in the graph associated with each node.
We here propose that node classes are also associated with topological features of the nodes.
We use this association to improve Graph machine learning in general and specifically, Graph Convolutional Networks (GCN).
First, we show that even in the absence of any external information on nodes, a good accuracy can be obtained on the prediction of the node class using either topological features, or using the neighbors class as an input to a GCN.
This accuracy is slightly less than the one that can be obtained using content based GCN.
Secondly, we show that explicitly adding the topology as an input to the GCN does not improve the accuracy when combined with external information on nodes.
However,  adding an additional adjacency matrix with edges between distant nodes with similar topology to the GCN does significantly improve its accuracy, leading to results better than all state of the art methods in multiple datasets.

assistant: text: Topology-Based Graph Convolutional Network (GCN)
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog

Paper content:
Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment.
This is a critical shortcoming for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction.
Thus, we develop a novel class of off-policy batch RL algorithms which use KL-control to penalize divergence from a pre-trained prior model of probable actions.
This KL-constraint reduces extrapolation error, enabling effective offline learning, without exploration, from a fixed batch of data.
We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning.
This Way Off-Policy (WOP) algorithm is tested on both traditional RL tasks from OpenAI Gym, and on the problem of open-domain dialog generation; a challenging reinforcement learning problem with a 20,000 dimensional action space.
WOP allows for the extraction of multiple different reward functions post-hoc from collected human interaction data, and can learn effectively from all of these.
We test real-world generalization by deploying dialog models live to converse with humans in an open-domain setting, and demonstrate that WOP achieves significant improvements over state-of-the-art prior methods in batch deep RL.

assistant: text: We show that KL-control from a pre-trained prior can allow RL models to learn from a static batch of collected data, without the ability to explore online in the environment.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Linguistically-Informed Self-Attention for Semantic Role Labeling

Paper content:
The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features.
However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax.
In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines
multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL.
For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token.
Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.
In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data.
On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error.

assistant: text: Our combination of multi-task learning and self-attention, training the model to attend to parents in a syntactic parse tree, achieves state-of-the-art CoNLL-2005 and CoNLL-2012 SRL results for models using predicted predicates.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Objective Mismatch in Model-based Reinforcement Learning

Paper content:
Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks.
Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception.
In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue.
Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized.
In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t. the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task.
For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task.
In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of the one-step ahead prediction is not always correlated with downstream control performance.
This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed.
We propose an initial method to mitigate the mismatch issue by re-weighting dynamics model training.
Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.

assistant: text: We define, explore, and begin to address the objective mismatch issue in model-based reinforcement learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Network Bandit Learning by Last Layer Marginalization

Paper content:
We propose a new method for training neural networks online in a bandit setting.
Similar to prior work, we model the uncertainty only in the last layer of the network, treating the rest of the network as a feature extractor.
This allows us to successfully balance between exploration and exploitation due to the efficient, closed-form uncertainty estimates available for linear models.
To train the rest of the network, we take advantage of the posterior we have over the last layer, optimizing over all values in the last layer distribution weighted by probability.
We derive a closed form, differential approximation to this objective and show empirically over various models and datasets that training the rest of the network in this fashion leads to both better online and offline performance when compared to other methods.

assistant: text: This paper proposes a new method for neural network learning in online bandit settings by marginalizing over the last layer
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS

Paper content:
Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE).
However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised.
Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions.
The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new  representations at different levels of abstraction.
We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark datasets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap.
We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations.
By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

assistant: text: We proposed a progressive learning method to improve learning and disentangling latent representations at different levels of abstraction.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Topics using Semantic Locality

Paper content:
The topic modeling discovers the latent topic probability of given the text documents.
To generate the more meaningful topic that better represents the given document, we proposed a universal method which can be used in the data preprocessing stage.
The method consists of three steps.
First, it generates the word/word-pair from every single document.
Second, it applies a two way parallel TF-IDF algorithm to word/word-pair for semantic filtering.
Third, it uses the k-means algorithm to merge the word pairs that have the similar semantic meaning.
Experiments are carried out on the Open Movie Database (OMDb), Reuters Dataset and 20NewsGroup Dataset and use the mean Average Precision score as the evaluation metric.
Comparing our results with other state-of-the-art topic models, such as Latent Dirichlet allocation and traditional Restricted Boltzmann Machines.
Our proposed data preprocessing can improve the generated topic accuracy by up to 12.99\%.
How the number of clusters and the number of word pairs should be adjusted for different type of text document is also discussed.

assistant: text: We proposed a universal method which can be used in the data preprocessing stage to generate the more meaningful topic that better represents the given document
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Architecture Search for Natural Language Understanding

Paper content:
Neural architecture search (NAS) has made rapid progress incomputervision,wherebynewstate-of-the-artresultshave beenachievedinaseriesoftaskswithautomaticallysearched neural network (NN) architectures.
In contrast, NAS has not made comparable advances in natural language understanding (NLU).
Corresponding to encoder-aggregator meta architecture of typical neural networks models for NLU tasks (Gong et al. 2018), we re-deﬁne the search space, by splittingitinto twoparts:encodersearchspace,andaggregator search space.
Encoder search space contains basic operations such as convolutions, RNNs, multi-head attention and its sparse variants, star-transformers.
Dynamic routing is included in the aggregator search space, along with max (avg) pooling and self-attention pooling.
Our search algorithm is then fulﬁlled via DARTS, a differentiable neural architecture search framework.
We progressively reduce the search space every few epochs, which further reduces the search time and resource costs.
Experiments on ﬁve benchmark data-sets show that, the new neural networks we generate can achieve performances comparable to the state-of-the-art models that does not involve language model pre-training.

assistant: text: Neural Architecture Search for a series of Natural Language Understanding tasks. Design the search space for NLU tasks. And Apply differentiable architecture search to discover new models
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Statistical Interactions from Neural Network Weights

Paper content:
Interpreting neural networks is a crucial and challenging task in machine learning.
In this paper, we develop a novel framework for detecting statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.
Depending on the desired interactions, our method can achieve significantly better or similar interaction detection performance compared to the state-of-the-art without searching an exponential solution space of possible interactions.
We obtain this accuracy and efficiency by observing that interactions between input features are created by the non-additive effect of nonlinear activation functions, and that interacting paths are encoded in weight matrices.
We demonstrate the performance of our method and the importance of discovered interactions via experimental results on both synthetic datasets and real-world application datasets.

assistant: text: We detect statistical interactions captured by a feedforward multilayer neural network by directly interpreting its learned weights.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fix your classifier: the marginal value of training the last weight layer

Paper content:
Neural networks are commonly used as models for classification for a wide variety of tasks.
Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification.
This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources.
In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits.
Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well.
We discuss the implications for current understanding of neural network models.

assistant: text: You can fix the classifier in neural networks without losing accuracy
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Off-Policy Actor-Critic with Shared Experience Replay

Paper content:
We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges:
(a) efficient actor-critic learning with experience replay
(b) stability of very off-policy learning.
We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module.
To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods.
Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable.
We provide extensive empirical validation of the proposed solution.
We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

assistant: text: We investigate and propose solutions for two challenges in reinforcement learning: (a) efficient actor-critic learning with experience replay (b) stability of very off-policy learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dynamic Scale Inference by Entropy Minimization

Paper content:
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field.
Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly.
The degree of variation and diversity of inputs makes this a difficult task.
Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data.
We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity.
We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input.
Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.

assistant: text: Unsupervised optimization during inference gives top-down feedback to iteratively adjust feedforward prediction of scale variation for more equivariant recognition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Pruning at a Glance: A Structured Class-Blind Pruning Technique for Model Compression

Paper content:
High performance of deep learning models typically comes at cost of considerable model size and computation time.
These factors limit applicability for deployment on memory and battery constraint devices such as mobile phones or embedded systems.
In this work we propose a novel pruning technique that eliminates entire filters and neurons according to their relative L1-norm as compared to the rest of the network, yielding more compression and decreased redundancy in the parameters.
The resulting network is non-sparse, however, much more compact and requires no special infrastructure for its deployment.
We prove the viability of our method by achieving 97.4%, 47.8% and 53% compression of LeNet-5, ResNet-56 and ResNet-110 respectively, exceeding state-of-the-art compression results reported on ResNet without losing any performance compared to the baseline.
Our approach does not only exhibit good performance, but is also easy to implement on many architectures.

assistant: text: We propose a novel structured class-blind pruning technique to produce highly compressed neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learnable Embedding Space for Efficient Neural Architecture Compression

Paper content:
We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search.
Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation.
We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018).
The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018).
We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.

assistant: text: We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling the roles of dimensionality and cell classes in neural computations

Paper content:
The description of neural computations in the field of neuroscience relies on two competing views:
(i) a classical single-cell view that relates the activity of individual neurons to sensory or behavioural variables, and focuses on how different cell classes map onto computations;
(ii) a more recent population view that instead characterises computations in terms of collective neural trajectories, and focuses on the dimensionality of these trajectories as animals perform tasks.
How the two key concepts of cell classes and low-dimensional trajectories interact to shape neural computations is however currently not understood.
Here we address this question by combining machine-learning tools for training RNNs with reverse-engineering and theoretical analyses of network dynamics.
We introduce a novel class of theoretically tractable recurrent networks: low-rank, mixture of Gaussian RNNs.
In these networks, the rank of the connectivity controls the dimensionality of the dynamics, while the number of components in the Gaussian mixture corresponds to the number of cell classes.
Using back-propagation, we determine the minimum rank and number of cell classes needed to implement neuroscience tasks of increasing complexity.
We then exploit mean-field theory to reverse-engineer the obtained solutions and identify the respective roles of dimensionality and cell classes.
We show that the rank determines the phase-space available for dynamics that implement input-output mappings, while having multiple cell classes allows networks to flexibly switch between different types of dynamics in the available phase-space.
Our results have implications for the analysis of neuroscience experiments and the development of explainable AI.

assistant: text: A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentanglement Challenge: From Regularization to Reconstruction

Paper content:
The challenge of learning disentangled representation has recently attracted much attention and boils down to a competition.
Various methods based on variational auto-encoder have been proposed to solve this problem, by enforcing the independence between the representation and modifying the regularization term in the variational lower bound.
However recent work by Locatello et al. (2018) has demonstrated that the proposed methods are heavily influenced by randomness and the choice of the hyper-parameter.
This work is built upon the same framework in Stage 1 (Li et al., 2019), but with different settings; to make it self-contained, we provide this manuscript, which is unavoidably very similar to the report for Stage 1.
In detail, in this work, instead of designing a new regularization term, we adopt the FactorVAE but improve the reconstruction performance and increase the capacity of network and the training step.
The strategy turns out to be very effective in achieving disentanglement.

assistant: text: disentangled representation learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience

Paper content:
The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -- minima where the output of the network is resilient to small random noise added to its parameters.
So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either \textit{stochastic} or \textit{compressed}.
In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -- a network that is deterministic and uncompressed.
What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves {\em generalize} to the interactions between the matrices on test data, thereby implying a wide test loss minimum.
We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data).
In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -- a guarantee that would not have been possible with prior approaches.

assistant: text: We provide a PAC-Bayes based generalization guarantee for uncompressed, deterministic deep networks by generalizing noise-resilience of the network on the training data to the test data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework.
VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator.
We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance.
This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion.
By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.

assistant: text: A novel Bayesian deep learning framework that captures and relates hierarchical semantic and visual concepts, performing well on a variety of image and text modeling and generation tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intensity-Free Learning of Temporal Point Processes

Paper content:
Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals.
The standard way of learning in such models is by estimating the conditional intensity function.
However, parameterizing the intensity function usually incurs several trade-offs.
We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times.
We draw on the literature on normalizing flows to design models that are flexible and efficient.
We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form.
The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data.

assistant: text: Learn in temporal point processes by modeling the conditional density, not the conditional intensity.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sample-Based Point Cloud Decoder Networks

Paper content:
Point clouds are a flexible and ubiquitous way to represent 3D objects with arbitrary resolution and precision.
Previous work has shown that adapting encoder networks to match the semantics of their input point clouds can significantly improve their effectiveness over naive feedforward alternatives.
However, the vast majority of work on point-cloud decoders are still based on fully-connected networks that map shape representations to a fixed number of output points.
In this work, we investigate decoder architectures that more closely match the semantics of variable sized point clouds.
Specifically, we study sample-based point-cloud decoders that map a shape representation to a point feature distribution, allowing an arbitrary number of sampled features to be transformed into individual output points.
We develop three sample-based decoder architectures and compare their performance to each other and show their improved effectiveness over feedforward architectures.
In addition, we investigate the learned distributions to gain insight into the output transformation.
Our work is available as an extensible software platform to reproduce these results and serve as a baseline for future work.

assistant: text: We present and evaluate sampling-based point cloud decoders that outperform the baseline MLP approach by better matching the semantics of point clouds.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Growing Action Spaces

Paper content:
In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress.
In this work, we use a curriculum of progressively growing action spaces to accelerate learning.
We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space.
Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data,  value estimates, and state representations from restricted action spaces to the full task.
We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.

assistant: text: Progressively growing the available action space is a great curriculum for learning agents
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergence of Compositional Language with Deep Generational Transmission

Paper content:
Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate to solve a task.
Of particular interest are the factors that cause language to be compositional---i.e., express meaning by combining words which themselves have meaning.
Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute significantly to the emergence of  compositionality.
In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language.
We show that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization.

assistant: text: We use cultural transmission to encourage compositionality in languages that emerge from interactions between neural agents.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Symmetry and Systematicity

Paper content:
We argue that symmetry is an important consideration in addressing the problem
of systematicity and investigate two forms of symmetry relevant to symbolic processes.
We implement this approach in terms of convolution and show that it can
be used to achieve effective generalisation in three toy problems: rule learning,
composition and grammar learning.

assistant: text: We use convolution to make neural networks behave more like symbolic systems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergent Coordination Through Competition

Paper content:
We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics.
We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation.
Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control.
In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior.
We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.

assistant: text: We introduce a new MuJoCo soccer environment for continuous multi-agent reinforcement learning research, and show that population-based training of independent reinforcement learners can learn cooperative behaviors