ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments

Paper content:
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay.
In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW.
Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.
The most popular algorithms for training Neural Networks (NNs) are Stochastic Gradient Descent (SGD) with momentum (Polyak, 1964; Sutskever et al., 2013) and Adam (Kingma & Ba, 2015) .
SGD with momentum is the preferred algorithm for computer vision, while Adam is the most commonly used for natural language processing (NLP) and speech problems.
Compared to SGD, Adam is perceived as safer and more robust to weight initialization and learning rate.
However, Adam has certain drawbacks.
First, as noted in the original paper (Kingma & Ba, 2015) , the second moment can vanish or explode, especially during the initial phase of training.
To alleviate this problem, a learning rate (LR) warmup (Goyal et al., 2017 ) is typically used.
Adam often leads to solutions that generalize worse than SGD (Wilson et al., 2017) , and to improve Adam regularization, Loshchilov & Hutter (2019) proposed AdamW with decoupled weight decay.
Our motivation for this work was to find an algorithm which: (1) performs equally well for image classification, speech recognition, machine translation, and language modeling, (2) is robust to learning rate and weight initialization, (3) has strong regularization properties.
We start with Adam, and then (1) replace the element-wise second moment with the layer-wise moment, (2) compute the first moment using gradients normalized by layer-wise second moment, (3) and decouple weight decay (similar to AdamW) from normalized gradients.
The resulting algorithm, NovoGrad, combines SGD's and Adam's strengths.
We applied NovoGrad to a variety of large scale problems -image classification, neural machine translation, language modeling, and speech recognition -and found that in all cases, it performs as well or better than Adam/AdamW and SGD with momentum.

assistant: text: NovoGrad -  an adaptive SGD method with layer-wise gradient normalization and decoupled weight decay.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Using Ontologies To Improve Performance In Massively Multi-label Prediction

Paper content:
Massively multi-label prediction/classification problems arise in environments like health-care or biology where it is useful to make very precise predictions.
One challenge with massively multi-label problems is that there is often a long-tailed frequency distribution for the labels, resulting in few positive examples for the rare labels.
We propose a solution to this problem by modifying the output layer of a neural network to create a Bayesian network of sigmoids which takes advantage of ontology relationships between the labels to help share information between the rare and the more common labels.
We apply this method to the two massively multi-label tasks of disease prediction (ICD-9 codes) and protein function prediction (Gene Ontology terms) and obtain significant improvements in per-label AUROC and average precision.

assistant: text: We propose a new method for using ontology information to improve performance on massively multi-label prediction/classification problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Transfer via Modelling Multi-level Task Dependency

Paper content:
Multi-task learning has been successful in modeling multiple related tasks with large, carefully curated labeled datasets.
By leveraging the relationships among different tasks, multi-task learning framework can improve the performance significantly.
However, most of the existing works are under the assumption that the predefined tasks are related to each other.
Thus, their applications on real-world are limited, because rare real-world problems are closely related.
Besides, the understanding of relationships among tasks has been ignored by most of the current methods.
Along this line, we propose a novel multi-task learning framework - Learning To Transfer Via Modelling Multi-level Task Dependency, which constructed attention based dependency relationships among different tasks.
At the same time, the dependency relationship can be used to guide what knowledge should be transferred, thus the performance of our model also be improved.
To show the effectiveness of our model and the importance of considering multi-level dependency relationship, we conduct experiments on several public datasets, on which we obtain significant improvements over current methods.

assistant: text: We propose a novel multi-task learning framework which extracts multi-view dependency relationship automatically and use it to guide the knowledge transfer among different tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-Advisor Reinforcement Learning

Paper content:
We consider tackling a single-agent RL problem by distributing it to $n$ learners.
These learners, called advisors, endeavour to solve the problem from a different focus.
Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system.
We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the \textit{egocentric} planning overestimates values of states where the other advisors disagree, and the \textit{agnostic} planning is inefficient around danger zones.
We introduce a novel approach called \textit{empathic} and discuss its theoretical aspects.
We empirically examine and validate our theoretical findings on a fruit collection task.

assistant: text: We consider tackling a single-agent RL problem by distributing it to $n$ learners.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Quantitatively Evaluating GANs With Divergences Proposed for Training

Paper content:
Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application.
However, we currently lack quantitative methods for model assessment.
Because of this, while many GAN variants being proposed, we have relatively little understanding of their relative abilities.
In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training.
We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion.
We also compare the proposed metrics to human perceptual scores.

assistant: text: An empirical evaluation on generative adversarial networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Data-Driven and Distributed Approach to Sparse Signal Representation and Recovery

Paper content:
In this paper, we focus on two challenges which offset the promise of sparse signal representation, sensing, and recovery.
First, real-world signals can seldom be described as perfectly sparse vectors in a known basis, and traditionally used random measurement schemes are seldom optimal for sensing them.
Second, existing signal recovery algorithms are usually not fast enough to make them applicable to real-time problems.
In this paper, we address these two challenges by presenting a novel framework based on deep learning.
For the first challenge, we cast the problem of finding informative measurements by using a maximum likelihood (ML) formulation and show how we can build a data-driven dimensionality reduction protocol for sensing signals using convolutional architectures.
For the second challenge, we discuss and analyze a novel parallelization scheme and show it significantly speeds-up the signal recovery process.
We demonstrate the significant improvement our method obtains over competing methods through a series of experiments.

assistant: text: We use deep learning techniques to solve the sparse signal representation and recovery problem.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ROBUST DISCRIMINATIVE REPRESENTATION LEARNING VIA GRADIENT RESCALING: AN EMPHASIS REGULARISATION PERSPECTIVE

Paper content:
It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist.
Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused and how much more should they be emphasised to achieve robust learning?
In this work, we study this question and propose gradient rescaling (GR) to solve it.
GR modifies the magnitude of logit vector’s gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs.
Apart from regularisation, we connect GR to examples weighting and designing robust loss functions.
We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR100 with 40% noisy labels.
It is also significantly superior to standard regularisers in both clean and abnormal settings.
Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios.

assistant: text: ROBUST DISCRIMINATIVE REPRESENTATION LEARNING VIA GRADIENT RESCALING: AN EMPHASIS REGULARISATION PERSPECTIVE
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scalable Model Compression by Entropy Penalized Reparameterization

Paper content:
We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a “latent” space, amounting to a reparameterization.
This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training.
Classification accuracy and model compressibility is maximized jointly, with the bitrate--accuracy trade-off specified by a hyperparameter.
We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures.
Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.

assistant: text: An end-to-end trainable model compression method optimizing accuracy jointly with the expected model size.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Reach Goals Without Reinforcement Learning

Paper content:
Imitation learning algorithms provide a simple and straightforward approach for training control policies via standard supervised learning methods.
By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator -- typically a person -- to provide the demonstrations.
In this paper, we ask: can we use imitation learning to train effective policies without any expert demonstrations?
The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks.
In particular, in the setting where the tasks correspond to different goals, every trajectory is a successful demonstration for the state that it actually reaches.
Informed by this observation, we propose a very simple algorithm for learning behaviors without any demonstrations, user-provided reward functions, or complex reinforcement learning methods.
Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached.
Although related variants of this approach have been proposed previously in imitation learning settings with example demonstrations, we present the first instance of this approach as a method for learning goal-reaching policies entirely from scratch.
We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems.

assistant: text: Learning how to reach goals from scratch by using imitation learning with data relabeling
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking the Value of Network Pruning

Paper content:
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings.
A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning.
During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy.
In this work, we make several surprising observations which contradict common beliefs.
For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights.
For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch.
Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned ``important'' weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited ``important'' weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.
Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods.
We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

assistant: text: In structured network pruning, fine-tuning a pruned model only gives comparable performance with training it from scratch.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Model Inversion Networks for Model-Based Optimization

Paper content:
In this work, we aim to solve data-driven optimization problems, where the goal is to find an input that maximizes an unknown score function given access to a dataset of input, score pairs.
Inputs may lie on extremely thin manifolds in high-dimensional spaces, making the optimization prone to falling-off the manifold.
Further, evaluating the unknown function may be expensive, so the algorithm should be able to exploit static, offline data.
We propose model inversion networks (MINs) as an approach to solve such problems.
Unlike prior work, MINs scale to extremely high-dimensional input spaces and can efficiently leverage offline logged datasets for optimization in both contextual and non-contextual settings.
We show that MINs can also be extended to the active setting, commonly studied in prior work, via a simple, novel and effective scheme for active data collection.
Our experiments show that MINs act as powerful optimizers on a range of contextual/non-contextual, static/active problems including optimization over images and protein designs and learning from logged bandit feedback.

assistant: text: We propose a novel approach to solve data-driven model-based optimization problems in both passive and active settings that can scale to high-dimensional input spaces.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fatty and Skinny: A Joint Training Method of Watermark Encoder and Decoder

Paper content:
Watermarks have been used for various purposes.
Recently, researchers started to look into using them for deep neural networks.
Some works try to hide attack triggers on their adversarial samples when attacking neural networks and others want to watermark neural networks to prove their ownership against plagiarism.
Implanting a backdoor watermark module into a neural network is getting more attention from the community.
In this paper, we present a general purpose encoder-decoder joint training method, inspired by generative adversarial networks (GANs).
Unlike GANs, however, our encoder and decoder neural networks cooperate to find the best watermarking scheme given data samples.
In other words, we do not design any new watermarking strategy but our proposed two neural networks will find the best suited method on their own.
After being trained, the decoder can be implanted into other neural networks to attack or protect them (see Appendix for their use cases and real implementations).
To this end, the decoder should be very tiny in order not to incur any overhead when attached to other neural networks but at the same time provide very high decoding success rates, which is very challenging.
Our joint training method successfully solves the problem and in our experiments maintain almost 100\% encoding-decoding success rates for multiple datasets with very little modifications on data samples to hide watermarks.
We also present several real-world use cases in Appendix.

assistant: text: We propose a novel watermark encoder-decoder neural networks. They perform a cooperative game to define their own watermarking scheme. People do not need to design watermarking methods any more.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TRAINING GENERATIVE ADVERSARIAL NETWORKS VIA PRIMAL-DUAL SUBGRADIENT METHODS: A LAGRANGIAN PERSPECTIVE ON GAN

Paper content:
We relate the minimax game of generative adversarial networks (GANs) to finding the saddle points of the Lagrangian function for a convex optimization problem, where the discriminator outputs and the distribution of generator outputs play the roles of primal variables and dual variables, respectively.
This formulation shows the connection between the standard GAN training process and the primal-dual subgradient methods for convex optimization.
The inherent connection does not only provide a theoretical convergence proof for training GANs in the function space, but also inspires a novel objective function for training.
The modified objective function forces the distribution of generator outputs to be updated along the direction according to the primal-dual subgradient methods.
A toy example shows that the proposed method is able to resolve mode collapse, which in this case cannot be avoided by the standard GAN or Wasserstein GAN.
Experiments on both Gaussian mixture synthetic data and real-world image datasets demonstrate the performance of the proposed method on generating diverse samples.

assistant: text: We propose a primal-dual subgradient method for training GANs and this method effectively alleviates mode collapse.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Adaptive Stress Testing Formulation

Paper content:
Validation is a key challenge in the search for safe autonomy.
Simulations are often either too simple to provide robust validation, or too complex to tractably compute.
Therefore, approximate validation methods are needed to tractably find failures without unsafe simplifications.
This paper presents the theory behind one such black-box approach: adaptive stress testing (AST).
We also provide three examples of validation problems formulated to work with AST.
An open question when robots operate autonomously in uncertain, real-world environments is how to tractably validate that the agent will act safely.
Autonomous robotic systems may be expected to interact with a number of other actors, including humans, while handling uncertainty in perception, prediction and control.
Consequently, scenarios are often too high-dimensional to tractably simulate in an exhaustive manner.
As such, a common approach is to simplify the scenario by constraining the number of non-agent actors and the range of actions they can take.
However, simulating simplified scenarios may compromise safety by eliminating the complexity needed to find rare, but important failures.
Instead, approximate validation methods are needed to elicit agent failures while maintaining the full complexity of the simulation.One possible approach to approximate validation is adaptive stress testing (AST) BID6 .
In AST, the validation problem is cast as a Markov decision process (MDP).
A specific reward function structure is then used with reinforcement learning algorithms in order to identify the most-likely failure of a system in a scenario.
Knowing the most-likely failure is useful for two reasons:
1) all other failures are at most as-likely, so it provides a bound on the likelihood of failures, and
2) it uncovers possible failure modes of an autonomous system so they can be addressed.
AST is not a silver bullet: it requires accurate models of all actors in the scenario and is susceptible to local convergence.
However, it allows failures to be identified tractably in simulation for complicated autonomous systems acting in high-dimensional spaces.
This paper briefly presents the latest methodology for using AST and includes example validation scenarios formulated as AST problems.
This paper presents the latest formulation of adaptive stress testing, and examples of how it can be applied.
AST is an approach to validation that can tractably find failures in autonomous systems in simulation without reducing scenario complexity.
Autonomous systems are difficult to validate because they interact with many other actors in high-dimensional spaces according to complicated policies.
However, validation is essential for producing autonomous systems that are safe, robust, and reliable.

assistant: text: A formulation for a black-box, reinforcement learning method to find the most-likely failure of a system acting in complex scenarios.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Matrix capsules with EM routing

Paper content:
A capsule is a group of neurons whose outputs represent different properties of the same entity.
Each layer in a capsule network contains many capsules.
We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose).
A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.
Each of these votes is weighted by an assignment coefficient.
These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes.
The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers.
On the smallNORB benchmark, capsules reduce the number of test errors by 45\% compared to the state-of-the-art.
Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.

assistant: text: Capsule networks with learned pose matrices and EM routing improves state of the art classification on smallNORB, improves generalizability to new view points, and white box adversarial robustness.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Defense Via Data Dependent Activation Function and  Total Variation Minimization

Paper content:
We improve the robustness of deep neural nets  to adversarial attacks by using an interpolating function as the output activation.
This data-dependent activation function remarkably improves both classification accuracy and stability to adversarial perturbations.
Together with the total variation minimization of adversarial images and augmented training, under the strongest attack, we achieve up to 20.6%, 50.7%, and 68.7% accuracy improvement w.r.t.
the fast gradient sign method, iterative fast gradient sign method, and Carlini-WagnerL2attacks, respectively.
Our defense strategy is additive to many of the existing methods.
We give an intuitive explanation of our defense strategy via analyzing the geometry of the feature space.
For reproducibility, the code will be available on GitHub.
The adversarial vulnerability BID26 of deep neural nets (DNNs) threatens their applicability in security critical tasks, e.g., autonomous cars BID0 , robotics BID8 , DNN-based malware detection systems BID20 BID7 .
Since the pioneering work by BID26 , many advanced adversarial attack schemes have been devised to generate imperceptible perturbations to sufficiently fool the DNNs BID6 BID19 BID5 BID29 BID11 BID2 .
And not only are adversarial attacks successful in white-box attacks, i.e. when the adversary has access to the DNN parameters, but they are also successful in black-box attacks, i.e. it has no access to the parameters.
Black-box attacks are successful because one can perturb an image so it misclassifies on one DNN, and the same perturbed image also has a significant chance to be misclassified by another DNN; this is known as transferability of adversarial examples BID22 ).
Due to this transferability, it is very easy to attack neural nets in a blackbox fashion BID14 BID4 .
In fact, there exist universal perturbations that can imperceptibly perturb any image and cause misclassification for any given network (MoosaviDezfooli et al. (2017) ).
There is much recent research on designing advanced adversarial attacks and defending against adversarial perturbation.In this work, we propose to defend against adversarial attacks by changing the DNNs' output activation function to a manifold-interpolating function, in order to seamlessly utilize the training data's information when performing inference.
Together with the total variation minimization (TVM) and augmented training, we show state-of-the-art defense results on the CIFAR-10 benchmark.
Moreover, we show that adversarial images generated from attacking the DNNs with an interpolating function are more transferable to other DNNs, than those resulting from attacking standard DNNs.

assistant: text: We proposal strategies for adversarial defense based on data dependent activation function, total variation minimization, and training data augmentation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning a Spatio-Temporal Embedding for Video Instance Segmentation

Paper content:
Understanding object motion is one of the core problems in computer vision.
It requires segmenting and tracking objects over time.
Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time.
We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation.
Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time.
Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding.
Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.

assistant: text: We introduce a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation, even with occlusions and missed detections, using appearance, geometry, and temporal context.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing Transformation-Based Defenses Against Adversarial Attacks with a Distribution Classifier

Paper content:
Adversarial attacks on convolutional neural networks (CNN) have gained significant attention and there have been active research efforts on defense mechanisms.
Stochastic input transformation methods have been proposed, where the idea is to recover the image from adversarial attack by random transformation, and to take the majority vote as consensus among the random samples.
However, the transformation improves the accuracy on adversarial images at the expense of the accuracy on clean images.
While it is intuitive that the accuracy on clean images would deteriorate, the exact mechanism in which how this occurs is unclear.
In this paper, we study the distribution of softmax induced by stochastic transformations.
We observe that with random transformations on the clean images, although the mass of the softmax distribution could shift to the wrong class, the resulting distribution of softmax could be used to correct the prediction.
Furthermore, on the adversarial counterparts, with the image transformation, the resulting shapes of the distribution of softmax are similar to the distributions from the clean images.
With these observations, we propose a method to improve existing transformation-based defenses.
We train a separate lightweight distribution classifier to recognize distinct features in the distributions of softmax outputs of transformed images.
Our empirical studies show that our distribution classifier, by training on distributions obtained from clean images only, outperforms majority voting for both clean and adversarial images.
Our method is generic and can be integrated with existing transformation-based defenses.

assistant: text: We enhance existing transformation-based defenses by using a distribution classifier on the distribution of softmax obtained from transformed images.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks.
Existing techniques rely on two stages: searching over the architecture space and validating the best architecture.
NAS algorithms are currently compared solely based on their results on the downstream task.
While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies.
In this paper, we propose to evaluate the NAS search phase.
To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection.
We find that: (i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy; (ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process.
We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.

assistant: text: We empirically disprove a fundamental hypothesis of the widely-adopted weight sharing strategy in neural architecture search and explain why the state-of-the-arts NAS algorithms performs similarly to random search.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: UaiNets: From Unsupervised to Active Deep Anomaly Detection

Paper content:
This work presents a method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection.
We show that a prior needs to be assumed on what the anomalies are, in order to have performance guarantees in unsupervised anomaly detection.
We argue that active anomaly detection has, in practice, the same cost of unsupervised anomaly detection but with the possibility of much better results.
To solve this problem, we present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method, presenting results on both synthetic and real anomaly detection datasets.

assistant: text: A method for active anomaly detection. We present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarially Robust Representations with Smooth Encoders

Paper content:
This paper studies the undesired phenomena of over-sensitivity of representations learned by deep networks to semantically-irrelevant changes in data.
We identify a cause for this shortcoming in the classical Variational Auto-encoder (VAE) objective, the evidence lower bound (ELBO).
We show that the ELBO fails to control the behaviour of the encoder out of the support of the empirical data distribution and this behaviour of the VAE can lead to extreme errors in the learned representation.
This is a key hurdle in the effective use of representations for data-efficient learning and transfer.
To address this problem, we propose to augment the data with specifications that enforce insensitivity of the representation with respect to families of transformations.
To incorporate these specifications, we propose a regularization method that is based on a selection mechanism that creates a fictive data point by explicitly perturbing an observed true data point.
For certain choices of parameters, our formulation naturally leads to the minimization of the entropy regularized Wasserstein distance between representations.
We illustrate our approach on standard datasets and experimentally show that significant improvements in the downstream adversarial accuracy can be achieved by learning robust representations completely in an unsupervised manner, without a reference to a particular downstream task and without a costly supervised adversarial training procedure.

assistant: text: We propose a method for computing adversarially robust representations in an entirely unsupervised way.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Tensor Decompositions for Temporal Knowledge Base Completion

Paper content:
Most algorithms for representation learning and link prediction in relational data have been designed for static data.
However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems.
This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama, [2009-2017]) that are valid only at certain points in time.
For the problem of link prediction under temporal constraints, i.e., answering queries of the form (US, has president, ?, 2012), we propose a solution inspired by the canonical decomposition of tensors of order 4.
We introduce new regularization schemes and present an extension of ComplEx that achieves state-of-the-art performance.
Additionally, we propose a new dataset for knowledge base completion constructed from Wikidata, larger than previous benchmarks by an order of magnitude, as a new reference for evaluating temporal and non-temporal link prediction methods.

assistant: text: We propose new tensor decompositions and associated regularizers to obtain state of the art performances on temporal knowledge base completion.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Word Mover's Embedding: From Word2Vec to Document Embedding

Paper content:
Learning effective text representations is a key foundation for numerous machine learning and NLP applications.
While the celebrated Word2Vec technique yields semantically rich word representations, it is less clear whether sentence or document representations should be built upon word representations or from scratch.
Recent work has demonstrated that a distance measure between documents called \emph{Word Mover's Distance} (WMD) that aligns semantically similar words, yields unprecedented KNN classification accuracy.
However, WMD is very expensive to compute, and is harder to apply beyond simple KNN than feature embeddings.
In this paper, we propose the \emph{Word Mover's Embedding } (WME), a novel approach to building an unsupervised document (sentence) embedding from pre-trained word embeddings.
Our technique extends the theory of \emph{Random Features} to show convergence of the inner product between WMEs to a positive-definite kernel that can be interpreted as a soft version of (inverse) WMD.
The proposed embedding is more efficient and flexible than WMD in many situations.
As an example, WME with a simple linear classifier reduces the computational cost of WMD-based KNN \emph{from cubic to linear} in document length and \emph{from quadratic to linear} in number of samples, while simultaneously improving accuracy.
In experiments on 9 benchmark text classification datasets and 22 textual similarity tasks the proposed technique consistently matches or outperforms state-of-the-art techniques, with significantly higher accuracy on problems of short length.

assistant: text: A novel approach to building an unsupervised document (sentence) embeddings from pre-trainedword embeddings
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering

Paper content:
Very recently, it comes to be a popular approach for answering open-domain questions by first searching question-related passages, then applying reading comprehension models to extract answers.
Existing works usually extract answers from single passages independently, thus not fully make use of the multiple searched passages, especially for the some questions requiring several evidences, which can appear in different passages, to be answered.
The above observations raise the problem of evidence aggregation from multiple passages.
In this paper, we deal with this problem as answer re-ranking.
Specifically, based on the answer candidates generated from the existing state-of-the-art QA model, we propose two different re-ranking methods, strength-based and coverage-based re-rankers, which make use of the aggregated evidences from different passages to help entail the ground-truth answer for the question.
Our model achieved state-of-the-arts on three public open-domain QA datasets, Quasar-T, SearchQA and the open-domain version of TriviaQA, with about 8\% improvement on the former two datasets.

assistant: text: We propose a method that can make use of the multiple passages information for open-domain QA.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DyNet: Dynamic Convolution for Accelerating Convolution Neural Networks

Paper content:
Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost.
To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models.
Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between convolution kernels.
To address this issue, we propose a novel dynamic convolution method named \textbf{DyNet} in this paper, which can adaptively generate convolution kernels based on image contents.
To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs.
The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged.
Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 40.0%, 56.7%, 68.2% and 72.4% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1.0%, -0.27%, -0.6% and -0.08%.
Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87x,1.32x and 1.48x on CPU platform respectively.
To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

assistant: text: We propose a dynamic convolution method to significantly accelerate inference time of CNNs while maintaining the accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DNN Feature Map Compression using Learned Representation over GF(2)

Paper content:
In this paper, we introduce a method to compress intermediate feature maps of deep neural networks (DNNs) to decrease memory storage and bandwidth requirements during inference.
Unlike previous works, the proposed method is based on converting fixed-point activations into vectors over the smallest GF(2) finite field followed by nonlinear dimensionality reduction (NDR) layers embedded into a DNN.
Such an end-to-end learned representation finds more compact feature maps by exploiting quantization redundancies within the fixed-point activations along the channel or spatial dimensions.
We apply the proposed network architecture to the tasks of ImageNet classification and PASCAL VOC object detection.
Compared to prior approaches, the conducted experiments show a factor of 2 decrease in memory requirements with minor degradation in accuracy while adding only bitwise computations.

assistant: text: Feature map compression method that converts quantized activations into binary vectors followed by nonlinear dimensionality reduction layers embedded into a DNN
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reparameterized Variational Divergence Minimization for Stable Imitation

Paper content:
State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior.
Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes.
Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in significant differences in model performance.
Choices including Wasserstein distance and various $f$-divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning.
Unfortunately, we find that in practice this existing imitation-learning framework for using $f$-divergences suffers from numerical instabilities stemming from the combination of function approximation and policy-gradient reinforcement learning.
In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as $f$-divergence minimization before further extending the framework to handle the problem of imitation from observations only.
Empirically, we demonstrate that our design choices for coupling imitation learning and $f$-divergences are critical to recovering successful imitation policies.
Moreover, we find that with the appropriate choice of $f$-divergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continous-control tasks with low-dimensional observation spaces.
With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work.

assistant: text: The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DEEP-TRIM: REVISITING L1 REGULARIZATION FOR CONNECTION PRUNING OF DEEP NETWORK

Paper content:
State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones.
The compression of DNN models has therefore become an active area of research recently, with \emph{connection pruning} emerging as one of the most successful strategies.
A very natural approach is to prune connections of DNNs via $\ell_1$ regularization, but recent empirical investigations have suggested that this does not work as well in the context of DNN compression.
In this work, we revisit this simple strategy and analyze it rigorously, to show that:
(a) any \emph{stationary point} of an $\ell_1$-regularized layerwise-pruning objective has its number of non-zero elements bounded by the number of penalized prediction logits, regardless of the strength of the regularization;
(b) successful pruning highly relies on an accurate optimization solver, and there is a trade-off between compression speed and distortion of prediction accuracy, controlled by the strength of regularization.
Our theoretical results thus suggest that $\ell_1$ pruning could be successful provided we use an accurate optimization solver.
We corroborate this in our experiments, where we show that simple $\ell_1$ regularization with an Adamax-L1(cumulative) solver gives pruning ratio competitive to the state-of-the-art.

assistant: text: We revisit the simple idea of pruning connections of DNNs through $\ell_1$ regularization achieving state-of-the-art results on multiple datasets with theoretic guarantees.