ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Incorporating Horizontal Connections in Convolution by Spatial Shuffling

Paper content:
Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.
The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.
In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF.
As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.
However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.
In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).
In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.
We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

assistant: text: Proposes SS convulation which uses information outside of its RF, showing improved results when tested on multiple CNN models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: POLYNOMIAL ACTIVATION FUNCTIONS

Paper content:
Activation is a nonlinearity function that plays a predominant role in the convergence and performance of deep neural networks.
While Rectified Linear Unit (ReLU) is the most successful activation function, its derivatives have shown superior performance on benchmark datasets.
In this work, we explore the polynomials as activation functions (order ≥ 2) that can approximate continuous real valued function within a given interval.
Leveraging this property, the main idea is to learn the nonlinearity, accepting that the ensuing function may not be monotonic.
While having the ability to learn more suitable nonlinearity, we cannot ignore the fact that it is a challenge to achieve stable performance due to exploding gradients - which is prominent with the increase in order.
To handle this issue, we introduce dynamic input scaling, output scaling, and lower learning rate for the polynomial weights.
Moreover, lower learning rate will control the abrupt fluctuations of the polynomials between weight updates.
In experiments on three public datasets, our proposed method matches the performance of prior activation functions, thus providing insight into a network’s nonlinearity preference.

assistant: text: We propose polynomial as activation functions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Function-Specific Word Representations

Paper content:
We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.
Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together.
The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure.
We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference (i.e., thematic fit) and event similarity.
The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%.
The proposed framework is versatile and holds promise to support learning function-specific representations beyond the SVO structures.

assistant: text: The paper proposed a method for training function-specific word vectors, in which each word is represented with three vectors each in a different category (Subject-Verb-Object).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Paper content:
Recent improvements in large-scale language models have driven progress on automatic generation of syntactically and semantically consistent text for many real-world applications.
Many of these advances leverage the availability of large corpora.
While training on such corpora encourages the model to understand long-range dependencies in text, it can also result in the models internalizing the social biases present in the corpora.
This paper aims to quantify and reduce biases exhibited by language models.
Given a conditioning context (e.g. a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g. country names, occupations, genders, etc.) in the conditioning context, a.k.a. counterfactual evaluation.
We quantify these biases by adapting individual and group fairness metrics from the fair machine learning literature.
Extensive evaluation on two different corpora (news articles and Wikipedia) shows that state-of-the-art Transformer-based language models exhibit biases learned from data.
We propose embedding-similarity and sentiment-similarity regularization methods that improve both individual and group fairness metrics without sacrificing perplexity and semantic similarity---a positive step toward development and deployment of fairer language models for real-world applications.

assistant: text: This paper measures sentiment bias in language models as reflected by text generated by the models, and adds other objective terms to the usual language modeling objective to reduce bias.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Statestream: A toolbox to explore layerwise-parallel deep neural networks

Paper content:
Building deep neural networks to control autonomous agents which have to interact in real-time with the physical world, such as robots or automotive vehicles, requires a seamless integration of time into a network’s architecture.
The central question of this work is, how the temporal nature of reality should be reflected in the execution of a deep neural network and its components.
Most artificial deep neural networks are partitioned into a directed graph of connected modules or layers and the layers themselves consist of elemental building blocks, such as single units.
For most deep neural networks, all units of a layer are processed synchronously and in parallel, but layers themselves are processed in a sequential manner.
In contrast, all elements of a biological neural network are processed in parallel.
In this paper, we define a class of networks between these two extreme cases.
These networks are executed in a streaming or synchronous layerwise-parallel manner, unlocking the layers of such networks for parallel processing.
Compared to the standard layerwise-sequential deep networks, these new layerwise-parallel networks show a fundamentally different temporal behavior and flow of information, especially for networks with skip or recurrent connections.
We argue that layerwise-parallel deep networks are better suited for future challenges of deep neural network design, such as large functional modularized and/or recurrent architectures as well as networks allocating different network capacities dependent on current stimulus and/or task complexity.
We layout basic properties and discuss major challenges for layerwise-parallel networks.
Additionally, we provide a toolbox to design, train, evaluate, and online-interact with layerwise-parallel networks.

assistant: text: A new toolbox for deep neural networks learning and evaluation, and proposal for a paradigm switch from layerwise-sequential networks to layer-wise parallel networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS

Paper content:
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.
The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.
We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
The introduction of normalizing layers to neural networks has in no small part contributed to the deep learning revolution in machine learning.
The most successful of these techniques in the image classification domain is the batch normalization (BatchNorm) layer BID4 , which works by normalizing the univariate first and second order statistics between layers.Batchnorm has seen near universal adoption in image classification tasks due to its surprisingly multifaceted benefits.
Compared to an unnormalized network, its has been widely observed that using batch norm empirically results in:• Stability over a wide range of step sizes • Faster convergence (particularly with larger step sizes)• Improved generalizationThe multiple effects of BatchNorm make it both hard to replace and hard to analyze.
In this paper we introduce Equilibrium Normalization (EquiNorm), a normalization that works in weight space and still uses a form of batch statistics unlike previous weight space approaches.
EquiNorm results in very rapid convergence, even more so than BatchNorm, however as we will show in our experiments, this also results in a tendency to overfit.
When combined with additional regularisation, EquiNorm can significantly outperform BatchNorm, which benefits less from this additional regularisation.

assistant: text: This manuscript introduces a new layer-wise transform, EquiNorm, to improve upon batch normalization that does not modify the inputs to the layers but rather the layer weights.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spectral Capsule Networks

Paper content:
In search for more accurate predictive models, we customize capsule networks for the learning to diagnose problem.
We also propose Spectral Capsule Networks, a novel variation of capsule networks, that converge faster than capsule network with EM routing.
Spectral capsule networks  consist of spatial coincidence filters that detect entities based on the alignment of extracted features on a one-dimensional linear subspace.
Experiments on a public benchmark learning to diagnose dataset not only shows the success of capsule networks on this task, but also confirm the faster convergence of the spectral capsule networks.

assistant: text: The paper proposes an improved routing method, which employs tools of eigendecomposition to find capsule activation and pose.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Evidential Uncertainty

Paper content:
Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial.
While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions.
In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target.
We accomplish this by  placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution.
We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output.
Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters.
We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.

assistant: text: This paper proposed deep evidential regression, a method for training neural networks to not only estimate the output but also the associated evidence in support of that output.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue

Paper content:
Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.
As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter.
The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge.
Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation.
We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks.
We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe et al., 2018).

assistant: text: Our approach is the first attempt to leverage a sequential latent variable model for knowledge selection in the multi-turn knowledge-grounded dialogue. It achieves the new state-of-the-art performance on Wizard of Wikipedia benchmark.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Do deep neural networks learn shallow learnable examples first?

Paper content:
In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models.
We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples.
We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process.
Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable.
In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/

assistant: text: We analyze the training process for Deep Networks and show that they start from rapidly learning shallow classifiable examples and slowly generalize to harder data points.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Deep Learning Models Resistant to Adversarial Attacks

Paper content:
Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.
To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.
This approach provides us with a broad and unifying view on much prior work on this topic.
Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.
In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.
These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.
They also suggest robustness against a first-order adversary as a natural security guarantee.
We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

assistant: text: This paper proposes to look at making neural networks resistant to adversarial loss through the framework of saddle-point problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Projection Based Constrained Policy Optimization

Paper content:
In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs.
We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set.
We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update.
We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence.
Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

assistant: text: We propose a new algorithm that learns constraint-satisfying policies, and provide theoretical analysis and empirical demonstration in the context of reinforcement learning with constraints.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarially Robust Neural Networks via Optimal Control: Bridging Robustness with Lyapunov Stability

Paper content:
Deep neural networks are known to be vulnerable to adversarial perturbations.
In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems.
From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets.
This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust.
The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently.
Experiments show that our method effectively improves deep model's adversarial robustness.

assistant: text: This paper uses the theoretical view of a neural network as a discretized ODE to develop a robust control theory aimed at training the network while enforcing robustness.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Consistent Performance on Atari using Expert Demonstrations

Paper content:
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games.
We identify three key challenges that any algorithm needs to master in order to perform well on all games:  processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently.
In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games.
A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of 0.999 (instead of 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states.
When tested on a set of 42 Atari games, our algorithm exceeds the performance  of an average human on 40 games using a common set of hyper parameters.

assistant: text: Ape-X DQfD = Distributed (many actors + one learner + prioritized replay) DQN with demonstrations optimizing the unclipped 0.999-discounted return on Atari.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: When is a Convolutional Filter Easy to Learn?

Paper content:
We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.
Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.
We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.
To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.
Our theory also justifies the two-stage learning rate strategy in deep neural networks.
While our focus is theoretical, we also present experiments that justify our theoretical findings.

assistant: text: This paper extends the Gaussian distribution assumption to a more general angular smoothness assumption, which covers a wider family of input distributions
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Value Propagation Networks

Paper content:
We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments.
We show that the modules enable learning to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems.
We evaluate on static and dynamic configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes, and on a StarCraft navigation scenario, with more complex dynamics, and pixels as input.

assistant: text: Proposes methods, which can be seen as modifications of Value Iteration Networks (VIN), with some improvements aimed at improving sample efficiency and generalization to large environment sizes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Open-Set Domain Adaptation with Category-Agnostic Clusters

Paper content:
Unsupervised domain adaptation has received significant attention in recent years.
Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories.
In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class).
The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source.
In this paper, we address this problem by augmenting the state-of-the-art domain adaptation technique, Self-Ensembling, with category-agnostic clusters in target domain.
Specifically, we present Self-Ensembling with Category-agnostic Clusters (SE-CC) --- a novel architecture that steers domain adaptation with the additional guidance of category-agnostic clusters that are specific to target domain.
These clustering information provides domain-specific visual cues, facilitating the generalization of Self-Ensembling for both closed-set and open-set scenarios.
Technically, clustering is firstly performed over all the unlabeled target samples to obtain the category-agnostic clusters, which reveal the underlying data space structure peculiar to target domain.
A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample.
Furthermore, SE-CC enhances the learnt representation with mutual information maximization.
Extensive experiments are conducted on Office and VisDA datasets for both open-set and closed-set domain adaptation, and superior results are reported when comparing to the state-of-the-art approaches.

assistant: text: A new approach to open set domain adaptation, where the source domain categories are contained in the target domain categories in order to filter out outlier categories and enable adaptation within the shared classes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fixing Weight Decay Regularization in Adam

Paper content:
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.
We provide empirical evidence that our proposed modification
(i) 
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and
(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).
We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.
Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32.
Our source code will become available after the review process.

assistant: text: Proposes idea to decouple the weight decay from the number of steps taken by the optimization process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Seed-Augment-Train Framework for Universal Digit Classification 

Paper content:
In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that contains a synthetic seed image dataset generation procedure for languages with different numeral systems using freely available open font file datasets.
This seed dataset of images is then augmented to create a purely synthetic training dataset, which is in turn used to train a deep neural network and test on held-out real world handwritten digits dataset spanning five Indic scripts, Kannada, Tamil, Gujarati, Malayalam, and Devanagari.
We showcase the efficacy of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN) that generates realistic digit images in the five languages, and also qualitatively by testing a CNN trained on the synthetic data on the real-world datasets.
This establishes not only an interesting nexus between the font-datasets-world and transfer learning but also provides a recipe for universal-digit classification in any script.

assistant: text: Is seeding and augmentation all you need for classifying digits in any language?
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compositional Transfer in Hierarchical Reinforcement Learning

Paper content:
The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency.
To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time.
To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL).
We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference.
Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions.
Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives.
We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting.
Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

assistant: text: We develop a hierarchical, actor-critic algorithm for compositional transfer by sharing policy components and demonstrate component specialization and related direct benefits in multitask domains as well as its adaptation for single tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Mixed-curvature Variational Autoencoders

Paper content:
It has been shown that using geometric spaces with non-zero curvature instead of plain Euclidean spaces with zero curvature improves performance on a range of Machine Learning tasks for learning representations.
Recent work has leveraged these geometries to learn latent variable models like Variational Autoencoders (VAEs) in spherical and hyperbolic spaces with constant curvature.
While these approaches work well on particular kinds of data that they were designed for e.g.~tree-like data for a hyperbolic VAE, there exists no generic approach unifying all three models.
We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature can be learned.
This generalizes the Euclidean VAE to curved latent spaces, as the model essentially reduces to the Euclidean VAE if curvatures of all latent space components go to 0.

assistant: text: This paper introduces a general formulation of the notion of a VAE with a latent space composed by a curved manifold.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms

Paper content:
Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles.
As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood.
In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma.
By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance.
In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality".
The Stein Variational Gradient Descent (SVGD) (Liu and Wang, 2016 ) is a deterministic particle-based inference algorithm that iteratively transports the particles by the functional gradient in the reproducing kernel Hilbert space (RKHS) of KL-divergence, which takes the form of a kernelized Stein's operator.
In contrast to the empirical successes (Liu et al., 2017; Haarnoja et al., 2017; Kim et al., 2018) , very few convergence guarantees have been established for SVGD except for the mean-field regime (Liu and Wang, 2018; Lu et al., 2019) .
Moreover, it has been observed that the variance estimated by SVGD scales inversely with the dimensionality of the problem.
This is a highly undesirable property for two reasons:
1) underestimating the variance leads to failures of explaining the uncertainty of model predictions;
2) modern inference problems are usually high-dimensional.
For example, Bayesian neural networks (MacKay, 1992) could be more than millions of dimensions.
We study the algorithmic bias of SVGD that leads to the variance underestimation in high dimensions.
We construct another kernel-based inference algorithm termed MMDdescent, which closely resembles SVGD but estimate the variance accurately.
By comparing their updates, we identify the cause of variance collapse in SVGD as a combination of high variance due to Stein's lemma, and deterministic bias, i.e. the inability to resample particles.
We empirically verify that removing either of these two factors, while computationally expensive, leads to accurate variance estimation.
Then, under mild assumptions, we derive the equilibrium variance of SVGD and MMD-descent in matching high-dimensional Gaussians, and confirm that variance estimated by SVGD scales inversely with the dimensionality.

assistant: text: Analyze the underlying mechanisms of variance collapse of SVGD in high dimensions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Correcting Nuisance Variation using Wasserstein Distance

Paper content:
Profiling cellular phenotypes from microscopic imaging can provide meaningful biological information resulting from various factors affecting the cells.
One motivating application is drug development: morphological cell features can be captured from images, from which similarities between different drugs applied at different dosages can be quantified.
The general approach is to find a function mapping the images to an embedding space of manageable dimensionality whose geometry captures relevant features of the input images.
An important known issue for such methods is separating relevant biological signal from nuisance variation.
For example, the embedding vectors tend to be more correlated for cells that were cultured and imaged during the same week than for cells from a different week, despite having identical drug compounds applied in both cases.
In this case, the particular batch a set of experiments were conducted in constitutes the domain of the data; an ideal set of image embeddings should contain only the relevant biological information (e.g. drug effects).
We develop a general framework for adjusting the image embeddings in order to `forget' domain-specific information while preserving relevant biological information.
To do this, we minimize a loss function based on distances between marginal distributions (such as the Wasserstein distance) of embeddings across domains for each replicated treatment.
For the dataset presented, the replicated treatment is the negative control.
We find that for our transformed embeddings (1) the underlying geometric structure is not only preserved but the embeddings also carry improved biological signal (2) less domain-specific information is present.

assistant: text: Discusses a method for adjusting image embeddings in order tease apart technical variation from biological signal.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SGD Learns One-Layer Networks in WGANs

Paper content:
Generative adversarial networks (GANs) are a widely used framework for learning generative models.
Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.
In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.

assistant: text: Attempts to prove that the Stochastic Gradient Decent-Ascent could converge to a global solution for the min-max problem of WGAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Transfer Learning to Learn with Multitask Neural Model Search

Paper content:
Deep learning models require extensive architecture design exploration and hyperparameter optimization to perform well on a given task.
The exploration of the model design space is often made by a human expert, and optimized using a combination of grid search and search heuristics over a large space of possible choices.
Neural Architecture Search (NAS) is a Reinforcement Learning approach that has been proposed to automate architecture design.
NAS has been successfully applied to generate Neural Networks that rival the best human-designed architectures.
However, NAS requires sampling, constructing, and training hundreds to thousands of models to achieve well-performing architectures.
This procedure needs to be executed from scratch for each new task.
The application of NAS to a wide set of tasks currently lacks a way to transfer generalizable knowledge across tasks.
In this paper, we present the Multitask Neural Model Search (MNMS) controller.
Our goal is to learn a generalizable framework that can condition model construction on successful model searches for previously seen tasks, thus significantly speeding up the search for new tasks.
We demonstrate that MNMS can conduct an automated architecture search for multiple tasks simultaneously while still learning well-performing, specialized models for each task.
We then show that pre-trained MNMS controllers can transfer learning to new tasks.
By leveraging knowledge from previous searches, we find that pre-trained MNMS models start from a better location in the search space and reduce search time on unseen tasks, while still discovering models that outperform published human-designed models.

assistant: text: This paper extends Neural Architecture Search to the multi-task learning problem where a task conditioned model search controller is learned to handle multiple tasks simultaneously.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning

Paper content:
Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery.
However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations.
To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?"
We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity.
We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation.

assistant: text: This paper introduces Morpho-MNIST, a collection of shape metrics and perturbations, in a step towards quantitative evaluation of representation learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TequilaGAN: How To Easily Identify GAN Samples

Paper content:
In this paper we show strategies to easily identify fake samples generated with the Generative Adversarial Network framework.
One strategy is based on the statistical analysis and comparison of raw pixel values and features extracted from them.
The other strategy learns formal specifications from the real data and shows that fake samples violate the specifications of the real data.
We show that fake samples produced with GANs have a universal signature that can be used to identify fake samples.
We provide results on MNIST, CIFAR10, music and speech data.

assistant: text: The paper proposes statistics to identify fake data generated using GANs based on simple marginal statistics or formal specifications automatically generated from real data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intriguing Properties of Adversarial Examples

Paper content:
It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples.
In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear.
Here we show that distributions of logit differences have a universal functional form.
This functional form is independent of architecture, dataset, and training protocol; nor does it change during training.
This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation.
We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD).
Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness.
Finally, we study the effect of network architectures on adversarial sensitivity.
To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10.
Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

assistant: text: Adversarial error has similar power-law form for all datasets and models studied, and architecture matters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MMA Training: Direct Input Space Margin Maximization through Adversarial Training

Paper content:
We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary.
Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins.
We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness.
Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint.
In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins.
Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

assistant: text: An adaptive margin-based adversarial training approach to train robust DNNs, by maximizing the shortest margin of inputs to the decision boundary, that makes adversarial training with large perturbation possible.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generating Multi-Agent Trajectories using Programmatic Weak Supervision

Paper content:
We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.
When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.
Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way.
We present a hierarchical framework that can effectively learn such sequential generative models.
Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.
In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.
We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

assistant: text: The paper proposes training generative models that produce multi-agent trajectories using heuristic functions that label variables that would otherwise be latent in training data
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sample-efficient policy learning in multi-agent Reinforcement Learning via meta-learning

Paper content:
To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions.
We can solve these tasks by first building models for other agents and then finding the optimal policy with these models.
To get an accurate model, many observations are needed and this can be sample-inefficient.
What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents.
In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution.
Thus we can treat the task against some specific agents as a task sampled from a task distribution.
We apply meta-learning method to build models and learn policies.
Therefore when new agents come, we can adapt to them efficiently.
Experiments on grid games show that our method can quickly get high rewards.

assistant: text: This paper focuses on fast adaptation to new behaviour of the other agents of the environment using a method based on MAML
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Text Classifier on Test-Time Budgets

Paper content:
In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-time
budget constraints.
We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words.
In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words.
Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus.
Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.

assistant: text: The authors consider training a RNN-based text classification where there is a resource restriction on test-time prediction, and provide an approach using a masking mechanism to reduce words/phrases/sentences used in prediction followed by a classifier to handle those components.