ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Online Explanation Generation for Human-Robot Teaming

Paper content:
As Artificial Intelligence (AI) becomes an integral part of our life, the development of explainable AI, embodied in the decision-making process of an AI or robotic agent, becomes imperative.
For a robotic teammate, the ability to generate explanations to explain its behavior is one of the key requirements of an explainable agency.
Prior work on explanation generation focuses on supporting the reasoning behind the robot's behavior.
These approaches, however, fail to consider the mental workload needed to understand the received explanation.
In other words, the human teammate is expected to understand any explanation provided, often before the task execution, no matter how much information is presented in the explanation.
In this work, we argue that an explanation, especially complex ones, should be made in an online fashion during the execution, which helps spread out the information to be explained and thus reducing the mental workload of humans.
However, a challenge here is that the different parts of an explanation are dependent on each other, which must be taken into account when generating online explanations.
To this end, a general formulation of online explanation generation is presented along with three different implementations satisfying different online properties.
We base our explanation generation method on a model reconciliation setting introduced in our prior work.
Our approaches are evaluated both with human subjects in a standard planning competition (IPC) domain, using NASA Task Load Index (TLX), as well as in simulation with ten different problems across two IPC domains.

assistant: text: We introduce online explanation to consider the cognitive requirement of the human for understanding the generated explanation by the agent.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Simplicity in Deep Reinforcement Learning: Streamlined Off-Policy Learning

Paper content:
The field of Deep Reinforcement Learning (DRL) has recently seen a surge in the popularity of maximum entropy reinforcement learning algorithms.
Their popularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks.
In this paper, we seek to understand the primary contribution  of the entropy term to the performance of maximum entropy algorithms.
For the Mujoco benchmark, we demonstrate that the entropy term in Soft Actor Critic (SAC) principally addresses the bounded nature of the action spaces.
With this insight, we propose a simple normalization scheme which allows a streamlined algorithm without entropy maximization match the performance of SAC.
Our experimental results demonstrate a need to revisit the benefits of entropy regularization in DRL.
We also propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training.
We further show that the streamlined algorithm with the simple non-uniform sampling scheme outperforms SAC and achieves state-of-the-art performance on challenging continuous control tasks.

assistant: text: We propose a new DRL off-policy algorithm achieving state-of-the-art performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep neuroethology of a virtual rodent

Paper content:
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems.
However, this interaction between fields is less developed in the study of motor control.
In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control.
We then use this platform to study motor activity across contexts by training a model to solve four complex tasks.
Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals.
We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics.
These representations are reflected in the sequential activity and population dynamics of neural subpopulations.
Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.

assistant: text: We built a physical simulation of a rodent, trained it to solve a set of tasks, and analyzed the resulting networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

Paper content:
Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese--English news translation task.
We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents.
In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences.
Our findings emphasise the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs.

assistant: text: Raters prefer adequacy in human over machine translation when evaluating entire documents, but not when evaluating single sentences.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TWIN GRAPH CONVOLUTIONAL NETWORKS: GCN WITH DUAL GRAPH SUPPORT FOR SEMI-SUPERVISED LEARNING

Paper content:
Graph Neural Networks as a combination of Graph Signal Processing and Deep Convolutional Networks shows great power in pattern recognition in non-Euclidean domains.
In this paper, we propose a new method to deploy two pipelines based on the duality of a graph to improve accuracy.
By exploring the primal graph and its dual graph where nodes and edges can be treated as one another, we have exploited the benefits of both vertex features and edge features.
As a result, we have arrived at a framework that has great potential in both semisupervised and unsupervised learning.

assistant: text: A primal dual graph neural network model for semi-supervised learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Out-of-Distribution Detection Using Layerwise Uncertainty in Deep Neural Networks

Paper content:
In this paper, we tackle the problem of detecting samples that are not drawn from the training distribution, i.e., out-of-distribution (OOD) samples, in classification.
Many previous studies have attempted to solve this problem by regarding samples with low classification confidence as OOD examples using deep neural networks (DNNs).
However, on difficult datasets or models with low classification ability, these methods incorrectly regard in-distribution samples close to the decision boundary as OOD samples.
This problem arises because their approaches use only the features close to the output layer and disregard the uncertainty of the features.
Therefore, we propose a method that extracts the uncertainties of features in each layer of DNNs using a reparameterization trick and combines them.
In experiments, our method outperforms the existing methods by a large margin, achieving state-of-the-art detection performance on several datasets and classification models.
For example, our method increases the AUROC score of prior work (83.8%) to 99.8% in DenseNet on the CIFAR-100 and Tiny-ImageNet datasets.

assistant: text: We propose a method that extracts the uncertainties of features in each layer of DNNs and combines them for detecting OOD samples when solving classification tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Word translation without parallel data

Paper content:
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora.
Recent studies showed that the need for parallel data supervision can be alleviated with character-level information.
While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet.
In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way.
Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs.
Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese.
We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation.
Our code, embeddings and dictionaries are publicly available.

assistant: text: Aligning languages without the Rosetta Stone: with no parallel data, we construct bilingual dictionaries using adversarial training, cross-domain local scaling, and an accurate proxy criterion for cross-validation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hybed: Hyperbolic Neural Graph Embedding

Paper content:
Neural embeddings have been used with great success in Natural Language Processing (NLP) where they provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks.
The success of neural embeddings has prompted significant amounts of research into applications in domains other than language.
One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks including edge prediction and vertex labelling.
For both NLP and graph-based tasks, embeddings in high-dimensional Euclidean spaces have been learned.
However, recent work has shown that the appropriate isometric space for embedding complex networks is not the flat Euclidean space, but a negatively curved hyperbolic space.
We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space.
We provide experimental evidence that hyperbolic embeddings significantly outperform Euclidean embeddings on vertex classification tasks for several real-world public datasets.

assistant: text: We learn neural embeddings of graphs in hyperbolic instead of Euclidean space
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Parametric Constraints in High Dimensions from Demonstrations

Paper content:
We extend the learning from demonstration paradigm by providing a method for learning unknown constraints shared across tasks, using demonstrations of the tasks, their cost functions, and knowledge of the system dynamics and control constraints.
Given safe demonstrations, our method uses hit-and-run sampling to obtain lower cost, and thus unsafe, trajectories.
Both safe and unsafe trajectories are used to obtain a consistent representation of the unsafe set via solving a mixed integer program.
Additionally, by leveraging a known parameterization of the constraint, we modify our method to learn parametric constraints in high dimensions.
We show that our method can learn a six-dimensional pose constraint for a 7-DOF robot arm.

assistant: text: We can learn high-dimensional constraints from demonstrations by sampling unsafe trajectories and leveraging a known constraint parameterization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Explainable Models Using Attribution Priors

Paper content:
Two important topics in deep learning both involve incorporating humans into the modeling process: Model priors transfer information from humans to a model by regularizing the model's parameters; Model attributions transfer information from a model to humans by explaining the model's behavior.
Previous work has taken important steps to connect these topics through various forms of gradient regularization.
We find, however, that existing methods that use attributions to align a model's behavior with human intuition are ineffective.
We develop an efficient and theoretically grounded feature attribution method, expected gradients, and a novel framework, attribution priors, to enforce prior expectations about a model's behavior during training.
We demonstrate that attribution priors are broadly applicable by instantiating them on three different types of data: image data, gene expression data, and health care data.
Our experiments show that models trained with attribution priors are more intuitive and achieve better generalization performance than both equivalent baselines and existing methods to regularize model behavior.

assistant: text: A method for encouraging axiomatic feature attributions of a deep model to match human intuition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Egregious Responses in Neural Sequence-to-sequence Models

Paper content:
In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq)  model to generate egregious outputs (aggressive, malicious, attacking, etc.).
And if such inputs exist, how to find them efficiently.
We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them.
Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users.
In our experiments, we apply this approach to  dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses.
We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.

assistant: text: This paper aims to provide an empirical answer to the question of whether well-trained dialogue response model can output malicious responses.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning

Paper content:
Most approaches in generalized zero-shot learning rely on cross-modal mapping between an image feature space and a class embedding space or on generating artificial image features.
However, learning a shared cross-modal embedding by aligning the latent spaces of modality-specific autoencoders is shown to be promising in (generalized) zero-shot learning.
While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier.
We evaluate our learned latent features on conventional benchmark datasets and establish a new state of the art on generalized zero-shot as well as on few-shot learning.
Moreover, our results on ImageNet with various zero-shot splits show that our latent features generalize well in large-scale settings.

assistant: text: We use VAEs to learn a shared latent space embedding between image features and attributes and thereby achieve state-of-the-art results in generalized zero-shot learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bias-Resilient Neural Network

Paper content:
Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years.
Such challenges range from spurious associations of confounding variables in medical studies to the bias of race in gender or face recognition systems.
One solution is to enhance datasets and organize them such that they do not reflect biases, which is a cumbersome and intensive task.
The alternative is to make use of available data and build models considering these biases.
Traditional statistical methods apply straightforward techniques such as residualization or stratification to precomputed features to account for confounding variables.
However, these techniques are not in general applicable to end-to-end deep learning methods.
In this paper, we propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s).
This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features.
We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset.
Our results show that the learned features by our method not only result in superior prediction performance but also are uncorrelated with the bias or confounder variables.
The code is available at http://blinded_for_review/.

assistant: text: We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Supersymmetric Artificial Neural Network

Paper content:
The “Supersymmetric Artificial Neural Network” in deep learning (denoted (x; θ, bar{θ})Tw), espouses the importance of considering biological constraints in the aim of further generalizing backward propagation.
Looking at the progression of ‘solution geometries’; going from SO(n) representation (such as Perceptron like models) to SU(n) representation (such as UnitaryRNNs) has guaranteed richer and richer representations in weight space of the artificial neural network, and hence better and better hypotheses were generatable.
The Supersymmetric Artificial Neural Network explores a natural step forward, namely SU(m|n) representation.
These supersymmetric biological brain representations (Perez et al.) can be represented by supercharge compatible special unitary notation SU(m|n), or (x; θ, bar{θ})Tw parameterized by θ, bar{θ}, which are supersymmetric directions, unlike θ seen in the typical non-supersymmetric deep learning model.
Notably, Supersymmetric values can encode or represent more information than the typical deep learning model, in terms of “partner potential” signals for example.

assistant: text: Generalizing backward propagation, using formal methods from supersymmetry.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Paper content:
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales.
Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments.
Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave.
This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms.
The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

assistant: text: We train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure, enabled by the new MAESTRO dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks.
Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception.
In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue.
Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized.
In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t.
the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task.
For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task.
In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of the one-step ahead prediction is not always correlated with downstream control performance.
This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed.
We propose an initial method to mitigate the mismatch issue by re-weighting dynamics model training.
Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.

assistant: text: We define, explore, and begin to address the objective mismatch issue in model-based reinforcement learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Permutation Processes

Paper content:
We introduce a neural architecture to perform amortized approximate Bayesian inference over latent random permutations of two sets of objects.
The method involves approximating permanents of matrices of pairwise probabilities using recent ideas on functions defined over sets.
Each sampled permutation comes with a probability estimate, a quantity unavailable in MCMC approaches.
We illustrate the method in sets of 2D points and MNIST images.
Posterior inference in generative models with discrete latent variables presents well-known challenges when the variables live in combinatorially large spaces.
In this work we focus on the popular and non-trivial case where the latent variables represent random permutations.
While inference in these models has been studied in the past using MCMC techniques (Diaconis, 2009 ) and variational methods , here we propose an amortized approach, whereby we invest computational resources to train a model, which later is used for very fast posterior inference (Gershman and Goodman, 2014) .
Unlike the variational autoencoder approach (Kingma and Welling, 2013) , in our case we do not learn a generative model.
Instead, the latter is postulated (through its samples) and posterior inference is the main focus of the learning phase.
This approach has been recently explored in sundry contexts, such as Bayesian networks (Stuhlmüller et al., 2013) , sequential Monte Carlo (Paige and Wood, 2016) , probabilistic programming (Ritchie et al., 2016; Le et al., 2016) , neural decoding (Parthasarathy et al., 2017) and particle tracking (Sun and Paninski, 2018) .
Our method is inspired by the technique introduced in (Pakman and Paninski, 2018 ) to perform amortized inference over discrete labels in mixture models.
The basic idea is to use neural networks to express posteriors in the form of multinomial distributions (with varying support) in terms of fixed-dimensional, distributed representations that respect the permutation symmetries imposed by the discrete variables.
After training the neural architecture using labeled samples from a particular generative model, we can obtain independent approximate posterior samples of the permutation posterior for any new set of observations of arbitrary size.
These samples can be used to compute approximate expectations, as high quality importance samples, or as independent Metropolis-Hastings proposals.
Our results on simple datasets validate this approach to posterior inference over latent permutations.
More complex generative models with latent permutations can be approached using similar tools, a research direction we are presently exploring.
The curves show mean training negative log-likelihood/iteration in the MNIST example.
f = 0 is a baseline model, were we ignore the unassigned points in (9).
The other two curves correspond to encoding the symmetry p(y n , x cn ) = p(x cn , y n ) as f (g(H x,cn ) + g(H y )) or as f (H x,cn H y ).

assistant: text: A novel neural architecture for efficient amortized inference over latent permutations
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GPU Memory Management for Deep Neural Networks Using Deep Q-Network

Paper content:
Deep neural networks use deeper and broader structures to achieve better performance and consequently, use increasingly more GPU memory as well.
However, limited GPU memory restricts many potential designs of neural networks.
In this paper, we propose a reinforcement learning based variable swapping and recomputation algorithm to reduce the memory cost, without sacrificing the accuracy of models.
Variable swapping can transfer variables between CPU and GPU memory to reduce variables stored in GPU memory.
Recomputation can trade time for space by removing some feature maps during forward propagation.
Forward functions are executed once again to get the feature maps before reuse.
However, how to automatically decide which variables to be swapped or recomputed remains a challenging problem.
To address this issue, we propose to use a deep Q-network(DQN) to make plans.
By combining variable swapping and recomputation, our results outperform several well-known benchmarks.

assistant: text: We propose a reinforcement learning based variable swapping and recomputation algorithm to reduce the memory cost.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neocortical plasticity: an unsupervised cake but no free lunch

Paper content:
The fields of artificial intelligence and neuroscience have a long history of fertile bi-directional interactions.
On the one hand, important inspiration for the development of artificial intelligence systems has come from the study of natural systems of intelligence, the mammalian neocortex in particular.
On the other, important inspiration for models and theories of the brain have emerged from artificial intelligence research.
A central question at the intersection of these two areas is concerned with the processes by which neocortex learns, and the extent to which they are analogous to the back-propagation training algorithm of deep networks.
Matching the data efficiency, transfer and generalisation properties of neocortical learning remains an area of active research in the field of deep learning.
Recent advances in our understanding of neuronal, synaptic and dendritic physiology of the neocortex suggest new approaches for unsupervised representation learning, perhaps through a new class of objective functions, which could act alongside or in lieu of back-propagation.
Such local learning rules have implicit rather than explicit objectives with respect to the training data, facilitating domain adaptation and generalisation.
Incorporating them into deep networks for representation learning could better leverage unlabelled datasets to offer significant improvements in data efficiency of downstream supervised readout learning, and reduce susceptibility to adversarial perturbations, at the cost of a more restricted domain of applicability.

assistant: text: Inspiration from local dendritic processes of neocortical learning to make unsupervised learning great again.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CZ-GEM:  A  FRAMEWORK  FOR DISENTANGLED REPRESENTATION LEARNING

Paper content:
Learning disentangled representations of  data is one of the central themes in unsupervised learning in general and generative modelling in particular.
In this work,  we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate.
To this end, we present a hierarchical model and a training method (CZ-GEM) that leverages some of the recent developments in likelihood-based and likelihood-free generative models.
We show that by formulation, CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variate disentangled.
This is achieved without compromising on the quality of the generated samples.
Our approach is simple, general, and can be applied both in supervised and unsupervised settings.

assistant: text: Hierarchical generative model (hybrid of VAE and GAN) that learns a disentangled representation of data without compromising the generative quality.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

Paper content:
We study the properties of common loss surfaces through their Hessian matrix.
In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk.
We present numerical evidence and mathematical justifications to the following conjectures laid out by Sagun et.
al. (2016): Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers.
We believe that our observations have striking implications for non-convex optimization in high dimensions.
First, the *flatness* of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading.
And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create *large* connected components at the bottom of the landscape.
Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs.
With this in mind, we may reevaluate the connections within the data-architecture-algorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications.
In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

assistant: text: The loss surface is *very* degenerate, and there are no barriers between large batch and small batch solutions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generating Differentially Private Datasets Using GANs

Paper content:
In this paper, we present a technique for generating artificial datasets that retain statistical properties of the real data while providing differential privacy guarantees with respect to this data.
We include a Gaussian noise layer in the discriminator of a generative adversarial network to make the output and the gradients differentially private with respect to the training data, and then use the generator component to synthesise privacy-preserving artificial dataset.
Our experiments show that under a reasonably small privacy budget we are able to generate data of high quality and successfully train machine learning models on this artificial data.

assistant: text: Train GANs with differential privacy to generate artificial privacy-preserving datasets.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MMD GAN with Random-Forest Kernels

Paper content:
In this paper, we propose a novel kind of kernel, random forest kernel, to enhance the empirical performance of MMD GAN.
Different from common forests with deterministic routings, a probabilistic routing variant is used in our innovated random-forest kernel, which is possible to merge with the CNN frameworks.
Our proposed random-forest kernel has the following advantages: From the perspective of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning.
In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure.
Besides, being an asymmetric kernel, our random-forest kernel is much more flexible, in terms of capturing the differences between distributions.
Sharing the advantages of CNN, kernel method, and ensemble learning, our random-forest kernel based MMD GAN obtains desirable empirical performances on CIFAR-10, CelebA and LSUN bedroom data sets.
Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.

assistant: text: Equip MMD GANs with a new random-forest kernel.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Metropolis-Hastings view on variational inference and adversarial training

Paper content:
In this paper we propose to view the acceptance rate of the Metropolis-Hastings algorithm as a universal objective for learning to sample from target distribution -- given either as a set of samples or in the form of unnormalized density.
This point of view unifies the goals of such approaches as Markov Chain Monte Carlo (MCMC), Generative Adversarial Networks (GANs), variational inference.
To reveal the connection we derive the lower bound on the acceptance rate and treat it as the objective for learning explicit and implicit samplers.
The form of the lower bound allows for doubly stochastic gradient optimization in case the target distribution factorizes (i.e. over data points).
We empirically validate our approach on Bayesian inference for neural networks and generative models for images.

assistant: text: Learning to sample via lower bounding the acceptance rate of the Metropolis-Hastings algorithm
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robustness Verification for Transformers

Paper content:
Robustness verification that aims to formally certify the prediction behavior of  neural networks has become an important tool for understanding the behavior of a given model and for obtaining safety guarantees.
However, previous methods are usually limited to relatively simple neural networks.
In this paper, we consider the robustness verification problem for Transformers.
Transformers have complex self-attention layers that pose many challenges for verification, including cross-nonlinearity and cross-position dependency, which have not been discussed in previous work.
We resolve these challenges and develop the first verification algorithm for Transformers.
The certified robustness bounds computed by our method are significantly tighter than those by naive Interval Bound Propagation.
These bounds also shed light on interpreting Transformers as they consistently reflect the importance of words in sentiment analysis.

assistant: text: We propose the first algorithm for verifying the robustness of Transformers.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Selective Convolutional Units: Improving CNNs via Channel Selectivity

Paper content:
Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently.
In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels.
In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks.
During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of
(a) pruning unimportant channels, and
(b) rewiring the pruned parameters to important channels.
The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it.
Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.

assistant: text: We propose a new module that improves any ResNet-like architectures by enforcing "channel selective" behavior to convolutional layers
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Neural Tangents is a library for working with infinite-width neural networks.
It provides a high-level API for specifying complex and hierarchical neural network architectures.
These networks can then be trained and evaluated either at finite-width as usual, or in their infinite-width limit.
For the infinite-width networks, Neural Tangents performs exact inference either via Bayes' rule or gradient descent, and generates the corresponding Neural Network Gaussian Process and Neural Tangent kernels.
Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks.
The entire library runs out-of-the-box on CPU, GPU, or TPU.
All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices.
In addition to the repository below, we provide an accompanying interactive Colab notebook at
https://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb

assistant: text: Keras for infinite neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Predicting the accuracy of neural networks from final and intermediate layer outputs

Paper content:
We show that information about whether a neural network's output will be correct or incorrect is present in the outputs of the network's intermediate layers.
To demonstrate this effect, we train a new "meta" network to predict from either the final output of the underlying "base" network or the output of one of the base network's intermediate layers whether the base network will be correct or incorrect for a particular input.
We find that, over a wide range of tasks and base networks, the meta network can achieve accuracies ranging from 65% - 85% in making this determination.

assistant: text: Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Weak contraction mapping and optimization

Paper content:
The weak contraction mapping is a self mapping that the range is always a subset of the domain, which admits a unique fixed-point.
The iteration of weak contraction mapping is a Cauchy sequence that yields the unique fixed-point.
A gradient-free optimization method as an application of weak contraction mapping is proposed to achieve global minimum convergence.
The optimization method is robust to local minima and initial point position.

assistant: text: A gradient-free method is proposed for non-convex optimization problem
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Capturing Human Category Representations by Sampling in Deep Feature Spaces

Paper content:
Understanding how people represent categories is a core problem in cognitive science, with the flexibility of human learning remaining a gold standard to which modern artificial intelligence and machine learning aspire.
Decades of psychological research have yielded a variety of formal theories of categories, yet validating these theories with naturalistic stimuli remains a challenge.
The problem is that human category representations cannot be directly observed and running informative experiments with naturalistic stimuli such as images requires having a workable representation of these stimuli.
Deep neural networks have recently been successful in a range of computer vision tasks and provide a way to represent the features of images.
In this paper, we introduce a method for estimating the structure of human categories that draws on ideas from both cognitive science and machine learning, blending human-based algorithms with state-of-the-art deep representation learners.
We provide qualitative and quantitative results as a proof of concept for the feasibility of the method.
Samples drawn from human distributions rival the quality of current state-of-the-art generative models and outperform alternative methods for estimating the structure of human categories.

assistant: text: using deep neural networks and clever algorithms to capture human mental visual concepts
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Confidence Calibration in Deep Neural Networks through Stochastic Inferences

Paper content:
We propose a generic framework to calibrate accuracy and confidence (score) of a prediction through stochastic inferences in deep neural networks.
We first analyze relation between variation of multiple model parameters for a single example inference and variance of the corresponding prediction scores by Bayesian modeling of stochastic regularization.
Our empirical observation shows that accuracy and score of a prediction are highly correlated with variance of multiple stochastic inferences given by stochastic depth or dropout.
Motivated by these facts, we design a novel variance-weighted confidence-integrated loss function that is composed of two cross-entropy loss terms with respect to ground-truth and uniform distribution, which are balanced by variance of stochastic prediction scores.
The proposed loss function enables us to learn deep neural networks that predict confidence calibrated scores using a single inference.
Our algorithm presents outstanding confidence calibration performance and improves classification accuracy with two popular stochastic regularization techniques---stochastic depth and dropout---in multiple models and datasets; it alleviates overconfidence issue in deep neural networks significantly by training networks to achieve prediction accuracy proportional to confidence of prediction.

assistant: text: We propose a framework to learn confidence-calibrated networks by designing a novel loss function that incorporates predictive uncertainty estimated through stochastic inferences.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Inferring Dynamical Systems with Long-Range Dependencies through Line Attractor Regularization

Paper content:
Vanilla RNN with ReLU activation have a simple structure that is amenable to systematic dynamical systems analysis and interpretation, but they suffer from the exploding vs. vanishing gradients problem.
Recent attempts to retain this simplicity while alleviating the gradient problem are based on proper initialization schemes or orthogonality/unitary constraints on the RNN’s recurrency matrix, which, however, comes with limitations to its expressive power with regards to dynamical systems phenomena like chaos or multi-stability.
Here, we instead suggest a regularization scheme that pushes part of the RNN’s latent subspace toward a line attractor configuration that enables long short-term memory and arbitrarily slow time scales.
We show that our approach excels on a number of benchmarks like the sequential MNIST or multiplication problems, and enables reconstruction of dynamical systems which harbor widely different time scales.

assistant: text: We develop a new optimization approach for vanilla ReLU-based RNN that enables long short-term memory and identification of arbitrary nonlinear dynamical systems with widely differing time scales.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules

Paper content:
The brain performs unsupervised learning and (perhaps) simultaneous supervised learning.
This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning.
Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal.
We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights.
We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone.
This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods.
We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training.
We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence.

assistant: text: Metalearning unsupervised update rules for neural networks improves performance and potentially demonstrates how neurons in the brain learn without access to global labels.