ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MixUp as Directional Adversarial Training

Paper content:
MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients.
Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained.
Because samples are preferentially moved in the direction of other classes \iffalse -- which are typically clustered in input space -- \fi we refer to this method as directional adversarial training, or DAT.
We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT.
We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples.
We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes.
Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp.
In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.

assistant: text: We present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An information-theoretic analysis of deep latent-variable models

Paper content:
We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference.
This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate).
We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier.
However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable.
Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.

assistant: text: We provide an information theoretic and experimental analysis of state-of-the-art variational autoencoders.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Working memory facilitates reward-modulated Hebbian learning in recurrent neural networks

Paper content:
Reservoir computing is a powerful tool to explain how the brain learns temporal sequences, such as movements, but existing learning schemes are either biologically implausible or too inefficient to explain animal performance.
We show that a network can learn complicated sequences with a reward-modulated Hebbian learning rule if the network of reservoir neurons is combined with a second network that serves as a dynamic working memory and provides a spatio-temporal backbone signal to the reservoir.
In combination with the working memory, reward-modulated Hebbian learning of the readout neurons performs as well as FORCE learning, but with the advantage of a biologically plausible interpretation of both the learning rule and the learning paradigm.

assistant: text: We show that a working memory input to a reservoir network makes a local reward-modulated Hebbian rule perform as well as recursive least-squares (aka FORCE)
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Subtask Discovery with Non-Negative Matrix Factorization

Paper content:
Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains.
However, learning an appropriate hierarchical decomposition of a domain into subtasks remains a substantial challenge.
We present a novel algorithm for subtask discovery, based on the recently introduced multitask linearly-solvable Markov decision process (MLMDP) framework.
The MLMDP can perform never-before-seen tasks by representing them as a linear combination of a previously learned basis set of tasks.
In this setting, the subtask discovery problem can naturally be posed as finding an optimal low-rank approximation of the set of tasks the agent will face in a domain.
We use non-negative matrix factorization to discover this minimal basis set of tasks, and show that the technique learns intuitive decompositions in a variety of domains.
Our method has several qualitatively desirable features: it is not limited to learning subtasks with single goal states, instead learning distributed patterns of preferred states; it learns qualitatively different hierarchical decompositions in the same domain depending on the ensemble of tasks the agent will face; and it may be straightforwardly iterated to obtain deeper hierarchical decompositions.

assistant: text: We present a novel algorithm for hierarchical subtask discovery which leverages the multitask linear Markov decision process framework.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Clustering By Predicting And Copying Noise

Paper content:
We propose a neural clustering model that jointly learns both latent features and how they cluster.
Unlike similar methods our model does not require a predefined number of clusters.
Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids.
To show the behavior of our model across different modalities we apply our model on both text and image data and very competitive results on MNIST.
Finally, we also provide results against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create.

assistant: text: Neural clustering without needing a number of clusters
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Paper content:
Deep neural networks are known to be annotation-hungry.
Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks.
Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data.
In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques.
In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner.
To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network.
During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively.
Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods.
Code is available at https://github.com/LiJunnan1992/DivideMix .

assistant: text: We propose a novel semi-supervised learning approach with SOTA performance on combating learning with noisy labels.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Distribution Regression Network

Paper content:
We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.
Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.
On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.
Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).
In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

assistant: text: A learning network which generalizes the MLP framework to perform distribution-to-distribution regression
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Missing Ingredient in Zero-Shot Neural Machine Translation

Paper content:
Multilingual Neural Machine Translation (NMT) systems are capable of translating between multiple source and target languages within a single system.
An important indicator of generalization within these systems is the quality of zero-shot translation - translating between language pairs that the system has never seen during training.
However, until now, the zero-shot performance of multilingual models has lagged far behind the quality that can be achieved by using a two step translation process that pivots through an intermediate language (usually English).
In this work, we diagnose why multilingual models under-perform in zero shot settings.
We propose explicit language invariance losses that guide an NMT encoder towards learning language agnostic representations.
Our proposed strategies significantly improve zero-shot translation performance on WMT English-French-German and on the IWSLT 2017 shared task, and for the first time, match the performance of pivoting approaches while maintaining performance on supervised directions.

assistant: text: Simple similarity constraints on top of multilingual NMT enables high quality translation between unseen language pairs for the first time.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning an off-policy predictive state representation for deep reinforcement learning for vision-based steering in autonomous driving

Paper content:
An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning.
There are three components being learned simultaneously:  (1) the off-policy predictions as a compact representation of state, (2) the behavior policy distribution for estimating the off-policy predictions, and (3) the deterministic policy gradient for learning to act.
A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs).
A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned.
All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment.
Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller.
Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model.

assistant: text: An algorithm to learn a predictive state representation with general value functions and off-policy learning is applied to the problem of vision-based steering in autonomous driving.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs

Paper content:
The vertebrate visual system is hierarchically organized to process visual information in successive stages.
Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation.
There is currently no unified theory explaining these differences in representations across layers.
Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the first time we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing.
The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck.
Second, we find that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks.
This result predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli.
These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.

assistant: text: We reproduced neural representations found in biological visual systems by simulating their neural resource constraints in a deep convolutional model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems.

Paper content:
Human-computer conversation systems have attracted much attention in Natural Language Processing.
Conversation systems can be roughly divided into two categories: retrieval-based and generation-based systems.
Retrieval systems search a user-issued utterance (namely a query) in a large conversational repository and return a reply that best matches the query.
Generative approaches synthesize new replies.
Both ways have certain advantages but suffer from their own disadvantages.
We propose a novel ensemble of retrieval-based and generation-based conversation system.
The retrieved candidates, in addition to the original query, are fed to a reply generator via a neural network, so that the model is aware of more information.
The generated reply together with the retrieved ones then participates in a re-ranking process to find the final reply to output.
Experimental results show that such an ensemble system outperforms each single module by a large margin.

assistant: text: A novel ensemble of retrieval-based and generation-based for open-domain conversation systems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Span Recovery for Deep Neural Networks with Applications to Input Obfuscation

Paper content:
The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical results proposed have only been for shallow networks.
In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery.
For $k<n$, let $\mathbf{A} \in \mathbb{R}^{k \times n}$ be the innermost weight matrix of an arbitrary feed forward neural network $M: \mathbb{R}^n \to  \mathbb{R}$, so $M(x)$ can be written as $M(x) = \sigma(\mathbf{A} x)$, for some network $\sigma: \mathbb{R}^k \to  \mathbb{R}$.
The goal is then to recover the row span of $\mathbf{A}$ given only oracle access to the value of $M(x)$. We show that if $M$ is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover $k/2$ linearly independent vectors in the row span of $\mathbf{A}$ using poly$(n)$ non-adaptive queries to $M(x)$.  Furthermore, if $M$ has differentiable activation functions, we demonstrate that \textit{full} span recovery is possible even when the output is first passed through a sign or $0/1$ thresholding function; in this case our algorithm is adaptive.
Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers.
For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data.
Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs.

assistant: text: We provably recover the span of a deep multi-layered neural network with latent structure and empirically apply efficient span recovery algorithms to attack networks by obfuscating inputs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SHREWD: Semantic Hierarchy Based Relational Embeddings For Weakly-Supervised Deep Hashing

Paper content:
Using class labels to represent class similarity is a typical approach to training deep hashing systems for retrieval; samples from the same or different classes take binary 1 or 0 similarity values.
This similarity does not model the full rich knowledge of semantic relations that may be present between data points.
In this work we build upon the idea of using semantic hierarchies to form distance metrics between all available sample labels; for example cat to dog has a smaller distance than cat to guitar.
We combine this type of semantic distance into a loss function to promote similar distances between the deep neural network embeddings.
We also introduce an empirical Kullback-Leibler divergence loss term to promote binarization and uniformity of the embeddings.
We test the resulting SHREWD method and demonstrate improvements in hierarchical retrieval scores using compact, binary hash codes instead of real valued ones, and show that in a weakly supervised hashing setting we are able to learn competitively without explicitly relying on class labels, but instead on similarities between labels.

assistant: text: We propose a new method for training deep hashing for image retrieval using only a relational distance metric between samples
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Connectivity Learning in Multi-Branch Networks

Paper content:
While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that  branching, i.e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance.
To combat the complexity of design choices in multi-branch architectures, prior work has adopted simple strategies, such as a fixed branching factor, the same input being fed to all parallel branches, and an additive combination of the outputs produced by all branches at aggregation points.
In this work we remove these predefined choices and propose an algorithm to learn the connections between branches in the network.
Instead of being chosen a priori by the human designer, the multi-branch connectivity is learned simultaneously with the weights of the network by optimizing a single loss function defined with respect to the end task.
We demonstrate our approach on the problem of multi-class image classification using four different datasets where it yields consistently higher accuracy compared to the state-of-the-art ``ResNeXt'' multi-branch network given the same learning capacity.

assistant: text: In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models

Paper content:
A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network.
Likelihood-based training  of  these  models  requires  restricting  their  architectures  to  allow  cheap computation of Jacobian determinants.
Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation.
In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density.
The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures.
We demonstrate our approach on high-dimensional density  estimation,  image  generation,  and  variational  inference,  achieving  the state-of-the-art among exact likelihood methods with efficient sampling.
We have presented FFJORD, a reversible generative model for high-dimensional data which can compute exact log-likelihoods and can be sampled from efficiently.
Our model uses continuoustime dynamics to produce a generative model which is parameterized by an unrestricted neural network.
All required quantities for training and sampling can be computed using automatic differentiation, Hutchinson's trace estimator, and black-box ODE solvers.
Our model stands in contrast to other methods with similar properties which rely on restricted, hand-engineered neural network architectures.
We demonstrated that this additional flexibility allows our approach to achieve on-par or improved performance on density estimation and variational inference.We believe there is much room for further work exploring and improving this method.
FFJORD is empirically slower to evaluate than other reversible models like Real NVP or Glow, so we are interested specifically in ways to reduce the number of function evaluations used by the ODE-solver without hurting predictive performance.
Advancements like these will be crucial in scaling this method to even higher-dimensional datasets.

assistant: text: We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Value of Incorporating Social Preferences in Dynamic Ridesharing

Paper content:
Dynamic ridesharing services (DRS) play a major role in improving the efficiency of urban transportation.
User satisfaction in dynamic ridesharing is determined by multiple factors such as travel time, cost, and social compatibility with co-passengers.
Existing DRS optimize profit by maximizing the operational value for service providers or minimize travel time for users but they neglect the social experience of riders, which significantly influences the total value of the service to users.
We propose DROPS, a dynamic ridesharing framework that factors the riders' social preferences in the matching process so as to improve the quality of the trips formed.
Scheduling trips for users is a multi-objective optimization that aims to maximize the operational value for the service provider, while simultaneously maximizing the value of the trip for the users.
The user value is estimated based on compatibility between co-passengers and the ride time.
We then present a real-time matching algorithm for trip formation.
Finally, we evaluate our approach empirically using real-world taxi trips data, and a population model including social preferences based on user surveys.
The results demonstrate improvement in riders' social compatibility, without significantly affecting the vehicle miles for the service provider and travel time for users.

assistant: text: We propose a novel dynamic ridesharing framework to form trips that optimizes both operational value for the service provider and user value to the passengers by factoring the users' social preferences into the decision-making process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Regularizing Trajectories to Mitigate Catastrophic Forgetting

Paper content:
Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective.
However in most practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters.
In this paper, we argue for the importance of regularizing optimization trajectories directly.
We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks.
We show that using the co-natural gradient systematically reduces forgetting in continual learning.
Moreover, it helps combat overfitting when learning a new task in a low resource scenario.

assistant: text: Regularizing the optimization trajectory with the Fisher information of old tasks reduces catastrophic forgetting greatly
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning of Sophisticated Curriculums by viewing them as Graphs over Tasks

Paper content:
Curriculum learning consists in learning a difficult task by first training on an easy version of it, then on more and more difficult versions and finally on the difficult task.
To make this learning efficient, given a curriculum and the current learning state of an agent, we need to find what are the good next tasks to train the agent on.
Teacher-Student algorithms assume that the good next tasks are the ones on which the agent is making the fastest progress or digress.
We first simplify and improve them.
However, two problematic situations where the agent is mainly trained on tasks it can't learn yet or it already learnt may occur.
Therefore, we introduce a new algorithm using min max ordered curriculums that assumes that the good next tasks are the ones that are learnable but not learnt yet.
It outperforms Teacher-Student algorithms on small curriculums and significantly outperforms them on sophisticated ones with numerous tasks.

assistant: text: We present a new algorithm for learning by curriculum based on the notion of mastering rate that outperforms previous algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MisGAN: Learning from Incomplete Data with Generative Adversarial Networks

Paper content:
Generative adversarial networks (GANs) have been shown to provide an effective way to model complex distributions and have obtained impressive results on various challenging tasks.
However, typical GANs require fully-observed data during training.
In this paper, we present a GAN-based framework for learning from complex, high-dimensional incomplete data.
The proposed framework learns a complete data generator along with a mask generator that models the missing data distribution.
We further demonstrate how to impute missing data by equipping our framework with an adversarially trained imputer.
We evaluate the proposed framework using a series of experiments with several types of missing data processes under the missing completely at random assumption.

assistant: text: This paper presents a GAN-based framework for learning the distribution from high-dimensional incomplete data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Group-Transformer: Towards A Lightweight Character-level Language Model

Paper content:
Character-level language modeling is an essential but challenging task in Natural Language Processing.
Prior works have focused on identifying long-term dependencies between characters and have built deeper and wider networks for better performance.
However, their models require substantial computational resources, which hinders the usability of character-level language models in applications with limited resources.
In this paper, we propose a lightweight model, called Group-Transformer, that reduces the resource requirements for a Transformer, a promising method for modeling sequence with long-term dependencies.
Specifically, the proposed method partitions linear operations to reduce the number of parameters and computational cost.
As a result, Group-Transformer only uses 18.2\% of parameters compared to the best performing LSTM-based model, while providing better performance on two benchmark tasks, enwik8 and text8.
When compared to Transformers with a comparable number of parameters and time complexity, the proposed model shows better performance.
The implementation code will be available.

assistant: text: This paper proposes a novel lightweight Transformer for character-level language modeling, utilizing group-wise operations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Minima and Regularization Effects

Paper content:
Understanding the behavior of  stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently.
Along this line, we theoretically study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics.
Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects.
A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function.
Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency.
We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to  escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well.
We verify our understanding through comparing
this anisotropic diffusion with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics) and other types of position-dependent noise.

assistant: text: We provide theoretical and empirical analysis on the role of anisotropic noise introduced by stochastic gradient on escaping from minima.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Optimize Neural Nets

Paper content:
Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning.
In this paper, we explore learning an optimization algorithm for training shallow neural nets.
Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms.
We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture.
More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100.

assistant: text: We learn an optimization algorithm that generalizes to unseen tasks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Stochastic Sign Descent Methods

Paper content:
Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models.
Sign-based methods, such as signSGD (Bernstein et al., 2018), have recently been gaining popularity because of their simple compression rule and connection to adaptive gradient methods, like ADAM.
In this paper, we perform a general analysis of sign-based methods for non-convex optimization.
Our analysis is built on intuitive bounds on success probabilities and does not rely on special noise distributions nor on the boundedness of the variance of stochastic gradients.
Extending the theory to distributed setting within a parameter server framework, we assure exponentially fast variance reduction with respect to number of nodes, maintaining 1-bit compression in both directions and using small mini-batch sizes.
We validate our theoretical findings experimentally.

assistant: text: General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Logic Machines

Paper content:
We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning.
NLMs exploit the power of both neural networks---as function approximators, and logic programming---as a symbolic processor for objects with properties, relations, logic connectives, and quantifiers.
After being trained on small-scale tasks (such as sorting short arrays), NLMs can recover lifted rules, and generalize to large-scale tasks (such as sorting longer arrays).
In our experiments, NLMs achieve perfect generalization in a number of tasks, from relational reasoning tasks on the family tree and general graphs, to decision making tasks including sorting arrays, finding shortest paths, and playing the blocks world.
Most of these tasks are hard to accomplish for neural networks or inductive logic programming alone.

assistant: text: We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeepXML: Scalable & Accurate Deep Extreme Classification for Matching User Queries to Advertiser Bid Phrases

Paper content:
The objective in deep extreme multi-label learning is to jointly learn feature representations and classifiers to automatically tag data points with the most relevant subset of labels from an extremely large label set.
Unfortunately, state-of-the-art deep extreme classifiers are either not scalable or inaccurate for short text documents.
This paper develops the DeepXML algorithm which addresses both limitations by introducing a novel architecture that splits training of head and tail labels
.  DeepXML increases accuracy
by (a) learning word embeddings on head labels and transferring them through a novel residual connection to data impoverished tail labels
; (b) increasing the amount of negative training data available by extending state-of-the-art negative sub-sampling techniques; and
(c) re-ranking the set of predicted labels to eliminate the hardest negatives for the original classifier.
All of these contributions are implemented efficiently by extending the highly scalable Slice algorithm for pretrained embeddings to learn the proposed DeepXML architecture.
As a result, DeepXML could efficiently scale to problems involving millions of labels that were beyond the pale of state-of-the-art deep extreme classifiers as it could be more than 10x faster at training than XML-CNN and AttentionXML.
At the same time, DeepXML was also empirically determined to be up to 19% more accurate than leading techniques for matching search engine queries to advertiser bid phrases.

assistant: text: Scalable and accurate deep multi label learning with millions of labels.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Characterizing the Accuracy/Complexity Landscape of Explanations of Deep Networks through Knowledge Extraction

Paper content:
Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models.
The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully.
The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods.
In this paper we examine this question systematically by proposing a knowledge extraction method using \textit{M-of-N} rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN).
Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal \textit{M-of-N} rule to describe its behaviour.
We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable.
The results shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability.

assistant: text: Systematically examines how well we can explain the hidden features of a deep network in terms of logical rules.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression

Paper content:
Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model.
Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size.
This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy.
We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process.
We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint.
Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

assistant: text: We demonstrate that large, but pruned models (large-sparse) outperform their smaller, but dense (small-dense) counterparts with identical memory footprint.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Feature Learning under Accuracy Constraint for Domain Generalization

Paper content:
Learning domain-invariant representation is a dominant approach for domain generalization.
However, previous methods based on domain invariance overlooked the underlying dependency of classes on domains, which is responsible for the trade-off between classification accuracy and the invariance.
This study proposes a novel method {\em adversarial feature learning under accuracy constraint (AFLAC)}, which maximizes domain invariance within a range that does not interfere with accuracy.
Empirical validations show that the performance of AFLAC is superior to that of baseline methods, supporting the importance of considering the dependency and the efficacy of the proposed method to overcome the problem.

assistant: text: Address the trade-off caused by the dependency of classes on domains by improving domain adversarial nets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs

Paper content:
We consider a simple and overarching representation for permutation-invariant functions of sequences (or set functions).
Our approach, which we call Janossy pooling, expresses a permutation-invariant function as the average of a permutation-sensitive function applied to all reorderings of the input sequence.
This allows us to leverage the rich and mature literature on permutation-sensitive functions to construct novel and flexible permutation-invariant functions.
If carried out naively, Janossy pooling can be computationally prohibitive.
To allow computational tractability, we consider three kinds of approximations: canonical orderings of sequences, functions with k-order interactions, and stochastic optimization algorithms with random permutations.
Our framework unifies a variety of existing work in the literature, and suggests possible modeling and algorithmic extensions.
We explore a few in our experiments, which demonstrate improved performance over current state-of-the-art methods.

assistant: text: We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Syntax-Directed Variational Autoencoder for Structured Data

Paper content:
Deep generative models have been enjoying success in modeling continuous data.
However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures.
How to generate both syntactically and semantically correct data still remains largely an open problem.
Inspired by the theory of compiler where syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes.
This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder.
Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable.
We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization.
The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches.

assistant: text: A new generative model for discrete structured data. The proposed stochastic lazy attribute converts the offline semantic check into online guidance for stochastic decoding, which effectively addresses the constraints in syntax and semantics, and also achieves superior performance
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Internal Dense But External Sparse Structures of Deep Neural Network

Paper content:
Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs).
On one hand, increasing the density of CNNs by adding cross-layer connections achieve higher accuracy.
On the other hand, creating sparsity structures through regularization and pruning methods enjoys lower computational costs.
In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections.
This new structure uses dense modules, as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process.
Experimental results demonstrate that the locally dense yet externally sparse structure could acquire competitive performance on benchmark tasks (CIFAR10, CIFAR100, and ImageNet) while keeping the network structure slim.

assistant: text: In this paper, we explore an internal dense yet external sparse network structure of deep neural networks and analyze its key properties.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Attention over Parameters for Dialogue Systems

Paper content:
Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans.
For example, different domains (e.g., restaurant reservation, train ticket booking) of goal-oriented dialogue systems can be viewed as different skills, and so does ordinary chatting abilities of chit-chat dialogue systems.
In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP).
The experimental results show that this approach achieves competitive performance on a combined dataset of MultiWOZ (Budzianowski et al., 2018), In-Car Assistant (Eric et al.,2017), and Persona-Chat (Zhang et al., 2018).
Finally, we demonstrate that each dialogue skill is effectively learned and can be combined with other skills to produce selective responses.

assistant: text: In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP).