ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LabelFool: A Trick in the Label Space

Paper content:
It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.
However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.
In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.
To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.
We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.
Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

assistant: text: A trick on adversarial samples so that the mis-classified labels are imperceptible in the label space to human observers
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ProxSGD: Training Structured Neural Networks under Regularization and Constraints

Paper content:
In this paper, we consider the problem of training neural networks (NN).
To promote a NN with specific structures, we explicitly take into consideration the nonsmooth regularization (such as L1-norm) and constraints (such as interval constraint).
This is formulated as a constrained nonsmooth nonconvex optimization problem, and we propose a convergent proximal-type stochastic gradient descent (Prox-SGD) algorithm.
We show that under properly selected learning rates, momentum eventually resembles the unknown real gradient and thus is crucial in analyzing the convergence.
We establish that with probability 1, every limit point of the sequence generated by the proposed Prox-SGD is a stationary point.
Then the Prox-SGD is tailored to train a sparse neural network and a binary neural network, and the theoretical analysis is also supported by extensive numerical tests.

assistant: text: This paper proposes Prox-SGD, a theoretical framework for stochastic optimization algorithms shown to converge asymptotically to stationarity for smooth non-convvex loss + convex constraint/regularizer.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Grassmannian initialization: Neural network initialization using sub-space packing

Paper content:
We recently observed that convolutional filters initialized
farthest apart from each other using offthe-
shelf pre-computed Grassmannian subspace
packing codebooks performed surprisingly well
across many datasets.
Through this short paper,
we’d like to disseminate some initial results in this
regard in the hope that we stimulate the curiosity
of the deep-learning community towards considering
classical Grassmannian subspace packing
results as a source of new ideas for more efficient
initialization strategies.

assistant: text: Initialize weights using off-the-shelf Grassmannian codebooks, get  faster training and better accuracy
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Not-So-Random Features

Paper content:
We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels.
Our method produces a sequence of feature maps, iteratively refining the SVM margin.
We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game.
Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.
Choosing the right kernel is a classic question that has riddled machine learning practitioners and theorists alike.
Conventional wisdom instructs the user to select a kernel which captures the structure and geometric invariances in the data.
Efforts to formulate this principle have inspired vibrant areas of study, going by names from feature selection to multiple kernel learning (MKL).We
present a new, principled approach for selecting a translation-invariant or rotation-invariant kernel to maximize the SVM classification margin. We
first describe a kernel-alignment subroutine, which finds a peak in the Fourier transform of an adversarially chosen data-dependent measure. Then
, we define an iterative procedure that produces a sequence of feature maps, progressively improving the margin. The
resulting algorithm is strikingly simple and scalable. Intriguingly
, our analysis interprets the main algorithm as no-regret learning dynamics in a zero-sum min-max game, whose value is the classification margin. Thus, we are
able to quantify convergence guarantees towards the largest margin realizable by a kernel with the assumed invariance. Finally, we
exhibit experiments on synthetic and benchmark datasets, demonstrating consistent improvements over related random features-based kernel methods.
We have presented an efficient kernel learning method that uses tools from Fourier analysis and online learning to optimize over two natural infinite families of kernels.
With this method, we show meaningful improvements on benchmark tasks, compared to related random features-based methods.
Many theoretical questions remain, such as accelerating the search for Fourier peaks (e.g. Hassanieh et al. FORMULA1 ; Kapralov FORMULA1 ).
These, in addition to applying our learned kernels to state-of-the-art methods (e.g. convolutional kernel networks (Mairal et al., 2014; BID21 Mairal, 2016) ), prove to be exciting directions for future work.

assistant: text: A simple and practical algorithm for learning a margin-maximizing translation-invariant or spherically symmetric kernel from training data, using tools from Fourier analysis and regret minimization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LEARNING TO ORGANIZE KNOWLEDGE WITH N-GRAM MACHINES

Paper content:
Deep neural networks (DNNs) had great success on NLP tasks such as language modeling, machine translation and certain question answering (QA) tasks.
However, the success is limited at more knowledge intensive tasks such as QA from a big corpus.
Existing end-to-end deep QA models (Miller et al., 2016; Weston et al., 2014) need to read the entire text after observing the question, and therefore their complexity in responding a question is linear in the text size.
This is prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web.
We propose to solve this scalability issue by using symbolic meaning representations, which can be indexed and retrieved efficiently with complexity that is independent of the text size.
More specifically, we use sequence-to-sequence models to encode knowledge symbolically and generate programs to answer questions from the encoded knowledge.
We apply our approach, called the N-Gram Machine (NGM), to the bAbI tasks (Weston et al., 2015) and a special version of them (“life-long bAbI”) which has stories of up to 10 million sentences.
Our experiments show that NGM can successfully solve both of these tasks accurately and efficiently.
Unlike fully differentiable memory models, NGM’s time complexity and answering quality are not affected by the story length.
The whole system of NGM is trained end-to-end with REINFORCE (Williams, 1992).
To avoid high variance in gradient estimation, which is typical in discrete latent variable models, we use beam search instead of sampling.
To tackle the exponentially large search space, we use a stabilized auto-encoding objective and a structure tweak procedure to iteratively reduce and refine the search space.

assistant: text: The authors propose the N-Gram machine to answer questions over long documents.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives

Paper content:
Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling 2013, Rezende et al. 2014).
These approaches maximize a variational lower bound on the intractable log likelihood of the observed data.
Burda et al. (2015) introduced a multi-sample variational bound, IWAE, that is at least as tight as the standard variational lower bound and becomes increasingly tight as the number of samples increases.
Counterintuitively, the typical inference network gradient estimator for the IWAE bound performs poorly as the number of samples increases (Rainforth et al. 2018, Le et al. 2018).
Roeder et a.
(2017) propose an improved gradient estimator, however, are unable to show it is unbiased.
We show that it is in fact biased and that the bias can be estimated efficiently with a second application of the reparameterization trick.
The doubly reparameterized gradient (DReG) estimator does not suffer as the number of samples increases, resolving the previously raised issues.
The same idea can be used to improve many recently introduced training techniques for latent variable models.
In particular, we show that this estimator reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS) (Bornschein & Bengio 2014), and the jackknife variational inference (JVI) gradient (Nowozin 2018).
Finally, we show that this computationally efficient, drop-in estimator translates to improved performance for all three objectives on several modeling tasks.

assistant: text: Doubly reparameterized gradient estimators provide unbiased variance reduction which leads to improved performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

Paper content:
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner.
Our scheme pits two versions of the same agent, Alice and Bob, against one another.
Alice proposes a task for Bob to complete; and then Bob attempts to complete the task.
In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset.
Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively.
Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent.
When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.

assistant: text: A new formulation for exploring the environment in an unsupervised way to aid a specific task later, where one agent proposes increasingly difficult tasks and the learning agent tries to accomplish them.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discrete flow posteriors for variational inference in discrete dynamical systems

Paper content:
Each training step for a variational autoencoder (VAE) requires us to sample from the approximate posterior, so we usually choose simple (e.g. factorised) approximate posteriors in which sampling is an efficient computation that fully exploits GPU parallelism.
However, such simple approximate posteriors are often insufficient, as they eliminate statistical dependencies in the posterior.
While it is possible to use normalizing flow approximate posteriors for continuous latents, there is nothing analogous for discrete latents.
The most natural approach to model discrete dependencies is an autoregressive distribution, but sampling from such distributions is inherently sequential and thus slow.
We develop a fast, parallel sampling procedure for autoregressive distributions based on fixed-point iterations which enables efficient and accurate variational inference in discrete state-space models.
To optimize the variational bound, we considered two ways to evaluate probabilities: inserting the relaxed samples directly into the pmf for the discrete distribution, or converting to continuous logistic latent variables and interpreting the K-step fixed-point iterations as a normalizing flow.
We found that converting to continuous latent variables gave considerable additional scope for mismatch between the true and approximate posteriors, which resulted in biased inferences, we thus used the former approach.
We tested our approach on the neuroscience problem of inferring discrete spiking activity from noisy calcium-imaging data, and found that it gave accurate connectivity estimates in an order of magnitude less time.

assistant: text: This paper has two main contributions: it extends normalizing flows to discrete settings and presents an approximate fixed-point update rule for autoregressive time-series that can exploit GPU parallelism.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning

Paper content:
Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery.
However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations.
To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?"
We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity.
We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation.

assistant: text: Authors present a set of criteria to categorize MNISt digists and a set of interesting perturbations to modify MNIST dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Optimization of Sorting Networks via Continuous Relaxations

Paper content:
Sorting input objects is an important step in many machine learning pipelines.
However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization.
In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting operator from permutation matrices to the set of unimodal row-stochastic matrices, where every row sums to one and has a distinct argmax.
This relaxation permits straight-through optimization of any computational graph involve a sorting operation.
Further, we use this relaxation to enable gradient-based stochastic optimization over the combinatorially large space of permutations by deriving a reparameterized gradient estimator for the Plackett-Luce family of distributions over permutations.
We demonstrate the usefulness of our framework on three tasks that require learning semantic orderings of high-dimensional objects, including a fully differentiable, parameterized extension of the k-nearest neighbors algorithm

assistant: text: This work builds on a sum(top k) identity to derive a pathwise differentiable sampler of 'unimodal row stochastic' matrices.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Does the neuronal noise in cortex help generalization?

Paper content:
Neural activity is highly variable in response to repeated stimuli.
We used an open dataset, the Allen Brain Observatory, to quantify the distribution of responses to repeated natural movie presentations.
A large fraction of responses are best fit by log-normal distributions or Gaussian mixtures with two components.
These distributions are similar to those from units in deep neural networks with dropout.
Using a separate set of electrophysiological recordings, we constructed a population coupling model as a control for state-dependent activity fluctuations and found that the model residuals also show non-Gaussian distributions.
We then analyzed responses across trials from multiple sections of different movie clips and observed that the noise in cortex aligns better with in-clip versus out-of-clip stimulus variations.
We argue that noise is useful for generalization when it moves along representations of different exemplars in-class, similar to the structure of cortical noise.
One of the hallmarks of neural codes is the high level of trial-to-trial variability [1, 2] .
This variability has been studied using multiple stimuli [3] , along with its relation to attention [4] and other behavioral variables [5] .
Previous theories on the possible role of noise center on its potential usefulness in inference [6] .
In the field of machine learning, noise can have a regularizing effect and enable better model generalization (e.g. dropout [7] ).
Here, we explore the hypothesis that networks of cortical neurons use noise with the goal of building general representations from a small number of exemplars.
First, we show that cortical noise is often non-Gaussian, and better captured by long-tailed distributions or mixtures of Gaussians.
This result was consistent across experiments using two-photon calcium imaging and electrophysiological recordings.
To control for possible state-dependent effects, we used a population coupling model where the activity of all other simultaneously recorded neurons is used to predict the activity of a single neuron (and as a proxy for brain state).
Finally, we defined a set of neural subspace measures and found that cortical noise aligns with in-class stimulus variations.
In the first part of the paper, we observed complex, non-Gaussian distributions in the responses of neurons even for their preferred stimulus.
In the second part of the paper, we found that trial-to-trial noise for an exemplar in a clip aligns better with exemplar-by-exemplar variation in the same clip than for other clips.
We believe that research into the structure and role of biological noise will be useful for developing new methods to train neural networks with better generalization capabilities.

assistant: text: We study the structure of noise in the brain and find it may help generalization by moving representations along in-class stimulus variations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Paper content:
This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations.
To this end, we propose a multi-modal learning framework that links the inference stage and generation stage.
First, the inference networks are trained to match the speaker identity between the two different modalities.
Then the pre-trained inference networks cooperate with the generation network by giving conditional information about the voice.

assistant: text: This paper presents a multi-modal learning framework that links the inference stage and generation stage for seeking the possibility of generating the human face from voice solely.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarially Robust Neural Networks via Optimal Control: Bridging Robustness with Lyapunov Stability

Paper content:
Deep neural networks are known to be vulnerable to adversarial perturbations.
In this paper, we bridge adversarial robustness of neural nets with Lyapunov stability of dynamical systems.
From this viewpoint, training neural nets is equivalent to finding an optimal control of the discrete dynamical system, which allows one to utilize methods of successive approximations, an optimal control algorithm based on Pontryagin's maximum principle, to train neural nets.
This decoupled training method allows us to add constraints to the optimization, which makes the deep model more robust.
The constrained optimization problem can be formulated as a semi-definite programming problem and hence can be solved efficiently.
Experiments show that our method effectively improves deep model's adversarial robustness.

assistant: text: The authors formulate training NNs as finding an optimal controller for a discrete dynamical system, allowing them to use method of successive approximations to train a NN in a way to be more robust to adversarial attacks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Style Memory: Making a Classifier Network Generative

Paper content:
Deep networks have shown great performance in classification tasks.
However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification.
We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network.
We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses.
The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct.
We further investigate the nature of the style memory, and how it relates to composing digits and letters.

assistant: text: Augmenting the top layer of a classifier network with a style memory enables it to be generative.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Fine-Grained Spectral Perspective on Neural Networks

Paper content:
Are neural networks biased toward simple functions?
Does depth always help learn more complex features?
Is training the last layer of a network as good as training all layers?
These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective.
We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*.
Roughly, the CK and the NTK tell us respectively ``"what a network looks like at initialization" and "``what a network looks like during and after training."
Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks.
By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks.
We believe the computational tools we develop here for analyzing the spectra of CK and NTK serve as a solid foundation for future studies of deep neural networks.
We have open-sourced the code for it and for generating the plots in this paper at github.com/jxVmnLgedVwv6mNcGCBy/NNspectra.

assistant: text: Eigenvalues of Conjugate (aka NNGP) and Neural Tangent Kernel can be computed in closed form over the Boolean cube and reveal the effects of hyperparameters on neural network inductive bias, training, and generalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Topological Defects in 2D Active Nematics Using Convolutional Neural Networks

Paper content:
Active matter consists of active agents which transform energy extracted from surroundings into momentum, producing a variety of collective phenomena.
A model, synthetic active system composed of microtubule polymers driven by protein motors spontaneously forms a liquid-crystalline nematic phase.
Extensile stress created by the protein motors precipitates continuous buckling and folding of the microtubules creating motile topological defects and turbulent fluid flows.
Defect motion is determined by the rheological properties of the material; however, these remain largely unquantified.
Measuring defects dynamics can yield fundamental insights into active nematics, a class of materials that include bacterial films and animal cells.
Current methods for defect detection lack robustness and precision, and require fine-tuning for datasets with different visual quality.
In this study, we applied Deep Learning to train a defect detector to automatically analyze microscopy videos of the microtubule active nematic.
Experimental results indicate that our method is robust and accurate.
It is expected to significantly increase the amount of video data that can be processed.

assistant: text: Apply a well known neural model (YOLO) to detect bounding boxes of objects in images.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Learning with Latent Embedding Optimization

Paper content:
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems.
However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes.
We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space.
The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters.
Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks.
Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

assistant: text: This paper proposes a latent embedding optimization method for meta-learning, and claims the contribution is to decouple optimization-based meta-learning techniques from high-dimensional space of model parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Mixed-curvature Variational Autoencoders

Paper content:
It has been shown that using geometric spaces with non-zero curvature instead of plain Euclidean spaces with zero curvature improves performance on a range of Machine Learning tasks for learning representations.
Recent work has leveraged these geometries to learn latent variable models like Variational Autoencoders (VAEs) in spherical and hyperbolic spaces with constant curvature.
While these approaches work well on particular kinds of data that they were designed for e.g.~tree-like data for a hyperbolic VAE, there exists no generic approach unifying all three models.
We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature can be learned.
This generalizes the Euclidean VAE to curved latent spaces, as the model essentially reduces to the Euclidean VAE if curvatures of all latent space components go to 0.

assistant: text: Variational Autoencoders with latent spaces modeled as products of constant curvature Riemannian manifolds improve on image reconstruction over single-manifold variants.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Graph2Seq: Graph to Sequence Learning with Attention-Based Neural Networks

Paper content:
The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks.
However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence.
To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors.
Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings.
We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs.
Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

assistant: text: Graph to Sequence Learning with Attention-Based Neural Networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Relational Multi-Instance Learning for Concept Annotation from Medical Time Series

Paper content:
Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.
Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.
Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.
However, these approaches are slow and do not use the accompanying medical time series data.
To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.
We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.
Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

assistant: text: We propose a deep Multi Instance Learning framework based on recurrent neural networks which uses pooling functions and attention mechanisms for the concept annotation tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TFGAN: Improving Conditioning for Text-to-Video Synthesis

Paper content:
Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning.
In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a GAN model with novel conditioning scheme that aids improving the text-video associations.
With a combination of this conditioning scheme and a deep GAN architecture, TFGAN generates photo-realistic videos from text on very challenging real-world video datasets.
In addition, we construct a benchmark synthetic dataset of moving shapes to systematically evaluate our conditioning scheme.
Extensive experiments demonstrate that TFGAN significantly outperforms the existing approaches, and can also generate videos of novel categories not seen during training.

assistant: text: This paper presents a GAN-based method for video generation conditioned on text description, with a new conditioning method that generates convolution filters from the encoded text, and uses them for a convolution in the discriminator.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks

Paper content:
Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence.
The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices.
This imposes an upper-bound on the reduction of complexity of multiply-accumulate units.
We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training.
Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation.
We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet.
In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline.
We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds.
Overall this analysis enables precise tailoring of computation hardware to the application, yielding area- and power-optimal systems.

assistant: text: The authors conduct a thorough analysis of the numeric precision required for the accumulation operations in neural network training and show the theoretical impact of reducing number of bits in the floating point accumulator.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Synthesizing realistic neural population activity patterns using Generative Adversarial Networks

Paper content:
The ability to synthesize realistic patterns of neural activity is crucial for studying neural information processing.
Here we used the Generative Adversarial Networks (GANs) framework to simulate the concerted activity of a population of neurons.
We adapted the Wasserstein-GAN variant to facilitate the generation of unconstrained neural population activity patterns while still benefiting from parameter sharing in the temporal domain.
We demonstrate that our proposed GAN, which we termed Spike-GAN, generates spike trains that match accurately the first- and second-order statistics of datasets of tens of neurons and also approximates well their higher-order statistics.
We applied Spike-GAN to a real dataset recorded from salamander retina and showed that it performs as well as state-of-the-art approaches based on the maximum entropy and the dichotomized Gaussian frameworks.
Importantly, Spike-GAN does not require to specify a priori the statistics to be matched by the model, and so constitutes a more flexible method than these alternative approaches.
Finally, we show how to exploit a trained Spike-GAN  to construct 'importance maps' to detect the most relevant statistical structures present in a spike train.
Spike-GAN provides a powerful, easy-to-use technique for generating realistic spiking neural activity and for describing the most relevant features of the large-scale neural population recordings studied in modern systems neuroscience.

assistant: text: A method for simulating spike trains from populations of neurons which match empirical data using a semi-convolutional GAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multiplicative Interactions and Where to Find Them

Paper content:
We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated.
We begin by showing that such layers strictly enrich the representable function classes of neural networks.
We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required.
We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation.
Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.

assistant: text: We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Rank Learning Curves

Paper content:
Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations.
In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training.
In contrast to existing methods, we consider this task as a ranking and transfer learning problem.
We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves.
We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture.
In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

assistant: text: The paper proposes a method to rank learning curves of neural networks that can model learning curves across different datasets, achieving higher speed-ups on image classification tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GENERATING HIGH FIDELITY IMAGES WITH SUBSCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING

Paper content:
The unconditional generation of high fidelity images is a longstanding benchmark
for testing the performance of image decoders.
Autoregressive image models
have been able to generate small images unconditionally, but the extension of
these methods to large images where fidelity can be more readily assessed has
remained an open problem.
Among the major challenges are the capacity to encode
the vast previous context and the sheer difficulty of learning a distribution that
preserves both global semantic coherence and exactness of detail.
To address the
former challenge, we propose the Subscale Pixel Network (SPN), a conditional
decoder architecture that generates an image as a sequence of image slices of equal
size.
The SPN compactly captures image-wide spatial dependencies and requires a
fraction of the memory and the computation.
To address the latter challenge, we
propose to use multidimensional upscaling to grow an image in both size and depth
via intermediate stages corresponding to distinct SPNs.
We evaluate SPNs on the
unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32
to 128.
We achieve state-of-the-art likelihood results in multiple settings, set up
new benchmark results in previously unexplored settings and are able to generate
very high fidelity large scale samples on the basis of both datasets.

assistant: text: We show that autoregressive models can generate high fidelity images.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Anomaly Detection with Generative Adversarial Networks

Paper content:
Many anomaly detection methods exist that perform well on low-dimensional problems however there is a notable lack of effective methods for high-dimensional spaces, such as images.
Inspired by recent successes in deep learning we propose a novel approach to anomaly detection using generative adversarial networks.
Given a sample under consideration, our method is based on searching for a good representation of that sample in the latent space of the generator; if such a representation is not found, the sample is deemed anomalous.
We achieve state-of-the-art performance on standard image benchmark datasets and visual inspection of the most anomalous samples reveals that our method does indeed return anomalies.

assistant: text: The authors propose using GAN for anomaly detection, a gradient-descent based method to iteratively update latent representations, and a novel parameter update to the generators.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-task Learning on MNIST Image Datasets

Paper content:
We apply multi-task learning to image classification tasks on MNIST-like datasets.
MNIST dataset has been referred to as the {\em drosophila} of machine learning and has been the testbed of many learning theories.
The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference.
In this work, we exploit these MNIST-like datasets for multi-task learning.
The datasets are pooled together for learning the parameters of joint classification networks.
Then the learned parameters are used as the initial parameters to retrain disjoint classification networks.
The baseline recognition model are all-convolution neural networks.
Without multi-task learning, the recognition accuracies for MNIST, NotMNIST and FashionMNIST are 99.56\%, 97.22\% and 94.32\% respectively.
With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70\%, 97.46\% and 95.25\%.
The results re-affirm that multi-task learning framework, even with data with different genres, does lead to significant improvement.

assistant: text: This paper presents a multi-task neural network for classification on MNIST-like datasets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Compressing U-net Using Knowledge Distillation

Paper content:
We study the use of knowledge distillation to compress the U-net architecture.
We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.
This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.

assistant: text: We present additional techniques to use knowledge distillation to compress U-net by over 1000x.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Paper content:
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019).
However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked.
In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work.
Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation.
The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements.
Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data.
One surprising observation is that they have a compound effect even when sequentially applied on the same data.
To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

assistant: text: Studies how self-supervised learning and knowledge distillation interact in the context of building compact models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Simple Geometric Proof for the Benefit of Depth in ReLU Networks

Paper content:
We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").
Specifically we present a sequence of classification problems f_i such that
(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and
(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.
While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.

assistant: text: A proof that deeper networks need less units than shallower ones for a family of problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Paper content:
One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth.
This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks.
For an $m$ hidden node shallow neural network with ReLU activation and $n$ training data, we show as long as $m$ is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function.
Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum.
We believe these insights are also useful in analyzing deep models and other first order methods.

assistant: text: This paper studies one hidden layer neural networks with square loss, where they show that in over-parameterized setting, random initialization and gradient descent gets to zero loss.