ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Connecting the Dots Between MLE and RL for Sequence Generation

Paper content:
Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms.
For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem.
Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency.
A variety of other algorithms such as RAML, SPG, and data noising, have also been developed in different perspectives.
This paper establishes a formal connection between these algorithms.
We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters.
The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency.
Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning.
Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

assistant: text: A unified perspective of various learning algorithms for sequence generation, such as MLE, RL, RAML, data noising, etc.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method

Paper content:
In recent years, three-dimensional convolutional neural network (3D CNN) are intensively applied in the video analysis and action recognition and receives good performance.
However, 3D CNN leads to massive computation and storage consumption, which hinders its deployment on mobile and embedded devices.
In this paper, we propose a three-dimensional regularization-based pruning method to assign different regularization parameters to different weight groups based on their importance to the network.
Our experiments show that the proposed method outperforms other popular methods in this area.

assistant: text: In this paper, we propose a three-dimensional regularization-based pruning method to accelerate the 3D-CNN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Recognizing Plans by Learning Embeddings from Observed Action Distributions

Paper content:
Plan recognition aims to look for target plans to best explain the observed actions based on plan libraries and/or domain models.
Despite the success of previous approaches on plan recognition, they mostly rely on correct action observations.
Recent advances in visual activity recognition have the potential of enabling applications such as automated video surveillance.
Effective approaches for such problems would require the ability to recognize the plans of agents from video information.
Traditional plan recognition algorithms rely on access to detailed planning domain models.
One recent promising direction involves learning approximate (or shallow) domain models directly from the observed activity sequences.
Such plan recognition approaches expect observed action sequences as inputs.
However, visual inference results are often noisy and uncertain, typically represented as a distribution over possible actions.
In this work, we develop a visual plan recognition framework that recognizes plans with an approximate domain model learned from uncertain visual data.

assistant: text: Handling Uncertainty in Visual Perception for Plan Recognition
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A rotation-equivariant convolutional neural network model of primary visual cortex

Paper content:
Classical models describe primary visual cortex (V1) as a filter bank of orientation-selective linear-nonlinear (LN) or energy models, but these models fail to predict neural responses to natural stimuli accurately.
Recent work shows that convolutional neural networks (CNNs) can be trained to predict V1 activity more accurately, but it remains unclear which features are extracted by V1 neurons beyond orientation selectivity and phase invariance.
Here we work towards systematically studying V1 computations by categorizing neurons into groups that perform similar computations.
We present a framework for identifying common features independent of individual neurons' orientation selectivity by using a rotation-equivariant convolutional neural network, which automatically extracts every feature at multiple different orientations.
We fit this rotation-equivariant CNN to responses of a population of 6000 neurons to natural images recorded in mouse primary visual cortex using two-photon imaging.
We show that our rotation-equivariant network outperforms a regular CNN with the same number of feature maps and reveals a number of common features, which are shared by many V1 neurons and are pooled sparsely to predict neural activity.
Our findings are a first step towards a powerful new tool to study the nonlinear functional organization of visual cortex.

assistant: text: A rotation-equivariant CNN model of V1 that outperforms previous models and suggest functional groupings of V1 neurons.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Inductive and Unsupervised Representation Learning on Graph Structured Objects

Paper content:
Inductive and unsupervised graph learning is a critical technique for predictive or information retrieval tasks where label information is difficult to obtain.
It is also challenging to make graph learning inductive and unsupervised at the same time, as learning processes guided by reconstruction error based loss functions inevitably demand graph similarity evaluation that is usually computationally intractable.
In this paper, we propose a general framework SEED (Sampling, Encoding, and Embedding Distributions) for inductive and unsupervised representation learning on graph structured objects.
Instead of directly dealing with the computational challenges raised by graph similarity evaluation, given an input graph, the SEED framework samples a number of subgraphs whose reconstruction errors could be efficiently evaluated, encodes the subgraph samples into a collection of subgraph vectors, and employs the embedding of the subgraph vector distribution as the output vector representation for the input graph.
By theoretical analysis, we demonstrate the close connection between SEED and graph isomorphism.
Using public benchmark datasets, our empirical study suggests the proposed SEED framework is able to achieve up to 10% improvement, compared with competitive baseline methods.

assistant: text: This paper proposed a novel framework for graph similarity learning in inductive and unsupervised scenario.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Awareness Models

Paper content:
We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world.
We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world.
In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body.
That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals.
Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape.
We show that active data collection by maximizing the entropy of predictions about the body---touch sensors, proprioception and vestibular information---leads to learning of dynamic models that show superior performance when used for control.
We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world.
Videos with qualitative results of our models are available at https://goo.gl/mZuqAV.

assistant: text: We train predictive models on proprioceptive information and show they represent properties of external objects.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving generalization by regularizing in $L^2$ function space

Paper content:
Learning rules for neural networks necessarily include some form of regularization.
Most regularization techniques are conceptualized and implemented in the space of parameters.
However, it is also possible to regularize in the space of functions.
Here, we propose to measure networks in an $L^2$ Hilbert space, and test a learning rule that regularizes the distance a network can travel through $L^2$-space each update.
This approach is inspired by the slow movement of gradient descent through parameter space as well as by the natural gradient, which can be derived from a regularization term upon functional change.
The resulting learning rule, which we call Hilbert-constrained gradient descent (HCGD), is thus closely related to the natural gradient but regularizes a different and more calculable metric over the space of functions.
Experiments show that the HCGD is efficient and leads to considerably better generalization.

assistant: text: It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Optimal Transport Maps For Distribution Preserving Operations on Latent Spaces of Generative Models

Paper content:
Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian.
After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample.
However, the latent space operations commonly used in the literature so far induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on.
Previous works have attempted to reduce this mismatch with heuristic modification to the operations or by changing the latent distribution and re-training models.
In this paper, we propose a framework for modifying the latent space operations such that the distribution mismatch is fully eliminated.
Our approach is based on optimal transport maps, which adapt the latent space operations such that they fully match the prior distribution, while minimally modifying the original operation.
Our matched operations are readily obtained for the commonly used operations and distributions and require no adjustment to the training procedure.

assistant: text: We propose a framework for modifying the latent space operations such that the distribution mismatch between the resulting outputs and the prior distribution the generative model was trained on is fully eliminated.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PAGANDA: An Adaptive Task-Independent Automatic Data Augmentation

Paper content:
Data augmentation is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce.
However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset.
In this paper we propose  \textit{Parallel Adaptive GAN Data Augmentation}(PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel.
We demonstrate by experiments that our data augmentation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks.
Our source code and experimental details are available at \url{https://github.com/miaojiang1987/k-folder-data-augmentation-gan/}.

assistant: text: We present an automated adaptive data augmentation that works for multiple different tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Music Transformer: Generating Music with Long-Term Structure

Paper content:
Music relies heavily on repetition to build structure and meaning.
Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure.
The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence.
This suggests that self-attention might also be well-suited to modeling music.
In musical composition and performance, however, relative timing is critically important.
Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018).
This is impractical for long sequences such as musical compositions since their memory complexity is quadratic in the sequence length.
We propose an algorithm that reduces the intermediate memory requirements to linear in the sequence length.
This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long (thousands of steps) compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies.
We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-competition, and obtain state-of-the-art results on the latter.

assistant: text: We show the first successful use of Transformer in generating music that exhibits long-term structure.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Abstractive Dialog Summarization with Semantic Scaffolds

Paper content:
The demand for abstractive dialog summary is growing in real-world applications.
For example, customer service center or hospitals would like to summarize customer service interaction and doctor-patient interaction.
However, few researchers explored abstractive summarization on dialogs due to the lack of  suitable datasets.
We propose an abstractive dialog summarization dataset based on MultiWOZ.
If we directly apply previous state-of-the-art document summarization methods on dialogs, there are two significant drawbacks: the informative entities such as restaurant names are difficult to preserve, and the contents from different dialog domains are sometimes mismatched.
To address these two drawbacks, we propose Scaffold Pointer Network (SPNet) to utilize the existing annotation on speaker role, semantic slot and dialog domain.
SPNet incorporates these semantic scaffolds for dialog summarization.
Since ROUGE cannot capture the two drawbacks mentioned, we also propose a new evaluation metric that considers critical informative entities in the text.
On MultiWOZ, our proposed SPNet outperforms state-of-the-art abstractive summarization methods on all the automatic and human evaluation metrics.

assistant: text: We propose a novel end-to-end model (SPNet) to incorporate semantic scaffolds for improving abstractive dialog summarization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization

Paper content:
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed on clusters to perform model fitting in parallel.
Alistarh et al. (2017) describe two variants of data-parallel SGD that quantize and encode gradients to lessen communication costs.
For the first variant, QSGD, they provide strong theoretical guarantees.
For the second variant, which we call QSGDinf, they demonstrate impressive empirical gains for distributed training of large neural networks.
Building on their work, we propose an alternative scheme for quantizing gradients and show that it yields stronger theoretical guarantees than exist for QSGD while matching the empirical performance of QSGDinf.

assistant: text: NUQSGD closes the gap between the theoretical guarantees of QSGD and the empirical performance of QSGDinf.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Accelerating SGD with momentum for over-parameterized learning

Paper content:
Nesterov SGD is widely used for training modern neural networks and other machine learning models.
Yet, its advantages over SGD have not been theoretically clarified.
Indeed, as we show  in this paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD.
Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD.
This is in contrast to the classical results in the deterministic setting, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent.
To address the non-acceleration issue, we  introduce a compensation term to Nesterov SGD.
The resulting  algorithm, which we call MaSS, converges  for same step sizes as SGD.
We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting.
For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method.
We also analyze the  practically important question of the dependence of the convergence rate and  optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation.
Experimental evaluation of MaSS for several standard  architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, Nesterov SGD  and Adam.

assistant: text: This work proves the non-acceleration of Nesterov SGD with any hyper-parameters, and proposes new algorithm which provably accelerates SGD in the over-parameterized setting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improve Training Stability of Semi-supervised Generative Adversarial Networks with Collaborative Training

Paper content:
Improved generative adversarial network (Improved GAN) is a successful method of using generative adversarial models to solve the problem of semi-supervised learning.
However, it suffers from the problem of unstable training.
In this paper, we found that the instability is mostly due to the vanishing gradients on the generator.
To remedy this issue, we propose a new method to use collaborative training to improve the stability of semi-supervised GAN with the combination of Wasserstein GAN.
The experiments have shown that our proposed method is more stable than the original Improved GAN and achieves comparable classification accuracy on different data sets.

assistant: text: Improve Training Stability of Semi-supervised Generative Adversarial Networks with Collaborative Training
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Progressive Memory Banks for Incremental Domain Adaptation

Paper content:
This paper addresses the problem of incremental domain adaptation (IDA).
We assume each domain comes sequentially, and that we could only access data in the current domain.
The goal of IDA is  to build a unified model performing well on all the encountered domains.
We propose to augment a recurrent neural network (RNN) with a directly parameterized memory bank, which is retrieved by an attention mechanism at each step of RNN transition.
The memory bank provides a natural way of IDA: when adapting our model to a new domain, we progressively add new slots to the memory bank, which increases the model capacity.
We learn the new memory slots and fine-tune existing parameters by back-propagation.
Experiments show that our approach significantly outperforms naive fine-tuning and previous work on IDA, including elastic weight consolidation and the progressive neural network.
Compared with expanding hidden states, our approach is more robust for old domains, shown by both empirical and theoretical results.

assistant: text: We present a neural memory-based architecture for incremental domain adaptation, and provide theoretical and empirical results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: iRDA Method for Sparse Convolutional Neural Networks

Paper content:
We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without reducing its accuracy.
The method has been tested for various data sets, and proven to be significantly more efficient than most existing compressing techniques in the deep learning literature.
For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy.
In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature.

assistant: text: A sparse optimization algorithm for deep CNN models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variance Regularized Counterfactual Risk Minimization via Variational Divergence Minimization

Paper content:
Off-policy learning, the task of evaluating and improving policies using historic data collected from a logging policy, is important because on-policy evaluation is usually expensive and has adverse impacts.
One of the major challenge of off-policy learning is to derive counterfactual estimators that also has low variance and thus low generalization error.
In this work, inspired by learning bounds for importance sampling problems, we present a new counterfactual learning principle for off-policy learning with bandit feedbacks.Our method regularizes the generalization error by minimizing the distribution divergence between the logging policy and the new policy, and removes the need for iterating through all training samples to compute sample variance regularization in prior work.
With neural network policies, our end-to-end training algorithms using variational divergence minimization showed significant improvement over conventional baseline algorithms and is also consistent with our theoretical results.

assistant: text: For off-policy learning with bandit feedbacks, we propose a new variance regularized counterfactual learning algorithm, which has both theoretical foundations and superior empirical performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design

Paper content:
Formal understanding of the inductive bias behind deep convolutional networks, i.e. the relation between the network's architectural features and the functions it is able to model, is limited.
In this work, we establish a fundamental connection between the fields of quantum physics and deep learning, and use it for obtaining novel theoretical observations regarding the inductive bias of convolutional networks.
Specifically, we show a structural equivalence between the function realized by a convolutional arithmetic circuit (ConvAC) and a quantum many-body wave function, which facilitates the use of quantum entanglement measures as quantifiers of a deep network's expressive ability to model correlations.
Furthermore, the construction of a deep ConvAC in terms of a quantum Tensor Network is enabled.
This allows us to perform a graph-theoretic analysis of a convolutional network, tying its expressiveness to a min-cut in its underlying graph.
We demonstrate a practical outcome in the form of a direct control over the inductive bias via the number of channels (width) of each layer.
We empirically validate our findings on standard convolutional networks which involve ReLU activations and max pooling.
The description of a deep convolutional network in well-defined graph-theoretic tools and the structural connection to quantum entanglement, are two interdisciplinary bridges that are brought forth by this work.

assistant: text: Employing quantum entanglement measures for quantifying correlations in deep learning, and using the connection to fit the deep network's architecture to correlations in the data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spike Sorting using the Neural Clustering Process

Paper content:
We present a novel approach to spike sorting for high-density multielectrode probes using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.
To optimally encode spike waveforms for clustering, we extended NCP by adding a convolutional spike encoder, which is learned end-to-end with the NCP network.
Trained purely on labeled synthetic spikes from a simple generative model, the NCP spike sorting model shows promising performance for clustering multi-channel spike waveforms.
The model provides higher clustering quality than an alternative Bayesian algorithm, finds more spike templates with clear receptive fields on real data and recovers more ground truth neurons on hybrid test data compared to a recent spike sorting algorithm.
Furthermore, NCP is able to handle the clustering uncertainty of ambiguous small spikes by GPU-parallelized posterior sampling.
The source code is publicly available.

assistant: text: We present a novel approach to spike sorting using the Neural Clustering Process (NCP), a recently introduced neural architecture that performs scalable amortized approximate Bayesian inference for efficient probabilistic clustering.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Large deep neural networks require huge memory to run and their running speed is sometimes too slow for real applications.
Therefore network size reduction with keeping accuracy is crucial for practical applications.
We present a novel neural network operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.
Chopout is easy to implement and integrate into most type of existing neural networks.
Furthermore it enables to reduce size of networks and latent representations even after training just by truncating layers.
We show its effectiveness through several experiments.
where where grad is a gradient and m is the number drawn in the forward pass.
DISPLAYFORM0
At test time, chopout is defined to behave as a identity function, that is, just pass through the input 45 vector without any modification.
This definition of chopout in prediction mode is contrastive to that 46 of dropout, which, in prediction time, dropout scale inputs to make it consistent with training time.
Training a fully-connected neural network with applying chopout can be interpreted as simultaneous 48 training of randomly sampled sub-networks which are obtained by cuttinng out former parts of the 49 original fully-connected neural network with sharing parameters.
In higher dimensional cases, chopout can be easily extended as a random truncation of channels 51 instead of dimensions.
For example, when applied to a tensor x ∈ R c×h×w , the forward-propagation 52 of chopout is defined as DISPLAYFORM0 where P ({0, 1, · · · , c}) is an arbitrary distribution.
Back-propagation is defined in the same way.
Throughout experiments, we use uniform distributions over {1, · · · , d} for P m ({0, 1, · · · , d}).
We train autoencoders on MNIST (LeCun et al. [1998] , Table 1 , FIG4 ).
We see that by applying 58 chopout on the hidden layer of the autoencoder, the reconstruction is kept well even after the hidden 59 layer is truncated.
We apply chopout for embeddings trained through skip-gram models (Mikolov et al. [2013a,b] ).
We 62 use text8 corpus 1 .
We set the window size to 5 and ignore infrequent words which appear less than 63 20 times in the corpus.
The result TAB2 shows the consistency of embeddings.
ran (512) ran (64) news (512) news (64) good (512)
(1) The distribution P m ({0, 1, · · · , d}) should be explored.
If we put chopouts in every layer of a 79 neural network, then, in training, there could be a layer where drawn m ∼ P m ({0, 1, · · · , d}) is very 80 small and it could be a bottleneck of the prediction accuracy.
(2) Chopout can be used for network pruning.

assistant: text: We present a novel simple operator, chopout, with which neural networks are trained, even in a single training process, so as to truncated sub-networks perform as well as possible.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Principled Weight Initialization for Hypernetworks

Paper content:
Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner.
Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date.
We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale.
We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

assistant: text: The first principled weight initialization method for hypernetworks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A fully automated periodicity detection in time series

Paper content:
This paper presents a method to autonomously find periodicities in a signal.
It is based on the same idea of using Fourier Transform and autocorrelation function presented in Vlachos et al. 2005.
While showing interesting results this method does not perform well on noisy signals or signals with multiple periodicities.
Thus, our method adds several new extra steps (hints clustering, filtering and detrending) to fix these issues.
Experimental results show that the proposed method outperforms the state of the art algorithms.

assistant: text: This paper presents a method to autonomously find multiple periodicities in a signal, using FFT and ACF and add three news steps (clustering/filtering/detrending)
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Attacks on Copyright Detection Systems

Paper content:
It is well-known that many machine learning models are susceptible to adversarial attacks, in which an attacker evades a classifier by making small perturbations to inputs.
This paper discusses how industrial copyright detection tools, which serve a central role on the web, are susceptible to adversarial attacks.
We discuss a range of copyright detection systems, and why they are particularly vulnerable to attacks.
These vulnerabilities are especially apparent for neural network based systems.
As proof of concept, we describe a well-known music identification method and implement this system in the form of a neural net.
We then attack this system using simple gradient methods.
Adversarial music created this way successfully fools industrial systems, including the AudioTag copyright detector and YouTube's Content ID system.
Our goal is to raise awareness of the threats posed by adversarial examples in this space and to highlight the importance of hardening copyright detection systems to attacks.

assistant: text: Adversarial examples can fool YouTube's copyright detection system
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dataset Distillation

Paper content:
Model distillation aims to distill the knowledge of a complex model into a simpler one.
In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one.
The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data.
For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization.
We evaluate our method in various initialization settings.
Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods.
Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images.

assistant: text: We propose to distill a large dataset into a small set of synthetic data that can train networks close to original performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

Paper content:
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions.
However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures.
Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data.
To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions.
We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.

assistant: text: We demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Customizing Sequence Generation with Multi-Task Dynamical Systems

Paper content:
Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application.
In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent  code z that specifies the customization to the
individual data sequence.
This enables style transfer, interpolation and morphing within generated sequences.
We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches.

assistant: text: Tailoring predictions from sequence models (such as LDSs and RNNs) via an explicit latent code.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scheduling the Learning Rate Via Hypergradients: New Insights and a New Algorithm

Paper content:
We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization.
This allows us to explicitly search for schedules that achieve good generalization.
We describe the structure of the gradient of a validation error w.r.t.
the learning rates, the hypergradient, and based on this we introduce a novel online algorithm.
Our method adaptively interpolates between two recently proposed techniques (Franceschi et al., 2017; Baydin et al.,2018), featuring increased stability and faster convergence.
We show empirically that the proposed technique compares favorably with baselines and related methodsin terms of final test accuracy.

assistant: text: MARTHE: a new method to fit task-specific learning rate schedules from the perspective of hyperparameter optimization
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Capsule Networks without Routing Procedures

Paper content:
We propose Pure CapsNets (P-CapsNets) without routing procedures.
Specifically, we make three modifications to CapsNets.
First, we remove routing procedures from CapsNets based on the observation that the coupling coefficients can be learned implicitly.
Second, we replace the convolutional layers in CapsNets to improve efficiency.
Third, we package the capsules into rank-3 tensors to further improve efficiency.
The experiment shows that P-CapsNets achieve better performance than CapsNets with varied routine procedures by using significantly fewer parameters on MNIST&CIFAR10.
The high efficiency of P-CapsNets is even comparable to some deep compressing models.
For example, we achieve more than 99% percent accuracy on MNIST by using only 3888 parameters.
We visualize the capsules as well as the corresponding correlation matrix to show a possible way of initializing CapsNets in the future.
We also explore the adversarial robustness of P-CapsNets compared to CNNs.

assistant: text: Routing procedures are not necessary for CapsNets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Paper content:
Exploration in sparse reward reinforcement learning remains an open challenge.
Many state-of-the-art methods use intrinsic motivation to complement the sparse extrinsic reward signal, giving the agent more opportunities to receive feedback during exploration.
Commonly these signals are added as bonus rewards, which results in a mixture policy that neither conducts exploration nor task fulfillment resolutely.
In this paper, we instead learn separate intrinsic and extrinsic task policies and schedule between these different drives to accelerate exploration and stabilize learning.
Moreover, we introduce a new type of intrinsic reward denoted as successor feature control (SFC), which is general and not task-specific.
It takes into account statistics over complete trajectories and thus differs from previous methods that only use local information to evaluate intrinsic motivation.
We evaluate our proposed scheduled intrinsic drive (SID) agent using three different environments with pure visual inputs: VizDoom, DeepMind Lab and DeepMind Control Suite.
The results show a substantially improved exploration efficiency with SFC and the hierarchical usage of the intrinsic drives.
A video of our experimental results can be found at https://gofile.io/?c=HpEwTd
.

assistant: text: A new intrinsic reward signal based on successor features and a novel way to combine extrinsic and intrinsic reward.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Domain Transfer When Predicting Intent in Text

Paper content:
In many domains, especially enterprise text analysis, there is an abundance of data which can be used for the development of new AI-powered intelligent experiences to improve people's productivity.
However, there are strong-guarantees of privacy which prevent broad sampling and  labeling of personal text data to learn or evaluate models of interest.
Fortunately, in some cases like enterprise email, manual annotation is possible on certain public datasets.
The hope is that models trained on these public datasets would perform well on the target private datasets of interest.
In this paper, we study the challenges of transferring information from one email dataset to another, for predicting user intent.
In particular, we present approaches to characterizing the transfer gap in text corpora from both an intrinsic and extrinsic point-of-view, and evaluate several proposed methods in the literature for bridging this gap.
We conclude with raising issues for further discussion in this arena.
Using publicly available text data to train predictive models for use in privacy-aware enterprise settings is a very fruitful direction in the area of document understanding.
However, when the labeled training dataset (source domain) is different from the unlabeled test (target domain), the two datasets likely follow different distributions.
This application setting violates the i.i.d. assumption made by classic supervised learning methods and calls for domain adaptation techniques to properly account for this difference.
State of the art domain adaptation techniques are generally developed and evaluated using a limited number of benchmark datasets and under constrained settings.
The extent to which these methods are applicable for predictive settings over enterprise text data has neither been explored nor characterized in detail.
To explore the effectiveness of state of the art domain adaptation methodology in enterprise text data, we focus on communication intent prediction in enterprise email.
In particular, we use two public enterprise email datasets (Avocado, an IT company, and Enron, an oil company) to systematically analyze the transfer problem in enterprise email.
The two intent prediction tasks that we focus on in this study are Meeting Intent (the email expresses an intent to meet with the recipient) and Commitment Intent (the email expresses an action the sender intends to take in the future) -both of which are binary classification tasks.

assistant: text: Insights on the domain adaptation challenge, when predicting user intent in enterprise email.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Autoencoding Documents for Topic Modeling with L-2 Sparsity Regularization

Paper content:
We propose a novel yet simple neural network architecture for topic modelling.
The method is based on training an autoencoder structure where the bottleneck represents the space of the topics distribution and the decoder outputs represent the space of the words distributions over the topics.
We exploit an auxiliary decoder to prevent mode collapsing in our model.
A key feature for an effective topic modelling method is having sparse topics and words distributions, where there is a trade-off between the sparsity level of topics and words.
This feature is implemented in our model by L-2 regularization and the model hyperparameters take care of the trade-off.
We show in our experiments that our model achieves competitive results compared to the state-of-the-art deep models for topic modelling, despite its simple architecture and training procedure.
The “New York Times” and “20 Newsgroups” datasets are used in the experiments.

assistant: text: A deep model for topic modelling
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks

Paper content:
We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks.
The local adaptation of activation function is achieved by introducing scalable hyper-parameters in each layer (layer-wise) and for every neuron separately (neuron-wise), and then optimizing it using the stochastic gradient descent algorithm.
Introduction of neuron-wise activation function acts like a vector activation function as opposed to the traditional scalar activation function given by fixed, global and layer-wise activations.
In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerate convergence, thereby reducing the training cost.
For numerical experiments, a nonlinear discontinuous function is approximated using a deep neural network with layer-wise and neuron-wise locally adaptive activation functions with and without the slope recovery term and compared with its global counterpart.
Moreover, solution of the nonlinear Burgers equation, which exhibits steep gradients, is also obtained using the proposed methods.
On the theoretical side, we prove that in the proposed method the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate.
Furthermore, the proposed adaptive activation functions with the slope recovery are shown to accelerate the training process in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets with and without data augmentation.

assistant: text: Proposing locally adaptive activation functions in deep and physics-informed neural networks for faster convergence
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks

Paper content:
New types of compute hardware in development and entering the market hold the promise of revolutionizing deep learning in a manner as profound as GPUs.
However, existing software frameworks and training algorithms for deep learning have yet to evolve to fully leverage the capability of the new wave of silicon.
In particular, models that exploit structured input via complex and instance-dependent control flow are difficult to accelerate using existing algorithms and hardware that typically rely on minibatching.
We present an asynchronous model-parallel (AMP) training algorithm that is specifically motivated by training on networks of interconnected devices.
Through an implementation on multi-core CPUs, we show that AMP training converges to the same accuracy as conventional synchronous training algorithms in a similar number of epochs, but utilizes the available hardware more efficiently, even for small minibatch sizes, resulting in shorter overall training times.
Our framework opens the door for scaling up a new class of deep learning models that cannot be efficiently trained today.

assistant: text: Using asynchronous gradient updates to accelerate dynamic neural network training