ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CoT: Cooperative Training for Generative Modeling of Discrete Data

Paper content:
We propose Cooperative Training (CoT) for training generative models that measure a tractable density for discrete data.
CoT coordinately trains a generator G and an auxiliary predictive mediator M. The training target of M is to estimate a mixture density of the learned distribution G and the target distribution P, and that of G is to minimize the Jensen-Shannon divergence estimated through M. CoT achieves independent success without the necessity of pre-training via Maximum Likelihood Estimation or involving high-variance algorithms like REINFORCE.
This low-variance algorithm is theoretically proved to be superior for both sample generation and likelihood prediction.
We also theoretically and empirically show the superiority of CoT over most previous algorithms in terms of generative quality and diversity, predictive generalization ability and computational cost.

assistant: text: We proposed Cooperative Training, a novel training algorithm for generative modeling of discrete data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exploring the Design of Patient-Generated Data Visualizations

Paper content:
We were approached by a group of healthcare providers who are involved in the care of chronic patients looking for potential technologies to facilitate the process of reviewing patient-generated data during clinical visits.
Aiming at understanding the healthcare providers' attitudes towards reviewing patient-generated data, we (1) conducted a focus group with a mixed group of healthcare providers.
Next, to gain the patients' perspectives, we (2) interviewed eight chronic patients, collected a sample of their data and designed a series of visualizations representing patient data we collected.
Last, we (3) sought feedback on the visualization designs from healthcare providers who requested this exploration.
We found four factors shaping patient-generated data: data & context, patient's motivation, patient's time commitment, and patient's support circle.
Informed by the results of our studies, we discussed the importance of designing patient-generated visualizations for individuals by considering both patient and healthcare provider rather than designing with the purpose of generalization and provided guidelines for designing future patient-generated data visualizations.

assistant: text: We explored the visualization designs that can support chronic patients to present and review their health data with healthcare providers during clinical visits.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sparsity Emerges Naturally in Neural Language Models

Paper content:
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse  neural  models for NLP tasks.
If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse?
Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients.
We find that  gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

assistant: text: We study the natural emergence of sparsity in the activations and gradients for some layers of a dense LSTM language model, over the course of training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Poisoning Attacks with Generative Adversarial Nets

Paper content:
Machine learning algorithms are vulnerable to poisoning attacks: An adversary can inject malicious points in the training dataset to influence the learning process and degrade the algorithm's performance.
Optimal poisoning attacks have already been proposed to evaluate worst-case scenarios, modelling attacks as a bi-level optimization problem.
Solving these problems is computationally demanding and has limited applicability for some models such as deep networks.
In this paper we introduce a novel generative model to craft systematic poisoning attacks against machine learning classifiers generating adversarial training examples, i.e. samples that look like genuine data points but that degrade the classifier's accuracy when used for training.
We propose a Generative Adversarial Net with three components: generator, discriminator, and the target classifier.
This approach allows us to model naturally the detectability constrains that can be expected in realistic attacks and to identify the regions of the underlying data distribution that can be more vulnerable to data poisoning.
Our experimental evaluation shows the effectiveness of our attack to compromise machine learning classifiers, including deep networks.

assistant: text: In this paper we propose a novel generative model to craft systematic poisoning attacks with detectability constraints against machine learning classifiers, including deep networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Inference Suboptimality in Variational Autoencoders

Paper content:
Amortized inference has led to efficient approximate inference for large datasets.
The quality of posterior inference is largely determined by two factors:
a) the ability of the variational distribution to model the true posterior and
b) the capacity of the recognition network to generalize inference over all datapoints.
We analyze approximate inference in variational autoencoders in terms of these factors.
We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution.
We show that this is due partly to the generator learning to accommodate the choice of approximation.
Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

assistant: text: We decompose the gap between the marginal log-likelihood and the evidence lower bound and study the effect of the approximate posterior on the true posterior distribution in VAEs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Kolmogorov Complexity Approach to Generalization in Deep Learning

Paper content:
Deep artificial neural networks can achieve an extremely small difference between training and test accuracies on identically distributed training and test sets, which is a standard measure of generalization.
However, the training and test sets may not be sufficiently representative of the empirical sample set, which consists of real-world input samples.
When samples are drawn from an underrepresented or unrepresented subset during inference, the gap between the training and inference accuracies can be significant.
To address this problem, we first reformulate a classification algorithm as a procedure for searching for a source code that maps input features to classes.
We then derive a necessary and sufficient condition for generalization using a universal cognitive similarity metric, namely information distance, based on Kolmogorov complexity.
Using this condition, we formulate an optimization problem to learn a more general classification function.
To achieve this end, we extend the input features by concatenating encodings of them, and then train the classifier on the extended features.
As an illustration of this idea, we focus on image classification, where we use channel codes on the input features as a systematic way to improve the degree to which the training and test sets are representative of the empirical sample set.
To showcase our theoretical findings, considering that corrupted or perturbed input features belong to the empirical sample set, but typically not to the training and test sets, we demonstrate through extensive systematic experiments that, as a result of learning a more general classification function, a model trained on encoded input features is significantly more robust to common corruptions, e.g., Gaussian and shot noise, as well as adversarial perturbations, e.g., those found via projected gradient descent, than the model trained on uncoded input features.

assistant: text: We present a theoretical and experimental framework for defining, understanding, and achieving generalization, and as a result robustness, in deep learning by drawing on algorithmic information theory and coding theory.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Clean-Label Backdoor Attacks

Paper content:
Deep neural networks have been recently demonstrated to be vulnerable to backdoor attacks.
Specifically, by altering a small set of training examples, an adversary is able to install a backdoor that can be used during inference to fully control the model’s behavior.
While the attack is very powerful, it crucially relies on the adversary being able to introduce arbitrary, often clearly mislabeled, inputs to the training set and can thus be detected even by fairly rudimentary data filtering.
In this paper, we introduce a new approach to executing backdoor attacks, utilizing adversarial examples and GAN-generated data.
The key feature is that the resulting poisoned inputs appear to be consistent with their label and thus seem benign even upon human inspection.

assistant: text: We show how to successfully perform backdoor attacks without changing training labels.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Recurrent Binary/Ternary Weights

Paper content:
Recurrent neural networks (RNNs) have shown excellent performance in processing sequence data.
However, they are both complex and memory intensive due to their recursive nature.
These limitations make RNNs difficult to embed on mobile devices requiring real-time processes with limited hardware resources.
To address the above issues, we introduce a method that can learn binary and ternary weights during the training phase to facilitate hardware implementations of RNNs.
As a result, using this approach replaces all multiply-accumulate operations by simple accumulations, bringing significant benefits to custom hardware in terms of silicon area and power consumption.
On the software side, we evaluate the performance (in terms of accuracy) of our method using long short-term memories (LSTMs) and gated recurrent units (GRUs) on various sequential models including sequence classification and language modeling.
We demonstrate that our method achieves competitive results on the aforementioned tasks while using binary/ternary weights during the runtime.
On the hardware side, we present custom hardware for accelerating the recurrent computations of LSTMs with binary/ternary weights.
Ultimately, we show that LSTMs with binary/ternary weights can achieve up to 12x memory saving and 10x inference speedup compared to the full-precision hardware implementation design.

assistant: text: We propose high-performance LSTMs with binary/ternary weights, that can greatly reduce implementation complexity
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Stochastic Derivative Free Optimization Method with Momentum

Paper content:
We consider the problem of unconstrained minimization of a smooth objective
function in $\mathbb{R}^d$ in setting where only function evaluations are possible.
We propose and analyze stochastic zeroth-order method with heavy ball momentum.
In particular, we propose, SMTP, a momentum version of the stochastic three-point method (STP) Bergou et al. (2019).
We show new complexity results for non-convex, convex and strongly convex functions.
We test our method on a collection of learning to continuous control tasks on several MuJoCo Todorov et al. (2012) environments with varying difficulty and compare against STP, other state-of-the-art derivative-free optimization algorithms and against policy gradient methods.
SMTP significantly outperforms STP and all other methods that we considered in our numerical experiments.
Our second contribution is SMTP with importance sampling which we call SMTP_IS.
We provide convergence analysis of this method for non-convex, convex and strongly convex objectives.

assistant: text: We develop and analyze a new derivative free optimization algorithm with momentum and importance sampling with applications to continuous control.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Defense Via Data Dependent Activation Function and  Total Variation Minimization

Paper content:
We improve the robustness of deep neural nets  to adversarial attacks by using an interpolating function as the output activation.
This data-dependent activation function remarkably improves both classification accuracy and stability to adversarial perturbations.
Together with the total variation minimization of adversarial images and augmented training, under the strongest attack, we achieve up to 20.6%, 50.7%, and 68.7% accuracy improvement w.r.t.
the fast gradient sign method, iterative fast gradient sign method, and Carlini-WagnerL2attacks, respectively.
Our defense strategy is additive to many of the existing methods.
We give an intuitive explanation of our defense strategy via analyzing the geometry of the feature space.
For reproducibility, the code will be available on GitHub.

assistant: text: We proposal strategies for adversarial defense based on data dependent activation function, total variation minimization, and training data augmentation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stabilizing Transformers for Reinforcement Learning

Paper content:
Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation.
Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially-observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting.
In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives.
We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant.
The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture.
We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical.
GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially-observable environments.

assistant: text: We succeed in stabilizing transformers for training in the RL setting and demonstrate a large improvement over LSTMs on DMLab-30, matching an external memory architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Program Planner for Structured Predictions

Paper content:
We consider the problem of weakly supervised structured prediction (SP) with reinforcement learning (RL) – for example, given a database table and a question, perform a sequence of computation actions on the table, which generates a response and  receives  a  binary  success-failure  reward.
This  line  of  research  has  been successful by leveraging RL to directly optimizes the desired metrics of the SP tasks – for example, the accuracy in question answering or BLEU score in machine translation.
However, different from the common RL settings, the environment dynamics is deterministic in SP, which hasn’t been fully utilized by the model-freeRL methods that are usually applied.
Since SP models usually have full access to the environment dynamics, we propose to apply model-based RL methods, which rely on planning as a primary model component.
We demonstrate the effectiveness of planning-based SP with a Neural Program Planner (NPP), which, given a set of candidate programs from a pretrained search policy, decides which program is the most promising considering all the information generated from executing these programs.
We evaluate NPP on weakly supervised program synthesis from natural language(semantic parsing) by stacked learning a planning module based on pretrained search policies.
On the WIKITABLEQUESTIONS benchmark, NPP achieves a new state-of-the-art of 47.2% accuracy.

assistant: text: A model-based planning component improves RL-based semantic parsing on WikiTableQuestions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: HYPE:  Human-eYe Perceptual Evaluation of Generative Models

Paper content:
Generative models often use human evaluations to determine and justify progress.
Unfortunately, existing human evaluation methods are ad-hoc: there is currently no standardized, validated evaluation that: (1) measures perceptual fidelity, (2) is reliable, (3) separates models into clear rank order, and (4) ensures high-quality measurement without intractable cost.
In response, we construct Human-eYe Perceptual Evaluation (HYPE), a human metric that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) results in separable model performances, and (4) efficient in cost and time.
We introduce two methods.
The first, HYPE-Time, measures visual perception under adaptive time constraints to determine the minimum length of time (e.g., 250ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake.
The second, HYPE-Infinity, measures human error rate on fake and real images with no time constraints, maintaining stability and drastically reducing time and cost.
We test HYPE across four state-of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebA and the newer higher-resolution FFHQ, and two sampling techniques of model outputs.
By simulating HYPE's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0%) on FFHQ.

assistant: text: HYPE is a reliable human evaluation metric for scoring generative models, starting with human face generation across 4 GANs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Metagross: Meta Gated Recursive Controller Units for Sequence Modeling

Paper content:
This paper proposes Metagross (Meta Gated Recursive Controller), a new neural sequence modeling unit.
Our proposed unit is characterized by recursive parameterization of its gating functions, i.e., gating mechanisms of Metagross are controlled by instances of itself, which are repeatedly called in a recursive fashion.
This can be interpreted as a form of meta-gating and recursively parameterizing a recurrent model.
We postulate that our proposed inductive bias provides modeling benefits pertaining to learning with inherently hierarchically-structured sequence data (e.g., language, logical or music tasks).
To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), sequential pixel-by-pixel classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving state-of-the-art (or close) performance on all tasks.

assistant: text: Recursive Parameterization of Recurrent Models improve performance
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning in Confusion: Batch Active Learning with Noisy Oracle

Paper content:
We study the problem of training machine learning models incrementally using active learning with access to imperfect or noisy oracles.
We specifically consider the setting of batch active learning, in which multiple samples are selected as opposed to a single sample as in classical settings so as to reduce the training overhead.
Our approach bridges between uniform randomness and score based importance sampling of clusters when selecting a batch of new samples.
Experiments on
benchmark image classification datasets (MNIST, SVHN, and CIFAR10) shows improvement over existing active learning strategies.
We introduce an extra denoising layer to deep networks to make active learning robust to label noises and show significant improvements.

assistant: text: We address the active learning in batch setting with noisy oracles and use model uncertainty to encode the decision quality of active learning algorithm during acquisition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enabling Continual Learning in Neural Networks with Meta Learning

Paper content:
Catastrophic forgetting in neural networks is one of the most well-known problems in continual learning.
Previous attempts on addressing the problem focus on preventing important weights from changing.
Such methods often require task boundaries to learn effectively and do not support backward transfer learning.
In this paper, we propose a meta-learning algorithm which learns to reconstruct the gradients of old tasks w.r.t. the current parameters and combines these reconstructed gradients with the current gradient to enable continual learning and backward transfer learning from the current task to previous tasks.
Experiments on standard continual learning benchmarks show that our algorithm can effectively prevent catastrophic forgetting and supports backward transfer learning.

assistant: text: We propose a meta learning algorithm for continual learning which can effectively prevent catastrophic forgetting problem and support backward transfer learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Epitome for Unravelling Generalized Hamming Network: A Fuzzy Logic Interpretation of Deep Learning

Paper content:
This paper gives a rigorous analysis of trained Generalized Hamming Networks (GHN) proposed by Fan (2017) and discloses an interesting finding about GHNs, i.e. stacked convolution layers in a GHN is equivalent to a single yet wide convolution layer.
The revealed equivalence, on the theoretical side, can be regarded as a constructive manifestation of the universal approximation theorem Cybenko (1989); Hornik (1991).
In practice, it has profound and multi-fold implications.
For network visualization, the constructed deep epitomes at each layer provide a visualization of network internal representation that does not rely on the input data.
Moreover, deep epitomes allows the direct extraction of features in just one step, without resorting to regularized optimizations used in existing visualization tools.

assistant: text: bridge the gap in soft computing
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning with Social Influence through  Interior Policy Differentiation

Paper content:
Animals develop novel skills not only through the interaction with the environment but also from the influence of the others.
In this work we model the social influence into the scheme of reinforcement learning, enabling the agents to learn both from the environment and from their peers.
Specifically, we first define a metric to measure the distance between policies then quantitatively derive the definition of uniqueness.
Unlike previous precarious joint optimization approaches, the social uniqueness motivation in our work is imposed as a constraint to encourage the agent to learn a policy different from the existing agents while still solve the primal task.
The resulting algorithm, namely Interior Policy Differentiation (IPD), brings about performance improvement as well as a collection of policies that solve a given task with distinct behaviors

assistant: text: A new RL algorithm called Interior Policy Differentiation is proposed to learn a collection of diverse policies for a given primal task.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Challenges in Computing and Optimizing Upper Bounds of Marginal Likelihood based on Chi-Square Divergences

Paper content:
Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood.
However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood.
This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks.
We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound.
We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies.

assistant: text: An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Quaternion Equivariant Capsule Networks for 3D Point Clouds

Paper content:
We present a 3D capsule architecture for processing of point clouds that is equivariant with respect to the SO(3) rotation group, translation and permutation of the unordered input sets.
The network operates on a sparse set of local reference frames, computed from an input point cloud and establishes end-to-end equivariance through a novel 3D quaternion group capsule layer, including an equivariant dynamic routing procedure.
The capsule layer enables us to disentangle geometry from pose, paving the way for more informative descriptions and a structured latent space.
In the process, we theoretically connect the process of dynamic routing between capsules to the well-known Weiszfeld algorithm, a scheme for solving iterative re-weighted least squares (IRLS) problems with provable convergence properties, enabling robust pose estimation between capsule layers.
Due to the sparse equivariant quaternion capsules, our architecture allows joint object classification and orientation estimation, which we validate empirically on common benchmark datasets.

assistant: text: Deep architectures for 3D point clouds that are equivariant to SO(3) rotations, as well as translations and permutations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Lifelong Learning with Dynamically Expandable Networks

Paper content:
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks.
DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them.
We validate DEN on multiple public datasets in lifelong learning scenarios on multiple public datasets, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch model with substantially fewer number of parameters.

assistant: text: We propose a novel deep network architecture that can dynamically decide its network capacity as it trains on a lifelong learning scenario.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Noisy Collaboration in Knowledge Distillation

Paper content:
Knowledge distillation is an effective model compression technique in which a smaller model is trained to mimic a larger pretrained model.
However in order to make these compact models suitable for real world deployment, not only do
we need to reduce the performance gap but also we need to make them more robust to commonly occurring and adversarial perturbations.
Noise permeates every level of the nervous system, from the perception of sensory signals to the
generation of motor responses.
We therefore believe that noise could be a crucial element in improving neural networks training and addressing the apparently contradictory goals of improving both the generalization and robustness of the
model.
Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise at either the input level or the supervision signals.
Our results show that noise can improve both the generalization and robustness of the model.
”Fickle Teacher” which uses dropout in teacher model as a source of response variation leads to significant generalization improvement.
”Soft Randomization”, which matches the output distribution of
the student model on the image with Gaussian noise to the output of the teacher on original image, improves the adversarial robustness manifolds compared to the student model trained with Gaussian noise.
We further show the surprising effect of random label corruption on a model’s adversarial robustness.
The study highlights the benefits of adding constructive noise in the knowledge distillation framework and hopes to inspire further work in the area.

assistant: text: Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: UNSUPERVISED METRIC LEARNING VIA NONLINEAR FEATURE SPACE TRANSFORMATIONS

Paper content:
In this paper, we propose a nonlinear unsupervised metric learning framework to boost of the performance of clustering algorithms.
Under our framework, nonlinear distance metric learning and manifold embedding are integrated and conducted simultaneously to increase the natural separations among data samples.
The metric learning component is implemented through feature space transformations, regulated by a nonlinear deformable model called Coherent Point Drifting (CPD).
Driven by CPD, data points can get to a higher level of linear separability, which is subsequently picked up by the manifold embedding component to generate well-separable sample projections for clustering.
Experimental results on synthetic and benchmark datasets show the effectiveness of our proposed approach over the state-of-the-art solutions in unsupervised metric learning.

assistant: text: a nonlinear unsupervised metric learning framework to boost the performance of clustering algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Zero-shot Cross-lingual Transfer of Multilingual Neural Machine Translation

Paper content:
Transferring representations from large-scale supervised tasks to downstream tasks have shown outstanding results in Machine Learning in both Computer Vision and natural language processing (NLP).
One particular example can be sequence-to-sequence models for Machine Translation (Neural Machine Translation - NMT).
It is because, once trained in a multilingual setup, NMT systems can translate between multiple languages and are also capable of performing zero-shot translation between unseen source-target pairs at test time.
In this paper, we first investigate if we can extend the zero-shot transfer capability of multilingual NMT systems to cross-lingual NLP tasks (tasks other than MT, e.g. sentiment classification and natural language inference).
We demonstrate a simple framework by reusing the encoder from a multilingual NMT system, a multilingual Encoder-Classifier, achieves remarkable zero-shot cross-lingual classification performance, almost out-of-the-box on three downstream benchmark tasks - Amazon Reviews, Stanford sentiment treebank (SST) and Stanford natural language inference (SNLI).
In order to understand the underlying factors contributing to this finding, we conducted a series of analyses on the effect of the shared vocabulary, the training data type for NMT models, classifier complexity, encoder representation power, and model generalization on zero-shot performance.
Our results provide strong evidence that the representations learned from multilingual NMT systems are widely applicable across languages and tasks, and the high, out-of-the-box classification performance is correlated with the generalization capability of such systems.

assistant: text: Zero-shot cross-lingual transfer by using multilingual neural machine translation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

Paper content:
Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios.
A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time.
In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption.
Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse.
Our approach takes two stages: first to adopt an end-to-end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned.
Our approach is mathematically appealing from an optimization perspective and easy to reproduce.
We experimented our approach through several image learning benchmarks and demonstrate its interest- ing aspects and competitive performance.

assistant: text: A CNN model pruning method using ISTA and rescaling trick to enforce sparsity of scaling parameters in batch normalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradient Estimators for Implicit Models

Paper content:
Implicit models, which allow for the generation of samples but not for point-wise evaluation of probabilities, are omnipresent in real-world problems tackled by machine learning and a hot topic of current research.
Some examples include data simulators that are widely used in engineering and scientific research, generative adversarial networks (GANs) for image synthesis, and hot-off-the-press approximate inference techniques relying on implicit distributions.
The majority of existing approaches to learning implicit models rely on approximating the intractable distribution or optimisation objective for gradient-based optimisation, which is liable to produce inaccurate updates and thus poor models.
This paper alleviates the need for such approximations by proposing the \emph{Stein gradient estimator}, which directly estimates the score function of the implicitly defined distribution.
The efficacy of the proposed estimator is empirically demonstrated by examples that include meta-learning for approximate inference and entropy regularised GANs that provide improved sample diversity.

assistant: text: We introduced a novel gradient estimator using Stein's method, and compared with other methods on learning implicit models for approximate inference and image generation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Non-reversibly updating a uniform [0,1] value for accept/reject decisions

Paper content:
I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform [0,1] value, and to then update this uniform value non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration.
This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions.
It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories.
This is of significance when some variables are updated by other methods, since if HMC is used, these updates can be done only between trajectories, whereas they can be done more often with Langevin updates.
This is seen for a Bayesian neural network model, in which connection weights are updated by persistent Langevin or HMC, while hyperparameters are updated by Gibbs sampling.

assistant: text: A non-reversible way of making accept/reject decisions can be beneficial
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exponential Family Word Embeddings: An Iterative Approach for Learning Word Vectors

Paper content:
GloVe and Skip-gram word embedding methods learn word vectors by decomposing a denoised matrix of word co-occurrences into a product of low-rank matrices.
In this work, we propose an iterative algorithm for computing word vectors based on modeling word co-occurrence matrices with Generalized Low Rank Models.
Our algorithm generalizes both Skip-gram and GloVe as well as giving rise to other embedding methods based on the specified co-occurrence matrix, distribution of co-occurences, and the number of iterations in the iterative algorithm.
For example, using a Tweedie distribution with one iteration results in GloVe and using a Multinomial distribution with full-convergence mode results in Skip-gram.
Experimental results demonstrate that multiple iterations of our algorithm improves results over the GloVe method on the Google word analogy similarity task.

assistant: text: We present a novel iterative algorithm based on generalized low rank models for computing and interpreting word embedding models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Domain Transfer When Predicting Intent in Text

Paper content:
In many domains, especially enterprise text analysis, there is an abundance of data which can be used for the development of new AI-powered intelligent experiences to improve people's productivity.
However, there are strong-guarantees of privacy which prevent broad sampling and  labeling of personal text data to learn or evaluate models of interest.
Fortunately, in some cases like enterprise email, manual annotation is possible on certain public datasets.
The hope is that models trained on these public datasets would perform well on the target private datasets of interest.
In this paper, we study the challenges of transferring information from one email dataset to another, for predicting user intent.
In particular, we present approaches to characterizing the transfer gap in text corpora from both an intrinsic and extrinsic point-of-view, and evaluate several proposed methods in the literature for bridging this gap.
We conclude with raising issues for further discussion in this arena.

assistant: text: Insights on the domain adaptation challenge, when predicting user intent in enterprise email.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergent Communication in Networked Multi-Agent Reinforcement Learning

Paper content:
With the ever increasing demand and the resultant reduced quality of services, the focus has shifted towards easing network congestion to enable more efficient flow in systems like traffic, supply chains and electrical grids.
A step in this direction is to re-imagine the traditional heuristics based training of systems as this approach is incapable of modelling the involved dynamics.
While one can apply Multi-Agent Reinforcement Learning (MARL) to such problems by considering each vertex in the network as an agent, most MARL-based models assume the agents to be independent.
In many real-world tasks, agents need to behave as a group, rather than as a collection of individuals.
In this paper, we propose a framework that induces cooperation and coordination amongst agents, connected via an underlying network, using emergent communication in a MARL-based setup.
We formulate the problem in a general network setting and demonstrate the utility of communication in networks with the help of a case study on traffic systems.
Furthermore, we study the emergent communication protocol and show the formation of distinct communities with grounded vocabulary.
To the best of our knowledge, this is the only work that studies emergent language in a networked MARL setting.

assistant: text: A framework for studying emergent communication in a networked multi-agent reinforcement learning setup.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fast and Accurate Text Classification: Skimming, Rereading and Early Stopping

Paper content:
Recent advances in recurrent neural nets (RNNs) have shown much promise in many applications in natural language processing.
For most of these tasks, such as sentiment analysis of customer reviews, a recurrent neural net model parses the entire review before forming a decision.
We argue that reading the entire input is not always necessary in practice, since a lot of reviews are often easy to classify, i.e., a decision can be formed after reading some crucial sentences or words in the provided text.
In this paper, we present an approach of fast reading for text classification.
Inspired by several well-known human reading techniques, our approach implements an intelligent recurrent agent which evaluates the importance of the current snippet in order to decide whether to make a prediction, or to skip some texts, or to re-read part of the sentence.
Our agent uses an RNN module to encode information from the past and the current tokens, and applies a policy module to form decisions.
With an end-to-end training algorithm based on policy gradient, we train and test our agent on several text classification datasets and achieve both higher efficiency and better accuracy compared to previous approaches.

assistant: text: We develop an end-to-end trainable approach for skimming, rereading and early stopping applicable to classification tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Are Few-shot Learning Benchmarks Too Simple ?

Paper content:
We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes, which defeats their intended purpose of evaluating few-shot classification methods.
The class semantics of Omniglot is invariably “characters” and the class semantics of miniImageNet, “object category”.
Because the class semantics are so similar, we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at metaevaluation time.
Our results suggest that those benchmarks are not adapted for supervised few-shot classification since the supervision itself is not necessary during meta-evaluation.
The Meta-Dataset, a collection of 10 datasets, was recently proposed as a harder few-shot classification benchmark.
Using our method, we derive a new metric, the Class Semantics Consistency Criterion, and use it to quantify the difficulty of Meta-Dataset.
Finally, under some restrictive assumptions, we show that Centroid Networks is faster and more accurate than a state-of-the-art learning-to-cluster method (Hsu et al., 2018).

assistant: text: Omniglot and miniImageNet are too simple for few-shot learning because we can solve them without using labels during meta-evaluation, as demonstrated with a method called centroid networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Differential Privacy in Adversarial Learning with Provable Robustness

Paper content:
In this paper, we aim to develop a novel mechanism to preserve differential privacy (DP) in adversarial learning for deep neural networks, with provable robustness to adversarial examples.
We leverage the sequential composition theory in DP, to establish a new connection between DP preservation and provable robustness.
To address the trade-off among model utility, privacy loss, and robustness, we design an original, differentially private, adversarial objective function, based on the post-processing property in DP, to tighten the sensitivity of our model.
An end-to-end theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of DP deep neural networks.

assistant: text: Preserving Differential Privacy in Adversarial Learning with Provable Robustness to Adversarial Examples
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Computation of Quantized Neural Networks by {−1, +1} Encoding Decomposition

Paper content:
Deep neural networks require extensive computing resources, and can not be efficiently applied to embedded devices such as mobile phones, which seriously limits their applicability.
To address this problem, we propose a novel encoding scheme by using {-1,+1} to decompose quantized neural networks (QNNs) into multi-branch binary networks, which can be efficiently implemented by bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving.
Our method can achieve at most ~59 speedup and ~32 memory saving over its full-precision counterparts.
Therefore, users can easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources.
Our mechanism is very suitable for the use of FPGA and ASIC in terms of data storage and computation, which provides a feasible idea for smart chips.
We validate the effectiveness of our method on both large-scale image classification (e.g., ImageNet) and object detection tasks.

assistant: text: A novel encoding scheme of using {-1, +1} to decompose QNNs into multi-branch binary networks, in which we used bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving.