ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DBA: Distributed Backdoor Attacks against Federated Learning

Paper content:
Backdoor attacks aim to manipulate a subset of training data by injecting adversarial triggers such that machine learning models trained on the tampered dataset will make arbitrarily (targeted) incorrect prediction on the testset with the same trigger embedded.
While federated learning (FL) is capable of aggregating information provided by different parties for training a better model, its distributed learning methodology and inherently heterogeneous data distribution across parties may bring new vulnerabilities.
In addition to recent centralized backdoor attacks on FL where each party embeds the same global trigger during training, we propose the distributed backdoor attack (DBA) --- a novel threat assessment framework developed by fully exploiting the distributed nature of FL.
DBA decomposes a global trigger pattern into separate local patterns and embed them into the training set of different adversarial parties respectively.
Compared to standard centralized backdoors, we show that DBA is substantially more persistent and stealthy against FL on diverse datasets such as finance and image data.
We conduct extensive experiments to show that the attack success rate of DBA is significantly higher than centralized backdoors under different settings.
Moreover, we find that distributed attacks are indeed more insidious, as DBA can evade two state-of-the-art robust FL algorithms against centralized backdoors.
We also provide explanations for the effectiveness of DBA via feature visual interpretation and feature importance ranking.
To further explore the properties of DBA, we test the attack performance by varying different trigger factors, including local trigger variations (size, gap, and location), scaling factor in FL, data distribution, and poison ratio and interval.
Our proposed DBA and thorough evaluation results shed lights on characterizing the robustness of FL.

assistant: text: We proposed a novel distributed backdoor attack on federated learning and show that it is not only more effective compared with standard centralized attacks, but also harder to be defended by existing robust FL methods
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variational Autoencoders for Text Modeling without Weakening the Decoder

Paper content:
Previous work (Bowman et al., 2015; Yang et al., 2017) has found difficulty developing generative models based on variational autoencoders (VAEs) for text.
To address the problem of the decoder ignoring information from the encoder (posterior collapse), these previous models weaken the capacity of the decoder to force the model to use information from latent variables.
However, this strategy is not ideal as it degrades the quality of generated text and increases hyper-parameters.
In this paper, we propose a new VAE for text utilizing a multimodal prior distribution, a modified encoder, and multi-task learning.
We show our model can generate well-conditioned sentences without weakening the capacity of the decoder.
Also, the multimodal prior distribution improves the interpretability of acquired representations.

assistant: text: We propose a model of variational autoencoders for text modeling without weakening the decoder, which improves the quality of text generation and interpretability of acquired representations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Benefits of Depth for Long-Term Memory of Recurrent Networks

Paper content:
The key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ever-improving ability to model intricate long-term temporal dependencies.
However, a well established measure of RNNs' long-term memory capacity is lacking, and thus formal understanding of their ability to correlate data throughout time is limited.
Though depth efficiency in convolutional networks is well established by now, it does not suffice in order to account for the success of deep RNNs on inputs of varying lengths, and the need to address their 'time-series expressive power' arises.
In this paper, we analyze the effect of depth on the ability of recurrent networks to express correlations ranging over long time-scales.
To meet the above need, we introduce a measure of the information flow across time that can be supported by the network, referred to as the Start-End separation rank.
Essentially, this measure reflects the distance of the function realized by the recurrent network from a function that models no interaction whatsoever between the beginning and end of the input sequence.
We prove that deep recurrent networks support Start-End separation ranks which are exponentially higher than those supported by their shallow counterparts.
Moreover, we show that the ability of deep recurrent networks to correlate different parts of the input sequence increases exponentially as the input sequence extends, while that of vanilla shallow recurrent networks does not adapt to the sequence length at all.
Thus, we establish that depth brings forth an overwhelming advantage in the ability of recurrent networks to model long-term dependencies, and provide an exemplar of quantifying this key attribute which may be readily extended to other RNN architectures of interest, e.g. variants of LSTM networks.
We obtain our results by considering a class of recurrent networks referred to as Recurrent Arithmetic Circuits (RACs), which merge the hidden state with the input via the Multiplicative Integration operation.

assistant: text: We propose a measure of long-term memory and prove that deep recurrent networks are much better fit to model long-term temporal dependencies than shallow ones.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Understanding Knowledge Distillation in Non-autoregressive Machine Translation

Paper content:
Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models.
Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance.
Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear.
In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training.
We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data.
Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality.
Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models.
We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

assistant: text: We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Network Regression with Beta, Dirichlet, and Dirichlet-Multinomial Outputs

Paper content:
We propose a method for quantifying uncertainty in neural network regression models when the targets are real values on a $d$-dimensional simplex, such as probabilities.
We show that each target can be modeled as a sample from a Dirichlet distribution, where the parameters of the Dirichlet are provided by the output of a neural network, and that the combined model can be trained using the gradient of the data likelihood.
This approach provides interpretable predictions in the form of multidimensional distributions, rather than point estimates, from which one can obtain confidence intervals or quantify risk in decision making.
Furthermore, we show that the same approach can be used to model targets in the form of empirical counts as samples from the Dirichlet-multinomial compound distribution.
In experiments, we verify that our approach provides these benefits without harming the performance of the point estimate predictions on two diverse applications: (1) distilling deep convolutional networks trained on CIFAR-100, and (2) predicting the location of particle collisions in the XENON1T Dark Matter detector.

assistant: text: Neural network regression should use Dirichlet output distribution when targets are probabilities in order to quantify uncertainty of predictions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

Paper content:
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained comput- ing systems.
Many network complexity reduction techniques have been proposed including fixed-point implementation.
However, a systematic approach for design- ing full fixed-point training and inference of deep neural networks remains elusive.
We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision.
The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori.
Thus, our work leads to a systematic methodology of determining suit- able precision for fixed-point training.
The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets.
The complexity reduction arising from our approach is compared with other fixed-point neural network designs.

assistant: text: We analyze and determine the precision requirements for training neural networks when all tensors, including back-propagated signals and weight accumulators, are quantized to fixed-point format.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Robust Representations via Multi-View Information Bottleneck

Paper content:
The information bottleneck method provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation.
The original formulation, however, requires labeled data in order to identify which information is superfluous.
In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label is unknown.
This enables us to identify superfluous information as that which is not shared by both views.
A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset.
We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.

assistant: text: We extend the information bottleneck method to the unsupervised multiview setting and show state of the art results on standard datasets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compressed Sensing and Overparametrized Networks: Overfitting Peaks in a Model of Misparametrized Sparse Regression in the Interpolation Limit

Paper content:
Current practice in machine learning is to employ deep nets in an overparametrized limit, with the nominal number of parameters typically exceeding the number of measurements.
This resembles the situation in compressed sensing, or in sparse regression with $l_1$ penalty terms, and provides a theoretical avenue for understanding phenomena that arise in the context of deep nets.
One such phenonemon is the success of deep nets in providing good generalization in an interpolating regime with zero training error.
Traditional statistical practice calls for regularization or smoothing to prevent "overfitting" (poor generalization performance).
However, recent work shows that there exist data interpolation procedures which are statistically consistent and provide good generalization performance\cite{belkin2018overfitting} ("perfect fitting").
In this context, it has been suggested that "classical" and "modern" regimes for machine learning are separated by a peak in the generalization error ("risk") curve, a phenomenon dubbed "double descent"\cite{belkin2019reconciling}.
While such overfitting peaks do exist and arise from ill-conditioned design matrices, here we challenge the interpretation of the overfitting peak as demarcating the regime where good generalization occurs under overparametrization.
We propose a model of Misparamatrized Sparse Regression (MiSpaR) and analytically compute the GE curves for $l_2$ and $l_1$ penalties.
We show that the overfitting peak arising in the interpolation limit is dissociated from the regime of good generalization.
The analytical expressions are obtained in the so called "thermodynamic" limit.
We find an additional interesting phenomenon: increasing overparametrization in the fitting model increases sparsity, which should intuitively improve performance of $l_1$ penalized regression.
However, at the same time, the relative number of measurements decrease compared to the number of fitting parameters, and eventually overparametrization does lead to poor generalization.
Nevertheless, $l_1$ penalized regression can show good generalization performance under conditions of data interpolation even with a large amount of overparametrization.
These results provide a theoretical avenue into studying inverse problems in the interpolating regime using overparametrized fitting functions such as deep nets.

assistant: text: Proposes an analytically tractable model and inference procedure (misparametrized sparse regression, inferred using L_1 penalty and studied in the data-interpolation limit) to study deep-net related phenomena in the context of inverse problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Task Level Data Augmentation for Meta-Learning

Paper content:
Data augmentation is one of the most effective approaches for improving the accuracy of modern machine learning models, and it is also indispensable to train a deep model for meta-learning.
However, most current data augmentation implementations applied in meta-learning are the same as those used in the conventional image classification.
In this paper, we introduce a new data augmentation method for meta-learning, which is named as ``Task Level Data Augmentation'' (referred to Task Aug).
The basic idea of Task Aug is to increase the number of image classes rather than the number of images in each class.
In contrast, with a larger amount of classes, we can sample more diverse task instances during training.
This allows us to train a deep network by meta-learning methods with little over-fitting.
Experimental results show that our approach achieves state-of-the-art performance on miniImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks.
Once paper is accepted, we will provide the link to code.

assistant: text: We propose a data augmentation approach for meta-learning and prove that it is valid.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

Paper content:
For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning.
One of the few well-known guidelines for architecture design is the avoidance of exploding or vanishing gradients.
However, even this guideline has remained relatively vague and circumstantial, because there exists no well-defined, gradient-based metric that can be computed {\it before} training begins and can robustly predict the performance of the network {\it after} training is complete.
We introduce what is, to the best of our knowledge, the first such metric: the nonlinearity coefficient (NLC).
Via an extensive empirical study, we show that the NLC, computed in the network's randomly initialized state, is a powerful predictor of test error and that attaining a right-sized NLC is essential for attaining an optimal test error, at least in fully-connected feedforward networks.
The NLC is also conceptually simple, cheap to compute, and is robust to a range of confounders and architectural design choices that comparable metrics are not necessarily robust to.
Hence, we argue the NLC is an important tool for architecture search and design, as it can robustly predict poor training outcomes before training even begins.

assistant: text: We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PAC-Bayesian Neural Network Bounds

Paper content:
Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to `effortlessly' extract desired representations from many large-scale datasets.
However, generalization bounds for this setting is still missing.
In this paper, we present a new PAC-Bayesian generalization bound for the negative log-likelihood loss which utilizes the \emph{Herbst Argument} for the log-Sobolev inequality to bound the moment generating function of the learners risk.

assistant: text: We derive a new PAC-Bayesian Bound for unbounded loss functions (e.g. Negative Log-Likelihood).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SALSA-TEXT : SELF ATTENTIVE LATENT SPACE BASED ADVERSARIAL TEXT GENERATION

Paper content:
Inspired by the success of self attention mechanism and Transformer architecture
in sequence transduction and image generation applications, we propose novel self
attention-based architectures to improve the performance of adversarial latent code-
based schemes in text generation.
Adversarial latent code-based text generation
has recently gained a lot of attention due to their promising results.
In this paper,
we take a step to fortify the architectures used in these setups, specifically AAE
and ARAE.
We benchmark two latent code-based methods (AAE and ARAE)
designed based on adversarial setups.
In our experiments, the Google sentence
compression dataset is utilized to compare our method with these methods using
various objective and subjective measures.
The experiments demonstrate the
proposed (self) attention-based models outperform the state-of-the-art in adversarial
code-based text generation.

assistant: text: We propose a self-attention based GAN architecture for unconditional text generation and improve on previous adversarial code-based results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: UW-NET: AN INCEPTION-ATTENTION NETWORK FOR UNDERWATER IMAGE CLASSIFICATION

Paper content:
The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning.
We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification.
In this model, we simulate the visual correlation of background attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module.
The experimental results demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet.
Moreover, we demonstrate the proposed IA module can be used to boost the performance of the existing object recognition networks.
By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the background attention in the image classifications.

assistant: text: A visual understanding mechanism for special environment
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Search Efficient DenseNet with Layer-wise Pruning

Paper content:
Deep neural networks have achieved outstanding performance in many real-world applications with the expense of huge computational resources.
The DenseNet, one of the recently proposed neural network architecture, has achieved the state-of-the-art performance in many visual tasks.
However, it has great redundancy due to the dense connections of the internal structure, which leads to high computational costs in training such dense networks.
To address this issue,  we design a reinforcement learning framework to search for efficient DenseNet architectures with layer-wise pruning (LWP) for different tasks, while retaining the original advantages of DenseNet, such as feature reuse, short paths, etc.
In this framework, an agent evaluates the importance of each connection between any two block layers, and prunes the redundant connections.
In addition, a novel reward-shaping trick is introduced to make DenseNet reach a better trade-off between accuracy and float point operations (FLOPs).
Our experiments show that DenseNet with LWP is more compact and efficient than existing alternatives.

assistant: text: Learning to Search Efficient DenseNet with Layer-wise Pruning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Continual Learning with Adaptive Weights (CLAW)

Paper content:
Approaches to continual learning aim to successfully learn a set of related tasks that arrive in an online manner.
Recently, several frameworks have been developed which enable deep learning to be deployed in this learning scenario.
A key modelling decision is to what extent the architecture should be shared across tasks.
On the one hand, separately modelling each task avoids catastrophic forgetting but it does not support transfer learning and leads to large models.
On the other hand, rigidly specifying a shared component and a task-specific part enables task transfer and limits the model size, but it is vulnerable to catastrophic forgetting and restricts the form of task-transfer that can occur.
Ideally, the network should adaptively identify which parts of the network to share in a data driven way.
Here we introduce such an approach called Continual Learning with Adaptive Weights (CLAW), which is based on probabilistic modelling and variational inference.
Experiments show that CLAW achieves state-of-the-art performance on six benchmarks in terms of overall continual learning performance, as measured by classification accuracy, and in terms of addressing catastrophic forgetting.

assistant: text: A continual learning framework which learns to automatically adapt its architecture based on a proposed variational inference algorithm.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The GAN Landscape: Losses, Architectures, Regularization, and Normalization

Paper content:
Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion.
While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks".
The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures.
In this work we take a sober view of the current state of GANs from a practical perspective.
We reproduce the current state of the art and go beyond fairly exploring the GAN landscape.
We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

assistant: text: A sober view on the current state of GANs from a practical perspective
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Coping With Simulators That Don’t Always Return

Paper content:
Deterministic models are approximations of reality that are often easier to build and interpret than stochastic alternatives.
Unfortunately, as nature is capricious, observational data can never be fully explained by deterministic models in practice.
Observation and process noise need to be added to adapt deterministic models to behave stochastically, such that they are capable of explaining and extrapolating from noisy data.
Adding process noise to deterministic simulators can induce a failure in the simulator resulting in no return value for certain inputs -- a property we describe as ``brittle.''
We investigate and address the wasted computation that arises from these failures, and the effect of such failures on downstream inference tasks.
We show that performing inference in this space can be viewed as rejection sampling, and train a conditional normalizing flow as a proposal over noise values such that there is a low probability that the simulator crashes, increasing computational efficiency and inference fidelity for a fixed sample budget when used as the proposal in an approximate inference algorithm.

assistant: text: We learn a conditional autoregressive flow to propose perturbations that don't induce simulator failure, improving inference performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Recurrent Layer Attention Network

Paper content:
Capturing long-range feature relations has been a central issue on convolutional neural networks(CNNs).
To tackle this, attempts to integrate end-to-end trainable attention module on CNNs are widespread.
Main goal of these works is to adjust feature maps considering spatial-channel correlation inside a convolution layer.
In this paper, we focus on modeling relationships among layers and propose a novel structure, 'Recurrent Layer Attention network,' which stores the hierarchy of features into recurrent neural networks(RNNs) that concurrently propagating with CNN and adaptively scales feature volumes of all layers.
We further introduce several structural derivatives for demonstrating the compatibility on recent attention modules and the expandability of proposed network.
For semantic understanding on learned features, we also visualize intermediate layers and plot the curve of layer scaling coefficients(i.e., layer attention).
Recurrent Layer Attention network achieves significant performance enhancement requiring a slight increase on parameters in an image classification task with CIFAR and ImageNet-1K 2012 dataset and an object detection task with Microsoft COCO 2014 dataset.

assistant: text: We propose a new type of end-to-end trainable attention module, which applies global weight balances among layers by utilizing co-propagating RNN with CNN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Audio Features for Singer Identification and Embedding

Paper content:
There has been an increasing use of neural networks for music information retrieval tasks.
In this paper, we empirically investigate different ways of improving the performance of convolutional neural networks (CNNs) on spectral audio features.
More specifically, we explore three aspects of CNN design: depth of the network, the use of residual blocks along with the use of grouped convolution, and global aggregation over time.
The application context is singer classification and singing performance embedding and we believe the conclusions extend to other types of music analysis using convolutional neural networks.
The results show that global time aggregation helps to improve the performance of CNNs the most.
Another contribution of this paper is the release of a singing recording dataset that can be used for training and evaluation.

assistant: text: Using deep learning techniques on singing voice related tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space

Paper content:
Challenges in natural sciences can often be phrased as optimization problems.
Machine learning techniques have recently been applied to solve such problems.
One example in chemistry is the design of tailor-made organic materials and molecules, which requires efficient methods to explore the chemical space.
We present a genetic algorithm (GA) that is enhanced with a neural network (DNN) based discriminator model to improve the diversity of generated molecules and at the same time steer the GA.
We show that our algorithm outperforms other generative models in optimization tasks.
We furthermore present a way to increase interpretability of genetic algorithms, which helped us to derive design principles

assistant: text: Tackling inverse design via genetic algorithms augmented with deep neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DELTA: DEEP LEARNING TRANSFER USING FEATURE MAP WITH ATTENTION FOR CONVOLUTIONAL NETWORKS

Paper content:
Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task.
To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied.
In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention.
Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network.
Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner.
We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP.
The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.

assistant: text: improving deep transfer learning with regularization using attention based feature maps
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Identifying Generalization Properties in Neural Networks

Paper content:
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian.
We connect model generalization with the local property of a solution under the PAC-Bayes paradigm.
In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters.
Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.

assistant: text: a theory connecting Hessian of the solution and the generalization power of the model
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness

Paper content:
This work provides theoretical and empirical evidence that invariance-inducing regularizers can increase predictive accuracy for worst-case spatial transformations (spatial robustness).
Evaluated on these adversarially transformed examples, we demonstrate that adding regularization on top of standard or adversarial training reduces the relative error by 20% for CIFAR10 without increasing the computational cost.
This outperforms handcrafted networks that were explicitly designed to be spatial-equivariant.
Furthermore, we observe for SVHN, known to have inherent variance in orientation, that robust training also improves standard accuracy on the test set.

assistant: text: for spatial transformations robust minimizer also minimizes standard accuracy; invariance-inducing regularization leads to better robustness than specialized architectures
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spatial Broadcast Decoder: A Simple Architecture for Disentangled Representations in VAEs

Paper content:
We present a neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations.
Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-“coordinate” channels, and apply a fully convolutional network with 1x1 stride.
This provides an architectural prior for dissociating positional from non-positional features in the latent space, yet without providing any explicit supervision to this effect.
We show that this architecture, which we term the Spatial Broadcast decoder, improves disentangling, reconstruction accuracy, and generalization to held-out regions in data space.
We show the Spatial Broadcast Decoder is complementary to state-of-the-art (SOTA) disentangling techniques and when incorporated improves their performance.
Knowledge transfer and generalization are hallmarks of human intelligence.
From grammatical generalization when learning a new language to visual generalization when interpreting a Picasso, humans have an extreme ability to recognize and apply learned patterns in new contexts.
Current machine learning algorithms pale in contrast, suffering from overfitting, adversarial attacks, and domain specialization BID12 BID16 .
We believe that one fruitful approach to improve generalization in machine learning is to learn compositional representations in an unsupervised manner.
A compositional representation consists of components that can be recombined, and such recombination underlies generalization.
For example, consider a pink elephant.
With a representation that composes color and object independently, imagining a pink elephant is trivial.
However, a pink elephant may not be within the scope of a representation that mixes color and object.
Compositionality comes in a variety of flavors, including feature compositionality (e.g. pink elephant), multi-object compositionality (e.g. elephant next to a penguin), and relational compositionality (e.g. the smallest elephant).
In this work we focus on feature compositionality.
Here we present the Spatial Broadcast decoder for Variational Autoencoders.
We demonstrate that it improves learned latent representations, most dramatically for datasets with objects varying in position.
It also improves generalization in latent space and can be incorporated into SOTA models to boost their performance in terms of both disentangling and reconstruction accuracy.
We believe that learning compositional representations is an important ingredient for flexibility and generalization in many contexts, from supervised learning to reinforcement learning, and the Spatial Broadcast decoder is one step towards robust compositional visual representation learning.

assistant: text: We introduce a neural rendering architecture that helps VAEs learn disentangled latent representations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Wasserstein is all you need

Paper content:
We propose a unified framework for building unsupervised representations of individual objects or entities (and their compositions), by associating with each object both a distributional as well as a point estimate (vector embedding).
This is made possible by the use of optimal transport, which allows us to build these associated estimates while harnessing the underlying geometry of the ground space.
Our method gives a novel perspective for building rich and powerful feature representations that simultaneously capture uncertainty (via a distributional estimate) and interpretability (with the optimal transport map).
As a guiding example, we formulate unsupervised representations for text, in particular for sentence representation and entailment detection.
Empirical results show strong advantages gained through the proposed framework.
This approach can be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data.
The key tools underlying the framework are Wasserstein distances and Wasserstein barycenters (and, hence the title!).

assistant: text: Represent each entity based on its histogram of contexts and then Wasserstein is all you need!
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Random Bias Initialization Improving Binary Neural Network Training

Paper content:
Edge intelligence especially binary neural network (BNN) has attracted considerable attention of the artificial intelligence community recently.
BNNs significantly reduce the computational cost, model size, and memory footprint.
However, there is still a performance gap between the successful full-precision neural network with ReLU activation and BNNs.
We argue that the accuracy drop of BNNs is due to their geometry.
We analyze the behaviour of the full-precision neural network with ReLU activation and compare it with its binarized counterpart.
This comparison suggests random bias initialization as a remedy to activation saturation in full-precision networks and  leads us towards an improved BNN training.
Our numerical experiments confirm our geometric intuition.

assistant: text: Improve saturating activations (sigmoid, tanh, htanh etc.) and Binarized Neural Network with Bias Initialization
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Data for free: Fewer-shot algorithm learning with parametricity data augmentation

Paper content:
We address the problem of teaching an RNN to approximate list-processing algorithms given a small number of input-output training examples.
Our approach is to generalize the idea of parametricity from programming language theory to formulate a semantic property that distinguishes common algorithms from arbitrary non-algorithmic functions.
This characterization leads naturally to a learned data augmentation scheme that encourages RNNs to learn algorithmic behavior and enables small-sample learning in a variety of list-processing tasks.
Since the earliest days of neural network research, some of the most important questions about neural models have focused on their ability to capture the crispness, systematicity and compositionality that characterize symbolic computation and human cognition BID2 BID11 , and to do so with a human-like number of examples BID10 .
While recent studies have demonstrated promising results in training recurrent neural networks (RNNs) to approximate symbolic algorithms in domains like list manipulation BID4 BID7 , binary arithmetic BID8 , graph traversal BID3 , and planar geometry BID12 , the question of sample efficiency remains very much open.
Difficult algorithmic problems may require tens or hundreds of thousands of labelled training examples, and even simple tasks on small inputs seem to require more data than should be necessary BID9 .Our
goal in this paper is to teach RNNs to approximate list-processing algorithms f :: DISPLAYFORM0 . Inspired
by the idea of parametricity BID13 ) from type theory and functional programming, we hypothesize that a feature that distinguishes many algorithms from arbitrary functions is that they commute with some family of element-wise changes to their inputs. We describe
a method for learning this family from the training set D, and show how this learned information can be used to create an augmented training set for an RNN. Our experiments
show that this augmentation scheme makes it possible to approximate algorithms from small training sets, in some cases requiring only a single example per input list length.

assistant: text: Learned data augmentation instills algorithm-favoring inductive biases that let RNNs learn list-processing algorithms from fewer examples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Score and Lyrics-Free Singing Voice Generation

Paper content:
Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics.
In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time.
In particular, we experiment with three different schemes:
1) free singer, where the model generates singing voices without taking any conditions;
2) accompanied singer, where the model generates singing voices over a waveform of instrumental music; and
3) solo singer, where the model improvises a chord sequence first and then uses that to generate voices.
We outline the associated challenges and propose a pipeline to tackle these new tasks.
This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.

assistant: text: Our models generate singing voices without lyrics and scores. They take accompaniment as input and output singing voices.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CRAP: Semi-supervised Learning via Conditional Rotation Angle Prediction

Paper content:
Self-supervised learning (SlfSL), aiming at learning feature representations through ingeniously designed pretext tasks without human annotation, has achieved compelling progress in the past few years.
Very recently, SlfSL has also been identified as a promising solution for semi-supervised learning (SemSL) since it offers a new paradigm to utilize unlabeled data.
This work further explores this direction by proposing a new framework to seamlessly couple SlfSL with SemSL.
Our insight is that the prediction target in SemSL can be modeled as the latent factor in the predictor for the SlfSL target.
Marginalizing over the latent factor naturally derives a new formulation which marries the prediction targets of these two learning processes.
By implementing this framework through a simple-but-effective SlfSL approach -- rotation angle prediction, we create a new SemSL approach called Conditional Rotation Angle Prediction (CRAP).
Specifically, CRAP is featured by adopting a module which predicts the image rotation angle \textbf{conditioned on the candidate image class}.
Through experimental evaluation, we show that CRAP achieves superior performance over the other existing ways of combining SlfSL and SemSL.
Moreover, the proposed SemSL framework is highly extendable.
By augmenting CRAP with a simple SemSL technique and a modification of the rotation angle prediction task, our method has already achieved the state-of-the-art SemSL performance.

assistant: text: Coupling semi-supervised learning with self-supervised learning and explicitly modeling the self-supervised task conditioned on the semi-supervised one
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Reactor: A fast and sample-efficient Actor-Critic agent for  Reinforcement Learning

Paper content:
In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2016) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016).
Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting.
The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones.
Next, we introduce the β-leaveone-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline.
Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization.
Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance.
Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.

assistant: text: Reactor combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN while giving better run-time performance than A3C.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model

Paper content:
Continuous Bag of Words (CBOW) is a powerful text embedding method.
Due to its strong capabilities to encode word content, CBOW embeddings perform well on a wide range of downstream tasks while being efficient to compute.
However, CBOW is not capable of capturing the word order.
The reason is that the computation of CBOW's word embeddings is commutative, i.e., embeddings of XYZ and ZYX are the same.
In order to address this shortcoming, we propose a
learning algorithm for the Continuous Matrix Space Model, which we call Continual Multiplication of Words (CMOW).
Our algorithm is an adaptation of word2vec, so that it can be trained on large quantities of unlabeled text.
We empirically show that CMOW better captures linguistic properties, but it is inferior to CBOW in memorizing word content.
Motivated by these findings, we propose a hybrid model that combines the strengths of CBOW and CMOW.
Our results show that the hybrid CBOW-CMOW-model retains CBOW's strong ability to memorize word content while at the same time substantially improving its ability to encode other linguistic information by 8%.
As a result, the hybrid also performs better on 8 out of 11 supervised downstream tasks with an average improvement of 1.2%.

assistant: text: We present a novel training scheme for efficiently obtaining order-aware sentence representations.