ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RANDOM MASK: Towards Robust Convolutional Neural Networks

Paper content:
Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs.
In this paper, we design a new CNN architecture that by itself has good robustness.
We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures.
We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training.
We next investigate the adversarial examples which “fool” a CNN with Random Mask.
Surprisingly, we find that these adversarial examples often “fool” humans as well.
This raises fundamental questions on how to define adversarial examples and robustness properly.

assistant: text: We propose a technique that modifies CNN structures to enhance robustness while keeping high test accuracy, and raise doubt on whether current definition of adversarial examples is appropriate by generating adversarial examples able to fool humans.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION

Paper content:
Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.
SGD estimates the gradient by uniform sampling with sample size one.
There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.
Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.
As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.
In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.
Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.
As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.
We demonstrate the benefits of our proposal on both SGD and AdaGrad.
In this paper, we proposed a novel LSH-based sampler with a reduction to the gradient estimation variance.
We achieved it by sampling with probability proportional to the L 2 norm of the instances gradients leading to an optimal distribution that minimizes the variance of estimation.
More remarkably, LSD is as computationally efficient as SGD but achieves faster convergence not only epoch wise but also time wise.Peilin Zhao and Tong Zhang.
Stochastic optimization with importance sampling for regularized loss minimization.
In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1-9, 2015.A EPOCH PLOTS AND PROOFS Theorem 3.
Let S be the bucket that sample x m is chosen from in Algorithm 2.
Let p m be the sampling probability associated with sample x m .
Suppose we query a sample with ✓ t .
Then we have an unbiased estimator of the full gradient: DISPLAYFORM0 Proof.
DISPLAYFORM1 Theorem 4.
The Trace of the covariance of our estimator is: DISPLAYFORM2 Proof.

assistant: text: We improve the running of all existing gradient descent algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Conversation Generation with Concept Flow

Paper content:
Human conversations naturally evolve around related entities and connected concepts, while may also shift from topic to topic.
This paper presents ConceptFlow, which leverages commonsense knowledge graphs to explicitly model such conversation flows for better conversation response generation.
ConceptFlow grounds the conversation inputs to the latent concept space and represents the potential conversation flow as a concept flow along the commonsense relations.
The concept is guided by a graph attention mechanism that models the possibility of the conversation evolving towards different concepts.
The conversation response is then decoded using the encodings of both utterance texts and concept flows, integrating the learned conversation structure in the concept space.
Our experiments on Reddit conversations demonstrate the advantage of ConceptFlow over previous commonsense aware dialog models and fine-tuned GPT-2 models, while using much fewer parameters but with explicit modeling of conversation structures.

assistant: text: This paper presents ConceptFlow that explicitly models the conversation flow in commonsense knowledge graph for better conversation generation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Novelty Detection Via Blurring

Paper content:
Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) are known to assign lower uncertainty to the OOD data than the target distribution.
In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images.
Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training.
Our detector is simple, efficient in test time, and outperforms baseline OOD detectors in various domains.
Further results show that SVD-RND learns a better target distribution representation than the baselines.
Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy in CelebA domain.

assistant: text: This paper tackles out-of-data distribution by leveraging RND applied to data augmentations by training a model to match the outputs of a random network with an augmentation as input.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training GANs with Optimism

Paper content:
We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs.
Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games.
WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics.
Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs.
We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle.
We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum.
We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences.
We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants.
Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam.
We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.

assistant: text: The paper proposes to use optimistic gradient descent for GAN training that avoids the cycling behavior observed with SGD and its variants and provides promising results in GAN training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Neural Machine Translation

Paper content:
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.
There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal.
In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora.
Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation.
Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation.
The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively.
Our implementation is released as an open source project.

assistant: text: We introduce the first successful method to train neural machine translation in an unsupervised manner, using nothing but monolingual corpora
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Text Classifier on Test-Time Budgets

Paper content:
In this paper, we design a generic framework for learning a robust text classification model that achieves accuracy comparable to standard full models under test-time
budget constraints.
We take a different approach from existing methods and learn to dynamically delete a large fraction of unimportant words by a low-complexity selector such that the high-complexity classifier only needs to process a small fraction of important words.
In addition, we propose a new data aggregation method to train the classifier, allowing it to make accurate predictions even on fragmented sequence of words.
Our end-to-end method achieves state-of-the-art performance while its computational complexity scales linearly with the small fraction of important words in the whole corpus.
Besides, a single deep neural network classifier trained by our framework can be dynamically tuned to different budget levels at inference time.
Recent advances in deep neural networks (DNN) has improved the performance of natural language processing tasks such as document classification, question answering, and sentiment analysis BID29 BID20 BID22 BID31 .
These approaches process the entire text and construct representations of words and phrases in order to perform target tasks.
While these models do realize high accuracy, their computational-time scales linearly with the size of the documents, which can be slow for documents containing many sentences.
In this context, various approaches based on modifying the existing RNN or LSTM architecture have been proposed BID21 ; BID31 to speed-up processing.
However, processing is still fundamentally sequential, which in turn requires loading entire documents to process, limiting compute gains.
We proposed a budgeted learning framework for learning a robust classifier under test-time budget constraints.
We demonstrated that training classifiers with data aggregation work well with low-complexity selectors based on word-embedding or bag-of-word model and achieve good performance with fragmented input.
The future work includes applying the proposed framework to other text reading tasks and improving the data aggregation strategy by applying learning to search approaches BID4 .

assistant: text: Modular framework for document classification and data aggregation technique for making the framework robust to various distortion, and noise and focus only on the important words.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Meaning-Preserving Adversarial Perturbations for Sequence-to-Sequence Models

Paper content:
Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance.
However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output.
Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general.
Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs.
Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

assistant: text: How you should evaluate adversarial attacks on seq2seq
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Uncertainty in Multitask Transfer Learning

Paper content:
Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks.
This results in a rich prior capable of few-shot learning on new tasks.
The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments.
Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them.
Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning.
Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.

assistant: text: A scalable method for learning an expressive prior over neural networks across multiple tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SGD Learns One-Layer Networks in WGANs

Paper content:
Generative adversarial networks (GANs) are a widely used framework for learning generative models.
Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.
In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.

assistant: text: We show that stochastic gradient descent ascent converges to a global optimum for WGAN with one-layer generator network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RANDOM MASK: Towards Robust Convolutional Neural Networks

Paper content:
Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed  perturbations which are imperceptible to humans but can cause the network to give incorrect outputs.
In this paper, we design a new CNN architecture that by itself has good robustness.
We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures.
We show that CNN with Random Mask achieves state-of-the-art performance against black-box adversarial attacks without applying any adversarial training.
We next investigate the adversarial examples which “fool” a CNN with Random Mask.
Surprisingly, we find that these adversarial examples often “fool” humans as well.
This raises fundamental questions on how to define adversarial examples and robustness properly.

assistant: text: The authors propose a simple method for increasing the robustness of convolutional neural networks against adversarial examples, with surprisingly good results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit

Paper content:
The loss of a few neurons in a brain rarely results in any visible loss of function.
However, the insight into what “few” means in this context is unclear.
How many random neuron failures will it take to lead to a visible loss of function?
In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces.
We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting.
We give provable guarantees on the robustness of the network to these crashes.
Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar.
The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks.
We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture.
We design an algorithm achieving fault tolerance using a reasonable number of neurons.
In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.

assistant: text: This contribution studies the impact of deletions of random neurons on prediction accuracy of trained architecture, with the application to failure analysis and the specific context of neuromorphic hardware.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: How transferable are features in convolutional neural network acoustic models across languages?

Paper content:
Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies.
Here we study convolutional neural network-based acoustic models in the context of automatic speech recognition.
Adapting a method proposed by Yosinski et al. [2014], we measure the transferability of each layer between German and English to assess the their language-specifity.
We observe three distinct regions of transferability: (1) the first two layers are entirely transferable between languages, (2) layers 2–8 are also highly transferable but we find evidence of some language specificity, (3) the subsequent fully connected layers are more language specific but can be successfully finetuned to the target language.
To further probe the effect of weight freezing, we performed follow-up experiments using freeze-training [Raghu et al., 2017].
Our results are consistent with the observation that CCNs converge 'bottom up' during training and demonstrate the benefit of freeze training, especially for transfer learning.

assistant: text: The paper measures the transferability of features for each layer in CNN-based acoustic models across languages, concluding that AMs trained with 'the freeze training' technique outperformed other transferred models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Document Embeddings With CNNs

Paper content:
This paper proposes a new model for document embedding.
Existing approaches either require complex inference or use recurrent neural networks that are difficult to parallelize.
We take a different route and use recent advances in language modeling to develop a convolutional neural network embedding model.
This allows us to train deeper architectures that are fully parallelizable.
Stacking layers together increases the receptive filed allowing each successive layer to model increasingly longer range semantic dependences within the document.
Empirically we demonstrate superior results on two publicly available benchmarks.
Full code will be released with the final version of this paper.

assistant: text: Convolutional neural network model for unsupervised document embedding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Foveated Downsampling Techniques

Paper content:
Foveation is an important part of human vision, and a number of deep networks have also used foveation.
However, there have been few systematic comparisons between foveating and non-foveating deep networks, and between different variable-resolution downsampling methods.
Here we define several such methods, and compare their performance on ImageNet recognition with a Densenet-121 network.
The best variable-resolution method slightly outperforms uniform downsampling.
Thus in our experiments, foveation does not substantially help or hinder object recognition in deep networks.
To help fill this gap, we compare several foveated downsampling approaches to uniform downsampling 23 in object recognition.
In this context, the different foveated methods perform fairly similarly to each 24 other, and the best performs slightly better than uniform downsampling (top-1 validation accuracy 25 48.95% vs. 47.72%; Table 1 ).
Therefore, foveation does not seem to be important for object 26 recognition (which is unsurprising given the good performance of standard deep networks), but it 27 does not greatly interfere either.
This suggests that foveation could be incorporated into more general 28 vision systems that perform multiple tasks, such as in robots that must recognize objects and also 29 read text in the environment.
We trained deep networks on the ImageNet recognition task, with various kinds of downsampled 33 images as input.
In each case we used a DenseNet-121 [1] network with original hyperparameters Figure 1 : Estimate of retinal ganglion cell (RGC) density as a function of degrees from the fovea.
We use estimates from [3] , which provides data along the nasal-temporal axis.
[4] shows that density is similar in temporal, dorsal, and ventral directions, but higher in the nasal direction.
To calculate radially symmetric mean values, we sum nasal and temporal fits from [3] with weights 0.25 and 0.75 (to account for the fact that nasal density is atypical). and training procedure, including random horizontal flips, batch size etc.
We trained each network 35 for 90 epochs, using SGD (initial learning rate 0.1, reduced by 10x every 30 epochs).
resulting crop went outside the image boundaries, we extrapolated by copying edge pixels.
We sometimes chose multiple points in a single image.
The highest-saliency points are typically close 61 together, and contain similar information.
To avoid selecting multiple similar points, we modified the 62 saliency maps after each selection.
Specifically, we subtracted a square-gaussian function from the 63 saliency map, with a peak equal to the saliency at the chosen point, and a width of 60 pixels.
interesting to explore potential benefits within more general vision systems.

assistant: text: We compare object recognition performance on images that are downsampled uniformly and with three different foveation schemes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A closer look at the approximation capabilities of neural networks

Paper content:
The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions σ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold ε, if and only if σ is non-polynomial.
In this paper, we give a direct algebraic proof of the theorem.
Furthermore we shall explicitly quantify the number of hidden units required for approximation.
Specifically, if X in R^n is compact, then a neural network with n input units, m output units, and a single hidden layer with {n+d choose d} hidden units (independent of m and ε), can uniformly approximate any polynomial function f:X -> R^m whose total degree is at most d for each of its m coordinate functions.
In the general case that f is any continuous function, we show there exists some N in O(ε^{-n}) (independent of m), such that N hidden units would suffice to approximate f.
We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights.
We highlight several consequences:
(i) For any δ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < δ.
(ii) There exists some λ>0 (depending only on f and σ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>λ.
(iii) If the non-bias weights in the first layer are *fixed* and randomly chosen from a suitable range, then the UAP holds with probability 1.

assistant: text: A new proof of Leshno's version of the universal approximation property for neural networks, and new insights into the universal approximation property.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Function changes in the Backpropagation equation is equivalent to an implicit learning rate

Paper content:
The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results.
Since its conception, variants of the backpropagation algorithm have emerged.
More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements.
Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm.
In this work, we show that function changes in the  backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network.
Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks.
Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.
Credit assignment BID10 is the task of identifying neurons and weights that are responsible for a desired prediction.
Currently, the backpropagation (BP) algorithm BID9 ) is the de-facto standard for credit assignment in artificial neural networks.
The backpropagation algorithm assigns credit by computing partial derivatives for weights and neurons with respect to the networks cost function.Variants of the backpropagation procedure have emerged since its conception.
More specifically variants that exploit function changes in the backpropagation procedure.
Feedback Alignment BID5 is considered a biologically plausible alternative to vanilla backpropagation.
Feedback alignment is a variant of the backpropagation algorithm that uses a random weight matrix instead of the weight transpose matrix in the backpropagation equation.
Despite not scaling to the ImageNet dataset BID1 , the algorithm relaxes BP weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets Similarly, BID0 produced unique backpropagation equations to train an artificial neural network by learning parts of the backpropagation equations.
BID0 report early convergence on unique backpropagation equations for CIFAR10.
In this work, we demonstrate that function changes in the backpropagation equations particularly activation function derivatives is equivalent to adding an implicit learning rate in stochastic gradient descent.

assistant: text: We demonstrate that function changes in the backpropagation is equivalent to an implicit learning rate
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Disentanglement using Mixture Latent Space with Continuous and Discrete Variables

Paper content:
Recent advances in deep learning techniques has shown the usefulness of the deep neural networks in extracting features required to perform the task at hand.
However, these features learnt are in particular helpful only for the initial task.
This is due to the fact that the features learnt are very task specific and does not capture the most general and task agnostic features of the input.
In fact the way humans are seen to learn is by disentangling features which task agnostic.
This indicates that leaning task agnostic features by disentangling only the most informative features from the input data.
Recently Variational Auto-Encoders (VAEs) have shown to be the de-facto models to capture the latent variables in a generative sense.
As these latent features can be represented as continuous and/or discrete variables, this indicates us to use VAE with a mixture of continuous and discrete variables for the latent space.
We achieve this by performing our experiments using a modified version of joint-vae to learn the disentangled features.

assistant: text: Mixture Model for Neural Disentanglement
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Déjà Vu: An Empirical Evaluation of the Memorization Properties of Convnets

Paper content:
Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate over- fitting.
This paper considers the related question of “membership inference”, where the goal is to determine if an image was used during training.
We con- sider membership tests over either ensembles of samples or individual samples.
First, we show how to detect if a dataset was used to train a model, and in particular whether some validation images were used at train time.
Then, we introduce a new approach to infer membership when a few of the top layers are not available or have been fine-tuned, and show that lower layers still carry information about the training samples.
To support our findings, we conduct large-scale experiments on Imagenet and subsets of YFCC-100M with modern architectures such as VGG and Resnet.

assistant: text: We analyze the memorization properties by a convnet of the training set and propose several use-cases where we can extract some information about the training set.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Style Memory: Making a Classifier Network Generative

Paper content:
Deep networks have shown great performance in classification tasks.
However, the parameters learned by the classifier networks usually discard stylistic information of the input, in favour of information strictly relevant to classification.
We introduce a network that has the capacity to do both classification and reconstruction by adding a "style memory" to the output layer of the network.
We also show how to train such a neural network as a deep multi-layer autoencoder, jointly minimizing both classification and reconstruction losses.
The generative capacity of our network demonstrates that the combination of style-memory neurons with the classifier neurons yield good reconstructions of the inputs when the classification is correct.
We further investigate the nature of the style memory, and how it relates to composing digits and letters.

assistant: text: The paper proposes training an autoencoder such that the middle layer representation consists of the class label of the input and a hidden vector representation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generalizing Natural Language Analysis through Span-relation Representations

Paper content:
A large number of natural language processing tasks exist to analyze syntax, semantics, and information content of human language.
These seemingly very different tasks are usually solved by specially designed architectures.
In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks.
We perform extensive experiments to test this insight on 10 disparate tasks as broad as dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving comparable performance as state-of-the-art specialized models.
We further demonstrate benefits in multi-task learning.
We convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.

assistant: text: This work presents a unified formulation of various phrase and token level NLP tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Deep Latent-variable MRFs with Amortized Bethe Free Energy Minimization

Paper content:
While much recent work has targeted learning deep discrete latent variable models with variational inference, this setting remains challenging, and it is often necessary to make use of potentially high-variance gradient estimators in optimizing the ELBO.
As an alternative, we propose to optimize a non-ELBO objective derived from the Bethe free energy approximation to an MRF's partition function.
This objective gives rise to a saddle-point learning problem, which we train inference networks to approximately optimize.
The derived objective requires no sampling, and can be efficiently computed for many MRFs of interest.
We evaluate the proposed approach in learning high-order neural HMMs on text, and find that it often outperforms other approximate inference schemes in terms of true held-out log likelihood.
At the same time, we find that all the approximate inference-based approaches to learning high-order neural HMMs we consider underperform learning with exact inference by a significant margin.

assistant: text: An objective for learning latent variable MRFs based on Bethe free energy and amortized inference, different from optimizing the standard ELBO.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Pooling Is Neither Necessary nor Sufficient for Appropriate Deformation Stability in CNNs

Paper content:
Many of our core assumptions about how neural networks operate remain empirically untested.
One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks.
For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers.
Recently, however, interleaved pooling has largely been abandoned.
This raises a number of questions: Are our intuitions about deformation stability right at all?
Is it important?
Is pooling necessary for deformation invariance?
If not, how is deformation invariance achieved in its absence?
In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers.
(2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters.
(3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification.
(4) Pooling confers \emph{too much} deformation stability for image classification at initialization, and during training, networks have to learn to \emph{counteract} this inductive bias.
Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.

assistant: text: We find that pooling alone does not determine deformation stability in CNNs and that filter smoothness plays an important role in determining stability.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Receptive Field Learning by Dynamic Gaussian Structure

Paper content:
The visual world is vast and varied, but its variations divide into structured and unstructured factors.
Structured factors, such as scale and orientation, admit clear theories and efficient representation design.
Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.
We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.
Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.
(Please see https://arxiv.org/abs/1904.11487 for the full edition.)

assistant: text: This paper proposes a structured convolution operator to model deformations of local regions of an image, which significantly reduced the number of parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Uncertainty-aware Variational-Recurrent Imputation Network for Clinical Time Series

Paper content:
Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.
Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.
In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.
In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.
Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.
We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

assistant: text: The paper presented a method that combines VAE and uncertainty aware GRU for sequential missing data imputation and outcome prediction.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training Structured Efficient Convolutional Layers

Paper content:
Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting.
We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods.
A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.

assistant: text: This work applies previous Structured Efficient Linear Layers to conv layers and proposes Structured Efficient Convolutional Layers as substitution of original conv layers.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeFINE: Deep Factorized Input Word Embeddings for Neural Sequence Modeling

Paper content:
For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers.
In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently.
Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.
DeFINE can be incorporated easily in new or existing sequence models.
Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.
On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance.
On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters,  achieving comparable performance to state-of-the-art methods with fewer parameters.
For machine translation, DeFINE improves a Transformer model by 2% while simultaneously reducing total parameters by 26%

assistant: text: DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better word embeddings efficiently.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Part-Based 3D Face Morphable Model with Anthropometric Local Control

Paper content:
We propose an approach to construct realistic 3D facial morphable models (3DMM) that allows an intuitive facial attribute
editing workflow.
Current face modeling methods using 3DMM suffer from the lack of local control.
We thus create a 3DMM by
combining local part-based 3DMM for the eyes, nose, mouth, ears, and facial mask regions.
Our local PCA-based approach
uses a novel method to select the best eigenvectors from the local 3DMM to ensure that the combined 3DMM is expressive
while allowing accurate reconstruction.
The editing controls we provide to the user are intuitive as they are extracted from
anthropometric measurements found in the literature.
Out of a large set of possible anthropometric measurements, we filter the
ones that have meaningful generative power given the face data set.
We bind the measurements to the part-based 3DMM through
mapping matrices derived from our data set of facial scans.
Our part-based 3DMM is compact yet accurate, and compared to
other 3DMM methods, it provides a new trade-off between local and global control.
We tested our approach on a data set of 135
scans used to derive the 3DMM, plus 19 scans that served for validation.
The results show that our part-based 3DMM approach
has excellent generative properties and allows intuitive local control to the user.

assistant: text: We propose an approach to construct realistic 3D facial morphable models (3DMM) that allows an intuitive facial attribute editing workflow by selecting the best sets of eigenvectors and anthropometric measurements.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Graph-to-Graph Translation for Molecules

Paper content:
The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties.
Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization.
In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph.
Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule.
We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

assistant: text: The authors present a hierarchical graph-to-graph translation method for generating novel organic molecules.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unravelling the neural signatures of dream recall in EEG: a deep learning approach

Paper content:
Dreams and our ability to recall them are among the most puzzling questions in sleep research.
Specifically, putative differences in brain network dynamics between individuals with high versus low dream recall rates, are still poorly understood.
In this study, we addressed this question as a classification problem where we applied deep convolutional networks (CNN) to sleep EEG recordings to predict whether subjects belonged to the high or low dream recall group (HDR and LDR resp.).
Our model achieves significant accuracy levels across all the sleep stages, thereby indicating subtle signatures of dream recall in the sleep microstructure.
We also visualized the feature space to inspect the subject-specificity of the learned features, thus ensuring that the network captured population level differences.
Beyond being the first study to apply deep learning to sleep EEG in order to classify HDR and LDR, guided backpropagation allowed us to visualize the most discriminant features in each sleep stage.
The significance of these findings and future directions are discussed.

assistant: text: We investigate the neural basis of dream recall using convolutional neural network and feature visualization techniques, like tSNE and guided-backpropagation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Tunability of Optimizers in Deep Learning

Paper content:
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.
In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?
We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.
Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

assistant: text: Introduction of a novel metric to capture the tunability of an optimizer, and a comprehensive empirical comparison of deep learning optimizers under different amounts of hyper-parameter tuning.