ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework

Paper content:
Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks.
There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly.
In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks.
Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent.
Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches.
Extensive experiments on various tasks demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark.
We further show that our proposed framework can generalize to contextualized representations and achieves state-of-the-art results on the CoNLL cross-lingual NER benchmark.

assistant: text: This paper compares approaches to bilingual lexicon induction and shows which method performs better on lexicon, induction, and NER and MT tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

Paper content:
While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages.
We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language.
XLDA enhances performance of all 14 tested languages of the cross-lingual natural language inference (XNLI) benchmark.
With improvements of up to 4.8, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu.
XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language.
On the SQuAD question answering task, we see that XLDA provides a 1.0 performance increase on the English evaluation set.
Comprehensive experiments suggest that most languages are effective as cross-lingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

assistant: text: The paper proposes a cross-lingual data augmentation method to improve the language inference and question answering tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Semi-Supervised Semantic Dependency Parsing Using CRF Autoencoders

Paper content:
Semantic dependency parsing, which aims to find rich bi-lexical relationships, allows words to have multiple dependency heads, resulting in graph-structured representations.
We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework.
Our encoder is a discriminative neural semantic dependency parser that predicts the latent parse graph of the input sentence.
Our decoder is a generative neural model that reconstructs the input sentence conditioned on the latent parse graph.
Our model is arc-factored and therefore parsing and learning are both tractable.
Experiments show our model achieves significant and consistent improvement over the supervised baseline.

assistant: text: We propose an approach to semi-supervised learning of semantic dependency parsers based on the CRF autoencoder framework.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints

Paper content:
In most practical settings and theoretical analyses, one assumes that a model can be trained until convergence.
However, the growing complexity of machine learning datasets and models may violate such assumptions.
Indeed, current approaches for hyper-parameter tuning and neural architecture search tend to be limited by practical resource constraints.
Therefore, we introduce a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training.
We analyze the following problem: "given a dataset, algorithm, and fixed resource budget, what is the best achievable performance?"
We focus on the number of optimization iterations as the representative resource.
Under such a setting, we show that it is critical to adjust the learning rate schedule according to the given budget.
Among budget-aware learning schedules, we find simple linear decay to be both robust and high-performing.
We support our claim through extensive experiments with state-of-the-art models on ImageNet (image classification), Kinetics (video classification), MS COCO (object detection and instance segmentation), and Cityscapes (semantic segmentation).
We also analyze our results and find that the key to a good schedule is budgeted convergence, a phenomenon whereby the gradient vanishes at the end of each allowed budget.
We also revisit existing approaches for fast convergence and show that budget-aware learning schedules readily outperform such approaches under (the practical but under-explored) budgeted training setting.

assistant: text: This paper analyzed which learning rate schedule should be used when the number of iteration is limited using an introduced concept of BAS (Budget-Aware Schedule).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spectral Capsule Networks

Paper content:
In search for more accurate predictive models, we customize capsule networks for the learning to diagnose problem.
We also propose Spectral Capsule Networks, a novel variation of capsule networks, that converge faster than capsule network with EM routing.
Spectral capsule networks  consist of spatial coincidence filters that detect entities based on the alignment of extracted features on a one-dimensional linear subspace.
Experiments on a public benchmark learning to diagnose dataset not only shows the success of capsule networks on this task, but also confirm the faster convergence of the spectral capsule networks.

assistant: text: A new capsule network that converges faster on our healthcare benchmark experiments.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Novelty Search in representational space for sample efficient exploration

Paper content:
We present a new approach for efficient exploration which leverages a low-dimensional encoding of the environment learned with a combination of model-based and model-free objectives.
Our approach uses intrinsic rewards that are based on a weighted distance of nearest neighbors in the low dimensional representational space to gauge novelty.
We then leverage these intrinsic rewards for sample-efficient exploration with planning routines in representational space.
One key element of our approach is that we perform more gradient steps in-between every environment step in order to ensure the model accuracy.
We test our approach on a number of maze tasks, as well as a control problem and show that our exploration approach is more sample-efficient compared to strong baselines.

assistant: text: This paper proposes a method of sample-efficient exploration for RL agent using a combination of model-based and model-free approaches with a novelty metric.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Structural Fingerprints for Graph Attention Networks

Paper content:
Many real-world data sets are represented as graphs, such as citation links, social media, and biological interaction.
The volatile graph structure makes it non-trivial to employ convolutional neural networks (CNN's) for graph data processing.
Recently, graph attention network (GAT) has proven a promising attempt by combining graph neural networks with attention mechanism, so as to achieve massage passing in graphs with arbitrary structures.
However, the attention in GAT is computed mainly based on the similarity between the node content, while the structures of the graph remains largely unemployed (except in masking the attention out of one-hop neighbors).
In this paper, we propose an `````````````````````````````"ADaptive Structural Fingerprint" (ADSF) model to fully exploit both topological details of the graph and  content features of the nodes.
The key idea is to contextualize each node with a weighted, learnable receptive field  encoding rich and diverse local graph structures.
By doing this, structural interactions between the nodes can  be inferred accurately, thus improving subsequent attention layer as well as the convergence of learning.
Furthermore, our model provides a useful platform  for different subspaces of node features and various scales of graph structures to ``cross-talk'' with each other through the learning of multi-head attention, being particularly useful in handling complex real-world data.
Encouraging performance is observed on a number of benchmark data sets in node classification.

assistant: text: A graph structure based methodology to augment the attention mechanism of graph neural networks, with the main idea to explore interactions between different types of nodes of the local neighborhood of a root node.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning a Convolutional Bilinear Sparse Code for Natural Videos

Paper content:
In contrast to the monolithic deep architectures used in deep learning today for computer vision, the visual cortex processes retinal images via two functionally distinct but interconnected networks: the ventral pathway for processing object-related information and the dorsal pathway for processing motion and transformations.
Inspired by this cortical division of labor and properties of the magno- and parvocellular systems, we explore an unsupervised approach to feature learning that jointly learns object features and their transformations from natural videos.
We propose a new convolutional bilinear sparse coding model that (1) allows independent feature transformations and (2) is capable of processing large images.
Our learning procedure leverages smooth motion in natural videos.
Our results show that our model can learn groups of features and their transformations directly from natural videos in a completely unsupervised manner.
The learned "dynamic filters" exhibit certain equivariance properties, resemble cortical spatiotemporal filters, and capture the statistics of transitions between video frames.
Our model can be viewed as one of the first approaches to demonstrate unsupervised learning of primary "capsules" (proposed by Hinton and colleagues for supervised learning) and has strong connections to the Lie group approach to visual perception.

assistant: text: We extend bilinear sparse coding and leverage video sequences to learn dynamic filters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exploring Cellular Protein Localization Through Semantic Image Synthesis

Paper content:
Cell-cell interactions have an integral role in tumorigenesis as they are critical in governing immune responses.
As such, investigating specific cell-cell interactions has the potential to not only expand upon the understanding of tumorigenesis, but also guide clinical management of patient responses to cancer immunotherapies.
A recent imaging technique for exploring cell-cell interactions, multiplexed ion beam imaging by time-of-flight (MIBI-TOF), allows for cells to be quantified in 36 different protein markers at sub-cellular resolutions in situ as high resolution multiplexed images.
To explore the MIBI images, we propose a GAN for multiplexed data with protein specific attention.
By conditioning image generation on cell types, sizes, and neighborhoods through semantic segmentation maps, we are able to observe how these factors affect cell-cell interactions simultaneously in different protein channels.
Furthermore, we design a set of metrics and offer the first insights towards cell spatial orientations, cell protein expressions, and cell neighborhoods.
Our model, cell-cell interaction GAN (CCIGAN), outperforms or matches existing image synthesis methods on all conventional measures and significantly outperforms on biologically motivated metrics.
To our knowledge, we are the first to systematically model multiple cellular protein behaviors and interactions under simulated conditions through image synthesis.

assistant: text: We explore cell-cell interactions across tumor environment contexts observed in highly multiplexed images, by image synthesis using a novel attention GAN architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Model-Agnostic Feature Selection with Additional Mutual Information

Paper content:
Answering questions about data can require understanding what parts of an input X influence the response Y. Finding such an understanding can be built by testing relationships between variables through a machine learning model.
For example, conditional randomization tests help determine whether a variable relates to the response given the rest of the variables.
However, randomization tests require users to specify test statistics.
We formalize a class of proper test statistics that are guaranteed to select a feature when it provides information about the response even when the rest of the features are known.
We show that f-divergences provide a broad class of proper test statistics.
In the class of f-divergences, the KL-divergence yields an easy-to-compute proper test statistic that relates to the AMI.
Questions of feature importance can be asked at the level of an individual sample.
We show that estimators from the same AMI test can also be used to find important features in a particular instance.
We provide an example to show that perfect predictive models are insufficient for instance-wise feature selection.
We evaluate our method on several simulation experiments, on a genomic dataset, a clinical dataset for hospital readmission, and on a subset of classes in ImageNet.
Our method outperforms several baselines in various simulated datasets, is able to identify biologically significant genes, can select the most important predictors of a hospital readmission event, and is able to identify distinguishing features in an image-classification task.

assistant: text: A model agnostic method to provide interpretation on the influence of input features on the response of a machine level model down to instance level, and proper test statistics for model agnostic feature selection.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: High Fidelity Speech Synthesis with Adversarial Networks

Paper content:
Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images.
However, their application in the audio domain has received limited attention,
and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech.
To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech.
Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes.
The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced.
To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS.
We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator.
Listen to GAN-TTS reading this abstract at http://tiny.cc/gantts.

assistant: text: We introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech, which achieves Mean Opinion Score (MOS) 4.2.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: STRUCTURED ALIGNMENT NETWORKS

Paper content:
Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity.
Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence.
When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors.
We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures.
Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective.
We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset.
We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.

assistant: text: Matching sentences by learning the latent constituency tree structures with a variant of the inside-outside algorithm embedded as a neural network layer.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Break-Even Point on the Optimization Trajectories of Deep Neural Networks

Paper content:
Understanding the optimization trajectory is critical to understand training of deep neural networks.
We show how the hyperparameters of stochastic gradient descent influence the covariance of the gradients (K) and the Hessian of the training loss (H) along this trajectory.
Based on a theoretical model, we predict that using a high learning rate or a small batch size in the early phase of training leads SGD to regions of the parameter space with (1) reduced spectral norm of K, and (2) improved conditioning of K and H. We show that the point on the trajectory after which these effects hold, which we refer to as the break-even point, is reached early during training.
We demonstrate these effects empirically for a range of deep neural networks applied to multiple different tasks.
Finally, we apply our analysis to networks with batch normalization (BN) layers and find that it is necessary to use a high learning rate to achieve loss smoothing effects attributed previously to BN alone.

assistant: text: In the early phase of training of deep neural networks there exists a "break-even point" which determines properties of the entire optimization trajectory.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Understanding and Training Deep Diagonal Circulant Neural Networks

Paper content:
In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.
Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks.
Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.
We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.
We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.
Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

assistant: text: We train deep neural networks based on diagonal and circulant matrices, and show that this type of networks are both compact and accurate on real world applications.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Image-guided Neural Object Rendering

Paper content:
We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis.
The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts).
A core component of our work is the handling of view-dependent effects.
Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.
As input data we are using an RGB video of the object.
This video is used to reconstruct a proxy geometry of the object via multi-view stereo.
Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering.
This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts.
To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects.
Based on these estimations, we are able to convert observed images to diffuse images.
These diffuse images can be projected into other views.
In the target view, our pipeline reinserts the new view-dependent effects.
To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results.
Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images.
We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

assistant: text: We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis while considering view-dependent effects.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling Improves VAEs' Robustness to Adversarial Attacks

Paper content:
This paper is concerned with the robustness of VAEs to adversarial attacks.
We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as β-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)).
This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

assistant: text: The authors propose a new VAE model called seatbelt-VAE, showing to be more robust for latent attack than benchmarks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dimensional Reweighting Graph Convolution Networks

Paper content:
In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs.
We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.
However, practically, we find that the degrees DrGCNs help vary severely on different datasets.
We revisit the problem and develop a new measure K to quantify the effect.
This measure guides when we should use dimensional reweighting in GCNs and how much it can help.
Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs.
The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants.
Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches.
Significant improvements can also be observed on a large scale industrial dataset.

assistant: text: A method, known as DrGCN, for reweighting the different dimensions of the node representations in graph convolutional networks by reducing variance between dimensions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Batch-shaping for learning conditional channel gated networks

Paper content:
We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost.
This is achieved by gating the deep-learning architecture on a fine-grained-level.
Individual convolutional maps are turned on/off conditionally on features in the network.
To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner.
We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution.
We use this novel technique to force gates to be more conditional on the data.
We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation.
Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.
In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.
We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

assistant: text: A method to train a network with large capacity, only parts of which are used at inference time dependent on input, using fine-grained conditional selection and a new method of regularization, "batch shaping."
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods

Paper content:
The geometric properties of loss surfaces, such as the local flatness of a solution, are associated with generalization in deep learning.
The Hessian is often used to understand these geometric properties.
We investigate the differences between the eigenvalues of the neural network Hessian evaluated over the empirical dataset, the Empirical Hessian, and the eigenvalues of the Hessian under the data generating distribution, which we term the True Hessian.
Under mild assumptions, we use random matrix theory to show that the True Hessian has eigenvalues of smaller absolute value than the Empirical Hessian.
We support these results for different SGD schedules on both a 110-Layer ResNet and VGG-16.
To perform these experiments we propose a framework for spectral visualization, based on GPU accelerated stochastic Lanczos quadrature.
This approach is an order of magnitude faster than state-of-the-art methods for spectral visualization, and can be generically used to investigate the spectral properties of matrices in deep learning.

assistant: text: This paper uses the random matrix theory to study the spectrum distribution of the empirical Hessian and true Hessian for deep learning, and proposes an efficient spectrum visualization methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Tunability of Optimizers in Deep Learning

Paper content:
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.
In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?
We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.
Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

assistant: text: This paper introduces a simple measure of tunability that allows to compare optimizers under resource constraints, finding that tuning Adam optimizers' learning rate is easiest to find well-performing hyperparameter configurations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improved Adversarial Image Captioning

Paper content:
In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions.
We investigate the viability of two discrete GAN training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST.

assistant: text: Image captioning as a conditional GAN training with novel architectures, also study two discrete GAN training methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Probing Emergent Semantics in Predictive Agents via Question Answering

Paper content:
Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.
We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling – action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).
After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.
The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.
Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.
By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

assistant: text: Proposes QA as a tool to investigate what agents learn about in the world, arguing this as an intuitive method for humans which allows for arbitrary complexity.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Counting the Paths in Deep Neural Networks as a Performance Predictor

Paper content:
We propose a novel quantitative measure to predict the performance of a deep neural network classifier, where the measure is derived exclusively from the graph structure of the network.
We expect that this measure is a fundamental first step in developing a method to evaluate new network architectures and reduce the reliance on the computationally expensive trial and error or "brute force" optimisation processes involved in model selection.
The measure is derived in the context of multi-layer perceptrons (MLPs), but the definitions are shown to be useful also in the context of deep convolutional neural networks (CNN), where it is able to estimate and compare the relative performance of different types of neural networks, such as VGG, ResNet, and DenseNet.
Our measure is also used to study the effects of some important "hidden" hyper-parameters of the DenseNet architecture, such as number of layers, growth rate and the dimension of 1x1 convolutions in DenseNet-BC.
Ultimately, our measure facilitates the optimisation of the DenseNet design, which shows improved results compared to the baseline.

assistant: text: The paper proposes a novel quantity that counts the number of path in the neural network which is predictive of the performance of neural networks with the same number of parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generative Question Answering: Learning to Answer the Whole Question

Paper content:
Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.
We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.
Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answer—allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.
Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.
The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.
We will release our code.

assistant: text: The authors proposes a generative QA model, which optimizes jointly the distribution of questions and answering given a document/context.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: One-shot and few-shot learning of word embeddings

Paper content:
Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.
By contrast, humans have an incredible ability to do one-shot or few-shot learning.
For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.
Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.
This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.

assistant: text: We highlight a technique by which natural language processing systems can learn a new word from context, allowing them to be much more flexible.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Three Mechanisms of Weight Decay Regularization

Paper content:
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.
Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ.
We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures.
We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization.
Our results provide insight into how to improve the regularization of neural networks.

assistant: text: We investigate weight decay regularization for different optimizers and identify three distinct mechanisms by which weight decay improves generalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scaling up Deep Learning for PDE-based Models

Paper content:
Across numerous applications, forecasting relies on numerical solvers for partial differential equations (PDEs).
Although the use of deep-learning techniques has been proposed, the uses have been restricted by the fact the training data are obtained using PDE solvers.
Thereby, the uses were limited to domains, where the PDE solver was applicable, but no further.
We present methods for training on small domains, while applying the trained models on larger domains, with consistency constraints ensuring the solutions are physically meaningful even at the boundary of the small domains.
We demonstrate the results on an air-pollution forecasting model for Dublin, Ireland.

assistant: text: We present RNNs for training surrogate models of PDEs, wherein consistency constraints ensure the solutions are physically meaningful, even when the training uses much smaller domains than the trained model is applied to.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RL-ST: Reinforcing Style, Fluency and Content Preservation for Unsupervised Text Style Transfer

Paper content:
Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training.
Style transfer systems are evaluated on their ability to generate sentences that
1) possess the target style,
2) are fluent and natural sounding, and
3) preserve the non-stylistic parts (content) of the source sentence.
We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same.
Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer.
Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets.

assistant: text: The authors propose a combination reward composed of fluency, content, and style for text style transfer.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving One-Shot NAS By Suppressing The Posterior Fading

Paper content:
There is a growing interest in automated neural architecture search (NAS).
To improve the efficiency of NAS, previous approaches adopt  weight sharing method to force all models share the same set of weights.
However, it has been observed that a model performing better with shared weights does not necessarily perform  better when trained alone.
In this paper, we analyse existing weight sharing one-shot NAS approaches from a Bayesian point of view and identify the posterior fading problem, which compromises the effectiveness of shared weights.
To alleviate this problem, we present a practical approach to guide the parameter posterior towards its true distribution.
Moreover, a hard latency constraint is introduced during the search so that the desired latency can be achieved.
The resulted method, namely Posterior Convergent NAS (PC-NAS), achieves state-of-the-art performance under standard GPU latency constraint on ImageNet.
In our small search space, our model PC-NAS-S attains76.8% top-1 accuracy, 2.1% higher than MobileNetV2 (1.4x) with the same latency.
When adopted to our large search space, PC-NAS-L achieves 78.1% top-1 accuracy within 11ms.
The discovered architecture also transfers well to other computer vision applications such as object detection and person re-identification.

assistant: text: Our paper identifies the issue of existing weight sharing approach in neural architecture search and propose a practical method, achieving strong results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Information Plane Analysis of Deep Neural Networks via Matrix--Based Renyi's Entropy and Tensor Kernels

Paper content:
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.
However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.
For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.
MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.
None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.
In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.
The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.
Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

assistant: text: First comprehensive information plane analysis of large scale deep neural networks using matrix based entropy and tensor kernels.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: YellowFin and the Art of Momentum Tuning

Paper content:
Hyperparameter tuning is one of the most time-consuming workloads in deep learning.
State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.
Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.
Motivated by this trend, we ask: can simple adaptive methods, based on SGD perform as well or better?
We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.
We then analyze its robustness to learning rate misspecification and objective curvature variation.
Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD.
YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly.
We empirically show YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to $3.28$x in synchronous and up to $2.69$x in asynchronous settings.

assistant: text: Proposes a method to automatically tuning the momentum parameter in momentum SGD methods, which achieves better results and fast convergence speed that state-of-the-art Adam algorithm.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Better Knowledge Retention through Metric Learning

Paper content:
In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories.
While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes.
This makes deep neural nets ill-suited to continual learning.
In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced.
We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

assistant: text: We show metric learning can help reduce catastrophic forgetting
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

Paper content:
Image-to-image translation has recently received significant attention due to advances in deep learning.
Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way.
However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations.
To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.
We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain.
Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain.
To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels.
Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

assistant: text: We propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain.