ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Probing Emergent Semantics in Predictive Agents via Question Answering

Paper content:
Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.
We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling – action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).
After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.
The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.
Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.
By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

assistant: text: The authors propose a framework to assess representations built by predictive models that contain sufficient information to answer questions about the environment they are trained on, showing those by SimCore contained sufficient information for the LSTM to answer questions accurately.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: META LEARNING SHARED HIERARCHIES

Paper content:
We develop a metalearning approach for learning hierarchically structured poli- cies, improving sample efficiency on unseen tasks through the use of shared primitives—policies that are executed for large numbers of timesteps.
Specifi- cally, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies.
We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks.
We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies.
We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes.
We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

assistant: text: This paper proposes a novel method for inducing temporal hierarchical structure in a specialized multi-task setting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning the Arrow of Time for Problems in Reinforcement Learning

Paper content:
We humans have an innate understanding of the asymmetric progression of time, which we use to efficiently and safely perceive and manipulate our environment.
Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.
We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.
Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).
Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer and Otto (1998).

assistant: text: We learn the arrow of time for MDPs and use it to measure reachability, detect side-effects and obtain a curiosity reward signal.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Langevin Dynamics as Nonparametric Variational Inference

Paper content:
Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased.
In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think.
In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to a variational inference procedure based on optimizing a nonparametric normalizing flow.
This result suggests that the transient bias of LD (due to too few warmup steps) may track that of VI (due to too few optimization steps), up to differences due to VI’s parameterization and asymptotic bias.
Empirically, we find that the transient biases of these algorithms (and momentum-accelerated versions) do evolve similarly.
This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if it’s far from burned in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).

assistant: text: The transient behavior of gradient-based MCMC and variational inference algorithms is more similar than one might think, calling into question the claim that variational inference is faster than MCMC.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Novelty Detection Via Blurring

Paper content:
Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) are known to assign lower uncertainty to the OOD data than the target distribution.
In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images.
Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training.
Our detector is simple, efficient in test time, and outperforms baseline OOD detectors in various domains.
Further results show that SVD-RND learns a better target distribution representation than the baselines.
Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy in CelebA domain.

assistant: text: We propose a novel OOD detector that employ blurred images as adversarial examples . Our model achieve significant OOD detection performance in various domains.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Perturbation Analysis of Input Transformations for Adversarial Attacks

Paper content:
The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today.
Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction.
While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature.
This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

assistant: text: The paper studies the robustness of adversarial attacks to transformations of their input.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Accelerating Convolutional Neural Networks using Iterative Two-Pass Decomposition

Paper content:
We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).
The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.
Our two-pass CP-decomposition helps prevent from the instability problem.
The iterative flow makes the decomposition of the deeper networks systematic.
The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.

assistant: text: The paper proposes a novel workflow for acceleration and compression of CNNs and also proposes a way to determine the target rank of each layer given the target overall acceleration.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Paper content:
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective.
We demonstrate a very universal Frequency Principle (F-Principle) --- DNNs often fit target functions from low to high frequencies --- on high-dimensional benchmark datasets, such as MNIST/CIFAR10, and deep networks, such as VGG16.
This F-Principle of DNNs is opposite to the learning behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibits faster convergence for higher frequencies, for various scientific computing problems.
With a naive theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions.
The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function.
This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

assistant: text: In real problems, we found that DNNs often fit target functions from low to high frequencies during the training process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Retrieving Signals with Deep Complex Extractors

Paper content:
Recent advances have made it possible to create deep complex-valued neural networks.
Despite this progress, many challenging learning tasks have yet to leverage the power of complex representations.
Building on recent advances, we propose a new deep complex-valued method for signal retrieval and extraction in the frequency domain.
As a case study, we perform audio source separation in the Fourier domain.
Our new method takes advantage of the convolution theorem which states that the Fourier transform of two convolved signals is the elementwise product of their Fourier transforms.
Our novel method is based on a complex-valued version of Feature-Wise Linear Modulation (FiLM) and serves as the keystone of our proposed signal extraction method.
We also introduce a new and explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram.
Using the Wall Street Journal Dataset, we compared our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss.
In this work, we introduced a new complex-valued framework for signal retrieval and signal separation in the Fourier domain.
As a case sudy, we considered audio source separation.
We proposed a new masking method based on a complex-valued version of the Feature-wise Linear Modulation (FiLM) model, allowing to perform local ensembling and yielding a beneficial regularization effect.
We also proposed a new phase-aware loss taking, explicitly, into account the magnitude and phase of the reference and estimated signals.
In our study, phase proved to be an important factor that should be taken into account in order to improve the quality of the separation in terms of SDR.
The phase-aware loss improves over other frequency and time-domain losses.
Our deep separator draws its power from the compelling properties of complex-valued neural networks and the proposed masking method.
Our finding might shed light on the deep complex-valued neural networks' tendency to solve challenging tasks where the data lie in the complex space and where it could be represented in the frequency domain.
We view these results as an opportunity to pursue a more systematic investigation of the underpinning of complex-valued representation success.
We believe that our proposed method could lead to new research directions where signal retrieval is needed.
6 Appendix

assistant: text: New Signal Extraction Method in the Fourier Domain
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Graph-to-Graph Translation for Molecules

Paper content:
The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties.
Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization.
In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph.
Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule.
We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

assistant: text: A hierarchical graph-to-graph translation model to generate molecular graphs using chemical substructures as building blocks that is fully autoregressive and learns coherent multi-resolution representations, outperforming previous models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Relational Multi-Instance Learning for Concept Annotation from Medical Time Series

Paper content:
Recent advances in computing technology and sensor design have made it easier to collect longitudinal or time series data from patients, resulting in a gigantic amount of available medical data.
Most of the medical time series lack annotations or even when the annotations are available they could be subjective and prone to human errors.
Earlier works have developed natural language processing techniques to extract concept annotations and/or clinical narratives from doctor notes.
However, these approaches are slow and do not use the accompanying medical time series data.
To address this issue, we introduce the problem of concept annotation for the medical time series data, i.e., the task of predicting and localizing medical concepts by using the time series data as input.
We propose Relational Multi-Instance Learning (RMIL) - a deep Multi Instance Learning framework based on recurrent neural networks, which uses pooling functions and attention mechanisms for the concept annotation tasks.
Empirical results on medical datasets show that our proposed models outperform various multi-instance learning models.

assistant: text: The paper addresses the classification of medical time-series data and proposes to model the temporal relationship between the instances in each series using a recurrent neural network architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A novel text representation which enables image classifiers to perform text classification

Paper content:
We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems.
We apply the technique to entity disambiguation of inventor names in US patents.
The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation.
We then train an image classification neural network to discriminate between such pairwise comparison images, and use the trained network to label each pair of records as either matched (same inventor) or non-matched (different inventors), obtaining highly accurate results (F1: 99.09%, precision: 99.41%, recall: 98.76%).
Our new text-to-image representation method could potentially be used more broadly for other NLP comparison problems, such as disambiguation of academic publications, or for problems that require simultaneous classification of both text and images.

assistant: text: The authors consider the problem of names disambiguisation for patent names inventors and propose to build an image page representation of the two name strings to compare and to apply an image classifier.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models

Paper content:
In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator.
We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task.
We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training.
We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer.
Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review''.
We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated.
Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

assistant: text: This paper analyzes the forgetting problem in the pretraining-finetuning framework from the perspective of context sensitivity and knowledge transfer, and proposes a fine-tuning strategy which outperforms the weight decay method.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

Paper content:
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients.
Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs).
We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices.
This is done by parametrizing with a skew-symmetric matrix using the Cayley transform.
Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones.
The proposed training scheme involves a straightforward gradient calculation and update step.
In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

assistant: text: This paper suggests an RNN reparametrization of the recurrent weights with a skew-symmetric matrix using Cayley transform to keep the recurrent weight matrix orthogonal.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Out-of-distribution Detection in Few-shot Classification

Paper content:
In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.
Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.
While both few-shot classification and out-of-distribution detection are popular topics,
their combination has not been studied.
In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.
Then, we propose two new methods for this task and investigate their performance.
In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

assistant: text: The paper proposes two new confidence scores which are more suitable for out-of-distribution detection of few-shot classification and shows that a distance metric-based approach improves performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: {COMPANYNAME}11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery

Paper content:
We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.
Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.
To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.
We provide a set of baselines for different feature extractors that can be built upon.
Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

assistant: text: This paper describes a large-scale ECG dataset the authors intend to publish and provides unsupervised analysis and visualization of the dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Revisiting Auxiliary Latent Variables in Generative Models

Paper content:
Extending models with auxiliary latent variables is a well-known technique to in-crease model expressivity.
Bachman & Precup (2015); Naesseth et al. (2018); Cremer et al. (2017); Domke & Sheldon (2018) show that Importance Weighted Autoencoders (IWAE) (Burda et al., 2015) can be viewed as extending the variational family with auxiliary latent variables.
Similarly, we show that this view encompasses many of the recent developments in variational bounds (Maddisonet al., 2017; Naesseth et al., 2018; Le et al., 2017; Yin & Zhou, 2018; Molchanovet al., 2018; Sobolev & Vetrov, 2018).
The success of enriching the variational family with auxiliary latent variables motivates applying the same techniques to the generative model.
We develop a generative model analogous to the IWAE bound and empirically show that it outperforms the recently proposed Learned Accept/Reject Sampling algorithm (Bauer & Mnih, 2018), while being substantially easier to implement.
Furthermore, we show that this generative process provides new insights on ranking Noise Contrastive Estimation (Jozefowicz et al.,2016; Ma & Collins, 2018) and Contrastive Predictive Coding (Oord et al., 2018).

assistant: text: Monte Carlo Objectives are analyzed using auxiliary variable variational inference, yielding a new analysis of CPC and NCE as well as a new generative model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Structured Neural Summarization

Paper content:
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input.
Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text.
In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.

assistant: text: This paper presents a structural summarization model with a graph-based encoder extended from RNN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks

Paper content:
Variational Bayesian Inference is a popular methodology for approximating posterior distributions over Bayesian neural network weights.
Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance.
In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization.
For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence.
This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance.
Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.

assistant: text: Mean field VB uses twice as many parameters; we tie variance parameters in mean field VB without any loss in ELBO, gaining speed and lower variance gradients.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring  for Minimax Problems and GANs

Paper content:
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.
First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.
Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.
Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

assistant: text: Convergence proof of stochastic sub-gradients method and variations on convex-concave minimax problems
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Singular Values of Convolutional Layers

Paper content:
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.
This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball.
We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

assistant: text: The paper is dedicated to computation of singular values of convolutional layers
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Graph Matching Consensus

Paper content:
This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs.
First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes.
Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs.
We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process.
Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently.
We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

assistant: text: We develop a deep graph matching architecture which refines initial correspondences in order to reach neighborhood consensus.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Modern Take on the Bias-Variance Tradeoff in Neural Networks

Paper content:
Recent empirical results on over-parameterized deep networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks.
Researchers are actively working on bridging this discrepancy by proposing better complexity measures.
Instead, we directly measure prediction bias and variance for four classification and regression tasks on modern deep networks.
We find that both bias and variance can decrease as the number of parameters grows.
Qualitatively, the phenomenon persists over a number of gradient-based optimizers.
To better understand the role of optimization, we decompose the total variance into variance due to training set sampling and variance due to initialization.
Variance due to initialization is significant in the under-parameterized regime.
In the over-parameterized regime, total variance is much lower and dominated by variance due to sampling.
We provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

assistant: text: We provide evidence against classical claims about the bias-variance tradeoff and propose a novel decomposition for variance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Locality and Compositionality in Zero-Shot Learning

Paper content:
In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL).
In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.
The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

assistant: text: An analysis of the effects of compositionality and locality on representation learning for zero-shot learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference

Paper content:
Computations for the softmax function in neural network models are expensive when the number of output classes is large.
This can become a significant issue in both training and inference for such models.
In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.
During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.
Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.
During inference, our method quickly locates the most probable expert to compute small-scale softmax.
Our method is learning-based and requires no knowledge of the output class partition space a priori.
We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of performance.

assistant: text: This paper proposes a fast approximation to the softmax computation when the number of classes is very large.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling Style and Content in Anime Illustrations

Paper content:
Existing methods for AI-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists.
We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular.
Training such model is challenging, since given a style, various content data may exist but not the other way round.
Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based on a dual-conditional generator.
We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over a thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer.
We show this unique capability as well as superior output to the current state-of-the-art.

assistant: text: An adversarial training-based method for disentangling two complementary sets of variations in a dataset where only one of them is labelled, tested on style vs. content in anime illustrations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively

Paper content:
The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training.
On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.
It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.
In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models.
This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition.
Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy.
Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them.
We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.

assistant: text: We present an efficient and adaptive framework for comparing image classifiers to maximize the discrepancies between the classifiers, in place of comparing on fixed test sets.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

Paper content:
While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages.
We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language.
XLDA enhances performance of all 14 tested languages of the cross-lingual natural language inference (XNLI) benchmark.
With improvements of up to 4.8, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu.
XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language.
On the SQuAD question answering task, we see that XLDA provides a 1.0 performance increase on the English evaluation set.
Comprehensive experiments suggest that most languages are effective as cross-lingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

assistant: text: This paper proposes to augment crosslingual data with heuristic swaps using aligned translations, like bilingual humans do in code-switching.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing experimental signals in single-cell RNA-sequencing data using graph signal processing

Paper content:
Single-cell RNA-sequencing (scRNA-seq) is a powerful tool for analyzing biological systems.
However, due to biological and technical noise, quantifying the effects of multiple experimental conditions presents an analytical challenge.
To overcome this challenge, we developed MELD: Manifold Enhancement of Latent Dimensions.
MELD leverages tools from graph signal processing to learn a latent dimension within the data scoring the prototypicality of each datapoint with respect to experimental or control conditions.
We call this dimension the Enhanced Experimental Signal (EES).
MELD learns the EES by filtering the noisy categorical experimental label in the graph frequency domain to recover a smooth signal with continuous values.
This method can be used to identify signature genes that vary between conditions and identify which cell types are most affected by a given perturbation.
We demonstrate the advantages of MELD analysis in two biological datasets, including T-cell activation in response to antibody-coated beads and treatment of human pancreatic islet cells with interferon gamma.

assistant: text: This paper introduces several methods to process experimental results on biological cells and proposes a MELD algorithm mapping hard group assignments to soft assignments, allowing relevant groups of cells to be clustered.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compression of Deep Neural Networks by combining pruning and low rank decomposition

Paper content:
Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud.
Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.
In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing.
We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.
We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet.
Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.

assistant: text: This paper proposes combining Tucker Decomposition with Filter pruning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Building Hierarchical Interpretations in Natural Language via Feature Interaction Detection

Paper content:
The interpretability of neural networks has become crucial for their applications in real world with respect to the reliability and trustworthiness.
Existing explanation generation methods usually provide important features by scoring their individual contributions to the model prediction and ignore the interactions between features, which eventually provide a bag-of-words representation as explanation.
In natural language processing, this type of explanations is challenging for human user to understand the meaning of an explanation and draw the connection between explanation and model prediction, especially for long texts.
In this work, we focus on detecting the interactions between features, and propose a novel approach to build a hierarchy of explanations based on feature interactions.
The proposed method is evaluated with three neural classifiers, LSTM, CNN, and BERT, on two benchmark text classification datasets.
The generated explanations are assessed by both automatic evaluation measurements and human evaluators.
Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models, and understandable to humans.

assistant: text: A novel method for providing explanations for predicitions made by text classifiers that outperforms baselines on word level importance scores, and a new metric, cohesion loss, to evaluate span-level importance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

Paper content:
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner.
Our scheme pits two versions of the same agent, Alice and Bob, against one another.
Alice proposes a task for Bob to complete; and then Bob attempts to complete the task.
In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset.
Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively.
Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent.
When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.

assistant: text: A self-play model where one agent learns to propose tasks that are easy for them but difficult for an opponent, creating a moving target of self-play objectives and learning curriculum.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ternary MobileNets via Per-Layer Hybrid Filter Banks

Paper content:
MobileNets family of computer vision neural networks have fueled tremendous progress in the design and organization of resource-efficient architectures in recent years.
New applications with stringent real-time requirements in highly constrained devices require further compression of MobileNets-like already computeefficient networks.
Model quantization is a widely used technique to compress and accelerate neural network inference and prior works have quantized MobileNets to 4 − 6 bits albeit with a modest to significant drop in accuracy.
While quantization to sub-byte values (i.e. precision ≤ 8 bits) has been valuable, even further quantization of MobileNets to binary or ternary values is necessary to realize significant energy savings and possibly runtime speedups on specialized hardware, such as ASICs and FPGAs.
Under the key observation that convolutional filters at each layer of a deep neural network may respond differently to ternary quantization, we propose a novel quantization method that generates per-layer hybrid filter banks consisting of full-precision and ternary weight filters for MobileNets.
The layer-wise hybrid filter banks essentially combine the strengths of full-precision and ternary weight filters to derive a compact, energy-efficient architecture for MobileNets.
Using this proposed quantization method, we quantized a substantial portion of weight filters of MobileNets to ternary values resulting in 27.98% savings in energy, and a 51.07% reduction in the model size, while achieving comparable accuracy and no degradation in throughput on specialized hardware in comparison to the baseline full-precision MobileNets.

assistant: text: The paper proposes layer-wise hybrid filter bank which only quantizes a fraction of convolutional filters to ternary values towards the MobileNets architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion

Paper content:
Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details.
Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views.
This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery.
To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion:
(i) co-registration,
(ii) fusion,
(iii) up-sampling, and
(iv) registration-at-the-loss.
Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism.
We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs.
We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet.
We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale.
Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

assistant: text: This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image.