ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fully differentiable full-atom protein backbone generation

Paper content:
The fast generation and refinement of protein backbones would constitute a major advancement to current methodology for the design and development of de novo proteins.
In this study, we train Generative Adversarial Networks (GANs) to generate fixed-length full-atom protein backbones, with the goal of sampling from the distribution of realistic 3-D backbone fragments.
We represent protein structures by pairwise distances between all backbone atoms, and present a method for directly recovering and refining the corresponding backbone coordinates in a differentiable manner.
We show that interpolations in the latent space of the generator correspond to smooth deformations of the output backbones, and that test set structures not seen by the generator during training exist in its image.
Finally, we perform sequence design, relaxation, and ab initio folding of a subset of generated structures, and show that in some cases we can recover the generated folds after forward-folding.
Together, these results suggest a mechanism for fast protein structure refinement and folding using external energy functions.

assistant: text: We train a GAN to generate and recover full-atom protein backbones , and we show that in select cases we can recover the generated proteins after sequence design and ab initio forward-folding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised  one-to-many image translation

Paper content:
We perform completely unsupervised one-sided image to image translation between a source domain $X$ and a target domain $Y$ such that we preserve relevant underlying shared semantics (e.g., class, size, shape, etc).
In particular, we are interested in a more difficult case than those typically addressed in the literature, where the source and target are ``far" enough that reconstruction-style or pixel-wise approaches fail.
We argue that transferring (i.e., \emph{translating}) said relevant information should involve both discarding source domain-specific information while incorporate target domain-specific information, the latter of which we model with a noisy prior distribution.
In order to avoid the degenerate case where the generated samples are only explained by the prior distribution, we propose to minimize an estimate of the mutual information between the generated sample and the sample from the prior distribution.
We discover that the architectural choices are an important factor to consider in order to preserve the shared semantic between $X$ and $Y$.
We show state of the art results on the MNIST to SVHN task for unsupervised image to image translation.

assistant: text: This paper formalizes the problem of unsupervised translation and proposes an augmented GAN framework which uses the mutual information to avoid the degenerate case
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to solve the credit assignment problem

Paper content:
Backpropagation is driving today's artificial neural networks (ANNs).
However, despite extensive research, it remains unclear if the brain implements this algorithm.
Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient.
However, the convergence rate of such learning scales poorly with the number of involved neurons.
Here we propose a hybrid learning approach.
Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.
We provide proof that our approach converges to the true gradient for certain classes of networks.
In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning.
Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

assistant: text: Perturbations can be used to train feedback weights to learn in fully connected and convolutional neural networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PDE-Net: Learning PDEs from Data

Paper content:
Partial differential equations (PDEs)  play a prominent role in many disciplines such as applied mathematics, physics, chemistry, material science, computer science, etc.
PDEs are commonly derived based on physical laws or empirical observations.
However, the governing equations for many complex systems in modern applications are still not fully known.
With the rapid development of sensors, computational power, and data storage in the past decade, huge quantities of data can be easily collected and efficiently stored.
Such vast quantity of data offers new opportunities for data-driven discovery of hidden physical laws.
Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulfill two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models.
The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (filters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses.
Comparing with existing approaches, which either assume the form of the nonlinear response is known or fix certain finite difference approximations of differential operators, our approach has the most flexibility by learning both differential operators and the nonlinear responses.
A special feature of the proposed PDE-Net is that all filters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network.
These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of filters (an important concept originated from wavelet theory).
We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet).
Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

assistant: text: This paper proposes a new feed-forward network, call PDE-Net, to learn PDEs from data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

Paper content:
While natural language processing systems often focus on a single language, multilingual transfer learning has the potential to improve performance, especially for low-resource languages.
We introduce XLDA, cross-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language.
XLDA enhances performance of all 14 tested languages of the cross-lingual natural language inference (XNLI) benchmark.
With improvements of up to 4.8, training with XLDA achieves state-of-the-art performance for Greek, Turkish, and Urdu.
XLDA is in contrast to, and performs markedly better than, a more naive approach that aggregates examples in various languages in a way that each example is solely in one language.
On the SQuAD question answering task, we see that XLDA provides a 1.0 performance increase on the English evaluation set.
Comprehensive experiments suggest that most languages are effective as cross-lingual augmentors, that XLDA is robust to a wide range of translation quality, and that XLDA is even more effective for randomly initialized models than for pretrained models.

assistant: text: Translating portions of the input during training can improve cross-lingual performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fluctuation-dissipation relations for stochastic gradient descent

Paper content:
The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.
In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.
Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.
These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.
We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity.
Our claims are empirically verified.

assistant: text: Paper's concepts work in the discrete-time formalism, use the master equation, and  remove reliance on a locally quadratic approximation of the loss function or on any Gaussian asumptions of the SGD noise.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Estimating Heterogeneous Treatment Effects Using Neural Networks With The Y-Learner

Paper content:
We develop the Y-learner for estimating heterogeneous treatment effects in experimental and observational studies.
The Y-learner is designed to leverage the abilities of neural networks to optimize multiple objectives and continually update, which allows for better pooling of underlying feature information between treatment and control groups.
We evaluate the Y-learner on three test problems: (1) A set of six simulated data benchmarks from the literature.
(2) A real-world large-scale experiment on voter persuasion.
(3) A task from the literature that estimates artificially generated treatment effects on MNIST didgits.
The Y-learner achieves state of the art results on two of the three tasks.
On the MNIST task, it gets the second best results.

assistant: text: We develop a CATE estimation strategy that takes advantage some of the intriguing properties of neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generalizing Across Domains via Cross-Gradient Training

Paper content:
We present CROSSGRAD , a method to use multi-domain training data to learn a classifier that generalizes to new domains.
CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain.
Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training.
In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains.
We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances.
CROSSGRAD jointly trains a label and a domain classifier on examples perturbed by loss gradients of each other’s objectives.
This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions.
Empirical evaluation on three different applications where this setting is natural establishes that
(1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and
(2) data augmentation is a more stable and accurate method than domain adversarial training.

assistant: text: Domain guided augmentation of data provides a robust and stable method of domain generalization
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Continuous Meta-Learning without Tasks

Paper content:
Meta-learning is a promising strategy for learning to efficiently learn within new tasks, using data gathered from a distribution of tasks.
However, the meta-learning literature thus far has focused on the task segmented setting, where at train-time, offline data is assumed to be split according to the underlying task, and at test-time, the algorithms are optimized to learn in a single task.
In this work, we enable the application of generic meta-learning algorithms to settings where this task segmentation is unavailable, such as continual online learning with a time-varying task.
We present meta-learning via online changepoint analysis (MOCA), an approach which augments a meta-learning algorithm with a differentiable Bayesian changepoint detection scheme.
The framework allows both training and testing directly on time series data without segmenting it into discrete tasks.
We demonstrate the utility of this approach on a nonlinear meta-regression benchmark as well as two meta-image-classification benchmarks.

assistant: text: Bayesian changepoint detection enables meta-learning directly from time series data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators

Paper content:
Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain.
We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.
The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning.
This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time.
Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way.
This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well.
Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains.
They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

assistant: text: We address multi-domain few-shot classification by building multiple models to represent this complex task distribution in a collective way and simplifying task-specific adaptation as a selection problem from these pre-trained models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradients as Features for Deep Representation Learning

Paper content:
We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.
Specifically, we propose to explore gradient-based features.
These features are gradients of the model parameters with respect to a task-specific loss given an input sample.
Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network.
We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight.
Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients.
Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures.
We demonstrate strong results in all settings.
And our results are well-aligned with our theoretical insight.

assistant: text: Given a pre-trained model, we explored the per-sample gradients of the model parameters relative to a task-specific loss, and constructed a linear model that combines gradients of model parameters and the activation of the model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML

Paper content:
An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning.
An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks.
Despite MAML's popularity, a fundamental open question remains -- is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse,  with the meta initialization already containing high quality features?
We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor.
This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of the underlying neural network.
ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML.
We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm).
We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.

assistant: text: The success of MAML relies on feature reuse from the meta-initialization, which also yields a natural simplification of the algorithm, with the inner loop removed for the network body, as well as other insights on the head and body.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Block-Sparse Recurrent Neural Networks

Paper content:
Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling.
Sparsity is a technique to reduce compute and memory requirements of deep learning models.
Sparse RNNs are easier to deploy on devices and high-end server processors.
Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms.
In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization with pruning to create blocks of weights with zeros.
Using these techniques, we can create block-sparse RNNs with sparsity ranging from 80% to 90% with a small loss in accuracy.
This technique allows us to reduce the model size by roughly 10x.
Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count.
Our technique works with a variety of block sizes up to 32x32.
Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

assistant: text: The authors propose a block sparsity pruning approach to compress RNNs, using group LASSO to promote sparsity and to prune, but with a very specialized schedule as to the pruning and pruning weight.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue

Paper content:
Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge.
As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter.
The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge.
Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation.
We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks.
We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe et al., 2018).

assistant: text: A sequential latent variable model for knowledge selection in dialogue generation that extends the posterior attention model to the latent knowledge selection problem and achieves higher performances than previous state-of-the-art models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ridge Regression: Structure, Cross-Validation, and Sketching

Paper content:
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator?
(2) how to correctly use cross-validation to choose the regularization parameter?
and (3) how to accelerate computation without losing too much accuracy?
We consider the three problems in a unified large-data linear model.
We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise.
We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.
We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.
Our results are illustrated by simulations and by analyzing empirical data.
Ridge or 2 -regularized regression is a widely used method for prediction and estimation when the data dimension p is large compared to the number of datapoints n.
This is especially so in problems with many good features, where sparsity assumptions may not be justified.
A great deal is known about ridge regression.
It is Bayes optimal for any quadratic loss in a Bayesian linear model where the parameters and noise are Gaussian.
The asymptotic properties of ridge have been widely studied (e.g., Tulino & Verdú, 2004; Serdobolskii, 2007; Couillet & Debbah, 2011; Dicker, 2016; Dobriban & Wager, 2018, etc) .
For choosing the regularization parameter in practice, cross-validation (CV) is widely used.
In addition, there is an exact shortcut (e.g., Hastie et al., 2009, p. 243) , which has good consistency properties (Hastie et al., 2019) .
There is also a lot of work on fast approximate algorithms for ridge, e.g., using sketching methods (e.g., el Alaoui & Mahoney, 2015; Chen et al., 2015; Wang et al., 2018; Chowdhury et al., 2018, among

assistant: text: We study the structure of ridge regression in a high-dimensional asymptotic framework, and get insights about cross-validation and sketching.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: One-shot and few-shot learning of word embeddings

Paper content:
Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily.
By contrast, humans have an incredible ability to do one-shot or few-shot learning.
For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us.
Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data.
This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.

assistant: text: A technique for exploiting prior knowledge to learn embedding representations for new words with minimal data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Relational State-Space Model for Stochastic Multi-Object Systems

Paper content:
Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other.
Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents.
This paper introduces the relational state-space model (R-SSM), a sequential hierarchical latent variable model that makes use of graph neural networks (GNNs) to simulate the joint state transitions of multiple correlated objects.
By letting GNNs cooperate with SSM, R-SSM provides a flexible way to incorporate relational information into the modeling of multi-object dynamics.
We further suggest augmenting the model with normalizing flows instantiated for vertex-indexed random variables and propose two auxiliary contrastive objectives to facilitate the learning.
The utility of R-SSM is empirically evaluated on synthetic and real time series datasets.

assistant: text: A hierarchical latent variable model of sequential dynamic processes of multiple objects when each object exhibits significant stochasticity.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeepSphere: a graph-based spherical CNN

Paper content:
Designing a convolution for a spherical neural network requires a delicate tradeoff between efficiency and rotation equivariance.
DeepSphere, a method based on a graph representation of the discretized sphere, strikes a controllable balance between these two desiderata.
This contribution is twofold.
First, we study both theoretically and empirically how equivariance is affected by the underlying graph with respect to the number of pixels and neighbors.
Second, we evaluate DeepSphere on relevant problems.
Experiments show state-of-the-art performance and demonstrates the efficiency and flexibility of this formulation.
Perhaps surprisingly, comparison with previous work suggests that anisotropic filters might be an unnecessary price to pay.

assistant: text: The authors use the existing graph CNN formulation and a pooling strategy that exploits hierarchical pixelations of the sphere to learn from the discretized sphere.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Alexandria: Unsupervised High-Precision Knowledge Base Construction using a Probabilistic Program

Paper content:
Creating a knowledge base that is accurate, up-to-date and complete remains a significant challenge despite substantial efforts in automated knowledge base construction.
In this paper, we present Alexandria -- a system for unsupervised, high-precision knowledge base construction.
Alexandria uses a probabilistic program to define a process of converting knowledge base facts into unstructured text.
Using probabilistic inference, we can invert this program and so retrieve facts, schemas and entities from web text.
The use of a probabilistic program allows uncertainty in the text to be propagated through to the retrieved facts, which increases accuracy and helps merge facts from multiple sources.
Because Alexandria does not require labelled training data, knowledge bases can be constructed with the minimum of manual input.
We demonstrate this by constructing a high precision (typically 97\%+) knowledge base for people from a single seed fact.

assistant: text: Overview about existing knowledge base that is constructed with a probabilistic model, with the knowledge base construction approach evaluated against other knowledge base approaches YAGO2, NELL, Knowledge Vault, and DeepDive.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Continuous-fidelity Bayesian Optimization with Knowledge Gradient

Paper content:
While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multi-fidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations.
In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations.
cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost.
Furthermore, cfKG can be generalized, following Wu et al. (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously.
Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.

assistant: text: Studies hyperparameter-optimization by Bayesian optimization, using the Knowledge Gradient framework and allowing the Bayesian optimizer to tune fidelity against cost.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Singular Values of Convolutional Layers

Paper content:
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.
This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball.
We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

assistant: text: We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: State-Denoised Recurrent Neural Networks

Paper content:
Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise.
We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance.
Attractor dynamics are incorporated into the hidden state to `clean up' representations at each step of a sequence.
The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states.
This state-denoised recurrent neural network (SDRNN) performs multiple steps of internal processing for each external sequence step.
On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss.
We argue that attractor dynamics---and corresponding connectivity constraints---are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.

assistant: text: We propose a mechanism for denoising the internal state of an RNN to improve generalization performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ISA-VAE: Independent Subspace Analysis with Variational Autoencoders

Paper content:
Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way.
These methods have focussed largely on modifying the variational cost function to achieve this goal.
However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components.
In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered.
Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space.
This means there is no pressure to identify each true factor of variation with a latent variable.
We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.
Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement.
The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

assistant: text: We present structured priors for unsupervised learning of disentangled representations in VAEs that significantly mitigate the trade-off between disentanglement and reconstruction loss.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scheduling with Complex Consumptive Resources for a Planetary Rover

Paper content:
Generating and scheduling activities is particularly challenging
when considering both consumptive resources and
complex resource interactions such as time-dependent resource
usage.We present three methods of determining valid
temporal placement intervals for an activity in a temporally
grounded plan in the presence of such constraints.
We introduce
the Max Duration and Probe algorithms which are
sound, but incomplete, and the Linear algorithm which is
sound and complete for linear rate resource consumption.
We apply these techniques to the problem of scheduling
awakes for a planetary rover where the awake durations
are affected by existing activities.
We demonstrate how the
Probe algorithm performs competitively with the Linear algorithm
given an advantageous problem space and well-defined
heuristics.
We show that the Probe and Linear algorithms
outperform the Max Duration algorithm empirically.
We then empirically present the runtime differences between
the three algorithms.
The Probe algorithm is currently base-lined
for use in the onboard scheduler for NASA’s next planetary
rover, the Mars 2020 rover.

assistant: text: This paper describes and analyzes three methods to schedule non-fixed duration activities in the presence of consumptive resources.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Novelty Detection Via Blurring

Paper content:
Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) are known to assign lower uncertainty to the OOD data than the target distribution.
In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images.
Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training.
Our detector is simple, efficient in test time, and outperforms baseline OOD detectors in various domains.
Further results show that SVD-RND learns a better target distribution representation than the baselines.
Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy in CelebA domain.

assistant: text: This paper presents the idea to use blurred images as regularizing examples to improve out-of-distribution detection performance based on Random Network Distillation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Program Guided Agent

Paper content:
Developing agents that can learn to follow natural language instructions has been an emerging research direction.
While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans.
To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks.
We then devise a modular framework that learns to perform a task specified by a program – as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task.
Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy.
We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

assistant: text: We propose a modular framework that can accomplish tasks specified by programs and achieve zero-shot generalization to more complex tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Optimization of Sorting Networks via Continuous Relaxations

Paper content:
Sorting input objects is an important step in many machine learning pipelines.
However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization.
In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting operator from permutation matrices to the set of unimodal row-stochastic matrices, where every row sums to one and has a distinct argmax.
This relaxation permits straight-through optimization of any computational graph involve a sorting operation.
Further, we use this relaxation to enable gradient-based stochastic optimization over the combinatorially large space of permutations by deriving a reparameterized gradient estimator for the Plackett-Luce family of distributions over permutations.
We demonstrate the usefulness of our framework on three tasks that require learning semantic orderings of high-dimensional objects, including a fully differentiable, parameterized extension of the k-nearest neighbors algorithm

assistant: text: Introduces a continuous relaxation of the sorting operator in order to construct an end-to-end gradient-based optimization and introduces a stochastic extension of its method using Placket-Luce distributions and Monte Carlo.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep connectomics networks: Results from neural network architectures inspired from network neuroscience

Paper content:
Claims from the fields of network neuroscience and connectomics suggest that topological models of the brain involving complex networks are of particular use and interest.
The field of deep neural networks has mostly left inspiration from these claims out.
In this paper, we propose three architectures and use each of them to explore the intersection of network neuroscience and deep learning in an attempt to bridge the gap between the two fields.
Using the teachings from network neuroscience and connectomics, we show improvements over the ResNet architecture, we show a possible connection between early training and the spectral properties of the network, and we show the trainability of a DNN based on the neuronal network of C.Elegans.

assistant: text: We explore the intersection of network neurosciences and deep learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Paper content:
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.
A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment.
Computing a Bayes-optimal policy is however intractable for all but the smallest tasks.
In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection.
In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty.
We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

assistant: text: This paper presents a new deep reinforcement learning method that can efficiently trade-off exploration and exploitation that combines meta-learning, variational inference, and bayesian RL.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Paper content:
One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth.
This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks.
For an $m$ hidden node shallow neural network with ReLU activation and $n$ training data, we show as long as $m$ is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function.
Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum.
We believe these insights are also useful in analyzing deep models and other first order methods.

assistant: text: We prove gradient descent achieves zero training loss with a linear rate on over-parameterized neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A General Logic-based Approach for Explanation Generation

Paper content:
In an explanation generation problem, an agent needs to identify and explain the reasons for its decisions to another agent.
Existing work in this area is mostly confined to planning-based systems that use automated planning approaches to solve the problem.
In this paper, we approach this problem from a new perspective, where we propose a general logic-based framework for explanation generation.
In particular, given a knowledge base $KB_1$ that entails a formula $\phi$ and a second knowledge base $KB_2$ that does not entail $\phi$, we seek to identify an explanation $\epsilon$ that is a subset of $KB_1$ such that the union of $KB_2$ and $\epsilon$ entails $\phi$.
We define two types of explanations, model- and proof-theoretic explanations, and use cost functions to reflect preferences between explanations.
Further, we present our algorithm implemented for propositional logic that compute such explanations and empirically evaluate it in random knowledge bases and a planning domain.

assistant: text: This paper provides a perspective on explanations between two knowledge bases, and runs parallel to work on model reconciliation in planning literature.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving One-Shot NAS By Suppressing The Posterior Fading

Paper content:
There is a growing interest in automated neural architecture search (NAS).
To improve the efficiency of NAS, previous approaches adopt  weight sharing method to force all models share the same set of weights.
However, it has been observed that a model performing better with shared weights does not necessarily perform  better when trained alone.
In this paper, we analyse existing weight sharing one-shot NAS approaches from a Bayesian point of view and identify the posterior fading problem, which compromises the effectiveness of shared weights.
To alleviate this problem, we present a practical approach to guide the parameter posterior towards its true distribution.
Moreover, a hard latency constraint is introduced during the search so that the desired latency can be achieved.
The resulted method, namely Posterior Convergent NAS (PC-NAS), achieves state-of-the-art performance under standard GPU latency constraint on ImageNet.
In our small search space, our model PC-NAS-S attains76.8% top-1 accuracy, 2.1% higher than MobileNetV2 (1.4x) with the same latency.
When adopted to our large search space, PC-NAS-L achieves 78.1% top-1 accuracy within 11ms.
The discovered architecture also transfers well to other computer vision applications such as object detection and person re-identification.

assistant: text: Author identifies an issue with NAS called posterior fading and introduces Posterior Convergent NAS to mitigate this effect
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Complement Objective Training

Paper content:
Hierarchical label structures widely exist in many machine learning tasks, ranging from those with explicit label hierarchies such as image classification to the ones that have latent label hierarchies such as semantic segmentation.
Unfortunately, state-of-the-art methods often utilize cross-entropy loss which in-explicitly assumes the independence among class labels.
Motivated by the fact that class members from the same hierarchy need to be similar to each others, we design a new training diagram called Hierarchical Complement Objective Training (HCOT).
In HCOT, in addition to maximizing the probability of the ground truth class, we also neutralize the probabilities of rest of the classes in a hierarchical fashion, making the model take advantage of the label hierarchy explicitly.
We conduct our method on both image classification and semantic segmentation.
Results show that HCOT outperforms state-of-the-art models in CIFAR100, Imagenet, and PASCAL-context.
Our experiments also demonstrate that HCOT can be applied on tasks with latent label hierarchies, which is a common characteristic in many machine learning tasks.

assistant: text: A method that regularizes the entropy of the posterior distribution over classes which can be useful for image classsification and segmentation tasks