ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Ensembles: A Loss Landscape Perspective

Paper content:
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty  and out-of-distribution robustness of deep learning models.
While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well.
Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift.
One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space.
We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions.
Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space.
We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions.
Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.

assistant: text: We study deep ensembles through the lens of loss landscape and the space of predictions, demonstrating that the decorrelation power of random initializations is unmatched by subspace sampling that only explores a single mode.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit

Paper content:
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search.
This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best fixed batch-size.
At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function.
Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.
After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.
Experimental results show that the RMGD achieves performance better than the best performing single batch size.
It is surprising that the RMGD achieves better performance than grid search.
Furthermore, it attains this performance in a shorter amount of time than grid search.

assistant: text: An optimization algorithm that explores various batch sizes based on probability and automatically exploits successful batch size which minimizes validation loss.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Smoothing the Geometry of Probabilistic Box Embeddings

Paper content:
There is growing interest in geometrically-inspired embeddings for learning hierarchies, partial orders, and lattice structures, with natural applications to transitive relational data such as entailment graphs.
Recent work has extended these ideas beyond deterministic hierarchies to probabilistically calibrated models, which enable learning from uncertain supervision and inferring soft-inclusions among concepts, while maintaining the geometric inductive bias of hierarchical embedding models.
We build on the Box Lattice model of Vilnis et al. (2018), which showed promising results in modeling soft-inclusions through an overlapping hierarchy of sets, parameterized as high-dimensional hyperrectangles (boxes).
However, the hard edges of the boxes present difficulties for standard gradient based optimization; that work employed a special surrogate function for the disjoint case, but we find this method to be fragile.
In this work, we present a novel hierarchical embedding model, inspired by a relaxation of box embeddings into parameterized density functions using Gaussian convolutions over the boxes.
Our approach provides an alternative surrogate to the original lattice measure that improves the robustness of optimization in the disjoint case, while also preserving the desirable properties with respect to the original lattice.
We demonstrate increased or matching performance on WordNet hypernymy prediction, Flickr caption entailment, and a MovieLens-based market basket dataset.
We show especially marked improvements in the case of sparse data, where many conditional probabilities should be low, and thus boxes should be nearly disjoint.

assistant: text: Improve hierarchical embedding models using kernel smoothing
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stability of Stochastic Gradient Method with Momentum for Strongly Convex Loss Functions

Paper content:
While momentum-based methods, in conjunction with the stochastic gradient descent, are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods.
In practice, the momentum parameter is often chosen in a heuristic fashion with little theoretical guidance.
In this work, we use the framework of algorithmic stability to provide an upper-bound on the generalization error for the class of strongly convex loss functions, under mild technical assumptions.
Our bound decays to zero inversely with the size of the training set, and increases as the momentum parameter is increased.
We also develop an upper-bound on the expected true risk,  in terms of the number of training steps, the size of the training set, and the momentum parameter.

assistant: text: Stochastic gradient method with momentum generalizes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Decentralized Deep Learning with Arbitrary Communication Compression

Paper content:
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters.
As current approaches are limited by network bandwidth, we propose the use of communication compression in the decentralized training context.
We show that Choco-SGD achieves linear speedup in the number of workers for arbitrary high compression ratios on general non-convex functions, and non-IID training data.
We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models
(i) over decentralized user devices, connected by a peer-to-peer network and
(ii) in a datacenter.

assistant: text: We propose Choco-SGD---decentralized SGD with compressed communication---for non-convex objectives and show its strong performance in various deep learning applications (on-device learning, datacenter case).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Multi-Manifold GANs with a Learned Noise Prior

Paper content:
Generative adversarial networks (GANs) learn to map samples from a noise distribution to a chosen data distribution.
Recent work has demonstrated that GANs are consequently sensitive to, and limited by, the shape of the noise distribution.
For example, a single generator struggles to map continuous noise (e.g. a uniform distribution) to discontinuous output (e.g. separate Gaussians) or complex output (e.g. intersecting parabolas).
We address this problem by learning to generate from multiple models such that the generator's output is actually the combination of several distinct networks.
We contribute a novel formulation of multi-generator models where we learn a prior over the generators conditioned on the noise, parameterized by a neural network.
Thus, this network not only learns the optimal rate to sample from each generator but also optimally shapes the noise received by each generator.
The resulting Noise Prior GAN (NPGAN) achieves expressivity and flexibility that surpasses both single generator models and previous multi-generator models.

assistant: text: A multi-generator GAN framework with an additional network to learn a prior over the input noise.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LEAF: A Benchmark for Federated Settings

Paper content:
Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day.
This wealth of data can help to learn models that can improve the user experience on each device.
However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning.
As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks.
To this end, we propose Leaf, a modular benchmarking framework for learning in federated settings.
Leaf includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.

assistant: text: We present Leaf, a modular benchmarking framework for learning in federated data, with applications to learning paradigms such as federated learning, meta-learning, and multi-task learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Undersensitivity in Neural Reading Comprehension

Paper content:
Neural reading comprehension models have recently achieved impressive gener- alisation results, yet still perform poorly when given adversarially selected input.
Most prior work has studied semantically invariant text perturbations which cause a model’s prediction to change when it should not.
In this work we focus on the complementary problem: excessive prediction undersensitivity where input text is meaningfully changed, and the model’s prediction does not change when it should.
We formulate a noisy adversarial attack which searches among semantic variations of comprehension questions for which a model still erroneously pro- duces the same answer as the original question – and with an even higher prob- ability.
We show that – despite comprising unanswerable questions – SQuAD2.0 and NewsQA models are vulnerable to this attack and commit a substantial frac- tion of errors on adversarially generated questions.
This indicates that current models—even where they can correctly predict the answer—rely on spurious sur- face patterns and are not necessarily aware of all information provided in a given comprehension question.
Developing this further, we experiment with both data augmentation and adversarial training as defence strategies: both are able to sub- stantially decrease a model’s vulnerability to undersensitivity attacks on held out evaluation data.
Finally, we demonstrate that adversarially robust models gener- alise better in a biased data setting with a train/evaluation distribution mismatch; they are less prone to overly rely on predictive cues only present in the training set and outperform a conventional model in the biased data setting by up to 11% F1.

assistant: text: We demonstrate vulnerability to undersensitivity attacks in SQuAD2.0 and NewsQA neural reading comprehension models, where the model predicts the same answer with increased confidence to adversarially chosen questions, and compare defence strategies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Revisiting Bayes by Backprop

Paper content:
In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks.
Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%.
Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs.
We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics.
We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks.
We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them.
We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

assistant: text: Variational Bayes scheme for Recurrent Neural Networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Neuro-AI Interface: Learning DNNs from the Human Brain

Paper content:
Deep neural networks (DNNs) are inspired from the human brain and the interconnection between the two has been widely studied in the literature.
However, it is still an open question whether DNNs are able to make decisions like the brain.
Previous work has demonstrated that DNNs, trained by matching the neural responses from inferior temporal (IT) cortex in monkey's brain, is able to achieve human-level performance on the image object recognition tasks.
This indicates that neural dynamics can provide informative knowledge to help DNNs accomplish specific tasks.
In this paper, we introduce the concept of a neuro-AI interface, which aims to use human's neural responses as supervised information for helping AI systems solve a task that is difficult when using traditional machine learning strategies.
In order to deliver the idea of neuro-AI interfaces, we focus on deploying it to one of the fundamental problems in generative adversarial networks (GANs): designing a proper evaluation metric to evaluate the quality of images produced by GANs.

assistant: text: Describe a neuro-AI interface technique to evaluate generative adversarial networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: M^3RL: Mind-aware Multi-agent Management Reinforcement Learning

Paper content:
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly learning a policy for each agent to maximize a common reward.
In this paper, we aim to address this from a different angle.
In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not want to do.
For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together.
The objective of the manager is to maximize the overall productivity as well as minimize payments made to the workers for ad-hoc worker teaming.
To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL), which consists of agent modeling and policy learning.
We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents.
The experimental results have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.

assistant: text: We propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL) for training a manager to motivate self-interested workers to achieve optimal collaboration by assigning suitable contracts to them.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Context-adaptive Entropy Model for End-to-end Optimized Image Compression

Paper content:
We propose a context-adaptive entropy model for use in end-to-end optimized image compression.
Our model exploits two types of contexts, bit-consuming contexts and bit-free contexts, distinguished based upon whether additional bit
allocation is required.
Based on these contexts, we allow the model to more accurately estimate the distribution of each latent representation with a more generalized form of the approximation models, which accordingly leads to an
enhanced compression performance.
Based on the experimental results, the proposed method outperforms the traditional image codecs, such as BPG and JPEG2000, as well as other previous artificial-neural-network (ANN) based approaches, in terms of the peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) index.
The test code is publicly available at https://github.com/JooyoungLeeETRI/CA_Entropy_Model.

assistant: text: Context-adaptive entropy model for use in end-to-end optimized image compression, which significantly improves compression performance
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Riemannian TransE: Multi-relational Graph Embedding in Non-Euclidean Space

Paper content:
Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion.
Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure.
To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold.
Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data.
Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters.

assistant: text: Multi-relational graph embedding with Riemannian manifolds and TransE-like loss function.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Gain

Paper content:
Adversarial examples can be defined as inputs to a model which induce a mistake -- where the model output is different than that of an oracle, perhaps in surprising or malicious ways.
Original models of adversarial attacks are primarily studied in the context of classification and computer vision tasks.
While several attacks have been proposed in natural language processing (NLP) settings, they often vary in defining the parameters of an attack and what a successful attack would look like.
The goal of this work is to propose a unifying model of adversarial examples suitable for NLP tasks in both generative and classification settings.
We define the notion of adversarial gain: based in control theory, it is a measure of the change in the output of a system relative to the perturbation of the input (caused by the so-called adversary) presented to the learner.
This definition, as we show, can be used under different feature spaces and distance conditions to determine attack or defense effectiveness across different intuitive manifolds.
This notion of adversarial gain not only provides a useful way for evaluating adversaries and defenses, but can act as a building block for future work in robustness under adversaries due to its rooted nature in stability and manifold theory.

assistant: text: We propose an alternative measure for determining effectiveness of adversarial attacks in NLP models according to a distance measure-based method like incremental L2-gain in control theory.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning temporal evolution of probability distribution with Recurrent Neural Network

Paper content:
We propose to tackle a time series regression problem by computing temporal evolution of a probability density function to provide a probabilistic forecast.
A Recurrent Neural Network (RNN) based model is employed to learn a nonlinear operator for temporal evolution of a probability density function.
We use a softmax layer for a numerical discretization of a smooth probability density functions, which transforms a function approximation problem to a classification task.
Explicit and implicit regularization strategies are introduced to impose a smoothness condition on the estimated probability distribution.
A Monte Carlo procedure to compute the temporal evolution of the distribution for a multiple-step forecast is presented.
The evaluation of the proposed algorithm on three synthetic and two real data sets shows advantage over the compared baselines.

assistant: text: Proposed RNN-based algorithm to estimate predictive distribution in one- and multi-step forecasts in time series prediction problems
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Isolating effects of age with fair representation learning when assessing dementia

Paper content:
One of the most prevalent symptoms among the elderly population, dementia, can be detected by classifiers trained on linguistic features extracted from narrative transcripts.
However, these linguistic features are impacted in a similar but different fashion by the normal aging process.
Aging is therefore a confounding factor, whose effects have been hard for machine learning classifiers to isolate.
In this paper, we show that deep neural network (DNN) classifiers can infer ages from linguistic features, which is an entanglement that could lead to unfairness across age groups.
We show this problem is caused by undesired activations of v-structures in causality diagrams, and it could be addressed with fair representation learning.
We build neural network classifiers that learn low-dimensional representations reflecting the impacts of dementia yet discarding the effects of age.
To evaluate these classifiers, we specify a model-agnostic score $\Delta_{eo}^{(N)}$ measuring how classifier results are disentangled from age.
Our best models outperform baseline neural network classifiers in disentanglement, while compromising accuracy by as little as 2.56\% and 2.25\% on DementiaBank and the Famous People dataset respectively.

assistant: text: Show that age confounds cognitive impairment detection + solve with fair representation learning + propose metrics and models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PVAE: Learning Disentangled Representations with Intrinsic Dimension via Approximated L0 Regularization

Paper content:
Many models based on the Variational Autoencoder are proposed to achieve disentangled latent variables in inference.
However, most current work is focusing on designing powerful disentangling regularizers, while the given number of dimensions for the latent representation at initialization could severely inﬂuence the disentanglement.
Thus, a pruning mechanism is introduced, aiming at automatically seeking for the intrinsic dimension of the data while promoting disentangled representations.
The proposed method is validated on MPI3D and MNIST to be advancing state-of-the-art methods in disentanglement, reconstruction, and robustness.
The code is provided on the https://github.com/WeyShi/FYP-of-Disentanglement.
To advance disentanglement, models based on the Variational Autoencoder (VAE) (Kingma and Welling, 2014) are proposed in terms of additional disentangling regularizers.
However, in this paper, we introduce an orthogonal mechanism that is applicable to most state-of-theart models, resulting in higher disentanglement and robustness for model configurationsespecially the choice of dimensionality for the latent representation.
Intuitively, both excessive and deficient latent dimensions can be detrimental to achieving the best disentangled latent representations.
For excessive dimensions, powerful disentangling regularizers, like the β-VAE (Higgins et al., 2017) , can force information to be split across dimensions, resulting in capturing incomplete features.
On the other hand, having too few dimensions inevitably leads to an entangled representation, such that each dimension could capture enough information for the subsequent reconstruction.
A pruning mechanism that is complementary to most current state-of-the-art VAE-based disentangling models is introduced and validated on MPI3D and MNIST.
The approximated L 0 regularization facilitates the model to capture better-disentangled representations with optimal size and increases the robustness to initialization.
Moreover, with the same hyperparameters, the model approaches the intrinsic dimension for several datasets including MNIST and MPI3D, even with an extra-large number of dimensions at initialization.
Even given the intrinsic dimension, the PVAE still outperforms other SOTA methods in terms of disentanglement and reconstruction.

assistant: text: The Pruning VAE is proposed to search for disentangled variables with intrinsic dimension.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
The backpropagation algorithm is the most popular algorithm training neural networks nowadays.
However, it suffers from the forward locking, backward locking and update locking problems, especially when a neural network is so large that its layers are distributed across multiple devices.
Existing solutions either can only handle one locking problem or lead to severe accuracy loss or memory inefficiency.
Moreover, none of them consider the straggler problem among devices.
In this paper, we propose \textbf{Layer-wise Staleness} and a novel efficient training algorithm, \textbf{Diversely Stale Parameters} (DSP), which can address all these challenges without loss of accuracy nor memory issue.
We also analyze the convergence of DSP with two popular gradient-based methods and prove that both of them are guaranteed to converge to critical points for non-convex problems.
Finally, extensive experimental results on training deep convolutional neural networks demonstrate that our proposed DSP algorithm can achieve significant training speedup with stronger robustness and better generalization than compared methods.

assistant: text: We propose Diversely Stale Parameters to break lockings of the backpropoagation algorithm and train a CNN in parallel.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation

Paper content:
Molecular graph generation is a fundamental problem for drug discovery and has been attracting growing attention.
The problem is challenging since it requires not only generating chemically valid molecular structures but also optimizing their chemical properties in the meantime.
Inspired by the recent progress in deep generative models, in this paper we propose a flow-based autoregressive model for graph generation called GraphAF.
GraphAF combines the advantages of both autoregressive and flow-based approaches and enjoys: (1) high model flexibility for data density estimation; (2) efficient parallel computation for training; (3) an iterative sampling process, which allows leveraging chemical domain knowledge for valency checking.
Experimental results show that GraphAF is able to generate 68\% chemically valid molecules even without chemical knowledge rules and 100\% valid molecules with chemical rules.
The training process of GraphAF is two times faster than the existing state-of-the-art approach GCPN.
After fine-tuning the model for goal-directed property optimization with reinforcement learning, GraphAF achieves state-of-the-art performance on both chemical property optimization and constrained property optimization.

assistant: text: A flow-based autoregressive model for molecular graph generation. Reaching state-of-the-art results on molecule generation and properties optimization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AdaGAN: Adaptive GAN for Many-to-Many Non-Parallel Voice Conversion

Paper content:
Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker.
Earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs.
Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning remains less explored areas in VC.
Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices.
In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i.e., case of zero-shot learning).
In particular, propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure help in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC.
We compare our results with the state-of-the-art StarGAN-VC architecture.
In particular, the AdaGAN achieves 31.73%, and 10.37% relative improvement compared to the StarGAN in MOS tests for speech quality and speaker similarity, respectively.
The key strength of the proposed architectures is that it yields these results with less computational complexity.
AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters.

assistant: text: Novel adaptive instance normalization based GAN framework for non parallel many-to-many and zero-shot VC.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Evaluations and Methods for Explanation through Robustness Analysis

Paper content:
Among multiple ways of interpreting a machine learning model, measuring the importance of a set of features tied to a prediction is probably one of the most intuitive way to explain a model.
In this paper, we establish the link between a set of features to a prediction with a new evaluation criteria, robustness analysis, which measures the minimum tolerance of adversarial perturbation.
By measuring the tolerance level for an adversarial attack, we can extract a set of features that provides most robust support for a current prediction, and also can extract a set of features that contrasts the current prediction to a target class by setting a targeted adversarial attack.
By applying this methodology to various prediction tasks across multiple domains, we observed the derived explanations are indeed capturing the significant feature set qualitatively and quantitatively.

assistant: text: We propose new objective measurement for evaluating explanations based on the notion of adversarial robustness. The evaluation criteria further allows us to derive new explanations which capture pertinent features qualitatively and quantitatively.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Distributional Bayesian optimisation for variational inference on black-box simulators

Paper content:
Inverse problems are ubiquitous in natural sciences and refer to the challenging task of inferring complex and potentially multi-modal posterior distributions over hidden parameters given a set of observations.
Typically, a model of the physical process in the form of differential equations is available but leads to intractable inference over its parameters.
While the forward propagation of parameters through the model simulates the evolution of the system, the inverse problem of finding the parameters given the sequence of states is not unique.
In this work, we propose a generalisation of the Bayesian optimisation framework to approximate inference.
The resulting method learns approximations to the posterior distribution by applying Stein variational gradient descent on top of estimates from a Gaussian process model.
Preliminary results demonstrate the method's performance on likelihood-free inference for reinforcement learning environments.

assistant: text: An approach to combine variational inference and Bayesian optimisation to solve complicated inverse problems
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Variational Inference For Embedding Knowledge Graphs

Paper content:
Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data.
In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model.
While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions.
The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising parameterisable distributions which permit training by back-propagation in the context of neural variational inference, resulting in a highly-scalable method.
Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound.
The generative frameworks are flexible enough to allow training under any prior distribution that permits a re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters.
Experiment results display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations.
Code publicly available on Github.

assistant: text: Working toward generative knowledge graph models to better estimate predictive uncertainty in knowledge inference.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LDMGAN: Reducing Mode Collapse in GANs with Latent Distribution Matching

Paper content:
Generative Adversarial Networks (GANs) have shown impressive results in modeling distributions over complicated manifolds such as those of natural images.
However, GANs often suffer from mode collapse, which means they are prone to characterize only a single or a few modes of the data distribution.
In order to address this problem, we propose a novel framework called LDMGAN.
We ﬁrst introduce Latent Distribution Matching (LDM) constraint which regularizes the generator by aligning distribution of generated samples with that of real samples in latent space.
To make use of such latent space, we propose a regularized AutoEncoder (AE) that maps the data distribution to prior distribution in encoded space.
Extensive experiments on synthetic data and real world datasets show that our proposed framework signiﬁcantly improves GAN’s stability and diversity.

assistant: text: We propose an AE-based GAN that alleviates mode collapse in GANs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning To Explore Using Active Neural Mapping

Paper content:
This work presents a modular and hierarchical approach to learn policies for exploring 3D environments.
Our approach leverages the strengths of both classical and learning-based methods, by using analytical path planners with learned mappers, and global and local policies.
Use of learning provides flexibility with respect to input modalities (in mapper), leverages structural regularities of the world (in global policies), and provides robustness to errors in state estimation (in local policies).
Such use of learning within each module retains its benefits, while at the same time, hierarchical decomposition and modular training allow us to sidestep the high sample complexities associated with training end-to-end policies.
Our experiments in visually and physically realistic simulated 3D environments demonstrate the effectiveness of our proposed approach over past learning and geometry-based approaches.

assistant: text: A modular and hierarchical approach to learn policies for exploring 3D environments.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Do Convolutional Neural Networks act  as Compositional Nearest Neighbors?

Paper content:
We present a simple approach based on pixel-wise nearest neighbors to understand and interpret the functioning of state-of-the-art neural networks for pixel-level tasks.
We aim to understand and uncover the synthesis/prediction mechanisms of state-of-the-art convolutional neural networks.
To this end, we primarily analyze the synthesis process of generative models and the prediction mechanism of discriminative models.
The main hypothesis of this work is that convolutional neural networks for pixel-level tasks learn a fast compositional nearest neighbor synthesis/prediction function.
Our experiments on semantic segmentation and image-to-image translation show qualitative and quantitative evidence supporting this hypothesis.

assistant: text: Convolutional Neural Networks behave as Compositional Nearest Neighbors!
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Automatically Composing Representation Transformations as a Means for Generalization

Paper content:
A generally intelligent learner should generalize to more complex tasks than it has previously encountered, but the two common paradigms in machine learning -- either training a separate learner per task or training a single learner for all tasks -- both have difficulty with such generalization because they do not leverage  the compositional structure of the task distribution.
This paper introduces the compositional problem graph as a broadly applicable formalism to relate tasks of different complexity in terms of problems with shared subproblems.
We propose the compositional generalization problem for measuring how readily old knowledge can be reused and hence built upon.
As a first step for tackling compositional generalization, we introduce the compositional recursive learner, a domain-general framework for learning algorithmic procedures for composing representation transformations, producing a learner that reasons about what computation to execute by making analogies to previously seen problems.
We show on a symbolic and a high-dimensional domain that our compositional approach can generalize to more complex problems than the learner has previously encountered, whereas baselines that are not explicitly compositional do not.

assistant: text: We explore the problem of compositional generalization and propose a means for endowing neural network architectures with the ability to compose themselves to solve these problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Incremental training of multi-generative adversarial networks

Paper content:
Generative neural networks map a standard, possibly distribution to a complex high-dimensional distribution, which represents the real world data set.
However, a determinate input distribution as well as a specific architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space.
To resolve this difficulty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space.
We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network's distribution space.
Generative Adversarial Nets (GAN) BID5 is a framework of estimating generative models.
The main idea BID4 is to train two target network models simultaneously, in which one, called the generator, aims to generate samples that resemble those from the data distribution, while the other, called the discriminator, aims to distinguish the samples by the generator from the real data.
Naturally, this type of training framework admits a nice interpretation as a twoperson zero-sum game and interesting game theoretical properties, such as uniqueness of the optimal solution, have been derived BID5 .
It is further proved that such adversarial process minimizes certain divergences, such as Shannon divergence, between the generated distribution and the data distribution.Simply put, the goal of training a GAN is to search for a distribution in the range of the generator that best approximates the data distribution.
The range is often defined by the input latent variable z and its specific architecture, i.e., Π = {G(z, θ), θ ∈ Θ}.
When the range is general enough, one could possibly find the real data distribution.
However, in practice, the range is usually insufficient to perfectly describe the real data, which is typically of high dimension.
As a result, what we search for is in fact the I-projection BID2 of the real data distribution on Π.

assistant: text: We propose a new method to incrementally train a mixture generative model to approximate the information projection of the real data distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Sparse-Winograd Convolutional Neural Networks

Paper content:
Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices.
Their energy is dominated by the number of multiplies needed to perform the convolutions.
Winograd’s minimal filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can reduce the operation count, but these two methods cannot be straightforwardly combined — applying the Winograd transform fills in the sparsity in both the weights and the activations.
We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity.
First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations.
Second, we prune the weights in the Winograd domain to exploit static weight sparsity.
For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by 10.4x, 6.8x and 10.8x respectively with loss of accuracy less than 0.1%, outperforming previous baselines by 2.0x-3.0x.
We also show that moving ReLU to the Winograd domain allows more aggressive pruning.

assistant: text: Prune and ReLU in Winograd domain for efficient convolutional neural network
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: HOW IMPORTANT ARE NETWORK WEIGHTS? TO WHAT EXTENT DO THEY NEED AN UPDATE?

Paper content:
In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss.
Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training.
This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated.
We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better).
We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19,
ResNet-110 and DenseNet-121).
On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy.
Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model.
Our source code can be found in the appendix.

assistant: text: An experimental paper that proves the amount of redundant weights that can be freezed from the third epoch only, with only a very slight drop in accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generative Models for Graph-Based Protein Design

Paper content:
Engineered proteins offer the potential to solve many problems in biomedicine, energy, and materials science, but creating designs that succeed is difficult in practice.
A significant aspect of this challenge is the complex coupling between protein sequence and 3D structure, and the task of finding a viable design is often referred to as the inverse protein folding problem.
We develop generative models for protein sequences conditioned on a graph-structured specification of the design target.
Our approach efficiently captures the complex dependencies in proteins by focusing on those that are long-range in sequence but local in 3D space.
Our framework significantly improves upon prior parametric models of protein sequences given structure, and takes a step toward rapid and targeted biomolecular design with the aid of deep generative models.

assistant: text: We learn to conditionally generate protein sequences given structures with a model that captures sparse, long-range dependencies.