ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies

Paper content:
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.
Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation.
We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities.
With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.
In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

assistant: text: An estimated mixture policy which takes ideas from off-policy policy evaluation infinite horizon estimators and regression importance sampling for importance weight, and extends them to many policies and unknown policies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis

Paper content:
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises.
In this work, we show that highly-structured semantic hierarchy emerges from the generative representations as the variation factors for synthesizing scenes.
By probing the layer-wise representations with a broad set of visual concepts at different abstraction levels, we are able to quantify the causality between the activations and the semantics occurring in the output image.
Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes.
The qualitative and quantitative results suggest that the generative representations learned by GAN are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme.
Identifying such a set of manipulatable latent semantics facilitates semantic scene manipulation.

assistant: text: Paper investigates the aspects encoded by the latent variables input into different layers in StyleGAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Representations and Generative Models for 3D Point Clouds

Paper content:
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.
In this paper, we look at geometric data represented as point clouds.
We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability.
The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation.
We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM).
Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.
To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds.

assistant: text: Approaches to learn GAN-type generative models using PointNet architecture and latent-space GAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion

Paper content:
Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details.
Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views.
This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery.
To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion:
(i) co-registration,
(ii) fusion,
(iii) up-sampling, and
(iv) registration-at-the-loss.
Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism.
We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs.
We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet.
We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale.
Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

assistant: text: The first deep learning approach to MFSR to solve registration, fusion, up-sampling in an end-to-end manner.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Loss Scaling for Mixed Precision Training

Paper content:
Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs.
MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training.
Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages.
We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter.
We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods.
We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

assistant: text: Proposal for an adaptive loss scaling method during backpropagation for mix precision training where scale rate is decided automatically to reduce the underflow.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Paper content:
A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.
These representations are typically used as general purpose features for words across a range of NLP problems.
However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.
Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.
In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model.
We train this model on several data sources with multiple training objectives on over 100 million sentences.
Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.
We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

assistant: text: This paper is about learning sentence embeddings by combining several training signals: skip-thought, predicting translation, classifying entailment relationships, and predicting the constituent parse.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SDGM: Sparse Bayesian Classifier Based on a Discriminative Gaussian Mixture Model

Paper content:
In probabilistic classification, a discriminative model based on Gaussian mixture exhibits flexible fitting capability.
Nevertheless, it is difficult to determine the number of components.
We propose a sparse classifier based on a discriminative Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM).
In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning.
This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components.
The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and can be trained in an end-to-end manner.
Experimental results indicated that the proposed method prevented overfitting by obtaining sparsity.
Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of a deep NN.

assistant: text: The paper presents a Gaussian mixture model trained via gradient descent arguments which allows for inducing sparsity and reducing the trainable model layer parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Approximation capability of neural networks on sets of probability measures and tree-structured data

Paper content:
This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.
By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.
The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.
The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.
This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

assistant: text: This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

Paper content:
In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training.
The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values.
But it cannot provide useful gradients and therefore requires complex optimization techniques.
The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent.
Yet it is not scale-invariant and causes the same shrinking rate to all parameters, which is inefficient in increasing sparsity.
Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant.
Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level.
We also show that DeepHoyer can be applied to both element-wise and structural pruning.

assistant: text: The paper proposes a scale-invariant regularizer (DeepHoyer) inspired by the Hoyer measure to enforce sparsity in neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving the robustness of ImageNet classifiers using elements of human visual cognition

Paper content:
We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale.
As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models against small-norm adversarial perturbations under some threat models.
It does not, however, improve the robustness against more natural, and typically larger, perturbations.
Learning more robust features during training appears to be necessary for robustness in this second sense.
We show that features derived from a model that was encouraged to learn global, shape-based representations (Geirhos et al., 2019) do not only improve the robustness against natural perturbations, but when used in conjunction with an episodic memory, they also provide additional robustness against adversarial perturbations.
Finally, we address three important design choices for the episodic memory: memory size, dimensionality of the memories and the retrieval method.
We show that to make the episodic memory more compact, it is preferable to reduce the number of memories by clustering them, instead of reducing their dimensionality.

assistant: text: systematic study of large-scale cache-based image recognition models, focusing particularly on their robustness properties
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Batch-shaping for learning conditional channel gated networks

Paper content:
We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost.
This is achieved by gating the deep-learning architecture on a fine-grained-level.
Individual convolutional maps are turned on/off conditionally on features in the network.
To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner.
We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution.
We use this novel technique to force gates to be more conditional on the data.
We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation.
Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.
In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.
We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

assistant: text: A method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Smooth Kernels Improve Adversarial Robustness and Perceptually-Aligned Gradients

Paper content:
Recent research has shown that CNNs are often overly sensitive to high-frequency textural patterns.
Inspired by the intuition that humans are more sensitive to the lower-frequency (larger-scale) patterns we design a regularization scheme that penalizes large differences between adjacent components within each convolutional kernel.
We apply our regularization onto several popular training methods, demonstrating that the models with the proposed smooth kernels enjoy improved adversarial robustness.
Further, building on recent work establishing connections between adversarial robustness and interpretability, we show that our method appears to give more perceptually-aligned gradients.

assistant: text: This paper proposes a new regularization scheme that encourages convolutional kernels to be smoother, arguing that reducing neural network reliance on high-frequency components helps robustness against adversarial examples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LEAP: Learning Embeddings for Adaptive Pace

Paper content:
Determining the optimal order in which data examples are presented to Deep Neural Networks during training is a non-trivial problem.
However, choosing a non-trivial scheduling method may drastically improve convergence.
In this paper, we propose a Self-Paced Learning (SPL)-fused Deep Metric Learning (DML) framework, which we call Learning Embeddings for Adaptive Pace (LEAP).
Our method parameterizes mini-batches dynamically based on the \textit{easiness} and \textit{true diverseness} of the sample within a salient feature representation space.
In LEAP, we train an \textit{embedding} Convolutional Neural Network (CNN) to learn an expressive representation space by adaptive density discrimination using the Magnet Loss.
The \textit{student} CNN classifier dynamically selects samples to form a mini-batch based on the \textit{easiness} from cross-entropy losses and \textit{true diverseness} of examples from the representation space sculpted by the \textit{embedding} CNN.
We evaluate LEAP using deep CNN architectures for the task of supervised image classification on MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and SVHN.
We show that the LEAP framework converges faster with respect to the number of mini-batch updates required to achieve a comparable or better test performance on each of the datasets.

assistant: text: Introduces a method for creating mini batches for a student network by using a second learned representation space to dynamically select examples by their 'easiness and true diverseness'.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GUIDEGAN:  ATTENTION  BASED  SPATIAL  GUIDANCE FOR  IMAGE-TO-IMAGE TRANSLATION

Paper content:
Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner.
However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice.
Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients.
To tackle this problem, we propose a GuideGAN based on attention mechanism.
More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction.
This attention map then assists the generator to produce more plausible and realistic images.
We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks.
Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

assistant: text: A general method that improves the image translation performance of GAN framework by using an attention embedded discriminator
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Connections Between Optimization in Machine Learning and Adaptive Control

Paper content:
This paper explores many immediate connections between adaptive control and machine learning, both through common update laws as well as common concepts.
Adaptive control as a field has focused on mathematical rigor and guaranteed convergence.
The rapid advances in machine learning on the other hand have brought about a plethora of new techniques and problems for learning.
This paper elucidates many of the numerous common connections between both fields such that results from both may be leveraged together to solve new problems.
In particular, a specific problem related to higher order learning is solved through insights obtained from these intersections.

assistant: text: History of parallel developments in update laws and concepts between adaptive control and optimization in machine learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Accelerating Convolutional Neural Networks using Iterative Two-Pass Decomposition

Paper content:
We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).
The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.
Our two-pass CP-decomposition helps prevent from the instability problem.
The iterative flow makes the decomposition of the deeper networks systematic.
The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.
Deep learning has become of vital importance in a variety of artificial intelligence applications.
Recently, convolutional neural networks (CNNs) have been widely applied to have the breakthrough in improving the recognition accuracy for challenging computer vision tasks such as image classification, localization, object detection, and so on BID19 ; BID11 ; BID20 ; BID3 ).
However, those significant achievements using CNNs come with the cost of larger network size and higher computational complexity, which leads to an increasing difficulty for deploying to resource constrained edge devices, or even for the fast computation on the cloud servers.
This paper addresses the acceleration of the existing CNN models to cope with such burden.
The iterative two-pass decomposition flow has been presented to accelerate existing deep CNNs.
Our two-pass decomposition effectively prevents from the CP instability.
The Rank Selection algorithm provides the fine-grained rank configuration to achieve the target speedup while maintaining the accuracy.
The experiment results show that VGG16 can be accelerated by 6.2 times with the accuracy drop of only 1.20% and the size reduction of 85%.
In addition, ResNet50 can be speeded up by 1.35 times with the accuracy drop of 1.51% and the size reduction of 48%.The
future works include the improvement of the grouping scheme and the decomposing order, and a smarter rank selection with non-linear fitness estimation. In
addition, accelerating 1×1 convolutional layers will also be considered for the further improvement on the advanced CNNs.APPENDIX A FIG6 shows the initial fitnesses of the baseline rank configuration and fitness-based configuration. Note
that the layer order is sorted by the fitnesses of the baseline rank configuration. Because
there is only a slight difference between the sorting of the two configuration, the fitness-based grouping scheme is based on the sorting of the baseline rank configuration. TAB1

assistant: text: We present the iterative two-pass CP decomposition flow to effectively accelerate existing convolutional neural networks (CNNs).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Joint autoencoders: a flexible meta-learning framework

Paper content:
The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.
Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.
Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a data-driven fashion.
We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the specific unique aspects of each task.
Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.
The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a unified fashion, and can easily deal with data arising from different types of sources.
Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the flexible and task-oriented representations arising in the network.

assistant: text: Proposes a generic framework for end-to-end transfer learning / domain adaptation with deep neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Non-Autoregressive Dialog State Tracking

Paper content:
Recent efforts in Dialogue State Tracking (DST) for task-oriented dialogues have progressed toward open-vocabulary or generation-based approaches where the models can generate slot value candidates from the dialogue history itself.
These approaches have shown good performance gain, especially in complicated dialogue domains with dynamic slot values.
However, they fall short in two aspects: (1) they do not allow models to explicitly learn signals across domains and slots to detect potential dependencies among \textit{(domain, slot)} pairs; and (2) existing models follow auto-regressive approaches which incur high time cost when the dialogue evolves over multiple domains and multiple turns.
In this paper, we propose a novel framework of Non-Autoregressive Dialog State Tracking (NADST) which can factor in potential dependencies among domains and slots to optimize the models towards better prediction of dialogue states as a complete set rather than separate slots.
In particular, the non-autoregressive nature of our method not only enables decoding in parallel to significantly reduce the latency of DST for real-time dialogue response generation, but also detect dependencies among slots at token level in addition to slot and domain level.
Our empirical results show that our model achieves the state-of-the-art joint accuracy across all domains on the MultiWOZ 2.1 corpus, and the latency of our model is an order of magnitude lower than the previous state of the art as the dialogue history extends over time.

assistant: text: Proposal for a model that is capable of tracking dialogue states in a non-recursive fashion.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Imagine Manipulation Goals for Robot Task Planning

Paper content:
Prospection is an important part of how humans come up with new task plans, but has not been explored in depth in robotics.
Predicting multiple task-level is a challenging problem that involves capturing both task semantics and continuous variability over the state of the world.
Ideally, we would combine the ability of machine learning to leverage big data for learning the semantics of a task, while using techniques from task planning to reliably generalize to new environment.
In this work, we propose a method for learning a model encoding just such a representation for task planning.
We learn a neural net that encodes the k most likely outcomes from high level actions from a given world.
Our approach creates comprehensible task plans that allow us to predict changes to the environment many time steps into the future.
We demonstrate this approach via application to a stacking task in a cluttered environment, where the robot must select between different colored blocks while avoiding obstacles, in order to perform a task.
We also show results on a simple navigation task.
Our algorithm generates realistic image and pose predictions at multiple points in a given task.

assistant: text: This paper proposes a method for learning a high-level transition function that is useful for task planning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Composition-based Multi-Relational Graph Convolutional Networks

Paper content:
Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data.
However, the primary focus has been on handling simple undirected graphs.
Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it.
Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only.
In this paper, we propose CompGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph.
CompGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations.
It also generalizes several of the existing multi-relational GCN methods.
We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results.
We make the source code of CompGCN available to foster reproducible research.

assistant: text: A Composition-based Graph Convolutional framework for multi-relational graphs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum

Paper content:
Long short-term memory networks (LSTMs) were introduced to combat vanishing gradients in simple recurrent neural networks (S-RNNs) by augmenting them with additive recurrent connections controlled by gates.
We present an alternate view to explain the success of LSTMs: the gates themselves are powerful recurrent models that provide more representational power than previously appreciated.
We do this by showing that the LSTM's gates can be decoupled from the embedded S-RNN, producing a restricted class of RNNs where the main recurrence computes an element-wise weighted sum of context-independent functions of the inputs.
Experiments on a range of challenging NLP problems demonstrate that the simplified gate-based models work substantially better than S-RNNs, and often just as well as the original LSTMs, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

assistant: text: This paper presents an analysis of LSTMS, showing that they have a form where the memory cell contents at each step is a weighted combination of the “content update” values computed at each time step and offers a simplification of LSTMs that compute the value by which the memory cell at each time step in terms of a deterministic function of the input rather than a function of the input and the current context.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Diversity and Depth in Per-Example Routing Models

Paper content:
Routing models, a form of conditional computation where examples are routed through a subset of components in a larger network, have shown promising results in recent works.
Surprisingly, routing models to date have lacked important properties, such as architectural diversity and large numbers of routing decisions.
Both architectural diversity and routing depth can increase the representational power of a routing network.
In this work, we address both of these deficiencies.
We discuss the significance of architectural diversity in routing models, and explain the tradeoffs between capacity and optimization when increasing routing depth.
In our experiments, we find that adding architectural diversity to routing models significantly improves performance, cutting the error rates of a strong baseline by 35% on an Omniglot setup.
However, when scaling up routing depth, we find that modern routing techniques struggle with optimization.
We conclude by discussing both the positive and negative results, and suggest directions for future research.

assistant: text: Per-example routing models benefit from architectural diversity, but still struggle to scale to a large number of routing decisions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Residual Non-local Attention Networks for Image Restoration

Paper content:
In this paper, we propose a residual non-local attention network for high-quality image restoration.
Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features.
To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts.
Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block.
The trunk branch is used to extract hierarchical features.
Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions.
The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map.
Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network.
Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution.
Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

assistant: text: The paper proposes a convolutional neural network architecture that includes blocks for local and non-local attention mechanisms, which are claimed to be responsible for achieving excellent results in four image restoration applications.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variance Reduction for Reinforcement Learning in Input-Driven Environments

Paper content:
We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system.
Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking.
Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns.
Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training.
We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines.
We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs.
Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.

assistant: text: This paper introduces the notion of input-dependent baselines in Policy Gradient Methods in RL, and proposes different methods to train the input dependent baseline function to help clear variance from external factor perturbation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Progressive Growing of GANs for Improved Quality, Stability, and Variation

Paper content:
We describe a new training methodology for generative adversarial networks.
The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.
This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2.
We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10.
Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator.
Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation.
As an additional contribution, we construct a higher-quality version of the CelebA dataset.

assistant: text: Introduces progressive growing and a simple parameter-free minibatch summary statistic feature for use in GAN training to enable synthesis of high-resolution images.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods

Paper content:
The geometric properties of loss surfaces, such as the local flatness of a solution, are associated with generalization in deep learning.
The Hessian is often used to understand these geometric properties.
We investigate the differences between the eigenvalues of the neural network Hessian evaluated over the empirical dataset, the Empirical Hessian, and the eigenvalues of the Hessian under the data generating distribution, which we term the True Hessian.
Under mild assumptions, we use random matrix theory to show that the True Hessian has eigenvalues of smaller absolute value than the Empirical Hessian.
We support these results for different SGD schedules on both a 110-Layer ResNet and VGG-16.
To perform these experiments we propose a framework for spectral visualization, based on GPU accelerated stochastic Lanczos quadrature.
This approach is an order of magnitude faster than state-of-the-art methods for spectral visualization, and can be generically used to investigate the spectral properties of matrices in deep learning.

assistant: text: Understanding the neural network Hessian eigenvalues under the data generating distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A∗ Search and Bound-Sensitive Heuristics for Oversubscription Planning

Paper content:
Oversubscription planning (OSP) is the problem of finding plans that maximize the utility value of their end state while staying within a specified cost bound.
Recently, it has been shown that OSP problems can be reformulated as classical planning problems with multiple cost functions but no utilities.
Here we take advantage of this reformulation to show that OSP problems can be solved optimally using the A* search algorithm, in contrast to previous approaches that have used variations on branch-and-bound search.
This allows many powerful techniques developed for classical planning to be applied to OSP problems.
We also introduce novel bound-sensitive heuristics, which are able to reason about the primary cost of a solution while taking into account secondary cost functions and bounds, to provide superior guidance compared to heuristics that do not take these bounds into account.
We implement two such bound-sensitive variants of existing classical planning heuristics, and show experimentally that the resulting search is significantly more informed than comparable heuristics that do not consider bounds.

assistant: text: We show that oversubscription planning tasks can be solved using A* and introduce novel bound-sensitive heuristics for oversubscription planning tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Contrastive Learning of Structured World Models

Paper content:
A structured understanding of our world in terms of objects, relations, and hierarchies is an important component of human cognition.
Learning such a structured world model from raw sensory data remains a challenge.
As a step towards this goal, we introduce Contrastively-trained Structured World Models (C-SWMs).
C-SWMs utilize a contrastive approach for representation learning in environments with compositional structure.
We structure each state embedding as a set of object representations and their relations, modeled by a graph neural network.
This allows objects to be discovered from raw pixel observations without direct supervision as part of the learning process.
We evaluate C-SWMs on compositional environments involving multiple interacting objects that can be manipulated independently by an agent, simple Atari games, and a multi-object physics simulation.
Our experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.

assistant: text: Contrastively-trained Structured World Models (C-SWMs) learn object-oriented state representations and a relational model of an environment from raw pixel input.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generalizing Natural Language Analysis through Span-relation Representations

Paper content:
A large number of natural language processing tasks exist to analyze syntax, semantics, and information content of human language.
These seemingly very different tasks are usually solved by specially designed architectures.
In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks.
We perform extensive experiments to test this insight on 10 disparate tasks as broad as dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving comparable performance as state-of-the-art specialized models.
We further demonstrate benefits in multi-task learning.
We convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.

assistant: text: This paper generalizes a wide range of natural language processing tasks as a single span-based framework and proposes a general architecture to solve all these problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An Explicitly Relational Neural Network Architecture

Paper content:
With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data.
In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity.
We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures.
The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

assistant: text: A network architecture based on the multi-head self-attention module to learn a new form of relational representations, which improves data efficiency and generalization ability on curriculum learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Compressing U-net Using Knowledge Distillation

Paper content:
We study the use of knowledge distillation to compress the U-net architecture.
We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.
This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.
where a y t ij indicates the label of sample t at location (i, j), andŷ t is the probability map predicted 41 by the network.
In the standard setting, this probability map is obtained via the softmax function.
to generate probabilitiesŷ t * for each sample x t in a validation set.
These probabilities are then H(y t ,ŷ t ), for mixed distillation.
After being trained at T > 1, the softmax temperature of the 48 student network is reduced back to 1.
Improving Distillation.
As evidenced by our results in Section 3, using standard distillation, whether 50 vanilla or mixed, did not prove sufficient to distill a standard U-net into a very small one.
To overcome 51 this, we therefore propose two modifications of the original strategy.
First, as indicated in FIG0

assistant: text: We present additional techniques to use knowledge distillation to compress U-net by over 1000x.