ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-Task Learning by Deep Collaboration and Application in Facial Landmark Detection

Paper content:
Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains.
While CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant.
Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem.
The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task.
While recent results in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem.
In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework.
We define connectivity in terms of two distinct non-linear transformation blocks.
One aggregates task-specific features into global features, while the other merges back the global features with each task-specific network.
Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features.
To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include.
Experimental results show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches.
We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related.

assistant: text: We propose a novel approach for connecting task-specific networks in a multi-task learning setting based on recent residual network advances.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Downsampling leads to Image Memorization in Convolutional Autoencoders

Paper content:
Memorization of data in deep neural networks has become a subject of significant research interest.
In this paper, we link memorization of  images in deep convolutional autoencoders to downsampling through strided convolution.
To analyze this mechanism in a simpler setting, we train linear convolutional autoencoders and show that linear combinations of training data are stored as eigenvectors in the linear operator corresponding to the network when downsampling is used.
On the other hand, networks without downsampling do not memorize training data.
We provide further evidence that the same effect happens in nonlinear networks.
Moreover, downsampling in nonlinear networks causes the model to not only memorize just linear combinations of images, but individual training images.
Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important phenomenon of memorization in over-parameterized deep networks.

assistant: text: We identify downsampling as a mechansim for memorization in convolutional autoencoders.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DEEP DENSITY NETWORKS AND UNCERTAINTY IN RECOMMENDER SYSTEMS

Paper content:
Building robust online content recommendation systems requires learning com- plex interactions between user preferences and content features.
The field has evolved rapidly in recent years from traditional multi-arm bandit and collabora- tive filtering techniques, with new methods integrating Deep Learning models that enable to capture non-linear feature interactions.
Despite progress, the dynamic nature of online recommendations still poses great challenges, such as finding the delicate balance between exploration and exploitation.
In this paper we provide a novel method, Deep Density Networks (DDN) which deconvolves measurement and data uncertainty and predicts probability densities of CTR, enabling us to perform more efficient exploration of the feature space.
We show the usefulness of using DDN online in a real world content recommendation system that serves billions of recommendations per day, and present online and offline results to eval- uate the benefit of using DDN.

assistant: text: We have introduced Deep Density Network, a unified DNN model to estimate uncertainty for exploration/exploitation in recommendation systems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Probabilistic Binary Neural Networks

Paper content:
Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks.
In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called PBNet.
By embracing stochasticity during training, we circumvent the need to approximate the gradient of functions for which the derivative is zero almost always, such as $\textrm{sign}(\cdot)$, while still obtaining a fully Binary Neural Network at test time.
Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution.
Since all operations in a layer of the PBNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time.
We evaluate two related training methods for the PBNet: one in which activation distributions are propagated throughout the network, and one in which binary activations are sampled in each layer.
Our experiments indicate that sampling the binary activations is an important element for stochastic training of binary Neural Networks.

assistant: text: We introduce a stochastic training method for training Binary Neural Network with both binary weights and activations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder

Paper content:
Human annotation for syntactic parsing is expensive, and large resources are available only for a  fraction of languages.
A question we ask is whether one can leverage abundant unlabeled texts to improve syntactic parsers, beyond just using the texts to obtain more generalisable lexical features (i.e. beyond word embeddings).
To this end, we propose a novel latent-variable generative model for semi-supervised syntactic dependency parsing.
As exact inference is intractable, we introduce a differentiable relaxation to obtain approximate samples and compute gradients with respect to the parser parameters.
Our method (Differentiable Perturb-and-Parse) relies on differentiable dynamic programming over stochastically perturbed edge scores.
We demonstrate effectiveness of our approach with experiments on English, French and Swedish.

assistant: text: Differentiable dynamic programming over perturbed input weights with application to semi-supervised VAE
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variational Hashing-based Collaborative Filtering with Self-Masking

Paper content:
Hashing-based collaborative filtering learns binary vector representations (hash codes) of users and items, such that recommendations can be computed very efficiently using the Hamming distance, which is simply the sum of differing bits between two hash codes.
A problem with hashing-based collaborative filtering using the Hamming distance, is that each bit is equally weighted in the distance computation, but in practice some bits might encode more important properties than other bits, where the importance depends on the user.
To this end, we propose an end-to-end trainable variational hashing-based collaborative filtering approach that uses the novel concept of self-masking: the user hash code acts as a mask on the items (using the Boolean AND operation), such that it learns to encode which bits are important to the user, rather than the user's preference towards the underlying item property that the bits represent.
This allows a binary user-level importance weighting of each item without the need to store additional weights for each user.
We experimentally evaluate our approach against state-of-the-art baselines on 4 datasets, and obtain significant gains of up to 12% in NDCG.
We also make available an efficient implementation of self-masking, which experimentally yields <4% runtime overhead compared to the standard Hamming distance.

assistant: text: We propose a new variational hashing-based collaborative filtering approach optimized for a novel self-mask variant of the Hamming distance, which outperforms state-of-the-art by up to 12% on NDCG.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Critical Learning Periods in Deep Networks

Paper content:
Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of a skill.
The extent of the impairment depends on the onset and length of the deficit window, as in animal models, and on the size of the neural network.
Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training.
To better understand this phenomenon, we use the Fisher Information of the weights to measure the effective connectivity between layers of a network during training.
Counterintuitively, information rises rapidly in the early phases of training, and then decreases, preventing redistribution of information resources in a phenomenon we refer to as a loss of "Information Plasticity".
Our analysis suggests that the first few epochs are critical for the creation of strong connections that are optimal relative to the input data distribution.
Once such strong connections are created, they do not appear to change during additional training.
These findings suggest that the initial learning transient, under-scrutinized compared to asymptotic behavior, plays a key role in determining the outcome of the training process.
Our findings, combined with recent theoretical results in the literature, also suggest that forgetting (decrease of information in the weights) is critical to achieving invariance and disentanglement in representation learning.
Finally, critical periods are not restricted to biological systems, but can emerge naturally in learning systems, whether biological or artificial, due to fundamental constrains arising from learning dynamics and information processing.

assistant: text: Sensory deficits in early training phases can lead to irreversible performance loss in both artificial and neuronal networks, suggesting information phenomena as the common cause, and point to the importance of the initial transient and forgetting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spectral Convolutional Networks on Hierarchical Multigraphs

Paper content:
Spectral Graph Convolutional Networks (GCNs) are a generalization of convolutional networks to learning on graph-structured data.
Applications of spectral GCNs have been successful, but limited to a few problems where the graph is fixed, such as shape correspondence and node classification.
In this work, we address this limitation by revisiting a particular family of spectral graph networks, Chebyshev GCNs, showing its efficacy in solving graph classification tasks with a variable graph structure and size.
Current GCNs also restrict graphs to have at most one edge between any pair of nodes.
To this end, we propose a novel multigraph network that learns from multi-relational graphs.
We explicitly model different types of edges: annotated edges, learned edges with abstract meaning, and hierarchical edges.
We also experiment with different ways to fuse the representations extracted from different edge types.
This restriction is sometimes implied from a dataset, however, we relax this restriction for all kinds of datasets.
We achieve state-of-the-art results on a variety of chemical, social, and vision graph classification benchmarks.

assistant: text: A novel approach to graph classification based on spectral graph convolutional networks and its extension to multigraphs with learnable relations and hierarchical structure. We show state-of-the art results on chemical, social and image datasets.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Non-Sequential Melody Generation

Paper content:
In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components.
Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies.
Here, we use DCGAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel.

assistant: text: Representing melodies as images with semantic units aligned we can generate them using a DCGAN without any recurrent components.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Learning with Warped Gradient Descent

Paper content:
Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning.
Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule.
Both these approaches pose challenges.
On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour.
On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation.
In this work we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations.
WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution.
Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner.
Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates.
WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems.
We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.

assistant: text: We propose a novel framework for meta-learning a gradient-based update rule that scales to beyond few-shot learning and is applicable to any form of learning, including continual learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Evolving intrinsic motivations for altruistic behavior

Paper content:
Multi-agent cooperation is an important feature of the natural world.
Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate.
Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory.
Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp.
By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way.
To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection.
We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.

assistant: text: We introduce a biologically-inspired modular evolutionary algorithm in which deep RL agents learn to cooperate in a difficult multi-agent social game, which could help to explain the evolution of altruism.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised  Video-to-Video Translation via Self-Supervised Learning

Paper content:
Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent.
In this work, we propose a novel unsupervised video-to-video translation model.
Our model decomposes the style and the content, uses specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units.
The style-content decomposition mechanism enables us to achieve long-term style-consistent video translation results as well as provides us with a good interface for modality flexible translation.
In addition, by changing the input frames and style codes incorporated in our translation, we propose a video interpolation loss, which captures temporal information within the sequence to train our building blocks in a self-supervised manner.
Our model can produce photo-realistic, spatio-temporal consistent translated videos in a multimodal way.
Subjective and objective experimental results validate the superiority of our model over the existing methods.

assistant: text: A temporally consistent and modality flexible unsupervised video-to-video translation framework trained in a self-supervised manner.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: EMS: End-to-End Model Search for Network Architecture, Pruning and Quantization

Paper content:
We present an end-to-end design methodology for efficient deep learning deployment.
Unlike previous methods that separately optimize the neural network architecture, pruning policy, and quantization policy, we jointly optimize them in an end-to-end manner.
To deal with the larger design space it brings, we train a quantization-aware accuracy predictor that fed to the evolutionary search to select the best fit.
We first generate a large dataset of <NN architecture, ImageNet accuracy> pairs without training each architecture, but by sampling a unified supernet.
Then we use these data to train an accuracy predictor without quantization, further using predictor-transfer technique to get the quantization-aware predictor, which reduces the amount of post-quantization fine-tuning time.
Extensive experiments on ImageNet show the benefits of the end-to-end methodology: it maintains the same accuracy (75.1%) as ResNet34 float model while saving 2.2× BitOps comparing with the 8-bit model; we obtain the same level accuracy as MobileNetV2+HAQ while achieving 2×/1.3× latency/energy saving; the end-to-end optimization outperforms separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy while reducing orders of magnitude GPU hours and CO2 emission.

assistant: text: We present an end-to-end design methodology for efficient deep learning deployment.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile

Paper content:
Owing to their connection with generative adversarial networks (GANs), saddle-point problems have recently attracted considerable interest in machine learning and beyond.
By necessity, most theoretical guarantees revolve around convex-concave (or even linear) problems; however, making theoretical inroads towards efficient GAN training depends crucially on moving beyond this classic framework.
To make piecemeal progress along these lines, we analyze the behavior of mirror descent (MD) in a class of non-monotone problems whose solutions coincide with those of a naturally associated variational inequality – a property which we call coherence.
We first show that ordinary, “vanilla” MD converges under a strict version of this condition, but not otherwise; in particular, it may fail to converge even in bilinear models with a unique solution.
We then show that this deficiency is mitigated by optimism: by taking an “extra-gradient” step, optimistic mirror descent (OMD) converges in all coherent problems.
Our analysis generalizes and extends the results of Daskalakis et al. [2018] for optimistic gradient descent (OGD) in bilinear problems, and makes concrete headway for provable convergence beyond convex-concave games.
We also provide stochastic analogues of these results, and we validate our analysis by numerical experiments in a wide array of GAN models (including Gaussian mixture models, and the CelebA and CIFAR-10 datasets).

assistant: text: We show how the inclusion of an extra-gradient step in first-order GAN training methods can improve stability and lead to improved convergence results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Extracting and Leveraging Feature Interaction Interpretations

Paper content:
Recommendation is a prevalent application of machine learning that affects many users; therefore, it is crucial for recommender models to be accurate and interpretable.
In this work, we propose a method to both interpret and augment the predictions of black-box recommender systems.
In particular, we propose to extract feature interaction interpretations from a source recommender model and explicitly encode these interactions in a target recommender model, where both source and target models are black-boxes.
By not assuming the structure of the recommender system, our approach can be used in general settings.
In our experiments, we focus on a prominent use of machine learning recommendation: ad-click prediction.
We found that our interaction interpretations are both informative and predictive, i.e., significantly outperforming existing recommender models.
What's more, the same approach to interpreting interactions can  provide new insights into domains even beyond recommendation.

assistant: text: Proposed a method to extract and leverage interpretations of feature interactions
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Clustering using Pseudo-semi-supervised Learning

Paper content:
In this paper, we propose a framework that leverages semi-supervised models to improve unsupervised clustering performance.
To leverage semi-supervised models, we first need to automatically generate labels, called pseudo-labels.
We find that prior approaches for generating pseudo-labels hurt clustering performance because of their low accuracy.
Instead, we use an ensemble of deep networks  to construct a similarity graph, from which we extract high accuracy pseudo-labels.
The approach of finding high quality pseudo-labels using ensembles and training the semi-supervised model is iterated, yielding continued improvement.
We show that our approach outperforms state of the art clustering results for multiple image and text datasets.
For example, we achieve 54.6% accuracy for CIFAR-10 and 43.9% for 20news, outperforming state of the art by 8-12% in absolute terms.

assistant: text: Using ensembles and pseudo labels for unsupervised clustering
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Normalization Gradients are Least-squares Residuals

Paper content:
Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks.
Discussions of why this normalization works so well remain unsettled.
We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN.
We recast the back-propagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations.
This view, which we term {\em gradient-least-squares}, is an extensible and arithmetically accurate description of BN.
To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.

assistant: text: Gaussian normalization performs a least-squares fit during back-propagation, which zero-centers and decorrelates partial derivatives from normalized activations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: FAST OBJECT LOCALIZATION VIA SENSITIVITY ANALYSIS

Paper content:
Deep Convolutional Neural Networks (CNNs) have been repeatedly shown to perform well on image classification tasks, successfully recognizing a broad array of objects when given sufficient training data.
Methods for object localization, however, are still in need of substantial improvement.
Common approaches to this problem involve the use of a sliding window, sometimes at multiple scales, providing input to a deep CNN trained to classify the contents of the window.
In general, these approaches are time consuming, requiring many classification calculations.
In this paper, we offer a fundamentally different approach to the localization of recognized objects in images.
Our method is predicated on the idea that a deep CNN capable of recognizing an object must implicitly contain knowledge about object location in its connection weights.
We provide a simple method to interpret classifier weights in the context of individual classified images.
This method involves the calculation of the derivative of network generated activation patterns, such as the activation of output class label units, with regard to each in- put pixel, performing a sensitivity analysis that identifies the pixels that, in a local sense, have the greatest influence on internal representations and object recognition.
These derivatives can be efficiently computed using a single backward pass through the deep CNN classifier, producing a sensitivity map of the image.
We demonstrate that a simple linear mapping can be learned from sensitivity maps to bounding box coordinates, localizing the recognized object.
Our experimental results, using real-world data sets for which ground truth localization information is known, reveal competitive accuracy from our fast technique.

assistant: text: Proposing a novel object localization(detection) approach based on interpreting the deep CNN using internal representation and network's thoughts
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Kronecker Recurrent Units

Paper content:
Our work addresses two important issues with recurrent neural networks: (1) they are over-parameterized, and (2) the recurrent weight matrix is ill-conditioned.
The former increases the sample complexity of learning and the training time.
The latter causes the vanishing and exploding gradient problem.
We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU).
KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix.
It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors.
Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient.
Our experimental results on seven standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance.
These results in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced.

assistant: text: Out work presents a Kronecker factorization of recurrent weight matrices for parameter efficient and well conditioned recurrent neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

Paper content:
State-of-the-art machine learning methods exhibit limited compositional generalization.
At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements.
We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks.
We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures.
We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy.
We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

assistant: text: Benchmark and method to measure compositional generalization by maximizing divergence of compound frequency at small divergence of atom frequency.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Multi-Task by Active Sampling

Paper content:
One of the long-standing challenges in Artificial Intelligence for learning goal-directed behavior is to build a single agent which can solve multiple tasks.
Recent progress in multi-task learning for goal-directed sequential problems has been in the form of distillation based learning wherein a student network learns from multiple task-specific expert networks by mimicking the task-specific policies of the expert networks.
While such approaches offer a promising solution to the multi-task learning problem, they require supervision from large expert networks which require extensive data and computation time for training.
In this work, we propose an efficient multi-task learning framework which solves multiple goal-directed tasks in an on-line setup without the need for expert supervision.
Our work uses active learning principles to achieve multi-task learning by sampling the harder tasks more than the easier ones.
We propose three distinct models under our active sampling framework.
An adaptive method with extremely competitive multi-tasking performance.
A UCB-based meta-learner which casts the problem of picking the next task to train on as a multi-armed bandit problem.
A meta-learning method that casts the next-task picking problem as a full Reinforcement Learning problem and uses actor-critic methods for optimizing the multi-tasking performance directly.
We demonstrate results in the Atari 2600 domain on seven multi-tasking instances: three 6-task instances, one 8-task instance, two 12-task instances and one 21-task instance.

assistant: text: Letting a meta-learner decide the task to train on for an agent in a multi-task setting improves multi-tasking ability substantially
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Long-term Forecasting using Tensor-Train RNNs

Paper content:
We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics.
Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation.
Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions.
Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance.
We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs.
We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.

assistant: text: Accurate forecasting over very long time horizons using tensor-train RNNs
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Residual Connections Encourage Iterative Inference

Paper content:
Residual networks (Resnets) have become a prominent architecture in deep learning.
However, a comprehensive understanding of Resnets is still a topic of ongoing research.
A recent view argues that Resnets perform iterative refinement of features.
We attempt to further expose properties of this aspect.
To this end, we study Resnets both analytically and empirically.
We formalize the notion of iterative refinement in Resnets by showing that residual architectures naturally encourage features to move along the negative gradient of loss during the feedforward phase.
In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement.
In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features.
Finally we observe that sharing residual layers naively leads to representation explosion and hurts generalization performance, and show that simple existing strategies can help alleviating this problem.

assistant: text: Residual connections really perform iterative inference
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality

Paper content:
Granger causality is a widely-used criterion for analyzing interactions in large-scale networks.
As most physical interactions are inherently nonlinear, we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements.
Our proposed approach relies on modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units (SRUs).
We make a case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes’ time series measurements.
We propose a variant of SRU, called economy-SRU, which, by design has considerably fewer trainable parameters, and therefore less prone to overfitting.
The economy-SRU computes a low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing.
Additionally, the internal weight parameters of the economy-SRU are strategically regularized in a group-wise manner to facilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events.
Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP, LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality.

assistant: text: A new recurrent neural network architecture for detecting pairwise Granger causality between nonlinearly interacting time series.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
A fundamental trait of intelligence is the ability to achieve goals in the face of novel circumstances.
In this work, we address one such setting which requires solving a task with a novel set of actions.
Empowering machines with this ability requires generalization in the way an agent perceives its available actions along with the way it uses these actions to solve tasks.
Hence, we propose a framework to enable generalization over both these aspects: understanding an action’s functionality, and using actions to solve tasks through reinforcement learning.
Specifically, an agent interprets an action’s behavior using unsupervised representation learning over a collection of data samples reflecting the diverse properties of that action.
We employ a reinforcement learning architecture which works over these action representations, and propose regularization metrics essential for enabling generalization in a policy.
We illustrate the generalizability of the representation learning method and policy, to enable zero-shot generalization to previously unseen actions on challenging sequential decision-making environments.
Our results and videos can be found at sites.google.com/view/action-generalization/

assistant: text: We address the problem of generalization of reinforcement learning to unseen action spaces.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Large Scale Optimal Transport and Mapping Estimation

Paper content:
This paper presents a novel two-step approach for the fundamental problem of learning an optimal map from one distribution to another.
First, we learn an optimal transport (OT) plan, which can be thought as a one-to-many map between the two distributions.
To that end, we propose a stochastic dual approach of regularized OT, and show empirically that it scales better than a recent related approach when the amount of samples is very large.
Second, we estimate a Monge map as a deep neural network learned by approximating the barycentric projection of the previously-obtained OT plan.
This parameterization allows generalization of the mapping outside the support of the input measure.
We prove two theoretical stability results of regularized OT which show that our estimations converge to the OT and Monge map between the underlying continuous measures.
We showcase our proposed approach on two applications: domain adaptation and generative modeling.

assistant: text: Learning optimal mapping with deepNN between distributions along with theoretical guarantees.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Weaknesses of Reinforcement Learning for Neural Machine Translation

Paper content:
Reinforcement learning (RL) is frequently used to increase performance in text generation tasks,
including machine translation (MT),
notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN).
However, little is known about what and how these methods learn in the context of MT.
We prove that one of the most common RL methods for MT does not optimize the
expected reward, as well as show that other methods take an infeasibly long time to converge.
In fact, our results suggest that RL practices in MT are likely to improve performance
only where the pre-trained parameters are already close to yielding the correct translation.
Our findings further suggest that observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve.

assistant: text: Reinforcment practices for machine translation performance gains might not come from better predictions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Simple Technique to Enable Saliency Methods to Pass the Sanity Checks

Paper content:
{\em Saliency methods} attempt to explain a deep net's decision by assigning a {\em score} to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input.
Recently \citet{adebayosan} questioned the validity of many of these methods since they do not pass simple {\em sanity checks}, which test whether the scores shift/vanish when  layers of the trained net are randomized, or when the net is retrained using random labels for inputs.
% for the inputs.
%Surprisingly, the tested methods did not pass these checks: the explanations were relatively unchanged
. 

We propose a simple fix to existing saliency methods that helps them pass sanity checks, which we call {\em competition for pixels}.
This involves computing saliency maps for all possible labels in the classification task, and using a simple competition among them to identify and remove less relevant pixels from the map.
Some theoretical justification is provided for it  and its performance is empirically demonstrated on several popular methods.

assistant: text: We devise a mechanism called competition among pixels that allows (approximately) complete saliency methods to pass the sanity checks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?

Paper content:
Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization.
The expense of producing these examples during training often precludes adversarial training from use on complex image datasets.
In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing.
Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples.

assistant: text: Achieving strong adversarial robustness comparable to adversarial training without training on adversarial examples
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Poincaré Wasserstein Autoencoder

Paper content:
This work presents the Poincaré Wasserstein Autoencoder, a reformulation of
the recently proposed Wasserstein autoencoder framework on a non-Euclidean
manifold, the Poincaré ball model of the hyperbolic space H n .
By assuming the
latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure
on the learned latent space representations.
We show that for datasets with latent
hierarchies, we can recover the structure in a low-dimensional latent space.
We
also demonstrate the model in the visual domain to analyze some of its properties
and show competitive results on a graph link prediction task.

assistant: text: Wasserstein Autoencoder with hyperbolic latent space
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Multiple Mappings: an Evaluation of Interference, Transfer, and Retention with Chorded Shortcut Buttons

Paper content:
Touch interactions with current mobile devices have limited expressiveness.
Augmenting devices with additional degrees of freedom can add power to the interaction, and several augmentations have been proposed and tested.
However, there is still little known about the effects of learning multiple sets of augmented interactions that are mapped to different applications.
To better understand whether multiple command mappings can interfere with one another, or affect transfer and retention, we developed a prototype with three pushbuttons on a smartphone case that can be used to provide augmented input to the system.
The buttons can be chorded to provide seven possible shortcuts or transient mode switches.
We mapped these buttons to three different sets of actions, and carried out a study to see if multiple mappings affect learning and performance, transfer, and retention.
Our results show that all of the mappings were quickly learned and there was no reduction in performance with multiple mappings.
Transfer to a more realistic task was successful, although with a slight reduction in accuracy.
Retention after one week was initially poor, but expert performance was quickly restored.
Our work provides new information about the design and use of augmented input in mobile interactions.

assistant: text: Describes a study investigating interference, transfer, and retention of multiple mappings with the same set of chorded buttons
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment.
This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large  temporal contexts into account.
Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the  state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset.
We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music.
We see this initial result as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems.
The remainder of this paper is structured as follows.
In section 2, we briefly review related work 27 from the literature.
In section 3, we introduce briefly the Wave-U-Net architecture and its application architectures like that of BID0 .
Recently, the U-Net architecture on magnitude spectrograms has for each prediction and is based on the repeated application of dilated convolutions with exponentially increasing dilation factors to factor in context information.
yield an estimate of the target sources, a tanh nonlinearity follows, succeeded by a final LeakyReLU.
In applying the Wave-U-Net architecture to the application of speech enhancement, our objective 58 is to separate a mixture waveform DISPLAYFORM0
To evaluate and compare the quality of the enhanced speech yielded by the Wave-U-Net, we mirror filter sizes, to the task and expanding to multi-channel audio and multi-source-separation.

assistant: text: The Wave-U-Net architecture, recently introduced by Stoller et al for music source separation, is highly effective for speech enhancement, beating the state of the art.