ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning From Noisy Singly-labeled Data

Paper content:
Supervised learning depends on annotated examples, which are taken to be the ground truth.
But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk.
Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem).
Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples.
This raises two fundamental questions: (1) How can we best learn from noisy workers?
(2) How should we allocate our labeling budget to maximize the performance of a classifier?
We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.
The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality.
Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality.
We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold.
Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

assistant: text: This paper proposes a method for learning from noisy labels, focusing on the case when data isn't redundantly labeled with theoretical and experimental validation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling Content and Style via Unsupervised Geometry Distillation

Paper content:
It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation in a different and unpredictable way.
It is rare for one to have access to a large number of data to help separate the influences.
In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner.
We address this problem in a two-branch Autoencoder framework.
For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge.
This encourages the branch to distill geometry information.
Another branch learns the complementary style information.
The two branches form an effective framework that can disentangle object's content-style representation without any human annotation.
We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data.
We are able to generate photo-realistic images with 256x256 resolution that are clearly disentangled in content and style.

assistant: text: We present a novel framework to learn the disentangled representation of content and style in a completely unsupervised manner.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Once for All: Train One Network and Specialize it for Efficient Deployment

Paper content:
We address the challenging problem of efficient deep learning model deployment, where the goal is to design neural network architectures that can fit different hardware platform constraints.
Most of the traditional approaches either manually design or use Neural Architecture Search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally expensive and unscalable.
Our key idea is to decouple model training from architecture search to save the cost.
To this end, we propose to train a once-for-all network (OFA) that supports diverse architectural settings (depth, width, kernel size, and resolution).
Given a deployment scenario, we can then quickly get a specialized sub-network by selecting from the OFA network without additional training.
To prevent interference between many sub-networks during training, we also propose a novel progressive shrinking algorithm, which can train a surprisingly large number of sub-networks ($> 10^{19}$) simultaneously.
Extensive experiments on various hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) show that OFA consistently outperforms SOTA NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3) while reducing orders of magnitude GPU hours and $CO_2$ emission.
In particular, OFA achieves a new SOTA 80.0% ImageNet top1 accuracy under the mobile setting ($<$600M FLOPs).
Code and pre-trained models are released at https://github.com/mit-han-lab/once-for-all.

assistant: text: We introduce techniques to train a single once-for-all network that fits many hardware platforms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

Paper content:
Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities.
This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors.
We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks.
Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization.
We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

assistant: text: We present a novel approach for learning to predict sets with unknown permutation and cardinality using feed-forward deep neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Optimization of Sorting Networks via Continuous Relaxations

Paper content:
Sorting input objects is an important step in many machine learning pipelines.
However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization.
In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting operator from permutation matrices to the set of unimodal row-stochastic matrices, where every row sums to one and has a distinct argmax.
This relaxation permits straight-through optimization of any computational graph involve a sorting operation.
Further, we use this relaxation to enable gradient-based stochastic optimization over the combinatorially large space of permutations by deriving a reparameterized gradient estimator for the Plackett-Luce family of distributions over permutations.
We demonstrate the usefulness of our framework on three tasks that require learning semantic orderings of high-dimensional objects, including a fully differentiable, parameterized extension of the k-nearest neighbors algorithm

assistant: text: The paper considers how to sort a number of items without explicitly necessarily learning their actual meanings or values and proposes a method to perform the optimization via a continuous relaxation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Refining the variational posterior through iterative optimization

Paper content:
Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.
A key challenge of variational inference is to approximate the posterior over model parameters with a distribution that is simpler and tractable yet sufficiently expressive.
In this work, we propose a method for training highly flexible variational distributions by starting with a coarse approximation and iteratively refining it.
Each refinement step makes cheap, local adjustments and only requires optimization of simple variational families.
We demonstrate theoretically that our method always improves a bound on the approximation (the Evidence Lower BOund) and observe this empirically across a variety of benchmark tasks.
In experiments, our method consistently outperforms recent variational inference methods for deep learning in terms of log-likelihood and the ELBO.
We see that the gains are further amplified on larger scale models, significantly outperforming standard VI and deep ensembles on residual networks on CIFAR10.

assistant: text: A method for training flexible variational posterior distributions, applied to Bayesian neural nets to perform variation inference (VI) over the weights.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Cakewalk Method

Paper content:
Combinatorial optimization is a common theme in computer science.
While in general such problems are NP-Hard, from a practical point of view, locally optimal solutions can be useful.
In some combinatorial problems however, it can be hard to define meaningful solution neighborhoods that connect large portions of the search space, thus hindering methods that search this space directly.
We suggest to circumvent such cases by utilizing a policy gradient algorithm that transforms the problem to the continuous domain, and to optimize a new surrogate objective that renders the former as generic stochastic optimizer.
This is achieved by producing a surrogate objective whose distribution is fixed and predetermined, thus removing the need to fine-tune various hyper-parameters in a case by case manner.
Since we are interested in methods which can successfully recover locally optimal solutions, we use the problem of finding locally maximal cliques as a challenging experimental benchmark, and we report results on a large dataset of graphs that is designed to test clique finding algorithms.
Notably, we show in this benchmark that fixing the distribution of the surrogate is key to consistently recovering locally optimal solutions, and that our surrogate objective leads to an algorithm that outperforms other methods we have tested in a number of measures.

assistant: text: The paper proposes an approach to construct surrogate objectives for the application of policy gradient methods to combinatorial optimization with the goal of reducing the need of hyper-parameter tuning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Not-So-Random Features

Paper content:
We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels.
Our method produces a sequence of feature maps, iteratively refining the SVM margin.
We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game.
Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.
Choosing the right kernel is a classic question that has riddled machine learning practitioners and theorists alike.
Conventional wisdom instructs the user to select a kernel which captures the structure and geometric invariances in the data.
Efforts to formulate this principle have inspired vibrant areas of study, going by names from feature selection to multiple kernel learning (MKL).We
present a new, principled approach for selecting a translation-invariant or rotation-invariant kernel to maximize the SVM classification margin. We
first describe a kernel-alignment subroutine, which finds a peak in the Fourier transform of an adversarially chosen data-dependent measure. Then
, we define an iterative procedure that produces a sequence of feature maps, progressively improving the margin. The
resulting algorithm is strikingly simple and scalable. Intriguingly
, our analysis interprets the main algorithm as no-regret learning dynamics in a zero-sum min-max game, whose value is the classification margin. Thus, we are
able to quantify convergence guarantees towards the largest margin realizable by a kernel with the assumed invariance. Finally, we
exhibit experiments on synthetic and benchmark datasets, demonstrating consistent improvements over related random features-based kernel methods.
We have presented an efficient kernel learning method that uses tools from Fourier analysis and online learning to optimize over two natural infinite families of kernels.
With this method, we show meaningful improvements on benchmark tasks, compared to related random features-based methods.
Many theoretical questions remain, such as accelerating the search for Fourier peaks (e.g. Hassanieh et al. FORMULA1 ; Kapralov FORMULA1 ).
These, in addition to applying our learned kernels to state-of-the-art methods (e.g. convolutional kernel networks (Mairal et al., 2014; BID21 Mairal, 2016) ), prove to be exciting directions for future work.

assistant: text: The authors propose an interesting algorithm for learning the l1-SVM and the Fourier represented kernel together
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing

Paper content:
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.
Recent works have developed accurate MI estimators through provably low-bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation.
In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018).
Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparamters, reduce overfitting and improve accuracy.
With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes.
We demonstrate the effectiveness of DEMINE on synthetic benchmarks and a real world fMRI dataset, with application of inter-subject correlation analysis.

assistant: text: A new & practical statistical test of dependency using neural networks, benchmarked on synthetic and a real fMRI datasets.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Paper content:
Domain adaptation is critical for success in new, unseen environments.
Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difficult to visualize and sometimes fail to capture pixel-level and low-level domain shifts.
Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at  mapping images between domains, even without the use of aligned image pairs.
We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain Adaptation model.
CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs.
Our model can be applied in a variety of visual recognition and prediction settings.
We show new state-of-the-art results across multiple adaptation tasks, including digit classification and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains.

assistant: text: This paper proposes the use of CycleGANs for Domain Adaptation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Compact Embedding Layers via Differentiable Product Quantization

Paper content:
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings.
Despite their effectiveness, the number of parameters in an embedding layer increases linearly with the number of symbols and poses a critical challenge on memory and storage constraints.
In this work, we propose a generic and end-to-end learnable compression framework termed differentiable product quantization (DPQ).
We present two instantiations of DPQ that leverage different approximation techniques to enable differentiability in end-to-end learning.
Our method can readily serve as a drop-in alternative for any existing embedding layer.
Empirically, DPQ offers significant compression ratios (14-238x) at negligible or no performance cost on 10 datasets across three different language tasks.

assistant: text: We propose a differentiable product quantization framework that can reduce the size of embedding layer in an end-to-end training at no performance cost.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Identifying and Controlling Important Neurons in Neural Machine Translation

Paper content:
Neural machine translation (NMT) models learn representations containing substantial linguistic information.
However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons.
We develop unsupervised methods for discovering important neurons in NMT models.
Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision.
We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena.
Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.

assistant: text: Unsupervised methods for finding, analyzing, and controlling important neurons in NMT
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Natural Language Representation Learning for Natural Language Inference by Projecting Superficial Words out

Paper content:
In natural language inference, the semantics of some words do not affect the inference.
Such information is considered superficial and brings overfitting.
How can we represent and discard such superficial information?
In this paper, we use first order logic (FOL) - a classic technique from meaning representation language – to explain what information is superficial for a given sentence pair.
Such explanation also suggests two inductive biases according to its properties.
We proposed a neural network-based approach that utilizes the two inductive biases.
We obtain substantial improvements over extensive experiments.

assistant: text: An approach to treat natural language inference using first-order logic and to infuse NLI models with logical information to be more robust at inference.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep connectomics networks: Results from neural network architectures inspired from network neuroscience

Paper content:
Claims from the fields of network neuroscience and connectomics suggest that topological models of the brain involving complex networks are of particular use and interest.
The field of deep neural networks has mostly left inspiration from these claims out.
In this paper, we propose three architectures and use each of them to explore the intersection of network neuroscience and deep learning in an attempt to bridge the gap between the two fields.
Using the teachings from network neuroscience and connectomics, we show improvements over the ResNet architecture, we show a possible connection between early training and the spectral properties of the network, and we show the trainability of a DNN based on the neuronal network of C.Elegans.
We have demonstrated three distinct approaches to applying work from network neurosciences and connectomics to deep learning.
Our experiments show improvements over ResNet by the inclusion of skip connections which follow a connectivity pattern with small world properties, a possible connection between early training performance and spectral gap when using expander graphs as the participant graph topology with the node model proposed by BID14 , and the trainability of a DNN based on the neuronal network of C.Elegans with and without freezing the parameters of the convolutional and fully connected layers.In future work, we will examine the impact of other spectral properties of the graph topologies used both in the architectures we proposed and in the RandWire architecture proposed by BID14 .
Additionally, we will explore parameter efficient connectivity patterns which could achieve similar performance to related networks with more parameters TAB0 Deep connectomics networks Figure 5 .
Performance of C.ElegansNet with all parameters frozen except the C.Elegans graph edge weights.

assistant: text: We explore the intersection of network neurosciences and deep learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Factorized Multimodal Representations

Paper content:
Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information.
Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data:
1) models must learn the complex intra-modal and cross-modal interactions for prediction and
2) models must be robust to unexpected missing or noisy modalities during testing.
In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels.
We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors.
Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction.
Modality-specific generative factors are unique for each modality and contain the information required for generating data.
Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets.
Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance.
Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

assistant: text: We propose a model to learn factorized multimodal representations that are discriminative, generative, and interpretable.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Consistent Performance on Atari using Expert Demonstrations

Paper content:
Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games.
We identify three key challenges that any algorithm needs to master in order to perform well on all games:  processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently.
In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games.
A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of 0.999 (instead of 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states.
When tested on a set of 42 Atari games, our algorithm exceeds the performance  of an average human on 40 games using a common set of hyper parameters.

assistant: text: The paper proposes three extensions (Bellman update, temporal consistency loss, and expert demonstration) to DQN to improve the learning performance on Atari games, achieving outperformance over the state-of-the-art results for Atari games.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Global Relational Models of Source Code

Paper content:
Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program.
Recent state-of-the-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g. data-flow relations), which are precise and abundantly available for code.
This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models.
Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer.
In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types.
By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.
Starting with a  graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

assistant: text: Models of source code that combine global and structural features learn more powerful representations of programs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial AutoAugment

Paper content:
Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks.
Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy.
Through finding the best policy in well-designed search space of data augmentation, AutoAugment (Cubuk et al., 2019) can significantly improve validation accuracy on image classification tasks.
However, this approach is not computationally practical for large-scale problems.
In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss.
The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization.
In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network.
Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet.
We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art.
On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model.
On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

assistant: text: A technique called Adversarial AutoAugment which dynamically learns good data augmentation policies during training using an adversarial approach.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Match prediction from group comparison data using neural networks

Paper content:
We explore the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data.
Challenges arise in practice.
As existing state-of-the-art algorithms are tailored to certain statistical models, we have different best algorithms across distinct scenarios.
Worse yet, we have no prior knowledge on the underlying model for a given scenario.
These call for a unified approach that can be universally applied to a wide range of scenarios and achieve consistently high performances.
To this end, we incorporate deep learning architectures so as to reflect the key structural features that most state-of-the-art algorithms, some of which are optimal in certain settings, share in common.
This enables us to infer hidden models underlying a given dataset, which govern in-group interactions and statistical patterns of comparisons, and hence to devise the best algorithm tailored to the dataset at hand.
Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-art algorithms.
It turns out that our framework consistently leads to the best performance across all datasets in terms of cross entropy loss and prediction accuracy, while the state-of-the-art algorithms suffer from inconsistent performances across different datasets.
Furthermore, we show that it can be easily extended to attain satisfactory performances in rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.

assistant: text: We investigate the merits of employing neural networks in the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Branched Multi-Task Networks: Deciding What Layers To Share

Paper content:
In the context of multi-task learning, neural networks with branched architectures have often been employed to jointly tackle the tasks at hand.
Such ramified networks typically start with a number of shared layers, after which different tasks branch out into their own sequence of layers.
Understandably, as the number of possible network configurations is combinatorially large, deciding what layers to share and where to branch out becomes cumbersome.
Prior works have either relied on ad hoc methods to determine the level of layer sharing, which is suboptimal, or utilized neural architecture search techniques to establish the network design, which is considerably expensive.
In this paper, we go beyond these limitations and propose a principled approach to automatically construct branched multi-task networks, by leveraging the employed tasks' affinities.
Given a specific budget, i.e. number of learnable parameters, the proposed approach generates architectures, in which shallow layers are task-agnostic, whereas deeper ones gradually grow more task-specific.
Extensive experimental analysis across numerous, diverse multi-tasking datasets shows that, for a given budget, our method consistently yields networks with the highest performance, while for a certain performance threshold it requires the least amount of learnable parameters.

assistant: text: A method for the automated construction of branched multi-task networks with strong experimental evaluation on diverse multi-tasking datasets.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Few-Shot Regression via Learned Basis Functions

Paper content:
The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.
Previous few-shot learning works have mainly focused on classification and reinforcement learning.
In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.
Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of appropriate basis functions.
This enables a few labelled samples to approximate the function.
We design a Feature Extractor network to encode basis functions for a task distribution, and a  Weights Generator to generate the weight vector for a novel task.
We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.
Regression deals with the problem of learning a model relating a set of inputs to a set of outputs.
The learned model can be thought as function y = F (x) that will give a prediction y ∈ R dy given input x ∈ R dx where d y and d x are dimensions of the output and input respectively.
Typically, a regression model is trained on a large number of data points to be able to provide accurate predictions of new inputs.
Recently, there have been a surge in popularity on few-shot learning methods (Vinyals et al., 2016; BID7 BID4 .
Few-shot learning methods require only a few examples from each task to be able to quickly adapt and perform well on a new task.
The fewshot learning model in essence is learning to learn i.e. the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample.In this work, we propose a few shot learning model that targets few-shot regression tasks.
We evaluate our model on the sinusoidal regression tasks and compare our model's performance to several meta-learning algorithms.
We further introduce two more regression tasks, namely the 1D heat equation task modeled by partial differential equations and the 2D Gaussian distribution task.
We propose a few-shot meta learning system that focuses exclusively on regression tasks.
Our model is based on the idea of linear representation of basis functions.
We design a Feature extractor network to encode the basis functions for the entire task distribution.
We design a Weight generator network to generate the weights from the K training samples of a novel task drawn from the same task distribution.
We show that our model has competitive performance in in various few short regression tasks.
A TECHNICAL DETAILS

assistant: text: A method that learns a regression model with a few samples and outperforms other methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intrinsic Motivation for Encouraging Synergistic Behavior

Paper content:
We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually.
Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent.
We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy.
While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken.
We validate our approach in robotic bimanual manipulation tasks with sparse rewards; we find that our approach yields more efficient learning than both
1) training with only the sparse reward and
2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior.
Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.

assistant: text: The paper focuses on using intrinsic motivation to improve the exploration process of reinforcement learning agents in tasks that require multi-agent to achieve.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Functional Annotation of Human Cognitive States using Graph Convolution Networks

Paper content:
A key goal in neuroscience is to understand brain mechanisms of cognitive functions.
An emerging approach is to study “brain states” dynamics using functional magnetic resonance imaging (fMRI).
So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identified from very short time series.
In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used.
Starting with a populational brain graph with nodes defined by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-specific graph representations, and then predicts the corresponding cognitive state.
We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data.
Using a 10-second window, the 21 cognitive states were identified with an excellent average test accuracy of 89% (chance level 4.8%).
As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains.

assistant: text: Using a 10s window of fMRI signals, our GCN model identified 21 different task conditions from HCP dataset with a test accuracy of 89%.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition

Paper content:
In this paper we study the problem of learning the weights of a deep convolutional neural network.
We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.
We develop an algorithm for simultaneously learning all the kernels from the training data.
Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.
We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.
We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.
Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

assistant: text: We consider a simplified deep convolutional neural network model. We show that all layers of this network can be approximately learned with a proper application of tensor decomposition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Counting the Paths in Deep Neural Networks as a Performance Predictor

Paper content:
We propose a novel quantitative measure to predict the performance of a deep neural network classifier, where the measure is derived exclusively from the graph structure of the network.
We expect that this measure is a fundamental first step in developing a method to evaluate new network architectures and reduce the reliance on the computationally expensive trial and error or "brute force" optimisation processes involved in model selection.
The measure is derived in the context of multi-layer perceptrons (MLPs), but the definitions are shown to be useful also in the context of deep convolutional neural networks (CNN), where it is able to estimate and compare the relative performance of different types of neural networks, such as VGG, ResNet, and DenseNet.
Our measure is also used to study the effects of some important "hidden" hyper-parameters of the DenseNet architecture, such as number of layers, growth rate and the dimension of 1x1 convolutions in DenseNet-BC.
Ultimately, our measure facilitates the optimisation of the DenseNet design, which shows improved results compared to the baseline.

assistant: text: The paper presents a method for counting paths in deep neural networks that arguably can be used to measure the performance of the network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Rewiring: Training very sparse deep networks

Paper content:
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them.
But also generic hardware and software implementations of deep learning run more efficiently for sparse networks.
Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints.
We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network.
DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded.
We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance.
DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

assistant: text: The paper presents Deep Rewiring, an algorithm that can be used to train deep neural networks when the network connectivity is severely constrained during training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Few-Shot Regression via Learned Basis Functions

Paper content:
The recent rise in popularity of few-shot learning algorithms has enabled models to quickly adapt to new tasks based on only a few training samples.
Previous few-shot learning works have mainly focused on classification and reinforcement learning.
In this paper, we propose a few-shot meta-learning system that focuses exclusively on regression tasks.
Our model is based on the idea that the degree of freedom of the unknown function can be significantly reduced if it is represented as a linear combination of a set of appropriate basis functions.
This enables a few labelled samples to approximate the function.
We design a Feature Extractor network to encode basis functions for a task distribution, and a  Weights Generator to generate the weight vector for a novel task.
We show that our model outperforms the current state of the art meta-learning methods in various regression tasks.
Regression deals with the problem of learning a model relating a set of inputs to a set of outputs.
The learned model can be thought as function y = F (x) that will give a prediction y ∈ R dy given input x ∈ R dx where d y and d x are dimensions of the output and input respectively.
Typically, a regression model is trained on a large number of data points to be able to provide accurate predictions of new inputs.
Recently, there have been a surge in popularity on few-shot learning methods (Vinyals et al., 2016; BID7 BID4 .
Few-shot learning methods require only a few examples from each task to be able to quickly adapt and perform well on a new task.
The fewshot learning model in essence is learning to learn i.e. the model learns to quickly adapt itself to new tasks rather than just learning to give the correct prediction for a particular input sample.In this work, we propose a few shot learning model that targets few-shot regression tasks.
We evaluate our model on the sinusoidal regression tasks and compare our model's performance to several meta-learning algorithms.
We further introduce two more regression tasks, namely the 1D heat equation task modeled by partial differential equations and the 2D Gaussian distribution task.
We propose a few-shot meta learning system that focuses exclusively on regression tasks.
Our model is based on the idea of linear representation of basis functions.
We design a Feature extractor network to encode the basis functions for the entire task distribution.
We design a Weight generator network to generate the weights from the K training samples of a novel task drawn from the same task distribution.
We show that our model has competitive performance in in various few short regression tasks.
A TECHNICAL DETAILS

assistant: text: This paper proposes a novel shot-learning method for small sample regression problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Paper content:
This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations.
To this end, we propose a multi-modal learning framework that links the inference stage and generation stage.
First, the inference networks are trained to match the speaker identity between the two different modalities.
Then the pre-trained inference networks cooperate with the generation network by giving conditional information about the voice.

assistant: text: This work aims to build one conditional face image generation framework from the audio signal.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Provably Correct Driver Assistance Systems through Stochastic Cognitive Modeling

Paper content:
The aim of this study is to introduce a formal framework for analysis and synthesis of driver assistance systems.
It applies formal methods to the verification of a stochastic human driver model built using the cognitive architecture ACT-R, and then bootstraps safety in semi-autonomous vehicles through the design of provably correct Advanced Driver Assistance Systems.
The main contributions include the integration of probabilistic ACT-R models in the formal analysis of semi-autonomous systems and an abstraction technique that enables a finite representation of a large dimensional, continuous system in the form of a Markov model.
The effectiveness of the method is illustrated in several case studies under various conditions.

assistant: text: Verification of a human driver model based on a cognitive architecture and synthesis of a correct-by-construction ADAS from it.