ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Understanding and Improving Information Transfer in Multi-Task Learning

Paper content:
We investigate multi-task learning approaches which use a shared feature representation for all tasks.
To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task.
We study the theory of this setting on linear and ReLU-activated models.
Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning.
We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer.
Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtained a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT LARGE using our alignment method.
We also design an SVD-based task re-weighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.

assistant: text: A Theoretical Study of Multi-Task Learning with Practical Implications for Improving Multi-Task Training and Transfer Learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Data Augmentation for Rumor Detection Using Context-Sensitive Neural Language Model With Large-Scale Credibility Corpus

Paper content:
In this paper, we address the challenge of limited labeled data and class imbalance problem for machine learning-based rumor detection on social media.
We present an offline data augmentation method based on semantic relatedness for rumor detection.
To this end, unlabeled social media data is exploited to augment limited labeled data.
A context-aware neural language model and a large credibility-focused Twitter corpus are employed to learn effective representations of rumor tweets for semantic relatedness measurement.
A language model fine-tuned with the a large domain-specific corpus shows a dramatic improvement on training data augmentation for rumor detection over pretrained language models.
We conduct experiments on six different real-world events based on five publicly available data sets and one augmented data set.
Our experiments show that the proposed method allows us to generate a larger training data with reasonable quality via weak supervision.
We present preliminary results achieved using a state-of-the-art neural network model with augmented data for rumor detection.

assistant: text: We propose a methodology of augmenting publicly available data for rumor studies based on samantic relatedness between limited labeled and unlabeled data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields

Paper content:
Generative adversarial networks (GANs) evolved into one of the most successful unsupervised techniques for generating realistic images.
Even though it has recently been shown that GAN training converges, GAN models often end up in local Nash equilibria that are associated with mode collapse or otherwise fail to model the target distribution.
We introduce Coulomb GANs, which pose the GAN learning problem as a potential field, where generated samples are attracted to training set samples but repel each other.
The discriminator learns a potential field while the generator decreases the energy by moving its samples along the vector (force) field determined by the gradient of the potential field.
Through decreasing the energy, the GAN model learns to generate samples according to the whole target distribution and does not only cover some of its modes.
We prove that Coulomb GANs possess only one Nash equilibrium which is optimal in the sense that the model distribution equals the target distribution.
We show the efficacy of Coulomb GANs on LSUN bedrooms, CelebA faces, CIFAR-10 and the Google Billion Word text generation.

assistant: text: Coulomb GANs can optimally learn a distribution by posing the distribution learning problem as optimizing a potential field
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Paper content:
For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset.
In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks.
By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks.
GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models.
We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task.
However, the low absolute performance of our best model indicates the need for improved general NLU systems.

assistant: text: We present a multi-task benchmark and analysis platform for evaluating generalization in natural language understanding systems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradients explode - Deep Networks are shallow - ResNet explained

Paper content:
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice.
We explain why exploding gradients occur and highlight the {\it collapsing domain problem}, which can arise in architectures that avoid exploding gradients.
ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property.
By noticing that {\it any neural network is a residual network}, we devise the {\it residual trick}, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

assistant: text: We show that in contras to popular wisdom, the exploding gradient problem has not been solved and that it limits the depth to which MLPs can be effectively trained. We show why gradients explode and how ResNet handles them.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Paper content:
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years.
It could be time to take a step back and analyze the good and bad aspects in the field of NAS.
A variety of algorithms search architectures under different search space.
These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization.
This raises a comparability problem when comparing the performance of various NAS algorithms.
NAS-Bench-101 has shown success to alleviate this problem.
In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information.
NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms.
The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph.
Each edge here is associated with an operation selected from a predefined operation set.
For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total.
The training log using the same setup and the performance for each architecture candidate are provided for three datasets.
This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself.
The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers.
We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.
In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

assistant: text: A NAS benchmark applicable to almost any NAS algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spreading vectors for similarity search

Paper content:
Discretizing floating-point vectors is a fundamental step of modern indexing methods.
State-of-the-art techniques learn parameters of the quantizers on training data for optimal performance, thus adapting quantizers to the data.
In this work, we propose to reverse this paradigm and adapt the data to the quantizer: we train a neural net whose last layers form a fixed parameter-free quantizer, such as pre-defined points of a sphere.
As a proxy objective, we design and train a neural network that favors uniformity in the spherical latent space, while preserving the neighborhood structure after the mapping.
For this purpose, we propose a new regularizer derived from the Kozachenko-Leonenko differential entropy estimator and combine it with a locality-aware triplet loss.
Experiments show that our end-to-end approach outperforms most learned quantization methods, and is competitive with the state of the art on widely adopted benchmarks.
Further more, we show that training without the quantization step results in almost no difference in accuracy, but yields a generic catalyser that can be applied with any subsequent quantization technique.

assistant: text: We learn a neural network that uniformizes the input distribution, which leads to competitive indexing performance in high-dimensional space
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sparse Deep Scattering Croisé Network

Paper content:
In this work, we propose the Sparse Deep Scattering Croisé Network (SDCSN) a novel architecture based on the Deep Scattering Network (DSN).
The DSN is achieved by cascading  wavelet transform convolutions with a complex modulus and a time-invariant operator.
We extend this work by first,
crossing multiple wavelet family transforms to increase the feature diversity while avoiding any learning.
Thus providing a more informative latent representation and benefit from the development of highly specialized wavelet filters over the last decades.
Beside, by combining all the different wavelet representations, we reduce the amount of prior information needed regarding the signals at hand.
Secondly, we develop an optimal thresholding strategy for over-complete filter banks that regularizes the network and controls instabilities such as inherent non-stationary noise in the signal.
Our systematic and principled solution sparsifies the latent representation of the network by acting as a local mask distinguishing between activity and noise.
Thus, we propose to enhance the DSN by increasing the variance of the scattering coefficients representation as well as improve its robustness with respect to non-stationary noise.
We show that our new approach is more robust and outperforms the DSN on a bird detection task.

assistant: text: We propose to enhance the Deep Scattering Network  in order to improve control and stability of any given machine learning pipeline by proposing a continuous wavelet thresholding scheme
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On learning visual odometry errors

Paper content:
This paper fosters the idea that deep learning methods can be sided to classical
visual odometry pipelines to improve their accuracy and to produce uncertainty
models to their estimations.
We show that the biases inherent to the visual odom-
etry process can be faithfully learnt and compensated for, and that a learning ar-
chitecture associated to a probabilistic loss function can jointly estimate a full
covariance matrix of the residual errors, defining a heteroscedastic error model.
Experiments on autonomous driving image sequences and micro aerial vehicles
camera acquisitions assess the possibility to concurrently improve visual odome-
try and estimate an error associated to its outputs.

assistant: text: This paper discusses different methods of pairing VO with deep learning and proposes a simultaneous prediction of corrections and uncertainty.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ``"Best-of-Many-Samples" Distribution Matching

Paper content:
Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem.
Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality.
Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success.
This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior.
The synthetic likelihood ratio term also shows instability during training.
We propose a novel objective with a ``"Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood.
This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.

assistant: text: We propose a new objective for training hybrid VAE-GANs which lead to significant improvement in mode coverage and quality.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Is my Deep Learning Model Learning more than I want it to?

Paper content:
Existing deep learning approaches for learning visual features tend to extract more information than what is required for the task at hand.
From a privacy preservation perspective, the input visual information is not protected from the model; enabling the model to become more intelligent than it is trained to be.
Existing approaches for suppressing additional task learning assume the presence of ground truth labels for the tasks to be suppressed during training time.
In this research, we propose a three-fold novel contribution:
(i) a novel metric to measure the trust score of a trained deep learning model,
(ii) a model-agnostic solution framework for trust score improvement by suppressing all the unwanted tasks, and
(iii) a simulated benchmark dataset, PreserveTask, having five different fundamental image classification tasks to study the generalization nature of models.
In the first set of experiments, we measure and improve the trust scores of five popular deep learning models: VGG16, VGG19, Inception-v1, MobileNet, and DenseNet and demonstrate that Inception-v1 is having the lowest trust score.
Additionally, we show results of our framework on color-MNIST dataset and practical applications of face attribute preservation in Diversity in Faces (DiF) and IMDB-Wiki dataset.

assistant: text: Can we trust our deep learning models? A framework to measure and improve a deep learning model's trust during training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration

Paper content:
Reinforcement learning (RL) agents improve through trial-and-error, but when reward is sparse and the agent cannot discover successful action sequences, learning stagnates.
This has been a notable problem in training deep RL agents to perform web-based tasks, such as booking flights or replying to emails, where a single mistake can ruin the entire sequence of actions.
A common remedy is to "warm-start" the agent by pre-training it to mimic expert demonstrations, but this is prone to overfitting.
Instead, we propose to constrain exploration using demonstrations.
From each demonstration, we induce high-level "workflows" which constrain the allowable actions at each time step to be similar to those in the demonstration (e.g., "Step 1: click on a textbox; Step 2: enter some text").
Our exploration policy then learns to identify successful workflows and samples actions that satisfy these workflows.
Workflows prune out bad exploration directions and accelerate the agent’s ability to discover rewards.
We use our approach to train a novel neural policy designed to handle the semi-structured nature of websites, and evaluate on a suite of web tasks, including the recent World of Bits benchmark.
We achieve new state-of-the-art results, and show that workflow-guided exploration improves sample efficiency over behavioral cloning by more than 100x.

assistant: text: We solve the sparse rewards problem on web UI tasks using exploration guided by demonstrations
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Machine Translation Using Monolingual Corpora Only

Paper content:
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora.
There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences.
In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data.
We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space.
By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data.
We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.

assistant: text: We propose a new unsupervised machine translation model that can learn without using parallel corpora; experimental results show impressive performance on multiple corpora and pairs of languages.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Parsing-based Approaches for Verification and Recognition of Hierarchical Plans

Paper content:
Hierarchical planning, in particular, Hierarchical Task Networks, was proposed as a method to describe plans by decomposition of tasks to sub-tasks until primitive tasks, actions, are obtained.
Plan verification assumes a complete plan as input, and the objective is finding a task that decomposes to this plan.
In plan recognition, a prefix of the plan is given and the objective is finding a task that decomposes to the (shortest) plan with the given prefix.
This paper describes how to verify and recognize plans using a common method known from formal grammars, by parsing.

assistant: text: The paper describes methods to verify and recognize HTN plans by parsing of attribute grammars.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: End-to-End Multi-Domain Task-Oriented Dialogue Systems with Multi-level Neural Belief Tracker

Paper content:
It has been an open research challenge for developing an end-to-end multi-domain task-oriented dialogue system, in which a human can converse with the dialogue agent to complete tasks in more than one domain.
First, tracking belief states of multi-domain dialogues is difficult as the dialogue agent must obtain the complete belief states from all relevant domains, each of which can have shared slots common among domains as well as unique slots specifically for the domain only.
Second, the dialogue agent must also process various types of information, including contextual information from dialogue context, decoded dialogue states of current dialogue turn, and queried results from a knowledge base, to semantically shape context-aware and task-specific responses to human.
To address these challenges, we propose an end-to-end neural architecture for task-oriented dialogues in multiple domains.
We propose a novel Multi-level Neural Belief Tracker which tracks the dialogue belief states by learning signals at both slot and domain level independently.
The representations are combined in a Late Fusion approach to form joint feature vectors of (domain, slot) pairs.
Following recent work in end-to-end dialogue systems, we incorporate the belief tracker with generation components to address end-to-end dialogue tasks.
We achieve state-of-the-art performance on the MultiWOZ2.1 benchmark with 50.91% joint goal accuracy and competitive measures in task-completion and response generation.

assistant: text: We proposed an end-to-end dialogue system with a novel multi-level dialogue state tracker and achieved consistent performance on MultiWOZ2.1 in state tracking, task completion, and response generation performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness

Paper content:
Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks.
In this work, we propose to employ mode connectivity in loss landscapes to study the adversarial robustness of deep neural networks, and provide novel methods for improving this robustness.
Our experiments cover various types of adversarial attacks applied to different network architectures and datasets.
When network models are tampered with backdoor or error-injection attacks, our results demonstrate that the path connection learned using limited amount of bonafide data can effectively mitigate adversarial effects while maintaining the original accuracy on clean data.
Therefore, mode connectivity provides users with the power to repair backdoored or error-injected models.
We also use mode connectivity to investigate the loss landscapes of regular and robust models against evasion attacks.
Experiments show that there exists a barrier in adversarial robustness loss on the path connecting regular and adversarially-trained models.
A high correlation is observed between the adversarial robustness loss and the largest eigenvalue of the input Hessian matrix, for which theoretical justifications are provided.
Our results suggest that mode connectivity offers a holistic tool and practical means for evaluating and improving adversarial robustness.

assistant: text: A novel approach using mode connectivity in loss landscapes to mitigate adversarial effects, repair tampered models and evaluate adversarial robustness
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Learn with Conditional Class Dependencies

Paper content:
Neural networks can learn to extract statistical properties from data, but they seldom make use of structured information from the label space to help representation learning.
Although some label structure can implicitly be obtained when training on huge amounts of data, in a few-shot learning context where little data is available, making explicit use of the label structure can inform the model to reshape the representation space to reflect a global sense of class dependencies.
We propose a meta-learning framework, Conditional class-Aware Meta-Learning (CAML), that conditionally transforms feature representations based on a metric space that is trained to capture inter-class dependencies.
This enables a conditional modulation of the feature representations of the base-learner to impose regularities informed by the label space.
Experiments show that the conditional transformation in CAML leads to more disentangled representations and achieves competitive results on the miniImageNet benchmark.

assistant: text: CAML is an instance of MAML with conditional class dependencies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Convergence and Stability of GANs

Paper content:
We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions.
We analyze the convergence of GAN training from this new point of view to understand why mode collapse happens.
We hypothesize the existence of undesirable local equilibria in this non-convex game to be responsible for mode collapse.
We observe that these local equilibria often exhibit sharp gradients of the discriminator function around some real data points.
We demonstrate that these degenerate local equilibria can be avoided with a gradient penalty scheme called DRAGAN.
We show that DRAGAN enables faster training, achieves improved stability with fewer mode collapses, and leads to generator networks with better modeling performance across a variety of architectures and objective functions.

assistant: text: Analysis of convergence and mode collapse by studying GAN training process as regret minimization
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generative Models from the perspective of Continual Learning

Paper content:
Which generative model is the most suitable for Continual Learning?
This paper aims at evaluating and comparing generative models on disjoint sequential image generation tasks.
We investigate how several models learn and forget, considering various strategies: rehearsal, regularization, generative replay and fine-tuning.
We used two quantitative metrics to estimate the generation quality and memory ability.
We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10).
We found that among all models, the original GAN performs best and among Continual Learning strategies, generative replay outperforms all other methods.
Even if we found satisfactory combinations on MNIST and Fashion MNIST, training generative models sequentially on CIFAR10 is particularly instable, and remains a challenge.

assistant: text: A comparative study of generative models on Continual Learning scenarios.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Overcoming the vanishing gradient problem in plain recurrent networks

Paper content:
Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs.
This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs.
We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.
We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks.
The RINs demonstrate competitive performance and converge faster in all tasks.
Notably, small RIN models produce 12%–67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

assistant: text: We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Two-layer Neural Networks with Symmetric Inputs

Paper content:
We give a new algorithm for learning a two-layer neural network under a very general class of input distributions.
Assuming there is a ground-truth two-layer network
y = A \sigma(Wx) + \xi,
where A, W are weight matrices, \xi represents noise, and the number of neurons in the hidden layer is no larger than the input or output,  our algorithm is guaranteed to recover the parameters A, W of the ground-truth network.
The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input.
Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions.
We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks.
Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.

assistant: text: We give an algorithm for learning a two-layer neural network with symmetric input distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergence of Implicit Filter Sparsity in Convolutional Neural Networks

Paper content:
We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained using adaptive gradient descent techniques with L2 regularization or weight decay.
Through an extensive empirical study (Anonymous, 2019) we hypothesize the mechanism be hind the sparsification process.
We find that the interplay  of  various  phenomena  influences  the strength of L2 and weight decay regularizers, leading the supposedly non sparsity inducing regularizers to induce filter sparsity.
In this workshop article we summarize some of our key findings and experiments, and present additional results on modern network architectures such as ResNet-50.

assistant: text: Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-Domain Adversarial Learning

Paper content:
Multi-domain learning (MDL) aims at obtaining a model with minimal average risk across multiple domains.
Our empirical motivation is automated microscopy data, where cultured cells are imaged after being exposed to known and unknown chemical perturbations, and each dataset displays significant experimental bias.
This paper presents a multi-domain adversarial learning approach, MuLANN, to leverage multiple datasets with overlapping but distinct class sets, in a semi-supervised setting.
Our contributions include:
i) a bound on the average- and worst-domain risk in MDL, obtained using the H-divergence;
ii) a new loss to accommodate semi-supervised multi-domain learning and domain adaptation;
iii) the experimental validation of the approach, improving on the state of the art on two standard image benchmarks, and a novel bioimage dataset, Cell.

assistant: text: Adversarial Domain adaptation and Multi-domain learning: a new loss to handle multi- and single-domain classes in the semi-supervised setting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AntMan: Sparse Low-Rank Compression To Accelerate RNN Inference

Paper content:
Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements.
To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy.
AntMan extends knowledge distillation based training to learn the compressed models efficiently.
Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models.
Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-of-art.
Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware.

assistant: text: Reducing computation and memory complexity of RNN models by up to 100x using sparse low-rank compression modules, trained via knowledge distillation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discriminative k-shot learning using probabilistic models

Paper content:
This paper introduces a probabilistic framework for k-shot image classification.
The goal is to generalise from an initial large-scale classification task to a separate task comprising new  classes and small numbers of examples.
The new approach not only leverages the feature-based representation learned by a neural network from the initial task (representational transfer), but also information about the classes (concept transfer).
The concept information is encapsulated in a probabilistic model for the final layer weights of the neural network which acts as a prior for probabilistic k-shot learning.
We show that even a simple probabilistic model achieves state-of-the-art on a standard k-shot learning dataset by a large margin.
Moreover, it is able to accurately model uncertainty, leading to well calibrated classifiers, and is easily extensible and flexible, unlike many recent approaches to k-shot learning.

assistant: text: This paper introduces a probabilistic framework for k-shot image classification that achieves state-of-the-art results
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
When data arise from multiple latent subpopulations, machine learning frameworks typically estimate parameter values independently for each sub-population.
In this paper, we propose
to overcome these limits by considering samples as tasks in a multitask learning framework.

assistant: text: We present a method to estimate collections of regression models in which each model is personalized to a single sample.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Additive function approximation in the brain

Paper content:
Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons.
For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned.
A network with d inputs per neuron is found to be equivalent to an additive model of order d, whereas with a degree distribution the network combines additive terms of different orders.
We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable.
Thus, even simple brain architectures can be powerful function approximators.
Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists.

assistant: text: We advocate for random features as a theory of biological neural networks, focusing on sparsely connected networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Overcoming the Lexical Overlap Bias Using Predicate-Argument Structures

Paper content:
Recent pretrained transformer-based language models have set state-of-the-art performances on various NLP datasets.
However, despite their great progress, they suffer from various structural and syntactic biases.
In this work, we investigate the lexical overlap bias, e.g., the model classifies two sentences that have a high lexical overlap as entailing regardless of their underlying meaning.
To improve the robustness, we enrich input sentences of the training data with their automatically detected predicate-argument structures.
This enhanced representation allows the transformer-based models to learn different attention patterns by focusing on and recognizing the major semantically and syntactically important parts of the sentences.
We evaluate our solution for the tasks of natural language inference and grounded commonsense inference using the BERT, RoBERTa, and XLNET models.
We evaluate the models' understanding of syntactic variations, antonym relations, and named entities in the presence of lexical overlap.
Our results show that the incorporation of predicate-argument structures during fine-tuning considerably improves the robustness, e.g.,  about 20pp on discriminating different named entities, while it incurs no additional cost at the test time and does not require changing the model or the training procedure.

assistant: text: Enhancing the robustness of pretrained transformer models against the lexical overlap bias by extending the input sentences of the training data with their corresponding predicate-argument structures
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ASGen: Answer-containing Sentence Generation to Pre-Train Question Generator for Scale-up Data in Question Answering

Paper content:
Numerous machine reading comprehension (MRC) datasets often involve manual annotation, requiring enormous human effort, and hence the size of the dataset remains significantly smaller than the size of the data available for unsupervised learning.
Recently, researchers proposed a model for generating synthetic question-and-answer data from large corpora such as Wikipedia.
This model is utilized to generate synthetic data for training an MRC model before fine-tuning it using the original MRC dataset.
This technique shows better performance than other general pre-training techniques such as language modeling, because the characteristics of the generated data are similar to those of the downstream MRC data.
However, it is difficult to have high-quality synthetic data comparable to human-annotated MRC datasets.
To address this issue, we propose Answer-containing Sentence Generation (ASGen), a novel pre-training method for generating synthetic data involving two advanced techniques, (1) dynamically determining K answers and (2) pre-training the question generator on the answer-containing sentence generation task.
We evaluate the question generation capability of our method by comparing the BLEU score with existing methods and test our method by fine-tuning the MRC model on the downstream MRC data after training on synthetic data.
Experimental results show that our approach outperforms existing generation methods and increases the performance of the state-of-the-art MRC models across a range of MRC datasets such as SQuAD-v1.1, SQuAD-v2.0, KorQuAD and QUASAR-T without any architectural modifications to the original MRC model.

assistant: text: We propose Answer-containing Sentence Generation (ASGen), a novel pre-training method for generating synthetic data for machine reading comprehension.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fix-Net: pure fixed-point representation of deep neural networks

Paper content:
Deep neural networks (DNNs) dominate current research in machine learning.
Due to massive GPU parallelization DNN training is no longer a bottleneck, and large models with many parameters and high computational effort lead common benchmark tables.
In contrast, embedded devices have a very limited capability.
As a result, both model size and inference time must be significantly reduced if DNNs are to achieve suitable performance on embedded devices.
We propose a soft quantization approach to train DNNs that can be evaluated using pure fixed-point arithmetic.
By exploiting the bit-shift mechanism, we derive fixed-point quantization constraints for all important components, including batch normalization and ReLU.
Compared to floating-point arithmetic, fixed-point calculations significantly reduce computational effort whereas low-bit representations immediately decrease memory costs.
We evaluate our approach with different architectures on common benchmark data sets and compare with recent quantization approaches.
We achieve new state of the art performance using 4-bit fixed-point models with an error rate of 4.98% on CIFAR-10.

assistant: text: Soft quantization approach to learn pure fixed-point representations of deep neural networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Identifying Bias in AI using Simulation

Paper content:
Machine learned models exhibit bias, often because the datasets used to train them are biased.
This presents a serious problem for the deployment of such technology, as the resulting models might perform poorly on populations that are minorities within the training set and ultimately present higher risks to them.
We propose to use high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers.
We present a framework that leverages Bayesian parameter search to efficiently characterize the high dimensional feature space and more quickly identify weakness in performance.
We apply our approach to an example domain, face detection, and show that it can be used to help identify demographic biases in commercial face application programming interfaces (APIs).

assistant: text: We present a framework that leverages high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Differentially Private Recurrent Language Models

Paper content:
We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy.
Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent.
In particular, we add user-level privacy protection to the federated averaging algorithm, which makes large step updates from user-level data.
Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work.
We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.

assistant: text: User-level differential privacy for recurrent neural network language models is possible with a sufficiently large dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Interactive Boosting of Neural Networks for Small-sample Image Classification

Paper content:
Neural networks have recently shown excellent performance on numerous classi- fication tasks.
These networks often have a large number of parameters and thus require much data to train.
When the number of training data points is small, however, a network with high flexibility will quickly overfit the training data, resulting in a large model variance and a poor generalization performance.
To address this problem, we propose a new ensemble learning method called InterBoost for small-sample image classification.
In the training phase, InterBoost first randomly generates two complementary datasets to train two base networks of the same structure, separately, and then next two complementary datasets for further training the networks are generated through interaction (or information sharing) between the two base networks trained previously.
This interactive training process continues iteratively until a stop criterion is met.
In the testing phase, the outputs of the two networks are combined to obtain one final score for classification.
Detailed analysis of the method is provided for an in-depth understanding of its mechanism.

assistant: text: In the paper, we proposed an ensemble method called InterBoost for training neural networks for small-sample classification. The method has better generalization performance than other ensemble methods, and reduces variances significantly.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DyRep: Learning Representations over Dynamic Graphs

Paper content:
Representation Learning over graph structured data has received significant attention recently due to its ubiquitous applicability.
However, most advancements have been made in static graph settings while efforts for jointly learning dynamic of the graph and dynamic on the graph are still in an infant stage.
Two fundamental questions arise in learning over dynamic graphs:
(i) How to elegantly model dynamical processes over graphs?
(ii) How to leverage such a model to effectively encode evolving graph information into low-dimensional representations?
We present DyRep - a novel modeling framework for dynamic graphs that posits representation learning as a latent mediation process bridging two observed processes namely -- dynamics of the network (realized as topological evolution) and dynamics on the network (realized as activities between nodes).
Concretely, we propose a two-time scale deep temporal point process model that captures the interleaved dynamics of the observed processes.
This model is further parameterized by a temporal-attentive representation network that encodes temporally evolving structural information into node representations which in turn drives the nonlinear evolution of the observed graph dynamics.
Our unified framework is trained using an efficient unsupervised procedure and has capability to generalize over unseen nodes.
We demonstrate that DyRep outperforms state-of-the-art baselines for dynamic link prediction and time prediction tasks and present extensive qualitative insights into our framework.

assistant: text: Models Representation Learning over dynamic graphs as latent hidden process bridging two observed processes of Topological Evolution of and Interactions on dynamic graphs.