ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Automated Relational Meta-learning

Paper content:
In order to efficiently learn with small amount of data on new tasks, meta-learning transfers knowledge learned from previous tasks to the new ones.
However, a critical challenge in meta-learning is the task heterogeneity which cannot be well handled by traditional globally shared meta-learning methods.
In addition, current task-specific meta-learning methods may either suffer from hand-crafted structure design or lack the capability to capture complex relations between tasks.
In this paper, motivated by the way of knowledge organization in knowledge bases, we propose an automated relational meta-learning (ARML) framework that automatically extracts the cross-task relations and constructs the meta-knowledge graph.
When a new task arrives, it can quickly find the most relevant structure and tailor the learned structure knowledge to the meta-learner.
As a result, the proposed framework not only addresses the challenge of task heterogeneity by a learned meta-knowledge graph, but also increases the model interpretability.
We conduct extensive experiments on 2D toy regression and few-shot image classification and the results demonstrate the superiority of ARML over state-of-the-art baselines.

assistant: text: Addressing task heterogeneity problem in meta-learning by introducing meta-knowledge graph
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Curriculum Learning for Deep Generative Models with Clustering

Paper content:
Training generative models like Generative Adversarial Network (GAN)  is challenging for noisy data.
A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper.
The curriculum construction is based on the centrality of underlying clusters in data points.
The data points of high centrality takes priority of being fed into generative models during training.
To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality.
Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models.
The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data.
An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper.

assistant: text: A novel cluster-based algorithm of curriculum learning is proposed to solve the robust training of generative models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Three factors influencing minima in SGD

Paper content:
We study the statistical properties of the endpoint of stochastic gradient descent (SGD).
We approximate SGD as a stochastic differential equation (SDE) and consider its Boltzmann Gibbs equilibrium distribution under the assumption of isotropic variance in loss gradients..
Through this analysis, we find that three factors – learning rate, batch size and the variance of the loss gradients – control the trade-off between the depth and width of the minima found by SGD, with wider minima favoured by a higher ratio of learning rate to batch size. In the equilibrium distribution only the ratio of learning rate to batch size appears, implying that it’s invariant under a simultaneous rescaling of each by the same amount. 
We experimentally show how learning rate and batch size affect SGD from two perspectives: the endpoint of SGD and the dynamics that lead up to it. For the endpoint, the experiments suggest the endpoint of SGD is similar under simultaneous rescaling of batch size and learning rate, and also that a higher ratio leads to flatter minima, both findings are consistent with our theoretical analysis. We note experimentally that the dynamics also seem to be similar under the same rescaling of learning rate and batch size, which we explore showing that one can exchange batch size and learning rate in a cyclical learning rate schedule. Next, we illustrate how noise affects memorization, showing that high noise levels lead to better generalization. Finally, we find experimentally that the similarity under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small.

assistant: text: Three factors (batch size, learning rate, gradient noise) change in predictable way the properties (e.g. sharpness) of minima found by SGD.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Domain Structure in HGNs for Nondeterministic planning 

Paper content:
This paper presents preliminary ideas of our work for auto- mated learning of Hierarchical Goal Networks in nondeter- ministic domains.
We are currently implementing the ideas expressed in this paper.

assistant: text: Learning HGNs, ND domains
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Analyzing analytical methods: The case of phonology in neural models of spoken language

Paper content:
Given the fast development of analysis techniques for NLP and speech
processing systems, few systematic studies have been conducted to
compare the strengths and weaknesses of each method.
As a step in
this direction we study the case of representations of phonology in
neural network models of spoken language.
We use two commonly applied
analytical techniques, diagnostic classifiers and representational
similarity analysis, to quantify to what extent neural activation
patterns encode phonemes and phoneme sequences.
We manipulate two
factors that can affect the outcome of analysis.
First, we investigate
the role of learning by comparing neural activations extracted from
trained versus randomly-initialized models.
Second, we examine the
temporal scope of the activations by probing both local activations
corresponding to a few milliseconds of the speech signal, and global
activations pooled over the whole utterance.
We conclude that
reporting analysis results with randomly initialized models is
crucial, and that global-scope methods tend to yield more consistent
and interpretable results and we recommend their use as a complement
to local-scope diagnostic methods.

assistant: text: We study representations of phonology in neural network models of spoken language with several variants of analytical techniques.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Irrationality can help reward inference

Paper content:
Specifying reward functions is difficult, which motivates the area of reward inference: learning rewards from human behavior.
The starting assumption in the area is that human behavior is optimal given the desired reward function, but in reality people have many different forms of irrationality, from noise to myopia to risk aversion and beyond.
This fact seems like it will be strictly harmful to reward inference: it is already hard to infer the reward from rational behavior, and noise and systematic biases make actions have less direct of a relationship to the reward.
Our insight in this work is that, contrary to expectations, irrationality can actually help rather than hinder reward inference.
For some types and amounts of irrationality, the expert now produces more varied policies compared to rational behavior, which help disambiguate among different reward parameters -- those that otherwise correspond to the same rational behavior.
We put this to the test in a systematic analysis of the effect of irrationality on reward inference.
We start by covering the space of irrationalities as deviations from the Bellman update, simulate expert behavior, and measure the accuracy of inference to contrast the different types and study the gains and losses.
We provide a mutual information-based analysis of our findings, and wrap up by discussing the need to accurately model irrationality, as well as to what extent we might expect (or be able to train) real people to exhibit helpful irrationalities when teaching rewards to learners.

assistant: text: We find that irrationality from an expert demonstrator can help a learner infer their preferences.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Infer

Paper content:
Inference models, which replace an optimization-based inference procedure with a learned model, have been fundamental in advancing Bayesian deep learning, the most notable example being variational auto-encoders (VAEs).
In this paper, we propose iterative inference models, which learn how to optimize a variational lower bound through repeatedly encoding gradients.
Our approach generalizes VAEs under certain conditions, and by viewing VAEs in the context of iterative inference, we provide further insight into several recent empirical findings.
We demonstrate the inference optimization capabilities of iterative inference models, explore unique aspects of these models, and show that they outperform standard inference models on typical benchmark data sets.

assistant: text: We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information

Paper content:
Sequence-to-sequence (seq2seq) neural models have been actively investigated for abstractive summarization.
Nevertheless, existing neural abstractive systems frequently generate factually incorrect summaries and are vulnerable to adversarial information, suggesting a crucial lack of semantic understanding.
In this paper, we propose a novel semantic-aware neural abstractive summarization model that learns to generate high quality summaries through semantic interpretation over salient content.
A novel evaluation scheme with adversarial samples is introduced to measure how well a model identifies off-topic information, where our model yields significantly better performance than the popular pointer-generator summarizer.
Human evaluation also confirms that our system summaries are uniformly more informative and faithful as well as less redundant than the seq2seq model.

assistant: text: We propose a semantic-aware neural abstractive summarization model and a novel automatic summarization evaluation scheme that measures how well a model identifies off-topic information from adversarial samples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GENERATIVE LOW-SHOT NETWORK EXPANSION

Paper content:
Conventional deep learning classifiers are static in the sense that they are trained on
a predefined set of classes and learning to classify a novel class typically requires
re-training.
In this work, we address the problem of Low-shot network-expansion
learning.
We introduce a learning framework which enables expanding a pre-trained
(base) deep network to classify novel classes when the number of examples for the
novel classes is particularly small.
We present a simple yet powerful distillation
method where the base network is augmented with additional weights to classify
the novel classes, while keeping the weights of the base network unchanged.
We
term this learning hard distillation, since we preserve the response of the network
on the old classes to be equal in both the base and the expanded network.
We
show that since only a small number of weights needs to be trained, the hard
distillation excels for low-shot training scenarios.
Furthermore, hard distillation
avoids detriment to classification performance on the base classes.
Finally, we
show that low-shot network expansion can be done with a very small memory
footprint by using a compact generative model of the base classes training data
with only a negligible degradation relative to learning with the full training set.

assistant: text: In this paper, we address the problem of Low-shot network-expansion learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Task Analysis

Paper content:
There is mounting evidence that pretraining can be valuable for neural network language understanding models, but we do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn.
With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information, holding constant the genre and quantity of training data.
We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data, which suggests that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information.
We also find that a randomly-initialized, frozen model can perform strikingly well on our auxiliary tasks, but that this effect disappears when the amount of training data for the auxiliary tasks is reduced.

assistant: text: Representations from language models consistently perform better than translation encoders on syntactic auxiliary prediction tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Optimal Attacks against Multiple Classifiers

Paper content:
We study the problem of designing provably optimal adversarial noise algorithms that induce misclassification in settings where a learner aggregates decisions from multiple classifiers.
Given the demonstrated vulnerability of state-of-the-art models to adversarial examples, recent efforts within the field of robust machine learning have focused on the use of ensemble classifiers as a way of boosting the robustness of individual models.
In this paper, we design provably optimal attacks against a set of classifiers.
We demonstrate how this problem can be framed as finding strategies at equilibrium in a two player, zero sum game between a learner and an adversary and consequently illustrate the need for randomization in adversarial attacks.
The main technical challenge we consider is the design of best response oracles that can be implemented in a Multiplicative Weight Updates framework to find equilibrium strategies in the zero-sum game.
We develop a series of scalable noise generation algorithms for deep neural networks, and show that it outperforms state-of-the-art attacks on various image classification tasks.
Although there are generally no guarantees for deep learning, we show this is a well-principled approach in that it is provably optimal for linear classifiers.
The main insight is a geometric characterization of the decision space that reduces the problem of designing best response oracles to minimizing a quadratic function over a set of convex polytopes.

assistant: text: Paper analyzes the problem of designing adversarial attacks against multiple classifiers, introducing algorithms that are optimal for linear classifiers and which provide state-of-the-art results for deep learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Are adversarial examples inevitable?

Paper content:
A wide range of defenses have been proposed to harden neural networks against adversarial attacks.
However, a pattern has emerged in which the majority of adversarial defenses are quickly broken by new attacks.
Given the lack of success at generating robust defenses, we are led to ask a fundamental question:  Are adversarial attacks inevitable?
This paper analyzes adversarial examples from a theoretical perspective, and identifies fundamental bounds on the susceptibility of a classifier to adversarial attacks.
We show that, for certain classes of problems, adversarial examples are inescapable.
Using experiments, we explore the implications of theoretical guarantees for real-world problems and discuss how factors such as dimensionality and image complexity limit a classifier's robustness against adversarial examples.

assistant: text: This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Explaining Neural Networks Semantically and Quantitatively

Paper content:
This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically.
How to analyze the specific rationale of each prediction made by the CNN presents one of key issues of understanding neural networks, but it is also of significant practical values in certain applications.
In this study, we propose to distill knowledge from the CNN into an explainable additive model, so that we can use the explainable model to provide a quantitative explanation for the CNN prediction.
We analyze the typical bias-interpreting problem of the explainable model and develop prior losses to guide the learning of the explainable additive model.
Experimental results have demonstrated the effectiveness of our method.

assistant: text: This paper presents a method to explain the knowledge encoded in a convolutional neural network (CNN) quantitatively and semantically.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Keep Doing What Worked: Behavior Modelling Priors for Offline Reinforcement Learning

Paper content:
Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired.
This property makes these algorithms appealing for real world problems such as robot control.
In practice, however, standard off-policy algorithms fail in the batch setting for continuous control.
In this paper, we propose a simple solution to this problem.
It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task.
Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources.
We find  improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.

assistant: text: We develop a method for stable offline reinforcement learning from logged data. The key is to regularize the RL policy towards a learned "advantage weighted" model of the data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification

Paper content:
Person re-identification (re-ID) aims at identifying the same persons' images across different cameras.
However, domain diversities between different datasets pose an evident challenge for adapting the re-ID model trained on one dataset to another one.
State-of-the-art unsupervised domain adaptation methods for person re-ID transferred the learned knowledge from the source domain by optimizing with pseudo labels created by clustering algorithms on the target domain.
Although they achieved state-of-the-art performances, the inevitable label noise caused by the clustering procedure was ignored.
Such noisy pseudo labels substantially hinders the model's capability on further improving feature representations on the target domain.
In order to mitigate the effects of noisy pseudo labels, we propose to softly refine the pseudo labels in the target domain by proposing an unsupervised framework, Mutual Mean-Teaching (MMT), to learn better features from the target domain via off-line refined hard pseudo labels and on-line refined soft pseudo labels in an alternative training manner.
In addition, the common practice is to adopt both the classification loss and the triplet loss jointly for achieving optimal performances in person re-ID models.
However, conventional triplet loss cannot work with softly refined labels.
To solve this problem, a novel soft softmax-triplet loss is proposed to support learning with soft pseudo triplet labels for achieving the optimal domain adaptation performance.
The proposed MMT framework achieves considerable improvements of 14.4%, 18.2%, 13.1% and 16.4% mAP on Market-to-Duke, Duke-to-Market, Market-to-MSMT and Duke-to-MSMT unsupervised domain adaptation tasks.

assistant: text: A framework that conducts online refinement of pseudo labels with a novel soft softmax-triplet loss for unsupervised domain adaptation on person re-identification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Selective Self-Training for semi-supervised Learning

Paper content:
Semi-supervised learning (SSL) is a study that efficiently exploits a large amount of unlabeled data to improve performance in conditions of limited labeled data.
Most of the conventional SSL methods assume that the classes of unlabeled data are included in the set of classes of labeled data.
In addition, these methods do not sort out useless unlabeled samples and use all the unlabeled data for learning, which is not suitable for realistic situations.
In this paper, we propose an SSL method called selective self-training (SST), which selectively decides whether to include each unlabeled sample in the training process.
It is also designed to be applied to a more real situation where classes of unlabeled data are different from the ones of the labeled data.
For the conventional SSL problems which deal with data where both the labeled and unlabeled samples share the same class categories, the proposed method not only performs comparable to other conventional SSL algorithms but also can be combined with other SSL algorithms.
While the conventional methods cannot be applied to the new SSL problems where the separated data do not share the classes, our method does not show any performance degradation even if the classes of unlabeled data are different from those of the labeled data.

assistant: text: Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Permutation Processes

Paper content:
We introduce a neural architecture to perform amortized approximate Bayesian inference over latent random permutations of two sets of objects.
The method involves approximating permanents of matrices of pairwise probabilities using recent ideas on functions defined over sets.
Each sampled permutation comes with a probability estimate, a quantity unavailable in MCMC approaches.
We illustrate the method in sets of 2D points and MNIST images.

assistant: text: A novel neural architecture for efficient amortized inference over latent permutations
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations

Paper content:
Deep networks have achieved impressive results across a variety of important tasks.
However, a known weakness is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples.
We propose \emph{Fortified Networks}, a simple transformation of existing networks, which “fortifies” the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well.
Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good results due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space.
We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner).
We show that these improvements are achieved across a wide variety of hyperparameters.

assistant: text: Better adversarial training by learning to map back to the data manifold with autoencoders in the hidden states.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Capsule Graph Neural Network

Paper content:
The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance.
However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representation may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings.
Inspired by the Capsule Neural Network (CapsNet), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms.
By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level.
As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects.
The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs.
Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven.
It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.

assistant: text: Inspired by CapsNet, we propose a novel architecture for graph embeddings on the basis of node features extracted from GNN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An Information-Theoretic Metric of Transferability for Task Transfer Learning

Paper content:
An important question in task transfer learning is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task.
Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning.
In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems.
Inspired by a principled information theoretic approach, H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature.
This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies.
Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice, but also it can generalize to inference problems beyond classification, such as recognition tasks for 3D indoor-scene understanding.

assistant: text: We present a provable and easily-computable evaluation function that estimates the performance of transferred representations from one learning task to another in task transfer learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Memory Augmented Control Networks

Paper content:
Planning problems in partially observable environments cannot be solved directly with convolutional networks and require some form of memory.
But, even memory networks with sophisticated addressing schemes are unable to learn intelligent reasoning satisfactorily due to the complexity of simultaneously learning to access memory and plan.
To mitigate these challenges we propose the Memory Augmented Control Network (MACN).
The network splits planning into a hierarchical process.
At a lower level, it learns to plan in a locally observed space.
At a higher level, it uses a collection of policies computed on locally observed spaces to learn an optimal plan in the global environment it is operating in.
The performance of the network is evaluated on path planning tasks in environments in the presence of simple and complex obstacles and in addition, is tested for its ability to generalize to new environments not seen in the training set.

assistant: text: Memory Augmented Network to plan in partially observable environments.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting

Paper content:
This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources.
We design a Dynamic Point-cloud Convolution (D-Conv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input.
This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -- an important aspect in spatiotemporal predictive learning.
The D-Conv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms.
We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e. mobile service traffic forecasting and air quality indicator forecasting.
Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of neural network models.

assistant: text: This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RPGAN: random paths as a latent space for GAN interpretability

Paper content:
In this paper, we introduce Random Path Generative Adversarial Network (RPGAN) --- an alternative scheme of GANs that can serve as a tool for generative model analysis.
While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network.
As we show, this design allows to associate different layers of the generator with different regions of the latent space, providing their natural interpretability.
With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about roles that different layers play in the image generation process.
Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data.

assistant: text: We introduce an alternative GAN design based on random routes in generator, which can serve as a tool for generative models interpretability.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Phase-Aware Speech Enhancement with Deep Complex U-Net

Paper content:
Most deep learning-based models for speech enhancement have mainly focused on estimating the magnitude of spectrogram while reusing the phase from noisy speech for reconstruction.
This is due to the difficulty of estimating the phase of clean speech.
To improve speech enhancement performance, we tackle the phase estimation problem in three ways.
First, we propose Deep Complex U-Net, an advanced U-Net structured model incorporating well-defined complex-valued building blocks to deal with complex-valued spectrograms.
Second, we propose a polar coordinate-wise complex-valued masking method to reflect the distribution of complex ideal ratio masks.
Third, we define a novel loss function, weighted source-to-distortion ratio (wSDR) loss, which is designed to directly correlate with a quantitative evaluation measure.
Our model was evaluated on a mixture of the Voice Bank corpus and DEMAND database, which has been widely used by many deep learning models for speech enhancement.
Ablation experiments were conducted on the mixed dataset showing that all three proposed approaches are empirically valid.
Experimental results show that the proposed method achieves state-of-the-art performance in all metrics, outperforming previous approaches by a large margin.

assistant: text: This paper proposes a novel complex masking method for speech enhancement along with a loss function for efficient phase estimation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the importance of single directions for generalization

Paper content:
Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance.
However, the differences between the learned solutions of networks which generalize and those which do not remain unclear.
Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated.
Here, we connect these lines of inquiry to demonstrate that a network’s reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyper- parameters, and over the course of training.
While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units.
Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance.

assistant: text: We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Semantic Structure Extraction for Spreadsheet Tables with a Multi-task Learning Architecture

Paper content:
Semantic structure extraction for spreadsheets includes detecting table regions, recognizing structural components and classifying cell types.
Automatic semantic structure extraction is key to automatic data transformation from various table structures into canonical schema so as to enable data analysis and knowledge discovery.
However, they are challenged by the diverse table structures and the spatial-correlated semantics on cell grids.
To learn spatial correlations and capture semantics on spreadsheets, we have developed a novel learning-based framework for spreadsheet semantic structure extraction.
First, we propose a multi-task framework that learns table region, structural components and cell types jointly; second, we leverage the advances of the recent language model to capture semantics in each cell value; third, we build a large human-labeled dataset with broad coverage of table structures.
Our evaluation shows that our proposed multi-task framework is highly effective that outperforms the results of training each task separately.

assistant: text: We propose a novel multi-task framework that learns table detection, semantic component recognition and cell type classification for spreadsheet tables with promising results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: FairFace: A Novel Face Attribute Dataset for Bias Measurement and Mitigation

Paper content:
Existing public face image datasets are strongly biased toward Caucasian faces, and other races (e.g., Latino) are significantly underrepresented.
The models trained from such datasets suffer from inconsistent classification accuracy, which limits the applicability of face analytic systems to non-White race groups.
To mitigate the race bias problem in these datasets, we constructed a novel face image dataset containing 108,501 images which is balanced on race.
We define 7 race groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino.
Images were collected from the YFCC-100M Flickr dataset and labeled with race, gender, and age groups.
Evaluations were performed on existing face attribute datasets as well as novel image datasets to measure the generalization performance.
We find that the model trained from our dataset is substantially more accurate on novel datasets and the accuracy is consistent across race and gender groups.
We also compare several commercial computer vision APIs and report their balanced accuracy across gender, race, and age groups.

assistant: text: A new face image dataset for balanced race, gender, and age which can be used for bias measurement and mitigation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Language Modeling by Jointly Learning Syntax and Lexicon

Paper content:
We propose a neural language model capable of unsupervised syntactic structure induction.
The model leverages the structure information to form better semantic representations and better language modeling.
Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information.
On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation.
In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model.
In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network.
Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

assistant: text: In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeepArchitect: Automatically Designing and Training Deep Architectures

Paper content:
In deep learning, performance is strongly affected by the choice of architecture
and hyperparameters.
While there has been extensive work on automatic hyperpa-
rameter optimization for simple spaces, complex spaces such as the space of deep
architectures remain largely unexplored.
As a result, the choice of architecture is
done manually by the human expert through a slow trial and error process guided
mainly by intuition.
In this paper we describe a framework for automatically
designing and training deep models.
We propose an extensible and modular lan-
guage that allows the human expert to compactly represent complex search spaces
over architectures and their hyperparameters.
The resulting search spaces are tree-
structured and therefore easy to traverse.
Models can be automatically compiled to
computational graphs once values for all hyperparameters have been chosen.
We
can leverage the structure of the search space to introduce different model search
algorithms, such as random search, Monte Carlo tree search (MCTS), and sequen-
tial model-based optimization (SMBO).
We present experiments comparing the
different algorithms on CIFAR-10 and show that MCTS and SMBO outperform
random search.
We also present experiments on MNIST, showing that the same
search space achieves near state-of-the-art performance with a few samples.
These
experiments show that our framework can be used effectively for model discov-
ery, as it is possible to describe expressive search spaces and discover competitive
models without much effort from the human expert.
Code for our framework and
experiments has been made publicly available

assistant: text: We describe a modular and composable language for describing expressive search spaces over architectures and simple model search algorithms applied to these search spaces.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Escaping Saddle Points Faster with Stochastic Momentum

Paper content:
Stochastic gradient descent (SGD) with stochastic momentum is popular in nonconvex stochastic optimization and particularly for the training of deep neural networks.
In standard SGD, parameters are updated by improving along the path of the gradient at the current iterate on a batch of examples, where the addition of a ``momentum'' term biases the update in the direction of the previous change in parameters.
In non-stochastic convex optimization one can show that a momentum adjustment provably reduces convergence time in many settings, yet such results have been elusive in the stochastic and non-convex settings.
At the same time, a widely-observed empirical phenomenon is that in training deep networks stochastic momentum appears to significantly improve convergence time, variants of it have flourished in the development of other popular update methods, e.g. ADAM, AMSGrad, etc.
Yet theoretical justification for the use of stochastic momentum has remained a significant open question.
In this paper we propose an answer: stochastic momentum improves deep network training because it modifies SGD to escape saddle points faster and, consequently, to more quickly find a second order stationary point.
Our theoretical results also shed light on the related question of how to choose the ideal momentum parameter--our analysis suggests that $\beta \in [0,1)$ should be large (close to 1), which comports with empirical findings.
We also provide experimental findings that further validate these conclusions.

assistant: text: Higher momentum parameter $\beta$ helps for escaping saddle points faster
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Unified Framework for Lifelong Learning in Deep Neural Networks

Paper content:
Humans can learn a variety of concepts and skills incrementally over the course of their lives while exhibiting an array of desirable properties, such as non-forgetting, concept rehearsal, forward transfer and backward transfer of knowledge, few-shot learning, and selective forgetting.
Previous approaches to lifelong machine learning can only demonstrate subsets of these properties, often by combining multiple complex mechanisms.
In this Perspective, we propose a powerful unified framework that can demonstrate all of the properties by utilizing a small number of weight consolidation parameters in deep neural networks.
In addition, we are able to draw many parallels between the behaviours and mechanisms of our proposed framework and those surrounding human learning, such as memory loss or sleep deprivation.
This Perspective serves as a conduit for two-way inspiration to further understand lifelong learning in machines and humans.

assistant: text: Drawing parallels with human learning, we propose a unified framework to exhibit many lifelong learning abilities in neural networks by utilizing a small number of weight consolidation parameters.