ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta Module Network for Compositional Visual Reasoning

Paper content:
There are two main lines of research on visual reasoning: neural module network (NMN) with explicit multi-hop reasoning through handcrafted neural modules, and monolithic network with implicit reasoning in the latent feature space.
The former excels in interpretability and compositionality, while the latter usually achieves better performance due to model flexibility and parameter efficiency.
In order to bridge the gap of the two, we present Meta Module Network (MMN), a novel hybrid approach that can efficiently utilize a Meta Module to perform versatile functionalities, while preserving compositionality and interpretability through modularized design.
The proposed model first parses an input question into a functional program through a Program Generator.
Instead of handcrafting a task-specific network to represent each function like traditional NMN, we use Recipe Encoder to translate the functions into their corresponding recipes (specifications), which are used to dynamically instantiate the Meta Module into Instance Modules.
To endow different instance modules with designated functionality, a Teacher-Student framework is proposed, where a symbolic teacher pre-executes against the scene graphs to provide guidelines for the instantiated modules (student) to follow.
In a nutshell, MMN adopts the meta module to increase its parameterization efficiency, and uses recipe encoding to improve its generalization ability over NMN.
Experiments conducted on the GQA benchmark demonstrates that: (1) MMN achieves significant improvement over both NMN and monolithic network baselines; (2) MMN is able to generalize to unseen but related functions.

assistant: text: We propose a new Meta Module Network to resolve some of the restrictions of previous Neural Module Network to achieve strong performance on realistic visual reasoning dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Can gradient clipping mitigate label noise?

Paper content:
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
This intuition has been made precise in a line of recent works, which show that suitable clipping  can yield significantly faster convergence than vanilla gradient descent.
In this paper, we propose a new lens for studying gradient clipping, namely, robustness: informally, one expects clipping to provide robustness to noise, since one does not overly trust any single sample.
Surprisingly, we prove that  for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness.
On the other hand, we show that  a simple variant of gradient clipping is provably robust, and corresponds to suitably modifying the underlying loss function.
This yields a simple, noise-robust alternative to the standard cross-entropy loss which performs well empirically.
We established that gradient clipping by itself does not suffice to endow even simple models with label noise robustness; however, a simple variant resolves this issue.
Experiments confirm that our composite loss-based gradient clipping performs well on datasets corrupted with label noise.
One interesting direction for future work is to analyse the behaviour of gradient-clipping inspired losses for the more general problem of distributionally robust learning (Shafieezadeh-Abadeh et al., 2015; Namkoong & Duchi, 2016; Sinha et al., 2018; Duchi & Namkoong, 2019) .

assistant: text: Gradient clipping doesn't endow robustness to label noise, but a simple loss-based variant does.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Corpus Based Amharic Sentiment Lexicon Generation

Paper content:
Sentiment classification is an active research area with several applications including analysis of political opinions, classifying comments, movie reviews, news reviews and product reviews.
To employ rule based sentiment classification, we require sentiment lexicons.
However, manual construction of sentiment lexicon is time consuming and costly for resource-limited languages.
To bypass manual development time and costs, we tried to build Amharic Sentiment Lexicons relying on corpus based approach.
The intention of this approach is to handle sentiment terms specific to Amharic language from Amharic Corpus.
Small set of seed terms are manually prepared from three parts of speech such as noun, adjective and verb.
We developed algorithms for constructing Amharic sentiment lexicons automatically from Amharic news corpus.
Corpus based approach is proposed relying on the word co-occurrence distributional embedding including frequency based embedding (i.e. Positive Point-wise Mutual Information PPMI).
First we build word-context unigram frequency count matrix and transform it to point-wise mutual Information matrix.
Using this matrix, we computed the cosine distance of mean vector of seed lists and each word in the corpus vocabulary.
Based on the threshold value, the top closest words to the mean vector of seed list are added to the lexicon.
Then the mean vector of the new sentiment seed list is updated and process is repeated until we get sufficient terms in the lexicon.
Using PPMI with threshold value of 100 and 200, we got corpus based Amharic Sentiment lexicons of size 1811 and 3794 respectively by expanding 519 seeds.
Finally, the lexicon generated in corpus based approach is evaluated.

assistant: text: Corpus based Algorithm is developed generate Amharic Sentiment lexicon relying on corpus
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Federated Learning Personalization via Model Agnostic Meta Learning

Paper content:
Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data.
A natural scenario arises with data created on mobile phones by the activity of their users.
Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually.
In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL.
We present FL as a natural source of practical applications for MAML algorithms, and make the following observations.
1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm.
2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize.
However, solely optimizing for the global model accuracy yields a weaker personalization result.
3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim.
These results raise new questions for FL, MAML, and broader ML research.

assistant: text: Federated Averaging already is a Meta Learning algorithm, while datacenter-trained methods are significantly harder to personalize.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Doc2Dial: a Framework for Dialogue Composition Grounded in Business Documents

Paper content:
We introduce Doc2Dial, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing.
Such data can be used to train automated dialogue agents performing customer care tasks for the enterprises or organizations.
In particular, the framework takes the documents as input and generates the tasks for obtaining the annotations for simulating dialog flows.
The dialog flows are used to guide the collection of utterances produced by crowd workers.
The outcomes include dialogue data grounded in the given documents, as well as various types of annotations that help ensure the quality of the data and the flexibility to (re)composite dialogues.

assistant: text: We introduce Doc2Dial, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing for train automated dialogue agents
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Monotonic Multihead Attention

Paper content:
Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence.
Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure.
In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention.
We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions.
We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.
Code will be released upon publication.

assistant: text: Make the transformer streamable with monotonic attention.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Sentence Representations with Multi-view Frameworks

Paper content:
Multi-view learning can provide self-supervision when different views are available of the same data.
Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora.
Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion.
One framework uses a generative objective and the other a discriminative one.
In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model.
We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

assistant: text: Multi-view learning improves unsupervised sentence representation learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Identifying Analogies Across Domains

Paper content:
Identifying analogies across domains without supervision is a key task for artificial intelligence.
Recent advances in cross domain image mapping have concentrated on translating images across domains.
Although the progress made is impressive, the visual fidelity many times does not suffice for identifying the matching sample from the other domain.
In this paper, we tackle this very task of finding exact analogies between datasets i.e. for every image from domain A find an analogous image in domain B. We present a matching-by-synthesis approach: AN-GAN, and show that it outperforms current techniques.
We further show that the cross-domain mapping task can be broken into two parts: domain alignment and learning the mapping function.
The tasks can be iteratively solved, and as the alignment is improved, the unsupervised translation function reaches quality comparable to full supervision.

assistant: text: Finding correspondences between domains by performing matching/mapping iterations
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Paper content:
The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation.
However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint.
We propose a general technique for replacing the softmax layer with a continuous embedding layer.
Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax.
We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation.
We show that our models obtain upto 2.5x speed-up in training time while performing on par with the state-of-the-art models in terms of translation quality.
These models are capable of handling very large vocabularies without compromising on translation quality.
They also produce more meaningful errors than in the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations.

assistant: text: Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DARE: Data Augmented Relation Extraction with GPT-2

Paper content:
Real-world Relation Extraction (RE) tasks are challenging to deal with, either due to limited training data or class imbalance issues.
In this work, we present Data Augmented Relation Extraction (DARE), a simple method to augment training data by properly finetuning GPT2 to generate examples for specific relation types.
The generated training data is then used in combination with the gold dataset to train a BERT-based RE classifier.
In a series of experiments we show the advantages of our method, which leads in improvements of up to 11 F1 score points compared to a strong baseline.
Also, DARE achieves new state-of-the-art in three widely used biomedical RE datasets surpassing the previous best results by 4.7 F1 points on average.

assistant: text: Data Augmented Relation Extraction with GPT-2
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Music Source Separation in the Waveform Domain

Paper content:
Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song.
Such components include voice, bass, drums and any other accompaniments.
While end-to-end models that directly generate the waveform are state-of-the-art in many audio synthesis problems, the best multi-instrument source separation models generate masks on the magnitude spectrum and achieve performances far above current end-to-end, waveform-to-waveform models.
We present an in-depth analysis of a new architecture, which we will refer to as Demucs, based on a (transposed) convolutional autoencoder, with a bidirectional LSTM at the bottleneck layer and skip-connections as in U-Networks (Ronneberger et al., 2015).
Compared to the state-of-the-art waveform-to-waveform model, Wave-U-Net (Stoller et al., 2018), the main features of our approach in addition of the bi-LSTM are the use of trans-posed convolution layers instead of upsampling-convolution blocks, the use of gated linear units, exponentially growing the number of channels with depth and a new careful initialization of the weights.
Results on the MusDB dataset show that our architecture achieves a signal-to-distortion ratio (SDR) nearly 2.2 points higher than the best waveform-to-waveform competitor (from 3.2 to 5.4 SDR).
This makes our model match the state-of-the-art performances on this dataset, bridging the performance gap between models that operate on the spectrogram and end-to-end approaches.

assistant: text: We match the performance of spectrogram based model with a model trained end-to-end in the waveform domain
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discovering the compositional structure of vector representations with Role Learning Networks

Paper content:
Neural networks (NNs) are able to perform tasks that rely on compositional structure even though they lack obvious mechanisms for representing this structure.
To analyze the internal representations that enable such success, we propose ROLE, a technique that detects whether these representations implicitly encode symbolic structure.
ROLE learns to approximate the representations of a target encoder E by learning a symbolic constituent structure and an embedding of that structure into E’s representational vector space.
The constituents of the approximating symbol structure are defined by structural positions — roles — that can be filled by symbols.
We show that when E is constructed to explicitly embed a particular type of structure (e.g., string or tree), ROLE successfully extracts the ground-truth roles defining that structure.
We then analyze a seq2seq network trained to perform a more complex compositional task (SCAN), where there is no ground truth role scheme available.
For this model, ROLE successfully discovers an interpretable symbolic structure that the model implicitly uses to perform the SCAN task, providing a comprehensive account of the link between the representations and the behavior of a notoriously hard-to-interpret type of model.
We verify the causal importance of the discovered symbolic structure by showing that, when we systematically manipulate hidden embeddings based on this symbolic structure, the model’s output is also changed in the way predicted by our analysis.
Finally, we use ROLE to explore whether popular sentence embedding models are capturing compositional structure and find evidence that they are not; we conclude by discussing how insights from ROLE can be used to impart new inductive biases that will improve the compositional abilities of such models.

assistant: text: We introduce a new analysis technique that discovers interpretable compositional structure in notoriously hard-to-interpret recurrent neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Lipschitz regularized Deep Neural Networks generalize

Paper content:
We show that if the usual training loss is augmented by a Lipschitz regularization term, then the networks generalize.
We prove generalization by first establishing a stronger convergence result, along with a rate of convergence.
A second result resolves a question posed in Zhang et al. (2016): how can a model distinguish between the case of clean labels, and randomized labels?
Our answer is that Lipschitz regularization using the Lipschitz constant of the clean data makes this distinction.
In this case, the model learns a different function which we hypothesize correctly fails to learn the dirty labels.

assistant: text: We prove generalization of DNNs by adding a Lipschitz regularization term to the training loss. We resolve a question posed in Zhang et al. (2016).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Predict Responsibly: Increasing Fairness by Learning to Defer

Paper content:
When machine learning models are used for high-stakes decisions, they should predict accurately, fairly, and responsibly.
To fulfill these three requirements, a model must be able to output a reject option (i.e. say "``I Don't Know") when it is not qualified to make a prediction.
In this work, we propose learning to defer, a method by which a model can defer judgment to a downstream decision-maker such as a human user.
We show that learning to defer generalizes the rejection learning framework in two ways: by considering the effect of other agents in the decision-making process, and by allowing for optimization of complex objectives.
We propose a learning algorithm which accounts for potential biases held by decision-makerslater in a pipeline.
Experiments on real-world datasets demonstrate that learning
to defer can make a model not only more accurate but also less biased.
Even when
operated by highly biased users, we show that
deferring models can still greatly improve the fairness of the entire pipeline.

assistant: text: Incorporating the ability to say I-don't-know can improve the fairness of a classifier without sacrificing too much accuracy, and this improvement magnifies when the classifier has insight into downstream decision-making.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

Paper content:
An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data.
We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other.
Thus changes to the network parameters during training are biased towards those that (locally) simultaneously benefit many examples when such similarity exists.
We support this hypothesis with heuristic arguments and perturbative experiments and outline how this can explain several common empirical observations about Deep Learning.
Furthermore, our analysis is not just descriptive, but prescriptive.
It suggests a natural modification to gradient descent that can greatly reduce overfitting.

assistant: text: We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Assessing the scalability of biologically-motivated deep learning algorithms and architectures

Paper content:
The backpropagation of error algorithm (BP) is often said to be impossible to implement in a real brain.
The recent success of deep networks in machine learning and AI, however, has inspired a number of proposals for understanding how the brain might learn across multiple layers, and hence how it might implement or approximate BP.
As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks.
Here we present the first results on scaling up a biologically motivated model of deep learning to datasets which need deep networks with  appropriate architectures to achieve good performance.
We present results on CIFAR-10 and ImageNet.
For CIFAR-10 we show that our algorithm, a straightforward, weight-transport-free variant of difference target-propagation (DTP) modified to remove backpropagation from the penultimate layer, is competitive with BP in training deep networks with locally defined receptive fields that have untied weights.
For ImageNet we find that both DTP and our algorithm perform significantly worse than BP, opening questions about whether different architectures or algorithms are required to scale these approaches.
Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

assistant: text: Benchmarks for biologically plausible learning algorithms on complex datasets and architectures
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Local and global model interpretability via backward selection and clustering

Paper content:
Local explanation frameworks aim to rationalize particular decisions made by a black-box prediction model.
Existing techniques are often restricted to a specific type of predictor or based on input saliency, which may be undesirably sensitive to factors unrelated to the model's decision making process.
We instead propose sufficient input subsets that identify minimal subsets of features whose observed values alone suffice for the same decision to be reached, even if all other input feature values are missing.
General principles that globally govern a model's decision-making can also be revealed by searching for clusters of such input patterns across many data points.
Our approach is conceptually straightforward, entirely model-agnostic, simply implemented using instance-wise backward selection, and able to produce more concise rationales than existing techniques.
We demonstrate the utility of our interpretation method on neural network models trained on text and image data.

assistant: text: We present a method for interpreting black-box models by using instance-wise backward selection to identify minimal subsets of features that alone suffice to justify a particular decision made by the model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: INTERPRETATION OF NEURAL NETWORK IS FRAGILE

Paper content:
In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions.
For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification.
How to interpret black-box predictors is thus an important and active area of research.
A fundamental question is: how much can we trust the interpretation itself?
In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense:  two perceptively indistinguishable inputs with the same predicted label can be assigned very different}interpretations.
We systematically characterize the fragility of the interpretations generated by several widely-used feature-importance interpretation methods (saliency maps, integrated gradient, and DeepLIFT) on ImageNet and CIFAR-10.
Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label.
We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile.
Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

assistant: text: Can we trust a neural network's explanation for its prediction? We examine the robustness of several popular notions of interpretability of neural networks including saliency maps and influence functions and design adversarial examples against them.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks

Paper content:
The point estimates of ReLU classification networks, arguably the most widely used neural network architecture, have recently been shown to have arbitrarily high confidence far away from the training data.
This architecture is thus not robust, e.g., against out-of-distribution data.
Approximate Bayesian posteriors on the weight space have been empirically demonstrated to improve predictive uncertainty in deep learning.
The theoretical analysis of such Bayesian approximations is limited, including for ReLU classification networks.
We present an analysis of approximate Gaussian posterior distributions on the weights of ReLU networks.
We show that even a simplistic (thus cheap), non-Bayesian Gaussian distribution fixes the asymptotic overconfidence issue.
Furthermore, when a Bayesian method, even if a simple one, is employed to obtain the Gaussian, the confidence becomes better calibrated.
This theoretical result motivates a range of Laplace approximations along a fidelity-cost trade-off.
We validate these findings empirically via experiments using common deep ReLU networks.

assistant: text: We argue theoretically that by simply assuming the weights of a ReLU network to be Gaussian distributed (without even a Bayesian formalism) could fix this issue; for a more calibrated uncertainty, a simple Bayesian method could already be sufficient.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Generative Dual Memory Network for Continual Learning

Paper content:
Despite advances in deep learning, artificial neural networks do not learn the same way as humans do.
Today, neural networks can learn multiple tasks when trained on them jointly, but cannot maintain performance on learnt tasks when tasks are presented one at a time -- this phenomenon called catastrophic forgetting is a fundamental challenge to overcome before neural networks can learn continually from incoming data.
In this work, we derive inspiration from human memory to develop an architecture capable of learning continuously from sequentially incoming tasks, while averting catastrophic forgetting.
Specifically, our model consists of a dual memory architecture to emulate the complementary learning systems (hippocampus and the neocortex) in the human brain and maintains a consolidated long-term memory via generative replay of past experiences.
We
(i) substantiate our claim that replay should be generative,
(ii) show the benefits of generative replay and dual memory via experiments, and
(iii) demonstrate improved performance retention even for small models with low capacity.
Our architecture displays many important characteristics of the human memory and provides insights on the connection between sleep and learning in humans.

assistant: text: A dual memory architecture inspired from human brain to learn sequentially incoming tasks, while averting catastrophic forgetting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Multivariate Mixture of Gaussians for Object Detection under Occlusion

Paper content:
In this paper, we consider the problem of detecting object under occlusion.
Most object detectors formulate bounding box regression as a unimodal task (i.e., regressing a single set of bounding box coordinates independently).
However, we observe that the bounding box borders of an occluded object can have multiple plausible configurations.
Also, the occluded bounding box borders have correlations with visible ones.
Motivated by these two observations, we propose a deep multivariate mixture of Gaussians model for bounding box regression under occlusion.
The mixture components potentially learn different configurations of an occluded part, and the covariances between variates help to learn the relationship between the occluded parts and the visible ones.
Quantitatively, our model improves the AP of the baselines by 3.9% and 1.2% on CrowdHuman and MS-COCO respectively with almost no computational or memory overhead.
Qualitatively, our model enjoys explainability since we can interpret the resulting bounding boxes via the covariance matrices and the mixture components.

assistant: text: a deep multivariate mixture of Gaussians model for bounding box regression under occlusion
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Graph Neural Networks with Generated Parameters for Relation Extraction

Paper content:
Recently, progress has been made towards improving relational reasoning in machine learning field.
Among existing models, graph neural networks (GNNs) is one of the most effective approaches for multi-hop relational reasoning.
In fact, multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction.
In this paper, we propose to generate the parameters of graph neural networks (GP-GNNs) according to natural language sentences, which enables GNNs to process relational reasoning on unstructured text inputs.
We verify GP-GNNs in relation extraction from text.
Experimental results on a human-annotated dataset and two distantly supervised datasets show that our model achieves significant improvements compared to the baselines.
We also perform a qualitative analysis to demonstrate that our model could discover more accurate relations by multi-hop relational reasoning.

assistant: text: A graph neural network model with parameters generated from natural languages, which can perform multi-hop reasoning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Internal Dense But External Sparse Structures of Deep Neural Network

Paper content:
Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs).
On one hand, increasing the density of CNNs by adding cross-layer connections achieve higher accuracy.
On the other hand, creating sparsity structures through regularization and pruning methods enjoys lower computational costs.
In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections.
This new structure uses dense modules, as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process.
Experimental results demonstrate that the locally dense yet externally sparse structure could acquire competitive performance on benchmark tasks (CIFAR10, CIFAR100, and ImageNet) while keeping the network structure slim.
In this paper, we firstly create locally dense and externally sparse structures by prefixing some dense modules and add sparse connections between them.
Experiment results demonstrate that evolving sparse connections could reach competitive results on benchmark datasets.
In order to give properties of these biologically plausible structures, we apply several sets of contrast experiments as shown in Experiment.
By equally changing the input feature groups of each module during the whole training process, this strategy could alleviate the risk of the weights being trapped in local optimal point.
Same to most of the related works, redundancy of each dense module is not 'the larger the better', where the test accuracy will first increase within the growth rate increases, but finally drop while the growth is above some threshold.The combination of being dense and being sparse is an interesting area, and the internal dense and externally sparse structure also coincide with the modularity in human brain.
We prove the feasibility of these structures and give a simple algorithm to search best connections.
We also noticed that the connection matrix is not unique for reaching good performance.
We will concentrate on revealing the relationship between these similar connection matrices and the representing features behind it.In this case, we may acquire state of the art performance on other datasets and tasks in our future work.
Moreover, as these structures have various direct paths between input and output, separating a network into several small networks without any accuracy loss is also a promising topic.

assistant: text: In this paper, we explore an internal dense yet external sparse network structure of deep neural networks and analyze its key properties.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Parametric Manifold Learning Via Sparse Multidimensional Scaling

Paper content:
We propose a metric-learning framework for computing distance-preserving maps that generate low-dimensional embeddings for a certain class of manifolds.
We employ Siamese networks to solve the problem of least squares multidimensional scaling for generating mappings that preserve geodesic distances on the manifold.
In contrast to previous parametric manifold learning methods we show a substantial reduction in training effort enabled by the computation of geodesic distances in a farthest point sampling strategy.
Additionally, the use of a network to model the distance-preserving map reduces the complexity of the multidimensional scaling problem and leads to an improved non-local generalization of the manifold compared to analogous non-parametric counterparts.
We demonstrate our claims on point-cloud data and on image manifolds and show a numerical analysis of our technique to facilitate a greater understanding of the representational power of neural networks in modeling manifold data.

assistant: text: Parametric Manifold Learning with Neural Networks in a Geometric Framework
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Federated Learning with Matched Averaging

Paper content:
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud.
We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs.
FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures.
Our experiments indicate that FedMA outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, while improving the communication efficiency.

assistant: text: Communication efficient federated learning with layer-wise matching
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Latent Variable Session-Based Recommendation

Paper content:
We present a probabilistic framework for session based recommendation.
A latent variable for the user state is updated as the user views more items and we learn more about their interests.
We provide computational solutions using both the re-parameterization trick and  using the Bouchard bound for the softmax function, we further explore employing a variational auto-encoder and a variational Expectation-Maximization algorithm for tightening the variational bound.
Finally we show that the Bouchard bound causes the denominator of the softmax to decompose into a sum enabling fast noisy gradients of the bound giving a fully probabilistic algorithm reminiscent of word2vec and a fast online EM algorithm.

assistant: text: Fast variational approximations for approximating a user state and learning product embeddings
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Do Attention Heads in BERT Track Syntactic Dependencies?

Paper content:
We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations.
We employ two methods—taking the maximum attention weight and computing the maximum spanning tree—to extract implicit dependency relations from the attention weights of each layer/head, and compare them to the ground-truth Universal Dependency (UD) trees.
We show that, for some UD relation types, there exist heads that can recover the dependency type significantly better than baselines on parsed English text, suggesting that some self-attention heads act as a proxy for syntactic structure.
We also analyze BERT fine-tuned on two datasets—the syntax-oriented CoLA and the semantics-oriented MNLI—to investigate whether fine-tuning affects the patterns of their self-attention, but we do not observe substantial differences in the overall dependency relations extracted using our methods.
Our results suggest that these models have some specialist attention heads that track individual dependency types, but no generalist head that performs holistic parsing significantly better than a trivial baseline, and that analyzing attention weights directly may not reveal much of the syntactic knowledge that BERT-style models are known to learn.

assistant: text: Attention weights don't fully expose what BERT knows about syntax.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropy

Paper content:
We study model recovery for data classification, where the training labels are generated from a one-hidden-layer fully -connected neural network with sigmoid activations, and the goal is to recover the weight vectors of the neural network.
We prove that under Gaussian inputs, the empirical risk function using cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large.
This implies that if initialized in this neighborhood, which can be achieved via the tensor method, gradient descent converges linearly to a critical point that is provably close to the ground truth without requiring a fresh set of samples at each iteration.
To the best of our knowledge, this is the first global convergence guarantee established for the empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension.

assistant: text: We provide the first theoretical analysis of guaranteed recovery of one-hidden-layer neural networks under cross entropy loss for classification problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Confidence-based Graph Convolutional Networks for Semi-Supervised Learning

Paper content:
Predicting properties of nodes in a graph is an important problem with applications in a variety of domains.
Graph-based Semi Supervised Learning (SSL) methods aim to address this problem by labeling a small subset of the nodes as seeds, and then utilizing the graph structure to predict label scores for the rest of the nodes in the graph.
Recently, Graph Convolutional Networks (GCNs) have achieved impressive performance on the graph-based SSL task.
In addition to label scores, it is also desirable to have a confidence score associated with them.
Unfortunately, confidence estimation in the context of GCN has not been previously explored.
We fill this important gap in this paper and propose ConfGCN, which estimates labels scores along with their confidences jointly in GCN-based setting.
ConfGCN uses these estimated confidences to determine the influence of one node on another during neighborhood aggregation, thereby acquiring anisotropic capabilities.
Through extensive analysis and experiments on standard benchmarks, we find that ConfGCN is able to significantly outperform state-of-the-art baselines.
We have made ConfGCN’s source code available to encourage reproducible research.

assistant: text: We propose a confidence based Graph Convolutional Network for Semi-Supervised Learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Video-to-Video Translation

Paper content:
Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time.
In this paper, we formulate a new task of unsupervised video-to-video translation, which poses its own unique challenges.
Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames.
We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem.
We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset.
Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance.

assistant: text: Proposed new task, datasets and baselines; 3D Conv CycleGAN preserves object properties across frames; batch structure in frame-level methods matters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Optimal transport maps for distribution preserving operations on latent spaces of Generative Models

Paper content:
Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian.
After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample.
In this paper, we show that the latent space operations used in the literature so far induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on.
To address this, we propose to use distribution matching transport maps to ensure that such  latent space operations preserve the prior distribution, while minimally modifying the original operation.
Our experimental results validate that the proposed operations give higher quality samples compared to the original operations.

assistant: text: Operations in the GAN latent space can induce a distribution mismatch compared to the training distribution, and we address this using optimal transport to match the distributions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Online Semi-Supervised Learning with Bandit Feedback

Paper content:
We formulate a new problem at the intersection of semi-supervised learning and contextual bandits, motivated by several applications including clinical trials and dialog systems.
We demonstrate how contextual bandit and graph convolutional networks can be adjusted to the new problem formulation.
We then take the best of both approaches to develop multi-GCN embedded contextual bandit.
Our algorithms are verified on several real world datasets.

assistant: text: Synthesis of GCN and LINUCB algorithms for online learning with missing feedbacks