ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Embedded Deep Learning for Face Detection and Emotion Recognition with Intel© Movidius (TM) Neural Compute Stick

Paper content:
Nowadays deep learning is one of the main topics in almost every field.
It helped to get amazing results in a great number of tasks.
The main problem is that this kind of learning and consequently neural networks, that can be defined deep, are resource intensive.
They need specialized hardware to perform a computation in a reasonable time.
Unfortunately, it is not sufficient to make deep learning "usable" in real life.
Many tasks are mandatory to be as much as possible real-time.
So it is needed to optimize many components such as code, algorithms, numeric accuracy and hardware, to make them "efficient and usable".
All these optimizations can help us to produce incredibly accurate and fast learning models.

assistant: text: Embedded architecture for deep learning on optimized devices for face detection and emotion recognition
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Semi-parametric topological memory for navigation

Paper content:
We introduce a new memory architecture for navigation in previously unseen environments, inspired by landmark-based navigation in animals.
The proposed semi-parametric topological memory (SPTM) consists of a (non-parametric) graph with nodes corresponding to locations in the environment and a (parametric) deep network capable of retrieving nodes from the graph based on observations.
The graph stores no metric information, only connectivity of locations corresponding to the nodes.
We use SPTM as a planning module in a navigation system.
Given only 5 minutes of footage of a previously unseen maze, an SPTM-based navigation agent can build a topological map of the environment and use it to confidently navigate towards goals.
The average success rate of the SPTM agent in goal-directed navigation across test environments is higher than the best-performing baseline by a factor of three.

assistant: text: We introduce a new memory architecture for navigation in previously unseen environments, inspired by landmark-based navigation in animals.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning DNA folding patterns with Recurrent Neural Networks 

Paper content:
The recent expansion of machine learning applications to molecular biology proved to have a significant contribution to our understanding of biological systems, and genome functioning in particular.
Technological advances enabled the collection of large epigenetic datasets, including information about various DNA binding factors (ChIP-Seq) and DNA spatial structure (Hi-C).
Several studies have confirmed the correlation between DNA binding factors and Topologically Associating Domains (TADs) in DNA structure.
However, the information about physical proximity represented by genomic coordinate was not yet used for the improvement of the prediction models.
In this research, we focus on Machine Learning methods for prediction of folding patterns of DNA in a classical model organism Drosophila melanogaster.
The paper considers linear models with four types of regularization, Gradient Boosting and Recurrent Neural Networks for the prediction of chromatin folding patterns from epigenetic marks.
The bidirectional LSTM RNN model outperformed all the models and gained the best prediction scores.
This demonstrates the utilization of complex models and the importance of memory of sequential DNA states for the chromatin folding.
We identify informative epigenetic features that lead to the further conclusion of their biological significance.

assistant: text: We apply RNN to solve the biological problem of chromatin folding patterns prediction from epigenetic marks and demonstrate for the first time that utilization of memory of sequential states on DNA molecule is significant for the best performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems

Paper content:
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks.
Second-order methods, despite their better convergence rate, are rarely used in practice due to the pro- hibitive computational cost in calculating the second-order information.
In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss.
Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK).
Different from typical second-order methods that have heavy computational cost in each iteration, GGN only has minor overhead compared to first-order methods such as SGD.
We also give theoretical results to show that for sufficiently wide neural networks, the convergence rate of GGN is quadratic.
Furthermore, we provide convergence guarantee for mini-batch GGN algorithm, which is, to our knowledge, the first convergence result for the mini-batch version of a second-order method on overparameterized neural net- works.
Preliminary experiments on regression tasks demonstrate that for training standard networks, our GGN algorithm converges much faster and achieves better performance than SGD.

assistant: text: A novel Gram-Gauss-Newton method to train neural networks, inspired by neural tangent kernel and Gauss-Newton method, with fast convergence speed both theoretically and experimentally.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variational Predictive Information Bottleneck

Paper content:
In classic papers, Zellner (1988, 2002) demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional.
Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective.
This generalized functional encompasses most modern inference procedures and suggests novel ones.

assistant: text: Rederive a wide class of inference procedures from an global information bottleneck objective.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning

Paper content:
The physical design of a robot and the policy that controls its motion are inherently coupled.
However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs.
In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision.
Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller.
Throughout training, we refine the robot distribution to maximize the expected reward.
This results in an assignment to the robot parameters and neural network policy that are jointly optimal.
We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs.

assistant: text: Use deep reinforcement learning to design the physical attributes of a robot jointly with a control policy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Treat Sepsis with Multi-Output Gaussian Process Deep Recurrent Q-Networks

Paper content:
Sepsis is a life-threatening complication from infection and a leading cause of mortality in hospitals.
While early detection of sepsis improves patient outcomes, there is little consensus on exact treatment guidelines, and treating septic patients remains an open  problem.
In this work we present a new deep reinforcement learning method that we use to learn optimal personalized treatment policies for septic patients.
We model patient continuous-valued physiological time series using multi-output Gaussian processes, a probabilistic model that easily handles missing values and irregularly spaced observation times while maintaining estimates of uncertainty.
The Gaussian process is directly tied to a deep recurrent Q-network that learns clinically interpretable treatment policies, and both models are learned together end-to-end.
We evaluate our approach on a heterogeneous dataset of septic spanning 15 months from our university health system, and find that our learned policy could reduce patient mortality by as much as 8.2\% from an overall baseline mortality rate of 13.3\%.
Our algorithm could be used to make treatment recommendations to physicians as part of a decision support tool, and the framework readily applies to other reinforcement learning problems that rely on sparsely sampled and frequently missing multivariate time series data.

assistant: text: We combine Multi-output Gaussian processes with deep recurrent Q-networks to learn optimal treatments for sepsis and show improved performance over standard deep reinforcement learning methods,
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Global-to-local Memory Pointer Networks for Task-Oriented Dialogue

Paper content:
End-to-end task-oriented dialogue is challenging since knowledge bases are usually large, dynamic and hard to incorporate into a learning framework.
We propose the global-to-local memory pointer (GLMP) networks to address this issue.
In our model, a global memory encoder and a local memory decoder are proposed to share external knowledge.
The encoder encodes dialogue history, modifies global contextual representation, and generates a global memory pointer.
The decoder first generates a sketch response with unfilled slots.
Next, it passes the global memory pointer to filter the external knowledge for relevant information, then instantiates the slots via the local memory pointers.
We empirically show that our model can improve copy accuracy and mitigate the common out-of-vocabulary problem.
As a result, GLMP is able to improve over the previous state-of-the-art models in both simulated bAbI Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on automatic and human evaluation.

assistant: text: GLMP: Global memory encoder (context RNN, global pointer) and local memory decoder (sketch RNN, local pointer) that share external knowledge (MemNN) are proposed to strengthen response generation in task-oriented dialogue.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exploring the Correlation between Likelihood of Flow-based Generative Models and Image Semantics

Paper content:
Among deep generative models, flow-based models, simply referred as \emph{flow}s in this paper, differ from other models in that they provide tractable likelihood.
Besides being an evaluation metric of synthesized data, flows are supposed to be robust against out-of-distribution~(OoD) inputs since they do not discard any information of the inputs.
However, it has been observed that flows trained on FashionMNIST assign higher likelihoods to OoD samples from MNIST.
This counter-intuitive observation raises the concern about the robustness of flows' likelihood.
In this paper, we explore the correlation between flows' likelihood and image semantics.
We choose two typical flows as the target models: Glow, based on coupling transformations, and pixelCNN, based on autoregressive transformations.
Our experiments reveal surprisingly weak correlation between flows' likelihoods and image semantics: the predictive likelihoods of flows can be heavily affected by trivial transformations that keep the image semantics unchanged, which we call semantic-invariant transformations~(SITs).
We explore three SITs~(all small pixel-level modifications): image pixel translation, random noise perturbation, latent factors zeroing~(limited to flows using multi-scale architecture, e.g. Glow).
These findings, though counter-intuitive, resonate with the fact that the predictive likelihood of a flow is the joint probability of all the image pixels.
So flows' likelihoods, modeling on pixel-level intensities, is not able to indicate the existence likelihood of the high-level image semantics.
We call for attention that it may be \emph{abuse} if we use the predictive likelihoods of flows for OoD samples detection.

assistant: text: show experimental evidences about the weak correlation between flows' likelihoods and image semantics.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Covariant Compositional Networks For Learning Graphs

Paper content:
Most existing neural networks for learning graphs deal with the issue of permutation invariance by conceiving of the network as a message passing scheme, where each node sums the feature vectors coming from its neighbors.
We argue that this imposes a limitation on their representation power, and instead propose a new general architecture for representing objects consisting of a hierarchy of parts, which we call Covariant Compositional Networks (CCNs).
Here covariance means that the activation of each neuron must transform in a specific way under permutations, similarly to steerability in CNNs.
We achieve covariance by making each activation transform according to a tensor representation of the permutation group, and derive the corresponding tensor aggregation rules that each neuron must implement.
Experiments show that CCNs can outperform competing methods on some standard graph learning benchmarks.

assistant: text: A general framework for creating covariant graph neural networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Curious Case of Neural Text Degeneration

Paper content:
Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g.  to generate a story).
The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops.
To address this we propose Nucleus Sampling, a simple but effective method to draw considerably  higher  quality  text  out  of  neural  language  models.
Our  approach avoids text degeneration by truncating the unreliable tail of the probability distribution, sampling from the dynamic  nucleus  of  tokens containing the vast majority of the probability mass.
To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.
Our results show that (1) maximization is an inappropriate decoding objective for open-ended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is the best decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.

assistant: text: Current language generation systems either aim for high likelihood and devolve into generic repetition or miscalibrate their stochasticity—we provide evidence of both and propose a solution: Nucleus Sampling.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compressing Word Embeddings via Deep Compositional Code Learning

Paper content:
Natural language processing (NLP) models often require a massive number of parameters for word embeddings, resulting in a large storage or memory footprint.
Deploying neural NLP models to mobile devices requires compressing the word embeddings without any significant sacrifices in performance.
For this purpose, we propose to construct the embeddings with few basis vectors.
For each word, the composition of basis vectors is determined by a hash code.
To maximize the compression rate, we adopt the multi-codebook quantization approach instead of binary coding scheme.
Each code is composed of multiple discrete numbers, such as (3, 2, 1, 8), where the value of each component is limited to a fixed range.
We propose to directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick.
Experiments show the compression rate achieves 98% in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss.
In both tasks, the proposed method can improve the model performance by slightly lowering the compression rate.
Compared to other approaches such as character-level segmentation, the proposed method is language-independent and does not require modifications to the network architecture.

assistant: text: Compressing the word embeddings over 94% without hurting the performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Visuomotor Control of Humanoids

Paper content:
We aim to build complex humanoid agents that integrate perception, motor control, and memory.
In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision.
We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies.
The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment.
Supplementary video link:  https://youtu.be/fBoir7PNxPk

assistant: text: Solve tasks involving vision-guided humanoid locomotion, reusing locomotion behavior from motion capture data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks

Paper content:
Recent work suggests goal-driven training of neural networks can be used to model neural activity in the brain.
While response properties of neurons in artificial neural networks bear similarities to those in the brain, the network architectures are often constrained to be different.
Here we ask if a neural network can recover both neural representations and, if the architecture is unconstrained and optimized, also the anatomical properties of neural circuits.
We demonstrate this in a system where the connectivity and the functional organization have been characterized, namely, the head direction circuit of the rodent and fruit fly.
We trained recurrent neural networks (RNNs) to estimate head direction through integration of angular velocity.
We found that the two distinct classes of neurons observed in the head direction system, the Ring neurons and the Shifter neurons, emerged naturally in artificial neural networks as a result of training.
Furthermore, connectivity analysis and in-silico neurophysiology revealed structural and mechanistic similarities between artificial networks and the head direction system.
Overall, our results show that optimization of RNNs in a goal-driven task can recapitulate the structure and function of biological circuits, suggesting that artificial neural networks can be used to study the brain at the level of both neural activity and anatomical organization.

assistant: text: Artificial neural networks trained with gradient descent are capable of recapitulating both realistic neural activity and the anatomical organization of a biological circuit.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Functional vs. parametric equivalence of ReLU networks

Paper content:
We address the following question: How redundant is the parameterisation of ReLU networks?
Specifically, we consider transformations of the weight space which leave the function implemented by the network intact.
Two such transformations are known for feed-forward architectures: permutation of neurons within a layer, and positive scaling of all incoming weights of a neuron coupled with inverse scaling of its outgoing weights.
In this work, we show for architectures with non-increasing widths that permutation and scaling are in fact the only function-preserving weight transformations.
For any eligible architecture we give an explicit construction of a neural network such that any other network that implements the same function can be obtained from the original one by the application of permutations and rescaling.
The proof relies on a geometric understanding of boundaries between linear regions of ReLU networks, and we hope the developed mathematical tools are of independent interest.

assistant: text: We prove that there exist ReLU networks whose parameters are almost uniquely determined by the function they implement.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: No Spurious Local Minima in a Two Hidden Unit ReLU Network

Paper content:
Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this.
A key question in optimization is to understand when the optimization landscape of a neural network is amenable to gradient-based optimization.
We focus on a simple neural network two-layer ReLU network with two hidden units, and show that all local minimizers are global.
This combined with recent work of Lee et al. (2017); Lee et al. (2016) show that  gradient descent converges to the global minimizer.
In this paper, we provided recovery guarantee of stochastic gradient descent with random initialization for learning a two-layer neural network with two hidden nodes, unit-norm weights, ReLU activation functions and Gaussian inputs.
Experiments are also done to verify our results.
For future work, here we list some possible directions.
In conclusion, based on the assumption that θ 1 ≤ θ 2 there are four critical points in the 2D case: Assume the manifold is R = {(w 1 , w 2 ) : w 1 2 = w 2 2 = 1}, then the Hessian on the manifold is DISPLAYFORM0 DISPLAYFORM1 where z = (z 1 , z 2 ) satisfies w DISPLAYFORM2 and DISPLAYFORM3 Then we can get when w 1 = w 2 and w 1 = −w 2 , DISPLAYFORM4 So this point is a saddle point.
In conclusion, we have four critical points: one is global maximal, the other three are saddle points.
DISPLAYFORM0 ) is a critical point, then there exists a set of standard orthogonal basis (e 1 , e 2 , e 3 ) such that e 1 = w * 1 , e 2 = w * 2 and w 1 , w 2 lies in span{e 1 , e 2 , e 3 }.Proof
. If (
w 1 , w 2 ) is a critical point, then DISPLAYFORM1 where matrix (I − w 1 w T 1 ) projects a vector onto the tangent space of w 1 . Since
DISPLAYFORM2 we get DISPLAYFORM3 DISPLAYFORM4 )w * 2 lies in the direction of w 1 . If θ
w1,w2 = π, i.e., w 1 = −w 2 , then of course the four vectors have rank at most 3, so we can find the proper basis. If θ
w1,w2 < π, then we know that there exists a real number r such that DISPLAYFORM5 Since θ w1,w2 < π, we know that the four vectors w 1 , w 2 , w * 1 and w * 2 are linear dependent. Thus
, they have rank at most 3 and we can find the proper basis.

assistant: text: Recovery guarantee of stochastic gradient descent with random initialization for learning a two-layer neural network with two hidden nodes, unit-norm weights, ReLU activation functions and Gaussian inputs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SesameBERT: Attention for Anywhere

Paper content:
Fine-tuning with pre-trained models has achieved exceptional results for many language tasks.
In this study, we focused on one such self-attention network model, namely BERT, which has performed well in terms of stacking layers across diverse language-understanding benchmarks.
However, in many downstream tasks, information between layers is ignored by BERT for fine-tuning.
In addition, although self-attention networks are well-known for their ability to capture global dependencies, room for improvement remains in terms of emphasizing the importance of local contexts.
In light of these advantages and disadvantages, this paper proposes SesameBERT, a generalized fine-tuning method that (1) enables the extraction of global information among all layers through Squeeze and Excitation and (2) enriches local information by capturing neighboring contexts via Gaussian blurring.
Furthermore, we demonstrated the effectiveness of our approach in the HANS dataset, which is used to determine whether models have adopted shallow heuristics instead of learning underlying generalizations.
The experiments revealed that SesameBERT outperformed BERT with respect to GLUE benchmark and the HANS evaluation set.

assistant: text: We proposed SesameBERT, a generalized fine-tuning method that enables the extraction of global information among all layers through Squeeze and Excitation and enriches local information by capturing neighboring contexts via Gaussian blurring.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Transfer Learn

Paper content:
We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.
L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses.
The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set.
We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets.
In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.

assistant: text: We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Training with Voronoi Constraints

Paper content:
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models.
Adversarial training, one of the most successful empirical defenses to adversarial examples, refers to training on adversarial examples generated within a geometric constraint set.
The most commonly used geometric constraint is an $L_p$-ball of radius $\epsilon$ in some norm.
We introduce adversarial training with Voronoi constraints, which replaces the $L_p$-ball constraint with the Voronoi cell for each point in the training set.
We show that adversarial training with Voronoi constraints produces robust models which significantly improve over the state-of-the-art on MNIST and are competitive on CIFAR-10.

assistant: text: We replace the Lp ball constraint with the Voronoi cells of the training data to produce more robust models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Effect of Network Depth on the Optimization Landscape

Paper content:
It is well-known that deeper neural networks are harder to train than shallower ones.
In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network gets deeper, and as residual connections are added to the architecture.
Computing a series of quantitative measures on the Hessian spectrum, we show that the Hessian eigenvalue distribution in deeper networks has substantially heavier tails (equivalently, more outlier eigenvalues), which makes the network harder to optimize with first-order methods.
We show that adding residual connections mitigates this effect substantially, suggesting a mechanism by which residual connections improve training.

assistant: text: Network depth increases outlier eigenvalues in the Hessian. Residual connections mitigate this.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Interpretable and Pedagogical Examples

Paper content:
Teachers intentionally pick the most informative examples to show their students.
However, if the teacher and student are neural networks, the examples that the teacher network learns to give, although effective at teaching the student, are typically uninterpretable.
We show that training the student and teacher iteratively, rather than jointly, can produce interpretable teaching strategies.
We evaluate interpretability by (1) measuring the similarity of the teacher's emergent strategies to intuitive strategies in each domain and (2) conducting human experiments to evaluate how effective the teacher's strategies are at teaching humans.
We show that the teacher network learns to select or generate interpretable, pedagogical examples to teach rule-based, probabilistic, boolean, and hierarchical concepts.

assistant: text: We show that training a student and teacher network iteratively, rather than jointly, can produce emergent, interpretable teaching strategies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Variational Inequality Perspective on Generative Adversarial Networks

Paper content:
Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train.
One common way to tackle this issue has been to propose new formulations of the GAN objective.
Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training.
In this work, we cast GAN optimization problems in the general variational inequality framework.
Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend methods designed for variational inequalities to the training of GANs.
We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

assistant: text: We cast GANs in the variational inequality framework and import techniques from this literature to optimize GANs better; we give algorithmic extensions and empirically test their performance for training GANs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Supervised Contextual Embeddings for Transfer Learning in Natural Language Processing Tasks

Paper content:
Pre-trained word embeddings are the primary
method for transfer learning in several Natural Language Processing (NLP) tasks.
Recent
works have focused on using unsupervised
techniques such as language modeling to obtain these embeddings.
In contrast, this work
focuses on extracting representations from
multiple pre-trained supervised models, which
enriches word embeddings with task and domain specific knowledge.
Experiments performed in cross-task, cross-domain and crosslingual settings indicate that such supervised
embeddings are helpful, especially in the lowresource setting, but the extent of gains is dependent on the nature of the task and domain.

assistant: text: extract contextual embeddings from off-the-shelf supervised model. Helps downstream NLP models in low-resource settings
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sound event classification using ontology-based neural networks

Paper content:
State of the art sound event classification relies in neural networks to learn the associations between class labels and audio recordings within a dataset.
These datasets typically define an ontology to create a structure that relates these sound classes with more abstract super classes.
Hence, the ontology serves as a source of domain knowledge representation of sounds.
However, the ontology information is rarely considered, and specially under explored to model neural network architectures.
We propose two ontology-based neural network architectures for sound event classification.
We defined a framework to design simple network architectures that preserve an ontological structure.
The networks are trained and evaluated using two of the most common sound event classification datasets.
Results show an improvement in classification performance demonstrating the benefits of including the ontological information.

assistant: text: We present ontology-based neural network architectures for sound event classification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dissecting an Adversarial framework for Information Retrieval

Paper content:
Recent advances in Generative Adversarial Networks facilitated by improvements to the framework and successful application to various problems has resulted in extensions to multiple domains.
IRGAN attempts to leverage the framework for Information-Retrieval (IR), a task that can be described as modeling the correct conditional probability distribution p(d|q) over the documents (d), given the query (q).
The work that proposes IRGAN claims that optimizing their minimax loss function will result in a generator which can learn the distribution, but their setup and baseline term steer the model away from an exact adversarial formulation, and this work attempts to point out certain inaccuracies in their formulation.
Analyzing their loss curves gives insight into possible mistakes in the loss functions and better performance can be obtained by using the co-training like setup we propose, where two models are trained in a co-operative rather than an adversarial fashion.

assistant: text: Points out problems in loss function used in IRGAN, a recently proposed GAN framework for Information Retrieval. Further, a model motivated by co-training is proposed, which achieves better performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SNODE: Spectral Discretization of Neural ODEs for System Identification

Paper content:
This paper proposes the use of spectral element methods \citep{canuto_spectral_1988} for fast and accurate training of Neural Ordinary Differential Equations (ODE-Nets; \citealp{Chen2018NeuralOD}) for system identification.
This is achieved by expressing their dynamics as a truncated series of Legendre polynomials.
The series coefficients, as well as the network weights, are computed by minimizing the weighted sum of the loss function and the violation of the ODE-Net dynamics.
The problem is solved by coordinate descent that alternately minimizes, with respect to the coefficients and the weights, two unconstrained sub-problems using standard backpropagation and gradient methods.
The resulting optimization scheme is fully time-parallel and results in a low memory footprint.
Experimental comparison to standard methods, such as backpropagation through explicit solvers and the adjoint technique \citep{Chen2018NeuralOD}, on training surrogate models of small and medium-scale dynamical systems shows that it is at least one order of magnitude faster at reaching a comparable value of the loss function.
The corresponding testing MSE is one order of magnitude smaller as well, suggesting generalization capabilities increase.

assistant: text: This paper proposes the use of spectral element methods for fast and accurate training of Neural Ordinary Differential Equations for system identification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Paper content:
Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments.
These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces.
However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy.
In this work, we propose an approach using deep representation learning algorithms to learn an adequate goal space.
This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space.
We present experiments with a simulated robot arm interacting with an object, and we show that exploration algorithms using such learned representations can closely match, and even sometimes improve, the performance obtained using engineered representations.

assistant: text: We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SENSE: SEMANTICALLY ENHANCED NODE SEQUENCE EMBEDDING

Paper content:
Effectively capturing graph node sequences in the form of vector embeddings is critical to many applications.
We achieve this by
(i) first learning vector embeddings of single graph nodes and
(ii) then composing them to compactly represent node sequences.
Specifically, we propose SENSE-S (Semantically Enhanced Node Sequence Embedding - for Single nodes), a skip-gram based novel embedding mechanism, for single graph nodes that co-learns graph structure as well as their textual descriptions.
We demonstrate that SENSE-S vectors increase the accuracy of multi-label classification tasks by up to 50% and link-prediction tasks by up to 78% under a variety of scenarios using real datasets.
Based on SENSE-S, we next propose generic SENSE to compute composite vectors that represent a sequence of nodes, where preserving the node order is important.
We prove that this approach is efficient in embedding node sequences, and our experiments on real data confirm its high accuracy in node order decoding.

assistant: text: Node sequence embedding mechanism that captures both graph and text properties.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling

Paper content:
A common way to speed up training of large convolutional networks is to add  computational units.
Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with a mini-batch divided between computational units.
With an increase in the number of nodes, the batch size grows.
However,  training with a large batch  often results in lower model accuracy.
We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge.
To overcome these optimization difficulties, we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS).
Using LARS, we scaled AlexNet  and ResNet-50 to a batch size of 16K.

assistant: text: A new large batch training algorithm  based on Layer-wise Adaptive Rate Scaling (LARS); using LARS, we scaled AlexNet  and ResNet-50 to a batch of 16K.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Likelihood Contribution based Multi-scale Architecture for Generative Flows

Paper content:
Deep generative modeling using flows has gained popularity owing to the tractable exact log-likelihood estimation with efficient training and synthesis process.
However, flow models suffer from the challenge of having high dimensional latent space, same in dimension as the input space.
An effective solution to the above challenge as proposed by Dinh et al. (2016) is a multi-scale architecture, which is based on iterative early factorization of a part of the total dimensions at regular intervals.
Prior works on generative flows involving a multi-scale architecture perform the dimension factorization based on a static masking.
We propose a novel multi-scale architecture that performs data dependent factorization to decide which dimensions should pass through more flow layers.
To facilitate the same, we introduce a heuristic based on the contribution of each dimension to the total log-likelihood which encodes the importance of the dimensions.
Our proposed heuristic is readily obtained as part of the flow training process, enabling versatile implementation of our likelihood contribution based multi-scale architecture for generic flow models.
We present such an implementation for the original flow introduced in Dinh et al. (2016), and demonstrate improvements in log-likelihood score and sampling quality on standard image benchmarks.
We also conduct ablation studies to compare proposed method with other options for dimension factorization.

assistant: text: Data-dependent factorization of dimensions in a multi-scale architecture based on contribution to the total log-likelihood
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Translating neural signals to text using a Brain-Computer Interface

Paper content:
Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing.
However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces.
To this end, we aim to create a BCI that decodes text directly from neural signals.
We implement a framework that initially isolates frequency bands in the input signal encapsulating differential information regarding production of various phonemic classes.
These bands form a feature set that feeds into an LSTM which discerns at each time point probability distributions across all phonemes uttered by a subject.
Finally, a particle filtering algorithm temporally smooths these probabilities incorporating prior knowledge of the English language to output text corresponding to the decoded word.
Further, in producing an output, we abstain from constraining the reconstructed word to be from a given bag-of-words, unlike previous studies.
The empirical success of our proposed approach, offers promise for the employment of such an interface by patients in unfettered, naturalistic environments.

assistant: text: We present an open-loop brain-machine interface whose performance is unconstrained to the traditionally used bag-of-words approach.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Batch Normalization is a Cause of Adversarial Vulnerability

Paper content:
Batch normalization (BN) is often used in an attempt to stabilize and accelerate training in deep neural networks.
In many cases it indeed decreases the number of parameter updates required to achieve low training error.
However, it also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages, as we show on five standard datasets.
Furthermore, we find that substituting weight decay for BN is sufficient to nullify a relationship between adversarial vulnerability and the input dimension.
A recent mean-field analysis found that BN induces gradient explosion when used on multiple layers, but this cannot fully explain the vulnerability we observe, given that it occurs already for a single BN layer.
We argue that the actual cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance.
As a result, the constant introduced for numerical stability in the BN step acts as an important hyperparameter that can be tuned to recover some robustness at the cost of standard test accuracy.
We explain this mechanism explicitly on a linear ``toy model and show in experiments that it still holds for nonlinear ``real-world models.

assistant: text: Batch normalization reduces robustness at test-time to common corruptions and adversarial examples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Deep neural networks (DNNs) have attained surprising achievement during the last decade due to the advantages of automatic feature learning and freedom of expressiveness.
However,  their interpretability remains mysterious because DNNs are complex combinations of linear and nonlinear transformations.
Even though many models have been proposed to explore the interpretability of DNNs,  several challenges remain unsolved: 1) The lack of interpretability quantity measures for DNNs,  2) the lack of theory for stability of DNNs, and 3) the difficulty to solve nonconvex DNN problems with interpretability constraints.
To address these challenges simultaneously, this paper presents a novel intrinsic interpretability evaluation framework for DNNs.
Specifically, Four independent properties of interpretability are defined based on existing works.
Moreover, we investigate the theory for the stability of DNNs, which is an important aspect of interpretability, and prove that DNNs are generally stable given different activation functions.
Finally, an extended version of deep learning Alternating Direction Method of Multipliers (dlADMM) are proposed to solve DNN problems with interpretability constraints efficiently and accurately.
Extensive experiments on several benchmark datasets validate several DNNs by our proposed interpretability framework.

assistant: text: We propose a novel framework to evaluate the interpretability of neural network.