ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Paper content:
Deep neural networks have become the state-of-the-art models in numerous machine learning tasks.
However, general guidance to network architecture design is still missing.
In our work, we bridge deep neural network design with numerical differential equations.
We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations.
This finding brings us a brand new perspective on the design of effective deep architectures.
We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks.
As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations.
The LM-architecture is an effective structure that can be used on any ResNet-like networks.
In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters.
In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (>50%) the original networks while maintaining a similar performance.
This can be explained mathematically using the concept of modified equation from numerical analysis.
Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks.
Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture.
As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

assistant: text: This paper bridges deep network architectures with numerical (stochastic) differential equations. This new perspective enables new designs of more effective deep neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Irregularly Sampled Time Series Learning with Dense Descriptors of Time

Paper content:
Supervised learning with irregularly sampled time series have been a challenge to Machine Learning methods due to the obstacle of dealing with irregular time intervals.
Some papers introduced recently recurrent neural network models that deals with irregularity, but most of them rely on complex mechanisms to achieve a better performance.
This work propose a novel method to represent timestamps (hours or dates) as dense vectors using sinusoidal functions, called Time Embeddings.
As a data input method it and can be applied to most machine learning models.
The method was evaluated with two predictive tasks from MIMIC III, a dataset of irregularly sampled time series of electronic health records.
Our tests showed an improvement to LSTM-based and classical machine learning models, specially with very irregular data.

assistant: text: A novel method to create dense descriptors of time (Time Embeddings) to make simple models understand temporal structures
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stackelberg GAN: Towards Provable Minimax Equilibrium via Multi-Generator Architectures

Paper content:
We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design.
The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs.
In this work, we give new results on the benefits of multi-generator architecture of GANs.
We show that the minimax gap shrinks to \epsilon as the number of generators increases with rate O(1/\epsilon).
This improves over the best-known result of O(1/\epsilon^2).
At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem.
Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Frechet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets.

assistant: text: We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design, with theoretical guarantees.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Thinking like a machine — generating visual rationales through latent space optimization

Paper content:
Interpretability and small labelled datasets are key issues in the practical application of deep learning, particularly in areas such as medicine.
In this paper, we present a semi-supervised technique that addresses both these issues simultaneously.
We learn dense representations from large unlabelled image datasets, then use those representations to both learn classifiers from small labeled sets and generate visual rationales explaining the predictions.
Using chest radiography diagnosis as a motivating application, we show our method has good generalization ability by learning to represent our chest radiography dataset while training a classifier on an separate set from a different institution.
Our method identifies heart failure and other thoracic diseases.
For each prediction, we generate visual rationales for positive classifications by optimizing a latent representation to minimize the probability of disease while constrained by a similarity measure in image space.
Decoding the resultant latent representation produces an image without apparent disease.
The difference between the original and the altered image forms an interpretable visual rationale for the algorithm's prediction.
Our method simultaneously produces visual rationales that compare favourably to previous techniques and a classifier that outperforms the current state-of-the-art.

assistant: text: We propose a method of using GANs to generate high quality visual rationales to help explain model predictions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unleashing the Potential of CNNs for Interpretable Few-Shot Learning

Paper content:
Convolutional neural networks (CNNs) have been generally acknowledged as one of the driving forces for the advancement of computer vision.
Despite their promising performances on many tasks, CNNs still face major obstacles on the road to achieving ideal machine intelligence.
One is that CNNs are complex and hard to interpret.
Another is that standard CNNs require large amounts of annotated data, which is sometimes very hard to obtain, and it is desirable to be able to learn them from few examples.
In this work, we address these limitations of CNNs by developing novel, simple, and interpretable models for few-shot learn- ing. Our models are based on the idea of encoding objects in terms of visual concepts, which are interpretable visual cues represented by the feature vectors within CNNs.
We first adapt the learning of visual concepts to the few-shot setting, and then uncover two key properties of feature encoding using visual concepts, which we call category sensitivity and spatial pattern.
Motivated by these properties, we present two intuitive models for the problem of few-shot learning.
Experiments show that our models achieve competitive performances, while being much more flexible and interpretable than alternative state-of-the-art few-shot learning methods.
We conclude that using visual concepts helps expose the natural capability of CNNs for few-shot learning.

assistant: text: We enable ordinary CNNs for few-shot learning by exploiting visual concepts which are interpretable visual cues learnt within CNNs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Integer Networks for Data Compression with Latent-Variable Models

Paper content:
We consider the problem of using variational latent-variable models for data compression.
For such models to produce a compressed binary sequence, which is the universal data representation in a digital world, the latent representation needs to be subjected to entropy coding.
Range coding as an entropy coding technique is optimal, but it can fail catastrophically if the computation of the prior differs even slightly between the sending and the receiving side.
Unfortunately, this is a common scenario when floating point math is used and the sender and receiver operate on different hardware or software platforms, as numerical round-off is often platform dependent.
We propose using integer networks as a universal solution to this problem, and demonstrate that they enable reliable cross-platform encoding and decoding of images using variational models.

assistant: text: We train variational models with quantized networks for computational determinism. This enables using them for cross-platform data compression.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
We develop a new algorithm for imitation learning from a single expert demonstration.
In contrast to many previous one-shot imitation learning approaches, our algorithm does not assume access to more than one expert demonstration during the training phase.
Instead, we leverage an exploration policy to acquire unsupervised trajectories, which are then used to train both an encoder and a context-aware imitation policy.
The optimization procedures for the encoder, imitation learner, and exploration policy are all tightly linked.
This linking creates a feedback loop wherein the exploration policy collects new demonstrations that challenge the imitation learner, while the encoder attempts to help the imitation policy to the best of its abilities.
We evaluate our algorithm on 6 MujoCo robotics tasks.

assistant: text: Unsupervised self-imitation algorithm capable of inference from a single expert demonstration.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep RL

Paper content:
Saliency maps are often used to suggest explanations of the behavior of deep rein- forcement learning (RL) agents.
However, the explanations derived from saliency maps are often unfalsifiable and can be highly subjective.
We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and show that explanations suggested by saliency maps are often not supported by experiments.
Our experiments suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.

assistant: text: Proposing a new counterfactual-based methodology to evaluate the hypotheses generated from saliency maps about deep RL agent behavior.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach

Paper content:
The robustness of neural networks to adversarial examples has received great attention due to security implications.
Despite various attack approaches to crafting visually imperceptible adversarial examples, little has been developed towards a comprehensive measure of robustness.
In this paper, we provide theoretical justification for converting robustness analysis into a local Lipschitz constant estimation problem, and propose to use the Extreme Value Theory for efficient evaluation.
Our analysis yields a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness.
The proposed CLEVER score is attack-agnostic and is computationally feasible for large neural networks.
Experimental results on various networks, including ResNet, Inception-v3 and MobileNet, show that
(i) CLEVER is aligned with the robustness indication measured by the $\ell_2$ and $\ell_\infty$ norms of adversarial examples from powerful attacks, and
(ii) defended networks using defensive distillation or bounded ReLU indeed give better CLEVER scores.
To the best of our knowledge, CLEVER is the first attack-independent robustness metric that can be applied to any neural network classifiers.

assistant: text: We propose the first attack-independent robustness metric, a.k.a CLEVER, that can be applied to any neural network classifier.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reward Estimation via State Prediction

Paper content:
Reinforcement learning typically requires carefully designed reward functions in order to learn the desired behavior.
We present a novel reward estimation method that is based on a finite sample of optimal state trajectories from expert demon- strations and can be used for guiding an agent to mimic the expert behavior.
The optimal state trajectories are used to learn a generative or predictive model of the “good” states distribution.
The reward signal is computed by a function of the difference between the actual next state acquired by the agent and the predicted next state given by the learned generative or predictive model.
With this inferred reward function, we perform standard reinforcement learning in the inner loop to guide the agent to learn the given task.
Experimental evaluations across a range of tasks demonstrate that the proposed method produces superior performance compared to standard reinforcement learning with both complete or sparse hand engineered rewards.
Furthermore, we show that our method successfully enables an agent to learn good actions directly from expert player video of games such as the Super Mario Bros and Flappy Bird.

assistant: text: Reward Estimation from Game Videos
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ReNeg and Backseat Driver: Learning from demonstration with continuous human feedback

Paper content:
Reinforcement learning (RL) is a powerful framework for solving problems by exploring and learning from mistakes.
However, in the context of autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world.
For this reason, AV RL is generally only viable in simulation.
Because these simulations have imperfect representations, particularly with respect to graphics, physics, and human interaction, we find motivation for a framework similar to RL, suitable to the real world.
To this end, we formulate a learning framework that learns from restricted exploration by having a human demonstrator do the exploration.
Existing work on learning from demonstration typically either assumes the collected data is performed by an optimal expert, or requires potentially dangerous exploration to find the optimal policy.
We propose an alternative framework that learns continuous control from only safe behavior.
One of our key insights is that the problem becomes tractable if the feedback score that rates the demonstration applies to the atomic action, as opposed to the entire sequence of actions.
We use human experts to collect driving data as well as to label the driving data through a framework we call ``Backseat Driver'', giving us state-action pairs matched with scalar values representing the score for the action.
We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples.
We empirically validate several models in the ReNeg framework, testing on lane-following with limited data.
We find that the best solution in this context outperforms behavioral cloning has strong connections to stochastic policy gradient approaches.

assistant: text: We introduce a novel framework for learning from demonstration that uses continuous human feedback; we evaluate this framework on continuous control for autonomous vehicles.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Unsupervised and semi-supervised learning are important problems that are especially challenging with complex data like natural images.
Progress on these problems would accelerate if we had access to appropriate generative models under which to pose the associated inference tasks.
Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Neural Rendering Model (NRM), a new hierarchical probabilistic generative model whose inference calculations correspond to those in a CNN.
The NRM introduces a small set of latent variables at each level of the model and enforces dependencies among all the latent variables via a conjugate prior distribution.
The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN).
We demonstrate that this regularizer improves generalization both in theory and in practice.
Likelihood estimation in the NRM yields the new Max-Min cross entropy training loss, which suggests a new deep network architecture–the Max- Min network–which exceeds or matches the state-of-art for semi-supervised and supervised learning on SVHN, CIFAR10, and CIFAR100.

assistant: text: We develop a new deep generative model for semi-supervised learning and propose a new Max-Min cross-entropy for training CNNs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Pay Less Attention with Lightweight and Dynamic Convolutions

Paper content:
Self-attention is a useful mechanism to build generative models for language and images.
It determines the importance of context elements by comparing each element to the current time step.
In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results.
Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention.
We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements.
The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic.
Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models.
On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

assistant: text: Dynamic lightweight convolutions are competitive to self-attention on language tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

Paper content:
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time.
Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks.
To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning.
Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context.
Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents.
We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments.
We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot: We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

assistant: text: A model-based meta-RL algorithm that enables a real robot to adapt online in dynamic environments
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spike-based causal inference for weight alignment

Paper content:
In artificial neural networks trained with gradient descent, the weights used for processing stimuli are also used during backward passes to calculate gradients.
For the real brain to approximate gradients, gradient information would have to be propagated separately, such that one set of synaptic weights is used for processing and another set is used for backward passes.
This produces the so-called "weight transport problem" for biological models of learning, where the backward weights used to calculate gradients need to mirror the forward weights used to process stimuli.
This weight transport problem has been considered so hard that popular proposals for biological learning assume that the backward weights are simply random, as in the feedback alignment algorithm.
However, such random weights do not appear to work well for large networks.
Here we show how the discontinuity introduced in a spiking system can lead to a solution to this problem.
The resulting algorithm is a special case of an estimator used for causal inference in econometrics, regression discontinuity design.
We show empirically that this algorithm rapidly makes the backward weights approximate the forward weights.
As the backward weights become correct, this improves learning performance over feedback alignment on tasks such as Fashion-MNIST and CIFAR-10.
Our results demonstrate that a simple learning rule in a spiking network can allow neurons to produce the right backward connections and thus solve the weight transport problem.

assistant: text: We present a learning rule for feedback weights in a spiking neural network that addresses the weight transport problem.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Paper content:
The recent “Lottery Ticket Hypothesis” paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keep the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights.
The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood.
In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results.
Ablating these factors leads to new insights for why LT networks perform as well as they do.
We show why setting weights to zero is important, how signs are all you need to make the re-initialized network train, and why masking behaves like training.
Finally, we discover the existence of Supermasks, or masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).
Many neural networks are over-parameterized BID0 BID1 , enabling compression of each layer BID1 BID14 BID4 or of the entire network BID9 .
Some compression approaches enable more efficient computation by pruning parameters, by factorizing matrices, or via other tricks BID4 BID5 BID8 BID10 BID11 BID12 BID13 BID14 BID15 BID16 .
A recent work by Frankle & Carbin BID2 presented a simple algorithm for finding sparse subnetworks within larger networks that can meet or exceed the performance of the original network.
Their approach is as follows: after training a network, set all weights smaller than some threshold to zero BID2 , rewind the rest of the weights to their initial configuration BID3 , and then retrain the network from this starting configuration but with the zero weights frozen (not trained).
See Section S1 for a more formal description of this algorithm.In this paper we perform ablation studies along the above three dimensions of variability, considering alternate mask criteria (Section 2), alternate mask-1 actions (Section 3), and alternate mask-0 actions (Section 4).
These studies in aggregate reveal new insights for why lottery ticket networks work as they do.
Along the way we also discover the existence of Supermasks-masks that produce above-chance performance when applied to untrained networks (Section 5).

assistant: text: In neural network pruning, zeroing pruned weights is important, sign of initialization is key, and masking can be thought of as training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Kanerva Machine: A Generative Distributed Memory

Paper content:
We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them.
Inspired by Kanerva's sparse distributed memory, it has a robust  distributed reading and writing mechanism.
The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule.
We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution.
Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation.
Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets.
Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.

assistant: text: A generative memory model that combines slow-learning neural networks and a fast-adapting linear Gaussian model as memory.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Reinforcement Learning with Implicit Human Feedback

Paper content:
We consider the following central question in the field of Deep Reinforcement Learning (DRL):
How can we use implicit human feedback to accelerate and optimize the training of a DRL algorithm?
State-of-the-art methods rely on any human feedback to be provided explicitly, requiring the active participation of humans (e.g., expert labeling, demonstrations, etc.).
In this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assessing) the agent interacting with the environment.
The human's intrinsic reactions to the agent's behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials.
The implicit feedback is then used to augment the agent's learning in the RL tasks.
We develop a system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment.
As a baseline contribution, we demonstrate the feasibility of capturing error-potentials  of a human observer watching an agent learning to play several different Atari-games using an electroencephalogram (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm with the intent of accelerating its learning of the game.
Building atop the baseline, we then make the following novel contributions in our work:
(i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error-potentials.
(ii) We propose two different frameworks to combine recent advances in DRL into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide implicit feedback while training in the loop, or prior to the training of the RL agent.
(iii) Finally, we scale the implicit human feedback (via ErrP) based RL to reasonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments.

assistant: text: We use implicit human feedback (via error-potentials, EEG) to accelerate and optimize the training of a DRL algorithm, in a practical manner.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RedSync : Reducing Synchronization Traffic for Distributed Deep Learning

Paper content:
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes.
Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently.
Among several recent proposed compression algorithms, 
Residual Gradient Compression (RGC) is one of the most successful approaches---it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy.
However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated.
In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems.
Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead.
We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server.
Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets.
For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement.

assistant: text: We proposed an implementation to accelerate DNN data parallel training by reducing communication bandwidth requirement.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scalable Gradients and Variational Inference for Stochastic Differential Equations

Paper content:
We derive reverse-mode (or adjoint) automatic differentiation for solutions of stochastic differential equations (SDEs), allowing time-efficient and constant-memory computation of pathwise gradients, a continuous-time analogue of the reparameterization trick.
Specifically, we construct a backward SDE whose solution is the gradient and provide conditions under which numerical solutions converge.
We also combine our stochastic adjoint approach with a stochastic variational inference scheme for continuous-time SDE models, allowing us to learn distributions over functions using stochastic gradient descent.
Our latent SDE model achieves competitive performance compared to existing approaches on time series modeling.

assistant: text: We present a constant memory gradient computation procedure through solutions of stochastic differential equations (SDEs) and apply the method for learning latent SDE models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Understanding and Improving Sequence-Labeling NER with Self-Attentive LSTMs

Paper content:
This paper improves upon the line of research that formulates named entity recognition (NER) as a sequence-labeling problem.
We use so-called black-box long short-term memory (LSTM) encoders to achieve state-of-the-art results while providing insightful understanding of what the auto-regressive model learns with a parallel self-attention mechanism.
Specifically, we decouple the sequence-labeling problem of NER into entity chunking, e.g., Barack_B Obama_E was_O elected_O, and entity typing, e.g., Barack_PERSON Obama_PERSON was_NONE elected_NONE, and analyze how the model learns to, or has difficulties in, capturing text patterns for each of the subtasks.
The insights we gain then lead us to explore a more sophisticated deep cross-Bi-LSTM encoder, which proves better at capturing global interactions given both empirical results and a theoretical justification.

assistant: text: We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks

Paper content:
Training with larger number of parameters while keeping fast iterations is an increasingly
adopted strategy and trend for developing better performing Deep Neural
Network (DNN) models.
This necessitates increased memory footprint and
computational requirements for training.
Here we introduce a novel methodology
for training deep neural networks using 8-bit floating point (FP8) numbers.
Reduced bit precision allows for a larger effective memory and increased computational
speed.
We name this method Shifted and Squeezed FP8 (S2FP8).
We
show that, unlike previous 8-bit precision training methods, the proposed method
works out of the box for representative models: ResNet50, Transformer and NCF.
The method can maintain model accuracy without requiring fine-tuning loss scaling
parameters or keeping certain layers in single precision.
We introduce two
learnable statistics of the DNN tensors - shifted and squeezed factors that are used
to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in
information due to quantization.

assistant: text: We propose a novel 8-bit format that eliminates the need for loss scaling, stochastic rounding, and other low precision techniques
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep-Net: Deep Neural Network for Cyber Security Use Cases

Paper content:
Deep neural networks (DNNs) have witnessed as a powerful approach in this year by solving long-standing Artificial
intelligence (AI) supervised and unsupervised tasks exists in natural language processing, speech processing, computer vision and others.
In this paper, we attempt to apply DNNs on three different cyber security use cases: Android malware classification, incident detection and fraud detection.
The data set of each use case contains real known benign and malicious activities samples.
These use cases are part of Cybersecurity Data Mining Competition (CDMC) 2017.
The efficient network architecture for DNNs is chosen by conducting various trails of experiments for network parameters and network structures.
The experiments of such chosen efficient configurations of DNNs are run up to 1000 epochs with learning rate set in the range [0.01-0.5].
Experiments of DNNs performed well in comparison to the classical machine learning algorithm in all cases of experiments of cyber security use cases.
This is due to the fact that DNNs implicitly extract and build better features, identifies the characteristics of the data that lead to better accuracy.
The best accuracy obtained by DNNs and XGBoost on Android malware classification 0.940 and 0.741, incident detection 1.00 and 0.997, and fraud detection 0.972 and 0.916 respectively.
The accuracy obtained by DNNs varies -0.05%, +0.02%, -0.01% from the top scored system in CDMC 2017 tasks.

assistant: text: Deep-Net: Deep Neural Network for Cyber Security Use Cases
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Vulnerability of Neural Networks Increases with Input Dimension

Paper content:
Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs.
For most current network architectures, we prove that the L1-norm of these gradients grows as the square root of the input size.
These nets therefore become increasingly vulnerable with growing image size.
Our proofs rely on the network’s weight distribution at initialization, but extensive experiments confirm that our conclusions still hold after usual training.

assistant: text: Neural nets have large gradients by design; that makes them adversarially vulnerable.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Attention Interpretability Across NLP Tasks

Paper content:
The attention layer in a neural network model provides insights into the model’s reasoning behind its prediction, which are usually criticized for being opaque.
Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019).
Amid such confusion arises the need to understand attention mechanism more systematically.
In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not).
Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

assistant: text: Analysis of attention mechanism across diverse NLP tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters

Paper content:
While deep neural networks are a highly successful model class, their large memory footprint puts considerable strain on energy consumption, communication bandwidth, and storage requirements.
Consequently, model size reduction has become an utmost goal in deep learning.
Following the classical bits-back argument, we encode the network weights using a random sample, requiring only a number of bits corresponding to the Kullback-Leibler divergence between the sampled variational distribution and the encoding distribution.
By imposing a constraint on the Kullback-Leibler divergence, we are able to explicitly control the compression rate, while optimizing the expected loss on the training set.
The employed encoding scheme can be shown to be close to the optimal information-theoretical lower bound, with respect to the employed variational family.
On benchmarks LeNet-5/MNIST and VGG-16/CIFAR-10, our approach yields the best test performance for a fixed memory budget, and vice versa, it achieves the highest compression rates for a fixed test performance.

assistant: text: This paper proposes an effective coding scheme for neural networks that encodes a random set of weights from a variational distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency

Paper content:
As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents.
Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action.
Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent.
Our approach generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency.
The first captures the impact of perturbation on the relative expected reward of the action to be explained.
The second downweights irrelevant features that alter the relative expected rewards of actions other than the action to be explained.
We compare our approach with existing approaches on agents trained to play board games (Chess and Go) and Atari games (Breakout, Pong and Space Invaders).
We show through illustrative examples (Chess, Atari, Go), human studies (Chess), and automated evaluation methods (Chess) that our approach generates saliency maps that are more interpretable for humans than existing approaches.

assistant: text: We propose a model-agnostic approach to explain the behaviour of black-box deep RL agents, trained to play Atari and board games, by highlighting relevant features of an input state.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scalable Neural Methods for Reasoning With a Symbolic Knowledge   Base

Paper content:
We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB.
This representation enables neural modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically large KBs.
The sparse-matrix reified KB can be distributed across multiple GPUs, can scale to tens of millions of entities and facts, and is orders of magnitude faster than naive sparse-matrix implementations.
The reified KB enables very simple end-to-end architectures to obtain competitive performance on several benchmarks representing two families of tasks: KB completion, and learning semantic parsers from denotations.

assistant: text: A scalable differentiable neural module that implements reasoning on symbolic KBs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Robust, Locally Linear Deep Networks

Paper content:
Deep networks realize complex mappings that are often understood by their locally linear behavior at or around points of interest.
For example, we use the derivative of the mapping with respect to its inputs for sensitivity analysis, or to explain (obtain coordinate relevance for) a prediction.
One key challenge is that such derivatives are themselves inherently unstable.
In this paper, we propose a new learning problem to encourage deep networks to have stable derivatives over larger regions.
While the problem is challenging in general, we focus on networks with piecewise linear activation functions.
Our algorithm consists of an inference step that identifies a region around a point where linear approximation is provably stable, and an optimization step to expand such regions.
We propose a novel relaxation to scale the algorithm to realistic models.
We illustrate our method with residual and recurrent networks on image and sequence datasets.

assistant: text: A scalable algorithm to establish robust derivatives of deep networks w.r.t. the inputs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hallucinating brains with artificial brains

Paper content:
Human brain function as measured by functional magnetic resonance imaging
(fMRI), exhibits a rich diversity.
In response, understanding the individual variability
of brain function and its association with behavior has become one of the
major concerns in modern cognitive neuroscience.
Our work is motivated by the
view that generative models provide a useful tool for understanding this variability.
To this end, this manuscript presents two novel generative models trained
on real neuroimaging data which synthesize task-dependent functional brain images.
Brain images are high dimensional tensors which exhibit structured spatial
correlations.
Thus, both models are 3D conditional Generative Adversarial networks
(GANs) which apply Convolutional Neural Networks (CNNs) to learn an
abstraction of brain image representations.
Our results show that the generated
brain images are diverse, yet task dependent.
In addition to qualitative evaluation,
we utilize the generated synthetic brain volumes as additional training data to improve
downstream fMRI classifiers (also known as decoding, or brain reading).
Our approach achieves significant improvements for a variety of datasets, classifi-
cation tasks and evaluation scores.
Our classification results provide a quantitative
evaluation of the quality of the generated images, and also serve as an additional
contribution of this manuscript.

assistant: text: Two novel GANs are constructed to generate high-quality 3D fMRI brain images and synthetic brain images greatly help to improve downstream classification tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Revisiting the Information Plane

Paper content:
There has recently been a heated debate (e.g. Schwartz-Ziv & Tishby (2017), Saxe et al. (2018), Noshad et al. (2018), Goldfeld et al. (2018)) about measuring the information flow in Deep Neural Networks using techniques from information theory.
It is claimed that Deep Neural Networks in general have good generalization capabilities since they not only learn how to map from an input to an output but also how to compress information about the training data input (Schwartz-Ziv & Tishby, 2017).
That is, they abstract the input information and strip down any unnecessary or over-specific information.
If so, the message compression method, Information Bottleneck (IB), could be used as a natural comparator for network performance, since this method gives an optimal information compression boundary.
This claim was then later denounced as well as reaffirmed (e.g.  Saxe et al. (2018), Achille et al. (2017), Noshad et al. (2018)), as the employed method of mutual information measuring is not actually measuring information but clustering of the internal layer representations (Goldfeld et al. (2018)).
In this paper, we will present a detailed explanation of the development in the Information Plain (IP), which is a plot-type that compares mutual information to judge compression (Schwartz-Ziv & Tishby (2017)), when noise is retroactively added (using binning estimation).
We also explain why different activation functions show different trajectories on the IP.
Further, we have looked into the effect of clustering on the network loss through early and perfect stopping using the Information Plane and how clustering can be used to help network pruning.

assistant: text: We give a detailed explanation of the trajectories in the information plane and investigate its usage for neural network design (pruning)