ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Boosting of Diverse Experts

Paper content:
In this paper, a deep boosting algorithm is developed to
learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs (base experts)
with diverse capabilities, e.g., these base deep CNNs are
sequentially trained to recognize a set of
object classes in an easy-to-hard way according to their
learning complexities.
Our experimental results have demonstrated
that our deep boosting algorithm can significantly improve the
accuracy rates on large-scale visual recognition.

assistant: text: A deep boosting algorithm is developed to learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generating Realistic Stock Market Order Streams

Paper content:
We propose an approach to generate realistic and high-fidelity stock market data based on generative adversarial networks.
We model the order stream as a stochastic process with finite history dependence, and employ a conditional Wasserstein GAN to capture history dependence of orders in a stock market.
We test our approach with actual market and synthetic data on a number of different statistics, and find the generated data to be close to real data.

assistant: text: We propose an approach to generate realistic and high-fidelity stock market data based on generative adversarial networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: 

Paper content:
Key equatorial climate phenomena such as QBO and ENSO have never been adequately explained as deterministic processes.
This in spite of recent research showing growing evidence of predictable behavior.
This study applies the fundamental Laplace tidal equations with simplifying assumptions along the equator — i.e. no Coriolis force and a small angle approximation.
The solutions to the partial differential equations are highly non-linear related to Navier-Stokes and only search approaches can be used to fit to the data.

assistant: text: Analytical Formulation of Equatorial Standing Wave Phenomena: Application to QBO and ENSO
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spatial Information is Overrated for Image Classification

Paper content:
Intuitively, image classification should profit from using spatial information.
Recent work, however, suggests that this might be overrated in standard CNNs.
In this paper, we are pushing the envelope and aim to further investigate the reliance on and necessity of spatial information.
We propose and analyze three methods, namely Shuffle Conv, GAP+FC and 1x1 Conv, that destroy spatial information during both training and testing phases.
We extensively evaluate these methods on several object recognition datasets (CIFAR100, Small-ImageNet, ImageNet) with a wide range of CNN architectures (VGG16, ResNet50, ResNet152, MobileNet, SqueezeNet).
Interestingly, we consistently observe that spatial information can be completely deleted from a significant number of layers with no or only small performance drops.

assistant: text: Spatial information at last layers is not necessary for a good classification accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Traditional and Heavy Tailed Self Regularization in Neural Network Models

Paper content:
Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet.
Empirical and theoretical results clearly indicate that the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of regularization, such as Dropout or Weight Norm constraints.
Building on recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of Implicit Self-Regularization.
For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a "size scale" separating signal from noise.
For state-of-the-art DNNs, however, we identify a novel form of Heavy-Tailed Self-Regularization, similar to the self-organization seen in the statistical physics of disordered systems.
This implicit Self-Regularization can depend strongly on the many knobs of the training process.
By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size.

assistant: text: See the abstract.  (For the revision, the paper is identical, except for a 59 page Supplementary Material, which can serve as a stand-along technical report version of the paper.)
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Paper content:
The convergence rate and final performance of common deep learning models have significantly benefited from recently proposed heuristics such as learning rate schedules, knowledge distillation, skip connections and normalization layers.
In the absence of theoretical underpinnings, controlled experiments aimed at explaining the efficacy of these strategies can aid our understanding of deep learning landscapes and the training dynamics.
Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations.
Instead, we revisit the empirical analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz. mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons why the heuristics succeed.
In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA.
Our empirical analysis suggests that
: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice
; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and
(c) that the latent knowledge shared by the teacher is primarily disbursed in the deeper layers.

assistant: text: We use empirical tools of mode connectivity and SVCCA to investigate neural network training heuristics of learning rate restarts, warmup and knowledge distillation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generalizing Deep Multi-task Learning with Heterogeneous Structured Networks

Paper content:
Many real applications show a great deal of interest in learning multiple tasks from different data sources/modalities with unbalanced samples and dimensions.
Unfortunately, existing cutting-edge deep multi-task learning (MTL) approaches cannot be directly applied to these settings, due to either heterogeneous input dimensions or the heterogeneity in the optimal network architectures of different tasks.
It is thus demanding to develop knowledge-sharing mechanism to handle the intrinsic discrepancies among network architectures across tasks.
To this end, we propose a flexible knowledge-sharing framework for jointly learning multiple tasks from distinct data sources/modalities.
The proposed framework allows each task to own its task (data)-specific network design, via utilizing a compact tensor representation, while the sharing is achieved through the partially shared latent cores.
By providing more elaborate sharing control with latent cores, our framework is effective in transferring task-invariant knowledge, yet also being efficient in learning task-specific features.
Experiments on both single and multiple data sources/modalities settings display the promising results of the proposed method, especially favourable in insufficient data scenarios.

assistant: text: a distributed latent-space based knowledge-sharing framework for deep multi-task learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Can I Trust the Explainer? Verifying Post-Hoc Explanatory Methods

Paper content:
For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks.
In this work, we identify two issues of current explanatory methods.
First, we show that two prevalent perspectives on explanations—feature-additivity and feature-selection—lead to fundamentally different instance-wise explanations.
In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals.
The second issue is that current post-hoc explainers have only been thoroughly validated on simple models, such as linear regression, and, when applied to real-world neural networks, explainers are commonly evaluated under the assumption that the learned models behave reasonably.
However, neural networks often rely on unreasonable correlations, even when producing correct decisions.
We introduce a verification framework for explanatory methods under the feature-selection perspective.
Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings.
We validate the efficacy of our evaluation by showing the failure modes of current explainers.
We aim for this framework to provide a publicly available,1 off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

assistant: text: An evaluation framework based on a real-world neural network for post-hoc explanatory methods
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Line attractor dynamics in recurrent networks for sentiment classiﬁcation

Paper content:
Recurrent  neural  networks  (RNNs)  are  a  powerful tool for modeling sequential data.
Despite their widespread usage, understanding how RNNs solve complex problems remains elusive.
Here, we characterize how popular RNN architectures perform document-level sentiment classification.
Despite their theoretical capacity to implement complex, high-dimensional computations, we find that  trained  networks  converge  to  highly  interpretable, low-dimensional representations.
We identify a simple mechanism, integration along an approximate line attractor, and find this mechanism present across RNN architectures (including LSTMs, GRUs, and vanilla RNNs).
Overall, these results demonstrate that surprisingly universal and human interpretable computations can arise across a range of recurrent networks.
Recurrent neural networks (RNNs) are a popular tool for sequence modelling tasks.
These architectures are thought to learn complex relationships in input sequences, and exploit this structure in a nonlinear fashion.
RNNs are typically viewed as black boxes, despite considerable interest in better understanding how they function.Here, we focus on studying how recurrent networks solve document-level sentiment analysis-a simple, but longstanding benchmark task for language modeling BID6 BID13 .
We demonstrate that popular RNN architectures, despite having the capacity to implement high-dimensional and nonlinear computations, in practice converge to low-dimensional representations when trained against this task.
Moreover, using analysis techniques from dynamical systems theory, we show that locally linear approximations to the nonlinear RNN dynamics are highly interpretable.
In particular, they all involve approximate low-dimensional line attractor dynamics-a useful dynamical feature that can be implemented by linear dynamics and used to store an analog value BID10 .
Furthermore, we show that this mechanism is surprisingly consistent across a range of RNN architectures.
In this work we applied dynamical systems analysis to understand how RNNs solve sentiment analysis.
We found a simple mechanismintegration along a line attractorpresent in multiple architectures trained to solve the task.
Overall, this work provides preliminary, but optimistic, evidence that different, highly intricate network models can converge to similar solutions that may be reduced and understood by human practitioners.

assistant: text: We analyze recurrent networks trained on sentiment classification, and find that they all exhibit approximate line attractor dynamics when solving this task.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-Class Few Shot Learning Task and Controllable Environment

Paper content:
Deep learning approaches usually require a large amount of labeled data to generalize.
However, humans can learn a new concept only by a few samples.
One of the high cogntition human capablities is to learn several concepts at the same time.
In this paper, we address the task of classifying multiple objects by seeing only a few samples from each category.
To the best of authors' knowledge, there is no dataset specially designed for few-shot multiclass classification.
We design a task of mutli-object few class classification and an environment for easy creating controllable datasets for this task.
We demonstrate that the proposed dataset is sound using a method which is an extension of prototypical networks.

assistant: text: We introduce a diagnostic task which is a variation of few-shot learning and introduce a dataset for it.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Model Aggregation via Good-Enough Model Spaces

Paper content:
In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints.
In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global satisficing (i.e. "good-enough") model within a few communication rounds by carefully combining the space of local nodes' satisficing models.
In experiments on benchmark and medical datasets, our approach outperforms other baseline aggregation techniques such as ensembling or model averaging, and performs comparably to the ideal non-distributed models.

assistant: text: We present Good-Enough Model Spaces (GEMS), a framework for learning an aggregate model over distributed nodes within a small number of communication rounds.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Amortized Nesterov's Momentum: Robust and Lightweight  Momentum for Deep Learning

Paper content:
Stochastic Gradient Descent (SGD) with Nesterov's momentum is a widely used optimizer in deep learning, which is observed to have excellent generalization performance.
However, due to the large stochasticity, SGD with Nesterov's momentum is not robust, i.e., its performance may deviate significantly from the expectation.
In this work, we propose Amortized Nesterov's Momentum, a special variant of Nesterov's momentum which has more robust iterates, faster convergence in the early stage and higher efficiency.
Our experimental results show that this new momentum achieves similar (sometimes better) generalization performance with little-to-no tuning.
In the convex case, we provide optimal convergence rates for our new methods and discuss how the theorems explain the empirical results.

assistant: text: Amortizing Nesterov's momentum for more robust, lightweight and fast deep learning training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training Neural Machines with Partial Traces

Paper content:
We present a novel approach for training neural abstract architectures which in- corporates (partial) supervision over the machine’s interpretable components.
To cleanly capture the set of neural architectures to which our method applies, we introduce the concept of a differential neural computational machine (∂NCM) and show that several existing architectures (e.g., NTMs, NRAMs) can be instantiated as a ∂NCM and can thus benefit from any amount of additional supervision over their interpretable components.
Based on our method, we performed a detailed experimental evaluation with both, the NTM and NRAM architectures, and showed that the approach leads to significantly better convergence and generalization capabilities of the learning phase than when training using only input-output examples.

assistant: text: We increase the amount of trace supervision possible to utilize when training fully differentiable neural machine architectures.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control

Paper content:
In this paper, we introduce Symplectic ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system from observed state trajectories.
To achieve better generalization with fewer training samples, SymODEN incorporates appropriate inductive bias by designing the associated computation graph in a physics-informed manner.
In particular, we enforce Hamiltonian dynamics with control to learn the underlying dynamics in a transparent way which can then be leveraged to draw insight about relevant physical aspects of the system, such as mass and potential energy.
In addition, we propose a parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data is embedded in a high-dimensional space or we can only access velocity data instead of generalized momentum.
This framework, by offering interpretable, physically-consistent models for physical systems, opens up new possibilities for synthesizing model-based control strategies.

assistant: text: This work enforces Hamiltonian dynamics with control to learn system models from embedded position and velocity data, and exploits this physically-consistent dynamics to synthesize model-based control via energy shaping.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Characterizing the Capacity of Neural Networks Using Algebraic Topology

Paper content:
The learnability of different neural architectures can be characterized directly by computable measures of data complexity.
In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias.
After suggesting algebraic topology as a measure for data complexity, we show that the power of a network to express the topological complexity of a dataset in its decision boundary is a strictly limiting factor in its ability to generalize.
We then provide the first empirical characterization of the topological capacity of neural networks.
Our empirical analysis shows that at every level of dataset complexity, neural networks exhibit topological phase transitions and stratification.
This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for a single hidden layer neural networks.

assistant: text: We show that the learnability of different neural architectures can be characterized directly by computable measures of data complexity.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR

Paper content:
We address two challenges of probabilistic topic modelling in order to better estimate
the probability of a word in a given context, i.e., P(wordjcontext) : (1) No
Language Structure in Context: Probabilistic topic models ignore word order by
summarizing a given context as a “bag-of-word” and consequently the semantics
of words in the context is lost.
In this work, we incorporate language structure
by combining a neural autoregressive topic model (TM) with a LSTM based language
model (LSTM-LM) in a single probabilistic framework.
The LSTM-LM
learns a vector-space representation of each word by accounting for word order
in local collocation patterns, while the TM simultaneously learns a latent representation
from the entire document.
In addition, the LSTM-LM models complex
characteristics of language (e.g., syntax and semantics), while the TM discovers
the underlying thematic structure in a collection of documents.
We unite two complementary
paradigms of learning the meaning of word occurrences by combining
a topic model and a language model in a unified probabilistic framework, named
as ctx-DocNADE.
(2) Limited Context and/or Smaller training corpus of documents:
In settings with a small number of word occurrences (i.e., lack of context)
in short text or data sparsity in a corpus of few documents, the application of TMs
is challenging.
We address this challenge by incorporating external knowledge
into neural autoregressive topic models via a language modelling approach: we
use word embeddings as input of a LSTM-LM with the aim to improve the wordtopic
mapping on a smaller and/or short-text corpus.
The proposed DocNADE
extension is named as ctx-DocNADEe.
We present novel neural autoregressive topic model variants coupled with neural
language models and embeddings priors that consistently outperform state-of-theart
generative topic models in terms of generalization (perplexity), interpretability
(topic coherence) and applicability (retrieval and classification) over 6 long-text
and 8 short-text datasets from diverse domains.

assistant: text: Unified neural model of topic and language modeling to introduce language structure  in topic models for contextualized topic vectors
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Randomness in Deconvolutional Networks for Visual Representation

Paper content:
To understand the inner work of deep neural networks and provide possible theoretical explanations, we study the deep representations through the untrained, random weight CNN-DCN architecture.
As a convolutional AutoEncoder, CNN indicates the portion of a convolutional neural network from the input to an intermediate convolutional layer, and DCN indicates the corresponding deconvolutional portion.
As compared with DCN training for pre-trained CNN, training the DCN for random-weight CNN converges more quickly and yields higher quality image reconstruction.
Then, what happens for the overall random CNN-DCN?
We gain intriguing results that the image can be reconstructed with good quality.
To gain more insight on the intermediate random representation, we investigate the impact of network width versus depth, number of random channels, and size of random kernels on the reconstruction quality, and provide theoretical justifications on empirical observations.
We further provide a fast style transfer application using the random weight CNN-DCN architecture to show the potential of our observation.

assistant: text: We investigate the deep representation of untrained, random weight CNN-DCN architectures, and show their image reconstruction quality and possible applications.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks

Paper content:
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time.
To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process.
However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency).
In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA (Espeholt et al., 2018)), sample efficiency drops significantly.
To address this, we propose a new distributed reinforcement learning algorithm, IMPACT.
IMPACT extends PPO with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling.
In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA.
For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.

assistant: text: IMPACT helps RL agents train faster by decreasing training wall-clock time and increasing sample efficiency simultaneously.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Representing Model Uncertainty of Neural Networks in Sparse Information Form

Paper content:
This paper addresses the problem of representing a system's belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN).
The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs.
To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form.
Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis.
As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank   approximation of this eigenbasis and a memory-efficient sampling scheme.
We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.

assistant: text: An approximate inference algorithm for deep learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dynamic Neural Program Embeddings for Program Repair

Paper content:
Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, code completion, and fault localization.
However, most existing program embeddings are based on syntactic features of programs, such as token sequences or abstract syntax trees.
Unlike images and text, a program has well-deﬁned semantics that can be difﬁcult to capture by only considering its syntax (i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited.
We propose a novel semantic program embedding that is learned from program execution traces.
Our key insight is that program states expressed as sequential tuples of live variable values not only capture program semantics more precisely, but also offer a more natural ﬁt for Recurrent Neural Networks to model.
We evaluate different syntactic and semantic program embeddings on the task of classifying the types of errors that students make in their submissions to an introductory programming class and on the CodeHunt education platform.
Our evaluation results show that the semantic program embeddings signiﬁcantly outperform the syntactic program embeddings based on token sequences and abstract syntax trees.
In addition, we augment a search-based program repair system with predictions made from our semantic embedding and demonstrate signiﬁcantly improved search efﬁciency.

assistant: text: A new way of learning semantic program embedding
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: BA-Net: Dense Bundle Adjustment Networks

Paper content:
This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error.
The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable.
Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth.
The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA.
The basis depth maps generator is also learned via end-to-end training.
The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem.
Experiments on large scale real data prove the success of the proposed method.

assistant: text: This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature bundle adjustment (BA)
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Deep Reinforced Model for Abstractive Summarization

Paper content:
Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences.
For longer documents and summaries however these models often include repetitive and incoherent phrases.
We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL).
Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training.
However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable.
We evaluate this model on the CNN/Daily Mail and New York Times datasets.
Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models.
Human evaluation also shows that our model produces higher quality summaries.

assistant: text: A summarization model combining a new intra-attention and reinforcement learning method to increase summary ROUGE scores and quality for long sequences.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Simplicial Complex Networks

Paper content:
Universal approximation property of neural networks is one of the motivations to use these models in various real-world problems.
However, this property is not the only characteristic that makes neural networks unique as there is a wide range of other approaches with similar property.
Another characteristic which makes these models interesting is that they can be trained with the backpropagation algorithm which allows an efficient gradient computation and gives these universal approximators the ability to efficiently learn complex manifolds from a large amount of data in different domains.
Despite their abundant use in practice, neural networks are still not well understood and a broad range of ongoing research is to study the interpretability of neural networks.
On the other hand, topological data analysis (TDA) relies on strong theoretical framework of (algebraic) topology along with other mathematical tools for analyzing possibly complex datasets.
In this work, we leverage a universal approximation theorem originating from algebraic topology to build a connection between TDA and common neural network training framework.
We introduce the notion of automatic subdivisioning and devise a particular type of neural networks for regression tasks: Simplicial Complex Networks (SCNs).
SCN's architecture is defined with a set of bias functions along with a particular policy during the forward pass which alternates the common architecture search framework in neural networks.
We believe the view of SCNs can be used as a step towards building interpretable deep learning models.
Finally, we verify its performance on a set of regression problems.

assistant: text: A novel method for supervised learning through subdivisioning the input space along with function approximation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Answer-based Adversarial Training for Generating Clarification Questions

Paper content:
We propose a generative adversarial training approach for the problem of clarification question generation.
Our approach generates clarification questions with the goal of eliciting new information that would make the given context more complete.
We develop a Generative Adversarial Network (GAN) where the generator is a sequence-to-sequence model and the discriminator is a utility function that models the value of updating the context with the answer to the clarification question.
We evaluate on two datasets, using both automatic metrics and human judgments of usefulness, specificity and relevance, showing that our approach outperforms both a retrieval-based model and ablations that exclude the utility model and the adversarial training.

assistant: text: We propose an adversarial training approach to the problem of clarification question generation which uses the answer to the question to model the reward.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Perception-Driven Curiosity with Bayesian Surprise

Paper content:
Intrinsic rewards in reinforcement learning provide a powerful algorithmic capability for agents to learn how to interact with their environment in a task-generic way.
However, increased incentives for motivation can come at the cost of increased fragility to stochasticity.
We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics.
Ultimately, an estimate of the conditional probability of observed states is used as our intrinsic reward for curiosity.
In our experiments, a video game agent uses our model to autonomously learn how to play Atari games using our curiosity reward in combination with extrinsic rewards from the game to achieve improved performance on games with sparse extrinsic rewards.
When stochasticity is introduced in the environment, our method still demonstrates improved performance over the baseline.

assistant: text: We introduce a method for computing an intrinsic reward for curiosity using metrics derived from sampling a latent variable model used to estimate dynamics.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Lattice Representation Learning

Paper content:
We introduce the notion of \emph{lattice representation learning}, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space.
Our main contribution is a result for replacing an objective function which employs lattice quantization with an objective function in which quantization is absent, thus allowing optimization techniques based on gradient descent to apply; we call the resulting algorithms \emph{dithered stochastic gradient descent} algorithms as they are designed explicitly to allow for an optimization procedure where only local information is employed.
We also argue that a technique commonly used in Variational Auto-Encoders (Gaussian priors and Gaussian approximate posteriors) is tightly connected with the idea of lattice representations, as the quantization error in good high dimensional lattices can be modeled as a Gaussian distribution.
We use a traditional encoder/decoder architecture to explore the idea of latticed valued representations, and provide experimental evidence of the potential of using lattice representations by modifying the \texttt{OpenNMT-py} generic \texttt{seq2seq} architecture  so that it can implement not only Gaussian dithering of representations, but also the well known straight-through estimator and its application to vector quantization.

assistant: text: We propose to use lattices to represent objects and prove a fundamental result on how to train networks that use them.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Decoupling the Layers in Residual Networks

Paper content:
We propose a Warped Residual Network (WarpNet) using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network.
We apply a perturbation theory on residual networks and decouple the interactions between residual units.
The resulting warp operator is a first order approximation of the output over multiple layers.
The first order perturbation theory exhibits properties such as binomial path lengths and exponential gradient scaling found experimentally by Veit et al (2016).
We demonstrate through an extensive performance study that the proposed network achieves comparable predictive performance to the original residual network with the same number of parameters, while achieving a significant speed-up on the total training time.
As WarpNet performs model parallelism in residual network training in which weights are distributed over different GPUs, it offers speed-up and capability to train larger networks compared to original residual networks.

assistant: text: We propose the Warped Residual Network using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: What Information Does a ResNet Compress?

Paper content:
The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective.
However, this claim was established on toy data.
The goal of the work we present here is to test these claims in a realistic setting using a larger and deeper convolutional architecture, a ResNet model.
We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding.
We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder.
Sampling images by conditioning on hidden layers’ activations offers an intuitive visualisation to understand what a ResNets learns to forget.

assistant: text: The Information Bottleneck Principle applied to ResNets, using PixelCNN++ models to decode mutual information and conditionally generate images for information illustration
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Information Bottleneck Theory of Deep Learning

Paper content:
The practical successes of deep neural networks have not been matched by theoretical progress that satisfyingly explains their behavior.
In this work, we study the information bottleneck (IB) theory of deep learning, which makes three specific claims: first, that deep networks undergo two distinct phases consisting of an initial fitting phase and a subsequent compression phase; second, that the compression phase is causally related to the excellent generalization performance of deep networks; and third, that the compression phase occurs due to the diffusion-like behavior of stochastic gradient descent.
Here we show that none of these claims hold true in the general case.
Through a combination of analytical results and simulation, we demonstrate that the information plane trajectory is predominantly a function of the neural nonlinearity employed: double-sided saturating nonlinearities like tanh yield a compression phase as neural activations enter the saturation regime, but linear activation functions and single-sided saturating nonlinearities like the widely used ReLU in fact do not.
Moreover, we find that there is no evident causal connection between compression and generalization: networks that do not compress are still capable of generalization, and vice versa.
Next, we show that the compression phase, when it exists, does not arise from stochasticity in training by demonstrating that we can replicate the IB findings using full batch gradient descent rather than stochastic gradient descent.
Finally, we show that when an input domain consists of a subset of task-relevant and task-irrelevant information, hidden representations do compress the task-irrelevant information, although the overall information about the input may monotonically increase with training time, and that this compression happens concurrently with the fitting process rather than during a subsequent compression period.

assistant: text: We show that several claims of the information bottleneck theory of deep learning are not true in the general case.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective

Paper content:
This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piece-wise linear non-linearity activations.
We use tropical geometry, a new development in the area of algebraic geometry, to provide a characterization of the decision boundaries of a simple neural network of the form (Affine, ReLU, Affine).
Specifically, we show that the decision boundaries are a subset of a tropical hypersurface, which is intimately related to a polytope formed by the convex hull of two zonotopes.
The generators of the zonotopes are precise functions of the neural network parameters.
We utilize this geometric characterization to shed light and new perspective on three tasks.
In doing so, we propose a new tropical perspective for the lottery ticket hypothesis, where we see the effect of different initializations on the tropical geometric representation of the decision boundaries.
Also, we leverage this characterization as a new set of tropical regularizers, which  deal directly  with the decision boundaries of a network.
We investigate the use of these regularizers  in neural network pruning (removing network parameters that do not contribute to the tropical geometric representation of the decision boundaries) and in generating adversarial input attacks (with input perturbations explicitly perturbing the decision boundaries geometry to change the network prediction of the input).

assistant: text: Tropical geometry can be leveraged to represent the decision boundaries of neural networks and bring to light interesting insights.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Model-based imitation learning from state trajectories

Paper content:
Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions.
However, in real life expert demonstrations, often the action information is missing and only state trajectories are available.
We present a model-based imitation learning method that can learn environment-specific optimal actions only from expert state trajectories.
Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability.
Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics.
Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods.
We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations.

assistant: text: Learning to imitate an expert in the absence of optimal actions learning a dynamics model while exploring the environment.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference

Paper content:
Nonlinearity is crucial to the performance of a deep (neural) network (DN).
To date there has been little progress understanding the menagerie of available  nonlinearities, but recently progress has been made on understanding the r\^{o}le played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling.
In particular, DN layers constructed from these operations can be interpreted as {\em max-affine spline operators} (MASOs) that have an elegant link to vector quantization (VQ) and $K$-means.
While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax.
{\em This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).
}
We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural ``hard'' VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding ``soft'' VQ inference problems.
We further extend the framework by hybridizing the hard and soft VQ optimizations to create a $\beta$-VQ inference that interpolates between hard, soft, and linear VQ inference.
A prime example of a $\beta$-VQ DN nonlinearity is the {\em swish} nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation.
Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.

assistant: text: Reformulate deep networks nonlinearities from a vector quantization scope and bridge most known nonlinearities together.