ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Function-Specific Word Representations

Paper content:
We present a neural framework for learning associations between interrelated groups of words such as the ones found in Subject-Verb-Object (SVO) structures.
Our model induces a joint function-specific word vector space, where vectors of e.g. plausible SVO compositions lie close together.
The model retains information about word group membership even in the joint space, and can thereby effectively be applied to a number of tasks reasoning over the SVO structure.
We show the robustness and versatility of the proposed framework by reporting state-of-the-art results on the tasks of estimating selectional preference (i.e., thematic fit) and event similarity.
The results indicate that the combinations of representations learned with our task-independent model outperform task-specific architectures from prior work, while reducing the number of parameters by up to 95%.
The proposed framework is versatile and holds promise to support learning function-specific representations beyond the SVO structures.

assistant: text: Task-independent neural model for learning associations between interrelated groups of words.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep 3D-Zoom Net: Unsupervised Learning of Photo-Realistic 3D-Zoom

Paper content:
The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane.
In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size.
In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image.
An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis.
Besides, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes.
We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net.
The Deep 3D-Zoom Net incorporates the following features:
(i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss;
(ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and
(iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas.
Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines.
We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets.

assistant: text: A novel network architecture to perform Deep 3D Zoom or close-ups.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Match prediction from group comparison data using neural networks

Paper content:
We explore the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data.
Challenges arise in practice.
As existing state-of-the-art algorithms are tailored to certain statistical models, we have different best algorithms across distinct scenarios.
Worse yet, we have no prior knowledge on the underlying model for a given scenario.
These call for a unified approach that can be universally applied to a wide range of scenarios and achieve consistently high performances.
To this end, we incorporate deep learning architectures so as to reflect the key structural features that most state-of-the-art algorithms, some of which are optimal in certain settings, share in common.
This enables us to infer hidden models underlying a given dataset, which govern in-group interactions and statistical patterns of comparisons, and hence to devise the best algorithm tailored to the dataset at hand.
Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-art algorithms.
It turns out that our framework consistently leads to the best performance across all datasets in terms of cross entropy loss and prediction accuracy, while the state-of-the-art algorithms suffer from inconsistent performances across different datasets.
Furthermore, we show that it can be easily extended to attain satisfactory performances in rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.

assistant: text: This paper proposes a deep neural network solution to the set ranking problem and designs a architecture for this task inspired by previous manually designed algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Passage Ranking with Weak Supervision

Paper content:
In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources.
Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities.
We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework.
Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets.

assistant: text: We propose a weak supervision training pipeline based on the data programming framework for ranking tasks, in which we train a BERT-base ranking model and establish the new SOTA.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Flexible degrees of connectivity under synaptic weight constraints

Paper content:
Biological neural networks face homeostatic and resource constraints that restrict the allowed configurations of connection weights.
If a constraint is tight it defines a very small solution space, and the size of these constraint spaces determines their potential overlap with the solutions for computational tasks.
We study the geometry of the solution spaces for constraints on neurons' total synaptic weight and on individual synaptic weights, characterizing the connection degrees (numbers of partners) that maximize the size of these solution spaces.
We then hypothesize that the size of constraints' solution spaces could serve as a cost function governing neural circuit development.
We develop analytical approximations and bounds for the model evidence of the maximum entropy degree distributions under these cost functions.
We test these on a published electron microscopic connectome of an associative learning center in the fly brain, finding evidence for a developmental progression in circuit structure.

assistant: text: We examine the hypothesis that the entropy of solution spaces for constraints on synaptic weights (the "flexibility" of the constraint) could serve as a cost function for neural circuit development.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: One-shot learning: From domain knowledge to action models

Paper content:
Most approaches to learning action planning models heavily rely on a significantly large volume of training samples or plan observations.
In this paper, we adopt a different approach based on deductive learning from domain-specific knowledge, specifically from logic formulae that specify constraints about the possible states of a given domain.
The minimal input observability required by our approach is a single example composed of a full initial state and a partial goal state.
We will show that exploiting specific domain knowledge enables to constrain the space of possible action models as well as to complete partial observations, both of which turn out helpful to learn good-quality action models.

assistant: text: Hybrid approach to model acquisition that compensates a lack of available data with domain specific knowledge provided by experts
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Good Policies By Learning Good Perceptual Models

Paper content:
Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years.
However, such complexity can be misleading and hides over-fitting.
We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization.
We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images.
Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.

assistant: text: This paper formulates curiosity based RL training as learning a visual representation model, arguing that focusing on better LR and maximising model loss for novel scenes will get better overall performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Evaluating the distribution learning capabilities of GANs

Paper content:
We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets.
The datasets include common distributions of points in $R^n$ space and images containing polygons of various shapes and sizes.
We find that by and large GANs fail to faithfully recreate point datasets which contain discontinous support or sharp bends with noise.
Additionally, on image datasets, we find that GANs do not seem to learn to count the number of objects of the same kind in an image.
We also highlight the apparent tension between generalization and learning in GANs.
Generative Adversarial Models (GANs) BID4 have been found to produce images of very high quality on some datasets BID6 .
However, their results on other datasets, while impressive, still lag behind BID1 .
This raises the question whether GANs are indeed the right choice to model some distributions.
This paper aims to test the distribution learning ability of GANs by evaluating them on synthetic datasets.
In this paper, we present the phenomenon of GANs being unable to count.
We support this hypothesis with experiments on synthetic datasets where the count of similar objects in a scene is kept constant while their location is varied.
We find that in their current form GANs are unable to learn semantic constraints even in the absence of noise introduced by natural image datasets.
We also emphasize the fine line between generalization and good learning outcomes in GANs.
Additionally, we conduct experiments on non-image data where we conclude that GANs tend to have difficulty learning discontinuous distributions which might necessitate the usage of mixtures of generators.
A thorough evaluation of such an approach is left as future work.
Each dataset has associated noise parameters corresponding to the noise parameters in the scikit-learn API.
We experiment with varying noise but we find that it does not affect learning outcomes too much.

assistant: text: GANs are evaluated on synthetic datasets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Uncertainty-aware Variational-Recurrent Imputation Network for Clinical Time Series

Paper content:
Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.
Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.
In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.
In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.
Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.
We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

assistant: text: A missing data imputation network to incorporate correlation, temporal relationships, and data uncertainty for the problem of data sparsity in EHRs, which yields higher AUC on mortality rate classification tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards an Adversarially Robust Normalization Approach

Paper content:
Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.
However, recently it has been shown that it is also vulnerable to adversarial perturbations.
In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.
We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.
We empirically proved this by experiments on various neural network architectures and datasets.
Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

assistant: text: Investigation of how BatchNorm causes adversarial vulnerability and how to avoid it.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variance Reduction for Reinforcement Learning in Input-Driven Environments

Paper content:
We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system.
Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking.
Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns.
Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training.
We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines.
We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs.
Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.

assistant: text: The authors consider the problem of learning in input-driven environments, show how the PG theorem still applies for an input-aware critic, and show that input-dependent baselines are the best to use in conjecture with that critic.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift

Paper content:
Few-Shot Learning (learning with limited labeled data) aims to overcome the limitations of traditional machine learning approaches which require thousands of labeled examples to train an effective model.
Considered as a hallmark of human intelligence, the community has recently witnessed several contributions on this topic, in particular through meta-learning, where a model learns how to learn an effective model for few-shot learning.
The main idea is to acquire prior knowledge from a set of training tasks, which is then used to perform (few-shot) test tasks.
Most existing work assumes that both training and test tasks are drawn from the same distribution, and a large amount of labeled data is available in the training tasks.
This is a very strong assumption which restricts the usage of meta-learning strategies in the real world where ample training tasks following the same distribution as test tasks may not be available.
In this paper, we propose a novel meta-learning paradigm wherein a few-shot learning model is learnt, which simultaneously overcomes domain shift between the train and test tasks via adversarial domain adaptation.
We demonstrate the efficacy the proposed method through extensive experiments.

assistant: text: This paper proposes a model combining unsupervised adversarial domain adaptation with prototypical networks that performs better than few-shot learning baselines on few-shot learning tasks with domain shift.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Robust Joint Representations for Multimodal Sentiment Analysis

Paper content:
Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities.
The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities.
However, existing work learns joint representations using multiple modalities as input and may be sensitive to noisy or missing modalities at test time.
With the recent success of sequence to sequence models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time.
In this paper, we propose a method to learn robust joint representations by translating between modalities.
Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input.
We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities.
Once our translation model is trained with paired multimodal data, we only need data from the source modality at test-time for prediction.
This ensures that our model remains robust from perturbations or missing target modalities.
We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube.
Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to perturbations of all other modalities.

assistant: text: This paper presents the Multimodal Cyclic Translation Network (MCTN) and evaluates it for multimodal sentiment analysis.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Weight-space symmetry in neural network loss landscapes revisited

Paper content:
Neural network training depends on the structure of the underlying loss landscape, i.e. local minima, saddle points, flat plateaus, and loss barriers.
In relation to the structure of the landscape, we study the permutation symmetry of neurons in each layer of a deep neural network, which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima.
In a network of $d-1$ hidden layers with $n_k$ neurons in layers $k = 1, \ldots, d$, we construct continuous paths between equivalent global minima that lead through a `permutation point' where the input and output weight vectors of two neurons in the same hidden layer $k$ collide and interchange.
We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss, contributing to the global flatness of the landscape.
We also find that a permutation point for the exchange of neurons $i$ and $j$ transits into a flat high-dimensional plateau that enables all $n_k!$
permutations of neurons in a given layer $k$ at the same loss value. Moreover
, we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks, and find that the number of $K$-th order permutation points is much larger than the (already huge) number of equivalent global minima -- at least by a polynomial factor of order $K$. In two
tasks, we demonstrate numerically with our path finding method that continuous paths between partner minima exist: first, in a toy network with a single hidden layer on a function approximation task and, second, in a multilayer network on the MNIST task. Our geometric
approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations.

assistant: text: The paper presented a low-loss method for studying the loss function with respect to parameters in a neural network from the perspective of weight-space symmetry.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Paper content:
Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.
While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs.
iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).
We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.
We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

assistant: text: The authors address the problem of signal propagation in recurrent neural networks by building an attractor system for the signal transition and checking whether it converges to an equilibrium.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing Batch Normalized Convolutional Networks using Displaced Rectifier Linear Units: A Systematic Comparative Study

Paper content:
In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently.
We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization.
Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models.
These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets.
The results showed DReLU speeded up learning in all models and datasets.
Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios.
Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance.
Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

assistant: text: The paper compares and suggests against the usage of batch normalization after using rectifier linear units
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bamboo: Ball-Shape Data Augmentation Against Adversarial Attacks from All Directions

Paper content:
Deep neural networks (DNNs) are widely adopted in real-world cognitive applications because of their high accuracy.
The robustness of DNN models, however, has been recently challenged by adversarial attacks where small disturbance on input samples may result in misclassification.
State-of-the-art defending algorithms, such as adversarial training or robust optimization, improve DNNs' resilience to adversarial attacks by paying high computational costs.
Moreover, these approaches are usually designed to defend one or a few known attacking techniques only.
The effectiveness to defend other types of attacking methods, especially those that have not yet been discovered or explored, cannot be guaranteed.
This work aims for a general approach of enhancing the robustness of DNN models under adversarial attacks.
In particular, we propose Bamboo -- the first data augmentation method designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.
Bamboo augments the training data set with a small amount of data uniformly sampled on a fixed radius ball around each training data and hence, effectively increase the distance between natural data points and decision boundary.
Our experiments show that Bamboo substantially improve the general robustness against arbitrary types of attacks and noises, achieving better results comparing to previous adversarial training methods, robust optimization methods and other data augmentation methods with the same amount of data points.

assistant: text: The first data augmentation method specially designed for improving the general robustness of DNN without any hypothesis on the attacking algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards an Adversarially Robust Normalization Approach

Paper content:
Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.
However, recently it has been shown that it is also vulnerable to adversarial perturbations.
In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.
We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.
We empirically proved this by experiments on various neural network architectures and datasets.
Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

assistant: text: This paper addresses vulnerability to adversarial perturbations in BatchNorm, and proposes an alternative called RobustNorm, using min-max rescaling instead of normalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discriminative out-of-distribution detection for semantic segmentation

Paper content:
Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes.
However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input.
These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive.
We propose to address this issue by discriminative detection
of OOD pixels in input data.
Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task.
Instead, we train a dedicated OOD model
which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world.
We perform our experiments on high resolution natural images in a dense prediction setup.
We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset.
We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.
The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

assistant: text: This paper aims to detect out-of-distribution pixels for semantic segmentation, and this work utilizes data from other domains to detect undetermined classes to model uncertainty better.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Influence-Directed Explanations for Deep Convolutional Networks

Paper content:
We study the problem of explaining a rich class of behavioral properties of deep neural networks.
Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent.
We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.
Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

assistant: text: A way to measure influence that satisfies certain axioms, and a notion of influence that can be used to identify what input part is most influential for the output of a neuron in a deep neural network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DISENTANGLED STATE SPACE MODELS: UNSUPERVISED LEARNING OF DYNAMICS ACROSS HETEROGENEOUS ENVIRONMENTS

Paper content:
Sequential data often originates from diverse environments.
Across them exist both shared regularities and environment specifics.
To learn robust cross-environment descriptions of sequences we introduce disentangled state space models (DSSM).
In the latent space of DSSM environment-invariant state dynamics is explicitly disentangled from environment-specific information governing that dynamics.
We empirically show that such separation enables robust prediction, sequence manipulation and environment characterization.
We also propose an unsupervised VAE-based training procedure to learn DSSM as Bayesian filters.
In our experiments, we demonstrate state-of-the-art performance in controlled generation and prediction of bouncing ball video sequences across varying gravitational influences.

assistant: text: DISENTANGLED STATE SPACE MODELS
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Paper content:
We explore the concept of co-design in the context of neural network verification.
Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily.
To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task.
We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones.
Then, improving ReLU stability leads to an additional 4-13x speedup in verification times.
An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

assistant: text: The paper presents several ways to regularize plain ReLU networks to opimize the adversarial robustness, provable adversarial robustness, and the verification speed.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TOWARDS ROBOT VISION MODULE DEVELOPMENT WITH EXPERIENTIAL ROBOT LEARNING

Paper content:
n this paper we present a thrust in three directions of visual development us- ing supervised and semi-supervised techniques.
The first is an implementation of semi-supervised object detection and recognition using the principles of Soft At- tention and Generative Adversarial Networks (GANs).
The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs).
The three thrusts to- gether are based on the approach of Experiential Robot Learning, introduced in previous publication.
While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic vi- sual modules.

assistant: text: Investigates is performance of existing image classifiers and object detectors.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Global Additive Explanations for Neural Nets Using Model Distillation

Paper content:
Interpretability has largely focused on local explanations, i.e. explaining why a model made a particular prediction for a sample.
These explanations are appealing due to their simplicity and local fidelity.
However, they do not provide information about the general behavior of the model.
We propose to leverage model distillation to learn global additive explanations that describe the relationship between input features and model predictions.
These global explanations take the form of feature shapes, which are more expressive than feature attributions.
Through careful experimentation, we show qualitatively and quantitatively that global additive explanations are able to describe model behavior and yield insights about models such as neural nets.
A visualization of our approach applied to a neural net as it is trained is available at https://youtu.be/ErQYwNqzEdc

assistant: text: We propose to leverage model distillation to learn global additive explanations in the form of feature shapes (that are more expressive than feature attributions) for models such as neural nets trained on tabular data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MEMO: A Deep Network for Flexible Combination of Episodic Memories

Paper content:
Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning.
Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures.
This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories.
Surprisingly, we found that current architectures struggle to reason over long distance associations.
Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path.
We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances.
This was accomplished with the addition of two novel components.
First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory.
Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of ‘memory hops’ before the answer is produced.
MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

assistant: text: A new task (paired associate inference) drawn from cognitive psychology, and proposal for a new memory architecture with features that allow for better performance on the paired associate task.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Experience replay for continual learning

Paper content:
Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster.
Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions.
This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills.
We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence.
Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience.
While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning.
We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities.
When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

assistant: text: Proposes a particular variant of experience replay with behavior cloning as a method for continual learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Universality Theorems for Generative Models

Paper content:
Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice.
In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well?
We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold.
We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

assistant: text: This paper provides certain basic guarantees on when manifolds can be written as the image of a map approximated by a neural net, and stitches together theorems from manifold geometry and standard universal approximation results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Through Limited Self-Supervision: Improving Time-Series Classification Without Additional Data via Auxiliary Tasks

Paper content:
Self-supervision, in which a target task is improved without external supervision, has primarily been explored in settings that assume the availability of additional data.
However, in many cases, particularly in healthcare, one may not have access to additional data (labeled or otherwise).
In such settings, we hypothesize that self-supervision based solely on the structure of the data at-hand can help.
We explore a novel self-supervision framework for time-series data, in which multiple auxiliary tasks (e.g., forecasting) are included to improve overall performance on a sequence-level target task without additional training data.
We call this approach limited self-supervision, as we limit ourselves to only the data at-hand.
We demonstrate the utility of limited self-supervision on three sequence-level classification tasks, two pertaining to real clinical data and one using synthetic data.
Within this framework, we introduce novel forms of self-supervision and demonstrate their utility in improving performance on the target task.
Our results indicate that limited self-supervision leads to a consistent improvement over a supervised baseline, across a range of domains.
In particular, for the task of identifying atrial fibrillation from small amounts of electrocardiogram data, we observe a nearly 13% improvement in the area under the receiver operating characteristics curve (AUC-ROC) relative to the baseline (AUC-ROC=0.55 vs. AUC-ROC=0.62).
Limited self-supervision applied to sequential data can aid in learning intermediate representations, making it particularly applicable in settings where data collection is difficult.

assistant: text: This paper propose an approach for self-supervised learning on time series.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A unified theory of adaptive stochastic gradient descent as Bayesian filtering

Paper content:
We formulate stochastic gradient descent (SGD) as a novel factorised Bayesian filtering problem, in which each parameter is inferred separately, conditioned on the corresopnding backpropagated gradient.
Inference in this setting naturally gives rise to BRMSprop and BAdam: Bayesian variants of RMSprop and Adam.
Remarkably, the Bayesian approach recovers many features of state-of-the-art adaptive SGD methods, including amongst others root-mean-square normalization, Nesterov acceleration and AdamW.  As such, the Bayesian approach provides one explanation for the empirical effectiveness of state-of-the-art adaptive SGD algorithms.
Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST, we find that Bayesian methods have the potential to considerably reduce test loss and classification error.

assistant: text: The paper analyzes stochastic gradient descent through Bayesian filtering as a framework for analyzing adaptive methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Event Discovery for History Representation in Reinforcement Learning

Paper content:
Environments in Reinforcement Learning (RL) are usually only partially observable.
To address this problem, a possible solution is to provide the agent with information about past  observations.
While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode.
Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision.
We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower.
Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

assistant: text: The authors study the problem of RL under partially observed settings, and propose a solution that uses a FFNN but provides a history representation, outperforming PPO.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discrete flow posteriors for variational inference in discrete dynamical systems

Paper content:
Each training step for a variational autoencoder (VAE) requires us to sample from the approximate posterior, so we usually choose simple (e.g. factorised) approximate posteriors in which sampling is an efficient computation that fully exploits GPU parallelism.
However, such simple approximate posteriors are often insufficient, as they eliminate statistical dependencies in the posterior.
While it is possible to use normalizing flow approximate posteriors for continuous latents, there is nothing analogous for discrete latents.
The most natural approach to model discrete dependencies is an autoregressive distribution, but sampling from such distributions is inherently sequential and thus slow.
We develop a fast, parallel sampling procedure for autoregressive distributions based on fixed-point iterations which enables efficient and accurate variational inference in discrete state-space models.
To optimize the variational bound, we considered two ways to evaluate probabilities: inserting the relaxed samples directly into the pmf for the discrete distribution, or converting to continuous logistic latent variables and interpreting the K-step fixed-point iterations as a normalizing flow.
We found that converting to continuous latent variables gave considerable additional scope for mismatch between the true and approximate posteriors, which resulted in biased inferences, we thus used the former approach.
We tested our approach on the neuroscience problem of inferring discrete spiking activity from noisy calcium-imaging data, and found that it gave accurate connectivity estimates in an order of magnitude less time.

assistant: text: This paper uses an autoregressive filtering variational approximation for parameter estimation in discrete dynamical systems by using fixed point iterations.