ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations

Paper content:
Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks.
However, understanding of their internal functioning is still insufficient and unsatisfactory.
In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states.
Unlike previous research, which mainly focuses on explaining Transformer models by their \hbox{attention} weights, we argue that hidden states contain equally valuable information.
Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task.
We inspect how QA models transform token vectors in order to find the correct answer.
To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer.
Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process.
Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks.
The system can therefore implicitly incorporate task-specific information into its token representations.
Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers.

assistant: text: We investigate hidden state activations of Transformer Models in Question Answering Tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: INFERENCE, PREDICTION, AND ENTROPY RATE OF CONTINUOUS-TIME, DISCRETE-EVENT PROCESSES

Paper content:
The inference of models, prediction of future symbols, and entropy rate estimation of discrete-time, discrete-event processes is well-worn ground.
However, many time series are better conceptualized as continuous-time, discrete-event processes.
Here, we provide new methods for inferring models, predicting future symbols, and estimating the entropy rate of continuous-time, discrete-event processes.
The methods rely on an extension of Bayesian structural inference that takes advantage of neural network’s universal approximation power.
Based on experiments with simple synthetic data, these new methods seem to be competitive with state-of- the-art methods for prediction and entropy rate estimation as long as the correct model is inferred.

assistant: text: A new method for inferring a model of, estimating the entropy rate of, and predicting continuous-time, discrete-event processes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CORD: A Consolidated Receipt Dataset for Post-OCR Parsing

Paper content:
OCR is inevitably linked to NLP since its final output is in text.
Advances in document intelligence are driving the need for a unified technology that integrates OCR with various NLP tasks, especially semantic parsing.
Since OCR and semantic parsing have been studied as separate tasks so far, the datasets for each task on their own are rich, while those for the integrated post-OCR parsing tasks are relatively insufficient.
In this study, we publish a consolidated dataset for receipt parsing as the first step towards post-OCR parsing tasks.
The dataset consists of thousands of Indonesian receipts, which contains images and box/text annotations for OCR, and multi-level semantic labels for parsing.
The proposed dataset can be used to address various OCR and parsing tasks.

assistant: text: We introduce a large-scale receipt dataset for post-OCR parsing tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Paper content:
Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice.
When the train and test distributions are mismatched, accuracy can plummet.
Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment.
In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers.
We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions.
AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.

assistant: text: We obtain state-of-the-art on robustness to data shifts, and we maintain calibration under data shift even though even when accuracy drops
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergence of grid-like representations by training recurrent neural networks to perform spatial localization

Paper content:
Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties.
The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns.
However, the mechanisms and functional significance of these spatial representations remain largely mysterious.
As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs.
Surprisingly, we find that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells.
All these different functional types of neurons have been observed experimentally.
The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies.
Together, our results suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efficiently given the predominant recurrent connections in the neural circuits.

assistant: text: To our knowledge, this is the first study to show how neural representations of space, including grid-like cells and border cells as observed in the brain, could emerge from training a recurrent neural network to perform navigation tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Needles in Haystacks: On Classifying Tiny Objects in Large Images

Paper content:
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images.
However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions.
To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images.
This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios.
Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.

assistant: text: We study low- and very-low-signal-to-noise classification scenarios, where objects that correlate with class label occupy tiny proportion of the entire image (e.g. medical or hyperspectral imaging).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Autoregressive Convolutional Neural Networks for Asynchronous Time Series

Paper content:
We propose Significance-Offset Convolutional Neural Network, a deep convolutional network architecture for regression of multivariate asynchronous time series.
The model is inspired by standard autoregressive (AR) models and gating mechanisms used in recurrent neural networks.
It involves an AR-like weighting system, where the final predictor is obtained as a weighted sum of adjusted regressors, while the weights are data-dependent functions learnt through a convolutional network.
The architecture was designed for applications on asynchronous time series and is evaluated on such datasets: a hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, an artificially generated noisy autoregressive  series  and  household  electricity  consumption  dataset.
The  pro-posed architecture achieves promising results as compared to convolutional and recurrent neural networks.
The code for the numerical experiments and the architecture implementation will be shared online to make the research reproducible.

assistant: text: Convolutional architecture for learning data-dependent weights for autoregressive forecasting of time series.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Task-agnostic Continual Learning via Growing Long-Term Memory Networks

Paper content:
As our experience shows, humans can learn and deploy a myriad of different skills to tackle the situations they encounter daily.
Neural networks, in contrast, have a fixed memory capacity that prevents them from learning more than a few sets of skills before starting to forget them.
In this work, we make a step to bridge neural networks with human-like learning capabilities.
For this, we propose a model with a growing and open-bounded memory capacity that can be accessed based on the model’s current demands.
To test this system, we introduce a continual learning task based on language modelling where the model is exposed to multiple languages and domains in sequence, without providing any explicit signal on the type of input it is currently dealing with.
The proposed system exhibits improved adaptation skills in that it can recover faster than comparable baselines after a switch in the input language or domain.

assistant: text: We introduce a continual learning setup based on language modelling where no explicit task segmentation signal is given and propose a neural network model with growing long term memory to tackle it.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CrossLang: the system of cross-lingual plagiarism detection

Paper content:
Plagiarism and text reuse become more available with the Internet development.
Therefore it is important to check scientific papers for the fact of cheating, especially in Academia.
Existing systems of plagiarism detection show the good performance and have a huge source databases.
Thus now it is not enough just to copy the text as is from the source document to get the original work.
Therefore, another type of plagiarism become popular -- cross-lingual plagiarism.
We present a CrossLang system for such kind of plagiarism detection for English-Russian language pair.
We introduced CrossLang -a framework for cross-lingual plagiarism detection for English Russian language pair.
We decomposed the problem of cross-lingual plagiarism detection into several stages and provide a service, consists of a set of microservices.
The CrossLang use a monolingual approachreducing the problem to the one language.
For this purpose we trained the neural machine translation system.
Another two main algoithmic components are Source Retrieval and Document Comparison stages.
For the Source Retrieval problem we used a modification of shingling method that allow us to deal with ambiguity after translation.
For the Document Comparison stage we used phrase embeddings that were trained with slight supervision.
We evaluated the effectiveness of main stages.

assistant: text: A system for cross-lingual (English-Russian) plagiarism detection
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning

Paper content:
Conservation laws are considered to be fundamental laws of nature.
It has broad application in many fields including physics, chemistry, biology, geology, and engineering.
Solving the differential equations associated with conservation laws is a major branch in computational mathematics.
Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods.
In this paper, we are the first to explore the possibility and benefit of solving nonlinear conservation laws using deep reinforcement learning.
As a proof of concept, we focus on 1-dimensional scalar conservation laws.
We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner.
We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy.
Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach.
In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts.
Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes.
Our code is released anomynously at \url{https://github.com/qwerlanksdf/L2D}.

assistant: text: We observe that numerical PDE solvers can be regarded as Markov Desicion Processes, and propose to use Reinforcement Learning to solve 1D scalar Conservation Laws
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering

Paper content:
We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.
In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data.
When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model.
We call our model the latent tree variational autoencoder (LTVAE).
Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable.
This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways.

assistant: text: We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variationally Inferred Sampling Through a Refined Bound

Paper content:
A framework for efficient Bayesian inference in probabilistic programs is introduced by embedding a sampler inside a variational posterior approximation.
Its strength lies in both ease of implementation and automatically tuning sampler parameters to speed up mixing time.
Several strategies to approximate the evidence lower bound (ELBO) computation are introduced, including a rewriting of the ELBO objective.
Experimental evidence is shown by performing experiments on an unconditional VAE on density estimation tasks; solving an influence diagram in a high-dimensional space with a conditional variational autoencoder (cVAE) as a deep Bayes classifier; and state-space models for time-series data.

assistant: text: We embed SG-MCMC samplers inside a variational approximation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improved Self-Supervised Deep Image Denoising

Paper content:
We describe techniques for training high-quality image denoising models that require only single instances of corrupted images as training data.
Inspired by a recent technique that removes the need for supervision through image pairs by employing networks with a "blind spot" in the receptive field, we address two of its  shortcomings: inefficient training and poor final denoising performance.
This is achieved through a novel blind-spot convolutional network architecture that allows efficient self-supervised training, as well as application of Bayesian distribution prediction on output colors.
Together, they bring the self-supervised model on par with fully supervised deep learning techniques in terms of both quality and training speed in the case of i.i.d. Gaussian noise.

assistant: text: We learn high-quality denoising using only single instances of corrupted images as training data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Error Correcting Algorithms for Sparsely Correlated Regressors

Paper content:
Autonomy and adaptation of machines requires that they be able to measure their own errors.
We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task.
How can a machine measure the error of regression sub-components when it does not have the ground truth for the correct predictions?
A
compressed sensing approach applied to the error signal of the regressors can recover their precision error without any ground truth.
It allows for some regressors to be strongly correlated as long as not too many are so related.
Its solutions, however, are not unique - a property of ground truth inference solutions.
Adding  $\ell_1$--minimization
as a condition can recover the correct solution in settings where error correction is possible.
We briefly discuss the similarity of the mathematics of ground truth inference for regressors to that for classifiers.

assistant: text: A non-parametric method to measure the error moments of regressors without ground truth can be used with biased regressors
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized NN

Paper content:
Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity.
In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation.
Our answer is "early stopping".
Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping.
This can be justified by a new concept, Anisotropic In- formation Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later.
Motivated by the recent development on theoretically analyzing overparame- terized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK).
AIR facilities a new understanding of distillation.
With that, we further utilize distillation to refine noisy labels.
We propose a self-distillation al- gorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels.
We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping.
Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous result was on convergence in 0-1 loss.
The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization.
Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.

assistant: text: theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Controlling generative models with continuous factors of variations

Paper content:
Recent deep generative models can provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing.
Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation.
To overcome these major issues, very recent works have shown the interest of studying the semantics of the latent space of generative models.
In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like position or scale of the object in the image.
Our method is weakly supervised and particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations.
We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

assistant: text: A model to control the generation of images with GAN and beta-VAE with regard to scale and position of the objects
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Recovering the Lowest Layer of Deep Networks with High Threshold Activations

Paper content:
Giving provable guarantees for learning neural networks is a core challenge of machine learning theory.
Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers.
In this work, we show how we can strengthen such results to deeper networks -- we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is gaussian.

assistant: text: We provably recover the lowest layer in a deep neural network assuming that the lowest layer uses a "high threshold" activation and the above network is a "well-behaved" polynomial.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks

Paper content:
We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks.
With the proposed method, the label embedding is adaptively and automatically learned through back propagation.
The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process.
As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed.
Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable.
The proposed method achieves comparable or even better results than the state-of-the-art systems.

assistant: text: Learning Label Representation for Deep Networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-Task Learning for Semantic Parsing with Cross-Domain Sketch

Paper content:
Semantic parsing which maps a natural language sentence into a formal machine-readable representation of its meaning, is highly constrained by the limited annotated training data.
Inspired by the idea of coarse-to-fine, we propose a general-to-detailed neural network(GDNN) by incorporating cross-domain sketch(CDS) among utterances and their logic forms.
For utterances in different domains, the General Network will extract CDS using an encoder-decoder model in a multi-task learning setup.
Then for some utterances in a specific domain, the Detailed Network will generate the detailed target parts using sequence-to-sequence architecture with advanced attention to both utterance and generated CDS.
Our experiments show that compared to direct multi-task learning, CDS has improved the performance in semantic parsing task which converts users' requests into meaning representation language(MRL).
We also use experiments to illustrate that CDS works by adding some constraints to the target decoding process, which further proves the effectiveness and rationality of CDS.

assistant: text: General-to-detailed neural network(GDNN)  with Multi-Task Learning by incorporating cross-domain sketch(CDS) for semantic parsing
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Selection of Initialization and Activation Function for Deep Neural Networks

Paper content:
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure.
An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation.
Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `edge of chaos' can lead to good performance.
We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos.
By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions.
This class includes the Swish activation, $\phi_{swish}(x) = x \cdot \text{sigmoid}(x)$, used in Hendrycks & Gimpel (2016),
Elfwing et al. (2017) and Ramachandran et al. (2017).
This provides a theoretical grounding for the excellent empirical performance of $\phi_{swish}$ observed in these contributions.
We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.

assistant: text: How to effectively choose Initialization and Activation function for deep neural networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: D2KE: From Distance to Kernel and Embedding via Random Features For Structured Inputs

Paper content:
We present a new methodology that constructs a family of \emph{positive definite kernels} from any given dissimilarity measure on structured inputs whose elements are either real-valued time series or discrete structures such as strings, histograms, and graphs.
Our approach, which we call D2KE (from Distance to Kernel and Embedding), draws from the literature of Random Features.
However, instead of deriving random feature maps from a user-defined kernel to approximate kernel machines, we build a kernel from a random feature map, that we specify given the distance measure.
We further propose use of a finite number of random objects to produce a random feature embedding of each instance.
We provide a theoretical analysis showing that D2KE enjoys better generalizability than universal Nearest-Neighbor estimates.
On one hand, D2KE subsumes the widely-used \emph{representative-set method} as a special case, and relates to the well-known \emph{distance substitution kernel} in a limiting case.
On the other hand, D2KE generalizes existing \emph{Random Features methods} applicable only to vector input representations to complex structured inputs of variable sizes.
We conduct classification experiments over such disparate domains as time series, strings, and histograms (for texts and images), for which our proposed framework compares favorably to existing distance-based learning methods in terms of both testing accuracy and computational time.

assistant: text: From Distance to Kernel and Embedding via Random Features For Structured Inputs
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Regularity Normalization: Constraining Implicit Space with Minimum Description Length

Paper content:
Inspired by the adaptation phenomenon of biological neuronal firing, we propose regularity normalization: a reparameterization of the activation in the neural network that take into account the statistical regularity in the implicit space.
By considering the neural network optimization process as a model selection problem, the implicit space is constrained by the normalizing factor, the minimum description length of the optimal universal code.
We introduce an incremental version of computing this universal code as normalized maximum likelihood and demonstrated its flexibility to include data prior such as top-down attention and other oracle information and its compatibility to be incorporated into batch normalization and layer normalization.
The preliminary results showed that the proposed method outperforms existing normalization methods in tackling the limited and imbalanced data from a non-stationary distribution benchmarked on computer vision task.
As an unsupervised attention mechanism given input data, this biologically plausible normalization has the potential to deal with other complicated real-world scenarios as well as reinforcement learning setting where the rewards are sparse and non-uniform.
Further research is proposed to discover these scenarios and explore the behaviors among different variants.

assistant: text: Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intriguing Properties of Adversarial Training at Scale

Paper content:
Adversarial training is one of the main defenses against adversarial attacks.
In this paper, we provide the first rigorous study on diagnosing elements of large-scale adversarial training on ImageNet, which reveals two intriguing properties.
First, we study the role of normalization.
Batch normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training.
One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%.
We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains.
This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging.
Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness.
Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness.
Second, we study the role of network capacity.
We find our so-called "deep" networks are still shallow for the task of adversarial learning.
Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness.
This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.

assistant: text: The first rigor diagnose of large-scale adversarial training on ImageNet
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Paper content:
Neural Tangents is a library for working with infinite-width neural networks.
It provides a high-level API for specifying complex and hierarchical neural network architectures.
These networks can then be trained and evaluated either at finite-width as usual, or in their infinite-width limit.
For the infinite-width networks, Neural Tangents performs exact inference either via Bayes' rule or gradient descent, and generates the corresponding Neural Network Gaussian Process and Neural Tangent kernels.
Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks.
The entire library runs out-of-the-box on CPU, GPU, or TPU.
All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices.
In addition to the repository below, we provide an accompanying interactive Colab notebook at
https://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb

assistant: text: Keras for infinite neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: When Agents Talk Back: Rebellious Explanations

Paper content:
As the area of Explainable AI (XAI), and Explainable AI Planning (XAIP), matures, the ability for agents to generate and curate explanations will likewise grow.
We propose a new challenge area in the form of rebellious and deceptive explanations.
We discuss how these explanations might be generated and then briefly discuss evaluation criteria.

assistant: text: Position paper proposing rebellious and deceptive explanations for agents.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hierarchical Image-to-image Translation with Nested Distributions Modeling

Paper content:
Unpaired image-to-image translation among category domains has achieved remarkable success in past decades.
Recent studies mainly focus on two challenges.
For one thing, such translation is inherently multimodal due to variations of domain-specific information (e.g., the domain of house cat has multiple fine-grained subcategories).
For another, existing multimodal approaches have limitations in handling more than two domains, i.e. they have to independently build one model for every pair of domains.
To address these problems, we propose the Hierarchical Image-to-image Translation (HIT) method which jointly formulates the multimodal and multi-domain problem in a semantic hierarchy structure, and can further control the uncertainty of multimodal.
Specifically, we regard the domain-specific variations as the result of the multi-granularity property of domains, and one can control the granularity of the multimodal translation by dividing a domain with large variations into multiple subdomains which capture local and fine-grained variations.
With the assumption of Gaussian prior, variations of domains are modeled in a common space such that translations can further be done among multiple domains within one model.
To learn such complicated space, we propose to leverage the inclusion relation among domains to constrain distributions of parent and children to be nested.
Experiments on several datasets validate the promising results and competitive performance against state-of-the-arts.

assistant: text: Granularity controled multi-domain and multimodal image to image translation method
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: (How) Can AI Bots Lie?

Paper content:
Recent work on explanation generation for decision-making problems has viewed the explanation process as one of model reconciliation where an AI agent brings the human mental model (of its capabilities, beliefs, and goals) to the same page with regards to a task at hand.
This formulation succinctly captures many possible types of explanations, as well as explicitly addresses the various properties -- e.g. the social aspects, contrastiveness, and selectiveness -- of explanations studied in social sciences among human-human interactions.
However, it turns out that the same process can be hijacked into producing "alternative explanations" -- i.e. explanations that are not true but still satisfy all the properties of a proper explanation.
In previous work, we have looked at how such explanations may be perceived by the human in the loop and alluded to one possible way of generating them.
In this paper, we go into more details of this curious feature of the model reconciliation process and discuss similar implications to the overall notion of explainable decision-making.
So far we have only considered explicit cases of deception.
Interestingly, existing approaches in model reconciliation already tend to allow for misconceptions to be ignored if not actively induced by the agent.

assistant: text: Model Reconciliation is an established framework for plan explanations, but can be easily hijacked to produce lies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Memorization in Overparameterized Autoencoders

Paper content:
Interpolation of data in deep neural networks has become a subject of significant research interest.
We prove that over-parameterized single layer fully connected autoencoders do not merely interpolate, but rather, memorize training data: they produce outputs in (a non-linear version of) the span of the training examples.
In contrast to fully connected autoencoders, we prove that depth is necessary for memorization in convolutional autoencoders.
Moreover, we observe that adding nonlinearity to deep convolutional autoencoders results in a stronger form of memorization: instead of outputting points in the span of the training images, deep convolutional autoencoders tend to output individual training images.
Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important question of the inductive bias in over-parameterized deep networks.

assistant: text: We identify memorization as the inductive bias of interpolation in overparameterized fully connected and convolutional auto-encoders.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius

Paper content:
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
In this paper, we propose the MACER algorithm, which learns robust models without using adversarial training but performs better than all existing provable l2-defenses.
Recent work shows that randomized smoothing can be used to provide certified l2 radius to smoothed classifiers, and our algorithm trains provably robust smoothed classifiers via MAximizing the CErtified Radius (MACER).
The attack-free characteristic makes MACER faster to train and easier to optimize.
In our experiments, we show that our method can be applied to modern deep neural networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and SVHN.
For all tasks, MACER spends less training time than state-of-the-art adversarial training algorithms, and the learned models achieve larger average certified radius.

assistant: text: We propose MACER: a provable defense algorithm that trains robust models by maximizing the certified radius. It does not use adversarial training but performs better than all existing provable l2-defenses.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Interactive Visual Exploration of Latent Space (IVELS) for peptide auto-encoder model selection

Paper content:
We present a tool for Interactive Visual Exploration of Latent Space (IVELS) for model selection.
Evaluating generative models of discrete sequences from a  continuous  latent  space  is  a  challenging  problem,  since  their  optimization involves multiple competing objective terms.
We introduce a model-selection pipeline  to  compare  and  filter  models  throughout  consecutive  stages  of  more complex and expensive metrics.
We present the pipeline in an interactive visual tool to enable the exploration of the metrics, analysis of the learned latent space, and selection of the best model for a given task.
We focus specifically on the variational auto-encoder family in a case study of modeling peptide sequences, which are short sequences of amino acids.
This task is especially interesting due to the presence of multiple attributes we want to model.
We demonstrate how an interactive visual comparison can assist in evaluating how well an unsupervised auto-encoder meaningfully captures the attributes of interest in its latent space.

assistant: text: We present a visual tool to interactively explore the latent space of an auto-encoder for peptide sequences and their attributes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving label efficiency through multi-task learning on auditory data

Paper content:
Collecting high-quality, large scale datasets typically requires significant resources.
The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through multitask learning with self-supervised tasks on unlabeled data.
To this end, we trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks.
We describe three self-supervised learning tasks that can operate on any large, unlabeled audio corpus.
We demonstrate that, in a scenario with limited labeled training data, one can significantly improve the performance of a supervised classification task by simultaneously training it with these additional self-supervised tasks.
We show that one can improve performance on a diverse sound events classification task by nearly 6\% when jointly trained with up to three distinct self-supervised tasks.
This improvement scales with the number of additional auxiliary tasks as well as the amount of unsupervised data.
We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance.

assistant: text: Improving label efficiency through multi-task learning on auditory data
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

Paper content:
In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components.
Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw.
Our game is grounded in a virtual world that contains movable clip art objects.
The game involves two players: a Teller and a Drawer.
The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces.
The two players communicate via two-way communication using natural language.
We collect the CoDraw dataset of ~10K dialogs consisting of ~138K messages exchanged between human agents.
We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel "crosstalk" condition which pairs agents trained independently on disjoint subsets of the training data for evaluation.
We present models for our task, including simple but effective baselines and neural network approaches trained using a combination of imitation learning and goal-driven training.
All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.

assistant: text: We introduce a dataset, models, and training + evaluation protocols for a collaborative drawing task that allows studying goal-driven and perceptually + actionably grounded language generation and understanding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Conditional Out-of-Sample Generation For Unpaired Data using trVAE

Paper content:
While generative models have shown great success in generating high-dimensional samples conditional on low-dimensional descriptors (learning e.g. stroke thickness in MNIST, hair color in CelebA, or speaker identity in Wavenet), their generation out-of-sample poses fundamental problems.
The conditional variational autoencoder (CVAE) as a simple conditional generative model does not explicitly relate conditions during training and, hence, has no incentive of learning a compact joint distribution across conditions.
We overcome this limitation by matching their distributions using maximum mean discrepancy (MMD) in the decoder layer that follows the bottleneck.
This introduces a strong regularization both for reconstructing samples within the same condition and for transforming samples across conditions, resulting in much improved generalization.
We refer to the architecture as transformer VAE (trVAE).
Benchmarking trVAE on high-dimensional image and tabular data, we demonstrate higher robustness and higher accuracy than existing approaches.
In particular, we show qualitatively improved predictions for cellular perturbation response to treatment and disease based on high-dimensional single-cell gene expression data, by tackling previously problematic minority classes and multiple conditions.
For generic tasks, we improve Pearson correlations of high-dimensional estimated means and variances with their ground truths from 0.89 to 0.97 and 0.75 to 0.87, respectively.

assistant: text: Generates never seen data during training from a desired condition