ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing

Paper content:
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.
Recent works have developed accurate MI estimators through provably low-bias approximations and tight variational lower bounds assuming abundant supply of samples, but require an unrealistic number of samples to guarantee statistical significance of the estimation.
In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE) that can provide a tight lower confident interval of MI under limited data, through adding cross-validation to the MINE lower bound (Belghazi et al., 2018).
Hyperparameter search is employed and a novel meta-learning approach with task augmentation is developed to increase robustness to hyperparamters, reduce overfitting and improve accuracy.
With improved data-efficiency, our DEMINE estimator enables statistical testing of dependency at practical dataset sizes.
We demonstrate the effectiveness of DEMINE on synthetic benchmarks and a real world fMRI dataset, with application of inter-subject correlation analysis.

assistant: text: Proposes a neural-network-based estimation of mutal information which can reliably work with small datasets, reducing the sample complexity by decoupling the network learning problem and the estimation problem.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Understanding and Training Deep Diagonal Circulant Neural Networks

Paper content:
In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones.
Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization scheme and proposed a smart use of non-linearity functions in order to train deep diagonal circulant networks.
Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers.
We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state of the art models based on structured matrices and with dense models.
We show that our models achieve better accuracy than other structured approaches while required 2x fewer weights as the next best approach.
Finally we train deep diagonal circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

assistant: text: The authors provide a theoretical analysis of the expressive power of diagonal circulant neural networks (DCNN) and propose an initialization scheme for deep DCNNs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DRASIC: Distributed Recurrent Autoencoder for Scalable Image Compression

Paper content:
We propose a new architecture for distributed image compression from a group of distributed data sources.
The work is motivated by practical needs of data-driven codec design, low power consumption, robustness, and data privacy.
The proposed architecture, which we refer to as Distributed Recurrent Autoencoder for Scalable Image Compression (DRASIC), is able to train distributed encoders and one joint decoder on correlated data sources.
Its compression capability is much better than the method of training codecs separately.
Meanwhile, for 10 distributed sources, our distributed system remarkably performs within 2 dB peak signal-to-noise ratio (PSNR) of that of a single codec trained with all data sources.
We experiment distributed sources with different correlations and show how our methodology well matches the Slepian-Wolf Theorem in Distributed Source Coding (DSC).
Our method is also shown to be robust to the lack of presence of encoded data from a number of distributed sources.
Moreover, it is scalable in the sense that codes can be decoded simultaneously at more than one compression quality level.
To the best of our knowledge, this is the first data-driven DSC framework for general distributed code design with deep learning.

assistant: text: The authors propose a method to train image compression models on multiple sources, with a separate encoder on each source, and a shared decoder.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Amharic Light Stemmer

Paper content:
Stemming is the process of removing affixes( i.e. prefixes, infixes and suffixes) that improve the accuracy and performance of information retrieval systems.This paper presents the reduction of Amharic words to corresponding stem where with the intention that it preserves semantic information.
The proposed approach efficiently removes affixes from an Amharic word.
The process of removing such affixes (prefixes, infixes and suffixes) from a word to its base form is called stemming.
While many stemmers exist for dominant languages such as English, under resourced languages such as Amharic which lacks such powerful tool support.
In this paper, we design a light Amharic stemmer relying on the rules that receives an Amharic word and then it finds a match to the beginning of a word to the possible prefixes and to its ending with the possible suffixes and finally it checks whether it has infix.
The final result is the stem if there is any prefix, infix or/and suffix, otherwise it remains in one of the earlier states.
The technique does not rely on any additional resource (e.g. dictionary) to verify the generated stem.
The performance of the generated stemmer is evaluated using manually annotated Amharic words.
The result is compared with current state-of-the-art stemmer for Amharic showing an increase of 7% in stemmer correctness.

assistant: text: Amharic Light Stemmer is designed for improving performance of  Amharic Sentiment Classification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Paper content:
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP.
In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels.
We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.
Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical.
As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case.
We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

assistant: text: Finite-width SGD trained CNNs vs. infinitely wide fully Bayesian CNNs. Who wins?
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A PRIVACY-PRESERVING IMAGE CLASSIFICATION FRAMEWORK WITH A LEARNABLE OBFUSCATOR

Paper content:
Real world images often contain large amounts of private / sensitive information that should be carefully protected without reducing their utilities.
In this paper, we propose a privacy-preserving deep learning framework with a learnable ob- fuscator for the image classification task.
Our framework consists of three mod- els: learnable obfuscator, classifier and reconstructor.
The learnable obfuscator is used to remove the sensitive information in the images and extract the feature maps from them.
The reconstructor plays the role as an attacker, which tries to recover the image from the feature maps extracted by the obfuscator.
In order to best protect users’ privacy in images, we design an adversarial training methodol- ogy for our framework to optimize the obfuscator.
Through extensive evaluations on real world datasets, both the numerical metrics and the visualization results demonstrate that our framework is qualified to protect users’ privacy and achieve a relatively high accuracy on the image classification task.

assistant: text: This paper proposes a framework which preserves the private information in the image and doesn’t compromise the usability of the image.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Value Propagation Networks

Paper content:
We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments.
We show that the modules enable learning to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems.
We evaluate on static and dynamic configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes, and on a StarCraft navigation scenario, with more complex dynamics, and pixels as input.

assistant: text: We present planners based on convnets that are sample-efficient and that generalize to larger instances of navigation and pathfinding problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Paper content:
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).
When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.
While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.
This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.
Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

assistant: text: We introduce a new inductive bias that integrates tree structures in recurrent neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Compact Embedding Layers via Differentiable Product Quantization

Paper content:
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings.
Despite their effectiveness, the number of parameters in an embedding layer increases linearly with the number of symbols and poses a critical challenge on memory and storage constraints.
In this work, we propose a generic and end-to-end learnable compression framework termed differentiable product quantization (DPQ).
We present two instantiations of DPQ that leverage different approximation techniques to enable differentiability in end-to-end learning.
Our method can readily serve as a drop-in alternative for any existing embedding layer.
Empirically, DPQ offers significant compression ratios (14-238x) at negligible or no performance cost on 10 datasets across three different language tasks.

assistant: text: This paper works on methods for compressing embedding layers for low memory inference, where compressed embeddings are learned together with the task-specific models in a differentiable end-to-end fashion.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Credible Sample Elicitation by Deep Learning, for Deep Learning

Paper content:
It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system).
In the literature, there is a line of studies on eliciting distributional information from self-interested agents who hold a relevant information.
Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in practice.
This is primarily due to the heavy cognitive loads required for human agents to reason and report this high dimensional information.
Consider the example where we are interested in building an image classifier via first collecting a certain category of  high-dimensional image data.
While classical elicitation results apply to eliciting a complex and generative (and continuous) distribution $p(x)$ for this image data, we are interested in eliciting samples $x_i \sim p(x)$ from agents.
This paper introduces a deep learning aided method to incentivize credible sample contributions from selfish and rational agents.
The challenge to do so is to design an incentive-compatible score function to score each reported sample to induce truthful reports, instead of an arbitrary or even adversarial one.
We show that with accurate estimation of a certain $f$-divergence function we are able to achieve approximate incentive compatibility in eliciting truthful samples.
We then present an efficient estimator with theoretical guarantee via studying the variational forms of $f$-divergence function.
Our work complements the literature of information elicitation via introducing the problem of \emph{sample elicitation}.  We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples.

assistant: text: The authors propose a sample elicitation framework for the problem of eliciting credible samples from agents for complex distributions, suggest that deep neural frameworks can be applied in this framework, and connect sample elicitation and f-GAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Examining Interpretable Feature Relationships in Deep Networks for Action recognition

Paper content:
A number of recent methods to understand neural networks have focused on quantifying the role of individual features.
One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).
Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.
We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.

assistant: text: We expand Network Dissection to include action interpretation and examine interpretable feature paths to understand the conceptual hierarchy used to classify an action.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Paper content:
Domain adaptation is critical for success in new, unseen environments.
Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difficult to visualize and sometimes fail to capture pixel-level and low-level domain shifts.
Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at  mapping images between domains, even without the use of aligned image pairs.
We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain Adaptation model.
CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs.
Our model can be applied in a variety of visual recognition and prediction settings.
We show new state-of-the-art results across multiple adaptation tasks, including digit classification and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains.

assistant: text: This paper proposes a domain adaptation approach by extending the CycleGAN with task specific loss functions and loss imposed over both pixels and features.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Domain Randomization for Reinforcement Learning

Paper content:
Producing agents that can generalize to a wide range of environments is a significant challenge in reinforcement learning.
One method for overcoming this issue is domain randomization, whereby at the start of each training episode some parameters of the environment are randomized so that the agent is exposed to many possible variations.
However, domain randomization is highly inefficient and may lead to policies with high variance across domains.
In this work, we formalize the domain randomization problem, and show that minimizing the policy's Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies.
We propose a method where the agent only needs to be trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant.
We conduct experiments that demonstrate that our technique leads to more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.

assistant: text: The paper introduces the high variance policies challenge in domain randomization for reinforcement learning and mainly focuses on the problem of visual randomization, where the different randomized domains differ only in state space and the underlying rewards and dynamics are the same.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring in Data

Paper content:
Equivariance is a nice property to have as it produces much more parameter efficient neural architectures and preserves the structure of the input through the feature mapping.
Even though some combinations of transformations might never appear (e.g. an upright face with a horizontal nose), current equivariant architectures consider the set of all possible transformations in a transformation group when learning feature representations.
Contrarily, the human visual system is able to attend to the set of relevant transformations occurring in the environment and utilizes this information to assist and improve object recognition.
Based on this observation, we modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.
We show that our proposed co-attentive equivariant neural networks consistently outperform conventional rotation equivariant and rotation & reflection equivariant neural networks on rotated MNIST and CIFAR-10.

assistant: text: We utilize attention to restrict equivariant neural networks to the set or co-occurring transformations in data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LabelFool: A Trick in the Label Space

Paper content:
It is widely known that well-designed perturbations can cause state-of-the-art machine learning classifiers to mis-label an image, with sufficiently small perturbations that are imperceptible to the human eyes.
However, by detecting the inconsistency between the image and wrong label, the human observer would be alerted of the attack.
In this paper, we aim to design attacks that not only make classifiers generate wrong labels, but also make the wrong labels imperceptible to human observers.
To achieve this, we propose an algorithm called LabelFool which identifies a target label similar to the ground truth label and finds a perturbation of the image for this target label.
We first find the target label for an input image by a probability model, then move the input in the feature space towards the target label.
Subjective studies on ImageNet show that in the label space, our attack is much less recognizable by human observers, while objective experimental results on ImageNet show that we maintain similar performance in the image space as well as attack rates to state-of-the-art attack algorithms.

assistant: text: A method for constructing adversarial attacks that are less detectable by humans without cost in image space by changing the target class to be similar to the original class of the image.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Goal-Conditioned Video Prediction

Paper content:
Many processes can be concisely represented as a sequence of events leading from a starting state to an end state.
Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe.
Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP).
Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.
GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames.
Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction
. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals
. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons
.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions
. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

assistant: text: We propose a new class of visual generative models: goal-conditioned predictors. We show experimentally that conditioning on the goal allows to reduce uncertainty and produce predictions over much longer horizons.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Winning the Lottery with Continuous Sparsification

Paper content:
The Lottery Ticket Hypothesis from Frankle & Carbin (2019) conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts.
The proposed algorithm to search for such sub-networks (winning tickets), Iterative Magnitude Pruning (IMP), consistently finds sub-networks with 90-95% less parameters which indeed train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning.
In this paper, we propose a new algorithm to search for winning tickets, Continuous Sparsification, which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies.
We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing up to 5 times faster search, when measured in number of training epochs.

assistant: text: We propose a new algorithm that quickly finds winning tickets in neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Augmenting Self-attention with Persistent Memory

Paper content:
Transformer networks have lead to important progress in language modeling and machine translation.
These models include two consecutive modules, a feed-forward layer and a self-attention layer.
The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.
Building upon this intuition, we propose a new model that solely consists of attention layers.
More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.
Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.
Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

assistant: text: This paper proposes a modification to the Transformer model by incorporating attention over "persistent" memory vectors into the self-attention layer, resulting in performance on par with existing models while using fewer parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GUIDEGAN:  ATTENTION  BASED  SPATIAL  GUIDANCE FOR  IMAGE-TO-IMAGE TRANSLATION

Paper content:
Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner.
However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice.
Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients.
To tackle this problem, we propose a GuideGAN based on attention mechanism.
More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction.
This attention map then assists the generator to produce more plausible and realistic images.
We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks.
Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

assistant: text: A feedback mechanism in the GAN framework which improves the quality of generated images in image-to-image translation, and whose discriminator outputs a map indicating where the generator should focus to make its results more convincing.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Alexandria: Unsupervised High-Precision Knowledge Base Construction using a Probabilistic Program

Paper content:
Creating a knowledge base that is accurate, up-to-date and complete remains a significant challenge despite substantial efforts in automated knowledge base construction.
In this paper, we present Alexandria -- a system for unsupervised, high-precision knowledge base construction.
Alexandria uses a probabilistic program to define a process of converting knowledge base facts into unstructured text.
Using probabilistic inference, we can invert this program and so retrieve facts, schemas and entities from web text.
The use of a probabilistic program allows uncertainty in the text to be propagated through to the retrieved facts, which increases accuracy and helps merge facts from multiple sources.
Because Alexandria does not require labelled training data, knowledge bases can be constructed with the minimum of manual input.
We demonstrate this by constructing a high precision (typically 97\%+) knowledge base for people from a single seed fact.

assistant: text: This paper presents a system for unsupervised, high-precision knowledge base construction using a probabilistic program to define a process of converting knowledge base facts into unstructured text.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions

Paper content:
The driving force behind deep networks is their ability to compactly represent rich classes of functions.
The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to replicate functions of another.
To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones.
In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways.
We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks.
By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency.
In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not.
Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy.
This leads us to believe that expressive efficiency may serve a key role in developing new tools for deep network design.

assistant: text: This paper theoretically validates that interconnecting networks with different dilations can lead to expressive efficiency using mixed tensor decomposition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Mining: Detecting Anomalous Patterns in Neural Network Activations with Subset Scanning

Paper content:
This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs.
Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs.
More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples.
This work introduces ``Subset Scanning methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks.
Subset Scanning allows us to answer the question: "``Which subset of inputs have larger-than-expected activations at which subset of nodes?
"  Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images.
Such images are ``"weird together".
Leveraging this common anomalous pattern, we show increased detection power as the proportion of noised images increases in a test set.
Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.

assistant: text: We efficiently find a subset of images that have higher than expected activations for some subset of nodes.  These images appear more anomalous and easier to detect when viewed as a group.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Paper content:
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019).
However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked.
In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work.
Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation.
The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements.
Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data.
One surprising observation is that they have a compound effect even when sequentially applied on the same data.
To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

assistant: text: This submission shows that pre-training a student directly on masked language modeling is better than distillation, and the best is to combine both and distill from that pre-trained student model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Augmentation via Data Subsampling

Paper content:
Data augmentation is commonly used to encode invariances in learning methods.
However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set.
The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply.
In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset.
We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

assistant: text: The authors study the problem of identifying subsampling strategies for data augmentation and propose strategies based on model influence and loss as well as empirical benchmarking of the proposed methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SuperCaptioning: Image Captioning Using Two-dimensional Word Embedding

Paper content:
Language and vision are processed as two different modal in current work for image captioning.
However, recent work on Super Characters method shows the effectiveness of two-dimensional word embedding, which converts text classification problem into image classification problem.
In this paper, we propose the SuperCaptioning method, which borrows the idea of two-dimensional word embedding from Super Characters method, and processes the information of language and vision together in one single CNN model.
The experimental results on Flickr30k data shows the proposed method gives high quality image captions.
An interactive demo is ready to show at the workshop.
Image captioning outputs a sentence related to the input image.
Current methods process the image and text separately BID3 BID8 BID10 BID9 BID4 BID5 BID0 BID1 .
Generally, the image is processed by a CNN model to extract the image feature, and the raw text passes through embedding layer to convert into one-dimensional wordembedding vectors, e.g. a 300x1 dimensional vector.
And then the extracted image feature and the word embedding vectors will be fed into another network, such as RNN, LSTM, or GRU model, to predict the next word in the image caption sequentially.Super Characters method ) is originally designed for text classification tasks.
It has achieved stateof-the-art results on benchmark datasets for multiple languages, including English, Chinese, Japanese, and Korean.
It is a two-step method.
In the first step, the text characters are printed on a blank image, and the generated image is called Super Characters image.
In the second step, the Super Characters image is fed into a CNN model for classification.
The CNN model is fine-tuned from pre-trained ImageNet model.In this paper, we address the image captioning problem by employing the two-dimensional word embedding from the Super Characters method, and the resulting method is named as SuperCaptioning method.
In this method, the input image and the raw text are combined together through two-dimensional embedding, and then fed into a CNN model to sequentially predict the words in the image caption.
The experimental results on Flickr30k shows that the proposed method gives high quality image captions.
Some examples given by SuperCaptioning method are shown in FIG2 .

assistant: text: Image captioning using two-dimensional word embedding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Convolutional neural networks with extra-classical receptive fields

Paper content:
In the visual system, neurons respond to a patch of the input known as their classical receptive field (RF), and can be modulated by stimuli in the surround.
These interactions are often mediated by lateral connections, giving rise to extra-classical RFs.
We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network.
These connections allow each unit to integrate information from its surround, generating extra-classical receptive fields for the units in our new proposed model (CNNEx).
We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets.
Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets.
Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable.

assistant: text: CNNs with biologically-inspired lateral connections learned in an unsupervised manner are more robust to noisy inputs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: What Can Neural Networks Reason About?

Paper content:
Neural networks have succeeded in many reasoning tasks.
Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, while less structured networks fail.
Theoretically, there is limited understanding of why and when a network structure generalizes better than other equally expressive ones.
We develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its structure aligns with the algorithmic structure of the relevant reasoning procedure.
We formally define algorithmic alignment and derive a sample complexity bound that decreases with better alignment.
This framework explains the empirical success of popular reasoning models and suggests their limitations.
We unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP).
We show that GNNs can learn DP and thus solve these tasks.
On several reasoning tasks, our theory aligns with empirical results.

assistant: text: The paper proposes a measure of classes of algorithmic alignment that measure how "close" neural networks are to known algorithms, proving the link between several classes of known algorithms and neural network architectures.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Winning the Lottery with Continuous Sparsification

Paper content:
The Lottery Ticket Hypothesis from Frankle & Carbin (2019) conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts.
The proposed algorithm to search for such sub-networks (winning tickets), Iterative Magnitude Pruning (IMP), consistently finds sub-networks with 90-95% less parameters which indeed train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning.
In this paper, we propose a new algorithm to search for winning tickets, Continuous Sparsification, which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies.
We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing up to 5 times faster search, when measured in number of training epochs.

assistant: text: This paper proposes a novel objective function that can be used to jointly optimize a classification objective while encouraging sparsification in a network that performs with high accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Tunability of Optimizers in Deep Learning

Paper content:
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD.
In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search?
We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.
Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

assistant: text: We provide a method to benchmark optimizers that is cognizant to the hyperparameter tuning process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Document Enhancement System Using Auto-encoders

Paper content:
The conversion of scanned documents to digital forms is performed using an Optical Character Recognition (OCR) software.
This work focuses on improving the quality of scanned documents in order to improve the OCR output.
We create an end-to-end document enhancement pipeline which takes in a set of noisy documents and produces clean ones.
Deep neural network based denoising auto-encoders are trained to improve the OCR quality.
We train a blind model that works on different noise levels of scanned text documents.
Results are shown for blurring and watermark noise removal from noisy scanned documents.

assistant: text: We designed and tested a REDNET (ResNet Encoder-Decoder) with 8 skip connections to remove noise from documents, including blurring and watermarks, resulting in a high performance deep network for document image cleanup.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Continual Learning and Self-Taught Associative Memory Hierarchies

Paper content:
We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time.
Given limited labeled data just before inference, those representations can also be associated with specific object types to perform classification.
To solve the UCL problem, we propose an architecture that involves a single module, called Self-Taught Associative Memory (STAM), which loosely models the function of a cortical column in the mammalian brain.
Hierarchies of STAM modules learn based on a combination of Hebbian learning, online clustering, detection of novel patterns and forgetting outliers, and top-down predictions.
We illustrate the operation of STAMs in the context of learning handwritten digits in a continual manner with only 3-12 labeled examples per class.
STAMs suggest a promising direction to solve the UCL problem without catastrophic forgetting.

assistant: text: Proposes using hierachies of STAM modules to solve the UCL problem, providing evidence that the representations the modules learn are well-suited for few-shot classification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Step Size Optimization

Paper content:
This paper proposes a new approach for step size adaptation in gradient methods.
The proposed method called step size optimization (SSO) formulates the step size adaptation as an optimization problem which minimizes the loss function with respect to the step size for the given model parameters and gradients.
Then, the step size is optimized based on alternating direction method of multipliers (ADMM).
SSO does not require the second-order information or any probabilistic models for adapting the step size, so it is efficient and easy to implement.
Furthermore, we also introduce stochastic SSO for stochastic learning environments.
In the experiments, we integrated SSO to vanilla SGD and Adam, and they outperformed state-of-the-art adaptive gradient methods including RMSProp, Adam, L4-Adam, and AdaBound on extensive benchmark datasets.

assistant: text: A new step size adaptation in first-order gradient methods that establishes a new optimization problem with the first-order expansion of the loss function and regularization, where step size is treated as variable.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SpaMHMM: Sparse Mixture of Hidden Markov Models for Graph Connected Entities

Paper content:
We propose a framework to model the distribution of sequential data coming from
a set of entities connected in a graph with a known topology.
The method is
based on a mixture of shared hidden Markov models (HMMs), which are trained
in order to exploit the knowledge of the graph structure and in such a way that the
obtained mixtures tend to be sparse.
Experiments in different application domains
demonstrate the effectiveness and versatility of the method.

assistant: text: A method to model the generative distribution of sequences coming from graph connected entities.