ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Non-Parametric Invariances from Data with Permanent Random Connectomes 

Paper content:
One of the fundamental problems in supervised classification and in machine learning in general, is the modelling of non-parametric invariances that exist in data.
Most prior art has focused on enforcing priors in the form of invariances to parametric nuisance transformations that are expected to be present in data.
However, learning non-parametric invariances directly from data remains an important open problem.
In this paper, we introduce a new architectural layer for convolutional networks which is capable of learning general invariances from data itself.
This layer can learn invariance to non-parametric transformations and interestingly, motivates and incorporates permanent random connectomes there by being called Permanent Random Connectome Non-Parametric Transformation Networks (PRC-NPTN).
PRC-NPTN networks are initialized with random connections (not just weights) which are a small subset of the connections in a fully connected convolution layer.
Importantly, these connections in PRC-NPTNs once initialized remain permanent throughout training and testing.
Random connectomes makes these architectures loosely more biologically plausible than many other mainstream network architectures which require highly ordered structures.
We motivate randomly initialized connections as a simple method to learn invariance from data itself while invoking invariance towards multiple nuisance transformations simultaneously.
We find that these randomly initialized permanent connections have positive effects on generalization, outperform much larger ConvNet baselines and the recently proposed Non-Parametric Transformation Network (NPTN) on benchmarks that enforce learning invariances from the data itself.
invariances directly from the data, with the only prior being the structure that allows them to do so.
with an enhanced ability to learn non-parametric invariances through permanent random connectivity.
we do not explore these biological connections in more detail, it is still an interesting observation.

assistant: text: A layer modelling local random connectomes in the cortex within deep networks capable of learning general non-parametric invariances from the data itself.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unaligned Image-to-Sequence Transformation with Loop Consistency

Paper content:
We tackle the problem of modeling sequential visual phenomena.
Given examples of a phenomena that can be divided into discrete time steps, we aim to take an input from any such time and realize this input at all other time steps in the sequence.
Furthermore, we aim to do this \textit{without} ground-truth aligned sequences --- avoiding the difficulties needed for gathering aligned data.
This generalizes the unpaired image-to-image problem from generating pairs to generating sequences.
We extend cycle consistency to \textit{loop consistency} and alleviate difficulties associated with learning in the resulting long chains of computation.
We show competitive results compared to existing image-to-image techniques when modeling several different data sets including the Earth's seasons and aging of human faces.

assistant: text: LoopGAN extends cycle length in CycleGAN to enable unaligned sequential transformation for more than two time steps.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TRL: Discriminative Hints for Scalable Reverse Curriculum Learning

Paper content:
Deep reinforcement learning algorithms have proven successful in a variety of domains.
However, tasks with sparse rewards remain challenging when the state space is large.
Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished.
In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum.
This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning.
We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space.
In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards.

assistant: text: We propose Tendency RL to efficiently solve goal-oriented tasks with large state space using automated curriculum learning and discriminative shaping reward, which has the potential to tackle robot manipulation tasks with perception.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods

Paper content:
For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence.
In real-world scenarios, future states become increasingly uncertain and multi-modal, particularly on long time horizons.
Dropout based Bayesian inference provides a computationally tractable, theoretically well grounded approach to learn different hypotheses/models to deal with uncertain futures and make predictions that correspond well to observations -- are well calibrated.
However, it turns out that such approaches fall short to capture complex real-world scenes, even falling behind in accuracy when compared to the plain deterministic approaches.
This is because the used log-likelihood estimate discourages diversity.
In this work, we propose a novel Bayesian formulation for anticipating future scene states which leverages synthetic likelihoods that encourage the learning of diverse models to accurately capture the multi-modal nature of future scene states.
We show that our approach achieves accurate state-of-the-art predictions and calibrated probabilities through extensive experiments for scene anticipation on Cityscapes dataset.
Moreover, we show that our approach generalizes across diverse tasks such as digit generation and precipitation forecasting.

assistant: text: Dropout based Bayesian inference is extended to deal with multi-modality and is evaluated on scene anticipation tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compact Encoding of Words for Efficient Character-level Convolutional Neural Networks Text Classification

Paper content:
This paper puts forward a new text to tensor representation that relies on information compression techniques to assign shorter codes to the most frequently used characters.
This representation is language-independent with no need of pretraining and produces an encoding with no information loss.
It provides an adequate description of the morphology of text, as it is able to represent prefixes, declensions, and inflections with similar vectors and are able to represent even unseen words on the training dataset.
Similarly, as it is compact yet sparse, is ideal for speed up training times using tensor processing libraries.
As part of this paper, we show that this technique is especially effective when coupled with convolutional neural networks (CNNs) for text classification at character-level.
We apply two variants of CNN coupled with it.
Experimental results show that it drastically reduces the number of parameters to be optimized, resulting in competitive classification accuracy values in only a fraction of the time spent by one-hot encoding representations, thus enabling training in commodity hardware.

assistant: text: Using Compressing tecniques to Encoding of Words is a possibility for faster training of CNN and dimensionality reduction of representation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving End-to-End Object Tracking Using Relational Reasoning

Paper content:
Relational reasoning, the ability to model interactions and relations between objects, is valuable for robust multi-object tracking and pivotal for trajectory prediction.
In this paper, we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning.
We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment.
We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs.
To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature.

assistant: text: MOHART uses a self-attention mechanism to perform relational reasoning in multi-object tracking.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Protein Structure with a Differentiable Simulator

Paper content:
The Boltzmann distribution is a natural model for many systems, from brains to materials and biomolecules, but is often of limited utility for fitting data because Monte Carlo algorithms are unable to simulate it in available time.
This gap between the expressive capabilities and sampling practicalities of energy-based models is exemplified by the protein folding problem, since energy landscapes underlie contemporary knowledge of protein biophysics but computer simulations are challenged to fold all but the smallest proteins from first principles.
In this work we aim to bridge the gap between the expressive capacity of energy functions and the practical capabilities of their simulators by using an unrolled Monte Carlo simulation as a model for data.
We compose a neural energy function with a novel and efficient simulator based on Langevin dynamics to build an end-to-end-differentiable model of atomic protein structure given amino acid sequence information.
We introduce techniques for stabilizing backpropagation under long roll-outs and demonstrate the model's capacity to make multimodal predictions and to, in some cases, generalize to unobserved protein fold types when trained on a large corpus of protein structures.

assistant: text: We use an unrolled simulator as an end-to-end differentiable model of protein structure and show it can (sometimes) hierarchically generalize to unseen fold topologies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Clipping Free Attacks Against Neural Networks

Paper content:
During the last years, a remarkable breakthrough has been made in AI domain
thanks to artificial deep neural networks that achieved a great success in many
machine learning tasks in computer vision, natural language processing, speech
recognition, malware detection and so on.
However, they are highly vulnerable
to easily crafted adversarial examples.
Many investigations have pointed out this
fact and different approaches have been proposed to generate attacks while adding
a limited perturbation to the original data.
The most robust known method so far
is the so called C&W attack [1].
Nonetheless, a countermeasure known as fea-
ture squeezing coupled with ensemble defense showed that most of these attacks
can be destroyed [6].
In this paper, we present a new method we call Centered
Initial Attack (CIA) whose advantage is twofold : first, it insures by construc-
tion the maximum perturbation to be smaller than a threshold fixed beforehand,
without the clipping process that degrades the quality of attacks.
Second, it is
robust against recently introduced defenses such as feature squeezing, JPEG en-
coding and even against a voting ensemble of defenses.
While its application is
not limited to images, we illustrate this using five of the current best classifiers
on ImageNet dataset among which two are adversarialy retrained on purpose to
be robust against attacks.
With a fixed maximum perturbation of only 1.5% on
any pixel, around 80% of attacks (targeted) fool the voting ensemble defense and
nearly 100% when the perturbation is only 6%.
While this shows how it is difficult
to defend against CIA attacks, the last section of the paper gives some guidelines
to limit their impact.

assistant: text: In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Abstract Diagrammatic Reasoning with Multiplex Graph Networks

Paper content:
Abstract reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems.
In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks.
MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks.
MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels.
MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates.
We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM).
For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%.
For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin.

assistant: text: MXGNet is a multilayer, multiplex graph based architecture which achieves good performance on various diagrammatic reasoning tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Memorization in ReLU Networks

Paper content:
We propose a new notion of 'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix.
We measure this non-linearity by applying non-negative factorization to the activation matrix.
Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization.
Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases.
We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets.
Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping.

assistant: text: We use the non-negative rank of ReLU activation matrices as a complexity measure and show it (negatively) correlates with good generalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Automatic Labeling of Data for Transfer Learning

Paper content:
Transfer learning uses trained weights from a source model as the initial weightsfor the training of a target dataset.
A well chosen source with a large numberof labeled data leads to significant improvement in accuracy.
We demonstrate atechnique that automatically labels large unlabeled datasets so that they can trainsource models for transfer learning.
We experimentally evaluate this method, usinga baseline dataset of human-annotated ImageNet1K labels, against five variationsof this technique.
We show that the performance of these automatically trainedmodels come within 17% of baseline on average.

assistant: text: A technique for automatically labeling large unlabeled datasets so that they can train source models for transfer learning and its experimental evaluation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Over-parameterization Improves Generalization in the XOR Detection Problem

Paper content:
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization.
However, there is currently no theoretical analysis that explains this observation.
In this work, we study a simplified learning task with over-parameterized convolutional networks that empirically exhibits the same qualitative phenomenon.
For this setting, we provide a theoretical analysis of the optimization and generalization performance of gradient descent.
Specifically, we prove data-dependent sample complexity bounds which show that over-parameterization improves the generalization performance of gradient descent.

assistant: text: We show in a simplified learning task that over-parameterization improves generalization of a convnet that is trained with gradient descent.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules

Paper content:
The brain performs unsupervised learning and (perhaps) simultaneous supervised learning.
This raises the question as to whether a hybrid of supervised and unsupervised methods will produce better learning.
Inspired by the rich space of Hebbian learning rules, we set out to directly learn the unsupervised learning rule on local information that best augments a supervised signal.
We present the Hebbian-augmented training algorithm (HAT) for combining gradient-based learning with an unsupervised rule on pre-synpatic activity, post-synaptic activities, and current weights.
We test HAT's effect on a simple problem (Fashion-MNIST) and find consistently higher performance than supervised learning alone.
This finding provides empirical evidence that unsupervised learning on synaptic activities provides a strong signal that can be used to augment gradient-based methods.
We further find that the meta-learned update rule is a time-varying function; thus, it is difficult to pinpoint an interpretable Hebbian update rule that aids in training.
We do find that the meta-learner eventually degenerates into a non-Hebbian rule that preserves important weights so as not to disturb the learner's convergence.
The HAT algorithm demonstrates that local, unsupervised signals can provide performance-improving weight updates.
Neural nets under HAT converge to better asymptotic losses as long as there is sufficient time (> 0.5 epochs) and a sufficient number of labels (> 20% of the data is labeled).
The latter finding is surprising since the addition of an unsupervised learning algorithm depends on the presence of labels in order to deliver marginal benefits over gradient descent.
The underlying form of the learned rule that makes HAT successful is still a mystery; we find that while the meta-learner may learn a useful update rule during training, the meta-learner does not converge to this useful rule in the long run and instead devolves into a linear function ConvergedRule.
This converged function preserves fully-converged weights by reinforcing incoming weights for neurons with high activations.

assistant: text: Metalearning unsupervised update rules for neural networks improves performance and potentially demonstrates how neurons in the brain learn without access to global labels.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SuperTML: Domain Transfer from Computer Vision to Structured Tabular Data through Two-Dimensional Word Embedding

Paper content:
Structured tabular data is the most commonly used form of data in industry according to a Kaggle ML and DS Survey.
Gradient Boosting Trees, Support Vector Machine, Random Forest, and Logistic Regression are typically used for classification tasks on tabular data.
The recent work of Super Characters method using two-dimensional word embeddings achieved state-of-the-art results in text classification tasks, showcasing the promise of this new approach.
In this paper, we propose the SuperTML method, which borrows the idea of Super Characters method and two-dimensional embeddings to address the problem of classification on tabular data.
For each input of tabular data, the features are first projected into two-dimensional embeddings like an image, and then this image is fed into fine-tuned ImageNet CNN models for classification.
Experimental results have shown that the proposed SuperTML method have achieved state-of-the-art results on both large and small datasets.

assistant: text: Deep learning on structured tabular data using two-dimensional word embedding with fine-tuned ImageNet pre-trained CNN model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY

Paper content:
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity.
In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility.
In this work, we present a new approach that prunes a given network once at initialization prior to training.
To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task.
This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations.
After pruning, the sparse network is trained in the standard way.
Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks.
Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

assistant: text: We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spiroplots: a new discrete-time dynamical system to generate curve patterns

Paper content:
We introduce a new procedural dynamic system that can generate a variety of shapes that often appear as curves, but technically, the figures are plots of many points.
We name them spiroplots and show how this new system relates to other procedures or processes that generate figures.
Spiroplots are an extremely simple process but with a surprising visual variety.
We prove some fundamental properties and analyze some instances to see how the geometry or topology of the input determines the generated figures.
We show that some spiroplots have a finite cycle and return to the initial situation, whereas others will produce new points infinitely often.
This paper is accompanied by a JavaScript app that allows anyone to generate spiroplots.

assistant: text: A new, very simple dynamic system is introduced that generates pretty patterns; properties are proved and possibilities are explored
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Greedy Approach to Max-Sliced Wasserstein GANs

Paper content:
Generative Adversarial Networks have made data generation possible in various use cases, but in case of complex, high-dimensional distributions it can be difficult to train them, because of convergence problems and the appearance of mode collapse.
Sliced Wasserstein GANs and especially the application of the Max-Sliced Wasserstein distance made it possible to approximate Wasserstein distance during training in an efficient and stable way and helped ease convergence problems of these architectures.
This method transforms sample assignment and distance calculation into sorting the one-dimensional projection of the samples, which results a sufficient approximation of the high-dimensional Wasserstein distance.
In this paper we will demonstrate that the approximation of the Wasserstein distance by sorting the samples is not always the optimal approach and the greedy assignment of the real and fake samples can result faster convergence and better approximation of the original distribution.

assistant: text: We apply a greedy assignment on the projected samples instead of sorting to approximate Wasserstein distance
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models

Paper content:
Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system.
However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data.
In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods.
We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison.
We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

assistant: text: We pose that generative models' likelihoods are excessively influenced by the input's complexity, and propose a way to compensate it when detecting out-of-distribution inputs
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Theoretical Analysis of Auto Rate-Tuning by Batch Normalization

Paper content:
Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization.
While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking.
Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates.
It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0.3), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T^{−1/2} in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates.
A similar result with convergence rate T^{−1/4} is also shown for stochastic gradient descent.

assistant: text: We give a theoretical analysis of the ability of batch normalization to automatically tune learning rates, in the context of finding stationary points for a deep learning objective.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Targeted sampling of enlarged neighborhood via Monte Carlo tree search for TSP

Paper content:
The travelling salesman problem (TSP) is a well-known combinatorial optimization problem with a variety of real-life applications.
We tackle TSP by incorporating machine learning methodology and leveraging the variable neighborhood search strategy.
More precisely, the search process is considered as a Markov decision process (MDP), where a 2-opt local search is used to search within a small neighborhood, while a Monte Carlo tree search (MCTS) method (which iterates through simulation, selection and back-propagation steps), is used to sample a number of targeted actions within an enlarged neighborhood.
This new paradigm clearly distinguishes itself from the existing machine learning (ML) based paradigms for solving the TSP, which either uses an end-to-end ML model, or simply applies traditional techniques after ML for post optimization.
Experiments based on two public data sets show that, our approach clearly dominates all the existing learning based TSP algorithms in terms of performance, demonstrating its high potential on the TSP.
More importantly, as a general framework without complicated hand-crafted rules, it can be readily extended to many other combinatorial optimization problems.

assistant: text: This paper combines Monte Carlo tree search with 2-opt local search in a variable neighborhood mode to solve the TSP effectively.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Invariant and Equivariant Graph Networks

Paper content:
Invariant and equivariant networks have been successfully used for learning images, sets, point clouds, and graphs.
A basic challenge in developing such networks is finding the maximal collection of invariant and equivariant \emph{linear} layers.
Although this question is answered for the first three examples (for popular transformations, at-least), a full characterization of invariant and equivariant linear layers for graphs is not known.
In this paper we provide a characterization of all permutation invariant and equivariant linear layers for (hyper-)graph data, and show that their dimension, in case of edge-value graph data, is $2$ and $15$, respectively.
More generally, for graph data defined on $k$-tuples of nodes, the dimension is the $k$-th and $2k$-th Bell numbers.
Orthogonal bases for the layers are computed, including generalization to multi-graph data.
The constant number of basis elements and their characteristics allow successfully applying the networks to different size graphs.
From the theoretical point of view, our results generalize and unify recent advancement in equivariant deep learning.
In particular, we show that our model is capable of approximating any message passing neural network.
Applying these new linear layers in a simple deep neural network framework is shown to achieve comparable results to state-of-the-art and to have better expressivity than previous invariant and equivariant bases.

assistant: text: The paper provides a full characterization of permutation invariant and equivariant linear layers for graph data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Represent Programs with Property Signatures

Paper content:
We introduce the notion of property signatures, a representation for programs and
program specifications meant for consumption by machine learning algorithms.
Given a function with input type τ_in and output type τ_out, a property is a function
of type: (τ_in, τ_out) → Bool that (informally) describes some simple property
of the function under consideration.
For instance, if τ_in and τ_out are both lists
of the same type, one property might ask ‘is the input list the same length as the
output list?’.
If we have a list of such properties, we can evaluate them all for our
function to get a list of outputs that we will call the property signature.
Crucially,
we can ‘guess’ the property signature for a function given only a set of input/output
pairs meant to specify that function.
We discuss several potential applications of
property signatures and show experimentally that they can be used to improve
over a baseline synthesizer so that it emits twice as many programs in less than
one-tenth of the time.

assistant: text: We represent a computer program using a set of simpler programs and use this representation to improve program synthesis techniques.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks

Paper content:
Deep neural networks, in particular convolutional neural networks, have become highly effective tools for compressing images and solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements.
This success can be attributed in part to their ability to represent and generate natural images well.
Contrary to classical tools such as wavelets, image-generating deep neural networks have a large number of parameters---typically a multiple of their output dimension---and need to be trained on large datasets.
In this paper, we propose an untrained simple image model, called the deep decoder, which is a deep neural network that can generate natural images from very few weight parameters.
The deep decoder has a simple architecture with no convolutions and fewer weight parameters than the output dimensionality.
This underparameterization enables the deep decoder to compress images into a concise set of network weights, which we show is on par with wavelet-based thresholding.
Further, underparameterization provides a barrier to overfitting, allowing the deep decoder to have state-of-the-art performance for denoising.
The deep decoder is simple in the sense that each layer has an identical structure that consists of only one upsampling unit, pixel-wise linear combination of channels, ReLU activation, and channelwise normalization.
This simplicity makes the network amenable to theoretical analysis, and it sheds light on the aspects of neural networks that enable them to form effective signal representations.

assistant: text: We introduce an underparameterized, nonconvolutional, and simple deep neural network that can, without training, effectively represent natural images and solve image processing tasks like compression and denoising competitively.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Thieves on Sesame Street! Model Extraction of BERT-based APIs

Paper content:
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model.
Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack.
In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering.
Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model.
Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones.

assistant: text: Outputs of modern NLP APIs on nonsensical text provide strong signals about model internals, allowing adversaries to steal the APIs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RaPP: Novelty Detection with Reconstruction along Projection Pathway

Paper content:
We propose RaPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder.
Precisely, RaPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces.
We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input.
In order to aggregate the hidden space activation values, we propose two metrics, which enhance the novelty detection performance.
Through extensive experiments using diverse datasets, we validate that RaPP improves novelty detection performances of autoencoder-based approaches.
Besides, we show that RaPP outperforms recent novelty detection methods evaluated on popular benchmarks.

assistant: text: A new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency

Paper content:
Convolutional neural networks (CNNs) are commonly trained using a fixed spatial image size predetermined for a given model.
Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
In this work, we describe and evaluate a novel mixed-size training regime that mixes several image sizes at training time.
We demonstrate that models trained using our method are more resilient to image size changes and generalize well even on small images.
This allows faster inference by using smaller images at test time.
For instance, we receive a 76.43% top-1 accuracy using ResNet50 with an image size of 160, which matches the accuracy of the baseline model with 2x fewer computations.
Furthermore, for a given image size used at test time, we show this method can be exploited either to accelerate training or the final test accuracy.
For example, we are able to reach a 79.27% accuracy with a model evaluated at a 288 spatial size for a relative improvement of 14% over the baseline.

assistant: text: Training convnets with mixed image size can improve results across multiple sizes at evaluation
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Entity Representations for Few-Shot Reconstruction of Wikipedia Categories

Paper content:
Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases.
Motivated by the observation that efforts to code
world knowledge into machine readable knowledge bases tend to be entity-centric,
we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the contexts in which those entities were mentioned.
We show that large scale training of neural models allows us to learn extremely high fidelity entity typing information, which we demonstrate with few-shot reconstruction of Wikipedia categories.
Our learning approach is powerful enough
to encode specialized topics such as Giro d’Italia cyclists.

assistant: text: We learn entity representations that can reconstruct Wikipedia categories with just a few exemplars.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MCTSBug: Generating Adversarial Text Sequences via Monte Carlo Tree Search and Homoglyph Attack

Paper content:
Crafting adversarial examples on discrete inputs like text sequences is fundamentally different from generating such examples for continuous inputs like images.
This paper tries to answer the question: under a black-box setting, can we create adversarial examples automatically to effectively fool deep learning classifiers on texts by making imperceptible changes?
Our answer is a firm yes.
Previous efforts mostly replied on using gradient evidence, and they are less effective either due to finding the nearest neighbor word (wrt meaning) automatically is difficult or relying heavily on hand-crafted linguistic rules.
We, instead, use Monte Carlo tree search (MCTS) for finding the most important few words to perturb and perform homoglyph attack by replacing one character in each selected word with a symbol of identical shape.
Our novel algorithm, we call MCTSBug, is black-box and extremely effective at the same time.
Our experimental results indicate that MCTSBug can fool deep learning classifiers at the success rates of 95% on seven large-scale benchmark datasets, by perturbing only a few characters.
Surprisingly, MCTSBug, without relying on gradient information at all, is more effective than the gradient-based white-box baseline.
Thanks to the nature of homoglyph attack, the generated adversarial perturbations are almost imperceptible to human eyes.

assistant: text: Use Monte carlo Tree Search and Homoglyphs to generate indistinguishable adversarial samples on text data
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Now I Remember! Episodic Memory For Reinforcement Learning

Paper content:
Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car.
Endowing reinforcement learning agents with episodic memory is a key step on the path toward replicating human-like general intelligence.
We analyze why standard RL agents lack episodic memory today, and why existing RL tasks don't require it.
We design a new form of external memory called Masked Experience Memory, or MEM, modeled after key features of human episodic memory.
To evaluate episodic memory we define an RL task based on the common children's game of Concentration.
We find that a MEM RL agent leverages episodic memory effectively to master Concentration, unlike the baseline agents we tested.

assistant: text: Implementing and evaluating episodic memory for RL.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Convolutional Networks as shallow Gaussian Processes

Paper content:
We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks.
For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN.
Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer.
The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.

assistant: text: We show that CNNs and ResNets with appropriate priors on the parameters are Gaussian processes in the limit of infinitely many convolutional filters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Paper content:
We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning.
We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task.
Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms.
We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the well-known finite-time bounds of Approximate Value-Iteration to the multi-task setting.
In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

assistant: text: A study on the benefit of sharing representation in Multi-Task Reinforcement Learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Temporal Difference Weighted Ensemble For Reinforcement Learning

Paper content:
Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with a single function.
In reinforcement learning, ensemble algorithms such as an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration.
In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal difference errors.
The advantage of this algorithm is that it improves ensemble performance by reducing weights of Q-functions unfamiliar with current trajectories.
We provide experimental results for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms.

assistant: text: Ensemble method for reinforcement learning that weights Q-functions based on accumulated TD errors.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Probability Calibration for Knowledge Graph Embedding Models

Paper content:
Knowledge graph embedding research has overlooked the problem of probability calibration.
We show popular embedding models are indeed uncalibrated.
That means probability estimates associated to predicted triples are unreliable.
We present a novel method to calibrate a model when ground truth negatives are not available, which is the usual case in knowledge graphs.
We propose to use Platt scaling and isotonic regression alongside our method.
Experiments on three datasets with ground truth negatives show our contribution leads to well calibrated models when compared to the gold standard of using negatives.
We get significantly better results than the uncalibrated models from all calibration methods.
We show isotonic regression offers the best the performance overall, not without trade-offs.
We also show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds.

assistant: text: We propose a novel method to calibrate knowledge graph embedding models without the need of negative examples.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DeepObfusCode: Source Code Obfuscation Through Sequence-to-Sequence Networks

Paper content:
The paper explores a novel methodology in source code obfuscation through the application of text-based recurrent neural network network (RNN) encoder-decoder models in ciphertext generation and key generation.
Sequence-to-sequence models
are incorporated into the model architecture to generate obfuscated code, generate the deobfuscation key, and live execution.
Quantitative benchmark comparison to existing obfuscation methods indicate significant improvement in stealth and execution cost for the proposed solution, and experiments regarding the model’s properties yield positive results regarding its character variation, dissimilarity to the original codebase, and consistent length of obfuscated code.

assistant: text: Obfuscate code using seq2seq networks, and execute using the obfuscated code and key pair