ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Critical initialisation in continuous approximations of binary neural networks

Paper content:
The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated.
We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks.
The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth.
Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do.
The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at initialisation.
Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.
This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

assistant: text: An in-depth exploration of stochastic binary networks, continuous surrogates, and their training dynamics, with insights on how to initialize weights for best performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

Paper content:
Transferring knowledge across tasks to improve data-efficiency is one of
the open key challenges in the area of global optimization algorithms.
Readily
available algorithms are typically designed to be universal optimizers and, thus,
often suboptimal for specific tasks.
We propose a novel transfer learning method to
obtain customized optimizers within the well-established framework of Bayesian
optimization, allowing our algorithm to utilize the proven generalization
capabilities of Gaussian processes.
Using reinforcement learning to meta-train an
acquisition function (AF) on a set of related tasks, the proposed method learns to
extract implicit structural information and to exploit it for improved data-efficiency.
We present experiments on a sim-to-real transfer task as well as on several simulated
functions and two hyperparameter search problems.
The results show that our
algorithm (1) automatically identifies structural properties of objective functions
from available source tasks or simulations, (2) performs favourably in settings with
both scarse and abundant source data, and (3) falls back to the performance level
of general AFs if no structure is present.

assistant: text: The authors present MetaBO which uses reinforcement learning to meta-learn the acquisition function for Bayesian Optimization, showing increasing sample efficiency on new tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Variational Semi-Supervised Novelty Detection

Paper content:
In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.
A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution.
In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies.
In this work,we propose two variational methods for training VAEs for SSAD.
The intuitive idea in both methods is to train the encoder to ‘separate’ between latent vectors for normal and outlier data.
We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.
Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture.
When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

assistant: text: The papers propose two methods of VAE-like approaches for semi-supervised novelty detection, MML-VAE and DP-VAE.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Neural Machine Translation

Paper content:
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.
There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal.
In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora.
Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation.
Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation.
The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively.
Our implementation is released as an open source project.

assistant: text: The authors present a model for unsupervised NMT which requires no parallel corpora between the two languages of interest.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Compressing U-net Using Knowledge Distillation

Paper content:
We study the use of knowledge distillation to compress the U-net architecture.
We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.
This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.

assistant: text: The authors introduced a modified distillation strategy to compress a U-net architecture by over 1000x while retaining an accuracy close to the original U-net.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Rank Learning Curves

Paper content:
Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations.
In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training.
In contrast to existing methods, we consider this task as a ranking and transfer learning problem.
We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves.
We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture.
In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

assistant: text: Learn to rank learning curves in order to stop unpromising training jobs early. Novelty: use of pairwise ranking loss to directly model the probability of improving and transfer learning across data sets to reduce required training data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem

Paper content:
Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in deep learning and approximation theory.
In a seminal paper, Telgarsky high- lighted the benefits of depth by presenting a family of functions (based on sim- ple triangular waves) for which DNNs achieve zero classification error, whereas shallow networks with fewer than exponentially many nodes incur constant error.
Even though Telgarsky’s work reveals the limitations of shallow neural networks, it doesn’t inform us on why these functions are difficult to represent and in fact he states it as a tantalizing open question to characterize those functions that cannot be well-approximated by smaller depths.
In this work, we point to a new connection between DNNs expressivity and Sharkovsky’s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks for representing functions based on the presence of a generalized notion of fixed points, called periodic points (a fixed point is a point of period 1).
Motivated by our observation that the triangle waves used in Telgarsky’s work contain points of period 3 – a period that is special in that it implies chaotic behaviour based on the celebrated result by Li-Yorke – we proceed to give general lower bounds for the width needed to represent periodic functions as a function of the depth.
Technically, the crux of our approach is based on an eigenvalue analysis of the dynamical systems associated with such functions.

assistant: text: In this work, we point to a new connection between DNNs expressivity and Sharkovsky’s Theorem from dynamical systems, that enables us to characterize the depth-width trade-offs of ReLU networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

Paper content:
Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture.
In this paper, we present a novel approach, namely  Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.
In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut.
This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels.
We solve it by introducing  edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search.
Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability.
Experiment results demonstrate the effectiveness of the proposed method.
Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search.
Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0
.

assistant: text: An extension of the neural architecture search method DARTS that addresses its shortcoming of immense memory cost by using a random subset of channels and a method to normalize edges.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking learning rate schedules for stochastic optimization

Paper content:
There is a stark disparity between the learning rate schedules used in the practice of large scale machine learning and what are considered admissible learning rate schedules prescribed in the theory of stochastic approximation.
Recent results, such as in the 'super-convergence' methods which use oscillating learning rates, serve to emphasize this point even more.
One plausible explanation is that non-convex neural network training procedures are better suited to the use of fundamentally different learning rate  schedules, such as the ``cut the learning rate every constant number of epochs'' method (which more closely resembles an exponentially decaying learning rate schedule); note that this widely used schedule is in stark contrast to the polynomial decay schemes prescribed in the stochastic approximation literature, which are indeed shown to be (worst case) optimal for classes of convex optimization problems.
The main contribution of this work shows that the picture is far more nuanced, where we do not even need to move to non-convex optimization to show other learning rate schemes can be far more effective.
In fact, even for the simple case of stochastic linear regression with a fixed time horizon, the rate achieved by any polynomial decay scheme is sub-optimal compared to the statistical minimax rate (by a factor of condition number); in contrast the ```''cut the learning rate every constant number of epochs'' provides an exponential improvement (depending only logarithmically on the condition number) compared to any polynomial decay scheme.
Finally, it is important to ask if our theoretical insights are somehow fundamentally tied to quadratic loss minimization (where we have circumvented minimax lower bounds for more general convex optimization problems)?
Here, we conjecture that recent results which make the gradient norm small at a near optimal rate, for both convex and non-convex optimization, may also provide more insights into learning rate schedules used in practice.

assistant: text: The paper studies the effect of learning-rate choices for stochastic optimization, focusing on least-mean-squares with decaying stepsizes
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Estimating Heterogeneous Treatment Effects Using Neural Networks With The Y-Learner

Paper content:
We develop the Y-learner for estimating heterogeneous treatment effects in experimental and observational studies.
The Y-learner is designed to leverage the abilities of neural networks to optimize multiple objectives and continually update, which allows for better pooling of underlying feature information between treatment and control groups.
We evaluate the Y-learner on three test problems: (1) A set of six simulated data benchmarks from the literature.
(2) A real-world large-scale experiment on voter persuasion.
(3) A task from the literature that estimates artificially generated treatment effects on MNIST didgits.
The Y-learner achieves state of the art results on two of the three tasks.
On the MNIST task, it gets the second best results.

assistant: text: The authors propose the Y-learner to estimate conditional average treatment effect(CATE), which simultaneously updates the parameters of the outcome functions and the CATE estimator.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

Paper content:
We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning.
In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach.
Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial.
To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation.
We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice.
Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods.

assistant: text: The authors proposes a novel approach in learning a representation for HRL and state an intriguing connection between representation learning and bounding the sub-optimality which results in a gradient based algorithm
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fixing Weight Decay Regularization in Adam

Paper content:
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.
We provide empirical evidence that our proposed modification
(i) 
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and
(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).
We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.
Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32.
Our source code will become available after the review process.

assistant: text: Fixing weight decay regularization in adaptive gradient methods such as Adam
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training Structured Efficient Convolutional Layers

Paper content:
Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting.
We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods.
A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.

assistant: text: It's possible to substitute the weight matrix in a convolutional layer to train it as a structured efficient layer; performing as well as low-rank decomposition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Modelling the influence of data structure on learning in neural networks

Paper content:
The lack of crisp mathematical models that capture the structure of real-world
data sets is a major obstacle to the detailed theoretical understanding of deep
neural networks.
Here, we first demonstrate the effect of structured data sets
by experimentally comparing the dynamics and the performance of two-layer
networks trained on two different data sets:
(i) an unstructured synthetic data
set containing random i.i.d. inputs, and
(ii) a simple canonical data set such
as MNIST images.
Our analysis reveals two phenomena related to the dynamics of
the networks and their ability to generalise that only appear when training on
structured data sets.
Second, we introduce a generative model for data sets,
where high-dimensional inputs lie on a lower-dimensional manifold and have
labels that depend only on their position within this manifold.
We call it the
*hidden manifold model* and we experimentally demonstrate that training
networks on data sets drawn from this model reproduces both the phenomena seen
during training on MNIST.

assistant: text: The paper studies how different settings of data structure affect learning of neural networks and how to mimic behavior on real datasets when learning on a synthetic one.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Passage Ranking with Weak Supervision

Paper content:
In this paper, we propose a \textit{weak supervision} framework for neural ranking tasks based on the data programming paradigm \citep{Ratner2016}, which enables us to leverage multiple weak supervision signals from different sources.
Empirically, we consider two sources of weak supervision signals, unsupervised ranking functions and semantic feature similarities.
We train a BERT-based passage-ranking model (which achieves new state-of-the-art performances on two benchmark datasets with full supervision) in our weak supervision framework.
Without using ground-truth training labels, BERT-PR models outperform BM25 baseline by a large margin on all three datasets and even beat the previous state-of-the-art results with full supervision on two of datasets.

assistant: text: The authors propose a combination of BERT and the weak supervision framework to tackle the problem of passage ranking, obtaining results better than the fully supervised state-of-the-art.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Examples Are a Natural Consequence of Test Error in Noise

Paper content:
Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input.
At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise.
In this work, we show that these are two manifestations of the same underlying phenomenon.
We establish this connection in several ways.
First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images.
Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions.
Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error.
All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions.
This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.

assistant: text: Small adversarial perturbations should be expected given observed error rates of models outside the natural data distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Generative Inpainting with Comparative Sample Augmentation

Paper content:
Recent advancements in deep learning techniques such as Convolutional Neural Networks(CNN) and Generative Adversarial Networks(GAN) have achieved breakthroughs in the problem of semantic image inpainting, the task of reconstructing missing pixels in given images.
While much more effective than conventional approaches, deep learning models require large datasets and great computational resources for training, and inpainting quality varies considerably when training data vary in size and diversity.
To address these problems, we present in this paper a inpainting strategy of \textit{Comparative Sample Augmentation}, which enhances the quality of training set by filtering out irrelevant images and constructing additional images using information about the surrounding regions of the images to be inpainted.
Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying sizes, while maintaining inpainting quality as measured by qualitative and quantitative metrics for a large class of deep models, with little need for model-specific consideration.

assistant: text: Help image inpainting using GANs by using a comparative augmenting filter and adding random noise to each pixel.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Paper content:
Recent improvements in large-scale language models have driven progress on automatic generation of syntactically and semantically consistent text for many real-world applications.
Many of these advances leverage the availability of large corpora.
While training on such corpora encourages the model to understand long-range dependencies in text, it can also result in the models internalizing the social biases present in the corpora.
This paper aims to quantify and reduce biases exhibited by language models.
Given a conditioning context (e.g. a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g. country names, occupations, genders, etc.) in the conditioning context, a.k.a. counterfactual evaluation.
We quantify these biases by adapting individual and group fairness metrics from the fair machine learning literature.
Extensive evaluation on two different corpora (news articles and Wikipedia) shows that state-of-the-art Transformer-based language models exhibit biases learned from data.
We propose embedding-similarity and sentiment-similarity regularization methods that improve both individual and group fairness metrics without sacrificing perplexity and semantic similarity---a positive step toward development and deployment of fairer language models for real-world applications.

assistant: text: We reduce sentiment biases based on counterfactual evaluation of text generation using language models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LEARNING TO PROPAGATE LABELS: TRANSDUCTIVE PROPAGATION NETWORK FOR FEW-SHOT LEARNING

Paper content:
The goal of few-shot learning is to learn a classifier that generalizes well even when trained with a limited number of training instances per class.
The recently introduced meta-learning approaches tackle this problem by learning a generic classifier across a large number of multiclass classification tasks and generalizing the model to a new task.
Yet, even with such meta-learning, the low-data problem in the novel classification task still remains.
In this paper, we propose Transductive Propagation Network (TPN), a novel meta-learning framework for transductive inference that classifies the entire test set at once to alleviate the low-data problem.
Specifically, we propose to learn to propagate labels from labeled instances to unlabeled test instances, by learning a graph construction module that exploits the manifold structure in the data.
TPN jointly learns both the parameters of feature embedding and the graph construction in an end-to-end manner.
We validate TPN on multiple benchmark datasets, on which it largely outperforms existing few-shot learning approaches and achieves the state-of-the-art results.

assistant: text: This paper proposes to address few-shot learning in a transductive way by learning a label propagation model in an end-to-end manner, the first to learn label propagation for transductive few-shot learning and produced effective empirical results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stable Recurrent Models

Paper content:
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks.
In this work, we conduct a thorough investigation of stable recurrent models.
Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent.
Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks.
Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime.
Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

assistant: text: Studies the stability of RNNs and investigation of spectral normalization to sequential predictions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks

Paper content:
Long training times of deep neural networks are a bottleneck in machine learning research.
The major impediment to fast training is the quadratic growth of both memory and compute requirements of dense and convolutional layers with respect to their information bandwidth.
Recently, training `a priori' sparse networks has been proposed as a method for allowing layers to retain high information bandwidth, while keeping memory and compute low.
However, the choice of which sparse topology should be used in these networks is unclear.
In this work, we provide a theoretical foundation for the choice of intra-layer topology.
First, we derive a new sparse neural network initialization scheme that allows us to explore the space of very deep sparse networks.
Next, we evaluate several topologies and show that seemingly similar topologies can often have a large difference in attainable accuracy.
To explain these differences, we develop a data-free heuristic that can evaluate a topology independently from the dataset the network will be trained on.
We then derive a set of requirements that make a good topology, and arrive at a single topology that satisfies all of them.

assistant: text: We investigate pruning DNNs before training and provide an answer to which topology should be used for training a priori sparse networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference

Paper content:
Computations for the softmax function in neural network models are expensive when the number of output classes is large.
This can become a significant issue in both training and inference for such models.
In this paper, we present Doubly Sparse Softmax (DS-Softmax), Sparse Mixture of Sparse of Sparse Experts, to improve the efficiency for softmax inference.
During training, our method learns a two-level class hierarchy by dividing entire output class space into several partially overlapping experts.
Each expert is responsible for a learned subset of the output class space and each output class only belongs to a small number of those experts.
During inference, our method quickly locates the most probable expert to compute small-scale softmax.
Our method is learning-based and requires no knowledge of the output class partition space a priori.
We empirically evaluate our method on several real-world tasks and demonstrate that we can achieve significant computation reductions without loss of

assistant: text: We present doubly sparse softmax, the sparse mixture of sparse of sparse experts, to improve the efficiency for softmax inference through exploiting the two-level overlapping hierarchy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Recurrent Convolutions: A Model Compression Point of View

Paper content:
Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals.
We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks.
RC reduces the redundancy across layers and is complementary to most existing model compression approaches.
However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.
This reduces the value of RC for model compression.
In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps.
We provide insights on why this works.
Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression.

assistant: text: The author modifies the recurrent convolution neural network (RCNN) with independent batch normalization, with the experimental results on RCNN compatible with the ResNet neural network architecture when it contains the same number of layers.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Preliminary theoretical troubleshooting in Variational Autoencoder

Paper content:
What would be learned by variational autoencoder(VAE) and what influence the disentanglement of VAE?
This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case.
On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension.
Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension.
On disentanglement issue,   subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made.
On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class.
On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors.
On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.

assistant: text: This paper tries to preliminarily address the disentanglement theoretically in the idealistic situation and practically through noise modelling perspective in the realistic case.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Neural Forests: An Architecture for Tabular Data

Paper content:
Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.
However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.
Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)
--  a novel architecture that combines elements from decision trees as well as dense residual connections.
We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks.
These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data.
To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task.

assistant: text: A novel neural network architecture mimicking how decision forests work to tackle the general problem of training deep models for tabular data and showcasing effectiveness on par with GBDT.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS

Paper content:
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.
The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.
We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.

assistant: text: This manuscript introduces a new layer-wise transform, EquiNorm, to improve upon batch normalization that does not modify the inputs to the layers but rather the layer weights.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks

Paper content:
Learning when to communicate and doing that effectively is essential in multi-agent tasks.
Recent works show that continuous communication allows efficient training with back-propagation in multi-agent scenarios, but have been restricted to fully-cooperative tasks.
In this paper, we present Individualized Controlled Continuous Communication Model (IC3Net) which has better training efficiency than simple continuous communication model, and can be applied to semi-cooperative and competitive settings along with the cooperative settings.
IC3Net controls continuous communication with a gating mechanism and uses individualized rewards foreach agent to gain better performance and scalability while fixing credit assignment issues.
Using variety of tasks including StarCraft BroodWars explore and combat scenarios, we show that our network yields improved performance and convergence rates than the baselines as the scale increases.
Our results convey that IC3Net agents learn when to communicate based on the scenario and profitability.

assistant: text: The authors propose an interesting gating scheme allowing agents to communicate in an multi-agent RL setting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SIMULTANEOUS ATTRIBUTED NETWORK EMBEDDING AND CLUSTERING

Paper content:
To deal simultaneously with both, the attributed network embedding and clustering, we propose a new model.
It exploits both content and structure information, capitalising on their simultaneous use.
The proposed model relies on the approximation of the relaxed continuous embedding solution by the true discrete clustering one.
Thereby, we show that incorporating an embedding representation provides simpler and more interpretable solutions.
Experiment results demonstrate that the proposed algorithm performs better, in terms of clustering and embedding, than the state-of-art algorithms, including deep learning methods devoted to similar tasks for attributed network datasets with different proprieties.

assistant: text: This paper propose a novel matrix decomposition framework for simultaneous attributed network data embedding and clustering.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Universal Deep Neural Network Compression

Paper content:
In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment.
Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding.
In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.
Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.
Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively.

assistant: text: Introduces a pipeline for network compression that is similar to deep compression and uses randomized lattice quantization instead of the classical vector quantization, and uses universal source coding (bzip2) instead of Huffman coding.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Diffusing Policies : Towards Wasserstein Policy Gradient Flows

Paper content:
Policy gradients methods often achieve better performance when the change in policy is limited to a small Kullback-Leibler divergence.
We derive policy gradients where the change in policy is limited to a small Wasserstein distance (or trust region).
This is done in the discrete and continuous multi-armed bandit settings with entropy regularisation.
We show that in the small steps limit with respect to the Wasserstein distance $W_2$, policy dynamics are governed by the heat equation, following the Jordan-Kinderlehrer-Otto result.
This means that policies undergo diffusion and advection, concentrating near actions with high reward.
This helps elucidate the nature of convergence in the probability matching setup, and provides justification for empirical practices such as Gaussian policy priors and additive gradient noise.

assistant: text: The authors studied policy gradient with change of policies limited by a trust region of Wasserstein distance in the multi-armed bandit setting, showing that in the small steps limit, the policy dynamics are governed by the heat equation (Fokker-Planck equation).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Benchmarking Adversarial Robustness

Paper content:
Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.
While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.
In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.
After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.
Based on the evaluation results, we draw several important findings and provide insights for future research.

assistant: text: This paper presents an evaluation of different kinds of classification models under various adversarial attack methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Projection Based Constrained Policy Optimization

Paper content:
In this paper, we consider the problem of learning control policies that optimize areward function while satisfying constraints due to considerations of safety, fairness, or other costs.
We propose a new algorithm - Projection Based ConstrainedPolicy Optimization (PCPO), an iterative method for optimizing policies in a two-step process - the first step performs an unconstrained update while the secondstep reconciles the constraint violation by projection the policy back onto the constraint set.
We theoretically analyze PCPO and provide a lower bound on rewardimprovement, as well as an upper bound on constraint violation for each policy update.
We further characterize the convergence of PCPO with projection basedon two different metrics - L2 norm and Kullback-Leibler divergence.
Our empirical results over several control tasks demonstrate that our algorithm achievessuperior performance, averaging more than 3.5 times less constraint violation andaround 15% higher reward compared to state-of-the-art methods.

assistant: text: This paper introduces a constrained policy optimization algorithm using a two-step optimization process, where policies that do not satisfy the constraint can be projected back into the constraint set.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: All SMILES Variational Autoencoder for Molecular Property Prediction and Optimization

Paper content:
Variational autoencoders (VAEs) defined over SMILES string and graph-based representations of molecules promise to improve the optimization of molecular properties, thereby revolutionizing the pharmaceuticals and materials industries.
However, these VAEs are hindered by the non-unique nature of SMILES strings and the computational cost of graph convolutions.
To efficiently pass messages along all paths through the molecular graph, we encode multiple SMILES strings of a single molecule using a set of stacked recurrent neural networks, harmonizing hidden representations of each atom between SMILES representations, and use attentional pooling to build a final fixed-length latent representation.
By then decoding to a disjoint set of SMILES strings of the molecule, our All SMILES VAE learns an almost bijective mapping between molecules and latent representations near the high-probability-mass subspace of the prior.
Our SMILES-derived but molecule-based latent representations significantly surpass the state-of-the-art in a variety of fully- and semi-supervised property regression and molecular property optimization tasks.

assistant: text: The authors describe a novel variational autoencoder like method for molecules which encode molecules as strings to reduce the operations needed to share information across atoms in the molecule.