ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing Generalization of First-Order Meta-Learning

Paper content:
In this study we focus on first-order meta-learning algorithms that aim to learn a parameter initialization of a network which can quickly adapt to new concepts, given a few examples.
We investigate two approaches to enhance generalization and speed of learning of such algorithms, particularly expanding on the Reptile (Nichol et al., 2018) algorithm.
We introduce a novel regularization technique called meta-step gradient pruning and also investigate the effects of increasing the depth of network architectures in first-order meta-learning.
We present an empirical evaluation of both approaches, where we match benchmark few-shot image classification results with 10 times fewer iterations using Mini-ImageNet dataset and with the use of deeper networks, we attain accuracies that surpass the current benchmarks of few-shot image classification using Omniglot dataset.

assistant: text: The paper presents an empirical study of the first-order meta-learning Reptile algorithm, investigating a proposed regularization technique and deeper networks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Perturbation Analysis of Input Transformations for Adversarial Attacks

Paper content:
The existence of adversarial examples, or intentional mis-predictions constructed from small changes to correctly predicted examples, is one of the most significant challenges in neural network research today.
Ironically, many new defenses are based on a simple observation - the adversarial inputs themselves are not robust and small perturbations to the attacking input often recover the desired prediction.
While the intuition is somewhat clear, a detailed understanding of this phenomenon is missing from the research literature.
This paper presents a comprehensive experimental analysis of when and why perturbation defenses work and potential mechanisms that could explain their effectiveness (or ineffectiveness) in different settings.

assistant: text: This paper discusses ways of destabilizing a given adversarial attack, what makes adversarial images non-robust, and if it's possible for attackers to use a universal model of perturbations to make their adversarial examples robust against such perturbations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Global Relational Models of Source Code

Paper content:
Models of code can learn distributed representations of a program's syntax and semantics to predict many non-trivial properties of a program.
Recent state-of-the-art models leverage highly structured representations of programs, such as trees, graphs and paths therein (e.g. data-flow relations), which are precise and abundantly available for code.
This provides a strong inductive bias towards semantically meaningful relations, yielding more generalizable representations than classical sequence-based models.
Unfortunately, these models primarily rely on graph-based message passing to represent relations in code, which makes them de facto local due to the high cost of message-passing steps, quite in contrast to modern, global sequence-based models, such as the Transformer.
In this work, we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias: Graph Sandwiches, which wrap traditional (gated) graph message-passing layers in sequential message-passing layers; and Graph Relational Embedding Attention Transformers (GREAT for short), which bias traditional Transformers with relational information from graph edge types.
By studying a popular, non-trivial program repair task, variable-misuse identification, we explore the relative merits of traditional and hybrid model families for code representation.
Starting with a  graph-based model that already improves upon the prior state-of-the-art for this task by 20%, we show that our proposed hybrid models improve an additional 10-15%, while training both faster and using fewer parameters.

assistant: text: A new method to model the source code for the bug repairing task using a sandwich model like [RNN GNN RNN] which significantly improves localization and repair accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Show, Attend and Translate: Unsupervised Image Translation with Self-Regularization and Attention

Paper content:
Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain.
It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training.
When paired training data is not accessible, image translation becomes an ill-posed problem.
We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term.
We further notice that existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input.
Thus we propose to add an attention module to predict an attention map to guide the image translation process.
The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes.
The predicted attention map also opens door to applications such as unsupervised segmentation and saliency detection.
Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.

assistant: text: We propose a simple generative model for unsupervised image translation and saliency detection.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Curiosity-driven Exploration by Bootstrapping Features

Paper content:
We introduce CBF, an exploration method that works in the absence of rewards or end of episode signal.
CBF is based on intrinsic reward derived from the error of a dynamics model operating in feature space.
It was inspired by (Pathak et al., 2017), is easy to implement, and can achieve results such as passing four levels of Super Mario Bros, navigating VizDoom mazes and passing two levels of SpaceInvaders.
We investigated the effect of combining the method with several auxiliary tasks, but find inconsistent improvements over the CBF baseline.

assistant: text: A simple intrinsic motivation method using forward dynamics model error in feature space of the policy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SpaMHMM: Sparse Mixture of Hidden Markov Models for Graph Connected Entities

Paper content:
We propose a framework to model the distribution of sequential data coming from
a set of entities connected in a graph with a known topology.
The method is
based on a mixture of shared hidden Markov models (HMMs), which are trained
in order to exploit the knowledge of the graph structure and in such a way that the
obtained mixtures tend to be sparse.
Experiments in different application domains
demonstrate the effectiveness and versatility of the method.

assistant: text: The authors propose a method to model sequential data from multiple interconnected sources using a mixture of common pool of HMM's.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets

Paper content:
Adaptive gradient algorithms perform gradient-based updates using the history of gradients and are ubiquitous in training deep neural networks.
While adaptive gradient methods theory is well understood for minimization problems, the underlying factors driving their empirical success in min-max problems such as GANs remain unclear.
In this paper, we aim at bridging  this gap from both theoretical and empirical perspectives.
First, we analyze a variant of Optimistic Stochastic Gradient (OSG) proposed in~\citep{daskalakis2017training} for solving a class of non-convex non-concave min-max problem and establish $O(\epsilon^{-4})$ complexity for finding $\epsilon$-first-order stationary point, in which the algorithm only requires invoking one stochastic first-order oracle while enjoying state-of-the-art iteration complexity achieved by stochastic extragradient method by~\citep{iusem2017extragradient}.
Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $\widetilde{O}\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$~\footnote{Here $\widetilde{O}(\cdot)$ compresses a logarithmic factor of $\epsilon$.
}, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.
To the best of our knowledge, this is the first work for establishing adaptive complexity in non-convex non-concave min-max optimization.
Empirically, our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training.
Moreover, this observation can be explained by the slow growth rate of the cumulative stochastic gradient, as observed empirically.

assistant: text: This paper provides novel analysis of adaptive gradient algorithms for solving non-convex non-concave min-max problems as GANs, and explains the reason why adaptive gradient methods outperform its non-adaptive counterparts by empirical studies.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Skip Connections Eliminate Singularities

Paper content:
Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures.
A completely satisfactory explanation for their success remains elusive.
Here, we present a novel explanation for the benefits of skip connections in training very deep networks.
The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model.
Several such singularities have been identified in previous works:
(i) overlap singularities caused by the permutation symmetry of nodes in a given layer,
(ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes,
(iii) singularities generated by the linear dependence of the nodes.
These singularities cause degenerate manifolds in the loss landscape that slow down learning.
We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent.
Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down.
These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

assistant: text: Paper examines the use of skip connections in deep networks as a way of alleviating singularities in the Hessian matrix during training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Riemannian Stochastic Gradient Descent for Tensor-Train Recurrent Neural Networks

Paper content:
The Tensor-Train factorization (TTF) is an efficient way to compress large weight matrices of fully-connected layers and recurrent layers in recurrent neural networks (RNNs).
However, high Tensor-Train ranks for all the core tensors of parameters need to be element-wise fixed, which results in an unnecessary redundancy of model parameters.
This work applies Riemannian stochastic gradient descent (RSGD) to train core tensors of parameters in the Riemannian Manifold before finding vectors of lower Tensor-Train ranks for parameters.
The paper first presents the RSGD algorithm with a convergence analysis and then tests it on more advanced Tensor-Train RNNs such as bi-directional GRU/LSTM and Encoder-Decoder RNNs with a Tensor-Train attention model.
The experiments on digit recognition and machine translation tasks suggest the effectiveness of the RSGD algorithm for Tensor-Train RNNs.

assistant: text: Proposes an algorithm for optimizing neural networks parametrized by Tensor Train decomposition based on the Riemannian optimization and rank adaptation, and designs a bidirectional TT LSTM architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Imbalanced Classification via Adversarial Minority Over-sampling

Paper content:
In most real-world scenarios, training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.
In this paper, we explore a novel yet simple way to alleviate this issue via synthesizing less-frequent classes with adversarial examples of other classes.
Surprisingly, we found this counter-intuitive method can effectively learn generalizable features of minority classes by transferring and leveraging the diversity of the majority information.
Our experimental results on various types of class-imbalanced datasets in image classification and natural language processing show that the proposed method not only improves the generalization of minority classes significantly compared to other re-sampling or re-weighting methods, but also surpasses other methods of state-of-art level for the class-imbalanced classification.

assistant: text: We develop a new method for imbalanced classification using adversarial examples
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Prediction Under Uncertainty with Error Encoding Networks

Paper content:
In this work we introduce a new framework for performing temporal predictions
in the presence of uncertainty.
It is based on a simple idea of disentangling com-
ponents of the future state which are predictable from those which are inherently
unpredictable, and encoding the unpredictable components into a low-dimensional
latent variable which is fed into the forward model.
Our method uses a simple su-
pervised training objective which is fast and easy to train.
We evaluate it in the
context of video prediction on multiple datasets and show that it is able to consi-
tently generate diverse predictions without the need for alternating minimization
over a latent space or adversarial training.

assistant: text: This paper introduces a times-series prediction model that that learns a deterministic mapping and trains another net to predict future frames given the input and residual error from the first network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Compressing U-net Using Knowledge Distillation

Paper content:
We study the use of knowledge distillation to compress the U-net architecture.
We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves the training process.
This allows us to compress a U-net by over 1000x, i.e., to 0.1% of its original number of parameters, at a negligible decrease in performance.
where a y t ij indicates the label of sample t at location (i, j), andŷ t is the probability map predicted 41 by the network.
In the standard setting, this probability map is obtained via the softmax function.
to generate probabilitiesŷ t * for each sample x t in a validation set.
These probabilities are then H(y t ,ŷ t ), for mixed distillation.
After being trained at T > 1, the softmax temperature of the 48 student network is reduced back to 1.
Improving Distillation.
As evidenced by our results in Section 3, using standard distillation, whether 50 vanilla or mixed, did not prove sufficient to distill a standard U-net into a very small one.
To overcome 51 this, we therefore propose two modifications of the original strategy.
First, as indicated in FIG0

assistant: text: The authors introduced a modified distillation strategy to compress a U-net architecture by over 1000x while retaining an accuracy close to the original U-net.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Approximation capability of neural networks on sets of probability measures and tree-structured data

Paper content:
This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.
By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.
The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.
The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.
This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.
This work has been motivated by recently proposed solutions to multi-instance learning BID28 ; BID19 ; BID5 and by mean-map embedding of probability measures BID23 .
It generalizes the universal approximation theorem of neural networks to compact sets of probability measures over compact subsets of Euclidean spaces.
Therefore, it can be seen as an adaptation of the mean-map framework to the world of neural networks, which is important for comparing probability measures and for multi-instance learning, and it proves the soundness of the constructions of BID19 ; BID5 .The
universal approximation theorem is extended to inputs with a tree schema (structure) which, being the basis of many data exchange formats like JSON, XML, ProtoBuffer, Avro, etc., are nowadays ubiquitous. This
theoretically justifies applications of (MIL) neural networks in this setting.As the presented proof relies on the Stone-Weierstrass theorem, it restricts non-linear functions in neural networks to be continuous in all but the last non-linear layer. Although
this does not have an impact on practical applications (all commonly use nonlinear functions within neural networks are continuous) it would be interesting to generalize the result to non-continuous non-linearities, as has been done for feed-forward neural networks in BID14 .

assistant: text: This paper generalizes the universal approximation theorem to real functions on the space of measures.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks

Paper content:
Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime,we introduce the Unrestricted Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent.
We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of differ-ent depths and widths) can emerge dynamically from the same URN.These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network.
We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.

assistant: text: We introduce a network framework which can modify its structure during training and show that it can converge to various ML network archetypes such as MLPs and LCNs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition

Paper content:
In this paper we study the problem of learning the weights of a deep convolutional neural network.
We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.
We develop an algorithm for simultaneously learning all the kernels from the training data.
Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.
We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.
We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.
Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

assistant: text: This paper proposes a learning method for a restricted case of deep convolutional networks, where the layers are limited to the non-overlapping case and have only one output channel per layer
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bounding and Counting Linear Regions of Deep Neural Networks

Paper content:
In this paper, we study the representational power of deep neural networks (DNN) that belong to the family of piecewise-linear (PWL) functions, based on PWL activation units such as rectifier or maxout.
We investigate the complexity of such networks by studying the number of linear regions of the PWL function.
Typically, a PWL function from a DNN can be seen as a large family of linear functions acting on millions of such regions.
We directly build upon the work of Mont´ufar et al. (2014), Mont´ufar (2017), and Raghu et al. (2017) by refining the upper and lower bounds on the number of linear regions for rectified and maxout networks.
In addition to achieving tighter bounds, we also develop a novel method to perform exact numeration or counting of the number of linear regions with a mixed-integer linear formulation that maps the input space to output.
We use this new capability to visualize how the number of linear regions change while training DNNs.

assistant: text: We empirically count the number of linear regions of rectifier networks and refine upper and lower bounds.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discriminative Particle Filter Reinforcement Learning for Complex Partial observations

Paper content:
Deep reinforcement learning has succeeded in sophisticated games such as Atari, Go, etc.
Real-world decision making, however, often requires reasoning with partial information extracted from complex visual observations.
This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for partial and complex observations.
DPFRL encodes a differentiable particle filter with learned transition and observation models in a neural network, which allows for reasoning with partial observations over multiple time steps.
While a standard particle filter relies on a generative observation model, DPFRL learns a discriminatively parameterized model that is training directly for decision making.
We show that the discriminative parameterization results in significantly improved performance, especially for tasks with complex visual observations, because it circumvents the difficulty of modelling observations explicitly.
In most cases, DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark that we introduce.
We further show that DPFRL performs well for visual navigation with real-world data.

assistant: text: This paper introduces a principled method for POMDP RL: Discriminative Particle Filter Reinforcement Learning that allows for reasoning with partial observations over multiple time steps, achieving state-of-the-art on benchmarks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Boosting Generative Models by Leveraging Cascaded Meta-Models

Paper content:
A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios.
However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures.
In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model.
Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation.
We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily.
We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework.

assistant: text: Propose an approach for boosting generative models by cascading hidden variable models
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Tensorized Embedding Layers for Efficient Model Compression

Paper content:
The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.
However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.
We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance.
We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

assistant: text: Embedding layers are factorized with Tensor Train decomposition to reduce their memory footprint.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Neural Representation of Sketch Drawings

Paper content:
We present sketch-rnn, a recurrent neural network able to construct stroke-based drawings of common objects.
The model is trained on a dataset of human-drawn images representing many different classes.
We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

assistant: text: This paper introduces a neural network architecture for generating sketch drawings inspired by the variational autoencoder.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Joint autoencoders: a flexible meta-learning framework

Paper content:
The incorporation of prior knowledge into learning is essential in achieving good performance based on small noisy samples.
Such knowledge is often incorporated through the availability of related data arising from domains and tasks similar to the one of current interest.
Ideally one would like to allow both the data for the current task and for previous related tasks to self-organize the learning system in such a way that commonalities and differences between the tasks are learned in a data-driven fashion.
We develop a framework for learning multiple tasks simultaneously, based on sharing features that are common to all tasks, achieved through the use of a modular deep feedforward neural network consisting of shared branches, dealing with the common features of all tasks, and private branches, learning the specific unique aspects of each task.
Once an appropriate weight sharing architecture has been established, learning takes place through standard algorithms for feedforward networks, e.g., stochastic gradient descent and its variations.
The method deals with meta-learning (such as domain adaptation, transfer and multi-task learning) in a unified fashion, and can easily deal with data arising from different types of sources.
Numerical experiments demonstrate the effectiveness of learning in domain adaptation and transfer learning setups, and provide evidence for the flexible and task-oriented representations arising in the network.

assistant: text: The paper focuses on learning common features from multiple domains data and ends up with a general architecture for multi-task, semi-supervised and transfer learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Cakewalk Method

Paper content:
Combinatorial optimization is a common theme in computer science.
While in general such problems are NP-Hard, from a practical point of view, locally optimal solutions can be useful.
In some combinatorial problems however, it can be hard to define meaningful solution neighborhoods that connect large portions of the search space, thus hindering methods that search this space directly.
We suggest to circumvent such cases by utilizing a policy gradient algorithm that transforms the problem to the continuous domain, and to optimize a new surrogate objective that renders the former as generic stochastic optimizer.
This is achieved by producing a surrogate objective whose distribution is fixed and predetermined, thus removing the need to fine-tune various hyper-parameters in a case by case manner.
Since we are interested in methods which can successfully recover locally optimal solutions, we use the problem of finding locally maximal cliques as a challenging experimental benchmark, and we report results on a large dataset of graphs that is designed to test clique finding algorithms.
Notably, we show in this benchmark that fixing the distribution of the surrogate is key to consistently recovering locally optimal solutions, and that our surrogate objective leads to an algorithm that outperforms other methods we have tested in a number of measures.

assistant: text: A new policy gradient algorithm designed to approach black-box combinatorial optimization problems. The algorithm relies only on function evaluations, and returns locally optimal solutions with high probability.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Critical initialisation in continuous approximations of binary neural networks

Paper content:
The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated.
We derive, using mean field theory, a set of scalar equations describing how input signals propagate through surrogate networks.
The equations reveal that depending on the choice of surrogate model, the networks may or may not exhibit an order to chaos transition, and the presence of depth scales that limit the maximum trainable depth.
Specifically, in solving the equations for edge of chaos conditions, we show that surrogates derived using the Gaussian local reparameterisation trick have no critical initialisation, whereas a deterministic surrogates based on analytic Gaussian integration do.
The theory is applied to a range of binary neuron and weight design choices, such as different neuron noise models, allowing the categorisation of algorithms in terms of their behaviour at initialisation.
Moreover, we predict theoretically and confirm numerically, that common weight initialization schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance.
This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to close to $\pm 1$ for deeper networks to be trainable.

assistant: text: signal propagation theory applied to continuous surrogates of binary nets;  counter intuitive initialisation; reparameterisation trick not helpful