user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discovering the mechanics of hidden neurons

Paper content:
Neural networks trained through stochastic gradient descent (SGD) have been around for more than 30 years, but they still escape our understanding.
This paper takes an experimental approach, with a divide-and-conquer strategy in mind: we start by studying what happens in single neurons.
While being the core building block of deep neural networks, the way they encode information about the inputs and how such encodings emerge is still unknown.
We report experiments providing strong evidence that hidden neurons behave like binary classifiers during training and testing.
During training, analysis of the gradients reveals that a neuron separates two categories of inputs, which are impressively constant across training.
During testing, we show that the fuzzy, binary partition described above embeds the core information used by the network for its prediction.
These observations bring to light some of the core internal mechanics of deep neural networks, and have the potential to guide the next theoretical and practical developments.
Deep neural networks are methods full of good surprises.
Today, to perform image classification, one can train a 100M parameters convolutional neural network (CNN) with 1M training examples.
Beyond raising questions about generalization (Zhang et al., 2017) , it appears that the classification models derived from those CNNs offer object detectors for free, simply by thresholding activation maps BID22 Zhou et al., 2015; BID5 .
The learned representations also appear to be universal enough to be re-used on new tasks even in an entirely different domain (e.g. from natural to medical images in BID10 ).
If memory or computation are bottlenecks, no problem, networks with binary weights and binary activations work just as well BID23 .
What characteristics of SGD trained neural networks allow these intriguing behaviour to emerge?Deep
neural networks also have their limitations. They
currently pose lots of difficulties with respect to continuous learning BID15 , robustness BID27 BID22 , or unsupervised learning BID6 . Are
there other good surprises to expect in those fields, or do those difficulties correspond to fundamental limitations of SGD trained deep neural networks?In order
to answer both questions, a better understanding of deep neural networks is definitely needed. Since the
intricate nature of the network hinders theoretical developments, we believe experiments offer a valuable alternative path to offer an insight into the key mechanisms supporting the success of neural networks, thereby paving the way both for future theoretical and practical developments. In other
words: analysing how something works helps understanding why it works, and gives ideas to make it work better.In particular, the workings of hidden neurons, while being the core building block of deep neural networks, are still a mystery. It is tempting
to associate hidden neurons to the detection of semantically relevant concepts. Accordingly, many
works studying neurons have focused on their interpretability. A common and generally
admitted conception consists in considering that they represent concepts with a level of abstraction that grows with the layer depth BID19 . This conception has been
supported by several works showing that intermediate feature maps in convolutional neural networks can be used to detect higher level objects through simple thresholding BID22 Zhou et al., 2015; BID5 . However, it is not clear
if these observations reflect the entire relevant information captured by that feature map, or, on the contrary, if this interpretation is ignoring important aspects of it. In other words, the complete
characterization of the way a neuron encodes information about the input remains unknown. Moreover, the dynamics of training
that lead to the encoding of information used by a neuron is -to our knowledge-unexplored. This paper uses an experimental approach
that advances the understanding of both these aspects of neurons. The main finding of our paper is the following
: the encodings and dynamics of a neuron can approximately be characterized by the behaviour of a binary classifier. More precisely:1. During training, we observe
that the sign of
the partial derivative of the loss with respect to the activation of a sample in a given neuron is impressively constant (except when the neuron is too far from the output layer). We observe experimentally that this leads a
neuron to push activation of samples either up, or down, partitioning the inputs in two categories of nearly equal size.2. During testing, quantization and binarization
experiments show that the fuzzy, binary partition observed in point 1. embeds the core information used by the network
for its predictions.This surprisingly simple behaviour has been observed across different layers, different networks and at different problem scales (MNIST, CIFAR-10 and ImageNet). It seems like hidden neurons have a clearly defined
behaviour that naturally emerges in neural networks trained with stochastic gradient descent. This behaviour has -to our knowledge-remained undiscovered
until now, and raises intriguing questions to address in future investigations.
In this paper, we try to validate an ambitious hypothesis describing the behaviour of a neuron in a neural network during training and testing.
Our hypothesis is surprisingly simple: a neuron behaves like a binary classifier, separating two categories of inputs.
The categories, of nearly equal size, are provided by the backpropagated gradients and are impressively consistent during training for layers close enough to the output.
While stronger validation is needed, our current experiments, ran on networks of different depths and widths, all validate this behaviour.Our results have direct implications on the interpretability of neurons.
Studies analysing interpretability focused on the highest activations, e.g. above the 99.5 percentile in BID5 .
While these activations are the ones who are the most clearly discriminated by the neuron, we show that they do not reflect the complete behaviour of the neuron at all.
Our experiments reveal that neurons tend to consistently learn concepts that distinguish half of the observed samples, which is fundamentally different.We expect that our observations stimulate further investigations in a number of intriguing research directions disclosed by our analysis.Firstly, since our analysis observes (in FIG2 but does not explain the binary behaviour of neurons in the first layers of a very deep network, it would be interesting to investigate further the regularity of gradients (cfr. Section 4.1), in layers far from the output.
This could potentially unveil simple training dynamics which are currently hidden by noise or, on the contrary, reveal that the unstable nature of the backpropagated gradients is a fundamental ingredient supporting the convergence of first layer neurons.
Ultimately, these results would provide the missing link for a complete characterization of training dynamics in deep networks.
Secondly, our work offers a new perspective on the role of activation functions.
Their current motivation is that adding non-linearities increases the expressivity of the network.
This, however, does not explain why one particular non-linearity is better than another.
Our lack of understanding of the role of activation functions heavily limits our ability to design them.
Our results suggest a local and precise role for activation functions: promoting and facilitating the emergence of a binary encoding in neurons.
This could be translated in activation functions with a forward pass consisting of well-positioned binarization thresholds, and a backward pass that takes into account how well a sample is partitioned locally, at the neuron level.Finally, we believe that our work provides a new angle of attack for the puzzle of the generalization gap observed in Zhang et al. (2017) .
Indeed, combining our observations with the works on neuron interpretability tells us that a neuron, while not able to finish its partitioning before convergence, seems to prioritize samples with common patterns (cfr. Figure 2) .
This prioritization effect during training has already been observed indirectly in BID3 , and we are now Figure 4: Sliding window binarization experiment: pre-activations inside a window with a width of percentile rank 10 are mapped to 1, pre-activations outside of it to 0.
Information that remains in the signal is only the fact that the pre-activation was inside or outside the window.
Observing if a new network can use this information for classification reveals structure about the encoding: which window positions provide the most important information for a classifier?
The results show a clear pattern across all layers and networks that confirms an encoding based on a fuzzy, binary partition of the inputs in two categories of nearly equal size.
As detailed in Section 3, the layers from the first two rows are part of a network trained on MNIST (with ReLU and sigmoid activation functions respectively), the third and fourth row on CIFAR-10 (with ReLU and no activation function respectively) and the fifth row on ImageNet (with ReLU activation).able
to localize and study it in depth. The
dynamics behind this prioritization between samples of a same category should provide insights about the generalization puzzle. While
most previous works have focused on the width of local minima BID16 , the regularity of the gradients and the prioritization effect suggest that the slope leading to it also matters: local minima with good generalization abilities are stronger attractors and are reached more rapidly.
Two main lessons emerge from our original experimental investigation.The first one arises from the observation that the sign of the loss function partial derivative with respect to the activation of a specific sample is constant along training for the neurons that are sufficiently close to the output, and states that those neurons simply aim at partitioning samples with positive/negative partial derivative sign.The second one builds on two experiments that challenge the partitioning behaviour of neurons in all network layers, and concludes that, as long as it separates large and small pre-activations, a binarization of the neuron's pre-activations in an arbitrary layer preserves most of the information embedded in this layer about the network task.As a main outcome, rather than supporting definitive conclusions, the unique observations made in our paper raise a number of intriguing and potentially very important questions about network learning capabilities.
Those include questions related to the convergence of first layer neurons in presence of noisy/unstable partial derivatives, the design of activation functions, and the generalization puzzle.

assistant: text: We report experiments providing strong evidence that a neuron behaves like a binary classifier during training and testing