Title: Inductive Bias of Deep Convolutional Networks through Pooling Geometry
Authors: Nadav Cohen
Amnon Shashua
Venue: ICLR 2017
Keywords: Theory
Deep learning

Abstract: Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network's pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.

TL;DR: We study the ability of convolutional networks to model correlations among regions of their input, showing that this is controlled by shapes of pooling windows.

Review discussion:
Could the authors say more about the invariances of separation rank?
That is, under which transformations will two classification tasks have identical separation rank?
The separation rank is a property that characterizes multivariate functions.
As such, it does not directly apply to classification tasks.
However, one could reason (as we do in the paper) about the separation ranks required by a function in order for it to effectively solve a given classification task.  Reviewer's question can thus be rephrased as follows (please let us know if this was not the intent): Under which geometrical transformations of an image classification task, will the separation ranks required for accurate classification remain unchanged?
In its raw mathematical form, the answer to this question would be that any geometric permutation of local image patches leaves required separation ranks unchanged, as all we have to do is maintain the labeling of patches, and from a mathematical perspective, nothing really changes.  In order for the geometry to be relevant, one may consider a convoutional network, which supports a fixed set of separation ranks, and reason about its pooling geometry, which links these separation ranks to geometrical patterns partitioning the input.  In this viewpoint, we could rephrase asked question as follows: Under which geometrical transformations of an image classification task, will an accurately classifying convolutional network with a fixed pooling geometry remain accurate?
The latter is a very interesting question, which we did not address in the paper, and view as a potential avenue for future research (thank you!).  We can say at this point that there is no definitive answer, and it really depends on the particular pooling geometry being considered.
For example, square contiguous pooling supports high separation ranks between interleaved regions, and so as long as geometrical transformations are continuous, this pooling geometry is expected to remain valid.
On the other hand, "mirror pooling" (pooling nodes with their spatial reflections) supports high separation ranks under symmetric partitions, thus in this case, geometrical transformations that maintain symmetry, even if non-continuous, will most likely admit invariance.  We will consider addressing this issue more formally in future work.
Please let us know if the above has addressed your question.  Thank you!
This paper addresses the question of which functions are well suited to deep networks, as opposed to shallow networks.  The basic intuition is convincing and fairly straightforward.  Pooling operations bring together information.
When information is correlated, it can be more efficiently used if the geometry of pooling regions matches the correlations so that it can be brought together more efficiently.  Shallow networks without layers of localized pooling lack this mechanism to combine correlated information efficiently.
The theoretical results are focused on convolutional arithmetic circuits, building on prior theoretical results of the authors.  The results make use of the interesting technical notion of separability, which in some sense measures the degree to which a function can be represented as the composition of independent functions.  Because separability is measured relative to a partition of the input, it is an appropriate mechanism for measuring the complexity of functions relative to a particular geometry of pooling operations.  Many of the technical notions are pretty intuitive, although the tensor analysis is pretty terse and not easy to follow without knowledge of the authors’ prior work.
In some sense the comparison between deep and shallow networks is somewhat misleading, since the shallow networks lack a hierarchical pooling structure.
For example, a shallow convolutional network with RELU and max pooling does not really make sense, since the max occurs over the whole image.
So it seems that the paper is really more of an analysis of the effect of pooling vs. not having pooling.
For example, it is not clear that a deep CNN without pooling would be any more efficient than a shallow network, from this work.
It is not clear how much the theoretical results depend on the use of a model with product pooling, and how they might be extended to the more common max pooling.
Even if theoretical results are difficult to derive in this case, simple illustrative examples might be helpful.
In fact, if the authors prepare a longer version of the paper for a journal I think the results could be made more intuitive if they could add a simple toy example of a function that can be efficiently represented with a convolutional arithmetic circuit when the pooling structure fits the correlations, and perhaps showing also how this could be represented with a convolutional network with RELU and max pooling.
I would also appreciate a more explicit discussion of how the depth of a deep network affects the separability of functions that can be represented.  A shallow network doesn’t have local pooling, so the difference between deep and shallow if perhaps mostly one of pooling vs. not pooling.
However, practitioners find that very deep networks seem to be more effective than “deep” networks with only a few convolutional layers and pooling.  The paper does not explicitly discuss whether their results provide insight into this behavior.
Overall, I think that the paper attacks an important problem in an interesting way.  It is not so convincing that this really gets to the heart of why depth is so important, because of the theoretical limitation to arithmetic circuits, and because the comparison is to shallow networks that are without localized pooling.
We thank reviewer for the time invested and the elaborate feedback.  Our response follows.
We first wish to emphasize that the primary focus of our work is not the importance of depth. This topic was covered extensively in previous works.
In particular, [1] analyzed convolutional arithmetic circuits, comparing shallow (single hidden layer) and deep networks, followed by an extension to networks of intermediate depths. [2] then adapted the analysis to convolutional networks with ReLU activation and max or average pooling. In a nutshell, the works show that a function realized by a network of depth L1 can only be replicated by a shallower network of depth L2 if the latter's size is double exponential in (L1-L2).
Our focus in this work lies on a different question.
Although it has been proven that depth brings forth new functions to convolutional networks ([1],[2]), it was not explained why these functions are so successful in practice. We address this problem by analyzing the capability of a fixed deep network to model correlations among regions of its input, and show how this depends on network architecture (pooling geometry). The comparison to a shallow network is essentially a byproduct of our analysis. Given the treatment of intermediate-depth networks in [1], extending our analysis to such networks is completely straightforward - we did not include this merely for simplicity of presentation, and to avoid cluttering the main message of the paper.
With regards to interpretation of our analysis, some of its results are indeed intuitive, for example (as pointed out by reviewer) that a deep network can model higher correlations between input regions that are pooled together earlier.
However, such interpretations are not always valid.
For instance, if a shallow network pools all input regions at once and immediately, why aren't the correlations it models high? Why is the locality important? This requires rigorous proof, and the techniques we apply can be used to address a wide variety of related questions, in particular the behavior of correlations modeled by intermediate-depth networks (these are stronger than those of a shallow network, and weaker than those of a deep network).
Regarding the application of our analysis to convolutional rectifier networks (convolutional networks with ReLU activation and max/average pooling), that requires following a path similar to [2].
Specifically, the generalized notion of tensor decompositions established in [2] needs to be adopted, and in a similar vein, a generalized version of the separation rank requires definition. This is a technical step we intend to pursue in future work, so as to avoid further complicating the paper. To validate the applicability of our findings to convolutional rectifier networks, we include them in our experiments, and show that they exhibit the same trends observed with convolutional arithmetic circuits, in full compliance with our analysis.
As for the suggestion to include "toy" illustrations, we will definitely add such to the appendixes of the final manuscript. Thank you!
References
[1] Nadav Cohen, Or Sharir and Amnon Shashua. On the Expressive Power of Deep Learning: A Tensor Analysis. Conference on Learning Theory (COLT) 2016.
[2] Nadav Cohen and Amnon Shashua. Convolutional Rectifier Networks as Generalized Tensor Decompositions. International Conference on Machine Learning (ICML) 2016.
The paper provides a highly complex algebraic machinery to analyze the type of functions covered by convolutional network. As in most attempts  in this direction in the literature, the ideal networks described in paper, which have to be interpretable as polynomials over tensors, do not match the type of CNNs used in practice: for instance the Relu non-linearity is replaced with a product of linear functions (or a sum of logs).
While the paper is very technical to read, every concept is clearly stated and mathematical terminology properly introduced.
Still, I think some the authors could make some effort to make the key concepts more accessible, and give a more intuitive understanding of what the separation rank means rather before piling up different mathematical interpretation.
My SVM-era algebra is quite rusted, and I am not familiar with the separation rank framework: it would have been much easier for me to first fully understand a simple and gentle case (shallow network in section 5.3), than the general deep case.
To summarize my understanding of the key theorem 1 result:
- The upper bound of the separation rank is used to show that in the shallow case, this rank grows AT MOST linearly with the network size (as measured by the only hidden layer).
So exponential network sizes are caused by this rank needing to grow exponentially, as required by the partition.
- In the deep case, one also uses the case that the upper bound is linear in the size of the network (as measured by the last hidden layer), however, this situation is caused by the selection of a partition (I^low, J^high), and the maximal rank induced by this partition is only linear anyway, hence the network size can remain linear.
If tried my best to summarize the key point of this paper and still probably failed at it, which shows how complex is this notion of partition rank, and that its linear growth with network size can either be a good or bad thing depending on the setting. Hopefully, someone will come one day with an explanation that holds in a single slide.
While this is worth publishing as conference paper in its present form, I have two suggestions that, IMHO, would make this work more significant:
On the theory side, we are still very far from the completeness of the PAC bound papers of the "shallow era".
In particular, the non-probabilistic lower and upper bound in theorem 1 are probably loose, and there is no PAC-like theory to tell us which one to use and what is the predicted impact on performance (not just the intuition).
Also, in the prediction of the inductive bias, the other half is missing. This paper attempts to predict the maximal representation capacity of a DNN under bounded network size constraints, but one of the reason why this size has to be bounded is overfitting (justified by PAC or VC-dim like bounds). If we consider the expected risk as  the sum of the empirical risk and the structural risk, this paper only seems to address fully the empirical risk minimization part, freezing the structural risk.
On the practice side, an issue is that experiments in this paper mostly confirm what is obvious through intuition, or some simpler form of reasonings.
For instance to use convolutions that join pixels which are symmetrical in images to detect symmetry. Basic hand-crafted pattern detectors, as they have been used in computer vision for decades, would just do the job. What would be a great motivation for using this framework is if it answered questions that simple human intuition cannot, and for which we are still in the dark: one example I could think of in the recent use of gated convolutions 'a trous' for 1D speech signal, popularized in Google WaveNet (https://deepmind.com/blog/wavenet-generative-model-raw-audio/). Note that 1D inputs would also simplify the notation!
We thank reviewer for the time and feedback. Following is our response.
The key points of the paper are:
++ The separation rank of a multivariate function w.r.t. a partition of its input measures the correlation modeled between sides of the partition. If the separation rank is high, partition sides are highly correlated, and vice versa.
++ Ideally, we would like to be able to realize functions with high separation rank under any given partition, meaning we could model high correlation between arbitrary parts of the input. Unfortunately, as we show, convolutional networks of polynomial size do not facilitate this ability.
++ A shallow network can only realize separation ranks linear in its size, thus if the latter is polynomial, separation ranks are no more than polynomial, regardless of the considered input partition.
++ A deep network is stronger than a shallow one - with polynomial size it can realize exponential separation ranks for some partitions, while being limited to polynomial separation ranks for others. This means that some correlations can be modeled efficiently, while others cannot.
++ What determines which correlations a deep network can efficiently model is its pooling geometry.  Standard square windows support local correlations, as required for natural images, while other window shapes allow tailoring a network to different types of data.
The above points have been summarized several times throughout the paper - in the abstract, introduction and discussion. We hope they resolve misunderstandings which may have arose. Please let us know if not and we will further elaborate.
As reviewer points out, our work analyzes representational properties of convolutional networks, without treating aspects of generalization (structural risk). The expressive power of deep learning, and convolutional networks in particular, is believed to be the driving force behind their success. Given the limited formal understanding of the matter, we view related questions as topics of interest. Generalization properties of convolutional networks are also an important area of research, but are outside the scope of our current work.
The practical conclusions from our analysis are that pooling windows should be small (giving rise to a deep network), and that pixels pooled together early are more correlated than ones joined deeper down the network. Retroactively this may come across as intuitive, but unlike many deep learning practices, we ground these principles with rigorous theory. We are currently working on a follow-up paper in which we use our theory to design dilated pooling geometries, similarly to the dilated convolutions in [1]. We believe some of the results there would be less intuitive, but this is outside the scope of the current paper, whose main purpose is to lay down theoretical foundations for the effect of pooling geometry on the inductive bias of deep convolutional networks.
References
[1] van den Oord et al.  WaveNet: A Generative Model for Raw Audio.
This paper investigates the fact why deep networks perform well in practice and how modifying the geometry of pooling can make the polynomially
sized deep network to provide a function with exponentially high separation rank (for certain partitioning.)
In the authors' previous works, they showed the superiority of deep networks over shallows when the activation function is ReLu and the pooling is max/mean pooling but in the current paper there is no activation function after conv and the pooling is just a multiplication of the node values.
Although for the experimental results they've considered both scenarios.
Actually, the general reasoning for this problem is hard, therefore, this drawback is not significant and the current contribution adds a reasonable amount of knowledge to the literature.
This paper studies the convolutional arithmetic circuits and shows how this model can address the inductive biases and how pooling can adjust these biases.
This interesting contribution gives an intuition about how deep network can capture the correlation between the input variables when its size is polynomial but and correlation is exponential.
It worth to note that although the authors tried to express their notation and definitions carefully where they were very successful, it would be helpful if they elaborate a bit more on their definitions, expressions, and conclusions in the sense to make them more accessible.
We thank reviewer for the time and the supporting feedback.
We will include additional illustrations and explanations in the final version of the manuscript.
This paper investigates the fact why deep networks perform well in practice and how modifying the geometry of pooling can make the polynomially
sized deep network to provide a function with exponentially high separation rank (for certain partitioning.)
In the authors' previous works, they showed the superiority of deep networks over shallows when the activation function is ReLu and the pooling is max/mean pooling but in the current paper there is no activation function after conv and the pooling is just a multiplication of the node values.
Although for the experimental results they've considered both scenarios.
Actually, the general reasoning for this problem is hard, therefore, this drawback is not significant and the current contribution adds a reasonable amount of knowledge to the literature.
This paper studies the convolutional arithmetic circuits and shows how this model can address the inductive biases and how pooling can adjust these biases.
This interesting contribution gives an intuition about how deep network can capture the correlation between the input variables when its size is polynomial but and correlation is exponential.
It worth to note that although the authors tried to express their notation and definitions carefully where they were very successful, it would be helpful if they elaborate a bit more on their definitions, expressions, and conclusions in the sense to make them more accessible.

Meta-review:
role: PC
data: comment: The paper uses the notion of separation rank from tensor algebra to analyze the correlations induced through convolution and pooling operations. They show that deep networks have exponentially larger separation ranks compared to shallow ones, and thus, can induce a much richer correlation structure compared to shallow networks. It is argued that this rich inductive bias is crucial for empirical success.
 
 The paper is technically solid. The reviewers note this, and also make a few suggestions on how to make the paper more accessible. The authors have taken this into account. In order to bridge the gap between theory and practice, it is essential for theory papers to be accessible.
 
 The paper covers related work pretty well. One aspect is misses is the recent geometric analysis of deep learning. Can the algebraic analysis be connected to geometric analysis of deep learning, e.g. in the following paper?
 https://arxiv.org/abs/1606.05340

Decision: Accept (Poster)