Title: Matrix capsules with EM routing
Authors: Geoffrey E Hinton
Sara Sabour
Nicholas Frosst
Venue: ICLR 2018
Keywords: Computer Vision
Deep Learning
Dynamic routing

Abstract: A capsule is a group of neurons whose outputs represent different properties of the same entity. Each layer in a capsule network contains many capsules. We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose). A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships. Each of these votes is weighted by an assignment coefficient. These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes. The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers. On the smallNORB benchmark, capsules reduce the number of test errors by 45\% compared to the state-of-the-art. Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.

TL;DR: Capsule networks with learned pose matrices and EM routing improves state of the art classification on smallNORB, improves generalizability to new view points, and white box adversarial robustness.  

Review discussion:
The paper proposes a novel architecture for capsule networks. Each capsule has a logistic unit representing the presence of an entity plus a 4x4 pose matrix representing the entity/viewer relationship. This new representation comes with a novel iterative routing scheme, based on the EM algorithm.
Evaluated on the SmallNORB dataset, the approach proves to be more accurate than previous work (beating also the recently proposed "routing-by-agreement" approach for capsule networks by Sabour et al.). It also generalizes well to new, unseen viewpoints and proves to be more robust to adversarial examples than traditional CNNs.
Capsule networks have recently gained attention from the community. The paper addresses important shortcomings exhibited by previous work (Sabour et al.), introducing a series of valuable technical novelties.
There are, however, some weaknesses. The proposed routing scheme is quite complex (involving an EM-based step at each layer); it's not fully clear how efficiently it can be performed / how scalable it is. Evaluation is performed on a small dataset for shape recognition; as noted in Sec. 6, the approach will need to be tested on larger, more challenging datasets. Clarity could be improved in some parts of the paper (e.g.: Sec. 1.1 may not be fully clear if the reader is not already familiar with (Sabour et al., 2017); the authors could give a better intuition about what is kept and what is discarded, and why, from that approach. Sec. 2: the sentence "this is incorrect because the transformation matrix..." could be elaborated more. V_{ih} in eq. 1 is defined only a few lines below; perhaps, defining the variables before the equations could improve clarity. Sec. 2.1 could be accompanied by mathematical formulation).
All in all, the paper brings an original contribution and will encourage further research / discussion on an important research question (how to effectively leverage knowledge about the part-whole relationships).
Other notes:
- There are a few typos (e.g. Sec. 1.2 "(Jaderberg et al. (2015)",  Sec. 2 "the the transformation", Sec. 4 "cetral crop" etc.).
- The authors could discuss in more detail why the approach does not show significant improvement on NORB with respect to the state of the art.
- The authors could provide more insights about why capsule gradients are smaller than CNN ones.
- It would be interesting to discuss how the network could potentially be adapted, in the future, to: 1. be more efficient 2. take into account other changes produced by viewpoint changes (pixel intensities, as noted in Sec. 1).
- In Sec, 4, the authors could provide more details about the network training.
- In Procedure 1, for indexing tensors and matrices it might be better to use a comma to separate dimensions (e.g. V_{:,c,:} instead of V_{:c:}).
Thank you for your detailed reading of the paper and suggestions!
As per your comments on the EM routing, we agree that it was not presented as best it could have been, and have added an appendix to present a gentle and thorough introduction to the free energy view of EM and the objective function which our routing operation minimizes. In response to the question about efficiency, we would like to draw your attention to the total number of arithmetic operations required for the routing procedure - each iteration of routing represents fewer arithmetic operations than a single layer feed forward pass, but due to architectural optimization decisions in tensorflow, our current capsule implementation is not as fast as it could be.
We agree that larger scale testing would ideal, but due to the aforementioned efficiency limitations were not able to include it in this paper.
In regards to your other comments we have done the following:
- To increase the clarity of the paper,  we have made several changes to the language used, and improved the mathematical notation.
- We have added section 2 which provides an intuitive explanation of capsules and makes clear when the routing occurs. We feel that improves the readers' ability to engage with the rest of the presented content. We also defined the variables and notation used in the rest of the paper more explicitly.
- We have expanded on the sentence "this is incorrect because the transformation matrix..." you mentioned which is now in the appendix.
- We have also made several changes to the nation and language throughout the paper to make it more comprehensible.
thank you for your feedback, and hope that we have addressed your comments to your satisfaction.
This paper proposes a new kind of capsules for CNN. The capsule contains a 4x4 pose matrix motivated by 3D geometric transformations describing the relationship between the viewer and the object (parts). An EM-type of algorithm is used to compute the routing.
The authors use the smallNORB dataset as an example. Since the scenes are simulated from different viewer angles, the pose matrix quite fits the motivation. It would be more beneficial to know if this kind of capsules is limited to the motivation or is general.
For example, the authors may consider reporting the results of the affNIST dataset where the digits undergo 2D affine transformations (in which case perhaps 3x3 pose matrices are enough?).
Minor: The arguments in line 5 of the procedure RM Routing(a,V) do not match those in line 1 of the procedure E-Step.
Section 2.1 (objective of EM) is unclear. The authors may want to explicitly write down the free energy function.
The section about robustness against adversarial attacks is interesting.
Overall the idea appears to be useful but needs more empirical validation (affNIST, ImageNet, etc).
thank you for the feedback! To address your comments we have done the following:
- To clarify the EM objective we have added an extended and thorough appendix which presents a gentle and intuitive explanation of the free energy view of EM, and explicit free energy function, and how our routing algorithm makes use of it.
- We believe that the benefit of capsules is not limited to smallNORB and will generalize. As suggested, we replicated the affNIST generalization experiment reported in the previous Capsule paper (Sabour et al. 2017). We found that our EM capsule model (the exact architecture used for smallNORB and MNIST in the paper), when trained to 0.8% test error on expanded MNIST (40x40 pixel MNIST images, created by padding and shifting MNIST), achieved 6.9% test error on affNIST. We trained a baseline CNN (with AlexNet architecture, without pooling) to 0.8% test error and it was only able to achieve 14.1% test error on affNIST. Our capsule model was able to half the test error of a CNN when trained on MNIST and tested on affNIST.  Due to time and space constraints these results are not reported in the paper as it is now.
- finally we address the minor issue raised in line 5 of the routing procedure.
we hope this has addressed your concerns, and thank you for your suggestions.
The paper describes another instantiation of "capsules" which attempt to learn part-whole relationships and the geometric pose transformations between them.  Results are presented on the smallNORB test set obtaining impressive performance.
Although I like very much this overall approach, this particular paper is so opaquely written that it is difficult to understand exactly what was done and how the network works.  It sounds like the main innovation here is using a 4x4 matrix for the pose parameters, and an iterative EM algorithm to find the correspondence between capsules (routing by agreement).
But what exactly the pose matrix represents, and how they get transformed from one layer to the next, is left almost entirely to the reader's imagination.
In addition, how EM factors in, what the probabilities P_ih represent, etc. is not clear.  I think the authors could do a much better job explaining this model, the rationale behind it, and how it works.
Perhaps the most interesting and compelling result is Figure 2, which shows how ambiguity in object class assignment is resolved with each iteration.  This is very intriguing, but it would be great to understand what is going on and how this is happening.
Although the results are impressive, if one can't understand how this was achieved it is hard to know what to make of it.
Thank you for your comments. upon reflection we agree that the paper was confusing and we have taken several steps to reduce the opacity of our work to the reader. To that end we have done the following:
- We have added section 2 which gives a general and intuitive explanation of the mechanism of capsule networks, paying close attention to how pose matrices get transformed from one layer to the next.
- Having identified the EM objective as another source of confusion, we added an extended appendix in which we provide a gentle and approachable explanation for the free energy view of EM and how our routing algorithm builds upon it.
- We have also added a paragraph to further explain figure 2 in the experiments section.
- Finally we have made several changes to the language of the paper, focusing in particular on the notation.
We believe that the comprehensibility of the paper has thus improved and appreciate your criticism.

Meta-review:
role: PC
data: comment: Authors present a new multi-layered capsule network architecture, implemented an EM routing procedure, and introduced "Coordinate Addition".  Capsule architectures are gaining interest because of their ability to achieve equivariance of parts, and employ a new form of pooling called "routing" (as opposed to max pooling) which groups parts that make similar predictions of the whole to which they belong, rather than relying on spatial co-locality. New state-of-art performances are being achieved on focused datasets, for which the authors have continued the trend.

Pros:
- New significant improvement to state-of-art performance is obtained on smallNORB, both in comparison to CNN structure as well as the most recent previous implementation of capsule network.

Cons:
- Some concern arose regarding the writing of the paper and the ability to understand the material, which authors have made an effort to address.

Given the general consensus of the reviewers that this work should be accepted, the general applicability of the technology to multiple domains, and the potential impact that improvements to capsule networks may have on an early field, area chair recommends this work be accepted as a poster presentation. 

Decision: Accept (Poster)