Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Authors improve upon dynamic routing between capsules by removing the squash function (norm normalization) and apply a layerNorm normalization instead. Furthermore, they experiment with concurrent routing rather than sequential routing (route all caps layers once, then all layers concurrently again and again). This is an interesting development since provides better gradient in conjunction with layerNorm. They report results on Cifar10 and Cifar100 and achieve similar to CNN (resnet) performance.

First, I want to point out that inverted attention is exactly what happens in dynamic routing (sabour et al 2017), proc. 1 line 4,5, and 7. In dynamic routing the dot product with the next layer capsule is calculated and then normalized over all next layer capsules. The only difference that I notice between alg. 1 here and proc. 1 there is replacement of squash with layer norm. There is no "reconstructing the layer bellow" in Dynamic routing as authors suggest in intro. 

Second, the Capsules are promised to have better viewpoint generalizability than CNNs while having comparable performance. Replacing the 1 convolution layer with a ResNet backbone and replacing the activation with a classifier on top seems reducing the proposed CapsNet to the level of CNNs in terms of Viewpoint Generalization. Why should someone use this network rather than the ResNet itself? Fewer number of parameters by itself is not interesting, the reason it is reported usually is that it indicates lower memory consumption or fewer flops. Is that the case when comparing the baseline ResNet with the proposed CapsNet? Otherwise, a set of experiments showcasing the viewpoint generalizability of proposed CapsuleNetworks might only justify the switch between resnets to the proposed capsnets.

Thirdly, Fig. 4 top images seems to indicate all 3 routing procedures are following the same Learning Rate schedule. In the text it is said that optimization hyperparameters are tuned individually. Did authors tune learning rate schedule individually?

Forth, the proper baseline for the current study is the dynamic routing CapsNet. Why the multiMNIST experiment lacks comparison with dynamic routing capsnet?

For the reasons above, the manuscript in its current format is not ready for publication.

------------------------------------------------------rebuttal
Thank you for your response. I acknowledged the novel contributions of this work. My comment was that some claims in the paper are not right. i.e. "inverted dot-product attention" is not new and "reconstructing the layer bellow" does not happen in Sabour et al . Parallel execution + layer norm definitely is novel and significant.

Regarding the LR-schedule, I am not sure how fair it is to use same hyper-params tuned for the proposed method on the baselines. 

Regarding the viewpoint, the diverseMultiMNIST is two over lapping MNIST digits shifted 6 pixels. There is no rotation or scale in this dataset. An example experiment verifying the viewpoint generalizability of the proposed model is training on MNIST testing on AFFNIST. 

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: 
This paper presents a new simpler routing mechanism for capsule networks and achieves good performance on real world data sets making use of this new capsule structure along with a restnet backbone. Strong performance on the cifar10 and cifar100 datasets are presented and the network outperforms earlier versions of capsule networks. This new structure also performs well on an augmented MNIST dataset of overlapping digits (similar to the one used by Sabour et al 2017). 

Overall the paper is well written and presents solid results. The paper also presents a thorough comparison of two earlier versions of capsules which is a worthwhile contribution in its own right.

The paper could be improved by clearing up a few ambiguities:

- is the learning rate schedule the same for all three models? in figure 4 it looks like the learning rate is decayed at two distinct points for your model, but only one distinct point for both the EM and Dynamic routing models.  

-"Notably, the prediction becomes random guess when the iteration number increases to 5." this sentence is a little confusing. Do you mean when the iteration number the performance is equivalent to not random assignments?  

- This new algorithm requires that the capsules in L+1 have initialized poses with which to compare agreement between the poses in L. This is initial value seems like it may greatly effect the performance of the model. In the paper it is set to 0 and not expanded upon. It would be interesting to see if randomizing this value, or learning a bias for it would effect performance.   

-unlike the two previous versions of capsules, the inverted dot product capsules show in figure 4 sudden huge decreases in test accuracy while training. These moments seem to be overcome quite quickly and the model ends up outperforming the other two. But it would be worth mentioning this behavior and perhaps attempting to explain it.

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a simple and effective routing algorithm for capsule networks. The paper is well written. A nice analysis of the proposed routing algorithm is provided. Experiments of varying the routing iterations demonstrate the stableability of proposed routing algorithm compared to others.

Here are some issues:
1. Would the authors release the code for reproducing the results in the paper? It will be helpful for future research in this area.

2. In Fig.5, it would be better to give some brief explanations about why CasNet (Matrix) occupies much more memory while possessing less parameters.
review_title: Official Blind Review #1

Meta-review: This work presents a routing algorithm for capsule networks, and demonstrates empirical evaluation on CIFAR-10 and CIFAR-100. The results outperform existing capsule networks and are at-par with CNNs. Reviewers appreciated the novelty, introducing a new simpler routing mechanism, and achieving good performance on real world datasets. In particular, removing the squash function and experimenting with concurrent routing was highlighted as significant progress. There were some concerns (e.g. claiming novelty for inverted dot-product attention) and clarification questions (e.g. same learning rate schedule for all models). The authors provided a response and revised the submission , which addresses most of these concerns. At the end, majority of reviewers recommended accept. Alongside with them, I acknowledge the novelty of using layer norm and parallel execution, and recommend accept.

Decision: Accept (Poster)