Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The submission details a novel technique to learn the routing in capsule networks for image classification tasks. Connecting capsules in such architectures typically requires iterative approaches which are computationally expensive. The main idea in this submission is to leverage a non-iterative attention mechanism to learn this routing and thus decreases computational cost. Furthermore, the experiments indicate that the proposed architecture leads to higher accuracy on a number of image classification tasks and datasets.

Overall, the submission is well written and easy to follow. The discussion of related work is thorough and to my best knowledge appears to be complete.

---
## Pros:

   * The paper clearly presents the proposed method, and it is easy to follow. 
   * The results show significant improvements over the baselines, especially on the Tiny-ImageNet dataset and the SmallNORB dataset in the case of novel poses.
   * The proposed learning algorithm does not require expensive iteration and thus allows for better scaling.  
   * The good results across several datasets suggest that the proposed model can be an important contribution to the field.

--- 
## Cons:

   * The authors discussion of related work mentions several papers that combine CapsNets with an attention mechanisms to address the issue of routing. 
   * However, the authors only provide a comparison to Hahn et al. (2019). The authors state that other methods are memory intensive and require knowledge of the number of concurrent iterations as an additional hyperparameter, which is probably why they do not compare to these methods. 
   * Although an important property, it would be beneficial to provide a comparison to other attention-based CapsNets. 
   * Furthermore, it would be interesting to compare to state-of-the-art image classification methods that are not based on the capsules concept. Adding these additional experiments would help readers to have a better overview of the task and where the proposed approach stands in the broader context of classification approaches. 
   * In other areas of deep-learning and representation learning in particular, use of GMMs in the architecture, e.g., in latent space, (as opposed to usage as a probabilistic output model) can lead to code-book collapse. I.e., it is hard to balance during pure back prop learning how to pick the correct gaussian to sample from and to shape it  (or in other words: should one update the parameters of a mixture component or penalize the selection of that component if the action leads to a high loss in the forward pass). It would be interesting to see more details on this aspect of the architecture and training. I.e. how was the number of mixtures selected, how does this influence performance? etc. 

--- 
Summary:
Overall, there is much to like about this submission but some questions remain (see cons). I'm looking forward to the authors' response.  
review_title: Review: Trans-Caps: Transformer Capsule Networks with Self-attention Routing

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to use the self-attention to find the agreement among the capsules of consecutive layers of a capsule network instead of iterative routing procedure. To reduce the computational and memory complexity of self-transformer capsules of each layer are considered as a set and follow set-transformer and apply inducing mixture of Gaussian distributions to compute the agreement among capsules of layer L and L+1. 
The proposed method have been evaluated on multiple dataset including large scale datasets like ImageNet and improved the performance compared to the baselines (except for  ImageNet)
pros:
1- The idea is interesting and the paper is well written and easy to follow.
2- The proposed method seems to be scalable to the large datasets. 
3- It achieves significant improvement compared to the baselines on the novel view point of SmallNorb.

cons(clarification)
1- There are related papers that are missing in the comparison section for instance 
-Capsule routing via variational Bayes.
2- Why is the performance the proposed method on the ImageNet is even less than the baseline Resnet50. 
3- It seems that increasing the number of attention head negatively affect the performance, what is your justification for that?
4- Results in the table 3 is counterintuitive. The proposed method has a significant improvement compare to the baseline on the novel view point of SmallNorb while it is on par with other method in familiar viewpoint.

Post Rebuttal

Thanks for the author(s)' responses. The rebuttal addressed some of my questions. I have a couple of  suggestions : 
1- Your proposed capsule network is not the first one that is applicable on large scale datasets like ImageNet, there are other capsule networks that are applicable on real world scenarios and also ImageNet dataset with improvements over the baseline 
- Dual Directed Capsule Network for Very Low Resolution Image Recognition (ICCV 2019)
-  Subspace Capsule Network (AAAI 2020)
please refer to them and also give intuitions about why your proposed Trans-Caps is not performing well on ImageNet. The intuition and analysis is valuable to the community.  

2- To support the generalizability claim of Trans-Caps, I highly recommend reporting results on Multi-MNIST and also affNIST. Specially when you train the model on MNIST and test it on these two datasets. 

review_title: The paper proposes a self-attention (learning based) routing method that improve the performance over the baselines but there are a couple of unjustified points that need clarification.

final_score: rating: 7: Good paper, accept
review_content: Authors propose a novel Capsule connection which is inspired by set transformer and induced points. They introduce log-likelihood based attention based on Gaussians centered at trainable fixed queries. They calculate the routing factors (attention) using the probability of a key projection of votes under trainable gaussians. Then they add the values weighted by the routing factor to the fixed mean. Their choice of modifying the mean based on the input rather than replacing it is quite interesting. It resonates with the concept of momentum as well. I am curious how much it affects the stability and convergence of their technique. 
Afterwards they linearly transform and add a skip connection + relu to get the output capsule parameters.
In this work the activation probability of the capsules are not calculated alongside the pose matrices. For the sake of classification (which needs the activation probabilities) they have an extra fully connected layer + cross entropy.
Their method surprisingly can generalize to new viewpoints, backed by experiments on smallNorb azimuth and elevation generalization much better than the CNN and previous Capsule Networks. 
Also by removing the iterative routing and replacing it with trainable parameters they are able to achieve competitive results on Cifar10-Cifar100-tiny imagenet and imageNet. 

Pros:
The paper is very well written and easy to follow. They provide a convincing set of experiments on reasonable datasets. Their method is novel and intuitive.
Cons:
Lack of ablation study to show the importance of their novel method IMoG. One baseline is just the attention used by set transformer. Essentially what is the effect of having a Gaussian (standard deviation). Is it necessary to modify the mean vs replacing it. 
It would be more convincing to add attention based vision models (bello et al 2019) to the tables for cifar10-cifar100.



------------------------Post Author Response


Thank you for adding the ablation study and the attention based models. I enjoyed reading your work and it has answered some of the questions we wanted to explore in Capsule Networks. 
review_title: Novel non-iterative scalable Capsule Network which improves viewpoint generalization too

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes to swap the typical routing mechanisms in capsules for a more standard attention mechanism. The attention mechanism is based on computing similarity scores using gaussians instead of dot-products . The authors show that this leads to better downstream performance of more natural tasks while preserving robustness to viewpoint changes, one of the main strengths of capsules.

My main concern is that the proposed solution is not a good fit for the general framework of capsules. One of the key ideas behind capsuels is that objects are made up of parts which are uniquely assigned to exactly one object. This assumption is broken in this paper (if I understood the paper correctly). That is, the attention mechanism doesn't enforce any kind of competition between objects for individual parts. Hence, this model should be thought of as a stacked set transformer instead. The results indicate that this is not necessary for downstream performance (it actually helps) and the models still remain somewhat robust to view point changes on synthetic data (NORB).

== Detailed Comments and Questions ==

Where is the "self" in self-attention? IIUC there is no attention between individual states of one layer, not even indirectly. There is only attention between some learnable "object" states (inducing points) to their individual parts.

Why is it necessary to model the likelihood of a part state in this model, when all we care about is the similarity between an objects inducing vector and a part representation. Why not using just simple dot-product attention to compute the similarities? The paper says it is used to "encode the second order interactions among points". What does that mean? Actually, the log probability used as similarity score can be written as a dot-product (ignoring the norms of the vectors) + some bias over j which doesn't matter when computing the softmax later on.

I don't understand the difference to self-routing capsules (Hahn et al.). The paper says they use stationary routing weights to specific locations, but I don't see how this approach differs from that? The routing weight in Hahn et al. basically computes dot products between inducing points (the rows in the learnable routing weight matrix) and the output of the part capsule which are used as similarity scores. The only difference here is that gaussians are used to model similarity, which, as I explained above is pretty much the same as a dot-product. Something that's different is the use of multiple heads and using a softmax over parts instead of over objects, which brings me to my biggest concern.

I am also not sure that this architecture fits well into the Capsules framework. CapsNets make the assumption that each part belongs to a single object. That's also the reason why typically there needs to be some iterative procedure to compute probable assignments. Here, however, parts can be part of multiple objects, because the softmax is taken over the parts and there are multiple heads. This kind of defeats the purpose of capsules.

Ultimately, the architecture is basically a stacked version of the set transformer with some potentially interesting deviations, and it should be presented as such with the necessary ablations. I think it is interesting to see that "competition" between objects for the individual parts is not necessary to achieve similar or better performance to capsnets, to achieve good performance on down stream tasks.
review_title: Bringing attention into capsules breaks unique object-part relations.

Meta-review: The paper proposes a new variant of capsule networks, where iterative routing is replaced by an attention-based procedure inspired by Induced Set Attention from Set Transformers. The method is competitive on several classification benchmarks and improves generalization to unseen views on SmallNORB.

The reviewers note that the method is presented well (R2, R3, R4), is more scalable than other capsules variants (R3, R4), and the results are good (R1, R2, R3, R4). However, the reviewers also point out missing relevant baselines (R2, R3, R4), limited amount of generalization experiments (R3), and issues with the positioning of the method and the details of the formulation (R1). In particular, R1 did a very thorough job at reading the paper and discussing with the authors.  The issue with missing baselines has been satisfactorily addressed in the updated version of the paper.

Considering all this feedback and after reading the paper myself, I would summarize the pros and cons of the paper as follows.

Pros:
1. Good presentation
2. The method is more scalable than prior capsule-based models
3. Competitive results on several small- to mid-scale classification datasets
4. Good results on viewpoint generalization on SmallNORB 

Cons:
1. Classification results on all datasets are worse than non-capsules models (SE-ResNet, AA-ResNet). I could not find a discussion of this fact either in the paper, or in the authors’ responses. Given this fact, superior generalization (or some other nice properties) would be a potential advantage of the proposed model. Which leads to the next point.
2. Generalization results on SmallNORB are encouraging, but it is just a single dataset. If these results are key to showing the benefit of the method (as argued in the previous point), it is crucial to demonstrate this generalization in more settings, e.g. at least on MultiMNIST and AffNIST, as suggested by R3.
3. Scalability of the method is only studied in limited detail (I do appreciate Figure 2). The best indication in the direction of scalability is that the model can be trained on ImageNet (which is great), but it performs worse than the ResNet-50 used as a backbone and it is not explained why (even after one of the reviewers asked about it) and how expensive computationally the model is.
4. I share the concerns of R1 regarding the use of the term “MoG”. It is a mathematical term, so one would expect mathematical precision when using it.
  4a. It is unclear how the mixing probabilities \phi are learned (IIUC they get no gradient, as described by R1) and if they are in some way actually learned, it is unclear how it is guaranteed that they sum to one.
  4b. MoG usually comes with the standard procedure of fitting it to data (EM), which IIUC the authors are not following here. This should be clearly explained. 
5. A relatively more minor concern: again, as pointed out by R1, the use of “self-” in “self-attention” does not seem accurate. Self-attention assumes inputs to the attention procedure attend to themselves in some sense. As one consequence, the output sequence has the same length as the input sequence. ISAB from Set Transformer can be seen as a factorized version of self-attention where first inducing points attend to the inputs and then the inputs attend to the inducing points, so the output of the whole block is still the same length as the input. But in the proposed model this second step of going back to the inputs is absent and the length of the output sequence is generally different from the length of the input sequence.

Note:
I partially share the doubts R1 raised on the positioning of the method as “capsules” as opposed to “attention”, but I believe it is not the authors’ fault that the definition of what capsules are is historically vague and that this term has been used in many different ways in the past. I would strongly recommend to discuss this point in the updated version of the paper and I hope the capsules community manages to get more clarity on what exactly capsules are. But I do not count this point as a weakness here.

Based on all this evidence, I recommend rejection at this point. The paper has its merit, but it has unfortunate gaps both on the experimental and the presentation sides, as listed above. Some of these have been mentioned during the discussion phase, but the authors have not quite addressed them. There is no mechanism to ensure these are fixed in the final version, so resubmission to a different venue is the only option.

Decision: Reject