Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes a neural architecture for summarizing trees inspired by capsule networks from computer vision. The authors re-use a tree convolution from previous work for the bottommost layer, and then propose adaptations to the dynamic routing from capsule networks so that it can be applied to variable-sized trees. The paper applies the proposed architecture to three different program classification datasets, which are in three different languages. The paper reports empirical gains compared to two architectures proposed by previous work.

I think that it's interesting to apply the capsule network architecture to tree classification, but unfortunately it doesn't appear that some of the motivation for capsule networks on images didn't seem to transfer neatly to this setting; for example, there is no equivalent of inverse graphics as there is no reconstruction loss (as pointed out by the authors in Section 6.4).

Also, the variable-to-static capsule routing indeed appears novel, but I was a bit confused by its internal details. It appears that the outputs of the previous layer which occur most often will get routed (considering lines 6-8 of Algorithm 1 which up-weights each of the $\hat{u}_i$ based on its similarity to $v_j$; the $v_j$ are initially a re-numbered subset of $\hat{u}_i$), without any prior transformation of the previous layer first. It seems to me that this doesn't allow for the prior layer to predict more complex features about the input that the subsequent layer is expected to capture. In fact, for certain code classification tasks, it may be that rare capsule outputs from the initial layer are the most important to preserve.

My biggest concern has to do with the empirical results. The source of Dataset C (Mou et al 2016, https://arxiv.org/pdf/1409.5718.pdf) reports 94.0% accuracy in Table 3 on their TBCNN method on the same dataset, whereas this paper reports 79.40% accuracy for TBCNN. I understand that the later result comes from a reimplementation, but it seems fairer to compare against (or additionally report) the results from the original authors of the method.

Also, the paper cites ASTNN (Zhang et al 2019, https://dl.acm.org/citation.cfm?id=3339604) in the introduction, and even though that paper reports (in table 2) 98.2% accuracy on Dataset C, the results table of the paper under review does not mention this in the evaluation section. I don't think that a paper necessarily has to achieve empirical results beating all previous ones in order to merit acceptance, but the way that the comparison is currently set up doesn't seem to facilitate a clear comparison of the pros and cons of this method versus other ones in the literature.

For the above reasons, I vote to reject the paper. For future submissions, it would be good to see a more comprehensive empirical comparison of the proposed method compared to others, and also to have more explanations about the design of the network.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper proposes a capsule-network-based architecture for predicting program properties and is evaluated on three tasks for predicting an algorithm from a code snippet.

Technically, the paper aims to transfer the idea of convolution from images and apply it to abstract syntax trees of programs. To do this, two dimensions describing the position of a node in a tree position are used - the depth of a node in a tree and its index in the list of children of its parent. This choice, however, is similar to image convolutions only at a very artificial level and drops significant amount of semantically-interesting information for programs from the index of the node at the parents, while keeping the total depth (which rarely matters in programs, as code is usually semantically similar no matter how nested in other code it is).

The experiments are small (on two small and one slightly larger dataset) and inconclusive:
1) Given the number of experiments done for tuning parameters on Dataset B (with ~640 examples), it is not clear that we are not observing some trivial case of overfitting. The improvement over GGNN is quite small and mostly due to ensembles.
2) The problem of small evaluation datasets make the results inconclusive. Only Dataset C is sufficiently large, if I assume no optimization like for Dataset B was performed.
3) Furthermore, it looks like the considered tasks are may be better handled by models such as code2vec or code2seq than by GGNN. The paper needs to include stronger baselines.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a tree-structured capsule network for program source code processing (essentially a program classification task with three datasets). 

The idea of incorporating tree structures into the design for capsule networks is not wrong. However, I am not sure why a capsule network is even needed in program classification. The authors follow the cliché of the importance of tree structures, but show little insight into the use of capsule networks in program analysis. The authors started rationalizing the capsule networks by saying "Capsule Networks itself is a promising concept ..." Being a promising concept itself doesn't necessarily mean it has/is suitable to be applied in program classification.

The treatment in Sec. 5.1 of the tree structures is pretty the same as in Mou et al. [2016], linearly weighting a token by its position. Sec. 5.2 is extremely hard to understand. It starts with presenting an algorithm and its line-by-line interpretation. I know how to program, but I wish to get some intuition of why capsule networks are needed for program classification, and how it is different from a generic capsule network and/or a graph capsule network [Xinyi & Chen, 2019]. Given a graph capsule network is in place, I found the contribution of this paper (tree capsule network) is limited. 

The experiments are very thin. The authors only compare their results to TreeCNN and Gated  Graph NN (GGNN). It's unclear if TreeCaps is better than other existing models, such as Transformer, TreeTransformer, GraphCap, etc.

While the authors experimented on three datasets, the evidence is actually limited. Dataset A is saturated (99.3%--100%). Dataset B shows some performance improvement (compared with GGNN and TreeCNN only). Dataset C basically shows TreeCaps is similar to GGNN. The gap between 89.41% and 86.52% is largely due to model ensembles. But the performance of GGNN ensembles is unknown. 

In summary, the paper applies Capsule Network to tree structures. The authors mainly follow the cliché of tree structures, but are not too excited about the capsule stuff. I am not excited either. 

==
Minor:

Nghi D. Q. BUI -> misformatted. Probably they are two people.
Zhang Xinyi and Lihui Chen --> Not sure if Xinyi is the last name. 
review_title: Official Blind Review #3

Meta-review: This paper proposes an application of capsule networks to code modeling.

I see the potential in this approach, but as the reviewers pointed out, in the current draft there are significant issues with respect to both clarity of motivating the work, and in the empirical results (which start at a much lower baseline than previous work). I am not recommending acceptance at this time, but would encourage the reviewers to clarify the issues raised in the reviews for future submission.
Decision: Reject