Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to apply contrastive self-supervised representation learning to tree-structured neuron morphology data. The authors adopted Tree LSTM (Ref. [51]) and MOCO (Ref. [16]), and proposed tree topology data augmentation. The proposed method was demonstrated on three public datasets (BIL, JML, and ACT). Learned representation was visualized with 2D t-SNE and then qualitatively inspected and analyzed. For quantitative evaluation, cell-type classification accuracy was measured with (1) K Nearest-Neighbor (KNN) accuracy from self-supervised pre-training and (2) supervised fine-tuning.
strengths_and_weaknesses: ### Strengths
This paper is written clearly and presented well. The problem of interest (representation learning on unlabeled neuron morphology data) is clearly significant. 


### Weaknesses
Novelty is limited. This paper has two novelties: (1) application of contrastive self-supervised learning to tree-structured neuron morphology data, and (2) topology data augmentation for tree data. However, the ablation study (Table 4 and Section 4.3) shows that the proposed topology data augmentation may not be effective nor important. Table 4 shows that the point transformation ("point") is the single-most effectective one, and the combination of "point" and attribute masking ("mask") achieved the best accuracy. The "point" and "mask" are both general graph augmentation, so the claimed novelty of the proposed tree topology augmentation becomes rather pointless.

Evaluation contains multiple problems. First, I don't understand why the Allen Cell Types (ACT) dataset was included in the evaluation. As is shown in Figure A2, ACT's neuron reconstruction is largely incomplete. I see no point in using incomplete reconstructions which could only complicate representation learning. Second, training and evaluation seem quite arbitrary. Why was the training/test set split only used in supervised fine-tuning, but not in pre-training? It is very weird to compute KNN accuracy **during, not after, pre-training and report the best one** while not even having a separate test set. Also why pre-training for 100 epochs for the main experiment and 50 epochs for the ablation study? Third, it is unclear whether the two baselines (MorphVAE and TRNN) were faithfully reproduced. Without any evidence for this, the main quantitative result of this paper in Table 3 cannot be fully justified.
questions: * (L224) Was the training/test set split only used in supervised fine-tuning? Why not in pre-training?
* (L235-236) The choice of number of neighbors for KNN is very arbitrary. 
* (L264) "..., with the best KNN evaluation accuracy during pre-training included."
    * I do not understand what it exactly means. Do you record KNN evaluation accuracy when each epoch is done, and report the best among them? That doesn't seem very reasonable to me. I think if you were to report cell-type classification accuracy from pre-training, you should strictly use separate training and test sets, and report KNN evaluation accuracy on the test set **after** pre-training is done.
* (L287-288) Why pre-training for 100 epochs for the main experiment and 50 epochs for the ablation study?
limitations: See "Weaknesses" in the above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a contrastive framework to learn features from neuron trees. The authors have devised a carefully engineered set of augmentation suitable for tree-structured data sets. The learned features were used to classify cell types that yield superior performance compared to the previous methods.
strengths_and_weaknesses: Strengths:
1. The proposed method is technically sound and novel.
2. The authors proposed several augmentations for the neuron trees, which turned out to be useful for self-supervised feature learning.
3. The results are convincing across multiple datasets showing generalizability.

Weaknesses:
1. While the proposed method yields superior performance in particular problems for neuron morphology learning, my concern is that it is a niche problem. How could it benefit the general graph/topology learning community in general? The authors should comment on the applicability of the proposed tree representation learning in other domains, for example, molecular properties classification.
2. The authors did not elaborate on the motivation for using InfoNCE loss over several other contrastive settings. Is there any particular reason to choose it?
3. The authors mentioned a few contrastive learning on graphs. However, a direct comparison is missing to demonstrate the advantage over the previous approaches.
4. It is unclear whether the baseline experiments [21,51] are performed in identical experimental settings. Further, if one has also to include the axons, would the tree structure assumption still hold? What is ‘m’ mentioned in the data augmentation?
questions: See the comments on weakness.
limitations: The authors have not adequately commented on the known limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents self-supervised learning for neuron morphology representation. In this paper, the neuron morphology is represented as tree-structured data. Then, the Tree-LSTM is used to represent the tree-structured data. Finally, the representations are learned through contrastive learning with the proposed tree data augmentation techniques. 
strengths_and_weaknesses: The proposed approach for representing neuron morphology is reasonable. Furthermore, this paper is well-written and easy to follow. Some experiments, including ablation studies, are well analyzed and discussed.

However, this paper leaves some negative or uncertain points. 
1. First, the proposed framework is too weak from the novelty viewpoint. The proposed representation learning is based on the MoCO [16, 6], one of the visual contrastive learning methods. The differences are that the data is tree-structured and uses tree-data augmentation. Even if the tree data augmentation is novel, some techniques in the augmentation are directly derived from graph data augmentation. As a result, in lines 44 and 307-308, the statement ``this paper proposes a **new SSL framework** for tree structure representation learning'' is an overstatement for the reviewer. 

2. Since the tree is a special type of graph, it is necessary to include graph-based SSL [48] as a comparative model in the quantitative evaluation.  

3. It would be better to mention/include details of baselines (morpheVAE and TRNN) used in the neuron cell-type classification experiment, such as how to implement and train the baselines. Without the details, it is hard to conclude that the proposed method achieves better performance than baselines.

4. Delineating the learned representations using TSNE was meaningful. However, the reviewer thinks it is not enough to show only the representations of the proposed method. It would be better to include the representations of baselines.
questions: In line 203, $m$ is empirically set to 10. What does setting $m$ to 10 mean? Would not changing this value make a difference in performance?
limitations: Adding some baselines such as graph-based SSL in the quantitative evaluation help show the proposed method's superiority.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors represent a neuron as a feature vector for each neuron node, and use an MLP and an Tree-LSTM to embed the neuron vectors in an embedding space using contrastive learning. Augmentations, dedicated to neuron morphology are proposed to enhance the contrastive learning procedure. 
strengths_and_weaknesses: Originality:

I don’t find the method very original or novel. The authors basically use SimCLR and adapt the architecture (inclusion of a Tree-LSTM) and augmentations to be suitable for tree structured data.
 
Quality:

I am doubting the validity of the performed experiments. There is no separate validation set, and the full training and test set is used to train CoNMoR. That results in two issues. How was the stopping criterium determined for CoNMoR training if no validation set was present? And second, the test set was already seen during CoNMoR pre-training which may highly boost the test set performance when subsequently training a supervised classifier. 
Also, in line 264 the authors report the best KNN evaluation accuracy that was acquired during pre-training. So I understand this is not the value after the 100 epochs of pre-training? The authors propose this approach as fully unsupervised, however if labels are used to determine the epoch where KNN accuracy is highest, it is still not fully unsupervised. It would be more fair to report the KNN accuracy at the end of pretraining.
Looking at Fig. 4, it seems that the L5 class is mostly very distinct from the other classes, since almost all metrics show ca clustering pattern around L5. So I’m not too sure whether the representations are really encoding all the features correctly, or simply distinguish L5 from the rest. 

Clarity:

Since the paper is aimed for NeurIPS audience, I think it deserves a bit better introduction in the field of neuron morphology and its analyses. Around line 44 I wondered, wat is the input to the model, a 3D reconstruction? But it’s a reconstruction of what? So try to give a better background about these reconstructions, what the data exactly is, how it is measured and what you are reconstructing here. Line 55: what is swc format? Better to explain that as well. 
Reading further, the first paragraph of 3.1 is also very unclear to me (I’m a reader without much background in neuron morphology). So if you want to target a broader audience, try to explain the concepts better. Remarks like “Also, the number of such 1-degree
 nodes is biased to neuron reconstruction pipeline and raw data resolution.” need extra explanation.
The last few sentences of this first paragraph in 3.1 might be better understable if a figure was added to illustrate the concept. 

Fig 3b is a bit hard to digest, and might need some improvement. Also, Table 4 could have a more descriptive caption. What is the meaning of the numbers and how should the reader compare them to Table 3?

Significance:

The idea of contrastive learning on tree structures is interesting in general. I think, however, that the paper could have had a broader signficance if it was written and tested as a more general method, for any type of tree-structured data. I also wonder whether it is needed to give this method a separate name CoNMoR, since it is basically SimCLR, adapted for tree structured data. 

questions: L108: V_i represents a 3D coordinate belonging to vertex i?

L133: What are the input states here? Child nodes?

Eq 1-4: the second subscript of the h-variables was not formally defined. I assume it’s the first and second layer of a stacked LSTM?

L170: are q and k the normalization factors, or the already normalized vectors. Looking at eq. 5 I guess, q and k are the normalized representations, but then the definitions in l170 seem incorrect and should be: q = h(z_q) / ||h(z_q)||_2 etc.

L193: which perturbations are you pointing to here?

L203: What is m?

L301: BN = batch norm, I assume? Why do you find it apparent that this is more needed for self-supervised training? I don’t find this very surprising actually, as there is no label that guides the training, and therefore structure must be sought in the data itself, which might require proper normalization. 

limitations: The training/testing strategy is doubtful (see the quality section above), and the authors did not present any limitations to their proposed work themselves. How is the model, e.g., dependent the type of features that were selected now?
review_title: null

Meta-review: metareview: The paper proposes a self-supervised learning method targeting representation learning for large-scale neuronal morphology. This is an important problem in connectomics, and systems neuroscience at large, and very little progress has been to date in applying modern machine learning methods to this problem. The authors combine a TreeLSTM with self-supervised learning.

The paper received borderline reviews, but the reviewers were unanimous in recommending acceptance. Two main concerns were raised. First, the reviewers questioned the novelty and applicability of the method for the machine learning field at large. Ultimately, there was some acknowledgement that some of the techniques proposed could have applicability beyond the field of ML for neuronal morphology. But even if this were not the case, neuroscience is an important focus area for the NeurIPS conference. Second, there were issues raised with the evaluation methodology. These were clarified by the authors, and additional material was added to the Appendix to enable a better understanding of the method.

I recommend acceptance.
confidence: Certain
Decision: Accept