Venue: NeurIPS 2022 Challenge CellSeg

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a framework that utilizes both CNN and Transformer for weakly supervised cell segmentation. CNN model is supposed to provide local information, Transformer is supposed to provide long range relation. 

The good thing is that the paper proposes a complete solution, and it utilizes both labeled data and unlabeled data. However, the reviewer still have the following concerns:

1. The paper proposes a method for weakly supervised. However, the motivation and the efficiency of the method is not discussed. There are many questions about the proposed method have not been answered in the paper. For example: Why the predictions of CNN and Transformer are good pseudo labels for the other model? How to deal with error propagation issue? With the proposed training scheme, it is very likely that both CNN and Transformer make the same errors in the final predication.  

2.  One of the challenges in cell segmentation is to segment cells correctly in crowded cases. Local information is always crucial to separate cells correctly. However, in the proposed method, how to deal with this challenge is not mentioned. It uses Transformer to provide ``long range relation''. What kind of long range relation the transformer provides? How does it help to segment cells?

3. Some details are missing. How to obtain the final instance segmentation results with two different outputs? 



review_title: A cross-teaching framework for weakly supervised cell segmentation

final_score: rating: 7: Good paper, accept
review_content: This paper provided a cross teaching frame for cell segmentation. With cross teaching the solution proposed in this paper outperformed the official baseline.
There are several expressions could be updated, which make this paper more rigorous and clearly reading:
1. 'the distance from the center of the cell' and 'the neighbor distance between cells' are not 'class'. They are properties of a pixel rather than a semantic class.
2. The words could be more brief. For example, section 2 (method) could be more clear with implementaion details moved to the experiment section, and the decription for U-Net structure in section 2.2 could be shorten to a cite in a sentense.
3. The paper didn't point out the difference between the proposed framework and the cited works.

review_title: Review comment of 'CNN and Transformer cross-teaching for multimodal image cell segmentation'

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a cross-teaching strategy for multi-modal cell segmentation. By leveraging both labeled and unlabeled data, the proposed model can effectively reduce the annotation burden. This paper is well written and easy to follow. The major concern is the references of the related works, details below. 

- Some milestone studies are missing [1][2]. And the reference is incomprehensive and inaccurate. For example, ‘Deep Learning (DL) algorithms have shown encouraging results in fully supervised image segmentation [3]’ In this sentence, authors mentioned fully supervised models, but cited a semi-supervised model. Appropriate references should be carefully added. I suggest citing the SOTA fully supervised nuclei or cell segmentation models here, such as [3][4]. In addition, authors should discuss more about how to reduce the annotation effort [5][6][7]. 

[1] Pachitariu, M., Stringer, C. Cellpose 2.0: how to train your own model. Nat Methods 19, 1634–1641 (2022). https://doi.org/10.1038/s41592-022-01663-4

[2] Greenwald, N.F., Miller, G., Moen, E. et al. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat Biotechnol 40, 555–565 (2022). https://doi.org/10.1038/s41587-021-01094-0

[3] Graham S, Vu QD, Raza SEA, Azam A, Tsang YW, Kwak JT, Rajpoot N. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal. 2019 Dec;58:101563. doi: 10.1016/j.media.2019.101563.

[4] Zhao B, Chen X, Li Z, Yu Z, Yao S, Yan L, Wang Y, Liu Z, Liang C, Han C. Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation. Med Image Anal. 2020 Oct;65:101786. doi: 10.1016/j.media.2020.101786.

[5] Zhao T, Yin Z. Weakly Supervised Cell Segmentation by Point Annotation. IEEE Trans Med Imaging. 2021 Oct;40(10):2736-2747. doi: 10.1109/TMI.2020.3046292.

[6] Qu H, Wu P, Huang Q, Yi J, Yan Z, Li K, Riedlinger GM, De S, Zhang S, Metaxas DN. Weakly Supervised Deep Nuclei Segmentation Using Partial Points Annotation in Histopathology Images. IEEE Trans Med Imaging. 2020 Nov;39(11):3655-3666. doi: 10.1109/TMI.2020.3002244.

[7] Han C, Yao H, Zhao B, Li Z, Shi Z, Wu L, Chen X, Qu J, Zhao K, Lan R, Liang C, Pan X, Liu Z. Meta multi-task nuclei segmentation with fewer training samples. Med Image Anal. 2022 Aug;80:102481. doi: 10.1016/j.media.2022.102481.

- ‘so the capacity of the developed model to process data never seen before’, this sentence seems not complete. I guess the authors want to say ‘the capacity of the developed model is limited to process the data it has never seen before’?

- ‘Our solution is based on a cross-teaching between a Convolutional Neural Network (CNN) and a Transformer. This framework takes both labeled and unlabeled images as inputs, and each image passes a CNN and a Transformer respectively to produce the prediction [3]. For the labeled data, the CNN and Transformer are supervised by the ground truth individually [3]. Then the predictions of unlabeled images generated by CNN/Transformer are used to update the parameters of the Transformer/CNN respectively. This framework benefits from the two different learning paradigms, CNNs focus on the local information and transformers model the long range relation, so the cross teaching can help to learn a unified segmenter with these two properties at the same time [3].’
Why authors cite [3] for three times in one paragraph? According to the title of [3], it seems that the idea of this pipeline comes from this paper. If so, authors should clearly point out this and shows the difference between the proposed method and the reference [3].

Authors should carefully proofread the paper, here are some typos:
- trasformer -> transformer
- The U-Net is a 5 layer network -> 5-layer network
- this require a more accurate -> requires

review_title: Review comment - CNN and Transformer cross–teaching for multimodal image cell segmentation

final_score: rating: 7: Good paper, accept
review_content: This paper focuses on the segmentation problem of biological imaging. Traditional algorithms for biological image segmentation must be tailored to the cell type and imaging environment. So semi-supervised or unsupervised multimodal segmentation draws more and more interest.

The authors propose a novel algorithm for transferring and aligning the knowledge between different models with the novel proposed Distance and Neighbour labels. They use UNet and Swin-UNet as the basis of cross-teaching architecture. The proposed method exploits both labelled and unlabeled images and combines the learning paradigms of the two different architectures used.

Minor Concerns:

1. Please cite Diceloss properly.

Major Concerns:

1. In Table 4, baseline and cross-teaching have different numbers of epochs. It would be great if the authors can provide an evaluation with regard to the same number of training epochs.
2. Besides, in Table 4, baseline and cross-teaching using supervised and semi-supervised training. It would be great if authors can provide a baseline with semi-supervised learning.

Questions:

1. (Ablation study) Would representation-based distillation be more effective in transferring knowledge?

This paper is clearly written and with minor grammatical errors which do not affect the reading. The proposed method with Distance and Neighbour labels is novel.
review_title: CNN and Transformer cross–teaching for multimodal image cell segmentation 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary
========

This paper proposes a framework based on cross-teaching between a CNN and a Transformer to exploit both labeled and unlabeled images as an alternative to fully supervised learning approaches. The clear advantage of such an approach is that it does not requires a lot of training data coming from a single imaging modality, can generalize well on images coming from varied imaging modalities and benefits from CNN and Transformer architectures to create a pseudo-label for the unlabeled images as a part of the training process. The network has a supervised branch (for labeled images) and a cross-teaching branch (for unlabeled images), the loss terms compare the pseudo-labels generated by the CNN and Transformer. The network also has regression loss terms to produce the distance and neighbor labels along with predicting background, interior and boundary pixels.

Training Methodology
-----------------------------
The authors train the CNN (U-Net) and the Transformer (Swin Transformer + U-Net) individually which are then used as pre-trained models during the cross-teaching between the two networks. During the base model training a combination of NeurIPS provided data with Cellpose and Omnipose data was used for model trainings, during the cross-teaching only the NeurIPS provided data was used.

Minor Comments
------------------------
1. Is the base library for applying the training and prediction only Pytorch based or is a Keras version also available? 
2. Including a link to the Github code, pypi package etc would be very helpful, this review does not represent a code review and we only focussed on the contents of the paper and the methodology presented.

Major Comments
------------------------
1. Maybe the authors can use recurrence based neural network architecture (ResNet `Kaiming_2016_CVPR`/DenseNet `Huang_2017_CVPR`) for the CNN branch as such networks have superior classification performance and that can potentially boost the performance of the cross teaching network as well. For the future work it would be interesting to see the effect of changing the base network architecture on the performance on the bad segmentation examples.

Quality
----------
The methodical work is of much better quality than the currently existing segmentation solutions in the field of bio-image analysis and it represents the next state of the art. 

Clarity
---------
The methods are described very well and the results of good and bad segmentation are presented as separate figures followed by a discussion of the strengths and shortcomings of their method. The paper is within the page limit and is presented in a way that is intuitive and easy to follow.

Conclusions
-----------------
It was a pleasure to read and review this paper, rather than being a delta increment publication this methodology provides a one level up than the current bio-medical image segmentation where iterations of manual relabelling to increment the ground truth training data are current state of the art, therefore this methodology sets the baseline for the next state of the art.

@InProceedings{Huang_2017_CVPR,
author = {Huang, Gao and Liu, Zhuang and van der Maaten, Laurens and Weinberger, Kilian Q.},
title = {Densely Connected Convolutional Networks},
booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {July},
year = {2017}
}

@INPROCEEDINGS{Kaiming_2016_CVPR,  author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},  booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},   title={Deep Residual Learning for Image Recognition},   year={2016},  volume={},  number={},  pages={770-778},  doi={10.1109/CVPR.2016.90}}

review_title: Cross-teaching as a way to use labelled and unlabelled data for bio-medical image segmentation 

Decision: Accept