Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: The premise of the work is that current image segmentation architectures lack the ability to learn and use global context. The work therefore integrates prior work on spatial dependency network architecture features into segmentation architectures to enable better inference of global context and evaluates the results on 4 medical imaging datasets. Improved results are seen compared the U-Net and U-Net++ consistently.
strengths: - Addresses the relevant and important problem of getting more global context into segmentation models.

- Spatial dependency networks have been used for generative models previously, so the application to segmentation is not a large change, but none the less novel.

- The described method shows impressive and consistently good results on multiple datasets in comparison to U-Net and U-Net++.

weaknesses: - A major weakness of the approach is that it lacks description and discussions of training or inference times or GPU memory usage. Granted some information is given in Appendix C and D, however, C lacks a comparison with the reference U-Net and U-Net++ models and only seems to give the GPU card memory details, not the algorithm usage. Appendix D shows that the proposed models do have larger number of parameters and require more floating point computations, but is still within the same order of magnitude. I would have preferred scalability to be considered in terms of both computational complexity and memory usage.

- No 3D processing is used either in comparison methods or in the proposed method as far as I understand. Considering that this is a medical imaging conference and 3 out of 4 of the datasets considered are in 3 dimensions, this would have been an obvious thing to discuss or try. This relates back to the scalability issue mentioned above. One of the primary challenges in medical imaging is dealing with context across larger and higher dimensional input images. If the proposed method does not scale well to larger images and 3D it is likely not relevant for many problems.

- No mention of newer architectures, such as vision transformers and combinations of vision transformers and CNNs, which attempts to solve similar problems of long range interactions and context.

- I find the SDU-Net explanation unclear. Particular section 2.1 and 2.2. Consider introducing more motivation and reasoning. What is the subscripts i and j indexing over in equation 2?

- Lack of publically available code repository makes reproducing results more difficult.

detailed_comments: - I think the claims that traditional approaches fail at, what the authors call holistic modelling, and in particular the descriptions of why should be expanded on. Why and at what do traditional approaches fail and what is it that the proposed approach is improving on?

- Figure 2 and receptive field points - The receptive field computation illustrated does not seem particularly relevant to me, as most modern neural networks for image processing do not just use convolutions, but also pooling layers, as also mentioned by the authors. The authors say pooling layers leads to an additive increase of (k-1)s in receptive field size, but to my understanding this is wrong. Pooling layers exponentially increase receptive field size.

- "But note that such a receptive may not fully encompass the original image.", I am not sure what the authors mean by this.

- "Also, such composition of layers only weakly models distant pixel relationships across different levels of the architecture. These limitations are unnecessarily restrictive for medical image segmentation.", please be specific here. How is it weak? Why is too restrictive? Moreover, these are unsupported statements on methods that have established themselves through 1000s of publications. Please be specific in your claims, whether done so or not, consider removing them or backing them up with solid references.

- "We found it sufficient to include spatial dependency layers at lower scales of the U-Net++", by what experiments, methods and data?

recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper introduces a novel spatial dependency layer to introduce spatial recurrence into convolutional neural networks. This is a novel concept which has recently been introduced in an ICLR 2021 paper and used in the domain of image generation (VAEs). In this paper, the layers are used in a segmentation architecture, a U-Net architecture. The paper presents experiments on four different datasets coming from four biomedical image analysis challenges and compares against the U-Net and U-Net++ as baseline methods and presents improvements for all applications.
strengths: - The paper is in general well written and provides extensive additional information in the supplementary material.
- This paper introduces a novel concept to incorporate global context into convolutional neural networks and show with simple experiments the benefit of the approach. This new concept is interesting to the field.
- Experiments are conducted across four different tasks.
weaknesses: -  I think the term 'holistic' is not the best; merriam-webster meaning of the word holistic is 'relating to or concerned with complete systems rather than with individual parts'. I would advise the authors to stick with the term spatial recurrence
- I am missing a good introduction and discussion in which the presented approach is put into context with other approaches that try to integrate long-range connections for image segmentation.
- I am concerned about computational cost, both during learning and inference. Appendix D is not linked in the text, but does provide figures on computational cost. The paper would benefit from a short discussion on this.
- This is a 2D approach applied to two 3D tasks, can the authors comment on extension of SDN layers to 3D?
 
detailed_comments: - No comparison to nnUnet, a pity in my opinion. In addition, the papers reference an older arxiv paper on the nnUnet framework, while a paper has already appeared in Nature Methods: https://www.nature.com/articles/s41592-020-01008-z
- No evaluation on the liver tumor category in the LITS2017 challenge, why not?
recommendation: Poster
review_title: null

final_score: rating: 3: Borderline
review_content: summary: The paper introduces a novel method that incorporates spatial recurrence for medical image segmentation via UNet and U-Net++ networks in order to take into account long-range spatial dependencies in medical data thus improving segmentation performance and enabling holistic modeling. The proposed method is evaluated on four segmentation tasks and demonstrates an increase in segmentation performance relative to U-Net and U-Net++ baselines.
strengths: The method is well evaluated on 4 segmentation tasks of different modalities and across 3 performance metrics. The authors have done a good job of defining the problems and objectives of the paper.  In addition, the authors have verified the significance and stability of the method through 5-folds cross-validation, multiple random seeding runs, and an ablation study across different model hyper-parameters. 
weaknesses: The main issue with the paper is the unclear presentation and mathematical formulation of the method. The reviewer has found it difficult to understand the methodology, and therefore the true added value of the proposed contribution relative to the already present state-of-the-art methods. Various annotations and variables in Section 2 are not clearly defined especially in eq. 1 and eq. 2 which makes it difficult to valorize the methodological added value of the paper. The reviewers would advise a proper and clearer formulation and demonstration of the method to be done as a condition of acceptance of the paper. 
detailed_comments: I am fairly confident that the work is suitable for publication to MIDL and as an extended version for a special issue conditioned on whether the authors address the clarity issues in the methodological formulation and demonstration of the method. 
recommendation: Poster
review_title: null

Meta-review: metareview: In this paper, a U-Net based segmentation method is proposed, that leverages spatial dependency layer, a new concept which has recently been introduced in an ICLR 2021 paper for image generation. These layers allow to incorporate global context into CNN.
Performance are compared on 4 medical imaging datasets to other state-of-the-art architectures (eg UNet and UNet++ initially).

Strengths:
- This paper introduces a novel concept (proposed elsewhere) into CNN to learn and use global context for image segmentation
- Experimental protocol is sound, presented results are convincing.

Weaknesses:
Initial concern raised by the reviewers focus both on the content and on the form of the papers:
- Comparison to other architectures were lacking (eg Transformers, nnUNet),
- 3D image segmentation was not addressed,
- Computational cost was not properly discussed.
- Some points in the description of the method were not clear

The authors have addressed most of the various questions raised by the reviewers, with additional & results, discussion and re-writing. 

Both great, complementary reviews and substantial effort by the authors have led to a significantly improved version of the paper, for which I now recommend acceptance at MIDL.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)