Semi-supervised learning with Noisy Students improves domain generalization in optic disc and cup segmentation in uncropped fundus images

Authors: eugenia  moris
ignacio  larrabide
josé ignacio orlando
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 20:24:34

Abstract: Automated optic disc (OD) and cup (OC) segmentation in fundus images has been widely explored for computer-aided diagnosis of glaucoma. However, existing models usually suffer from drops in performance when applied on images significantly different than those used for training.
Several domain generalization strategies have been introduced to mitigate this issue, although they are trained and evaluated using images manually cropped around the optic nerve head. This operation eliminates most sources of domain variation, therefore overestimating their actual ability to cope with new, unseen patterns. 
In this paper, we analyze the most recent and accurate methods for domain generalization in OD/OC segmentation by applying them on uncropped fundus pictures, observing notorious degradations in their performance when trained and evaluated under this setting.
To overcome their drawbacks, we also introduce a simple semi-supervised learning approach for domain generalization based on the Noisy Student framework.
Using a Teacher model trained on a combination of domains, we pseudo-labeled a dataset of 18.000 originally unlabeled images that are then used for training a Student model.
This semi-supervised setting allowed the Student network to capture additional sources of variability while retaining the original cues and patterns used by the Teacher through the weak annotations.
Our results on eight different public datasets show improvements in every unseen domain over all alternative methods, and are available in https://github.com/eugeniaMoris/Noisy_student_ODOC_MIDL_2024.

Main paper: 
 introduction: 
 Segmenting the optic disc (OD) and cup (OC) in fundus images is a common practice for detecting and characterizing glaucoma, one of the leading causes of irreversible blindness worldwide (Veena et al., 2020) . A significant effort has been made to automate this task, resulting in models with excellent performance in known databases (Alawad et al., 2022; Moris et al., 2023; Wang et al., 2019) . Nevertheless, their accuracy is frequently affected when applied on images from domains unseen during training, hampering their clinical application (Nan et al., 2022) . This is inherent to the natural diversity in the appearance of fundus images, e.g. due to the overall quality of the scan, variations in the acquisition protocol or device, the intrinsic retinal pigmentation associated to patient ethnicity, or the presence of lesions that were not considered in the training sets (Yoon et al., 2023) .Typically, practitioners aim to overcome this limitation with increased data augmentation (Lyu et al., 2022) , although modelling every possible scan (and disease) appearance with image transformations is unfeasible. Alternatively, several studies introduced novel domain generalization techniques (Yoon et al., 2023) that aim to improve OD/OC segmentation in unseen domains without requiring domain-specific information or adaptation, e.g. through domain alignment (Chen et al., 2021; Liu et al., 2021; Wang et al., 2020; Chen et al., 2022; Hu et al., 2022; Zhou et al., 2022) , meta-learning (Hu et al., 2023) , and augmentation techniques (Lyu et al., 2022; Yang et al., 2021; Kang et al., 2022; Gu et al., 2023) . While these have reported remarkable improvements when applied on unseen domains, we noticed that, in all cases, they are trained and validated using manual crops around the optic nerve head (ONH). This decision drastically eliminates most sources of image variability, implicitly overestimating their ability to cope with alterations outside this area. Furthermore, they also require users to perform the crop themselves when deployed in real clinical settings, hampering their automation and applicability for processing large databases.In this paper we perform an in-depth evaluation of the most accurate existing models for domain generalization in OD/OC segmentation, by training and applying them on uncropped fundus images. In line with our hypothesis, we observe notorious degradations in their results when compared with their reported numbers. Furthermore, we show that a semi-supervised learning strategy based on the Noisy Student (Xie et al., 2020) is already able to overcome this limitation. Semi-supervised learning is an active area of research for medical image segmentation (Jiao et al., 2023) , while weak supervision have shown promising results for OD segmentation e.g. through classification labels or bounding boxes in a multitask setting (Yin et al., 2023) and for source-free domain adaptation (Huai et al., 2023) . In our case, we leverage a Teacher model trained on diverse domains to pseudo-label a dataset comprising 18.000 initially unlabeled images. This massive set is then used for training a Student model, which effectively assimilate new sources of variability while preserving the intrinsic cues and patterns imparted by the Teacher via weak annotations. Our evaluation across six unseen domains reveals consistent performance enhancements, surpassing the alternative methodologies. Furthermore, we also observe a statistical preservation of performance on unseen scans belonging to the three domains used for training. 
domain generalization with noisy students in uncropped images: 
 Let S = {(x i , y i )} n i=1 represent a dataset with n pairs of uncropped fundus images x i ∈ X and their corresponding multiclass OD/OC annotations y i ∈ Y. In a standard supervised learning setting, this set is used to train a multiclass segmentation model f θ S : X → Y, with f denoting the neural network architecture and θ S the learned parameters. When applying the model f θ S on a new unseen target domain D T , it is desirable for f θ S to retain its original performance. To this end, networks are usually trained in datasets as big and diverse as possible, e.g. by crafting S using subsets of images X sampled from multiple source domains D (j) S . Notice that this requires scaling manual annotation for every new input sample, which is expensive and time consuming for segmentation tasks. As an alternative, one can stimulate the model to learn other alternative appearances through heavy data augmentation, although modelling every possible target scenario through image transformations is unfeasible. At the same time, it is likely that the model potentially learns to cope artificially produced artifacts, as there are no guarantees about the actual resembling of these altered images to those expected from any unseen target domain D T .Our hypothesis is that leveraging samples from a large enough unlabelled dataset U during training might mitigate and reduce the covariate shift between training and test domains, improving domain generalization with no extra manual annotation effort. Furthermore, feeding with uncropped fundus images should aid the network to capture sources of variation complementary to those around the ONH. To accomplish this, we propose to follow a Noisy Student approach (Xie et al., 2020) , originally introduced for image classification. In this case, our goal is to learn a Student network for OD/OC segmentation in color fundus pictures using pseudo-labelled uncropped samples from U. Figure 1 (a) depicts our methodology. First, we train a supervised Teacher model f θ S using the labelled dataset S, with uncropped images from a (combination of) source domain(s) D S . Afterwards, this network is applied on an unlabeled dataset U = {x i } m i=1 , with n << m, to generate weak, pseudo-labels ŷi of the OD and the OC. The resulting pairs (x i , ŷi ) can then be used to train a Student model, either individually (resulting in f θ U , Figure 1 (a)), or in combination with S (resulting in f θ U ∪S , Figure 1 (b) ), (see Section 2.2). Both the Teacher and the Student are based on the exact same architecture f , and the Student is trained with stronger data augmentation than the Teacher, as suggested by Xie et al. (2020) . This hardens the task solved by the Student model, forcing it to learn new patterns to obtain similar segmentations to those obtained by the Teacher, but in other different, more difficult images. This combination of new pseudo-labeled samples and data augmentation, and the fact that the Student is trained from scratch on this set, has been reported to improve results on out-ofdistribution samples within seen test sets (Xie et al., 2020) . Alternatively, we propose to apply this approach to improve results in unseen target domains D (j) T .A Student f θ U trained with pseudo-labels benefits from an implicit transfer of knowledge from the Teacher through weak targets. In practical terms, this should manifest in the Student having similar performance in the source domains D S . Nevertheless, recall that the Student is trained from scratch on U, meaning that there was no direct access to images from S. Therefore, the model might behave in domains D S as if they were target domains, experiencing a certain drop in performance. To alleviate this issue, one option is to include S in the training set of the Student, resulting in a model f θU ∪S that had access to both sets simultaneously (Figure 1 (b) ). Alternatively, we can think on an iterative process as the one in Figure 1 (c), in which the Student f θU is fine-tuned in S (resulting in f θ (2)S ) to become a Teacher, and then applied on U to produce new pseudo-labels to create a new f θ(2) U as in (Hao et al., 2022; Guan and Yuan, 2023) . This process can be repeated k times until reaching convergence, e.g. by evaluating performance increments on a held-out set. 
implementation details: 
 Code was implemented using PyTorch Lightning (v. 1.5.10), and all experiments were conducted using NVIDIA RTX 3060 GPUs with 12GB. Both Teacher and Student used the U-Net described in (Moris et al., 2023) (with 31 million parameters) as backbone.Training was performed minimizing a multiclass cross-entropy loss with Adam optimization. Learning rates were experimentally adjusted to each run, based on validation performance. Batch sizes of 20 images were used in all cases. For f θ S∪U , each batch included 18 samples from U and 2 from S, to ensure the model had access to both domains on each iteration. To differentiate errors on each domain, we used a convex sum of losses with a λ coefficient.Data augmentation was used following a custom adaptation of RandAugment (Cubuk et al., 2020), using vertical and horizontal flipping, Gaussian blur, rotation, rescaling, and color jittering as transformations, parameterized with a probability p of applying a transformation and a strength factor s to increase or reduce their limits (e.g. angles in rotations, size of resizing and blurring filters, etc.). To avoid overfitting the Teacher, we also applied data augmentation for training it, using p = 0.1 and s = 0.1. The Student, on the other hand, was trained using p = 0.5 and s = 0.5. In all cases, images were resized to 256 × 256 pixels before feeding the network. In test time, output segmentation were re-scaled to the original image resolution using nearest neighbor interpolation for metric computation.A summary of the datasets used for training, validation and test is provided in the appendix. We built the supervised training set S with images from DRISHTI (Sivaswamy et al., 2014) , REFUGE (Orlando et al., 2020) and RIGA (Almazroa et al., 2018) . In particular, we took all REFUGE training set (400 images), and 90% of DRISHTI training set (45 images) and RIGA (675 images). The remaining 10% of RIGA (74) and DRISHTI (5) were combined with the offsite set of REFUGE (400 images) to build a validation set. For U, we randomly sampled 18000 scans from AIROGS (De Vente et al., 2023) training set. To calibrate model hyperparameters and monitor performance during training, 10% of these images were separated as a validation set. Test partitions from DRISHTI (50 images) and REFUGE (400 images) were used for testing in known domains. As unknown domains covering variations in acquisition machine, ethnicity, lesions, and field of view (FOV), we used all scans from RIM-ONE V3 (Fumero et al., 2011) (159 images, Spanish sample, cropped around the ONH), ORIGA (Zhang et al., 2010) (650 images from Malay adults), and the three subsets We also used images from PALM (Fang et al., 2024) (400 scans from an Asian population) to evaluate performance for OD segmentation in pathological myopia. We used the Dice Similarity Coefficient (DSC) and Hausdorff distance (HD) as evaluation metrics, to account both for the overlap with the ground truth labels and for boundary consistency, respectively (Maier-Hein et al., 2022) . Statistical significance of the differences in metrics was assessed using one-tail Wilcoxon sign-rank tests with α = 0.05. 
results: 
 Quantitative results for OC/OD segmentation in unseen domains are reported in Tables 1. We include results obtained with other domain generalization techniques that publicly released usable implementations for comparison. To ensure a fair evaluation, we re-trained their models with uncropped images, using our supervised set S. Notice that none of them follows a semi-supervised learning approach. To our knowledge, there are no studies on domain generalization for OD/OC segmentation following this approach. Our Student f θ U reported statistically significant improvements for OC segmentation with respect to the Teacher in terms of HD, for all the unseen domains, except BOSCH and ORIGA. This also holds for DSC values obtained in RIMONE, FORUS and REMIDIO. A similar behavior is observed for OD segmentation, where this Student reported improvements over the Teacher in all sets except PALM evaluated in terms of DSC, and slightly higher HD values in BOSCH, with no statistically significant differences. On the other hand, the Student f θ U ∪S was able to statistically improve Teacher's DSC values for OC segmentation for all datasets except BOSCH and FORUS. When evaluated in terms of average HD, this Student also achieved significantly better OC segmentations in RIMONE, REMIDIO, and ORIGA, but less accurate results in BOSCH and FORUS. For OD segmentation, the Student f θ U ∪S showed a different behavior, reporting better DSC and HD values than the Teacher in all datasets except PALM, where average DSC and HD are lower but statistically comparable.When comparing the Students one another, we observe that f θ U ∪S performs statistically significantly better than f θ U in RIMONE and ORIGA when evaluated using DSC for OC segmentation, but comparable in BOSCH and statistically worse in FORUS and REMIDIO. When using the HD, on the other hand, f θ U performs statistically better in BOSCH, FORUS, and ORIGA, almost equivalently in REMIDIO, and slightly worse in RIMONE. Conversely, for OD segmentation we see f θ U reporting statistically better DSC values in FORUS, REMIDIO and PALM, and almost equivalent results in all other sets. In terms of HD, f θ U reported values slightly better than f θ U ∪S in all sets except for ORIGA, although the differences are only statistically significant in FORUS and REMIDIO.Compared with other existing approaches, both Students reported better DSC values for OC/OD segmentation in all the datasets, except for OD results in PALM. In terms of HD, we observe improvements over the literature for OC segmentation in all datasets except for RIMONE, and for OD segmentation in RIMONE, FORUS, REMIDIO and ORIGA. In PALM, both Students reported statistically higher HD values than the three evaluated counterparts, while in BOSCH the differences are not significant.Figure 2 provides qualitative examples of OC/OD segmentation results obtained on images from different unseen domains. Examples were chosen to illustrate the behavior under changes in the acquisition device, ethnicity, lesions, and diversities of FOVs. As expected, the Teacher showed poor results in the unseen domains due to the intrinsic problem of domain generalization. Conversely, results obtained with the method by Zhou et al. (2022) demonstrate shape consistency throughout datasets, except under changes in ethnicity or when extensive peripapilary atrophy is present. Notice also a completely misfit of the OD with respect to the OC detection in the last image. Alternatively, our Students present more accurate segmentation for most of the cases, except for segmenting OC/OD in images with peripapillary atrophy when incorporating S.Finally, we performed additional experiments focused on comparing differences in seen and unseen domains (Figure A.1) , evaluating the effect of sequentially repeating our framework for another iteration (k = 2, Table D .2 and D.3), and evaluating glaucoma detection results when using vertical cup-to-disc ratio estimates obtained from segmentations retrieved with the Teacher and the Student models (see Appendix C). 
discussion and conclusions: 
 Domain generalization remains being a challenge in OD/OC segmentation due to the significant variations observed between fundus pictures. While several approaches have been introduced to improve results under multiple imaging settings, we observed in our literature This drops out most sources of variability expected to occur in these images, reducing the space of patterns to model in training and overestimating their actual performance on full size images when testing. This can be observed in Table 1 , where state of the art models re-trained on uncropped scans obtained lower values than those reported in the original papers (Wang et al., 2020; Chen et al., 2022; Zhou et al., 2022) . Additionally, training with cropped images introduces the need of doing a manual crop in test time, or training a separate model to automate it, e.g. in a coarse-to-fine manner (Moris et al., 2023) . However, doing so was proved suboptimal for OD/OC segmentation if the coarse part lacks domain generalization capabilities (Moris et al., 2023) . Thus, we are facing a causal loop paradox, where to achieve domain generalization we rely on a coarse detection model that needs to be able to generalize well to unseen images to ensure proper results.In this paper we even show that a simple solution based on Noisy Students can outperform existing approaches. By training Students using a massive set of images pseudolabelled by a pre-trained Teacher model, we obtained networks that reported better results in unseen domains than those obtained with other more technically complex methods trained and evaluated in full size images (Tables 1). These counterparts only reported better DSC and HD values for OD segmentation in one dataset, PALM, which features images of patients with pathological myopia. When analyzing results qualitatively, we observe that this drop is caused mostly when peripapilary atrophies are present (Figure 2 ), which significantly alters the appeareance of the boundaries of the OD. Another interesting remark observed in Figure A.1 is that following this approach does not degrade results in domains seen by the Teacher (or used also by the Student, if using f θ U ∪S ), statistically retaining most of the original performance. We also empirically showed that this behavior holds when using our segmentations for glaucoma detection using vCDR estimates (Figure C .2), reaching results in line or even superior to those obtained using manual segmentations.Our results using Students with and without access to S were inconclusive, as both seems to be accurate in specific cases, sometimes reporting statistically comparable results. Considering this scenario, one could potentially combine their results e.g. in an ensemble setting, to take advantage of their individual results. This Noisy Student framework is general enough to be applied with any backbone neural network architecture and/or domain generalization technique. In our experiments, we used a standard U-Net due to computational limitation, but other architectures with much more capacity could be leveraged to capture additional patterns Yi et al. (2023) . Furthermore, our process could be leveraged in the context of any of the alternative approaches evaluated, potentially boosting their performance in unseen domains with uncropped images. Finally, notice that this technique could be extrapolated to any other fundus image segmentation task that requires domain generalization, e.g. diabetic retinopathy lesion segmentation. Nevertheless, as with any semi-supervised learning strategy based on weak labels, it must be considered that a poor Teacher model can degrade performance due to confirmation bias (Kwon and Kwak, 2022) . Discarding this approach in advance would require to somehow approximate the performance in U, which is challenging due to the intrinsic lack of labels on it. This particular limitation is an active research topic now, with many approaches being introduced e.g. to predict areas of error in the pseudo-labels or to rank labels based on the uncertainty of the model (Kwon and Kwak, 2022; Albert et al., 2023; Khan et al., 2024) . Future work will focus on analyzing the potential contribution of these approaches in domain generalization results.  
 
 Appendix: We evaluated also the performance of our Student models f θ U and f θ U ∪S when using CHAKSU training sets as S for training a Teacher, and evaluating them all in all other sets as unseen domains. Results for OC and OD segmentation evaluated using DSC are depicted in Figure D.3. The Student model f θ U improves results of the Teacher in almost all unseen datasets, except for OC segmentation in DRISHTI, although differences are not statistically significant. In seen domains, the model performs comparable to the Teacher, meaning that it is still able to retain its original performance. The Student f θ U ∪S , on the other hand, significantly improves results for OD segmentation in all datasets, both seen and unseen, and OC results in seen domains. In the unseen domains, however, OC segmentations in DRISHTI, REFUGE and ORIGA are worse than those obtained with the Teacher, experiencing a notorious drop in REFUGE and ORIGA.  


Decision: This submission investigates the use of semi-supervised learning with noisy student models to enhance domain generalization in medical imaging. While the core technical novelty might be somewhat limited, the primary strength of the work lies in its extensive evaluation and insights into the effectiveness of this approach.

The reviews have a consensus on the limited novelty (builds on existing methods, limited novelty), but appreciate a strong evaluation (thorough evaluation across multiple datasets, demonstrating the value of the approach in a practical setting - a contribution for MIDL).

Despite the incremental technical novelty, the strong evaluation, the focus on domain generalization, and the interest expressed by reviews warrant an acceptance of this paper. It could present a valuable discussion for the MIDL community.

The authors are encouraged to strengthen the related work section to better contextualize their contribution in light of existing techniques.

For all these reasons, and with respect to the other submissions, the recommendation is towards Acceptance.

To PC: This is the best paper of my batch (my batch was possibly of a lower quality this year)

Reviews: review: paper_summary: paper_summary: This work challenges the improvement of generalisation ability in the segmentation of optic disc and cup in fundus images. The authors indicate the existence of unfair manual preprocessing for these segmentation in the previous works' experimental evaluations, which prevents the evaluation of actual generalisation ability of a trained model. After omitting the unfair preprocessing in the segmentation problem, the authors apply the Noisy Student Framework to improve a model’s generalisatoin ability. In the experiments, the authors performed comparative evaluation with several open-access datasets.
main_review: main_review: I thank the authors’ feedback. The authors gave feedback to my questions. Even though technical novelty is incremental, some of conference attendees might have interest on this work. Therefore, I rate this submission as borderline accept.
strength_weakness: strength_weakness: -The technical novelty of the proposed method is incremental

-The survey for weak-supervised learning with pseudo labels seems missing. 

-Table summarising the details of training, validation and testing datasets is necessary for high readability.
questions: questions: For practical application of OC/OD segmentations, how accurate segmentation models are required for each? 

If the given pseudo annotations on U are accurate, i.e. the teacher model can accurately segment samples in U, this means that almost all samples in U have similar patterns to S. In this case, why a student model can achieve more accurate segmentation than a teacher model even though many patterns used in the training are almost the same in two sets?

If the given pseudo annotations on U are inaccurate, i.e. the teacher model can’t accurately segment sample in U, this means that many samples in U have different patterns from S. In this case, why larger noisy annotated training set can lead to a more-accurate segmentation model? Noises can lead to inaccurate segmentation model, where errors caused by noises are accumulated. Is there  theoretically convincing explanation?
paper_summary: paper_summary: This paper presents a “noisy student” framework for semi-supervised learning for domain generalization. A teacher model is first trained on a combination of all domains and used to pseudo label unlabeled images for training a student model. A key premise of the work is that many existing DG methods rely on manually cropping around the optic nerve head, which reduces most sources of domain variations and thus limiting their ability in real-world use scenarios. Experiments demonstrated clear improvements of the proposed approach on OC and OD segmentations in uncropped images across different datasets.
main_review: main_review: After going through my fellow reviewers' comments and the authors' response, I will maintain my earlier rating based on its interesting concept, promising empirical results, and the gap of research it is addressing in the current literature.
strength_weakness: strength_weakness: The baselines included in the comparison were not adequately discussed, especially whether they are semi-supervised nor not.

The proposed work is of limited technical innovation as it builds primarily on a previously published “noisy student” framework. This is not a major concern though as it’s offset by the insight provided and experimental results shown.
questions: questions: See above.
paper_summary: paper_summary: In this study, the authors explored the concept of domain generalization for optic cup and disc segmentation in fundus images. They employed a semi-supervised technique with a noisy student to enhance the generalization ability. The authors utilized multiple datasets to train the algorithms and tested the model on studies that were not included in the training dataset, in order to evaluate the model's robustness.
main_review: main_review: The authors observed an improvement in performance on unseen datasets after swapping the datasets. The authors swapped the training set to CHAKSU, and the student model improved performance on all the unseen datasets. The paper includes new experiments that show the algorithm performs well on the new unseen domains.
strength_weakness: strength_weakness: 1. The authors chose REFUGE, DRISTHTI, and RIGA datasets for training and the rest for testing the algorithms. However, it is unclear how the algorithm performs if some of the datasets in training and testing sets are swapped.
2. There is limited novelty in the paper as the semi-supervised learning with noisy student is quite popular in the field of classification tasks and deep learning.
questions: questions: NA

Comments: comment: I thank the authors for their comprehensive response and for conducting additional experiments especially with swapping the datasets. Upon reviewing the results, I am convinced that the proposed method has significant advantages in domain generalization. I am pleased to raise my score accordingly.
We thank the reviewer for taking their time to evaluate our manuscript. Below we address each of their comments:

Weakness 1. The authors chose REFUGE, DRISTHTI, and RIGA datasets for training and the rest for testing the algorithms. However, it is unclear how the algorithm performs if some of the datasets in training and testing sets are swapped.

We made this experimental choice to leverage the predefined training, validation and test partitions provided in REFUGE, DRISHTI and RIGA, using the non-partitioned ones for testing. Nevertheless, we perform an additional, complementary experiment swapping CHAKSU as a training set and using all the other test sets as unseen domains. These results are provided and discussed in Appendix D. Except for OC segmentation in DRISHTI, we observe that our Student models perform statistically better than the Teacher model in all unseen domains, while being comparable or even better than the Teacher in seen domains.

Weakness 2. There is limited novelty in the paper as the semi-supervised learning with noisy student is quite popular in the field of classification tasks and deep learning.

We agree with the reviewer in that leveraging Noisy Students is popular in the literature. Nevertheless, we believe ours is the first article applying it in the field of domain generalization for effective OD/OC segmentation from full size images. Furthermore, as discussed in the article, our goal was not to introduce a novel method but to show that, due to an experimental problem in the existing literature, most of technically novel approaches were not able to perform better than an already established semi-supervised learning baseline. We believe that the revised version of our manuscript better reflects this idea, e.g. thanks to our modifications in Sections 1, 2.1 and 5.
We thank the reviewer for their feedback on our manuscript. We have modified our manuscript according to their comments and suggestions, namely:

Weakness 1. The baselines included in the comparison were not adequately discussed, especially whether they are semi-supervised nor not.

Unfortunately we were unable to include a proper description and analysis of each of the three considered approaches due to space constraints. Nevertheless, we modified the first paragraph in Section 4 to include this particular request from the reviewer, explaining that none of the compared methods follows a semi-supervised learning approach.

Weakness 2. The proposed work is of limited technical innovation as it builds primarily on a previously published “noisy student” framework. This is not a major concern though as it’s offset by the insight provided and experimental results shown.

We thank the reviewer for pointing out this fact. The goal of our manuscript is not to introduce a novel technical methodology but to demonstrate a significant experimental problem in the literature of domain generalization for OD/OC segmentation, and at the same time show that a simple approach such as the Noisy Student framework can surpass the existing literature if applied on an uncropped setting. Furthermore, ours is the first study showing that the Noisy Student approach not only improves results for out-of-distribution samples in seen domains as reported by Xie et al. 2020, but also improves in fully unseen domains.
We thank the reviewer for the time and feedback on our article, and for pointing out its strengths. We analyzed all the comments and recommendations from the reviewer and modified our original manuscript to clarify and complete all missing points. 
Below we answer the specific concerns requested by the reviewer for this rebuttal:

-R1. For practical application of OC/OD segmentations, how accurate segmentation models are required for each?

OD/OC segmentations are clinically used for calculating glaucoma biomarkers such as the vertical cup-to-disc ratio (vCDR) or the disc damage likelihood scale (DDLS). Both features require accurate input segmentations for a proper estimate, yet there is no consensus about which Dice or Hausdorff Distance values ensure proper estimates of these features. Nevertheless, to illustrate the contribution of our domain generalization strategy to the final performance estimates, we included Appendix C illustrating the improvements in glaucoma assessment with vCDR values derived from segmentations obtained by applying our domain generalization technique (ROC curves in Figure C.2). We also updated Sec. 4 (par. 5) and 5 (par. 2) to refer to this incorporation.

R2. If the given pseudo annotations on U are accurate, i.e. the teacher model can accurately segment samples in U, this means that almost all samples in U have similar patterns to S. In this case, why a student model can achieve more accurate segmentation than a teacher model even though many patterns used in the training are almost the same in two sets?

Incorporating stronger data augmentation as noise when training the (Noisy) Student yields a model that differs from the Teacher. Data augmentation further hardens the task of the Student, forcing it to learn new, unseen patterns to keep producing results as those obtained with the Teacher model. If accurate labels are obtained in U using the Teacher, then the Student can increase this accuracy by being trained from scratch on this set of further transformed images. Xie et al. 2020 reported that this methodology improved results on out-of-distribution samples within seen domains. In our experiments, we show that this also holds for new samples from unseen domains, resulting in a valid alternative for reaching better domain generalization. We included a few lines in the last paragraph of Sec. 2.1 to summarize this rationale, and extended Sec. 5 with a new paragraph in the end, in line also with the comment R3.

R3. If the given pseudo annotations on U are inaccurate, i.e. the teacher model can’t accurately segment sample in U, this means that many samples in U have different patterns from S. In this case, why larger noisy annotated training set can lead to a more-accurate segmentation model? Noises can lead to inaccurate segmentation model, where errors caused by noises are accumulated. Is there theoretically convincing explanation?

Indeed, existing literature reports that having trustworthy annotations by the Teacher model in the set of unlabelled samples U used for semi-supervised learning is fundamental to ensure a good performance (e.g. Kwon and Kwak, 2022). However, as no annotations are available for U, it is not feasible for us to estimate the accuracy of these particular segmentations. To raise awareness about this particular topic, we extended Sec. 5 by incorporating a paragraph in the end referring to this observation, including also relevant literature and potential solutions.

We also addressed comments and potential weaknesses indicated by the reviewer:

W1. The technical novelty of the proposed method is incremental

We agree that the novelty of our method is incremental, although we believe our article contributes to the state of the art from multiple perspectives. As indicated by the reviewer, we are reporting a significant experimental problem in the existing literature in domain generalization for OD/OC segmentation. Furthermore, we are extending the Noisy Student approach to this particular segmentation domain, illustrating that this already established method is able to outperform other more complex approaches from a domain generalization perspective. Finally, we are releasing our results publicly so that new methods in the literature can effectively compare their segmentations with ours in a reproducible manner.

W2. The survey for weak-supervised learning with pseudo labels seems missing.
 
We have extended Sec. 1 (par. 3) to the best of our ability considering the space limitation, including references to recent studies on weak supervision and semi supervised learning in the context of fundus image analysis applications.

W3. Table summarising the details of training, validation and testing datasets is necessary for high readability.

We included a table in the Appendix (Table .1) that summarizes the number of samples on each training, validation, and test partition for each dataset, indicating also which ones were considered as seen and unseen domains.