As neural network models, the authors created a collection of deep learning models for different types of classification tasks based on different deep learning architectures. For disease risk detection, the DenseNet201, and EfficientNet B4 architecture were utilized, whereas the DenseNet201, EfficientNet B4, ResNet152, and InceptionV3 were used for disease-label annotation of abnormal images (Huang et al., 2017; Tan and Le, 2019; Szegedy et al., 2016; He et al., 2016). The model training process utilized a transfer learning strategy based on ImageNet, a dynamic learning rate starting from $10^{-4}$ to a maximum decrease of $10^{-7}$, an early stopping, as well as a model checkpoint approach to reduce the risk of overfitting and a class, weighted focal loss implementation (Russakovsky et al., 2015; Lin et al., 2017). Instead of the traditional epoch definition as a single run through the complete training data, the authors defined an epoch as 250 iterations (batches). The training was performed for 290 epochs with a batch size of 24 images.

In addition to multiple model architectures and individual tasks, the authors applied a stratified multi-label 5-fold cross-validation for model training. This bagging technique allowed a more efficient usage of training data which resulted in a collection of in total 30 deep learning models. Finally, the pipeline ensembles the predictions of all deep learning models through an ensemble learning stacking technique based on binary logistic regression models.

The pipeline was implemented with their in-house developed framework AUCMEDI using two NVIDIA Titan RTX GPUs. To ensure full reproducibility, the authors published their complete code on GitHub: https://github.com/frankkramer-lab/riadd.aucmedi.

## Appendix A.8. chizu & arai & okada (Hidehisa Arai et al.)

Arai et al. proposed a standard ensemble framework to combine predictions of several CNNs. To make an effective ensemble, the authors made models differ from each other by choosing models with diverging complexity and training them on different image resolutions. For each of the 29 targets, different blending weights were adjusted to maximize scores on local cross-validation. The images in the given data set were taken by three different cameras, each with different aspect ratios and retinal reflections. Therefore, the authors trimmed the background using a circle crop and then aligned the aspect ratio. Figure A.11 shows an example of fundus images taken by three different cameras and the pre-processed images applied to them. Then, the images were resized to 512 × 512 pixels.

Because of data scarcity especially, for rare classes, data augmentation plays an important role in this competition. The authors applied four different data augmentations to the images. To make models robust to the change of position, scale, and rotation, the authors used random resized crop and translate, scale, and rotate, and to make models invariant to the change of hue and brightness, the authors used random brightness contrast and hue saturation. Several models were chosen from EfficientNet (Tan and Le, 2019) families and NFNet (Brock et al., 2021) families and trained with several image resolution. Focal Loss (Lin et al., 2017) was employed to train models. Focal Loss dynamically weights the contribution of each sample

<figure><img src="image_30.png" /><figcaption>Examples of the difference in the position of the visible retina between cameras</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Examples after pre-processing</figcaption></figure>

Fig. A.11. Example of fundus images and results of pre-processing on them.

on the loss function based on prediction difficulty, thus resulting in better performance on the prediction to the classes with fewer samples. Adam wrapped was used in SAM (Foret et al., 2020) optimizer. SAM simultaneously minimizes loss value and loss sharpness, so the models became more robust. The initial learning rate was set to 0.001 and changed using a learning rate scheduling technique called Cosine Annealing. Cosine Annealing changes learning rate $\eta_t$ to follow the value of the following function:

$$ \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min}) \left(1 + \cos\left(\frac{T_{cur}}{T_{max}}\pi\right)\right) \quad (A.10) $$

The authors set $\eta_{min} = 0.0$, $\eta_{max} = 0.001$, $T_{max} = 10$ and trained the models for 55 epochs. To ensemble the predictions of each model, weighted blending was used. Blending weights are adjusted for each target. The random search was applied to optimize the weights by maximizing the local cross-validation score for each target. Implementation of this work was done in PyTorch v1.7. The training was performed using an NVIDIA GeForce RTX 3090, two NVIDIA Tesla V100 with 32GB vRAM, and one NVIDIA GeForce RTX 2080. The source code for training is available at https://github.com/koukyo1994/riadd-competition and https://github.com/Chizuchizu/riadd. The source code for inference is available at https://www.kaggle.com/hidehisaarai1213/riadd-inference.

## References

* Abràmoff, M.D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J.C., Niemeijer, M., 2016. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investigative ophthalmology & visual science 57, 5200–5206.

* Ahn, J.M., Kim, S., Ahn, K.S., Cho, S.H., Lee, K.B., Kim, U.S., 2018. A deep learning model for the detection of both advanced and early glaucoma using fundus photography. PloS one 13, e0207982.
Ben-Baruch, E., Ridnik, T., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L., 2020. Asymmetric loss for multi-label classification. arXiv preprint arXiv:2009.14119.

Brock, A., De, S., Smith, S.L., Simonyan, K., 2021. High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171.

Chang, Y.S., Weng, S.F., Chang, C., Wang, J.J., Tseng, S.H., Wang, J.Y., Jan, R.L., 2016. Risk of retinal vein occlusion following end-stage renal disease. Medicine 95.

Chen, T., Xu, M., Hui, X., Wu, H., Lin, L., 2019. Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition. arXiv:1908.07325 [cs] arXiv:1908.07325.

Choi, J.Y., Yoo, T.K., Seo, J.G., Kwak, J., Um, T.T., Rim, T.H., 2017. Multi-categorical deep learning neural network to classify retinal images: A pilot study employing small database. PloS one 12, e0187336.

Christopher, M., Belghith, A., Bowd, C., Proudfoot, J.A., Goldbaum, M.H., Weinreb, R.N., Girkin, C.A., Liebmann, J.M., Zangwill, L.M., 2018. Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs. Scientific reports 8, 1-13.

Cuadros, J., Bresnick, G., 2009. Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening. Journal of diabetes science and technology 3, 509-516.

Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V., 2019. Autoaugmentation: Learning augmentation strategies from data, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123.

Decencière, E., Zhang, X., Cazuguel, G., Lay, B., Cochener, B., Trone, C., Gain, P., Ordonez, R., Massin, P., Erginay, A., et al., 2014. Feedback on a publicly distributed image database: the messidor database. Image Analysis & Stereology 33, 231-234.

DeVries, T., Taylor, G.W., 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.

Diaz-Pinto, A., Morales, S., Naranjo, V., Köhler, T., Mossi, J.M., Navea, A., 2019. Cnn's for automatic glaucoma assessment using fundus images: an extensive validation. Biomedical engineering online 18, 1-19.

Farnell, D.J., Hatfield, F., Knox, P., Reakes, M., Spencer, S., Parry, D., Harding, S.P., 2008. Enhancement of blood vessels in digital fundus photographs via the application of multiscale line operators. Journal of the Franklin institute 345, 748-765.

Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B., 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412.

Fu, H., Li, F., Orlando, J., Bogunovic, H., Sun, X., Liao, J., Xu, Y., Zhang, S., Zhang, X., 2020. Adam: Automatic detection challenge on age-related macular degeneration. IEEE Dataport.

Fumero, F., Alayón, S., Sanchez, J.L., Sigut, J., Gonzalez-Hernandez, M., 2011. Rim-one: An open retinal image database for optic nerve evaluation, in: 2011 24th international symposium on computer-based medical systems (CBMS), IEEE. pp. 1-6.

Gargeya, R., Leng, T., 2017. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124, 962-969.

Grassmann, F., Mengelkamp, J., Brandl, C., Harsch, S., Zimmermann, M.E., Linkohr, B., Peters, A., Heid, I.M., Palm, C., Weber, B.H., 2018. A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 125, 1410-1420.

Group, A.R.E.D.S.R., et al., 2001. A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins c and e, beta carotene, and zinc for age-related macular degeneration and vision loss: Ares report no. 8. Archives of ophthalmology 119, 1417-1436.

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al., 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316, 2402-2410.

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.

Hoover, A., Kouznetsova, V., Goldbaum, M., 2000. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging 19, 203-210.

Hu, J., Chen, Y., Zhong, J., Ju, R., Yi, Z., 2018a. Automated analysis for

retinopathy of prematurity by deep neural networks. IEEE transactions on medical imaging 38, 269-279.

Hu, J., Shen, L., Sun, G., 2018b. Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141.

Hu, J., Shen, L., Sun, G., 2018c. Squeeze-and-excitation networks.

Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708.

Keel, S., Wu, J., Lee, P.Y., Scheetz, J., He, M., 2019. Visualizing deep learning models for the detection of referable diabetic retinopathy and glaucoma. JAMA ophthalmology 137, 288-292.

Kendall, M.G., 1938. A new measure of rank correlation. Biometrika 30, 81-93.

Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Li, Z., He, Y., Keel, S., Meng, W., Chang, R.T., He, M., 2018. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology 125, 1199-1206.

Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P., 2017. Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, pp. 2980-2988.

Liu, R., Wang, X., Wu, Q., Dai, L., Fang, X., Yan, T., Son, J., Tang, S., Li, J., Gao, Z., et al., 2022. Deepdrid: Diabetic retinopathy—grading and image quality estimation challenge. Patterns 10, 100512.

MacGillivray, T., Trucco, E., Cameron, J., Dhillon, B., Houston, J., Van Beek, E., 2014. Retinal imaging as a source of biomarkers for diagnosis, characterization and prognosis of chronic illness or long-term conditions. The British journal of radiology 87, 20130832.

Mansour, R.F., 2018. Deep-learning-based automatic computer-aided diagnosis system for diabetic retinopathy. Biomedical engineering letters 8, 41-57.

Matsunaga, K., Hamada, A., Minagawa, A., Koga, H., 2017. Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble. arXiv preprint arXiv:1703.03108.

Misra, D., 2019. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681.

Mookiah, M.R.K., Chua, C.K., Min, L.C., Ng, E., Laude, A., 2013. Computer aided diagnosis of diabetic retinopathy using multi-resolution analysis and feature ranking frame work. Journal of Medical Imaging and Health Informatics 3, 598-606.

Murphy, K.P., 2012. Machine learning: a probabilistic perspective. MIT press.

Nielsen, K.B., Lautrup, M.L., Andersen, J.K., Savarimuthu, T.R., Grauslund, J., 2019. Deep learning-based algorithms in screening of diabetic retinopathy: A systematic review of diagnostic performance. Ophthalmology Retina 3, 294-304.

Niemeijer, M., Van Ginneken, B., Cree, M.J., Mizutani, A., Quellec, G., Sánchez, C.I., Zhang, B., Hornero, R., Lamard, M., Muramatsu, C., et al., 2009. Retinopathy online challenge: automatic detection of microaneurysms in digital color fundus photographs. IEEE transactions on medical imaging 29, 185-195.

Niu, X., Han, H., Shan, S., Chen, X., 2020. Multi-label Co-regularization for Semi-supervised Facial Action Unit Recognition. arXiv:1910.11012 [cs] arXiv:1910.11012.

OAM, B.A.H.P.D., Resnikoff, S., 2002. The role of optometry in vision 2020. Community Eye Health 15, 33-36.

ODIR, 2019. challenge. https://odir2019.grand-challenge.org/.

ODIR-2019, 2019. Peking university international competition on ocular disease intelligent recognition (odir-2019). IEEE Dataport.

Orlando, J.I., Fu, H., Breda, J.B., van Keer, K., Bathula, D.R., Diaz-Pinto, A., Fang, R., Heng, P.A., Kim, J., Lee, J., et al., 2020. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical image analysis 59, 101570.

Pachade, S., Porwal, P., Kokare, M., Giancardo, L., Mériaudeau, F., 2021a. Neten: Nested efficientnet and adversarial learning for joint optic disc and cup segmentation. Medical Image Analysis 74, 102253.

Pachade, S., Porwal, P., Thulkar, D., Kokare, M., Deshmukh, G., Sahasrabud-dhe, V., Giancardo, L., Quellec, G., Mériaudeau, F., 2021b. Retinal fundus multi-disease image dataset (rfmid): a dataset for multi-disease detection research. Data 6, 14.

PALM-2019, 2019. Palm challenge. IEEE Dataport.

Patton, N., Aslam, T.M., MacGillivray, T., Deary, I.J., Dhillon, B., Eikelboom, R.H., Yogesan, K., Constable, I.J., 2006. Retinal image analysis: concepts,
applications and potential. Progress in retinal and eye research 25, 99-127.

Pead, E., Megaw, R., Cameron, J., Fleming, A., Dhillon, B., Trucco, E., MacGillivray, T., 2019. Automated detection of age-related macular degeneration in color fundus photography: a systematic review. survey of ophthalmology 64, 498-511.

Peng, Y., Keenan, T.D., Chen, Q., Agrón, E., Allot, A., Wong, W.T., Chew, E.Y., Lu, Z., 2020. Predicting risk of late age-related macular degeneration using deep learning. NPJ digital medicine 3, 1-10.

Phan, S., Satoh, S., Yoda, Y., Kashiwagi, K., Oshika, T., 2019. Evaluation of deep convolutional neural networks for glaucoma detection. Japanese journal of ophthalmology 63, 276-283.

Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabud-dhe, V., Meriaudeau, F., 2018a. Indian diabetic retinopathy image dataset (idrid). URL: https://dx.doi.org/10.21227/H25W98, doi:10.21227/H25W98.

Porwal, P., Pachade, S., Kokare, M., Deshmukh, G., Son, J., Bae, W., Liu, L., Wang, J., Liu, X., Gao, L., et al., 2020. Idrid: Diabetic retinopathy-segmentation and grading challenge. Medical image analysis 59, 101561.

Porwal, P., Pachade, S., Kokare, M., Giancardo, L., Mériaudeau, F., 2018b. Retinal image analysis for disease screening through local tetra patterns. Computers in biology and medicine 102, 200-210.

Quellec, G., Lamard, M., Conze, P.H., Massin, P., Cochener, B., 2020. Automatic detection of rare pathologies in fundus photographs using few-shot learning. Medical image analysis 61, 101660.

Quellec, G., Lamard, M., Lay, B., Guilcher, A.L., Erginay, A., Cochener, B., Massin, P., 2019. Instant automatic diagnosis of diabetic retinopathy. arXiv preprint arXiv:1906.11875.

Radosavovic, I., Dollár, P., Girshick, R., Gkioxari, G., He, K., 2018. Data distillation: Towards omni-supervised learning. in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4119-4128.

Raghavendra, U., Fujita, H., Bhandary, S.V., Gudigar, A., Tan, J.H., Acharya, U.R., 2018. Deep convolution neural network for accurate diagnosis of glaucoma using digital fundus images. Information Sciences 441, 41-49.

Raju, M., Pagidimarri, V., Barreto, R., Kadam, A., Kasivajjala, V., Aswath, A., 2017. Development of a deep learning algorithm for automatic diagnosis of diabetic retinopathy, in: MEDINFO 2017: Precision Healthcare through Informatics. IOS Press, pp. 559-563.

Ridnik, T., Lawen, H., Noy, A., Ben Baruch, E., Sharir, G., Friedman, I., 2021. Tresnet: High performance gpu-dedicated architecture, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1400-1409.

Roychowdhury, S., Koozekanani, D.D., Parhi, K.K., 2013. Dream: diabetic retinopathy analysis using machine learning. IEEE journal of biomedical and health informatics 18, 1717-1728.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al., 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211-252.

Sechidis, K., Tsoumakas, G., Vlahavas, I., 2011. On the stratification of multi-label data, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, pp. 145-158.

Shibata, N., Tanito, M., Mitsuhashi, K., Fujino, Y., Matsuura, M., Murata, H., Asaoka, R., 2018. Development of a deep residual learning algorithm to screen for glaucoma from fundus photography. Scientific reports 8, 1-9.

Shortliffe, E.H., Blois, M.S., 2006. The computer meets medicine and biology: emergence of a discipline, in: Biomedical Informatics. Springer, pp. 3-45.

Sivaswamy, J., Krishnadas, S., Chakravarty, A., Joshi, G., Tabish, A.S., et al., 2015. A comprehensive retinal image dataset for the assessment of glaucoma from the optic nerve head analysis. JSM Biomedical Imaging Data Papers 2, 1004.

Son, J., Shin, J.Y., Kim, H.D., Jung, K.H., Park, K.H., Park, S.J., 2020. Development and validation of deep learning models for screening multiple abnormal findings in retinal fundus images. Ophthalmology 127, 85-94.

Staal, J., Abrámoff, M.D., Niemeijer, M., Viergever, M.A., Van Ginneken, B., 2004. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging 23, 501-509.

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016. Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826.

Szymański, P., Kajdanowicz, T., 2017. A network perspective on stratification of multi-label data, in: First International Workshop on Learning with Imbalanced Domains: Theory and Applications, PMLR. pp. 22-35.

Tan, J.H., Bhandary, S.V., Sivaprasad, S., Hagiwara, Y., Bagchi, A., Raghavendra, U., Rao, A.K., Raju, B., Shetty, N.S., Gertych, A., et al., 2018. Age-related macular degeneration detection using deep convolutional neural network. Future Generation Computer Systems 87, 127-135.

Tan, M., Le, Q., 2019. Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR. pp. 6105-6114.

Ting, D.S.W., Cheung, C.Y.L., Lim, G., Tan, G.S.W., Quang, N.D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I.Y., Lee, S.Y., et al., 2017. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. Jama 318, 2211-2223.

de Vente, C., Vermeer, K.A., Jaccard, N., van Ginneken, B., Lemij, H.G., Sánchez, C.I., 2021. Rotterdam eyepacs airogs train set. URL: https://doi.org/10.5281/zenodo.5793241, doi:10.5281/zenodo.5793241. The previous version was split into two records. This new version contains all data and the second record is deprecated.

Wagner, M., Müller-Stich, B.P., Kisilenko, A., Tran, D., Heger, P., Mündermann, L., Lubotsky, D.M., Müller, B., Davitashvili, T., Capek, M., et al., 2023. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the heichole benchmark. Medical image analysis 86, 102770.

Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., 2019. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing 338, 34-45.

Wang, J., Ju, R., Chen, Y., Zhang, L., Hu, J., Wu, Y., Dong, W., Zhong, J., Yi, Z., 2018. Automated retinopathy of prematurity screening using deep neural networks. EBioMedicine 35, 361-368.

WHO, 2019. World report on vision. World Health Organization.

Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K., 2017. Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500.

Yang, W., Xu, H., Yu, X., Wang, Y., 2015. Association between retinal artery lesions and nonalcoholic fatty liver disease. Hepatology international 9, 278-282.

Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D., 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
# BIAS Reporting Guideline

## Media checklist to be submitted along with challenge reports

<figure><img src="image_32.png" /></figure>

The Biomedical Image Analysis Challenges (BIAS) statement [1] has been officially approved by the Enhancing the Quality and Transparency Of health Research (EQUATOR) initiative as a guideline for reporting challenge results (click here for details). The present checklist represents a light version, comprising the parameters that represent the main body of the paper. **Please report the page number(s) (or N/A) for each parameter, such that the reviewers can quickly find relevant information and assess the comprehensiveness of the report.** Please cite [1] when referring to/using the BIAS guideline and/or the light version of the checklist as basis of your paper.

[1] Maier-Hein, L. et al. BIAS: Transparent reporting of biomedical image analysis challenges. Medical image analysis, 66, 101796 (2020).

<table><thead><tr><th>Section/ Topic</th><th>No</th><th>Checklist Item</th><th>Reported on page</th></tr></thead><tbody><tr><td colspan="4"><strong>Introduction</strong></td></tr><tr><td></td><td>4a</td><td>General introduction to the topic from a <strong>biomedical point of view</strong>.</td><td>2</td></tr><tr><td></td><td>4b</td><td>General introduction to the topic from a <strong>technical point of view</strong>.</td><td>2</td></tr><tr><td></td><td>4c</td><td>Concise statement of the <strong>primary challenge objective</strong>, including a <strong>statement of the task</strong>.</td><td>2,3</td></tr><tr><td colspan="4"><strong>Methods</strong></td></tr><tr><td rowspan="11">Challenge organization</td><td>5</td><td><strong>Representative name</strong> of the challenge including an <strong>acronym</strong> (if any).</td><td>2</td></tr><tr><td>6</td><td>Information on the <strong>organizing team</strong> (names and affiliations).</td><td>1</td></tr><tr><td>7</td><td>Intended <strong>submission cycle</strong> of the challenge, including information on whether/how the challenge has been/will be continued after the present study.<br><em>Examples</em>: One-time event with fixed submission deadline; repeated event with annual fixed submission deadline</td><td>4.5</td></tr><tr><td>8a</td><td><strong>Event</strong> (e.g. conference) that was <strong>associated</strong> with the challenge (if any).</td><td>4</td></tr><tr><td>8b</td><td><strong>Platform</strong> (e.g. grand-challenge.org) used to run the challenge.</td><td>5</td></tr><tr><td>8c</td><td><strong>URL</strong> for the challenge website (if any).</td><td>N/A</td></tr><tr><td>9a</td><td><strong>Allowed user interaction</strong> of the algorithms (e.g. only (semi-) automatic methods).</td><td>5</td></tr><tr><td>9b</td><td>Policy on the <strong>usage of training data</strong>.<br><em>Examples</em>: Training data have been restricted to the data provided by the challenge or to publicly available data including (open) pre-trained nets.</td><td>5</td></tr><tr><td>9c</td><td><strong>Participation policy for members of the organizers' institutes</strong>.<br><em>Example</em>: Members of the organizers' institutes could participate but were not eligible for awards.</td><td>N/A</td></tr><tr><td>9d</td><td><strong>Award policy</strong>, including details with respect to challenge prizes.</td><td>5,6</td></tr><tr><td>9e</td><td>Policy for <strong>results announcement</strong>.<br><em>Example</em>: The top three performing methods were announced publicly</td><td>6</td></tr><tr><td>9f</td><td><strong>Publication policy</strong>, including details on who of the participating teams' members qualified as author, whether participating teams could publish their own results separately, and whether an embargo time was defined.</td><td>N/A</td></tr><tr><td>10a</td><td>Method used for result submission, including a link to the <strong>submission instructions</strong> (if any).<br><em>Examples</em>: Docker container on the Synapse platform. Link to submission instructions: &lt;URL&gt;; algorithm output was sent to organizers via e-mail. Submission instructions were sent by e-mail.</td><td>5</td></tr><tr><td>10b</td><td>Information on the possibility for participating teams to <strong>evaluate</strong> their <strong>algorithms before submitting</strong> final results.</td><td>5</td></tr><tr><td>11</td><td><strong>Timetable</strong> for the challenge, including the release date(s) of the training/validation/test cases, registration date/period, submission date(s), associated workshop days, release date(s) of the results.</td><td>5,6</td></tr></tbody></table>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>
<table><tbody><tr><td></td><td>12</td><td>Indication whether <strong>ethics approval</strong> was necessary for the data. If yes, details on the ethics approval and URL or <strong>reference to the document</strong> of the ethics approval (if available).</td><td>N/A</td></tr><tr><td></td><td>13</td><td>Information on how the data can be used and distributed by the participating teams as well as others, including the explicit <strong>listing of the license</strong>.<br><em>Examples:</em> CC BY (Attribution); CC BY-SA (Attribution-ShareAlike)</td><td>5</td></tr><tr><td></td><td>14a</td><td>Information on the <strong>accessibility of the organizers' evaluation software</strong>, preferably, including a link to the code.</td><td>N/A</td></tr><tr><td></td><td>14b</td><td>Information on the <strong>accessibility of the participating teams' code</strong> in an analogous manner.</td><td>10 - 14</td></tr><tr><td></td><td>15</td><td>Information related to conflicts of interest, including information related to <strong>sponsoring/funding</strong> of the challenge and an explicit statement on who had <strong>access to the test case labels</strong> and when.</td><td>10</td></tr><tr><td></td><td>16</td><td><strong>Contributions</strong> of all authors to the paper (preferably in the appendix).</td><td>N/A</td></tr><tr><td rowspan="10">Mission of the challenge</td><td>17</td><td><strong>Main field(s) of application</strong> that the participating algorithms target.<br><em>Examples:</em> Diagnosis; intervention planning; screening; training</td><td>4</td></tr><tr><td>18</td><td><strong>Task category/-ies</strong>.<br><em>Examples:</em> Classification; detection; segmentation; reconstruction</td><td>4</td></tr><tr><td>19a</td><td><strong>Target cohort</strong>, i.e. the subjects/objects from whom/which the data would be acquired in the final biomedical application.</td><td>4</td></tr><tr><td>19b</td><td><strong>Challenge cohort</strong>, i.e. the subject(s)/object(s) from whom/which the challenge data was acquired.</td><td>4</td></tr><tr><td>20</td><td><strong>Imaging technique(s)</strong> applied in the challenge.</td><td>4</td></tr><tr><td colspan="3">Additional information given along with the images, corresponding to the...</td></tr><tr><td>21a</td><td><strong>... image data</strong> (e.g. tumor volume).</td><td>4</td></tr><tr><td>21b</td><td><strong>... patient</strong> in general (e.g. gender, medical history).</td><td>N/A</td></tr><tr><td>21c</td><td><strong>... acquisition process</strong> (e.g. calibration data for an image modality).</td><td>4</td></tr><tr><td>22a</td><td><strong>Data origin</strong>, i.e. the region(s)/part(s) of subject(s)/object(s) from whom/which the image data would be acquired in the final biomedical application.</td><td>4</td></tr><tr><td>22b</td><td><strong>Algorithm target</strong>, i.e. the structure(s)/subject(s)/object(s)/ component(s) that the participating algorithms have been designed to focus on.</td><td>4</td></tr><tr><td>23</td><td><strong>Property/-ies of the algorithms to be optimized</strong> to perform well in the challenge. Should be reflected in the metrics (parameter 29) and ranking (parameter 30) applied.<br><ul><li><em>Example 1:</em> Find liver segmentation algorithm that processes CT images of a certain size in less than a minute with an error that reflects inter-rater variability of experts.</li><li><em>Example 2:</em> Find lung tumor detection algorithm with high sensitivity for mammography images.</li></ul></td><td>4</td></tr><tr><td rowspan="7">Challenge data sets</td><td>24a</td><td><strong>Device(s)</strong> used to acquire the challenge data, including details on the device(s) used to acquire the imaging data as well as information on additional devices used for performance assessment.</td><td>4</td></tr><tr><td>24b</td><td>Relevant details on the imaging process/<strong>data acquisition</strong> for each acquisition device (e.g. image acquisition protocol(s)).</td><td>4</td></tr><tr><td>24c</td><td><strong>Center(s)/institute(s)</strong> in which the data was acquired and/or <strong>data providing platform/source</strong> (e.g. previous challenge).</td><td>4</td></tr><tr><td>24d</td><td>Relevant <strong>characteristics</strong> (e.g. level of expertise) <strong>of the subjects</strong> (e.g. surgeon)/objects (e.g. robot) involved in the data acquisition process (if any).</td><td>N/A</td></tr><tr><td>25a</td><td>Information on the meaning of one <strong>case</strong> in this challenge. A case encompasses all data that is processed to produce one result that is then compared to the corresponding reference result (i.e. the desired algorithm output).<br><em>Example:</em> Training and test cases both represented a CT image of a human brain. Training cases had a weak annotation (tumor present or not and tumor volume) while the test cases were annotated with the tumor contour.</td><td>4,5</td></tr><tr><td>25b</td><td><strong>Total number</strong> of cases as well as the number of training, validation and test cases.</td><td>5</td></tr><tr><td>25c</td><td><strong>Justification why a total number of cases and the specific proportion of training, validation and test cases was chosen.</strong></td><td>5</td></tr><tr><td>25d</td><td><strong>Further important characteristics</strong> of the training, validation and test cases (e.g. class distribution in classification tasks) and justification of choice.</td><td>5</td></tr></tbody></table>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>
<table><tbody><tr><td></td><td>26a</td><td>Method for determining the reference annotation. If human annotation was involved, state the number of annotators.</td><td>4</td></tr><tr><td></td><td>26b</td><td>Instructions given to the annotators (if any) prior to the annotation, preferably, including a link to the annotation protocol.</td><td>4</td></tr><tr><td></td><td>26c</td><td>Details on the subject(s)/algorithm(s) that annotated the cases (e.g. information on level of expertise such as number of years of experience, medically-trained or not).</td><td>4</td></tr><tr><td></td><td>26d</td><td>Method(s) used to merge multiple annotations for one case (if any).</td><td>4</td></tr><tr><td></td><td>27</td><td>Method(s) used for pre-processing the raw training data before it is provided to the participating teams (if any).</td><td>N/A</td></tr><tr><td></td><td>28a</td><td>Most relevant possible error sources related to the image annotation, preferably including an estimate of the magnitude (range) of these errors, using inter-and intra-annotator variability, for example.</td><td>N/A</td></tr><tr><td></td><td>28b</td><td>In an analogous manner, other relevant sources of error.</td><td>N/A</td></tr><tr><td rowspan="6">Assessment method</td><td>29a</td><td>Metric(s) used to assess a property of an algorithm. Should reflect the desired properties described in parameter 23. Which metric(s) were used to compute the ranking(s) (if any).</td><td>6,7</td></tr><tr><td>29b</td><td>Justification why the metric(s) was/were chosen, preferably with reference to the biomedical application.</td><td>6</td></tr><tr><td>30a</td><td>Method used to compute a performance rank for all submitted algorithms, including how results were obtained per case and metrics are aggregated to get a final ranking.</td><td>7</td></tr><tr><td>30b</td><td>Method(s) used to manage submissions with missing results on test cases.</td><td>N/A</td></tr><tr><td>30c</td><td>Justification why the described ranking scheme(s) was/were used.</td><td>N/A</td></tr><tr><td>31a</td><td>Details for all statistical methods, including description of the missing data handling, details about assessment of variability of rankings, or indication of any software product used for data analysis.</td><td>N/A</td></tr><tr><td></td><td>31b</td><td>Justification why the described statistical method(s) was/were used.</td><td>N/A</td></tr><tr><td colspan="3"><strong>Results</strong></td><td></td></tr><tr><td rowspan="10">Challenge outcome</td><td>32a</td><td>Summarizing information on the number of registrations.</td><td></td></tr><tr><td>32b</td><td>Summarizing information on the number of participating teams that provided valid submissions.</td><td>7</td></tr><tr><td>32c</td><td>Summarizing information on the number of participating teams that the paper refers to (with justification).</td><td>7</td></tr><tr><td>33a</td><td>Team identifiers for the participating teams that are included in the paper.</td><td>6.7</td></tr><tr><td>33b</td><td>Method description including parameter instantiation and/or a reference/URL to a document containing this information for the participating teams that are included in the paper.</td><td>10 - 14</td></tr><tr><td>34</td><td>Raw and/or aggregated metric values (including measure of variability) for all participating teams and each metric and the numbers of test set submissions for each participating team.</td><td>8,9</td></tr><tr><td>35a</td><td>Ranking(s) (if any) including the number of test set submissions for each team.</td><td>8</td></tr><tr><td>35b</td><td>Results of the statistical analyses.</td><td>N/A</td></tr><tr><td>36</td><td>Results of further analyses (if applicable), e.g. related to combining algorithms via ensembling, inter-algorithm variability, common problems/biases of the submitted methods, or ranking variability.</td><td>N/A</td></tr><tr><td colspan="3"><strong>Discussion</strong></td><td></td></tr><tr><td>37</td><td>Main results of the challenge.</td><td>7</td></tr><tr><td>38</td><td>(Expected) Biomedical and technical impact of the challenge in the context of the state of the art with reference to the challenge motivation (parameter 1).</td><td>10</td></tr><tr><td>39a</td><td>Detailed discussion and conclusion whether the task is now solved in a satisfactory way (e.g. the remaining errors are comparable to inter-annotator variability).</td><td>10</td></tr><tr><td>39b</td><td>Detailed analysis of individual cases, in which the majority of algorithms performed poorly (if any).</td><td>N/A</td></tr><tr><td>39c</td><td>Discussion on advantages and disadvantages of the submitted methods. Include time and memory consumption comparison if time and memory were not among the metrics.</td><td>8,9</td></tr><tr><td>40</td><td>Limitations related to the challenge design and execution.</td><td>10</td></tr></tbody></table>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>
<table><tr><td>41</td><td>Recommendations for future work and maintenance plans for the challenge and its website (if any).</td><td>IU</td></tr><tr><td>42</td><td>Concise conclusion based on the results of the study.</td><td>I</td></tr></table>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

BIAS

BIAS Reporting Guideline