SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, we first measure the vessel diameters automatically via "Vessel Diameters" plugin in ImageJ. Then, we divide the arteries into multiple small vessel segments based on the diameters of the vessels. Based on the largest diameter among these artery segments, we define four levels of widths according to specific ratios. More specifically, a vessel segment within the range of  $0 - 25\%$  of the largest diameter is categorized as level 0. Similarly, levels 1, 2, and 3 correspond to vessel widths in the ranges of  $25\% - 50\%$ ,  $50\% - 75\%$ , and  $75\% - 100\%$  of the largest diameter, respectively. Afterward, we obtain four-class masks for arteries based on vessel widths. The same operation is also applied to veins. After the automatic processing, clinicians will validate the quality of the fine-grained
2https://imagej.net/software/imagej/
segmentation masks. This process ultimately yields eight-class masks for both arteries and veins (see Fig. 2 (d)). Those masks significantly enrich the granularity of our dataset.
3.3.2 Temporal localization
Existence of SVP: Based on the results of data cleaning and preprocessing, we utilize the stabilized videos and further annotate the dynamic state of vessel pulsations. Spontaneous retinal Venous Pulsation (SVP) plays a crucial role as a biomarker in retina assessments. Specifically, SVP is characterized by rhythmic pulsations evident in the central retinal vein and its branches, typically observable within the optic disc region (ODR) of the retinas. The absence of SVP holds substantial clinical significance, as it is correlated with certain pathologies. For example, the absence of SVP is associated with progressive glaucoma [53], and it is indicative of increased intracranial pressure [52]. Considering the requirement of specialized knowledge, we have invited multiple clinicians and finished the annotation of SVP presence or absence for each video in our dataset. Once the annotation process completes, we obtain 335 "SVP-present" videos and 300 "SVP-absent" videos respectively. This annotation establishes a fundamental task of SVP detection, facilitating further analysis and investigation on the relationship between SVP and eye diseases.
Temporal duration of SVP: After annotating the existence of SVP in the stabilized videos, some "SVP-present" videos may not contain SVP throughout the whole video. This means in some frames SVP is not visible. Using these videos to train an SVP classification model would suffer ambiguity especially when an entire video cannot be fed into a neural network. Therefore, we further provide temporal emergence annotations of SVP by indicating the starting and ending frames of retinal vessel fluctuation (see Fig. 2 (e)). The detailed duration of SVP serves two purposes: it acts as a valuable signal to improve the performance of SVP detection tasks and concurrently sets a new task for SVP temporal localization. We obtain videos in three distinct groups: 156 videos containing intermittent SVPs, 179 videos demonstrating persistent SVPs, and the remaining 300 videos without SVPs. These temporal annotations allow us to better understand retinal vessel dynamics.
"Peak" and "Trough" annotations of SVP: As discussed above, SVP reflects the temporal dilation and contraction in retinal vessels. The state with maximal dilation is characterized as "peak", whereas those with maximal contraction are termed "trough". Here, we select frames corresponding to the "peak" and "trough" states from each "SVP-present" video. Subsequently, we generate corresponding masks for these selected frames, yielding a total of 670 annotated masks. This annotation allows us to quantitatively measure the extent of pulsations and occurring positions of vessel pulsations.
3.4 Data Protocols
Data split: When partitioning our dataset for training and evaluation, we also take into account the similarity of the recorded videos of the same person. Specifically, we ensure that videos captured from the same patient are allocated to the same subset during the partitioning process. This strategy aims to decrease the similarity between training data and testing data and thus minimizes performance bias. In practice, we divide the data based on patient IDs. In this manner, the same patient's videos will not appear simultaneously in both the training and testing sets. Then, we select 517 videos from the 635 videos for training and validation, and the rest 118 videos are used for testing. We also cross-validate a method with three different data splits and will release the dataset and data splits.
Metrics: Based on our annotations, we can conduct tasks in two major categories. (1) Retinal vessel segmentation metrics: Since the binary, general artery-vein and fine-grained vessel segmentation tasks are essentially semantic segmentation, we adopt the mean Intersection over Union (mIoU), mean Accuracy (mAcc), and macro-averaged F1-score (mFscore) to evaluate the performance of models on our dataset. Note that, mFscore is the average of F1-score by treating all classes equally and the F1 score is also known as the DICE score in binary classification or segmentation tasks<sup>3</sup>, the mFscore thus essentially equals the mean of DICE scores for segmentation tasks. (2) SVP recognition and temporal localization metrics: SVP recognition is to classify whether SVP exists in a video. SVP localization task is to identify the time period where SVP appears in a video. We adopt the Accuracy (Acc), Area Under the Receiver Operating Characteristic Curve (AUROC), and Recall for SVP recognition. The frame-mAP (F-mAP), video-mAP (V-mAP) [27] under IoU threshold 0.5 and mean Intersection over Union (mIOU) are adopted for the task of SVP localization.
<sup>3</sup>https://en.wikipedia.org/wiki/F-score
Table 2: Segmentation results of different methods on our RVD dataset. "DLV3" denotes "DeepLabV3" and "M2F" denotes "Mask2Former".
Method	Backbone	mIoU	Binary mAcc	mFscore	General Artery-Vein	Fine-grained Artery-Vein
mIoU	mAcc	mFscore	mIoU	mAcc	mFscore
DLV3 [11]	UNet [63]	66.59±0.6	72.92±1.0	76.67±0.9	36.54±1.5	37.80±1.9	55.85±1.2	12.81±0.4	14.03±0.7	19.16±0.5
ResNet50 [30]	62.15±0.6	65.84±1.1	70.72±0.7	47.92±0.1	51.85±0.2	57.21±0.1	17.30±0.3	19.59±0.7	24.67±0.5
ResNet101	62.89±0.7	71.45±1.0	78.45±1.3	56.60±0.7	51.99±0.9	57.10±1.0	18.13±0.5	21.05±0.6	24.72±0.6
M2F [12]	ResNet50	70.27±0.3	77.65±0.2	79.51±0.3	57.60±0.1	66.80±0.2	69.06±0.1	24.88±0.8	32.58±1.5	34.11±1.1
ResNet101	70.74±0.3	78.78±0.3	79.99±0.3	59.43±1.7	68.58±1.5	70.73±1.7	31.62±4.0	41.96±5.1	42.89±6.4
Swin-T [45]	70.94±0.7	78.87±0.7	80.14±0.7	58.58±1.2	69.10±2.6	71.39±0.1	28.14±3.2	36.93±4.3	38.36±4.2
Swin-S	70.27±0.3	77.55±0.1	80.14±0.7	57.60±0.1	66.14±0.8	69.04±0.1	23.41±0.1	30.44±0.5	32.60±0.6
Swin-B-1k	71.20±0.6	78.74±0.9	80.38±0.6	58.66±0.3	68.43±0.4	71.29±0.8	25.30±0.3	34.50±1.0	34.81±0.4
Swin-B-22k	70.99±0.1	78.85±0.6	80.19±0.1	56.12±1.3	68.31±0.2	70.14±0.3	25.26±0.2	33.88±0.3	34.88±0.2
Swin-L	74.09±3.0	78.70±0.9	80.63±0.4	60.49±1.7	70.34±1.9	71.99±1.6	24.91±0.2	33.04±0.7	34.46±0.3
In this section, we employ state-of-the-art (SOTA) segmentation methods to examine the contributions and challenges of our newly curated RVD as well as establish a new benchmark for the dynamic vessel segmentation and localization tasks. As our data are collected from hand-held fundus imaging devices, we also investigate whether domain gaps between our dataset and existing ones which are captured by benchtop based devices.
4.1 Overall Results
We first focus on spatial segmentation tasks on our dataset. We conduct experiments of binary vessel segmentation, general artery-vein segmentation, and fine-grained artery-vein segmentation, respectively. We employ several popular used segmentation methods, including FCN [66], DeepLabV3 [11], Segmentor [73], and Mask2Former [12]. We also apply different backbones to these segmentation methods. The adopted backbones involves convolutional UNet [63], ResNet [30], ViT [20], and Swin Transformer [45]. We use the pre-trained parameters as initialization and train the networks on our training set. Due to the space limit, we only present the results of DeepLabV3 and Mask2Former in Table 2 and others are reported in the Appendix.
Even for the binary segmentation task, the highest mIoU results barely reach around  $70\%$ . For the more complex fine-grained artery-vein segmentation, the mIoU values further decline to approximately  $25\%$ . Some visual results are shown in Fig. 3. It is observed that current methods in general struggle to localize thin vessels. However, these thin vessels always play an important role in reflecting some diseases, e.g., atherosclerosis [80]. The performance of the SOTA methods implies the challenges of our dataset, but this also emphasizes the potential of our dataset for future studies.
We also conduct experiments on temporal SVP recognition and localization with our provided annotations. In SVP recognition, we train the models to predict whether SVP exists in a video. In SVP localization, we train the models to identify the time period where SVP appears in a video. We employ the models of LRCN [19], I3D [8], X3D [24],
Table 3: Performance of SVP recognition and localization.
Method	Acc	Recognition AUROC	Recall	F-mAP	Localization V-mAP	mIOU
LRCN [19]	52.68	56.79	45.00	64.62	59.06	50.62
I3D [8]	60.71	61.83	61.67	67.49	60.63	51.89
X3D [24]	52.68	51.60	75.00	61.12	52.60	50.53
TSN [77]	50.89	64.39	30.00	67.60	56.85	50.89
VTN [55]	58.93	65.58	86.67	68.08	57.64	51.25
TSN [77], and VTN [55]. We use the metrics in Section 3.4 and report the results in Table 3. To the best of our knowledge, we are the first to provide data and annotations for SVP recognition and localization. However, it is found that existing methods fail to recognize and localize SVP precisely on our real-clinic video data. For example, in SVP localization task, VTN only achieves  $51.25\%$  mIoU, which might not meet the needs of real-world applications. Such results indicate that more specific-designed methods are highly demanded.
4.2 Domain Gaps between RVD and Existing Datasets
We conduct a two-way evaluation process where models trained on our dataset are tested on previous datasets and models trained on existing datasets are also evaluated on our dataset. First, to evaluate
Figure 3: Visualization in the binary, general artery-vein, and fine-grained artery-vein segmentation.
Table 4: Evaluation of domain gaps between different datasets.
Method	Backbone	Binary Segmentation	General Artery-Vein
RVD ↓ C-DB.	C-DB. ↓ RVD	RVD ↓ DRI.	DRI. ↓ RVD	RVD ↓ HRF	HRF ↓ RVD	RVD ↓ FIVES	FIVES ↓ RVD	RVD ↓ RITE	RITE ↓ RVD
DLV3	UNet ResNet50 ResNet101	70.23±0.2 71.56±0.3 71.89±0.4	62.17±0.4 63.41±0.2 63.59±0.2	65.21±0.2 66.34±0.1 66.72±0.3	62.21±0.3 63.04±0.2 63.98±0.1	70.89±0.3 71.27±0.2 71.98±0.3	56.81±0.5 57.76±0.4 58.17±0.9	69.55±0.2 68.73±0.3 64.81±0.2	68.92±0.4 64.95±0.2 65.26±0.3	51.05±0.2 51.84±0.3 52.56±0.2	28.31±0.4 29.87±0.3 30.30±0.4
M2F	ResNet50 ResNet101 Swin-T Swin-S Swin-B-1k Swin-B-22k Swin-L	72.23±0.2 73.57±0.3 74.97±0.2 75.21±0.9 74.93±0.2 76.95±0.3 76.86±1.3	65.71±0.3 67.20±0.1 67.59±0.3 67.98±0.2 68.13±0.1 70.04±0.2 70.47±0.2	67.03±0.2 67.57±0.2 69.99±0.3 69.93±0.1 71.21±0.2 71.79±0.3 73.02±0.1	66.74±0.1 67.25±0.3 67.74±0.3 67.51±0.3 67.64±0.3 68.36±0.3 67.72±0.4	73.05±0.2 73.76±0.3 74.71±0.1 74.83±0.3 73.97±0.2 76.03±0.3 78.67±0.2 76.15±0.3	65.65±0.2 65.27±0.2 65.69±0.3 66.89±0.4 67.40±0.3 72.87±0.6 73.30±0.2 73.74±0.3 73.56±0.9	69.89±0.4 69.39±0.3 69.64±0.1 69.53±0.8 69.57±0.7 69.34±0.4 69.64±1.5	54.65±0.2 55.23±0.3 55.65±0.4 56.47±0.2 57.04±0.4 52.99±0.4 57.05±0.2 52.84±0.5 57.28±0.4 52.98±0.3
binary vessel segmentation performance, we include the following datasets: CHASE DB1 (C-DB.), DRIVE (DRI.), HRF, and STARE (STA.). Then, we also conduct general artery-vein segmentation on the RITE dataset. The results are shown in Table 4. Note that existing datasets do not support fine-grained eight-class segmentation, and thus we did not test our data in this setting. Due to the domain gap, the models suffer performance drop. The results also indicate that our dataset provides unique data samples. The visualization is illustrated in the Appendix. From our experimental results above, we can tell that the retinal vessel segmentation is far from being solved. Our RVD dataset will serve as a valuable resource, motivating future explorations in retinal vessel segmentation.
In this work, we propose the first video-based retinal vessel segmentation dataset by employing handheld devices for data acquisition. Our dataset significantly complements the current benchtop-based datasets for retinal vessel segmentation and enables SVP detection and localization. More importantly, it offers rich annotations for both spatial vessel segmentation and temporal SVP localization. In comparison to existing datasets, our dataset is not only the largest scale one with the most diverse annotations but also more challenging. The domain gaps between our dataset and existing ones allows researchers to investigate how to minimize the domain gaps in vessel segmentation. Therefore, our curated dataset RVD is valuable for retinal vessel segmentation and would facilitate the clinical diagnosis of eye-related diseases.
This research is funded in part by ARC-Discovery grant (DP220100800) and ARC-DECRA grant (DE230100477). We thank all anonymous reviewers and ACs for their constructive suggestions.
[85] Mo Zhang, Fei Yu, Jie Zhao, Li Zhang, and Quanzheng Li. Befd: Boundary enhancement and feature denoising for vessel segmentation. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23, pages 775–785. Springer, 2020.
[86] Yishuo Zhang and Albert CS Chung. Deep supervision with additional labels for retinal vessel segmentation task. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 83-91. Springer, 2018.
[87] Yitian Zhao, Jianyang Xie, Huaizhong Zhang, Yalin Zheng, Yifan Zhao, Hong Qi, Yangchun Zhao, Pan Su, Jiang Liu, and Yonghuai Liu. Retinal vascular network topology reconstruction and artery/vein classification via dominant set clustering. IEEE transactions on medical imaging, 39(2):341-356, 2019.
[88] Chao Zhou, Xiaogang Zhang, and Hua Chen. A new robust method for blood vessel segmentation in retinal fundus images based on weighted line detector and hidden markov model. Computer methods and programs in biomedicine, 187:105231, 2020.
[89] Lei Zhou, Qi Yu, Xun Xu, Yun Gu, and Jie Yang. Improving dense conditional random field for retinal vessel segmentation by discriminative feature learning and thin-vessel enhancement. Computer methods and programs in biomedicine, 148:13-25, 2017.
A Broader Impacts Statement
The creation and introduction of our video-based retinal vessel dataset (RVD) have profound effects on both the research community and the broader healthcare domain.
Research Impacts. By providing a comprehensive dataset with both spatial and temporal dimensions, the RVD significantly facilitates the analysis of retinal vessel segmentation and leads to improved understanding and modeling of ocular diseases. By incorporating dynamic video data, this dataset offers a broader and richer scope of retinal information than the traditional static image-based datasets. It fosters new research opportunities in retinal vessel segmentation that emphasize dynamic temporal characteristics and more granular vessel details. The introduced domain gaps with handheld devices may promote the development of robust and adaptable models, thereby advancing state-of-the-art image analysis methods. By providing this dataset as a resource for the research community, we hope to facilitate widespread collaboration and accelerate the exploration in retinal disease detection, understanding, and diagnosis.
Societal Impacts. As adopted in our RVD dataset, smartphone-based devices have the potential to democratize retinal vessel examination by making it more accessible and less reliant on expensive, specialized ophthalmic equipment. Such an enhancement in accessibility consequently leads to earlier detection and prevention of a spectrum of ocular diseases. Handheld devices potentially increase equity in healthcare services.
Limitations. Although our RVD is the largest dataset for retinal vessel segmentation to date (635 videos with annotations), its scale is still limited compared to other datasets in computer vision and thus our dataset can be further extended in the future. Compared to the data collected with bench-top devices, the original videos captured with handheld devices involve more realistic factors such as operator techniques, varying lighting conditions, and eye movement of the patient. These factors will require more sophisticated data cleaning and preprocessing strategies to avoid the degraded quality and reliability of the data.
B Building RVD
Equipment. We use self-designed handheld devices for data collection. Compared to bench-top devices which are cumbersome and expensive, the devices we adopt here are lightweight and portable. Our devices are much cheaper and easier to access. Our devices are built by connecting a smartphone to the fundus camera lens via an optical tube (see Fig. 1).
Operation details. Clinicians use the handheld devices in Fig. 1 (b) to amass a collection of videos during eye health examinations. To accommodate handheld operation, each video lasts at least 0.5 seconds and does not exceed 25 seconds. The process initiates with the random selection of participants and their consent prior to recording. Then, either the left eye, the right eye, or both eyes are randomly selected for video recording. This method ensures that our dataset comprises both healthy and diseased eyes. Each participant joins in the recording process either once or multiple times. In our dataset, we try to eliminate the potential bias towards specific eye conditions and ensure a broad representation of various ocular health states from different people.
Statistics of our RVD. We show some statistical characteristics of the patients in Fig. 3. Age distribution among patients exhibits a Gaussian-like one, and more videos are collected from males rather than females in our dataset. From Fig. 3 (c), we conclude that most of the videos are collected from clinic 0 and clinic 3. Finally, the videos collected from the left eye and right eye are nearly balanced, as shown in Fig. 3 (d). More detailed statistical information on the videos can be accessed on our website in the future.
Characteristics of our videos. We show more samples in Fig. 2 to illustrate the diversity of our collected videos, e.g., samples with different sizes of optic disc regions (ODR), different illumination, and different vessel density. The ODR contains most of the vessels in the retina and the size of ODR is vital in retinal vessel segmentation. Different illumination and vessel density will also make our dataset more challenging by increasing the variety of samples.
(a) Bench-top device
(b) An example of our handheld devices
Figure 1: (a): Bench-top device, which is cumbersome and expensive; (b): Our handheld device, which is lightweight and portable. It is built by connecting a smartphone to the fundus camera lens via a optical tube.
(a)
Figure 2: Diversity of our collected videos. (a): Samples with different sizes of ODR; (b): Different illumination; (c): Different vessel density.
Statistics of diseased and normal eyes. As indicated in Table 1, there are 317 videos of diseased eyes and 318 videos of normal eyes. Although the number of videos collected from different clinics varies, e.g., only dozens of videos are collected from clinics Q and R, while more than 200 videos are collected from clinics P and
S, the diseased and normal eye videos for each clinic are balanced.
Table 1: The statistics of diseased and normal eyes.
Clinics	P	Q	R	S	Total
Diseased	132	18	16	151	317
Normal	140	15	42	121	318
C Experimental settings
C.1 Implementation Details.
We implement the benchmark in PyTorch using the open-sourced MMSegmentation [1]. For all methods, we leverage the default settings of each method in MMSegmentation and implement them on 4090 GPUs. We train each model for 40,000 iterations and select the checkpoints with the best validation results. To ensure the accuracy and reliability of final results, cross-validation is employed across our experiments.
Figure 3: Statistics of our dataset. (a): The age distribution of the participants; (b): The ratio of males and females; (c): The number of videos and patients in each clinic; (d): The number of videos collected from the left eye and right eye.
Figure 4: Visualization in the binary, general artery-vein, and fine-grained artery-vein segmentation.
C.2 More Results of Models
In Table 2, we present the segmentation results produced by FCN, PSPNet, and Segmenter. Each method leverages different backbones, e.g., UNet, ResNet, and ViT. The models are trained and tested with binary vessel masks, general artery-vein masks, and fine-grained artery-vein masks. We observe a consistent pattern that even though the methods have achieved the best results with binary masks, the highest mIoU is under 70. These results underscore the difficulties of existing methods in dealing with our dataset. Such results show the challenges inherent to our dataset and imply the potential of our dataset to inspire future studies. In Fig. 4, we show the complete segmentation results of FCN, PSPNet, DeeplabV3, Segmenter, and Mask2Former in our dataset.
C.3 Better domain gap analysis
In this section, we investigate the potential domain gaps in our RVD dataset. Such variations in distribution could arise from multiple factors: (i) Different clinical conditions; (ii) Variations of vasculature due to the presence and absence of disease and (iii) Different devices used for video capture. Furthermore, we consider the evaluation of baseline models on a third dataset.
C.3.1 Test across disease and normal videos
We evaluate our best-performing model Mask2Former with the Swin Transformer backbone on diseased and normal retinal videos. The binary segmentation results are reported in Table 3. The segmentation results on both diseased and normal eye videos are close, indicating the domain discrepancy between the diseased and normal videos is marginal in the context of retinal vessel segmentation.
Table 2: Segmentation results of "FCN", "PSPNet", and "Segmenter" on our RVD dataset.
Method	Backbone	mIoU	Binary mAcc	mFscore	General Artery-Vein	Fine-grained Artery-Vein
mIoU	mAcc	mFscore	mIoU	mAcc	mFscore
FCN [5]	UNet [4]	67.82±0.6	73.22±0.6	77.08±0.5	38.29±1.0	39.85±0.9	64.59±0.7	13.47±0.5	14.88±0.9	19.95±1.2
ResNet50 [3]	62.12±0.5	66.05±0.7	70.76±0.3	49.22±0.1	53.93±0.2	58.99±0.1	18.38±0.6	21.41±0.8	26.62±0.4
ResNet101	62.79±0.3	66.77±0.4	71.54±0.3	48.24±0.6	51.98±0.5	57.90±0.5	18.53±0.3	21.44±0.1	24.07±0.2
PSPNet [7]	UNet	68.53±0.5	74.04±0.6	77.80±0.1	40.08±0.8	42.27±0.2	45.65±0.1	12.71±0.7	13.67±0.6	64.34±0.4
ResNet50	61.82±0.3	65.25±0.2	70.37±0.5	49.08±0.6	53.94±0.9	58.71±1.1	18.92±0.8	22.10±1.2	24.45±1.1
ResNet101	63.06±0.6	67.11±0.2	71.87±0.1	47.76±0.3	51.34±0.8	57.12±0.5	19.37±1.3	22.39±2.0	25.05±1.5
Segmenter [6]	ViT-T [2]	49.39±1.3	51.25±1.2	51.51±0.9	33.83±0.1	35.15±0.3	36.04±0.5	11.98±0.2	12.57±0.1	29.73±0.4
ViT-S	51.36±0.4	53.33±0.7	55.14±0.1	32.54±0.5	33.79±1.0	33.61±0.2	11.62±0.5	12.11±0.4	28.37±0.9
ViT-B	50.98±0.3	52.90±2.3	54.45±0.7	34.03±0.4	35.36±1.3	36.40±0.9	11.78±1.1	12.30±0.4	28.99±1.2
ViT-L	48.11±0.3	50.00±0.5	98.07±1.1	34.70±1.3	36.03±0.8	37.55±0.7	12.19±0.1	12.75±0.3	24.37±0.6
Table 3: Binary segmentation results of Mask2Former on diseased/normal videos.
Eye Condition	Diseased	Normal
mIoU	74.25	73.96
mAcc	79.23	78.16
mFscore	81.18	80.08
C.3.2 Test across different devices