A Medical Data-Effective Learning Benchmark for Highly Efficient Pre-training of Foundation Models

Authors: wenxuan  yang
weimin  tan
yuqi  sun
bo  yan
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:06:41

Abstract: Foundation models, pre-trained on massive datasets, have achieved unprecedented generalizability. However, is it truly necessary to involve such vast amounts of data in pre-training, consuming extensive computational resources? This paper introduces Data-effective learning, aiming to use data in the most impactful way to pre-train foundation models. This involves strategies that focus on data quality rather than quantity, ensuring the data used for training has high informational value. Data-effective learning plays a profound role in accelerating Foundation Model training, reducing computational costs, and saving data storage, which is very important as the volume of medical data in recent years has grown beyond many people's expectations. However, due to the lack of standards and comprehensive benchmark, research on medical data-effective learning is poorly studied. To address this gap, our paper introduces a comprehensive benchmark specifically for evaluating data-effective learning in the medical field. This benchmark includes a dataset with millions of data samples from 31 medical centers (DataDEL), a baseline method for comparison (MedDEL), and a new evaluation metric (NormDEL) to objectively measure data-effective learning performance. Our extensive experimental results show the baseline MedDEL can achieve performance comparable to the original large dataset with only 5% of the data. Establishing such an open data-effective learning benchmark is crucial for the medical AI research community because it facilitates efficient data use, promotes collaborative breakthroughs, and fosters the development of cost-effective, scalable, and impactful healthcare solutions.

Main paper: 
 introduction: 
 The effectiveness of foundation models depends on the abundance of pre-training data, a notion seemingly supported by consensus: more pre-training data leads to enhanced model performance. However, is this assumption truly accurate? To explore this issue, we introduce the concept of data-effective learning, which plays a significant role in the field of medical data, accelerating foundation model training and reducing storage burden in the era of big data (Figure 1 (a)). The global endoscopy surgery market is undergoing significant development, with an estimated daily addition of 22,546,800,000 video frames [1] . However, a substantial presence of disruptive and invalid data [45] significantly hampers training efficiency and occupies a considerable amount of storage space [60] . Therefore, achieving data-effective in endoscopy datasets holds the following special advantages:• Storage Savings: Assuming the use of traditional highdefinition endoscopes (1080p) [26] , the daily uncompressed endoscopy examination videos would require 12,756,493 TB (about 13.06 exabytes) of storage space. Condensing endoscopy datasets can result in substantial storage savings. • Enhanced Model Efficiency: [8] released the largest digestive system image dataset (Hyper-Kvasir) to date and showed that upon analyzing data sources, over 90% of video frames consist of disruptive and invalid data. Core critical data comprises only 2% of the entire dataset [30, 31] . Efficient utilization of core critical data can significantly improve the efficiency of model training, in order to achieve rapid convergence. • Computational Resource Savings: With the use of a single RTX 3090 graphics card and the VGG16 model [33] (approximately 138 million parameters) for 32-bit floating-point (FP32) calculations, processing 325.6 images per second is achievable [35] . In this configuration, training on the daily added video frames would require 19,200 hours. Utilizing only core critical data could save nearly 18,816 hours without compromising precision.The issue of data-effective learning is not exclusive to endoscopy datasets; it is also evident in other types of medical datasets. With the rapid expansion of future medical data, efficiently handling medical datasets is the next crucial research problem in data-driven learning methods [47] .In recent years, there has been relevant research on data-effective learning in natural image datasets such as adversarial samples [20] , and data biases [56] , aiming to enhance the robustness and generalization capabilities of deep learning models. These works in natural datasets have shown preliminary effectiveness, indicating significant potential advantages such as data storage, computational resource savings, and efficient model training. However, despite the exponential growth of medical datasets, there is currently a lack of such exploration in the medical field [58] . This is attributed to the absence of relevant benchmarks, encompassing unified datasets, baseline methods, and comprehensive evaluation metrics, providing the academic community with a basis for developing and comparing advanced methods.In addition to the absence of benchmark, achieving data-effective in the field of endoscopy remains a huge challenge [50] and is worth exploring in depth. Traditional data-effective techniques may struggle to work effectively with datasets that have high-dimensional features or sparse data. In endoscopy datasets, the challenge lies more in identifying subtle differences in images, which can be crucial for medical diagnosis. Therefore, accurately determining the similarity of images in endoscopy datasets is a top priority in research. Maintaining data quality and integrity is a key issue in data-effective learning [18] , as it directly affects the usability and accuracy of the dataset.Our research contributions can be summarized as follows:• We introduce the concept of data-effective learning and provide a corresponding medical benchmark (Figure 1 • In our benchmark, we introduce a baseline method called MedDEL for data-effective learning, which can outperform the use of 100% of the data in downstream tasks with 5% of the pretraining data in extreme cases. • We develop a new metric called NormDEL, to assess the performance of data-effective in datasets, which considers the relationship between the proportion of the dataset retained and the performance of downstream tasks. 
related work: 
 In the natural image domain, benchmarks for assessing data-effective learning typically involve various techniques such as structural similarity index [17] , convolutional neural networks [39] , and local feature descriptors [36] . These methods [44, 57] are evaluated on widely used image datasets like ImageNet [15, 49] and CIFAR [34] , or specialized datasets. PXDedup [59] explores the issue of dataeffective learning in JPEG images, pointing out that traditional binary stream-based techniques do not work well for compressed JPEG images. [9] introduces a high-precision image data effective method that identifies and eliminates duplicate images through feature extraction and high-dimensional indexing [32] . [10] uses wavelet decomposition [43] to extract feature vectors from images and calculate the Manhattan distance to determine image similarity, thus achieving the detection and removal of duplicate images. However, when handling datasets with high-dimensional features [48] or sparse data [59] , traditional data-effective techniques might not work effectively. MFDedup [61] surpasses existing technologies in improving data reduction rates and recovery throughput, while also reducing the cost of garbage collection. Storage and memory capacity [13] can also become limiting factors. [24] discusses how to accurately estimate the data reduction ratio achieved through data effective and compression techniques for specific datasets.Hash methods [46] indeed have a wide application in data effective processing. The CE-Dedup [38] framework combines hash-based image data-effective techniques with deep learning image classification tasks. By adjusting the deduplication threshold, effectively balances the trade-off between data reduction rate and model accuracy [21] . [40] proposes a deep supervised hash algorithm for image retrieval by providing the model with pairs of images labeled as similar/dissimilar. [14] proposed a Core-set selection method based on metric explanations (CSUME) for the classification of multi-class electrocardiograms. This work offers us an effective method for selecting representative datasets within an unsupervised learning framework, which holds significant importance in the fields of medical image computing and health informatics. However, in the medical field, there currently exists no such benchmark for the research of data-effective learning algorithms. 
data-effective learning medical benchmark: 
 In this section, we introduce the definition of benchmark tasks. Dataeffective learning refers to the practice of using a limited amount of data for the pre-training phase [19] . Therefore, this benchmark encourages researchers to develop advanced data-effective learning methods to generate a compact version of the originally collected large-scale dataset, thereby obtaining a compact small-scale new dataset for pre-training foundational models.In the era of big data, large models often require massive pretraining data [11] . However, our perspective challenges this conventional thinking, and we have undertaken the following work to explore this issue in depth.Our benchmark consists of three parts, a dataset with millions of data samples (DataDEL), a baseline method for comparison (Med-DEL) and a new evaluation metric (NormDEL).Given the urgent demand for a comprehensive benchmark in the medical field, we are facing the pressing task of integrating diverse large datasets. Currently, prevailing challenges within existing medical datasets encompass issues like the uniformity of data sources, the standardization of task execution, the limited scope of covered disease types, imbalances in dataset categories, and the uniformity of modalities [37] .These limitations hinder the comprehensive development of medical research. Therefore, there is an urgent need for us to construct a comprehensive medical dataset. Our efforts focus on integrating datasets from over 31 different centers and 23 different countries, spanning multiple modalities such as images and videos. Furthermore, our dataset surpasses the million-scale mark, becoming a large-scale collection that encompasses multiple tasks, modalities, sources, and diseases. This dataset can provide rich and diverse support for subsequent research, offering significant convenience for healthcare professionals and research institutions.We introduce a benchmark baseline method (MedDEL) in the endoscopic medical field, which is based on the principles of the SemDeDup method [2] . Endoscopic data holds significant importance in medical diagnosis and treatment. However, due to various factors such as organ morphological changes [53] , lighting conditions [12] , and noise interference [23] , the analysis of this data becomes complex and challenging.Specifically, the MedDEL method takes an image 𝐼 as an 𝑖 𝑡ℎ example, the encoder of Vision Transformer (ViT) [16] extracts deep features from the image, ultimately producing a 768-dimensional feature output. This output serves as our information embedding. The high-dimensional representation not only captures the semantic information of the image more effectively but also addresses semantic repetitiveness issues that are challenging to resolve in low-dimensional spaces. Considering the encoding process of image 𝐼 through the ViT model, we can represent it with the following formula:EQUATIONWhere 𝑥 represents the final layer of the 𝑉 𝑖𝑇 encoder. After obtaining the feature embeddings for each image in the dataset, we can explore the similarity between images in a highdimensional space. Let the embedding corresponding to the 𝑖 𝑡ℎ image be denoted as 𝐹 (𝑥, 𝑖), representing all image features as ). Taking our proposed upstream dataset as an example, which includes 88,282 images, the total number of calculations reaches 744 million. However, by introducing the K-means algorithm [42] , the overall computational complexity is reduced from 𝑂 (𝑛 2 ) to 𝑂 (𝑁 2 /𝑘), where 𝑘 is the number of clusters. After applying the K-means algorithm, we obtain 𝑘 clusters. Assuming that the feature 𝐹 (𝑥, 𝑖) belongs to the 𝑘 cluster, we can derive the following 𝑘 sequences:EQUATIONIn each cluster, data points that are far from the cluster centroid are considered disruptive or invalid data. To filter out interference and invalid data, we introduce a predefined threshold 𝜖. Specifically, in each cluster 𝐶 𝑗 we define the distance from image 𝑖 to the cluster centroid as 𝑑 𝑖 𝑗 . By using the following inequality, we classify image 𝑖 as potentially affected by disruptive or invalid data.Specifically, we introduce a similarity threshold 𝜂, which is used to define semantic duplicate information. In category 𝑗, we perform similarity calculations for each pair of embeddings using the following formula:EQUATIONThrough this similarity calculation, we retain a set of features that are closer to the cluster center, achieving the goal of filtering effective and core-important data. The pseudocode for a simplified algorithm regarding data effectiveness is presented in Algorithm 1.Currently, there is no comprehensive objective metric for evaluating the performance of data-effective learning algorithms, where comprehensive refers to integrating both the test accuracy of downstream tasks and the compactness of pre-training data. Hence, We propose a new data-effective metric, the Normalized Data-Effective Learning Index (NormDEL). Our objective is that within the same task, if a model can achieve the same level of performance in downstream tasks using fewer pre-training data, its data-effective performance should be considered superior. Taking the commonly used segmentation task in endoscopy as an example, where the general metric for measuring performance is often mIoU (mean Intersection over Union), we can formulate the following equation:EQUATIONWhere 𝛼 serves as a positive weight parameter used to adjust the influence of mIoU and the retention ratio in DEL, and 𝑅 represents the proportion of the retained dataset.Subsequently, we aim to normalize DEL to a range between 0 and 1 for a clearer definition of data-effective performance. Thus, we obtain the definition of NormDEL:EQUATIONThrough NormDEL, we can comprehensively evaluate the adaptability of the data-effective model with limited training data, emphasizing the critical factors of model data-effective in large-scale datasets. 
experiment: 
 To assess the capability of MedDEL, we train the Foundation Model on three pre-training datasets as presented in Table 1 . Subsequently, we conduct finetuning on downstream task datasets to evaluate performance.To achieve the pretraining of the Foundation Model, enabling it to provide a better initial value for the majority of endoscopy tasks, we use the unsupervised Masked Auto Encoder (MAE-ViT-Base) [25] .During the training of all models, we selectively filter the pretraining dataset capacity based on different thresholds of 𝜖 and 𝜂. Specifically, we set 𝜖 to 0.9 and 𝜂 from 0.7 to 0.9 with a step size of 0.05. Consequently, we obtain the dataset quantities as presented in Table 2 . The batch size is set to 16, and the training is conducted using AdamW [41] and a cosine learning rate [22] . The model maintains a peak learning rate of 1.5 × 10 -4 throughout the training process. In the validation of downstream tasks, we implement a simple Multi-Layer Perceptron (MLP) [54] to achieve segmentation heads.Figure 3 demonstrates that, under equivalent computational resources, the MedDEL method which operates on a reduced dataset (utilizing only 5% of the data, indicated by the red curve), generally performs comparably to the full dataset (using 100% of the data, indicated by the black curve) across various downstream tasks. Moreover, in certain specific scenarios, due to its smaller dataset size and higher data quality, the MedDEL method exhibits superior performance when trained on just 5% of the data. This advantage is particularly pronounced, as seen in the case of CVC-300, highlighting its effectiveness in efficiently leveraging a small amount of high-quality data. This underscores the MedDEL method's characteristic focus and refinement during the learning process, enabling it to more effectively capture crucial task features, resulting in improved performance in low-data scenarios. This result emphasizes the significance of MedDEL method in enhancing data effectiveness, particularly in the era of big data when high-performance models are needed, showcasing its capability for effective data utilization.Table 3 presents the impact of different amounts of pre-training data on mIoU and NormDEL across 8 downstream datasets. It is worth noting, in the majority of tasks, the mIoU achieved with only 5% of the data is comparable to that achieved with 100% of the data. To clearly assess the influence of pre-training data set proportion on performance, we computed NormDEL for each proportion. The results indicate that in some tasks, although the mIoU attained with 5% of the data may not be the highest when considering dataset size comprehensively, its NormDEL reaches the highest.Furthermore, To analyze and evaluate the accuracy and robustness of the baseline method under the same time performance, Figure 4 illustrates a box plot of different proportions of pre-trained data in various downstream datasets, based on the mIoU probabilities. Each box represents the predictive performance of the model during the several final epochs, employing the median, approximate quartiles, and the lowest and highest probabilities to intuitively display the level, spread, and symmetry of the mIoU distribution.Figure 4 demonstrates a unique trend in the performance of models (measured by the median mIoU) across most downstream tasks, in relation to the increase in the volume of pre-training data. Initially, performance improves with an increase in data volume, reaching a peak, and then starts to decline as data volume continues to grow. Specifically, when a smaller volume of data is used (as in the case of the 5% data volume), despite the limited quantity, the average information content per image is higher, enabling the model Figure 3 : Demonstration of the feasibility of data-effective learning. We compared the performance differences between using only 5% of the pre-training data (in red) and using 100% of the data (in black) in 8 datasets. The results indicate that using only 5% of the pre-training data can achieve results comparable to using 100% of the pre-training data, which fully demonstrates the validity of the MedDEL method.Table 3 : Demonstration of the rationality of NormDEL. The table shows the performance of different proportions of pre-trained data on mIoU (↑) as well as NormDEL (↑) in eight different datasets. The results indicate that the performance of mIoU is almost comparable in different scales, suggesting that it primarily measures performance without considering the scale of the pre-training data used. In contrast, NormDEL incorporates the factor of data scale, and can evaluate data compactness even with only 5% of the data used. This demonstrates the rationality of the NormDEL method, as it not only assesses performance but also effectively utilizes the data scale.to achieve relatively decent results. However, as the data volume increases, redundancy in information also grows. At a certain point, a balance is reached between information content and redundancy, culminating in peak model performance. Beyond this point, further increases in data volume lead to a decline in performance. This trend is reflected in the length of the boxes in the plot, where the box length first increases with the data volume, reaches a peak, and then starts to shorten. However, in the results from Figure 4 , the trends are not as stable, and there is fluctuation observed in some of the downstream tasks. This also reflects the limitations of the MedDEL method. These fluctuations may arise from differences in data characteristics, task complexities, or other factors. It is precisely this diversity that complicates the optimization of data effectiveness, requiring indepth research and fine-tuning.Nevertheless, these findings still underscore the effectiveness of MedDEL in reducing storage burden and achieving good results with smaller proportions of data. We encourage more researchers to engage with our study and propose more refined methods to make the most of limited data resources. 4 : Box plot of the baseline method (MedDEL) for testing mIoU under retaining different data ratios. Under the same computational resources, we plot the impact of varying proportions of pre-training data on downstream task performance, using the data proportions from Table 2 and models pre-trained on ImageNet for reference. The results demonstrate that the performance of downstream tasks initially improves with an increase in pre-training data but eventually decreases. This is because the initially added data is effective, however, as more data is introduced beyond a certain point, it includes redundancy or erroneous information, which adversely affects the model's performance. Remarkably, we observe that optimal performance is not consistently achieved with models trained using 100% of the pretraining time across nearly all datasets. Instead, we note the presence of a complex uncertainty and fluctuation, where the optimal result may be distributed among different pretraining time points in diverse datasets and with varying proportions of pretraining data. This underscores the effectiveness of MedDEL in conserving computational resources. However, it also suggests that optimizing pretraining time may require different strategies for various tasks and data contexts. Our study has not yet uncovered the specific relationships between these strategies, indicating the need for more in-depth research in the future.The MedDEL algorithm, as a benchmark method in the field of endoscopic, is based on the Vision Transformer (ViT) model [16] for extracting image features, generating a 768-dimensional feature output. By introducing K-means clustering [42] , it successfully reduces computational complexity and further effectively addresses irrelevant data in the endoscopic dataset by setting similarity thresholds and filtering conditions. Figure 5 visualizes the results of the MedDEL (DEL) algorithm, where upon observation, it is evident that MedDEL eliminates a significant portion of redundant data with highly similar features from the dataset. Considering their similarity, we suppose removing such redundant data is a reasonable strategy to optimize the dataset structure, enhancing dataset quality. This optimization improves the model's understanding of endoscopic images, making it more targeted and interpretable, and providing robust support for medical image analysis [16] .To validate the effectiveness of our model, we supplement the following generalization experiments in Fig 6 . Specifically, we select 5%, 20%, and 50% of the data and compare the test performance of random selection and MedDEL. We find that whether using the random selection method or MedDEL, as the amount of data increases, the model's generalization ability improves. Furthermore, at the same data volume, MedDEL shows superior performance to the random method. Additionally, MedDEL shows comparable generalization performance when selecting 50% data compared to using full data. 
conclusion: 
 In the era of foundation models, we are the first to propose whether larger pre-training data necessarily leads to improved model performance. In order to investigate this issue, we propose the first medical data-effective Benchmark, which involves a million-level dataset from 31 medical centers (DataDEL), a data-effective approach (MedDEL), and a comprehensive metric for evaluating pretraining data volume (NormDEL). The establishment of an open benchmark for data-effective learning is crucial for the medical artificial intelligence research community, which is an encouraging foundation for subsequent research endeavors. 
acm reference format:: 
 Wenxuan Yang, Weimin Tan, Yuqi Sun, and Bo Yan. 2024. A Medical Data-Effective Learning Benchmark for Highly Efficient Pre-training of Foundation Models. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24) , October 28-November 1, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3664647.3681313 Table 1 : Overview of the proposed DataDEL in our benchmark. We integrate 31 medical centers, including data on the scale of millions, to build a high-quality, large-scale, multi-disease comprehensive dataset, aiming to provide researchers with a better data platform.Model Pre-training Gastrovision [29] Detection and classification 2 None 8,000 GI Hyper-Kvasir [8] Classification 1 374 110,079 GI Kvasir-Capsule [52] Classification 1 117 47,238 GI Model downstream testing CVC-12k (CVC-ClinicDB) [6] Segmentation 1 None 612 polyp CVC-300 [55] Segmentation Not mentioned None 60 polyp CVC-ColonDB [7] Segmentation Not mentioned None 380 polyp EAD2019 [5] Endoscopic artifacts 6 None 2,991 cancers EDD2020 [3] Segmentation, detection, and localization 4 None 380 polyp ETIS [51] Segmentation Not mentioned None 196 polyp ImageCLEFmed [27] Polyp segmentation 1 None 66,662 polyp Kvasir-Instrument [28] Segmentation, detection, and localization 1 None 590 polyp Kvasir-SEG [31] Segmentation, detection, and localization 4 None 1,000 polyp Kvasir-Sessile [30] Segmentation, detection, and localization 4 None 196 polyp PolypGen2021 [4] Polyp segmentation 6 None 3,762 polyp 
 
 Appendix:  


Decision: This paper introduces a comprehensive benchmark for data-efficient learning in the medical field, including a large dataset, a baseline method, and a new evaluation metric. Reviewers acknowledge the paper's contributions, novelty, and clarity but raise concerns about the lack of comparison with other data efficiency approaches, limited discussion of related works, and the need for more extensive experiments. The paper's significance and impact on the multimedia/multimodal processing community are evident, but improvements are necessary to address the reviewers' concerns.

The paper's strengths outweigh its limitations. The authors have made significant contributions to the field, and the paper has the potential to inspire future research.

Reviews: review: paper_summary: paper_summary: The paper introduces a comprehensive benchmark, which includes a dataset (DataDEL) with millions of samples from 31 medical centers, a baseline comparison method (MedDEL), and a new evaluation metric (NormDEL). Experimental results demonstrate that the baseline method MedDEL can achieve performance comparable to that of the full dataset using only 5% of the original data.
strength_weakness: strength_weakness: 1.	The paper integrate an open-source dataset, sourced from a million-level dataset spanning 31 medical centers.
2.	The paper conducted a large number of experimental verifications.
limitations: limitations: 1.	In your MedDEL method, what is the innovation of the clustering method?
2.	Although methods based on feature extraction and high-dimensional indexing have been proposed for data deduplication, the ability to handle small differences in images still needs to be verified, especially in medical diagnosis, where these differences may be of important significance.
3.	In the medical field, model accuracy is crucial, so is it reasonable to reduce the amount of data at the expense of model accuracy? Specifically, in Table 3, if the NormDEL method is used, the best data set should be 5% pretraining data, but the highest mIoU is generally 30% pretraining data. There is a certain gap between the two mIoU. Whether your NormDEL method is too high depends on the proportion of data volume and ignores the accuracy of the model.
4.	If 5%, 20% and so on of the pretraining data are randomly selected for training, and then the NormDEL measurement standard is used to calculate the quality of the randomly sampled data set, will it still be concluded that the quality of the randomly sampled data is better than the full amount of data? Given this situation, how to explain the rationality of NormDEL and MedDEL?
paper_summary: paper_summary: The paper focuses on data efficient training of foundation models on healthcare. The large quantity of data requirement has proven to be a significant challenge in development of deep learning models in healthcare and thus the topic of research is highly relevant and of interest to broader community.  While the proposed approach is novel and the paper is well-written, certain limitations hinder the strength of the paper. The proposed dataset is extensive and would be a significant contribution to forward research, however this is not a dataset paper.  The author's claim "We introduce the concept of data-effective learning and provide a corresponding medical benchmark (Figure 1(b)) to guide data-effective algorithm research in the medical field" is not new as data efficient learning has been the focus on research for quite some time. The problem of core-set selection also translates to the same which has not been reviewed or discussed by the authors. The lack of comparison of the proposed approach with other techniques of data selection is missing with only random selection shown as comparison.
strength_weakness: strength_weakness: Comprehensive evaluation on multiple datasets.
Well-written
Novel Approach
limitations: limitations: While there may not be benchmarks on healthcare image datasets, there is a plethora of research on data efficient learning and core-set selection for healthcare data on various modalities of 2D images, 1D time-series signals and EHR tabular data. The lack of discussion of these works makes the related work not extensive enough.

Some Sample works: 
1.Weicheng Kuo, Christian Häne, Esther Yuh, Pratik Mukherjee, and Jitendra Malik. 2018. Cost-sensitive active learning for intracranial hemorrhage detection. In the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer,715–723
2. S. Dakshit, B. M. Maweu, S. Dakshit and B. Prabhakaran, "Core-set Selection Using Metrics-based Explanations (CSUME) for multiclass ECG," 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI), Rochester, MN, USA, 2022, pp. 217-225, doi: 10.1109/ICHI54592.2022.00041
3. Hossein Shokri Ghadikolaei, Hadi Ghauch, Carlo Fischione, and Mikael Skoglund. 2019. Learning and data selection in big datasets. In International Conference on Machine Learning. PMLR, 2191–2200

The proposed approach lacks comparison of other data efficiency approaches and core-set learning approaches which also have shown similar results. Without comparisons it is not possible to truly judge the efficiency or the benefit from the proposed approach.
paper_summary: paper_summary: This paper proposes a benchmark for data-effective learning in the medical field, including a dataset with millions of data samples from 31 medical centers and a baseline model and A way to evaluate parameters and experimentally verify that the performance of the original data set can still be achieved using part of the data in the large data set.
strength_weakness: strength_weakness: 1.Millions of data from 31 medical centers were collected to build a benchmark data set, covering a variety of diseases and tasks.
2.A simple but effective data-effective baseline learning method is proposed, and the K-means method is used to select the samples closest to the center point in different clusters.
3.Propose a new parameter for evaluating the relationship between the proportion of the data set used and the performance of downstream tasks
limitations: limitations: 1.The baseline model only uses ViT as the encoder, which lacks diversity and does not take into account the adaptability of the pre-trained model itself to different amounts of data.
2.The downstream tasks only verify Segmentation and lack experimental verification for other tasks.
paper_summary: paper_summary: The paper proposes a data-efficient learning benchmark for pre-training on medical data. The benchmark consists a dataset DataDEL, a baseline method for data-efficient learning (MedDEL) and a evaluation metric (NormDEL). Experiments and evaluations are performed to demonstrate the effectiveness of the benchmark.
strength_weakness: strength_weakness: 1. The technical details are sound as demonstrated by Algorithm 1 and 4.1 Settings.
2. The experiments are performed on multiple datasets with reported results on different percentages of pre-training data.
3. Figure 3 shows the feasibility of data-efficient learning on multiple datasets with trends on training epochs.
4. In general, the paper provides many details on methodology and experiments with clarity and numerical results.
limitations: limitations: 1. MedDEL is only compared with random selection in Table 5. Is there any other methods for data-efficient learning achieving the same purpose? If yes, how would them perform compared to MedDEL?

2. What is the purpose of DataDEL dataset? Is it a collection of existing datasets? It seems we can use proposed MedDEL and NormDEL to do the data-efficient learning on existing datasets, e.g. CVC-300, ETIS, as demonstrated in the experiments.