JSCDS: A Core Data Selection Method with Jason-Shannon Divergence for Caries RGB Images-Efficient Learning

Authors: peiliang  zhang
yujia  tong
chao  che
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-05-20 16:16:06

Abstract: Deep learning-based RGB caries detection improves the efficiency of caries identification and is crucial for preventing oral diseases. The performance of deep learning models depends on high-quality data and requires substantial training resources, making efficient deployment challenging. Core data selection, by eliminating low-quality and confusing data, aims to enhance training efficiency without significantly compromising model performance. However, distance-based data selection methods struggle to distinguish dependencies among high-dimensional caries data. To address this issue, we propose a Core Data Selection Method with Jensen-Shannon Divergence (JSCDS) for efficient caries image learning and caries classification. We describe the core data selection criterion as the distribution of samples in different classes. JSCDS calculates the cluster centers by sample embedding representation in the caries classification network and utilizes Jensen-Shannon Divergence to compute the mutual information between data samples and cluster centers, capturing nonlinear dependencies among high-dimensional data. The average mutual information is calculated to fit the above distribution, serving as the criterion for constructing the core set for model training. Extensive experiments on RGB caries datasets show that JSCDS outperforms other data selection methods in prediction performance and time consumption. Notably, JSCDS exceeds the performance of the full dataset model with only 50% of the core data, with its performance advantage becoming more pronounced in the 70% of core data.

Main paper: 
 introduction: 
 Dental caries is one of the chronic diseases affecting the health of children and adolescents. If not treated promptly, it can lead to complications such as pulpitis and periapical periodontitis, adversely affecting the development and quality of life of adolescents [9] . According to incomplete statistics, approximately 50% of children aged 6-11 have caries in their primary teeth, and over 34% of adolescents aged 12-19 have caries in their permanent teeth globally [2] . This underscores the importance of early detection and treatment of dental caries in maintaining the oral health of children and adolescents. Currently, the most widely used methods for caries detection are visual inspection, probing, and X-ray examination. However, these methods depend heavily on the professional expertise of dentists and increase the economic burden on patients. Therefore, efficient, accurate, and affordable caries detection methods are urgently needed.Previous Work: According to different detection methods, caries detection can be categorized into physical examination, auxiliary examination, and automated detection [4, 5, 11] . Physical examination refers to dentists visually inspecting the tooth surface for changes in color and shape to identify caries, with common methods including visual inspection and probing. While physical examinations are cost-effective, they rely heavily on the dentist's expertise, have a higher misdiagnosis rate, and require patients to visit the clinic, which is time-consuming and labor-intensive. Auxiliary examinations utilize modern medical equipment to scan or chemically test teeth for caries detection [11, 15] . Methods such as X-ray examination, laser fluorescence detection, electrical impedance measurement, and optical coherence tomography fall under this category. Although auxiliary examinations are more accurate compared to physical examinations, they are also significantly more expensive. Automated detection involves using automated tools to capture RGB images of teeth and intelligently determine the presence of caries [5] . With the advancement of Deep Learning, various caries recognition and detection models have been developed, making automated caries detection increasingly mainstream [6, 8] . Compared to the other two methods, deep learning-based automated caries detection offers significant advantages in terms of portability, high accuracy, and low cost.Limitation and Motivation: Deep learning methods based on RGB images have significantly improved the efficiency and accuracy of caries detection while reducing its costs [15] . However, deep learning methods require high-quality and large training datasets. Low-quality, confusing data in the training dataset can severely affect the model's performance. In caries detection, patients primarily use mobile devices to capture RGB images of their teeth. During this process, due to the oral cavity's unique structure and the photographers' varying skill levels, the captured images often contain a certain amount of unclear, mixed-boundary confusing data. These data can severely impact the performance of the detection model [4, 5] . Data core set selection is one effective way to address this issue. By quantifying the impact of different data on the model, core data selection identifies key data and reduces the influence of confusing data, thereby enhancing the performance and efficiency of model training [7, 10, 16] . As shown in Figure 1 (A), the Moderate method achieved superior performance and faster computational efficiency using 70% of the core dataset, with similar results observed in Figure 1(B) . Therefore, how to measure the impact of different caries data on the detection model and selecting the core dataset accordingly is crucial for improving caries detection performance and computational efficiency.To address the aforementioned problems, we propose a core data selection method with Jensen-Shannon Divergence (JSCDS) for efficient caries image learning and caries classification. Specifically, JSCDS calculates the cluster centers of different classes with sample embeddings in the caries classification network. The core data selection criterion is described as the distribution of samples within different classes. We utilize Jensen-Shannon Divergence (JSD) to compute the mutual information between data samples and cluster centers and use average mutual information (AvgMI) as the data selection criterion. JSCDS selects samples close to the AvgMI as the core set for model training. Unlike distance-based data importance selection methods, JSCDS captures mutual information gain between samples, enhancing the handling of high-dimensional data. We conduct extensive experiments using MobileNetV2 and ResNet18, and the results demonstrate that JSCDS has lower time costs in data selection and exceeds the performance of the full dataset model using only 50% of the core data.Our main contributions are as follows:• Different from distance-based data selection methods, we focus on information theory-based data selection to identify high-dimensional nonlinear dependencies in samples.We propose the concept of AvgMI to generate the core set, effectively distinguishing between different caries. • As a proof of concept, we design a mutual information calculation method with Jensen-Shannon Divergence. This method does not depend on model structure and selects core data only with the neural network's embedding. • The results demonstrate that JSCDS significantly outperforms others in prediction performance and time consumption. Notably, JSCDS exceeds the performance of the full dataset model using only 50% of the core data. 
methodology 2.1 definition: 
 We define the core data selection problem in caries classification. For a given full dataset of caries training data M = {𝑚 1 , 𝑚 ------→ M * . When the model's performance on M * is comparable to M, and its training efficiency significantly improves, we refer to M * as the core set and use it for model training.To construct the core set for caries, we propose JSCDS with Jensen-Shannon Divergence. The workflow is shown in Figure 2 . JSCDS primarily consists of two parts: data representation and data selection.For the neural network model for caries classification, it can be viewed as a black-box feature extractor. For the caries image F 𝑖 , the probability distribution P (Z, F 𝑖 ) of the image sample can be obtained through the mapping Z( * ) = H (ℎ( * )). Here, ℎ( * ) represents the final embedding representation of the input caries image in the hidden layer, and H ( * ) is the feature mapping function represented by 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 ( * ). The hidden layer embedding representation of the training data 𝐹 𝑖 can be obtained through the above calculation, and from this, the hidden embedding representation of all training data M can be derived, as shown in Equation ( 1). We calculate the cluster center C 𝑗 for each data class based on the hidden layer representations by Equation ( 2).EQUATIONHere, the numerator represents the sum of the sample embeddings in class T 𝑗 , and the denominator represents the number of samples in class T 𝑗 . The cluster center C 𝑗 is the reference for subsequent data selection processes.Relevant research primarily measures the relevance of samples by calculating the distance between them [10, 16] . While this method is computationally efficient, it heavily depends on the data size [7, 16] . Identifying high-dimensional nonlinear dependencies in the training data is challenging, making it difficult to distinguish between samples. In contrast, information theory-based methods can capture nonlinear relationships between samples, enabling effective high-dimensional data processing. Therefore, JSCDS designs a JSD-based data sample importance selection strategy from the information theory perspective.Based on the hidden layer representations {𝑒 1 , 𝑒 2 , • • • , 𝑒 𝑥 } of the training data and the cluster centers {C 1 , C 2 , • • • , C 𝑗 } of the classes, the mutual information 𝑗𝑠𝑑 (𝑒, C) between each image embedding representation and its corresponding cluster center is calculated. 𝑗𝑠𝑑 (𝑒, C) is obtained by computing the JSD:EQUATIONwhere 𝑒 and C represent the data samples and the class cluster centers, respectively. The JSD is computed by combining KL divergences, and 𝑤 denotes the embedding dimension of 𝑒.The set of mutual information between each sample in the training set and its class cluster center is denoted as 𝑀𝐼 (𝑚 1 ), • • • , 𝑀𝐼 (𝑚 𝑥 ), sorted in descending order to obtain {𝑀𝐼 ( m1 ), • • • , 𝑀𝐼 ( m𝑥 )}. We use the average mutual information 𝐴𝑣𝑔(𝑀𝐼 (𝑚)) as the metric, selecting samples close to 𝐴𝑣𝑔(𝑀𝐼 (𝑚)) as the core set 𝑀 * .EQUATIONEQUATION) Here, 𝛾 represents the selection ratio of data in the core set (To ensure clarity in our descriptions, we describe 𝛾 as "Fraction" in the experimental section.), and { m𝛾 , • • • , m𝑥 -𝛾 } is the core set after selection, which is used for model training.In information theory, the more considerable mutual information between a data sample and the cluster center indicates the higher posterior probability given by the neural network for the sample. However, obtaining embedding representations in neural networks is easier than computing posterior probabilities [1] . Therefore, our method is more practical in real-world applications and requires fewer resources, thereby improving the computational efficiency of deep learning. 
experimental 3.1 datasets: 
 We used the RGB dental caries dataset labeled and processed by professional dentists. This dataset has been widely used in caries object detection and classification [4, 5] . It contains 5619 caries images, each in 24-bit JPG format. Professional dentists classified the caries images into three categories: mild, moderate, and severe caries, based on the extent and severity of the lesions. Following the settings in previous related studies [4, 5, 17] , we divided the dataset into training, validation, and test set in the ratio of 8:1:1. (each with 24GB RAM), Ubuntu operating system and an Intel(R) Core(TM) i9-12900KF CPU. The backbone networks for the experiments were MobileNetV2 [12] and ResNet18 [3] . We initialize the model parameters using the model that has been pre-trained on the ImageNet-1K dataset. The training parameters for JSCDS are as follows: we train the network for 50 epochs using a learning rate of 0.001, no weight decay, batch size of 64, and Adam optimizer. Every 10 epochs,the core set is reselected.We compared the performance of JSCDS with four core set selection methods. The primary comparison methods include: random data selection, Moderate [16] , kCenterGreedy [13] , and Forgetting [14] .To comprehensively analyze the performance of JSCDS, we conducted extensive experiments on the dental caries classification dataset. The experimental results are shown in Table 1 . The parameter settings of the comparison methods all followed the principle of optimal performance in our experimental equipment. 1 , our proposed JSCDS achieves the best fine-grained caries prediction results in different fractions. In predictive performance, JSCDS's performance on 50% of the core set is already close to or even exceeds the predictive results using the full dataset. At the 70% fraction, its predictive performance significantly surpasses that of the full dataset, as highlighted by the red data in Table 1 . Figure 3 further visually confirms these findings. Regarding time overhead, JSCDS effectively reduces the training time, substantially enhancing training efficiency. In summary, JSCDS effectively selects high-quality data from the original dataset, improving the model's classification performance. The main reason is that JSCDS calculates the AvgMI with JSD and uses it to select the core set, capturing high-dimensional nonlinear dependencies among samples. Euclidean distance for data selection, which is computationally simpler. However, as the volume of selected data increases, Moderate's computational speed significantly decreases. This experimentation result aligns with the general knowledge that Euclidean distance incurs higher computational overhead in large-scale datasets. In contrast, JSCDS maintains its advantages in predictive performance and time overhead as the data scale grows. kCenterGreedy and Forgetting fail to achieve satisfactory predictive performance and training time, possibly due to the limited number of caries data classes. Moderate and JSCDS yield commendable results regarding training time overhead, significantly reducing the training time.The Performance Analysis in Different Fractions. As shown in Table 1 , all models achieve optimal predictive performance when the fraction is 50% or 70%, with minimal performance differences from the full dataset. JSCDS and Moderate even surpass the full dataset's performance at 70% fraction, indicating their effectiveness in core data selection. As the fraction increases from 30% to 70%, the training time for JSCDS only increases by 21% and 19.5% in MobilenetV2 and Resnet18, respectively. This demonstrates that JSCDS is efficient in data selection, with its time consumption not proportionally increasing with larger fractions. In contrast, kCen-terGreedy and Forgetting have consistently high training times, and Moderate's performance in this aspect is slightly inferior to JSCDS. This result further underscores JSCDS's efficiency. The results in Figure 3 indicate that JSCDS achieves the best performance at a 70% fraction, with its predictive performance in both backbone networks showing a near-normal distribution trend. This suggests that the core set size significantly impacts model performance. The core set that is too large may include some noisy data, while a core set that is too small lacks sufficient high-quality data, both of which can degrade the model's predictive performance.  
conclusion: 
 In this paper, we designed a core set selection method with Jensen-Shannon Divergence. By capturing high-dimensional dependencies in caries images, we construct a high-quality core set for model training to improve the model's predictive performance. Extensive experiments on RGB caries datasets show that JSCDS outperforms other data selection methods in prediction performance and time consumption. Notably, JSCDS exceeds the performance of the full dataset model with only 50% of the core data, with its performance advantage becoming more pronounced in the 70% of core data. 
 
 Appendix:  



Reviews: review: title: title: This paper makes incremental improvements on the Carie’s classification. The experimental results are preliminary.
main_review: main_review: This paper propose a Core Data Selection Method with Jensen-Shannon Divergence (JSCDS) for efficient caries image learning
and classification. More specifically, the proposed JDCDS selects the core data for model training by calculating the cluster centers of sample embedding representation. JSCDS captures nonlinear dependencies among high-dimensional data. 

# Quality:
It is an acceptable paper, although the quality of the contents (e.g. typos, etc.) and figures should be improved (e.g. the font in the figure should be larger, etc.).

# Clarity:
The algorithm and its analyzing idea are missing in the paper, making readers not easy to grasp a good understanding to this work.

# Originality and Significance:
This is an incremental work on the caries classification.

# Strengths:
1.This paper proposes an information theory-based data selection based on average mutual information (AvgMI) to generate the core set, effectively distinguishing between different caries.
2.This paper proposes strategy that selecting core data by a mutual information calculation method with Jensen-Shannon Divergence which does not depend on model structure.
3. The proposed method exceeds the performance of the full dataset model with only 31 50% of the core data.

# Limitations:
1.The authors could discuss the differences between the proposed method and previous  caries classification methods based on deep learning architecture.
2.Some famous metrics for similarity, such as Manhattan Distance, Cosine Similarity, Minkowski distance, ect, should be employed for comparisons.
3.The ablation studies on JSD using different model structure should be verified. 
4.The limitations and failure cases of the proposed method should be discussed.
title: title: Reviews from 3Q6W
main_review: main_review: This paper introduces a Core Data Selection Method based on Jensen-Shannon Divergence (JSCDS) to enhance the training efficiency of caries image classification. By calculating mutual information between data samples and cluster centers, JSCDS effectively captures nonlinear dependencies in high-dimensional data, addressing the limitations of traditional methods. Experimental results demonstrate that JSCDS outperforms other data selection methods in prediction performance and time efficiency, achieving superior results with only 50-70% of the core data compared to the full dataset.  
﻿
The paper presents the methodology clearly, and the experimental results are good.  
﻿
Questions:  
﻿
It is necessary to highlight the strengths and distinctive features of this method. What are the key advantages of this method compared to traditional and SOTA methods? Why does it outperform the methods cited [13][14][16], given that those methods can also model nonlinear relationships and high-dimensional data? Additionally, please conduct thorough research; the baseline methods from 2017, 2018, and 2020 need updating.  
﻿
Why was JSD chosen? Is it a universal method for image analysis or specifically designed for this RGB dental caries dataset? If it is intended for general image data, more experiments are necessary. If designed for dental caries data, clarify the rationale for choosing this measurement method based on data characteristics.  
﻿
Will reducing the amount of data affect the generalization of the deep learning models? Current evidence suggests that greater data quantity positively impacts model generalization and accuracy. In the experiments, the improved accuracy may be linked to dataset partitioning (as described in Section 3.1, with training, validation, and test sets in an 8:1:1 ratio). Consider incorporating cross-validation and external validation for further validation.