dataesr/scientific-paragraphs-categorization huggingface dataset:
For example, identifying a high-grade tumor in real-time could lead to the resection of a wider and deeper section of the tissue to avoid future recurrences. In recent years, Deep-Learning (DL)-based methods have shown promising results in the analysis of endoscopic images. Most of the currently available datasets for endoscopic image analysis focus on colonoscopy  [12] ,  [13]  and consist mainly of WLI data. Recently, few studies which include NBI data too have stressed on the advantage of using multi-domain data in the colonoscopy scenario  [14] -  [16]. In the case of the urinary system, only a few studies have been carried out in the task of tissue classification from endoscopic images  [17] -  [20]. Except for the study presented in  [20]  where BL imaging is used, the rest of the studies use only WLI.
Multi-domain image classification implies several challenges, especially when the data and annotations are not evenly distributed across the different domains and some of the classes are under-represented  [21]. In the specific case of TURBT some of these challenges include the fact that visually it is difficult to differentiate between lesions and the diagnosis is inconclusive  [22]. Furthermore, due to the fact that multi-imaging endoscopes can collect only one imaging type at the time, it is not possible to have equivalent pairs of WLI and NBI images. Usually, an initial examination of the bladder is carried out using WLI and the lesions and anatomical structures deemed to be potentially cancerous tissue are examined again with NBI, in case this modality is available which is not always the case.
An additional challenge is related to the imbalance of data in terms of the different classes and types of tissue. Non-Suspicious Tissue (NST) usually receives less attention during interventions, therefore fewer amount of image data is collected from it than from lesions, either in WLI or NBI. Furthermore, non-cancerous lesions such as cystitis or other types of bladder inflammations are less common to appear in the initial inspection during TURBT. All this contributes to the fact that most of the datasets (including ours) are imbalanced not only in terms of different image domains but also in terms of tissue classes. In this work, we focus on the task of bladder tissue classification in multi-domain images from TURBT procedures, with special emphasis in the fact that annotations only exist in one of these image domains.
A. Tissue Classification in Endoscopy The analysis of endoscopic images has been rapidly developing in recent years thanks to the recent availability of new public datasets  [24] ,  [25]. In the specific task of tissue classification different models and techniques have been proposed with a special focus on the gastrointestinal (GI) tract. The existing methods range from the proposal of feature extraction models  [26] ,  [27] , to the use of transfer learning and pre-trained CNNs  [28] ,  [29]  and to more complex methods that focus on targeting the specific challenges present when working with GI endoscopic images  [30] -  [33]. In the case of the bladder, Ikeda et al. [19]  proposed the use of 2-step transfer learning by first fine-tuning their models on 8728 gastroscopic images, and then re-training the models on 2102 cystoscopy WLI images, using the GoogLeNet model for the task of binary classification of images with and without NMIBC.
Yang et al. [18]  compared the use of 3 different Convolutional Neural Networks (CNNs) as well as the platform EasyDL. The models used were LeNet, AlexNet and GoogLeNet. Their dataset includes 1200 cystoscopy images with cancer and 1150 without. Shkolyar et al. [17]  proposed CystoNet, a CNN for bladder cancer detection and binary classification. In their study, they used 2335 WLI frames of normal benign bladder mucosa and 417 histologically confirmed papillary urothelial carcinoma to train the network. In  [34]  the use of a Generative Adversarial Network (GAN) is proposed to perform data augmentation, then AlexNet and VGG16 are trained with the real and augmented data. In total 202 images from a Confocal Laser Endomicroscope were used in their experiments. In  [20]  Ali et al. proposed the use of pre-trained models for the task of cancer malignancy, grading, and invasiveness classification on BL photodynamic cystoscopy images.
C. Semi-Supervised Image Classification A common characteristic of medical image datasets is the lack of large annotated sets  [44]. During the last few years semi-supervised learning methods have progressed as a good alternative to leverage this large amount of unlabeled data. One of the most common paradigms of semi-supervised learning is the use of Teacher-Student Networks (TSN)  [45]. In this type of model, a teacher network is trained on the labeled data, and a student network is trained on the unlabeled data using the predictions given by the teacher. Training in semi-supervised mode allows the student model to learn features from unlabeled datasets  [46]. In the endoscopic scenario, few studies have been carried out using semi-supervised learning. Du et al. [47]  implemented a semisupervised contrastive learning method for Esophageal Disease Classification in a small dataset.
Golhar et al. [48]  proposed the use an unsupervised jigsaw learning method for GI lesion classification obtaining an improvement in accuracy of 9. 8% with respect to supervised methods. Guo et al. [49]  proposed the use of a combination of a discriminative angular loss and Jensen-Shannon divergence loss for semi-supervised learning for wireless-capsule endoscopic image classification. Shi et al. [50]  implemented a TSN network for the 3D reconstruction of stereo endoscopic images. Recently, semi-supervised GAN-based models have been proposed for image classification in different fields such as natural images and hyper-spectral image classification  [51] -  [54]. However, in the field of endoscopic images it remains an unexplored topic. Unlike the studies presented in  [55] -  [59]  where cycle-consistency translation has been implemented as a way of augmenting their datasets, we use image-translation inside a semi-supervised training loop to improve the classification performance of the unlabeled domain.
A. Problem Statement The proposed method consists of three main components; 1) A cycle-consistency translation network to translate every image in the dataset and have equivalent paired images in both domains (NBI and WLI); 2) A teacher network trained on the labeled WLI data; and 3) A multi-input multi-domain classifier trained as student network in a TSN semi-supervised way. A schematic of the proposed model is depicted in Fig. 2. Let us define a dataset X = X A ∪ X B composed by the union of two subsets: X A = {(x A1 , y A1 ),. , (x An , y An )} composed by n labeled images x i belonging to domain A, and X B = {x B1 , x B2 ,. , x Bm } composed by m unlabeled images x j belonging to domain B. Initially, a classifier C is trained in a fully supervised fashion on X A. This classifier will work as a teacher model C T at a later stage.
We propose the use of cycle-consistency image translation to deal with the issue of an unpaired and imbalanced dataset. For each image in domain x A ∈ A we will generate an equivalent translation xAB ∈ B, and for every x B ∈ B we will generate an equivalent translation xBA ∈ A. The translated images xAB and xBA are produced by the generators G AB and G BA respectively. An advantage of using cycle-consistency GANs is that an additional image x is generated, which corresponds to the reconstruction back to the original image. This can be used as additional data to train the student classifier. Therefore for every image x A we have two extra images xAB and xABA and the same for x B where we have xBA and xBAB. Then we train a multi-input classifier C S which takes as input C S (x A , xAB , xABA ) or C S (x B , xBA , xBAB ), depending on the domain of the input data.
In the case of the branch that takes as input x, random data augmentation operations are applied which include random crop, random rotation, and flipping. Backbone networks b 1 , b 2 , and b 3 , are used to extract the features of each of the 3 input images. In our case, we used as backbone ResNet-101 trained on ImageNet. The extracted features from each of the backbones are processed separately using a shallow network composed of 3 Fully Connected (FC) layers. The outputs from these layers are concatenated together, from which finally the class prediction is performed in the final layer. The classifier was trained to optimize the categorical cross-entropy loss defined as: L C (ŷ T i , ŷi ) = - i ŷT i • log(ŷ i ) ( 7 ) where ŷi is the predicted output from the student model, ŷT i is the corresponding pseudo-label provided by the teacher network, and i refers to the index over the classes.
D. Dataset For this study, endoscopic videos from 23 patients undergoing TURBT were collected, as well as the respective histopathological analysis from the resected lesions. The matching between the visual data and the histological results was done with the aid of an expert surgeon. The matching was performed by analyzing frame-by-frame the videos. The sections of the bladder from which lesions were resected during the surgical intervention were then identified. To avoid ambiguities of having multiple lesions of multiple types, only the frames in which individual lesions appeared were used in the dataset. This procedure was performed on all the WLI video clips as well as 3 patients with NBI video data. In total 4 classes were defined. Taking into consideration the general classification of BC as defined in  [2]  by the WHO and the International Society of Urological Pathology (ISUP), two categories were considered for cancerous tissue: Low-Grade Cancer (LGC) and High-Grade Cancer (HGC).
Additionally, 2 extra categories were considered for No Tumor Lesion (NTL) which comprehends cystitis, caused by infections or other inflammatory agents, and Non-Suspicious Tissue (NST). The detailed statistics of the dataset are shown in Table  I. The videos were acquired at the European Institute of Oncology (IEO) at Milan, Italy. Each patient signed an informed consent document approved by the IEO and in accordance with the Helsinki Declaration. No personal data was recorded. *The total number of patient cases does not correspond to the sum of the second column since some of the patients had more than one type of lesion. To determine if the use of more data helps to achieve better generalization when training the GAN networks, we used additional data from the datasets presented in  [14] ,  [28]  which contains endoscopic images from colonoscopy in NBI and WLI domains, and  [62]  which contains unlabeled data from TURBT as well in NBI and WLI domains.
E. Model Implementation The model was trained in three steps. First, the cycle consistency GAN was trained for 150 epochs with an initial learning rate of 2e-4 and batch size of 1. The λ hyperparameters were set to λ 1 =λ 2 =2. 0, and λ 3 =λ 4 =1. 0 The second step consisted of training the teacher classifier using the labeled dataset X A. Once the GAN model and the teacher networks were trained, the multi-input classifier was trained setting the initial learning rate at 1e-5 using a batch size of 32. The models were implemented using Tensorflow 2. 5 in Python 3. 6 and deployed on an Nvidia GeForce GTX 1080 GPU. The training of the classifiers was repeated 10 times for each of the different experiments carried out in this study. For performance benchmarking of the classifiers, a hold-out strategy was used, 4 patient cases randomly chosen were held as test dataset.
The rest of the dataset was divided randomly in a 75/25 ratio for training/validation. In the case of the GAN models, only the train dataset used for supervised classification was used during the training of the different combinations described in Table  II. For the semi-supervised training apart from using the labeled WLI images and unlabeled NBI, all NBI cystoscopy images described in  [62]  were added to the training dataset. The test dataset for the semi-supervised task remained the same as the one used to test the performance of the teacher model.
F. Evaluation protocol Each of the different modules that comprise the proposed method was evaluated separately, and the best components of each one were chosen. D A : our dataset. D B : dataset described on  [62]. D C : dataset described on  [28]. D D : dataset described on  [14]  In contrast with other DL models that are trained to minimize a loss function, GAN models are trained to converge to an equilibrium between the generator and the discriminator networks. For this reason, there is no objective loss function to train this type of model, and compare their performance objectively  [51]. However, there are some quantitative techniques that have been proposed to assess the performance of GAN models  [63]. 1) Quantitative Evaluation of the Generators: Generator models are usually evaluated based on the quality of the images they generate.
However, this type of evaluation might not fully show the performance of the models and might be subjective due to biases of the reviewer  [63]. In this regard, some authors have proposed the use of different metrics such as the Inception score, to quantitatively evaluate the quality of the generated images  [51]. In our specific case, we have the limitation that the dataset does not correspond to natural images, such as the ones on ImageNet dataset, and therefore we can not apply the Inception score directly. We use instead the Fréchet Inception Distance (FID) proposed in  [64] , to quantify the performance of each generator trained and defined as: d 2 ((m, C), (mω, Cω)) = ∥m -mω∥ 2 2 + T r C + Cω -2(CCω) 1/2 (8) were m, C are the mean and covariance obtained from the last pooling layer of an Inception model using sample images produced by the generative model respectively, and mω, Cω are the corresponding ones using images from the original dataset.
We also analyze how the amount of data affects the quality of the images and the classifiers' performance. For this purpose, we use 3 different combinations of datasets coming from 4 different sources. The datasets composition is shown in Table  II. To measure the sensitivity of the models depending on the amount of data used, we analyze the sensitivity to noise for each of the generative models trained on the different datasets as proposed in  [65]. We added zero-mean Gaussian noise N (0, σ) in a range of σ = [0. 025, 0. 05, 0. 075, 0. 1, 0. 2] to the translation result before reconstruction. We compute the Mean Square pixel Error (MSE) of the reconstructed image with respect to the original image x i and calculate the sensitivity (SN) using the equation: SN = 1 N N i=1 M SE(Gp(Gq(x i ) + N (0, σ) -x i ) (9) We compared the sensitivity for each of the generators in the proposed Cycle Similarity network (CSi-GAN) and the baseline CycleGAN.
A. Evaluation of the GAN models The first set of results corresponds to the qualitative assessment of the synthetically generated images. Samples of randomly chosen generated images by the different GAN models trained are shown in Fig. 3. A visual comparison shows that the amount and diversity of training data improve the quality of the images. We can observe that the addition of data helps the network learn the existence of other objects which do not correspond to the anatomical structures in the body, such as tools or bubbles. This shortcoming where the networks tend to disappear external structures by coloring them with the same hue as the rest of the background is more perceptible when models are trained with small datasets (D 1 ). Furthermore, in these cases, the network also presents some noticeable flaws since sometimes the generated images present black dots scattered at diverse points.
Nevertheless, the use of only external data (D 2 ) also alters the hue of the translation. This could be linked to the fact that the external data comes mainly from GI images which present different tints and anatomical formations than the ones present in the bladder. In general, for both cases cycleGAN and CSi-GAN the use of the more general dataset (D 3 ) which comprises data from the same anatomical target and external data produce the best quality images. However, still some image artifacts such as specularities, reflections, interlacing, etc. appear in the generated images without being present in the original one. The most significant improvement comes from using the L sim loss to train the GANs. The fine-grain details, such as small vessels, are better preserved and highlighted after the translations, and it also helps to reduce the amount of noise in the image.
Similar behaviors can be observed in the video material attached to this manuscript. 1) Quantitative Evaluation of the GAN: To evaluate the quality of the images generated by the GAN models the FID score and the AUC of the sensitivity curve were used. The results obtained for both metrics are shown in Table  III. The model that obtains the best metrics for both cases, i. e. lower values, is the proposed CSi-GAN when trained on D 3. In the case of FID score there is a clear difference between CycleGAN and CSi-GAN regardless of the dataset used for training, with CSi-GAN obtaining in general better results. In the case of the AUC of the Sensitivity curve, the difference between the two models is not that obvious. This could be associated with the fact that neither of the networks is designed from the origin to be noise-resistant.
However, there is a clear tendency that the addition of data makes CSiGAN more resistant to the addition of noise than its counterpart CycleGAN. This might be related to the fact that even if the addition of more data helps CycleGAN to generalize better in domain translation the lack of a structural loss inhibits it to discern properly between the correct information to produce a satisfactory translation, and the information that seems useful but is just noise. This could also explain the reason why CycleGAN obtains better metrics when trained on dataset D 2 than on D 3 since the quality of the images of D 2 is higher and less noisy. 2) Evaluation by Medical Specialists: In order to perform a more exhaustive analysis, a protocol was implemented to acquire feedback from expert clinicians in the field of endoscopy as described in sec.
III-F. 2. A total of 20 physicians from 10 different institutions familiar with TURBT participated in the study. Of this, 15 corresponded to Expert Surgeons (ES) and 5 to Residents (RE). For this analysis we choose the generative model which obtained the best FID score and AUC values, i. e. CSi-GAN trained on dataset D 3 , to generate the synthetic images. The results regarding the ability of surgeons to discern between real and synthetic images are shown in table IV. The results are split in 3 categories to evaluate separately each translation (WLI → NBI and NBI → WLI) and therefore each generator independently, as well as the overall performance of the GAN (ALL). For both groups of participants (ES and RE), the results show slightly better results in the translation WLI → NBI for all metrics.
This might be related to the fact that there are more sample images in the WLI training dataset than in the NBI and therefore the generator G AB is able to generalize better and produce better quality images than its counterpart G BA. The overall AU C for ES is 0. 59 and 0. 52 for RE, meaning that their performance is marginally better than what a random binary classifier could achieve, confirming that the quality of the generated images is good enough to trick experts in the area. Concerning the tissue classification task, results are shown in Fig. 4. In the case of Acc there was an average improvement of 8% when using a pair of a real image and a synthetic one than when only 2 real images were shown. In the case of P rec the improvement was 19%, while no improvement or decrease was observed in the case of Rec.
1) GAN-based Tissue Classification: To test the generalization of our method, we compare the use of different networks (VGG16, VGG19, Inception V3, Desenet, ResNet-50, and ResNet-101) trained in a fine-tuning fashion against the implementation of these same networks in our GAN-based classification method. CSi-GAN trained on D 3 was chosen as the as the translation network. Results in terms of ACC, M CC and F-1 score are shown in Table  V. Overall the use of the proposed GAN-based method obtains better metrics than the baseline networks. In the majority of the cases, there is little improvement or no improvement when the input image is in the WLI domain. This uneven behavior in terms of the classification improvement might be related to the fact that WLI images are more similar to the natural images dataset in which the models were originally pre-trained (ImageNet).
However, there is a noticeable improvement when it comes to the classification of NBI images where most of the base-line shows poor performances. 2) Semi-supervised Classification: We compared the use of GAN-based classification trained in a fully supervised way against the use of semi-supervised classification. In both cases, only the Multi-Input classifier weights were trained while the ones of the Cycle-Consistency Network remained constant. For these experiments, CSi-GAN pre-trained on each of the D k datasets were used. The results of these experiments are shown in Fig. 5  in terms of ACC, F-1 score, and M CC. On average the improvement, in terms of ACC, F-1 score, and M CC, of using CSiGAN trained in a fully supervised way against the training in a semi-supervised fashion was of 8%, 6%, and 9% respectively.
From these results, we can see that in general, all the models obtain better results than the specialists, and the major improvement comes from the use of a semi-supervised approach. However, the improvement obtained in the domain for which there are no labels when using domain translation is also noticeable. As expected, the integration of both results in the best performance, and improves considerably the detection of classes that are underrepresented. This behavior is more clearly noticeable in the case of the NTL class which in our dataset has the smallest number of samples and in contrast to NST could be easily misclassified as a tumorous lesion. An additional analysis was performed in order to determine if the quality of the GAN-translated images influence the classifier performance.
Nevertheless, additional research needs to be carried out to find accurate metrics to assess the quality of generated images objectively and to determine to which point it might be related to the classification performances. Future work includes further validation of multi-center data, as well as the acquisition of data from other imaging domains which could help to assess better the generalization of the method, and the development of lesion detection methods that could differentiate specific image regions that correspond to the lesion and non-lesion tissue. By making available our dataset we hope to encourage further research in the field that could motivate the clinical translation of endoscopic image classification.
Ethical Approval The proposed study is a retrospective study. No personal data was  Informed consent Written informed consent was obtained from all patients included in the study. TABLE VI: Ablation results. The average± standard deviation for each metric is presented in terms of the type of data in the test dataset (WLI and NBI) and the combination of both (ALL), for each of the models. To have a reference point, the results obtained from physicians are shown too divided by specialists and residents. The table shows in which cases Domain Translation (DT) and Unlabeled Data (UD) were used during the training. The experiments to examine the impact of each of the branches (b 1 m, b 2 , b 3 ) in the multi-input classifier were performed in a fully supervised (FS) way in order to analyze the effects only of the translations performed by the GAN.