2.2. Step 2. Training process
In this project, a key focus was exploring different configurations for the fully connected
layers in order to optimize performance with pretrained models. By experimenting
with various layer architectures, the goal was to effectively adapt these pretrained
models (e.g., ResNet, DenseNet) to the specific task of Helicobacter pylori detection
in histological images. This approach allowed for leveraging the pretrained features
while customizing the final classification layers to improve sensitivity and accuracy in
detecting H. pylori.
Before training our deep learning models, a set of different configuartions are estab-
lished to explore different model architectures and hyperparameters (model config.py).
This systematic approach ensures that the most effective configurations can be iden-
tified.
6
These are the guidelines:
• Model Architectures (model options): Consider two pre-trained models as
the base of the architecture: resnet50 and densenet201.
• Number of Dense Layers (num layers options): We explore models with 1
dense layer, 2 dense layers, and 3 dense layers.
• Number of units (units options): For each dense layer consider 64 units,
128 units, and 256 units.
• Dropout Rate (dropout options): Finally, a fixed dropout rate of 0.25 is
used.
For each num layers (1, 2, or 3), we generate all possible combinations of units per-
layer options repeated num layers times. We then filter these combinations to re-
duce redundancy and focus on diverse architectures.
Before applying any filtering, there was a total of 90 different configurations. So,
we implemented a line of code removing configurations where all two layers or three
layers had identical units or insufficient diversity in unit values. The total number of
different was achieved to be reduced to 66 configurations.
Therefore, with the 66 configurations that had been generated, we proceeded to the
training phase of our project. The goal was to train each of these models individually
to evaluate their performance and identify the most effective architectures for our task.
During the training of each model, the key performance metrics were meticously
recorded, specifically the training accuracy and training loss at each epoch. These
metrics are essential for assessing how well the model is learning from the data and
for detecting any issues such as overfitting or underfitting.
Upon completion of the training phase for all configurations, a comparative analysis
of the collected performance metrics was undertaken. This analysis involves reviewing
the accuracy and loss trends to determine which models have achieved the highest levels
of accuracy while maintaining acceptable loss values. Models that exhibit superior
performance in terms of converging quickly to high accuracy and demonstrating stable
loss reduction are considered strong candidates.
In the following figure (Figure 2 ) it can be seen what we had to choose the best
models.
Figure 2: Accuracies and losses lists for all configurations.
7
With the chosen best models, we will undergo further phases of training, exper-
imenting with epochs, learning rate, etc. And after that, they will be tested and
evaluated.
For the model architecture, pre-trained models like ResNet50 and DenseNet201
have been used as backbones, leveraging transfer learning to fine-tune them for our
classification task.
Custom fully connected layers are appended to the backbone networks to adapt
them to our specific problem, with configurations based on task complexity and data
availability.
Cross-entropy loss with class weights is selected to address class imbalance, ensuring
the model pays equal attention to all classes. On the other side, the Adam optimizer
is chosen for its adaptive learning rate capabilities, enhancing convergence speed and
stability.
Training involves iterating over the dataset for multiple epochs, where the model
processes batches of data, computes loss, and updates weights through backpropaga-
tion. We record metrics such as training loss, accuracy, and recall after each epoch to
monitor progress and determine the optimal stopping point.
Upon completion of training, the model is evaluated on a validation set using met-
rics like validation loss, accuracy, precision, recall, and F1-score to assess performance
comprehensively.
The trained models are saved for future use, and plots of training and validation
metrics are generated to visualize learning dynamics.
By experimenting with various architectures and hyperparameters, and by making
informed choices based on data characteristics and theoretical considerations, the aim
was to identify the model configuration that achieves optimal performance. This
methodical approach is essential for developing robust models capable of accurate
predictions on complex datasets.
2.2.2. Metrics
The following figure (Figure 3 ) illustrates the plots of accuracy, loss, and recall that
were shown after each configuration was trained. This is an example.
Figure 3: Accuracy, Loss training and Recall plot of a DenseNet201 with 2 layers.
8
2.3. Step 3. Testing Process (patch classifier)
The testing process performs the patch-level recognition. By taking the loader for test
created by the stratified Kfold, it can be seen how each model is able to predict the
correct class for patches. This is done in the file evaluation test patces.py
2.3.1. Models
Based in the code created the evaluation is done sequencially in a loop for all the
configuartion models (previously saved) and each of the folds.
2.3.2. Metrics
There are several function related to the matrices computation:
plot confusion matrix(conf matrix, save path=None):
This function visualizes the confusion matrix using a heatmap generated by seaborn.
It labels the axes with ”True label” and ”Predicted label” and saves the confusion ma-
trix as an image if a save path is provided.
save evaluation results(evaluation results, save folder, model filename):
This function saves the evaluation metrics (accuracy, precision, recall, F1 score,
AUC) to a CSV file. It helps in tracking the performance of models across multiple
runs.
evaluate model with classifier(model, val loader, device, save curve folder, model name):
This function is used to evaluate classification models. The model’s predictions
are compared to the ground truth labels to compute traditional metrics like accuracy,
precision, recall, F1 score, and confusion matrix. Additionally, an ROC curve is gen-
erated, and the AUC is calculated to assess the modelˆas ability to distinguish between
classes.
They are crucial for understanding how a model performs on test data. Each of
these metrics has its own focus for measuring performance, and together they provide
a more complete picture of the model’s effectiveness. Below are detailed descriptions
of the main metrics used and their relevance:
Accuracy: It measures how well the model classifies both positive and negative
classes correctly. Its formula is:
Accuracy = Correct Predictions
Total Predictions
Although it is useful, it may not be a reliable metric in this case as we have an
imbalanced datasets. In these cases, a model may predict the majority class all the
time and still achieve high accuracy, without truly being effective.
9
Precision: Measures the proportion of positive cases correctly predicted in relation
to all cases the model classified as positive. Its formula is:
Precision = True Positives (TP)
True Positives (TP) + False Positives (FP)
High precision means the model has few false positives, which is important in situ-
ations where a false positive can have bad consequences, such as in disease detection.
Recall (Sensitivity): Measures the model’s ability to identify all the actual positive
cases. Its formula is:
Recall = True Positives (TP)
True Positives (TP) + False Negatives (FN)
A model with high recall is particularly useful when you want to minimize false
negatives, such as in the diagnosis of serious diseases, where missing a positive case
could have serious consequences.
F1 Score: This metric is especially useful when a balance between precision and
recall is needed, and is commonly used when classes are imbalanced. The F1 score is
particularly useful when it’s not possible to achieve both metrics at once. Its formula
is:
F1 Score = 2 × Precision × Recall
Precision + Recall
A higher F1 score indicates better model performance in terms of balancing preci-
sion and recall.
Confusion Matrix: The confusion matrix is a visual tool that shows the classifica-
tion results by comparing the model’s predictions to the actual values. This matrix
contains four key values:
- True Positives (TP): Cases correctly identified as positive.
- False Positives (FP): Cases incorrectly identified as positive.
- False Negatives (FN): Cases incorrectly identified as negative.
- True Negatives (TN): Cases correctly identified as negative.
The confusion matrix allows you to clearly observe the types of errors made by the
model and helps identify patterns in false positives or false negatives. It is a valuable
tool for evaluating model performance and adjusting it as needed.
ROC Curve (Receiver Operating Characteristic): illustrates the performance of a
binary classifier as the decision threshold varies. The curve shows the relationship
between the true positive rate (TPR) and the false positive rate (FPR) for different
thresholds.
The AUC (Area Under the Curve) is a value that summarizes the ROC curve’s
performance. An AUC of 1.0 indicates perfect performance, while an AUC of 0.5
suggests that the model has no discriminative ability, similar to random classification.
A high AUC indicates that the model is good at distinguishing between positive and
negative classes.
The ROC curve is useful for evaluating the model’s ability to differentiate between
classes under different threshold conditions and is particularly valuable when working
with imbalanced classification problems.
10
2.4. Step 4. Testing Process (patient diagnosis)
This step evaluates using a subset of cropped images and takes the Pat ID index
selection from the training and translate this into the new dataset.
2.4.1. Models
For validation same models are called for evaluation. However, for this case
2.4.2. Metrics
The matrics codified for better understanding the results of the predicted patients are
the same to the the patch-level ones.
3. Experimental Design
For this experiment all the annotated data has been for training all the models (2975
samples). However, the patches ground truth shows a high imbalance between positive
and negative values: 330 approximately for first class and 2650 for the negative one.
This is one of the main challenges of this project as being able to detect positive cases
is undoubtedly important. Furthermore, this project has two different levels: patch-
level and patient-level. With the annotated data we train the model and with the
kfold partitions for test it is verified how well the model learns at patch level. After
that we have the cropped data. This are all the patches for each patient but they are
annotated at the patient-level. In this case our experiment to speed up the acquisition
of results and all the computations. Our experiment selects randomly 10 patches for
each patient, and with this subset and the kfold index for train it predicts at patient
level. In addition, different methods are proposed. Densenet201 and Resnet50 have
been used as the pretrained networks and then a fully connected layer has been created.
3.1. Dataset Description
All the data is provided by QUIRON center and the annotations at patch level were
done by an specialist. At patch-level each image has its own window ID (section of
an image of a patient). In addition, they have the Presence value (contain Hyplori or
not) and all have a Pat ID to determine from who is the image. At patient-level only
the CODI (Pat ID) and the DENSITAT (Presence per image) is given.
Figure 4: Data distribution in annotated file (negatives and positives).
11
3.2. Evaluation and Metrics
In this analysis, various experiments and tests have been carried out to evaluate the
models used to classify the histopathological images.
Once the 66 configurations of the two models (ResNet50 and DenseNet201) were
obtained, the best models for each layer and models were chosen from the training
plots, two models were chosen for each type and for each layer, having a total of 12
models. The rule for choosing the models was to choose the best model from each
layer and then choose the other model from the same layer, so that they could be
compared. For example, if the DenseNet201 from a layer [256] had given good results,
the ResNet50 [256] was chosen to compare it.
These 12 models were chosen to train them with some changes: different learning
rate values, increasing the number of epochs, adding a more aggressive threshold to give
more importance and weight to the positive images. In order to have a comprehensive
evaluation and analysis of the 12 models (it must be taken into account that for each
configuration, the K-fold was applied, and in total there were 36 models for each test).
To analyze the results, the models were evaluated in two different ways. On the one
hand, the models were evaluated by patches (images). On the other hand, they were
evaluated by patient with the images available for each one (there are more than one
image per patient). For both methods, the ROC Curve, confusion matrix, accuracy,
recall, precision, f1 - score and AUC were used.
Each method was tested under specific configurations to ensure optimal perfor-
mance:
• Configurations that have been made in the testing of the chosen mod-
els:
– Configuration: Batch size of 32.
– Thresholds for binary cross-entropy: [0.5, 0.5, 0.35].
– Learning rates: [1e-4, 5e-7, 1e-4].
– Number of epochs: [25, 35, 25].
• DenseNet201:
• DenseNet201 [128] (Chosen as the best).
• DenseNet201 [256] (Chosen for comparison with resnet50).
• DenseNet201 [64, 128] (Chosen as the best).
• DenseNet201 [128, 256] (Chosen for comparison with resnet50).
• DenseNet201 [128, 64, 256] (Chosen as the best).
• DenseNet201 [64, 256, 64] (Chosen for comparison with resnet50).
• ResNet50:
• ResNet50 [128] (Chosen for comparison with densenet201).
• ResNet50 [256] (Chosen as the best).
• ResNet50 [64, 128] (Chosen for comparison with densenet201).
• ResNet50 [128, 256] (Chosen as the best).
• ResNet50 [128, 64, 256] (Chosen for comparison with densenet201).
• ResNet50 [64, 256, 64] (Chosen as the best)

SUMMARIZE THIS INTO 4 MINUTES TALK FOR A PRESENTATION. DEEP DETAIL?