MEDICAL IMAGING
HPylori in inmunohistochemical
histological images
Submitted by:
Miguel Arpa Robig, 1668833
Meritxell Carvajal Librado, 1671647
Lara Rodriguez Cuenca, 1667906
Vision and Learning
15th of October of 2024
1
1. Introduction
In recent years, detection and recognition systems have seen widespread application
across diverse fields, including medical diagnostics, security, and surveillance.
This project focuses on the development of an AI-based system for automated de-
tection and classification in histopathological images, specifically targeting the identi-
fication of the Helicobacter pylori bacterium.
Two distinct approaches are explored in this project: a classification-based ap-
proach using deep learning neural networks and an anomaly detection approach lever-
aging autoencoders. Pretrained models are employed in the classification approach
to address several challenges inherent in histopathological image analysis, such as the
limited sample size of whole slide images (WSIs), the large dimensions of these images,
and the unbalanced distribution of H. pylori across image patches.
GitHub [1] serves as the primary platform for storing and managing the code and
other files of the project. The main reasons of its use are because it allows collaboration
between multiple people in an efficient way and, at the same time, the progress is
stored online without the fear of losing it. However, GitHub’s memory limitations
posed challenges, as the increased size of the data caused slower commit processes
Additionally, MobaXterm [2] was used to streamline remote access to computa-
tional resources required for handling high-resolution histopathological images. This
tool proved essential for efficiently managing the computational demands of processing
these images
1.1. Goals
The primary goals of this project are divided into system implementation, comparative
performance evaluation, and reproducibility testing:
1. Implement two detection systems:
• Patch Classification: Develop models for detecting H. pylori presence at the
patch level, including both anomaly detection and classification approaches.
• Patient Diagnosis: Implement a model that aggregates patch-level outputs
for patient diagnosis, incorporating adaptive thresholding and attention
mechanisms.
2. Evaluate and compare model performance:
• Cross-validation: Perform cross-validation at both patch and patient levels
to evaluate model robustness.
• Reproducibility Testing: Test model reproducibility, focusing on achieving
consistent results at the patient level.
1
Further project goals include detailed data preparation, model training, and eval-
uation processes:
• Data Extraction and Image Transformation: Create several functions to extract
de zip folders, rename images based on how they were annotated (in csv) to
make them manageable for analysis.
• Model Training: Implement models based on ResNet [3], DenseNet [4], and au-
toencoders, exploring architectures suited to the unique features of histopatho-
logical images. Use cross-validation strategies, such as k-fold cross-validation, to
ensure robust performance across different data splits.
• Adaptive Thresholding and Attention Mechanisms: Use adaptive thresholding to
optimize sensitivity and specificity at the patch level, distinguishing H. pylori-
rich patches from non-infected regions. Implement attention mechanisms to
enhance model focus on relevant regions, improving interpretability by guiding
the model’s attention to essential image areas.
• Evaluation Metrics: Use a comprehensive set of evaluation metrics, includ-
ing ROC curves, accuracy, sensitivity, and specificity, to assess model perfor-
mance. Employ ROC-AUC and precision-recall analyses to better understand
the model’s effectiveness in distinguishing positive and negative cases. In addi-
tion consider using standard deviation for later reproducibility assumptions.
2. Methodology
This section provides a detailed explanation of the methodology used to address the
detection and classification of Helicobacter pylori (H. pylori) in immunohistochemical
(IHC) histological images. It outlines the complete workflow, from data preprocessing
to model evaluation, including the specific algorithms, strategies, and system design
choices implemented in this project.
The general workflow, depicted in Figure 1, encompasses the following stages:
1. Data Management:
(a) Extracting files and data transformations: Data has been organized in fold-
ers based on their ground truth names to better comparison after training.
Additionally, all images went through a minimal transformation process
to ensure all have same dimensions and were changed to tensors for later
combination with torch [5] functions.
(b) Load functions and Kfold usage: Images and labels is combined together
and DataLoader is created for training and validation set. Nevertheless,
data was separated by patients (to avoid leakage) and the sets are created
using folds so that the results are not dependable on the data.
2
2. Training process:
(a) Models: Select predefined models for medical imaging and create some
configurations for a fully connected block.
(b) Metrics: Loss, accuracy and recall are measured to control the results of
the models.
3. Testing Process (patch classifier):
(a) Models: Trained models are evaluated with the validation loader from the
kfold splitting.
(b) Metrics: Use accuracy, recall, precision, AUC and confusion matrics to
evaluate the performance of the model.
4. Validation Process (patient diagnosis):
(a) Models: Evaluate the models at a patient-level using train kfold to select
same patient ID.
(b) Metrics: Use accuracy, recall, precision, AUC and confusion matrics to
evaluate the performance of the model.
Figure 1: Methodology Workflow.
2.1. Step 1. Data Management
In this section it is presented how has the data been treated, all the different folders
and files that have been created, and their explanation for better understanding of the
process.
2.1.1. Extracting files and data transformations
In this project, a dataset comprising histopathological images and corresponding an-
notations has been used for classification and detection tasks. The data preparation
involves several steps, including extracting image files from compressed archives, or-
ganizing them into usable directories, and creating annotation files in CSV format.
Below is a detailed explanation of the data we use, how we obtain it, and the purpose
of each data file.
3
Data Files and Their Purpose
1. Compressed Archives (CrossValidation.zip and HoldOut.zip): These ZIP files
contain the raw image data for cross-validation and hold-out testing.
(a) CrossValidation.zip includes either Annotated.zip (patch-level images with
annotations) or Cropped.zip (patient-level images without annotations), de-
pending on the chosen analysis level.
(b) HoldOut.zip contains HoldOut.zip inside it, which further contains images
for final testing.
2. Image Directories (EXTRACT annotated, EXTRACT cropped, EXTRACT holdout):
After extracting the ZIP files, the images are organized into these directories for
further processing.
(a) EXTRACT annotated: Contains patch-level images extracted from Anno-
tated.zip.
(b) EXTRACT cropped: Contains patient-level images extracted from Cropped.zip.
(c) EXTRACT holdout: Contains hold-out images for final model evaluation.
3. Usable Image Directories (USABLE annotated, USABLE cropped): These di-
rectories contain filtered and processed images that are ready for model training.
(a) USABLE annotated: Contains images from EXTRACT annotated that
have corresponding annotations and meet certain criteria.
(b) USABLE cropped: Contains selected images from EXTRACT cropped af-
ter filtering based on specific conditions, such as density.
4. Annotation Files (TRAIN DATA annotated.csv, TRAIN DATA cropped.csv,
TRAIN DATA holdout.csv)
(a) TRAIN DATA annotated.csv: Contains patch-level annotations, including
patient IDs (Pat ID), window IDs (Window ID), and presence values (Pres-
ence), indicating the presence of specific features in the patches.
(b) TRAIN DATA cropped.csv: Contains patient-level annotations with pa-
tient codes (CODI) and density (DENSITAT), which is mapped to binary
values for classification.
(c) TRAIN DATA holdout.csv:
5. Excel Files (HP WSI-CoordAnnotatedPatches.xlsx, PatientDiagnosis cropped.csv,
all annotations with patient diagnosis.csv):
(a) HP WSI-CoordAnnotatedPatches.xlsx: Contains patch-level annotations
that are used to create TRAIN DATA annotated.csv.
(b) PatientDiagnosis cropped.csv: Contains patient-level diagnosis information,
used to create TRAIN DATA cropped.csv.
(c) all annotations with patient diagnosis.csv:
4
Data Preparation Steps
The data preparation workflow begins with the systematic extraction of images
from archival datasets, specifically CrossValidation.zip and HoldOut.zip. Utilizing
Python’s zipfile module ensures efficient and reliable decompression. Depending on
the analytical focusˆawhether at the patch-level or patient-levelˆawe extract either An-
notated.zip or Cropped.zip from the CrossValidation.zip archive. The images are
carefully deposited into designated directories, namely EXTRACT annotated or EX-
TRACT cropped, to streamline subsequent data handling processes. Post-extraction,
the images, initially nested within patient-specific directories, are reorganized to create
a flattened directory structure.
This involves renaming and relocating files, employing a consistent naming conven-
tion that combines patient identifiers with sequence numbers. Flattening the directory
hierarchy is crucial as it facilitates more straightforward data access patterns during
model training, thus enhancing computational efficiency. To curate datasets that are
both relevant and high-quality for training, a rigorous filtering criteria was applied.
For patch-level images, inclusion is ensured by cross-referencing image filenames
with records in HP WSI-CoordAnnotatedPatches.xlsx, thereby guaranteeing that only
images with valid annotations are selected. At the patient-level, density informa-
tion from PatientDiagnosis cropped.csv is used to select patients exhibiting specific
histopathological characteristics, thereby aligning the dataset with the intended diag-
nostic objectives. Annotation files undergo meticulous processing to produce struc-
tured CSV files required for training. In TRAIN DATA annotated.csv, this ensure a
comprehensive mapping between images and their corresponding patch-level annota-
tions. For patient-level data encapsulated in TRAIN DATA cropped.csv, ’DENSITAT’
is transform into a binary labels (0 or 1). This binarization is essential as it refor-
mulates the problem into a binary classification task, simplifying the model’s output
layer design and aligning with binary cross-entropy loss functions.
Recognizing that data quality is paramount, with the implementation of robust
data cleaning and filtering protocols. This includes handling missing values, correct-
ing discrepancies, and ensuring consistency across datasets. For example, we exclude
patch-level images lacking valid ’Presence’ annotations to prevent noise in training.
Additionally, class imbalance is adressed by balancing the dataset concerning the num-
ber of images per patient and class. Balancing is critical as it prevents the model from
becoming biased toward majority classes, ensuring equitable learning across all classes.
Data Exploration
Prior to model development, an in-depth data exploration phase is conducted to
comprehend the intrinsic properties of the dataset. This encompasses assessing the
distribution of classes within both patch-level and patient-level datasets. Understand-
ing class distribution is vital, as significant imbalances can bias the model’s learning
process. Identifying such imbalances informs the decision to adjust loss functions by
incorporating class weights or to employ resampling techniques, thereby enhancing
model fairness and performance. For that reason, since it was found that certain
classes are underrepresented, hyperparameters such as learning rate, batch size, and
the number of epochs were adjusted based on dataset size and complexity to optimize
model training.
5
2.1.2. Load functions and Kfold usage
K-Fold cross-validation is a powerful method for evaluating model performance, es-
pecially in contexts like medical imaging, where avoiding data leakage is crucial. By
implementing it at the patient level, we ensure that all data related to a single patient
is either in the training or validation set, never both. This avoids artificially inflating
performance metrics and better simulates real-world conditions.
The process begins by organizing the data into groups based on unique patient
identifiers such as Pat ID. Each group contains all images for a particular patient,
alongside the corresponding label or diagnosis. To maintain class balance across folds,
a stratified K-Fold splitter is employed. Stratification ensures that each fold has a
similar distribution of classes, which is essential in datasets with class imbalances.
For every fold, the splitter generates two sets of patient IDs: one for training
and the other for validation. These IDs are then used to filter the dataset, creating
training and validation subsets. Only images associated with the selected patient
IDs are included in their respective sets, ensuring the integrity of the patient-level
separation.
Class weights are computed based on the training set’s class distribution for each
fold. These weights are incorporated into the model’s loss function to mitigate the
effects of class imbalance, preventing the model from favoring the majority class.
To manage the iterative nature of K-Fold cross-validation, the indices, patient IDs,
and labels for each fold are saved. This not only ensures reproducibility but also
enables further analysis or experimentation using the same splits.
An appropriate number of folds (3-fold) was chosen based on the dataset size
and computational resources. The model is trained on the data from k-1 folds and
evaluated on the remaining fold. This process repeats for all folds, resulting in k sets
of performance metrics, such as accuracy, precision, recall, and loss. Aggregating these
metrics provides a comprehensive view of the model’s performance and generalizability.
2.2. Step 2. Training process
In this project, a key focus was exploring different configurations for the fully connected
layers in order to optimize performance with pretrained models. By experimenting
with various layer architectures, the goal was to effectively adapt these pretrained
models (e.g., ResNet, DenseNet) to the specific task of Helicobacter pylori detection
in histological images. This approach allowed for leveraging the pretrained features
while customizing the final classification layers to improve sensitivity and accuracy in
detecting H. pylori.
Before training our deep learning models, a set of different configuartions are estab-
lished to explore different model architectures and hyperparameters (model config.py).
This systematic approach ensures that the most effective configurations can be iden-
tified.
6
These are the guidelines:
• Model Architectures (model options): Consider two pre-trained models as
the base of the architecture: resnet50 and densenet201.
• Number of Dense Layers (num layers options): We explore models with 1
dense layer, 2 dense layers, and 3 dense layers.
• Number of units (units options): For each dense layer consider 64 units,
128 units, and 256 units.
• Dropout Rate (dropout options): Finally, a fixed dropout rate of 0.25 is
used.
For each num layers (1, 2, or 3), we generate all possible combinations of units per-
layer options repeated num layers times. We then filter these combinations to re-
duce redundancy and focus on diverse architectures.
Before applying any filtering, there was a total of 90 different configurations. So,
we implemented a line of code removing configurations where all two layers or three
layers had identical units or insufficient diversity in unit values. The total number of
different was achieved to be reduced to 66 configurations.
Therefore, with the 66 configurations that had been generated, we proceeded to the
training phase of our project. The goal was to train each of these models individually
to evaluate their performance and identify the most effective architectures for our task.
During the training of each model, the key performance metrics were meticously
recorded, specifically the training accuracy and training loss at each epoch. These
metrics are essential for assessing how well the model is learning from the data and
for detecting any issues such as overfitting or underfitting.
Upon completion of the training phase for all configurations, a comparative analysis
of the collected performance metrics was undertaken. This analysis involves reviewing
the accuracy and loss trends to determine which models have achieved the highest levels
of accuracy while maintaining acceptable loss values. Models that exhibit superior
performance in terms of converging quickly to high accuracy and demonstrating stable
loss reduction are considered strong candidates.
In the following figure (Figure 2 ) it can be seen what we had to choose the best
models.
Figure 2: Accuracies and losses lists for all configurations.
7
With the chosen best models, we will undergo further phases of training, exper-
imenting with epochs, learning rate, etc. And after that, they will be tested and
evaluated.
For the model architecture, pre-trained models like ResNet50 and DenseNet201
have been used as backbones, leveraging transfer learning to fine-tune them for our
classification task.
Custom fully connected layers are appended to the backbone networks to adapt
them to our specific problem, with configurations based on task complexity and data
availability.
Cross-entropy loss with class weights is selected to address class imbalance, ensuring
the model pays equal attention to all classes. On the other side, the Adam optimizer
is chosen for its adaptive learning rate capabilities, enhancing convergence speed and
stability.
Training involves iterating over the dataset for multiple epochs, where the model
processes batches of data, computes loss, and updates weights through backpropaga-
tion. We record metrics such as training loss, accuracy, and recall after each epoch to
monitor progress and determine the optimal stopping point.
Upon completion of training, the model is evaluated on a validation set using met-
rics like validation loss, accuracy, precision, recall, and F1-score to assess performance
comprehensively.
The trained models are saved for future use, and plots of training and validation
metrics are generated to visualize learning dynamics.
By experimenting with various architectures and hyperparameters, and by making
informed choices based on data characteristics and theoretical considerations, the aim
was to identify the model configuration that achieves optimal performance. This
methodical approach is essential for developing robust models capable of accurate
predictions on complex datasets.
2.2.2. Metrics
The following figure (Figure 3 ) illustrates the plots of accuracy, loss, and recall that
were shown after each configuration was trained. This is an example.
Figure 3: Accuracy, Loss training and Recall plot of a DenseNet201 with 2 layers.
8
2.3. Step 3. Testing Process (patch classifier)
The testing process performs the patch-level recognition. By taking the loader for test
created by the stratified Kfold, it can be seen how each model is able to predict the
correct class for patches. This is done in the file evaluation test patces.py
2.3.1. Models
Based in the code created the evaluation is done sequencially in a loop for all the
configuartion models (previously saved) and each of the folds.
2.3.2. Metrics
There are several function related to the matrices computation:
plot confusion matrix(conf matrix, save path=None):
This function visualizes the confusion matrix using a heatmap generated by seaborn.
It labels the axes with ”True label” and ”Predicted label” and saves the confusion ma-
trix as an image if a save path is provided.
save evaluation results(evaluation results, save folder, model filename):
This function saves the evaluation metrics (accuracy, precision, recall, F1 score,
AUC) to a CSV file. It helps in tracking the performance of models across multiple
runs.
evaluate model with classifier(model, val loader, device, save curve folder, model name):
This function is used to evaluate classification models. The model’s predictions
are compared to the ground truth labels to compute traditional metrics like accuracy,
precision, recall, F1 score, and confusion matrix. Additionally, an ROC curve is gen-
erated, and the AUC is calculated to assess the modelˆas ability to distinguish between
classes.
They are crucial for understanding how a model performs on test data. Each of
these metrics has its own focus for measuring performance, and together they provide
a more complete picture of the model’s effectiveness. Below are detailed descriptions
of the main metrics used and their relevance:
Accuracy: It measures how well the model classifies both positive and negative
classes correctly. Its formula is:
Accuracy = Correct Predictions
Total Predictions
Although it is useful, it may not be a reliable metric in this case as we have an
imbalanced datasets. In these cases, a model may predict the majority class all the
time and still achieve high accuracy, without truly being effective.
9

CAN YOU DO GENERAL CONCLUSIONS, ACHIEVEMENTS AND LIMITATIONS+FUTURE WORK? THANK YOU Despite the project’s successes, several areas for improvement and expansion have
been identified:
• More configurations can be added.
• Hyperparametrization using (HPO)
• Train the autoencoder for anomaly detection
• Improve patient diagnosis by modifing or tring new models, for instance VGG