# Transformer with Selective Shuffled Position Embedding and key-patch exchange strategy for early detection of Knee Osteoarthritis

Zhe Wang, Aladine Chetouani, Mohamed Jarraya, Didier Hans, Rachid Jennane

<figure><img src="image_1.png" /></figure>

## To cite this version:

Zhe Wang, Aladine Chetouani, Mohamed Jarraya, Didier Hans, Rachid Jennane. Transformer with Selective Shuffled Position Embedding and key-patch exchange strategy for early detection of Knee Osteoarthritis. Expert Systems with Applications, 2024, 255, pp.124614. 10.1016/j.eswa.2024.124614. hal-04643953

HAL Id: hal-04643953

https://inria.hal.science/hal-04643953v1

Submitted on 10 Jun 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
TRANSFORMER WITH SELECTIVE SHUFFLED POSITION
EMBEDDING AND KEY-PATCH EXCHANGE STRATEGY FOR
EARLY DETECTION OF KNEE OSTEOARTHRITIS

Zhe Wang

IDP Institute, UMR CNRS 7013
University of Orleans
Orleans, France

zhe.wang@etu.univ-orleans.fr

Aladine Chetouani

PRISME Laboratory, EA 4229
University of Orleans
Orleans, France

aladine.chetouani@univ-orleans.fr

Mohamed Jarraya

Department of Radiology, Massachusetts General Hospital
Harvard Medical School
Boston, USA

mjarraya@mgh.harvard.edu

Didier Hans

Nuclear Medicine Division
Geneva University Hospital
Geneva, Switzerland

didier.hans@chuv.ch

Rachid Jennane †

IDP Institute, UMR CNRS 7013
University of Orleans
Orleans, France

rachid.jennane@univ-orleans.fr

July 4, 2023

ABSTRACT

Knee OsteoArthritis (KOA) is a widespread musculoskeletal disorder that can severely impact the mobility of older individuals. Insufficient medical data presents a significant obstacle for effectively training models due to the high cost associated with data labelling. Currently, deep learning-based models extensively utilize data augmentation techniques to improve their generalization ability and alleviate overfitting. However, conventional data augmentation techniques are primarily based on the original data and fail to introduce substantial diversity to the dataset. In this paper, we propose a novel approach based on the Vision Transformer (ViT) model with original Selective Shuffled Position Embedding (SSPE) and key-patch exchange strategies to obtain different input sequences as a method of data augmentation for early detection of KOA (KL-0 vs KL-2). More specifically, we fix and shuffle the position embedding of key and non-key patches, respectively. Then, for the target image, we randomly select other candidate images from the training set to exchange their key patches and thus obtain different input sequences. Finally, a hybrid loss function is developed by incorporating multiple loss functions for different types of the sequences. According to the experimental results, the generated data are considered valid as they lead to a notable improvement in the model's classification performance.

**Keywords** Vision transformer · Position embedding · Hybrid loss · Knee osteoarthritis · Data augmentation

1 Introduction

Knee OsteoArthritis (KOA) is a highly prevalent joint disease with high societal and health burden. Its prevalence
is set to increase with the aging of the population [1]. The main symptoms of KOA include pain, swelling, stiffness
<figure><img src="image_3.png" /></figure>

Figure 1: The global flowchart of this study. The data-flow is illustrated using black arrows. Green and purple blocks represent the fixed and shuffled embedding positions, respectively. For each target image $I_t$, after applying our proposed SSPE (Section 2.2), N candidate images ($I_{c1}, I_{c2}, ...I_{cn}$) are introduced for the proposed key-patch exchange operation (Section 2.3). To simplify the representation, only one candidate image $I_{cn}$ and four resulting sequences are displayed in this flowchart. The encoder module learns these sequences along with their corresponding defined labels using the hybrid loss (Section 2.4).

and impaired movement. Patients usually feel pain around the knee joint, which becomes worse over time and can even persist at rest. KOA is usually caused by a number of factors including genetics, age, obesity, joint damage, and lifestyle [2]. Current treatments for KOA include medication, physiotherapy, and surgery [3]. Medication can relieve symptoms by reducing pain and inflammation, while physiotherapy can strengthen muscles and reduce stress through exercise, massage, and physical therapy. In severe cases, surgical treatment is necessary, such as joint replacement surgery [4].

Radiographic classification of KOA is an important step in KOA research including epidemiological cohort studies and clinical trial recruitment. The Kellgren-Lawrence (KL) grading system [5] was proposed in 1957 and has since remained the reference standard for radiographic assessment of KOA when using plain radiographs. As shown in Table 1, KOA severity can be defined into five grades depending on the existence and degree of symptoms. However, the diagnosis of KOA relies completely on the perception and judgement of each medical professional, with practitioners potentially differing in their assessments for the same knee X-ray [6].

<table><caption>Table 1: Description of the KL grading system</caption><thead><tr><td>Grade</td><td>Severity</td><td>Description</td></tr></thead><tbody><tr><td>KL-0</td><td>none</td><td>no signs of osteoarthritis</td></tr><tr><td>KL-1</td><td>doubtful</td><td>potential osteophytic lipping</td></tr><tr><td>KL-2</td><td>minimal</td><td>certain osteophytes and potential JSN</td></tr><tr><td>KL-3</td><td>moderate</td><td>moderate multiple osteophytes, certain JSN, and some sclerosis</td></tr><tr><td>KL-4</td><td>severe</td><td>large osteophytes, certain JSN, and severe sclerosis</td></tr></tbody></table>

Computer hardware advancements have enabled deep learning to play a greater role in Computer Vision (CV). Convolutional Neural Networks (CNNs) [7] have been able to effectively execute tasks such as detection [8], segmentation [9] and classification [10]. In recent years, with the successful application of Transformers in Natural Language Processing (NLP), the Transformer-based model, namely Vision Transformer (ViT), has also shown the ability to compete with CNNs in CV.
Several CNN-based and ViT-based learning models have been proposed for KOA diagnosis in the literature. In [11], Nasser et al. proposed the Discriminative Shape-Texture Convolutional Neural Network (DST-CNN), which incorporates a discriminative loss to enhance class separability and address the challenge of high inter-class similarities to achieve automatic detection of KOA. In [12], Tiuplin et al. cropped two patches from the lateral and medial parts of the knee joint X-ray images as the input pair of the Siamese network to classify KOA grades. In [13], Alshareef et al. used the ViT-L32 model to score KOA with an accuracy of 1.48% higher than that of the classical CNN.

In the realm of deep learning for CV, particularly in the medical domain, the scarcity of high-quality medical datasets produces a significant challenge for training deep learning models due to the costly labelling process [14]. To tackle this problem, deep neural networks rely heavily on data augmentation techniques to enhance the model's generalization capability and prevent over-fitting [15]. However, conventional augmentation methods, such as rotation and gamma correction, do not fundamentally change the data structure of the samples, resulting in limited sample diversity. To address this issue, we drew inspiration from [16], [17], and [18] to introduce the concept of data augmentation by applying the proposed novel training strategy. Notably, KOA is definitely present at KL-2, although of minimal severity. Patients at advanced stages of KOA (KL-3 and KL-4) usually have recourse to total knee replacement [19]. Hence, early detection of KOA is clinically more worthwhile. In addition, since the label data for KL-1 patients are often regarded as doubtful in the literature, they may contain high uncertainties that could lead to unstable model training and inaccurate prediction results. Therefore, in this study, we focused solely on KL-0 and KL-2 as the early detection of KOA. To do this, we introduce an approach combining: (i) the ViT-based learning model [20], (ii) a key-patch exchange strategy and (iii) a hybrid loss function for the early detection of KOA (KL-0 vs KL-2). More specifically, we use ViT-base-patch16-224-in21k as a pre-trained model. Then, applying our proposed Selective Shuffled Position Embedding (SSPE), we force the model to focus only on patches associated with characteristics of KL grade, namely key patches (framed in red in Fig. 1). Moreover, we exchange the key patches from candidate images to conduct the new input sequences as a data augmentation technique. Finally, a hybrid loss function consisting of Label Smoothing Cross-Entropy (LSCE) and Cross-Entropy (CE) losses is computed with optimized weights.

The main contributions of this paper are as follows:

* A novel ViT-based learning model with well-designed position embedding is proposed.

* A key-patch exchange strategy is introduced to force the model to focus on learning the features of key patches.

* The experimental data were obtained from the OsteoArthritis Initiative (OAI) database [21].

# 2 Proposed Method

The flowchart of the proposed approach is presented in Fig. 1. Before describing our method, we briefly present the structure of the classical Transformer network, which was first introduced in a seminal paper by Vaswani et al. [22].

## 2.1 Classical ViT model

Transformer networks are particularly well-suited for Natural Language Processing (NLP) tasks, such as language translation [23] and text classification [24]. Unlike traditional neural networks, which process input data sequentially and are therefore prone to losing information about long-term dependencies, Transformers are designed to be capable of parallel processing, making them more efficient and capable of modelling long-range dependencies [25]. The key innovation behind Transformers is the self-attention mechanism, which enables the network to selectively attend to different parts of the input sequence in order to capture the most relevant information for a given task. This mechanism has proven to be particularly effective in NLP [26], where the relationship between different words in a sentence can be complex and nuanced.
<figure><img src="image_4.png" /><figcaption>Figure 2: The structure of the classical ViT network.</figcaption></figure>

ViT is a neural network architecture introduced in [20] in 2020. It represents a breakthrough in CV, showing that Transformers, designed initially for NLP, could also be applied successfully to image-based tasks. As shown in Fig. 2, the key idea behind ViT is to treat an image as a sequence of patches rather than as a grid of pixels and to apply a Transformer network to these patches. The resulting network can then be trained using standard backpropagation techniques on a large-scale image dataset. One of the advantages of ViT over traditional CNNs is that it can capture global spatial relationships in an image [27], which is especially important for tasks such as object recognition and image classification. In addition, ViT is highly efficient since it can process an entire image in parallel, unlike CNNs, which require sequential processing. ViT has achieved State-Of-The-Art (SOTA) performance on a range of CV tasks, including image classification [28], object detection [29], and image segmentation [30]. As a result, ViT has quickly become one of the most popular architectures in CV research.

Usually, ViT uses learnable absolute positional embedding to encode different positions. For each input patch, a position encoding vector is assigned with the same dimension as the embedding vector of the Transformer. During input, the position embedding vector is added to the input vector to preserve the relative positional information between different positions. For each position $i$ and each position encoding dimension $j$, the position embedding, $PE_{i,j}$ is calculated by adding a sine encoding and a cosine encoding:

$$ PE_{i,j} = \begin{cases} \sin(\frac{i}{10000^{2j/d}}), & \text{if } j \text{ is even} \\ \cos(\frac{i}{10000^{2(j-1)/d}}), & \text{if } j \text{ is odd} \end{cases} \qquad (1) $$

where $d$ is the dimension of the position embedding, which is typically set to an even number.

<figure><img src="image_5.png" /><figcaption>Figure 3: The structure of the encoder module.</figcaption></figure>

The structure of the encoder module of the ViT model is briefly presented in Fig. 3, which consists of a stack of 12 identical Transformer blocks. As shown, each Transformer block is composed of two sub-layers, namely, the multi-head self-attention layer and the Multi-Layer Perceptron (MLP). The multi-head self-attention layer is responsible for capturing the relationships between different parts of the image. It consists of multiple parallel attention heads, each of which learns to attend to different parts of the image. The outputs of these attention heads are then concatenated and linearly projected to produce the final output of the self-attention layer. MLP, also named the feedforward neural network layer, is responsible for transforming the features learned by the self-attention layer into a form that can be easily used by subsequent layers. It consists of two linear transformations followed by a non-linear activation function.
## 2.2 Selective Shuffled Position Embedding

<figure><img src="image_6.png" /><figcaption>Figure 4: Classical position embedding (row 1), SSPE strategy and highlighted key patches (row 2), and the final position embedding of this sequence as input of ViT (row 3). Key patches #4 and #6 remain unchanged during the process.</figcaption></figure>

Considering that the symptoms of KOA (KL-2) (i.e., osteophytes and JSN) are only manifested in specific areas (patches) of the knee and inspired by our previous work [16] and [18], contrarily to classical position embedding, we propose a novel position embedding strategy, namely Selective Shuffled Position Embedding (SSPE). Specifically, we first fix the position embedding of the key patches (i.e., patches #4 and #6), then shuffle the position embedding of the remaining patches. In the following, non-key patches are used to design these remaining patches. As shown in Fig. 4, an input sequence composed of the fixed key patches (i.e., #4 and #6 red boxes in Fig. 4) and the shuffled non-key ones were built. In such a way, SSPE can force the ViT model to focus only on the key patches concerned by OA. This procedure was applied at each epoch during the training step.

## 2.3 Key-patch exchange strategy

As explained previously, our proposed SSPE strategy makes the model focus on the key patches. In this section, we introduce an additional step for data augmentation. Inspired by our work [18], we propose to exchange key patches of images having different KL grades. More specifically, for any target image $I_t$ from the training set $\mathcal{T}$, we further randomly select $N$ candidate images, $I_{c1}, I_{c2}, ..., I_{cn}, \forall cn \neq t$, from the same set and exchange their key patches, resulting in four different input sequences during each exchange operation. It is worth noting that key patches can only be exchanged according to the specific positions (#4 $\Leftrightarrow$ #4; #6 $\Leftrightarrow$ #6), while the non-key patches remain the same for the target image $I_t$. The match number, $N$, will be discussed in Section 4.2.2. The labels $l$ of the resulting input sequences are defined as follows:

$$l = \begin{cases} \text{KL-0}, & \text{if } \forall i \in \{1, 2\}, l_i = \text{KL-0} \\ \text{KL-2}, & \text{otherwise} \end{cases} \qquad (2)$$

where $l_i$ represents the label of each key patch.
<figure><img src="image_7.png" /><figcaption>Figure 5: Proposed key-patch exchange strategy. For convenience, only the image $I_{c1}$ among N candidates is shown during the key-patch exchange operation along with the target image $I_t$. As shown, blue and red patches are the key ones of the candidate and target images, respectively. After the key-patch exchange operation, four different sequences are obtained by each candidate image. The label for each obtained sequence is defined in the following Section 2.4.</figcaption></figure>

## 2.4 Hybrid loss strategy

To set the loss function, two types of sequences are defined after applying the key-patch exchange strategy: the sequences composed of key patches of the same KL-grade, so-called full-KL, $\mathcal{F}$, and those composed of the two KL-grades considered (KL-0 and KL-2), so-called mixed-KL, $\mathcal{M}$. In order to consider such diversity, we employed a hybrid loss strategy by using the Label Smoothing Cross-Entropy (LSCE) for $\mathcal{M}$ and the Cross-Entropy (CE) for $\mathcal{F}$.

Firstly, the LSCE loss function was used for the set $\mathcal{M}$ to avoid the learning model being overconfident. To this end, we used label smoothing [31], which is a regularization technique that consists in perturbing the target variable to make the learning model less certain of its prediction.

Label smoothing replaces the one-hot encoded label vector $y_s^{hot}$ by a mixture of $y_s^{hot}$, named $y_s^{LS}$:

$$y_s^{LS} = y_s^{hot}(1 - \varepsilon) + \frac{\varepsilon}{2}, \quad y_s^{hot} \in \{0, 1\}, \varepsilon \in (0, 1) \qquad (3)$$

$$y_s^{hot} = \begin{cases} 1 & \hat{Y} = T \\ 0 & \hat{Y} \neq T \end{cases} \qquad (4)$$

where $y_s^{hot}$ is the one-hot encoded ground-truth label of the sample $s$, and $\varepsilon$ is a hyper-parameter that determines the amount of smoothing.

The LSCE loss, $J_{LSCE}$ is thus computed as follows:

$$\begin{aligned} J_{LSCE} &= \sum_{s \in \mathcal{M}} -y_s^{LS} \log(p_s) \\ &= \sum_{s \in \mathcal{M}} -(y_s^{hot}(1 - \varepsilon) + \frac{\varepsilon}{2})\log(p_s) \end{aligned} \qquad (5)$$

$$p_s = P_{\hat{Y}|T}(\hat{Y} = k | T = k), \quad \forall k \in \mathcal{K}, \forall s \in \mathcal{D} \qquad (6)$$
where $\hat{Y}$ and $T$ are the predicted and the real labels, respectively, $k$ represents the KL grade of the sample $s$, $P_{\hat{Y}|T}$ is the conditional probability distribution, $\mathcal{K}$ is a set of KL grades (KL-0 and KL-2), and $\mathcal{D}$ is the overall dataset.

Secondly, for input sequences of full-KL (set $\mathcal{F}$), the classical CE loss, $J_{CE}$ was used and computed as follows:

$$J_{CE} = \sum_{s \in \mathcal{F}_k} -y_s^{\text{hot}} \log(p_s), \quad \forall k \in \mathcal{K}, \forall s \in \mathcal{F} \qquad (7)$$

Finally, the proposed hybrid loss function was defined as:

$$J_{\text{hybrid}} = \alpha J_{\text{LSCE}} + \beta J_{CE} \qquad (8)$$

where the hyper-parameters $\alpha$ and $\beta$ were used to weigh and better balance the loss functions, which is discussed in Section 4.2.1.

# 3 Experiments

## 3.1 Public knee database

We used knee X-rays from the OsteoArthritis Initiative (OAI) [21]. The OAI is a longitudinal study of 4796 individuals aged between 45 to 79 years of age followed over 96 months, with each participant having nine follow-up examinations. The OAI study aims to observe participants who already suffered from KOA or from an elevated risk of developing it. All the collected data are publicly available to accelerate research in the field of KOA.

## 3.2 Data preprocessing

<figure><img src="image_8.png" /><figcaption>Figure 6: A knee radiograph from the OAI database with the extracted knee joint in the red box (a). Obtained knee joint (b).</figcaption></figure>

As shown in Fig. 6, as in [32], knee joints were detected using the learning model, YOLOv2 [33] and served as inputs of the proposed model. As a result of the preprocessing steps, 3,185 KL-0 and 2,126 KL-2 images were collected. According to each KL grade, the dataset was randomly divided into training, validation, and test sets with a ratio of 7:1:2, respectively.

## 3.3 Experimental details

We used the pre-trained weights of the model ViT-base-patch16-224-in21k. Adam was employed for 100 epochs with a learning rate of 5e-06 and a batch size of 128. Common data augmentation techniques, such as random rotation, brightness, contrast, etc., were executed randomly during the training. To deal with the imbalanced dataset of the study, bootstrapping-based oversampling [34] was applied. We implemented our approach using PyTorch v1.8.1 [35] on Nvidia TESLA A100 graphic cards with 80 GB memory.
# 4 Results and discussion

In this section, experimental results are discussed. KL-2 was treated as the positive class to compute the F1-score.

## 4.1 Selection of position embedding and key patches

To assess the contribution of the proposed SSPE to performance more independently and intuitively, in this section, we do not consider the key-patch exchange strategy. Therefore, two ablation studies for the comparison of different embedding methods and the selection of key patches were carried out and are reported in Table 2. More specifically, for different embedding methods:

* 1-D position embedding: The classical position embedding is computed as Eq.1 (Section 2.1).

* 2-D position embedding: The coordinates of each pixel in an image are represented as a vector that encodes both the row and column positions of the pixel.

* Relative position embedding: It is calculated by encoding the difference vectors between patch coordinates. For each pair of patches $(i, j)$, the Relative Position Embedding $RPE(i, j)$ is computed as the positional encoding for the difference vector between the coordinates of patch i and patch j [20].

<table><caption>Table 2: Comparison of position embedding and selection of key patches</caption><thead><tr><th>Position Embedding</th><th>Acc (%)</th><th>Acc (%)</th><th>Diff (%)</th><th>Acc (%)</th><th>Diff (%)</th><th>Acc (%)</th><th>Diff (%)</th></tr></thead><tbody><tr><td>All patches</td><td></td><td></td><td></td><td>Patches #123</td><td>Patches #456</td><td>Patches #789</td><td></td></tr><tr><td>No Pos. Emb.</td><td>85.62</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>1-D Pos. Emb.</td><td><b>88.44</b></td><td>85.51</td><td>2.93 ↓</td><td><b>88.76</b></td><td>0.32 ↑</td><td>86.12</td><td>2.32 ↓</td></tr><tr><td>2-D Pos. Emb.</td><td>87.81</td><td>85.98</td><td>1.83 ↓</td><td>88.29</td><td><b>0.48</b> ↑</td><td>86.57</td><td>1.24 ↓</td></tr><tr><td>Rel. Pos. Emb.</td><td>87.69</td><td>86.15</td><td>1.54 ↓</td><td>87.99</td><td>0.30 ↑</td><td>86.02</td><td>1.67 ↓</td></tr></tbody></table>

with SSPE

Without losing any generality and as too many combinations exist, we selected three groups of patches among #123, #456, and #789 to determine the potential key patches. As shown in Table 2, compared to no position embedding, all evaluated position embedding methods perform better. Of those, the 1-D position embedding achieves the best performance (88.44%). Moreover, the application of our SSPE significantly improves the performance only for patches #456, from which it can be inferred that the key patches should be located within these areas, as the proposed SSPE forces the model to focus on learning the features of the key patches and increases the diversity of the input sequences to some extent changing the data structure. These results were expected as OA symptoms might be more visible on patches #456. On the contrary, shuffling the key patches is, to some extent, equivalent to the no-position embedding method, preventing the model from effectively attending to the crucial areas.

In order to apply the proposed key-patch exchange strategy more conveniently, we further conducted a masking experiment to select two out of three potential key patches (#456) as the final key patches. For each patch pair #45, #46, and #56, the obtained accuracies were 88.69%, 88.86%, and 88.72%, respectively. Therefore, the patches #46 were retained as the final key patches for the following key-patch exchange strategy.

## 4.2 Settings of the proposed key-patch exchange strategy

### 4.2.1 Selection of the hyper-parameters

To show the impact of the hyper-parameters ($\epsilon$, $\alpha$, and $\beta$) of the proposed hybrid loss function for the key-patch exchange strategy, different configurations were tested with $N = 1$. We first determined the level of smoothing $\epsilon \in [0.05, 0.3]$. Then, we evaluated the weight hyper-parameters through a small grid search with $\alpha, \beta \in [0.1, 1]$ ensuring that $\alpha + \beta = 1$. For greater convenience, we show only the performance obtained for each $\epsilon$ with the most optimized weight parameters $\alpha$ and $\beta$. From Table 3, several conclusions can be drawn. Firstly, using the key-patch exchange strategy generally improves the accuracy of the model. Secondly, while using the hybrid loss strategy during training may not constantly improve accuracy, it can be effective depending on the values of the hyper-parameters $\epsilon$, $\alpha$, and $\beta$. The highest accuracy of 89.80% was achieved with $\epsilon = 0.2$, $\alpha = 0.3$, and $\beta = 0.7$, indicating the importance of carefully selecting the hyper-parameters in achieving optimal performance.
<table><caption>Table 3: Contribution of different combinations of hyper-parameters</caption><thead><tr><th></th><th>ε</th><th>α</th><th>β</th><th>Acc (%)</th><th>Diff<sub>1</sub><sup>1</sup> (%)</th><th>Diff<sub>2</sub><sup>2</sup> (%)</th></tr></thead><tbody><tr><td rowspan="4">with key-patch exchange</td><td colspan="6">without hybrid loss<sup>3</sup></td></tr><tr><td>-</td><td>0</td><td>1</td><td>89.32</td><td>0.46 ↑</td><td>-</td></tr><tr><td colspan="6">with hybrid loss</td></tr><tr><td>0.05</td><td>0.2</td><td>0.8</td><td>89.11</td><td>0.25 ↑</td><td>0.21 ↓</td></tr><tr><td></td><td>0.10</td><td>0.4</td><td>0.6</td><td>89.29</td><td>0.43 ↑</td><td>0.03 ↓</td></tr><tr><td></td><td>0.15</td><td>0.3</td><td>0.7</td><td>89.57</td><td>0.71 ↑</td><td>0.25 ↑</td></tr><tr><td></td><td>0.20</td><td>0.3</td><td>0.7</td><td><b>89.80</b></td><td><b>0.94</b> ↑</td><td><b>0.48</b> ↑</td></tr><tr><td></td><td>0.25</td><td>0.4</td><td>0.6</td><td>88.96</td><td>0.10 ↑</td><td>0.36 ↓</td></tr><tr><td></td><td>0.30</td><td>0.5</td><td>0.5</td><td>88.91</td><td>0.05 ↑</td><td>0.41 ↓</td></tr></tbody></table>

### 4.2.2 Effects of the match number

In this subsection, we evaluate the effects of the match number, $N \in [0, 4]$, on the model's performance in terms of convergence and accuracy. Fig. 7(a) and Fig. 7(b) demonstrate that with an increasing value of $N$, the model's convergence accelerates and leads to higher accuracy, which is attributed to the greater diversity of input sequences, enabling the model to learn discriminative features more efficiently. Nevertheless, it is noteworthy that the benefits of further increasing $N$ diminish after $N = 2$. Considering the trade-off between effect and computational efficiency, we ultimately opted for $N = 2$ as the optimal match number for the key-patch exchange strategy.

<figure><img src="image_9.png" /><figcaption>Figure 7: Obtained convergence (a) and performance (b) curves using various match numbers, $N$, for 100 epochs.</figcaption></figure>

## 4.3 Comparison to different learning models

In this section, our proposed approach is compared to common deep learning-based models and SOTA approaches in terms of accuracy, F1-score and the number of parameters. Results are shown in Table 4. As can be seen, our proposed approach performs better than the evaluated models in terms of both metrics, which shows that the proposed global approach (SSPE and key-patch exchange) can effectively improve the model's performance, as the former makes the model focus on learning the features of the key patches and the latter performs data augmentation. However, the shortcoming is the relatively high number of parameters in our model, especially compared to the works of [12] and [16], which will be discussed in Section 4.6.

<footnote n="1">¹ Diff₁: The accuracy difference compared to the model using SSPE without key-patch exchange strategy (88.86%).</footnote>

<footnote n="2">² Diff₂: The accuracy difference compared to the model using SSPE without the hybrid loss (89.32%).</footnote>

<footnote n="3">³ Only the classical CE loss was used.</footnote>
<table><caption>Table 4: Comparison of the performance of different models</caption><thead><tr><th>Models</th><th>Accuracy (%)</th><th>F1 (%)</th><th>Params (M)<sup>1</sup></th></tr></thead><tbody><tr><td>Densenet-121</td><td>80.84</td><td>77.03</td><td>6.95</td></tr><tr><td>Densenet-161</td><td>79.06</td><td>75.09</td><td>26.47</td></tr><tr><td>Densenet-169</td><td>81.81</td><td>78.23</td><td>12.48</td></tr><tr><td>Densenet-201</td><td>84.43</td><td>81.29</td><td>18.09</td></tr><tr><td>Resnet-18</td><td>83.59</td><td>80.30</td><td>11.17</td></tr><tr><td>Resnet-34</td><td>80.14</td><td>76.35</td><td>21.28</td></tr><tr><td>Resnet-50</td><td>81.78</td><td>78.29</td><td>23.51</td></tr><tr><td>Resnet-101</td><td>77.60</td><td>72.98</td><td>42.50</td></tr><tr><td>Resnet-152</td><td>76.47</td><td>72.13</td><td>58.14</td></tr><tr><td>VGG-11</td><td>80.97</td><td>77.30</td><td>195.89</td></tr><tr><td>VGG-13</td><td>80.71</td><td>77.88</td><td>196.07</td></tr><tr><td>VGG-16</td><td>73.17</td><td>68.50</td><td>201.38</td></tr><tr><td>VGG-19</td><td>71.98</td><td>68.84</td><td>206.70</td></tr><tr><td>Tiuplin et al. [12]</td><td>87.33</td><td>84.82</td><td><b>0.15</b></td></tr><tr><td>Wang et al. [16]</td><td>88.38</td><td>85.93</td><td>2.71</td></tr><tr><td>Our model</td><td><b>89.80</b></td><td><b>87.66</b></td><td>85.95</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>Figure 8: Comparison of attention maps obtained from the last layer using different learning models.</figcaption></figure>

<footnote n="1">¹ Params is the number of parameters used by the model</footnote>
To visualise the regions that contributed to the decision of each model, the Grad-Cam technique [36] was used. The attention maps obtained are presented in Fig. 8. As can be noticed, all evaluated models show a certain sensitivity to regions that involve early KOA (KL-2) characteristics (osteophytes and JSN). However, DenseNet, ResNet, and VGG models also exhibit a reaction to background noise, which may negatively impact the classification performance. To a lesser extent, the Siamese-based models [12] and [16] exhibit more reaction to areas affected by OA. Conversely, our proposed approach focuses more on regions affected by OA, which demonstrates that with a well-designed position embedding layer, it is possible to make the model concentrate on specific areas concerned by OA. It is noteworthy that during the data preprocessing of Siamese-based networks ([12] and [16]), it is often necessary to crop key patches as an input pair to the network. While this significantly reduces the parameter size of the model, it results in a loss of information from the input image. Especially for medical images, the integrity of global image information is vital and can enhance the model's decision confidence in clinical applications. In comparison to those methods, our approach simply encourages the model to pay more attention to the key patches, but it also considers the texture information of other patches to some extent. Additionally, thanks to the ordered position embedding of key patches, our model handles their sequential information better. As can be seen in Fig. 8(d) and Fig. 8(e), compared to [12] and [16], our model is able to draw attention to locations of possible osteophytes in the joint space, which is more consistent with medical opinion.

## 4.4 Contribution of the hybrid loss

As presented in Section 2.4, our proposed hybrid loss combines the CE and the LSCE losses, which were applied to the full-KL, $\mathcal{F}$, and the mixed-KL, $\mathcal{M}$ sequences, respectively. In this section, using t-SNE scatter plots, we evaluate the contribution of the proposed hybrid loss using the optimised hyper-parameters discussed in Section 4.2.1. As shown in Fig. 9, compared to the CE loss, our proposed hybrid loss strategy ensures inter-class distance (i.e., centre points of two classes) and makes the distribution of samples of each class more continuous by appropriately increasing intra-class distance, which is consistent with the objective of the continuity of KL grades, which to some extent alleviates the problem of overconfidence in model training caused by the semi-quantitative nature of the KL grading system.

<figure><img src="image_11.png" /><figcaption>Figure 9: t-SNE scatter plots obtained using the CE loss (a) and the proposed hybrid loss (b).</figcaption></figure>

## 4.5 Analysis of the applicability

In this section, we analyze the applicability of our proposed global approach to the standard pre-trained ViT models. Table 5 presents the performance achieved using all proposed strategies (SSPE, key-patch exchange, and hybrid loss). For a fair comparison, the hyper-parameters that obtained the best performance in the hybrid loss strategy were used. As can be seen, our proposed global approach consistently improves the accuracy across all evaluated ViT models, indicating its broad applicability and potential interest in improving ViT models' performance.

## 4.6 Discussion

In this paper, we have introduced a novel approach for the early detection of KOA (KL-0 vs KL-2) based on the ViT network. Firstly, we fixed and shuffled the position encoding of key patches and non-key ones in the input sequences, respectively, to force the model to focus on learning the characteristics of the key patches. Then, we randomly selected other $N$ images as inputs of the model to exchange their key patches and thus obtain different input sequences,
<table><caption>Table 5: Applicability of the proposed global approach</caption><thead><tr><th>Pre-trained model</th><th>Acc (%)</th><th>Acc*<fnref n="1" /> (%)</th><th>Diff (%)</th><th>Param (M)</th></tr></thead><tbody><tr><td>ViT-base-patch16-224</td><td>88.31</td><td>89.11</td><td>0.80 ↑</td><td><strong>85.95</strong></td></tr><tr><td>ViT-base-patch16-224-in21k<fnref n="2" /></td><td>88.44</td><td>89.80</td><td><strong>1.36 ↑</strong></td><td><strong>85.95</strong></td></tr><tr><td>ViT-base-patch32-224</td><td>88.39</td><td>89.29</td><td>0.90 ↑</td><td>87.50</td></tr><tr><td>ViT-base-patch32-224-in21k</td><td>88.52</td><td>89.69</td><td>1.17 ↑</td><td>87.50</td></tr><tr><td>ViT-large-patch16-224</td><td>88.52</td><td>89.71</td><td>1.19 ↑</td><td>303.51</td></tr><tr><td>ViT-large-patch16-224-in21k</td><td>88.61</td><td><strong>89.84</strong></td><td>1.23 ↑</td><td>303.51</td></tr><tr><td>ViT-large-patch32-224-in21k</td><td><strong>88.73</strong></td><td>89.82</td><td>1.09 ↑</td><td>305.56</td></tr></tbody></table>

which leads to data augmentation. Finally, an adequate hybrid loss strategy was proposed, combining the CE and the LSCE losses. These two loss functions were optimised by adjusting the weights $\alpha$ and $\beta$. Our experimental findings demonstrated the validity of our proposed global approach, as it significantly improves the model's classification performance.

### 4.6.1 Details of the position embedding

As presented in Section 2.1, we introduced the SSPE to diversify the input sequences to make the model focus only on the key patches. Here, we also evaluated several random dropouts of 20%, 30%, and 50% for the position embedding of non-key patches. However, no improvement was observed, which is probably caused by the fact that the diversity of the input sequences decreases when the positional information of non-key patches is dropped randomly.

### 4.6.2 Strengths and limitations

This study has several notable strengths. From a clinical perspective, focusing on early KOA diagnosis (KL-0 vs KL-2) is particularly relevant as it enables timely physical interventions that can potentially delay the onset and progression of OA symptoms. To ensure the clinical applicability of our model, we undertook several steps: First, we fixed the position embedding of key patches, which is consistent with the attention areas that radiologists consider when diagnosing KOA. Secondly, during our key-patch exchange strategy, all obtained sequences comprised patches from real and valid data. Finally, common data augmentation techniques such as rotation, gamma correction, and jitter were also used to further enhance the stability and robustness of the model. We believe that such a more stable and explainable model in a Computer-Aided Diagnosis (CAD) system can make deep learning approaches gain more trust and acceptance from medical practitioners in daily clinical practice. There were also several limitations in our study. All the experiments were achieved using only the OAI database. Other large databases should be considered to evaluate and strengthen the proposed approach. Therefore, the use of other large datasets, such as the Multicenter Osteoarthritis Study (MOST), and continued optimisation of the baseline model for specific tasks could be of interest for future work. Moreover, as presented in Table 4, although our approach's accuracy is higher than that of Siamese-based models [12] and [16], the high computational cost is still a major impediment to clinical applications, given its high reliance on high-end hardware.

## 5 Acknowledgements

The authors would like to express their gratitude to the French National Research Agency (ANR) for supporting their work through the ANR-20-CE45-0013-01 project.

This manuscript was prepared using OAI data and does not necessarily reflect the opinions or views of the OAI investigators, the NIH, or the private funding partners. The authors would like to thank studies participants and clinical staff as well as the coordinating centre at UCSF.

## References

[1] Richard F Loeser, Steven R Goldring, Carla R Scanzello, and Mary B Goldring. Osteoarthritis: a disease of the joint as an organ. *Arthritis and rheumatism*, 64(6):1697, 2012.

<footnote n="1">¹ Acc*: Accuracy obtained using our proposed global approach.</footnote>

<footnote n="2">² The pre-trained model used in this study.</footnote>
[2] Anna Litwic, Mark H. Edwards, Elaine M. Dennison, and Cyrus Cooper. Epidemiology and burden of osteoarthritis. *British Medical Bulletin*, 105(1):185-199, 01 2013.

[3] Wei Boon Lim and Oday Al-Dadah. Conservative treatment of knee osteoarthritis: A review of the literature. *World Journal of Orthopedics*, 13(3):212, 2022.

[4] Jean-Pierre Raynauld, Johanne Martel-Pelletier, Marc Dorais, Boulos Haraoui, Denis Choquette, François Abram, André Beaulieu, Louis Bessette, Frédéric Morin, Lukas M Wildi, et al. Total knee replacement as a knee osteoarthritis outcome: predictors derived from a 4-year long-term observation following a randomized clinical trial using chondroitin sulfate. *Cartilage*, 4(3):219-226, 2013.

[5] J. H. Kellgren and J. S. Lawrence. Radiological Assessment of Osteo-Arthrosis. *Annals of the Rheumatic Diseases*, 16(4):494, 1957.

[6] Lior Shamir, Shari M Ling, William W Scott, Angelo Bos, Nikita Orlov, Tomasz J Macura, D Mark Eckley, Luigi Ferrucci, and Ilya G Goldberg. Knee x-ray image analysis method for automated detection of osteoarthritis. *IEEE Transactions on Biomedical Engineering*, 56(2):407-415, 2008.

[7] A. Krizhevsky, I. Sutskever, and G. E Hinton. Imagenet classification with deep convolutional neural networks. *Advances in neural information processing systems*, 25:1097-1105, 2012.

[8] Lucas C Ribas, Rabia Riad, Rachid Jennane, and Odemir M Bruno. A complex network based approach for knee osteoarthritis detection: Data from the osteoarthritis initiative. *Biomedical Signal Processing and Control*, 71:103133, 2022.

[9] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Drinet for medical image segmentation. *IEEE Transactions on Medical Imaging*, 37(11):2453-2462, 2018.

[10] Yassine Nasser, Rachid Jennane, Aladine Chetouani, Eric Lespessailles, and Mohammed El Hassouni. Discriminative regularized auto-encoder for early detection of knee osteoarthritis: data from the osteoarthritis initiative. *IEEE transactions on medical imaging*, 39(9):2976-2984, 2020.

[11] Yassine Nasser, Mohammed El Hassouni, Didier Hans, and Rachid Jennane. A discriminative shape-texture convolutional neural network for early diagnosis of knee osteoarthritis from x-ray images. *Physical and Engineering Sciences in Medicine*, pages 1-11, 2023.

[12] A. Tiulpin, J. Thevenot, E. Rahtu, P. Lehenkari, and S. Saarakkala. Automatic knee osteoarthritis diagnosis from plain radiographs: A deep learning-based approach. *Scientific Reports*, 2018.

[13] Esam Alsadiq Alshareef, Fawzi Omar Ebrahim, Yosra Lamami, Mohamed Burid Milad, Mohamed SA Eswani, Sedigh Abdalla Bashir, Salah AM Bshina, Anas Jakdoum, Asharaf Abourqeeqah, Mohamed O Elbasir, et al. Knee osteoarthritis severity grading using vision transformer. *Journal of Intelligent & Fuzzy Systems*, (Preprint):1-11, 2022.

[14] Julia Röglin, Katharina Ziegeler, Jana Kube, Franziska König, Kay-Geert Hermann, and Steffen Ortmann. Improving classification results on a small medical dataset using a gan; an outlook for dealing with rare disease datasets. *Frontiers in Computer Science*, page 102, 2022.

[15] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. *Journal of big data*, 6(1):1-48, 2019.

[16] Zhe Wang, Aladine Chetouani, Didier Hans, Eric Lespessailles, and Rachid Jennane. Siamese-gap network for early detection of knee osteoarthritis. In *2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)*, pages 1-4, 2022.

[17] Zhe Wang, Aladine Chetouani, and Rachid Jennane. A confident labelling strategy based on deep learning for improving early detection of knee osteoarthritis. *arXiv preprint arXiv:2303.13203*, 2023.

[18] Zhe Wang, Aladine Chetouani, and Rachid Jennane. Key-exchange convolutional auto-encoder for data augmentation in early knee osteoarthritis classification. *arXiv preprint arXiv:2302.13336*, 2023.

[19] J. A. D. van der Woude, S. C. Nair, R. J. H. Custers, J. M. van Laar, N. O. Kuchuck, F. P. J. G. Lafeber, and P. M. J. Welsing. Knee joint distraction compared to total knee arthroplasty for treatment of end stage osteoarthritis: Simulating long-term outcomes and cost-effectiveness. *PLOS ONE*, 11(5):1-13, 05 2016.

[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929*, 2020.

[21] G. Lester. The Osteoarthritis Initiative: A NIH Public-Private Partnership. *HSS Journal: The Musculoskeletal Journal of Hospital for Special Surgery*, 8(1):62-63, 2011.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *Advances in neural information processing systems*, 30, 2017.

[23] Mattia A Di Gangi, Matteo Negri, and Marco Turchi. Adapting transformer to end-to-end spoken language translation. In *Proceedings of INTERSPEECH 2019*, pages 1133–1137. International Speech Communication Association (ISCA), 2019.

[24] Zein Shaheen, Gerhard Wohlgenannt, and Erwin Filtz. Large scale legal text classification using transformer models. *arXiv preprint arXiv:2010.12871*, 2020.

[25] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. *ACM computing surveys (CSUR)*, 54(10s):1–41, 2022.

[26] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In *Proceedings of the AAAI conference on artificial intelligence*, volume 33, pages 3159–3166, 2019.

[27] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. *IEEE transactions on pattern analysis and machine intelligence*, 45(1):87–110, 2022.

[28] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 357–366, 2021.

[29] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16*, pages 213–229. Springer, 2020.

[30] Xiaohong Huang, Zhifang Deng, Dandan Li, and Xueguang Yuan. Missformer: An effective medical image segmentation transformer. *arXiv preprint arXiv:2109.07162*, 2021.

[31] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? *Advances in neural information processing systems*, 32, 2019.

[32] P. Chen, L. Gao, X. Shi, K. Allen, and L. Yang. Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss. *Computerized Medical Imaging and Graphics*, 75:84–92, 2019.

[33] J. Redmon and A. Farhadi. Yolo9000: better, faster, stronger. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 7263–7271, 2017.

[34] N. V Chawla, K. W Bowyer, L. O Hall, and W P. Kegelmeyer. Smote: synthetic minority over-sampling technique. *Journal of artificial intelligence research*, 16:321–357, 2002.

[35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.

[36] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In *2017 IEEE International Conference on Computer Vision (ICCV)*, pages 618–626, 2017.