# Deep metric learning for end-to-end document classification

Tri-Cong Pham, Mickaël Coustaty, Antoine Doucet, Aurélie Joseph, Vincent Poulain D'andecy

► To cite this version:

Tri-Cong Pham, Mickaël Coustaty, Antoine Doucet, Aurélie Joseph, Vincent Poulain D'andecy. Deep metric learning for end-to-end document classification. Neurocomputing, 2025, 653, 10.1016/j.neucom.2025.131241. hal-05498972

HAL Id: hal-05498972

https://hal.science/hal-05498972v1

Submitted on 7 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# Deep metric learning for end-to-end document classification

Tri-Cong Pham<sup>a, b,*</sup> <sup>●</sup>, Mickaël Coustaty<sup>a</sup> <sup>●</sup>, Antoine Doucet<sup>a</sup> <sup>●</sup>, Aurélie Joseph<sup>b</sup> <sup>●</sup>, Vincent Poulain d'Andecy<sup>b</sup> <sup>●</sup>

## ARTICLE INFO

Communicated by Y. Lai

**Keywords:**

Document classification

Deep metric learning

Novelty detection

Ambiguity rejection

KNN

## ABSTRACT

Document classification systems become more and more complex with the need to deal with new document formats or categories while obtaining a low error rate when classifying more and more documents. Such systems need to have important features including (1) the ability to eliminate ambiguity to improve precision or reduce the error rate, (2) the capability to detect and reject documents belonging to new categories or new variations. Previous studies often focused on closed datasets or solely on the problem of novelty detection, without evaluating the ability to reject ambiguous results after the novelty detection. In this paper, we propose an end-to-end document classification algorithm including both novelty and ambiguity rejection. The proposed algorithm utilizes deep metric learning to compact the knowledge space, and then uses the last hidden layer's features as input for an unsupervised KNN-based method for novelty and ambiguity rejection. Extensive experiments and analysis on private and public benchmark datasets demonstrate the effectiveness of our proposed algorithm. The algorithm provides the capability to handle new documents while effectively rejecting ambiguity to enhance the precision, recall of known categories, and coverage rate of the end-to-end document classification system.

## 1. Introduction

Documents are used every day for supporting different administrative tasks, for instance in the financial and administrative domains. Document analysis and understanding remain a challenge as they involve the need to read and understand the textual content and understand its spatial organization. As reading a document is a complex task, a first step in recent systems consists of classifying documents in order to detect their class/objective and then define the process to be used for the final task. This document classification step requires high accuracy because any error will lead to a waste of time and money, and finally call for a human operator to manually reject the documents or correct the mistake. Document classification faces two main difficulties which are, on the one hand, the large number of document categories (more than 200 document categories are generally used in industrial systems) and on the other hand, the evolution of these document categories as new and unseen formats of documents are generated every day. For example, in financial applications, invoices, purchase orders, or bank statements may have different templates, field locations, and formats depending on the company or software used. Even the format of the same category of

document from the same company may change over time. These difficulties make it very challenging to develop a system with high accuracy that is also able to detect new categories or formats.

Powered by parallel computing capabilities and big data, deep learning networks have achieved significant performance in various fields from NLP [1,2] to Computer Vision (CV) [3], leading to great impact on academic research and commercial applications. The algorithms are also used in document analysis systems [4,5], especially for document classification [6–8]. In the studies conducted so far, classifiers are trained on a dataset consisting of several categories and evaluated on a test dataset containing unseen documents belonging to the same set of categories. However, in reality, there are novel/unseen documents that do not belong to these categories nor have novel formats of existing categories (see Fig. 1). In this case, the classifiers need to categorize them as novel documents or an unknown category. Most novelty detection studies have been in the field of computer vision [9–11]. Although there are studies [12] on novelty detection for document images, the number is still limited.

While many studies have applied deep learning to document classification, the majority of them have concentrated on closed datasets

<footnote><sup>a</sup> L3i, La Rochelle Université, Avenue Michel Crépeau, La Rochelle, 17042, France</footnote>

<footnote><sup>b</sup> Yooz, 1 Rue Fleming, La Rochelle, 17000, France</footnote>

<footnote>* Corresponding author at: L3i, La Rochelle Université, Avenue Michel Crépeau, La Rochelle, 17042, France.
Email address: cong.pham@univ-lr.fr (T.-C. Pham).</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 1. Samples of different document categories in our private administrative dataset which illustrate the document images that are in distribution (the 1st row with blue border) and out of distribution (the 2nd row with red border). From left to right of the first row: Order, Account Due, Invoice, and Bank Statements; and of the second row: Debit Mandate, Debit Statement, Mail Dues, and Insurance Daily Allowance.</figcaption></figure>

of known categories and the few studies that have focused on novelty
detection have not, however, conducted a comprehensive evaluation, in-
cluding assessing the overall ambiguity rejection [13] capabilities of the
classification step when incorporating novelty detection into a complete
pipeline for practical applications. When these modules are combined
into a full pipeline, the fundamental question is whether the holistic al-
gorithm improves precision compared to the scenario where no rejection
module is used? Furthermore, for low baseline accuracy classification
algorithms, typically around 90 %, the critical question is whether a re-
jection module can boost precision to around 99 % to achieve a 1 %
error rate for real-world applications?

Thus, in this study, we propose a unified end-to-end document classi-
fication algorithm that integrates both novelty and ambiguity rejection.
The key contributions of this research are as follows:

1. A novel end-to-end framework for document classification that incorporates novelty and ambiguity rejection. The framework uses embedding features from the last hidden layers of a deep network and integrates a *k*th nearest neighbor (KNN) algorithm with Euclidean distance as the scoring function to effectively handle outliers and ambiguous cases.

2. A novel deep metric learning approach is proposed, incorporating a custom margin-based contrastive loss function, to optimize BERT-based networks, effectively compacting the knowledge space. This enhances the performance of the novelty and ambiguity rejector, leading to significant improvements in both precision and recall for the classification pipeline.

3. We perform thorough experimental evaluations comparing the proposed rejection method against existing approaches using both private and public datasets. Results demonstrate that the proposed method consistently outperforms baseline methods in both novelty detection and ambiguity rejection within an integrated pipeline.

4. The proposed algorithm achieves exceptional performance metrics, including high precision, recall, and coverage on private datasets. Additionally, it achieves state-of-the-art novelty detection results on the public RVL-CDIP dataset.

## 2. Related work

### 2.1. *Document classification*

Document classification methods use clues hidden in the document's
content to classify a given document into a category. This contextual

information can be text, layout, and visual information. This informa-
tion can be used separately or combined together by advanced machine
learning techniques such as Natural Language Processing (NLP) [1,2],
Deep Convolution Neural Networks (Deep CNN) [3], and multimodal
deep learning [4–6]. With NLP approaches [13], text information is
extracted from document images using Optical Character Recognition
(OCR), then using NLP such as BERT-based models are used to clas-
sify the text into category. Meanwhile, with the approach of Deep CNN
[12,14], the visual information is extracted by CNN and then classi-
fied into category by fully connected layers. Recently, many studies
[5,7] have used multimodal deep learning to extract both text and vi-
sual information, then combine them to classify documents. Some of
the studies [15,16] use fusion methods to leverage visual, textual docu-
ment images into a joint embedding space to perform entity recognition
on documents. The others [4,5] utilize an end-to-end trainable deep
architecture.

Most recent studies [4,5,8] have focused on combining positional,
textual, and visual features in an end-to-end trainable network to
perform document classification task and are evaluated on the pub-
licly available RVL-CDIP (Ryerson Vision Lab Complex Document
Information Processing) [14] dataset. This dataset is a subset of the IIT-
CDIP Test Collection [14], consists of gray-scale labeled documents of 16
categories, and includes 320 K training documents, 40 K documents for
validation and test sets. On the RVL-CDIP test dataset, the DocFormer
End-to-End Transformer published by Appalaraju et al. [5] performed
the highest accuracy at 96.17 %. The second place is LayoutLMv3 pub-
lished by Xu et al. [4] with 95.44 % accuracy. Next algorithm is the
SelfDoc published by Li et al. [8] with an accuracy of 93.81 %. These re-
searchers proposed cross-modal learning in the model pre-training stage
to leverage positional, textual, and visual information via self-supervised
learning from millions of unlabeled document images of public datasets
such as IIT-CDIP [14]. Afterward, the authors fine-tuned the models on
RVL-CDIP [14] dataset for downstream task of document classifica-
tion. Besides, by utilizing the visual and textual features of documents,
Bakkali et al. [15] employed a cross-modal deep network to perform
end-to-end document image classification. Their proposed cross-modal
network was trained and validated on RVL-CDIP [14] test dataset. The
algorithm outperformed LayoutLM [17] by 2.63 % in classification accu-
racy. In order to apply document classification method for small private
dataset, Mahamoud et al. [16] proposed a method using multimodality
and attention patterns on images and text. The model was trained and
tested on the French dataset and achieved very high accuracy at 98.2 %.
However, compared to BERT-based method that utilizes only textual information, the difference in accuracy is not significant, only 0.7 %.

Current research on document classification has primarily focused on optimizing algorithms for closed datasets, without emphasis on developing an end-to-end pipeline for real-world applications. In such applications, the input data consists not only of documents belonging to known categories but also includes novel documents from new or unknown categories. These applications need to be capable of novelty rejection while also achieving highly efficient classification on selected documents.

## 2.2. Novelty detection

Novelty detection is a critical task for deploying machine learning models in the real-world applications. We also call to it as out-of-distribution (OOD) detection where a given sample is detected as novelty (or OOD) if it is relatively far away from known in-distribution (ID) samples [18-20]. Depending on the accessibility of OOD data, novelty detection methods are categorized into three kinds: supervised [21], self-supervised [22], and unsupervised [9,18] approaches. In the supervised approach, through training on both ID and OOD data, models are expected to generalize information from both ID and OOD data, thereby being able to recognize OOD samples [21]. However, this approach faces the challenge of collecting a sufficiently large dataset that covers all possible OOD cases in reality, which is infeasible. This is not only because OOD data is highly diverse, but also because in the case of administrative documents, new data categories continuously emerge whenever new services are introduced. On the other hand, in order to generalize effectively on the ID data and exhibit distinctions from OOD data, the self-supervised methods [22] employ augmentation techniques (e.g., image rotation) on the ID dataset to enable the model to learn a feature space in which the inter-class separation is greater than the intra-class separation. However, this method may be challenging to apply in the context of natural language due to the limitations of text augmentation techniques. Unsupervised methods only utilize ID data during training [11,18,23]. Initially, a classifier is trained on the ID dataset. Then, this classifier is employed to extract features. A scoring function transforms these features of a given sample into a real number. By using the ID data, an appropriate threshold is chosen based on the desired true positive rate. The OOD detector classifies the input as OOD if the score is smaller than the selected threshold value. The extracted features can be the class probabilities [24-26] or other latent space metrics [9,10,27]. Recent studies on novelty detection in pattern recognition have achieved state-of-the-art efficacy, often focusing on utilizing embedding features and Euclidean distance [9,11,18].

Despite being extensively studied in the field of computer vision, out-of-distribution detection has not received much attention in the field of natural language processing and document analysis. Shu et al., 2017 [25] build a multi-class classifier with a 1-vs-rest final layer of sigmoid rather than softmax then use Gaussian fitting to evaluate thresholds for OOD Detection. Cho et al. [28] enhance OOD detection in NLP by leveraging intermediate representations and show the potential of intermediate representations in OOD detection. Wu et al. [29] and Zhou et al. [30] improve OOD detection performance by using contrastive learning. Gunel et al. [31] apply Supervised Contrastive Learning for Pre-trained Language Models to enhance OOD detection. Most recently, Larson et al. [12] in 2022, have introduced a novel dataset (namely RVL-CDIP-O) for evaluating the out-of-distribution performance of document classification. The authors conducted experiments to compare the effectiveness of novelty detection of different network architectures combined with OOD detection methods on this dataset after training the network on the popular RVL-CDIP dataset. Their highest AUC and FPR@TPR95 were 92.30 % and 46.10 %, respectively.

Research on novelty detection for document classification is still limited, primarily at a fundamental comparison level using the RVL-CDIP dataset. While many algorithms exist in computer vision, adapting them

to document classification is challenging. Supervised methods face difficulties due to the constant emergence of new document categories or formats, making it hard to collect diverse OOD datasets. Self-supervised approaches are restricted by the limitations of text augmentation techniques, unlike their broader algorithms in image tasks. Furthermore, no comprehensive study has evaluated the overall performance of an application when integrating novelty detection algorithms with classification, a crucial aspect for practical applications. The major questions for such an application are: (1) Does novelty detection contribute to increasing the precision of the classification application? (2) To achieve an acceptable precision level (such as 99 %), what percentage of the input documents can the application effectively process?

## 2.3. Deep metric learning

Overall, the Deep Metric Learning (namely DML) [32,33] is a family of methods that focus on optimizing neural networks based on the principle of enhancing the similarity or dissimilarity between semantic features of samples depending on whether the samples belong to the same object or different objects. DML was conceived based on the idea of a contrastive loss [34-36]. Bromley et al. [37] proposed the first contrastive loss function for Siamese Networks in 1993. The loss function measures the similarity between every pair of samples, where positive pairs belong to the same category, and negative pairs belong to different categories. This loss function was later formally defined by Chopra et al. [38] with application to Face Verification in 2005, where positive pairs belong to the same person. Similarly, the Triplet loss was proposed in studies by Schroff et al. [39], using triplets: an anchor, a positive example, and a negative example. By using Triplet Loss, the distance between an anchor and a positive example, both of which belong to the same identity, is minimized, while the distance between the anchor and a negative example of a different identity is maximized. Building on that, many studies have focused on applying contrastive learning to both self-supervised and supervised learning. In self-supervised contrastive learning [36], where data lacks labels, a positive pair often involves data augmentations of the same sample, and negative pairs are constructed using the anchor and randomly selected samples from the mini-batch. It has also been utilized and has achieved high effectiveness in supervised learning with a labeled dataset [32,35,40], where positive samples belong to the same label, and negative samples belong to different labels.

Recently, DML has been employed in numerous studies [9,18,26] to optimize neural networks, thereby enhancing the effectiveness of the novelty detection and Out-of-distribution detection problems. Cevikalp et al. [9] employed a supervised center contrastive loss function to train their neural networks, followed by utilizing the Euclidean distance to the class centers as a score function for novelty detection. This approach achieved state-of-the-art accuracies on the CIFAR-10, MNIST, and Fashion-MNIST datasets. Yang et al. [26] designed a novel one-vs-all loss to optimize their networks, followed by utilizing embedding feature distance-based rejection and probability-based rejection to improve novelty detection performance. Sun et al. [11] utilized supervised contrastive learning to optimize a CNN and subsequently combined it with K-nearest neighbors to improve the performance of Out-of-distribution detection on several image benchmark datasets. Zhou et al. [30] employed margin-based contrastive learning to optimize neural network then used the Mahalanobis distance in the model's penultimate layer to detect out-of-distribution instances in Sentiment Analysis, Topic Classification, and Question Classification. However, there has not yet been any research assessing the effectiveness of DML for the problem of novelty and ambiguity rejection.

# 3. Proposed method

## 3.1. Unified end-to-end document classification framework

In this study, we propose a novel end-to-end framework for document classification that integrates novelty and ambiguity rejection, as
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Overview of the proposed end-to-end document classification framework with novelty and ambiguity rejection. The classifier is created by BERT-based transformer, and the novelty and ambiguity rejector is built by the KNN algorithm, utilizing the last hidden layer's features and Euclidean distance in the score function. The final classification result for a given document is: (1) Novelty (OOD); or (2) ID along with the document's label. The colors cobalt blue, lavender pink, and alloy orange respectively represent the information and data flow of the documents in the train, test, and novelty sets.</figcaption></figure>

illustrated in Fig. 2. This framework is designed to accurately classify in-distribution documents while rejecting out-of-distribution or ambiguous documents. It consists of three main components: Classifier, Novelty & Ambiguity Rejector, and Consensus Agreement components. The classifier is trained using deep metric learning on $D_{in}^{train}$, enabling it to function as an encoder for extracting embedding features and logits. Initially, the features and logits of the documents in $D_{in}^{train}$ are extracted, and the features are indexed for k-nearest neighbor search, which is a crucial step for the rejector in subsequent processes. Then, the embedding features of each document are used to find the k-nearest neighbors, and the Euclidean distance to the kth nearest neighbor is used as the final score for the rejector. Finally, the consensus agreement component determines the final result by aggregating the novelty results and the consensus between the logits output and the true label of the kth nearest neighbor. ID test documents ($D_{in}^{test}$) and a pre-defined true positive rate (TPR) are used to determine the threshold for the novelty rejector. The threshold, $\theta$, must ensure that the acceptance rate of documents in the $D_{in}^{test}$ set is at least equal to this TPR. For each new document, such as those in the OOD documents ($D_{ood}$), the features and logits are first extracted, followed by finding the k-nearest neighbors, calculating the novelty score, and determining the label or rejecting it using the agreement module and the threshold $\theta$. In this study, we use $D_{in}^{test}$ and $D_{ood}$ to evaluate the performance of the proposed method with different pre-defined TPRs.

## 3.2. Classifier and deep metric learning

In this study, the classifier is a BERT-based transformer, with OCR text from document images as input. The embedding vector size is 768, while the logits vector size equals the number of categories of $D_{in}$. We design a custom loss function to enhance the discriminative power of learned representations by encouraging similar instances to be closer to each other while pushing dissimilar instances farther apart. In addition, to enhance the efficiency of the custom loss function during network optimization, we also customize the mini-batch logic to adapt to our private document dataset.

### 3.2.1. Custom margin-based loss

Fig. 3 illustrates the method of using contrastive loss to compress knowledge of documents in the training set. In this study, we propose a Custom Margin-based Loss function at the mini-batch level, as outlined in Algorithm 1. This is based on Margin-based contrastive loss [30]. In which, documents belonging to the same category are referred to as positive documents, while those belonging to different categories are referred to as negative documents. The main idea of the margin loss is to ensure that the distance between any two negative documents is greater than the maximum distance between any two positive documents. At each step of the training process, firstly, we calculate the Margin, which represents the maximum distance between two positive

documents (Line 2). Next, we compute $L_{pos}$ as the average loss of the document and its positive documents. The loss between two positive documents is computed as the Euclidean distance between the two embedding features of the documents. After that, we compute $L_{neg}$, which is the average loss of the document and its negative documents. To compute the loss between two negative documents, we calculate the Euclidean distance between the two embedding features of the documents. If this distance is greater than the Margin, the loss is zero; otherwise, the loss is the difference between the Euclidean distance and the Margin. When $L_{pos}$ is optimized closer to zero, positive documents become closer to each other. Conversely, when $L_{neg}$ is optimized closer to zero, negative documents are pushed further apart, with the minimum distance being the Margin. $L_{margin}$ is the combination of $L_{pos}$ and $L_{neg}$ (Line 6). The final loss $L$ is the combination of $L_{CE}$ (Cross Entropy loss) and $L_{margin}$. To adjust the impact of Margin, $L_{pos}$, $L_{neg}$, and $L_{margin}$ on the final loss $L$, we utilize parameters $\alpha$, $\beta$, and $\lambda$.

### 3.2.2. Custom mini-batch logic

Typically, when using contrastive loss functions, studies often employ large mini-batch sizes of around a thousand samples per mini-batch [35]. However, our private dataset is imbalanced and some minor categories have a maximum of only 20 documents so we chose a mini-batch size of 128 documents per step. To ensure the effectiveness of the custom contrastive loss functions, we customize mini-batch logic to

<caption>**Algorithm 1** Custom margin-based contrastive loss function.</caption>

A(i) = {1, 2, ..., M} \ {i} is the set of all instances. P(i) = {p ∈ A(i) : y_i = y_p} is the set of positive instances from the same category as i. N(i) = {n ∈ A(i) : y_i ≠ y_n} is the set of negative instances from the different category as i. h ∈ R^d is the unnormalized embedding features of the last hidden layer. z is the L2-normalized of h.

1: **function** CUSTOMMarginLoss(λ, α, β)

2:   $\xi \leftarrow \max_{i=1}^{M} \max_{p \in P(i)} \|z_i - z_p\|^2$ ▷ maximum distance between positive pairs

3:   $\xi \leftarrow \xi * \alpha$ ▷ Margin with adjustable parameter

4:   $L_{pos} \leftarrow \sum_{i=1}^{M} \frac{1}{|P(i)|} \sum_{p \in P(i)} \|z_i - z_p\|^2$ ▷ Positive documents are attracted as close as possible

5:   $L_{neg} \leftarrow \sum_{i=1}^{M} \frac{1}{|N(i)|} \sum_{n \in N(i)} (\xi - \|z_i - z_n\|^2)_+ \downarrow$ Negative documents are repelled as far as possible

6:   $L_{margin} \leftarrow \frac{1}{d^M} (L_{pos} * \beta + L_{neg})$ ▷ Margin loss with adjustable parameter

7:   $L \leftarrow L_{ce} + \lambda L_{margin}$ ▷ $L_{ce}$ is categorical cross-entropy loss

8:   **return** L

9: **end function**
<figure><img src="image_6.png" /><figcaption>Fig. 3. The overall process of deep metric learning uses contrastive loss to compact in-distribution knowledge. Documents of the same category are optimized to be as close to each other as possible, while being distant from documents of different categories.</figcaption></figure>

select at least 4 and 8 documents from each category at each step on private and public datasets, respectively.

### 3.3. Novelty and ambiguity rejector

In this study, the trained classifier is used as Encode : $D_{in} \rightarrow R^m$ ($m=768$), meaning that for each document $d$, we can determine the embedding feature $x = Encode(d)$. We use the Euclidean distance to the $k$th nearest neighbor as final score, as shown in Formulas 1 and 2, and apply this score to determine whether a document $d$ is novel or not, as a binary classification problem in Formula 3. In addition to detecting novelty, this algorithm also helps eliminate ambiguous documents in the in-distribution set, thereby improving the precision of the final result.

$$ \text{EuclideanDistance}(p, q) = \sqrt{\sum_{i=1}^{m} (p_i - q_i)^2} \qquad (1) $$

$$ \text{Score}(x) = -\text{EuclideanDistance}(x, x_{\text{nearest}}^{k}) \qquad (2) $$

$$ \text{Novelty}(d) = \begin{cases} \text{No (In)} & \text{if Score}(x) \ge \theta \\ \text{Yes (Out)} & \text{otherwise} \end{cases} \qquad (3) $$

where vector $p = [p_1, p_2, ..., p_m]$ and vector $q = [q_1, q_2, ..., q_m]$ are two $m$-dimensional embedding vectors; $x$ and $x_{\text{nearest}}^k$ are the embedding features of document $d$ and its $k$th nearest document, respectively.

Algorithm 2 details the steps for determining the novelty threshold $\theta$ (Function 1) and performing novelty detection (Function 2). In order to estimate $\theta$, the embedding features of the trained documents $D_{in}^{\text{train}}$ are extracted and denoted as $X_{in}^{\text{train}}$. These $X_{in}^{\text{train}}$ features are then used to compute Scores[i] for each document $d_i$ in $D_{in}^{\text{test}}$ according to Formula 2. To ensure the true positive rate (tpr), at least $n \cdot \text{tpr}$ documents in $D_{in}^{\text{test}}$ must be correctly classified, where $n$ is the total number of documents in $D_{in}^{\text{test}}$. Therefore, these scores are sorted in descending order, and the score of the $(n \cdot \text{tpr})$-th element is the threshold that satisfies this condition. We use the threshold $\theta$ for inference to detect novelty. Given a document $d$, the embedding feature $x$ is extracted, and then $X_{in}^{\text{train}}$ is used to find the $k$-nearest neighbors and compute the Score as described in Formula 2. If this Score is smaller than the threshold $\theta$, the document is considered novel.

### 3.4. Consensus agreement

In this study, we use a consensus score between the logits-based classification outcomes and the true label of the $k$th nearest neighbor. Specifically, if the classification output differs from the true label of the $k$th nearest neighbor, we classify the document as a novelty document or reject the document without considering the OOD score calculated in the Section 3.3. In the experimental section, we compare both KNN-based rejection methods: (1) $k$th nearest neighbor without an agreement score (namely KNN); and (2) $k$th nearest neighbor with an agreement score (namely KNN*).

**Algorithm 2** Novelty detection using $k$th NN and Euclidean distance.

**Function 1:** Threshold $\theta$ satisfies that True Positive Rate of $D_{in}^{\text{test}} \ge \text{tpr}$

1: **function** THRESHOLD_ESTIMATION($D_{in}^{\text{train}}, D_{in}^{\text{test}}, k, \text{tpr}$)

2: $X_{in}^{\text{train}} \leftarrow \text{Encode}(D_{in}^{\text{train}}) \triangleright \text{Encode } \forall d \in D_{in}^{\text{train}}$

3: $n \leftarrow \text{cardinality}(D_{in}^{\text{test}}) \triangleright \text{The number of documents } \in D_{in}^{\text{test}}$

4: Scores $\leftarrow [0] * n \triangleright \text{Initial scores array for } n \text{ documents } \in D_{in}^{\text{test}}$

5: **for** $d_i \in D_{in}^{\text{test}}$ **do**

6: $x_i \leftarrow \text{Encode}(d_i)$

7: $X_{\text{nearest}}^{i} \leftarrow k$ nearest neighbors $\in X_{in}^{\text{train}}$ of $x_i$

8: $x_{\text{nearest}}^{k} \leftarrow$ the $k$th neighbor $\in X_{\text{nearest}}^{k}$

9: Scores[i] $\leftarrow$ -EuclideanDistance($x_i, x_{\text{nearest}}^{k}$) $\triangleright$ Max score is 0

10: **end for**

11: Scores $\leftarrow$ sort(Scores) $\triangleright$ Descending sort

12: $\text{ind} \leftarrow \text{ceil}(\text{tpr} * n) \triangleright$ The index of the last accepted document

13: threshold $\leftarrow$ Scores[$\text{ind}$]

14: **return** threshold

15: **end function**

**Function 2:** Novelty Detection by threshold = $\theta$

1: **function** NOVELTY_DETECTION($D_{in}^{\text{train}}, k, d, \theta$)

2: $X_{in}^{\text{train}} \leftarrow \text{Encode}(D_{in}^{\text{train}})$

3: $x \leftarrow \text{Encode}(d)$

4: $X_{\text{nearest}}^{d} \leftarrow k$ nearest neighbors $\in X_{in}^{\text{train}}$ of $x$

5: $x_{\text{nearest}}^{k} \leftarrow$ the $k$th neighbor of $X_{\text{nearest}}^{k}$

6: Score $\leftarrow$ -EuclideanDistance($x, x_{\text{nearest}}^{k}$)

7: **if** Score $\ge \theta$ **then**

8: **novelty** $\leftarrow$ No (In)

9: **else**

10: **novelty** $\leftarrow$ Yes (Out)

11: **end if**

12: **return** novelty

13: **end function**

## 4. Experimental results

### 4.1. Materials

In this work, we use two datasets to evaluate the effectiveness of proposed method including: (1) our private French administrative document dataset; and (2) the well-known English, large, and public RVL-CDIP [12,14] dataset. In order to evaluate the novelty detection efficiency, we have three scenarios named Private Small, Private Tiny, and RVL-CDIP respectively. With each scenario, we need to split the datasets into two sets (1) the set of documents of known categories (namely $D_{in}$); and (2) the set of novel documents (namely $D_{ood}$). We only use the documents in $D_{in}^{\text{train}}$ to train the network, and $D_{ood}$ to evaluate the novelty detection efficiency. The detailed information of these scenarios is shown in Table 1.

Our private dataset has 23,539 images belonging to 47 categories (Account Dues, Account Invoice, Bank Statements, Insurance Daily
<caption>**Table 1**</caption>

Detail information (including: The number of documents in train, validation and test datasets; and the number of categories) of Private Small, Private Tiny, and RVL-CDIP datasets. $D_{in}$ represents in distribution data, $D_{ood}$ represents out-of-distribution data.

<table><thead><tr><th rowspan="2">#</th><th rowspan="2">Type</th><th colspan="2">Private small</th><th colspan="2">Private tiny</th><th colspan="2">RVL-CDIP</th></tr><tr><th>D<sub>in</sub></th><th>D<sub>ood</sub></th><th>D<sub>in</sub></th><th>D<sub>ood</sub></th><th>D<sub>in</sub></th><th>D<sub>ood</sub></th></tr></thead><tbody><tr><td>1</td><td>Train</td><td>15,898</td><td>0</td><td>15,287</td><td>0</td><td>300,162</td><td>0</td></tr><tr><td>2</td><td>Validation</td><td>1766</td><td>0</td><td>1699</td><td>0</td><td>37,541</td><td>0</td></tr><tr><td>3</td><td>Test</td><td>5825</td><td>50</td><td>5623</td><td>930</td><td>37,420</td><td>2799</td></tr><tr><td>4</td><td>Total</td><td><b>23,489</b></td><td><b>50</b></td><td><b>22,609</b></td><td><b>930</b></td><td><b>375,123</b></td><td><b>2,799</b></td></tr><tr><td>5</td><td>Categories</td><td>36</td><td>11</td><td>23</td><td>24</td><td>16</td><td>-</td></tr></tbody></table>

Allowance, Mail Account Dues, etc.). The dataset is imbalanced. The smallest category has only 25 documents while the largest one has 4540 documents. We can see that $D_{ood}$ has about 25 % and 50 % of categories in Private Small and Private Tiny scenarios, respectively.

Besides, RVL-CDIP [14] corpus is a well-known benchmark dataset for document classification. It has 400,000 document images covering 16 categories (e.g., resume, invoice, letter, memo, etc.). This is a balanced dataset, where each category consists of 25,000 documents, and it is divided into a training set of 320,000 documents (20,000 per category), validation and test sets of 40,000 documents each (2500 per category per set). In the pre-processing step, we keep only documents with at least 10 words for the Text-Modality of the study. This dataset is used as the $D_{in}$ set to train and evaluate Text-Modality. In addition, the RVL-CDIP-O [12] dataset consisting of 3,415 documents is used in this study to evaluate the effectiveness of the out-of-distribution detection. After removing documents with less than 10 words, the final $D_{ood}$ set has 2799 documents as shown in Table 1.

## 4.2. Evaluation metrics

In order to evaluate novelty detection, we use True Positive Rate (TPR), False Positive Rate (FPR), and the area under the receiver operating characteristic curve (AUROC) which is drawn with the y-axis as true positive rate and the x-axis as false positive rate. The AUROC of test documents $\in D_{in} \cup D_{ood}$ is AUC. Besides, the document classification performance of classifiers is evaluated by three measures: (1) Accuracy (ACC); (2) Weighted Precision (wPRE); and (3) Mean Recall (mREC). Finally, the end-to-end document classification performance is evaluated by four measures: (1) Precision of known categories (PRE); (2) Recall of known categories (REC); (3) F1 score; and (4) Coverage rate (COV);

$$TPR = \frac{TP}{TP + FN} \qquad (4)$$

$$FPR = \frac{FP}{FP + TN} \qquad (5)$$

$$REC = \frac{TP_{correct}}{TP + FN} \qquad (6)$$

$$PRE = \frac{TP_{correct}}{TP + FP} \qquad (7)$$

$$F1 = 2 \times \frac{PRE \times REC}{PRE + REC} \qquad (8)$$

$$COV = \frac{TP_{correct}}{TP + FP + TN + FN} \qquad (9)$$

where Positive means a given document $\in D_{in}$ (not novelty label) and Negative means a given document $\in D_{ood}$ (novelty label). In more detail, TP stands for True Positive, when a given ID document is detected as ID; $TP_{correct}$ stands for True Positive and correctly classified when a given ID document is detected as ID and correctly classified; FN stands for False Negative, when a given ID document is predicted as OOD; FP

stands for False Positive, when a given OOD document is predicted as ID; and TN stands for True Negative, when a given OOD document is predicted as OOD. The higher TPR value the model achieves, the better it is, on the contrary, the lower FPR value the model achieves, the better it is. Besides, (TP + FN) and (TP + FP + TN + FN) represent the total number of documents in the $D_{in}^{test}$ set and $D_{in}^{test} \cup D_{ood}$ set, respectively, and these are constant values. Therefore, REC and COV are directly proportional to each other. REC always is higher than COV, and the higher REC, the higher COV.

## 4.3. Experimental setting

In this study, in order to evaluate the effectiveness of our proposed novelty detection framework for documents, after training classifiers by supervised learning on only the $D_{in}^{train}$ dataset, we use the best accuracy model on validation dataset as the final model. Next, performance of the multi-class classification on the $D_{in}^{test}$ dataset and novelty detection on the $D_{ood}$ dataset are evaluated. The study assesses and analyzes experimental results of two BERT-based architectures, including FlauBERT [2] (for the French corpus) and BERT [1] (for the English corpus). The textual content of the documents is extracted by a third-party OCR engine. For novelty detection, we use KNN algorithm using Euclidean distance of the kth nearest-neighbor as score function. Meta's Faiss library [41] is used to optimize the speed of KNN. Our private dataset contains certain categories with a limited number of documents in the training set. Consequently, we have opted to set the value of k to 1 for evaluating our method. For the loss functions, we use Supervised Contrastive (SCL) [35], Category-Weighted Cross-Entropy (Weight), Cross-Entropy (CE), and two Margin-based contrastive losses, namely Margin ($\alpha = 1.0, \beta = 1.0$, and $\lambda = 2.0$) and Margin* ($\alpha = 1.5, \beta = 0.5$, and $\lambda = 4.0$). The goal of the Margin* is to repel negative documents further than base Margin and increase the influence of $L_{neg}$ compared to $L_{pos}$ in final loss $L$. For Category-Weighted loss, the weights are inversely proportional to frequency of categories in training datasets.

## 4.4. Performance evaluation

In this paper, we used five loss functions to optimize the BERT-based classifiers. The classifiers are trained by supervised learning using the losses on only documents $\in D_{in}^{train}$. Then, we select the best models, which have highest accuracy on $D_{in}^{validation}$, as final models. Finally, we use the final models to evaluate: (1) classification performance on the test documents of $D_{in}^{test}$; (2) novelty detection performance on $D_{in}^{test}$ and $D_{ood}$; (3) combination of rejection and classification performance on $D_{in}^{test}$ and $D_{ood}$;

### 4.4.1. Document classification performances

The classification performances on $D_{in}^{test}$ of classifiers are shown in Table 2. Overall, the classifiers are very effective on the private datasets of both Small and Tiny scenarios, with all metrics surpassing 98 %, except mRecalls of Small are around 94–95 %. Meanwhile, with the public RVL-CDIP dataset, its metrics are lower, at only about 90 %.

On private document dataset, Margin performs the best among all metrics in both Small and Tiny scenarios, but these metrics are not significantly higher than the other losses. Specifically on Small, ACC and PRE are 98.76 % and 98.77 %, respectively, only 0.41 % and 0.40 % higher than the worst one (SCL). Meanwhile, the gap in mRecall between the first and worst ones is higher, equaling 1.02 %, but compared to the second one, it's only 0.35 %. Similarly, on the Tiny dataset, the metrics of Margin also exhibit minimal deviations in comparison to other loss functions, with the gap being smaller than that observed on the Small dataset, only 0.1 % for ACC and wPRE, and ranging from 0.3 % to 0.5 % for mRecall. Last but not least, the classifiers achieve higher effectiveness on the Tiny dataset compared to the Small dataset, particularly with regard to the mREC metric. The reason for this is that the Small dataset is more imbalanced due to the presence of an additional 13 categories,
<table><caption>**Table 2**
Comparison of classification results on private datasets and RVL-CDIP [14] dataset. The performance is calculated based on test documents ∈ $D_{in}^{test}$ set by the final model which is supervised learning on $D_{in}^{train}$ set.</caption><thead><tr><th rowspan="2">#</th><th rowspan="2">Loss</th><th colspan="3">Private small</th><th colspan="3">Private tiny</th><th colspan="3">RVL-CDIP</th></tr><tr><th>ACC↑</th><th>wPRE↑</th><th>mREC↑</th><th>ACC↑</th><th>wPRE↑</th><th>mREC↑</th><th>ACC↑</th><th>wPRE↑</th><th>mREC↑</th></tr></thead><tbody><tr><td>1</td><td>Margin</td><td><strong>98.76</strong></td><td><strong>98.77</strong></td><td><strong>95.04</strong></td><td><strong>99.13</strong></td><td><strong>99.13</strong></td><td><strong>98.37</strong></td><td>90.55</td><td>90.62</td><td>90.21</td></tr><tr><td>2</td><td>Margin*</td><td>98.52</td><td>98.55</td><td>94.02</td><td>98.99</td><td>99.00</td><td>98.11</td><td>90.52</td><td>90.59</td><td>90.19</td></tr><tr><td>3</td><td>SCL</td><td>98.35</td><td>98.37</td><td>94.09</td><td>99.04</td><td>99.05</td><td>98.00</td><td><strong>90.80</strong></td><td><strong>90.83</strong></td><td><strong>90.47</strong></td></tr><tr><td>4</td><td>Weight</td><td>98.66</td><td>98.64</td><td>94.89</td><td>98.92</td><td>98.92</td><td>98.08</td><td>90.58</td><td>90.66</td><td>90.26</td></tr><tr><td>5</td><td>CE</td><td>98.63</td><td>98.60</td><td>94.82</td><td>99.08</td><td>99.07</td><td>97.88</td><td>90.49</td><td>90.59</td><td>90.19</td></tr></tbody></table>

and these categories have very limited training documents, with only tens to a hundred records per category.

On public RVL-CDIP dataset, performance is significantly lower than in the private dataset scenarios. The ACC, wPRE, and mREC metrics are all around 90 %, with the SCL performing the best metrics at 90.80 %, 90.83 %, and 90.47 %, respectively. However, there is not much disparity between the classifiers using the different loss functions, only about 0.2 %. These results also do not exhibit significant deviation when compared to the research by Larson et al. [12] which employed the Visual-Modality (90.50 %) and Multi-Modality of LayoutLMv2 (88.7 %, achieved by the author) on a similar training data resource. However, they are lower than the reported result of LayoutLMv2 (95.3 %) [7] with a larger training dataset.

**Compacted Knowledge** In this study, we employ a contrastive loss function to compact knowledge space. After training the model with each loss function, we utilize features to retrieve the most relevant document for each document using the K-nearest neighbors algorithm. Subsequently, we compute the distance between these two documents. The closer the distance approaches zero, the more similar the two documents are. Fig. 4 illustrates the variance and standard deviation of these distances for each category. In summary, compared to CE, all contrastive losses contribute to reducing the standard deviation and variance of each category in all three scenarios: Private Small, Private Tiny, and Public RVL-CDIP. Specifically, SCL achieves the lowest standard deviation, followed by Margin and Margin* losses. On the private dataset, Margin compresses knowledge better than Margin*. However, on RVL-CDIP, the difference is negligible. There is a notable observation that on the private dataset, three categories (namely Account Invoice, Account Due, and Burn Order) exhibit significantly different standard deviations compared to the remaining categories. On the RVL-CDIP dataset, the probability distribution of the categories is relatively uniform, with minimal variation.

### 4.4.2. Novelty detection performances

The effectiveness of novelty detection is evaluated based on two test datasets, including $D_{in}^{test}$ and $D_{ood}$. In this study, AUC is utilized for evaluation, similar to other binary classification problems. Additionally, we employ FPR, F1, and COV at a true positive rate of 95 % to compare with another method [12]. In Table 3, we compare our proposed KNN approach with other alternative outlier detection methods including: Mahalanobis [24], Cosine similarity [27], Energy [27], and Maximum Softmax Probability (MSP) [24]. In which, KNN*, KNN, Cosine, and Mahalanobis are using the embedding features of the last hidden layer, while the remaining methods use logits information from the output layer. The MSP method transforms logits vector into probability information using the softmax function.

Overall, methods utilizing embedding features are more effective than those employing logits. The AUC gap between these two groups depends on the loss function and ranges from 2 % to 6 %, 2 % to 9 %, and 4 % to 8 % on the Private Small, Private Tiny, and Public RVL-CDIP datasets, respectively. KNN performs better than KNN* in terms of novelty detection (higher AUC, lower FPR), but the difference is not substantial across all three scenarios. However, KNN lags behind KNN*

in terms of F1 and COV measurements on two private datasets. This is because KNN* attempts to eliminate ambiguous cases in classification results, so it has to reduce the threshold to achieve the same @TPR95. This leads to an increase in FPR.

On private document dataset, our proposed KNN-based rejection methods are the best algorithms. When combined with KNN, Margin-based contrastive losses perform the highest AUC, followed by CE, and lastly SCL on both Small and Tiny. For the FPR@TPR95, this order remains unchanged with Margin performing the lowest FPR, followed by Margin*, CE, and lastly SCL on the Tiny dataset. However, on the Small dataset, CE performs the lowest FPR, followed by Margin*, SCL, and Margin & Weight being the least effective.

On public RVL-CDIP [12,14] dataset, the KNN-CE combination achieves the highest AUC of 93.44 %. However, the Mahalanobis-Margin* combination records the lowest FPR@TPR95 at 35.58 %. Nonetheless, with an AUC of 93.44 %, this metric surpasses the current state-of-the-art [12] on the same dataset by 1.14 % (93.44 % vs. 92.3 %). Our proposed algorithm also significantly reduces the FPR compared to SOTA, achieving FPR values of 35.38 %, 39.80 %, and 39.7 % for Mahalanobis-Margin*, KNN-CE, and KNN-Margin*, respectively, as opposed to the SOTA's 46.1 % [12]. Nevertheless, the challenge with this dataset is that the ACC on $D_{in}^{test}$ is only around 90 %, so evaluating based on a TPR threshold of 95 % is relatively high, necessitating additional assessments for comparison.

### 4.4.3. End-to-end document classification performances

In a comprehensive document classification system in real-world scenarios, it involves a novelty detection step followed by correctly assigning labels to the selected documents. In such a system, apart from evaluating each step independently, which includes novelty detection and document classification, we need to assess the full pipeline when these two steps are combined. After rejecting out-of-distribution documents from the entire set of documents, we have a set of selected documents. It's essential to have the following evaluations: (1) recall of known categories; (2) precision on the selected documents. Figs. 5 and 6 respectively compare the effectiveness of AUC and Precision-Recall of six out-of-distribution methods when combined with five loss functions across three datasets. Fig. 7 compares the effectiveness through the metrics of AUC, PRE, and REC among five loss functions when combined with KNN* and KNN on three datasets. Finally, Table 4 provides a detailed comparison by these metrics of rejection methods for each important threshold of PRE corresponding to each dataset scenario. Table 5 shows the effectiveness of the best rejection method (KNN*) at important thresholds of PRE corresponding to each dataset scenario.

**Comparison by AUC and ROC Curve:** Fig. 5 illustrates that the feature-based OOD methods exhibit superior TPR performance compared to the two logits-based methods at the same FPR. Notably, these methods achieve higher TPR values compared to the other methods when FPR ranges from 0 to 20 % on the private dataset and to 60 % on the public dataset. Among the feature-based methods, KNN demonstrates the highest AUC across all scenarios, with KNN* trailing slightly behind with a negligible difference in AUC. Cosine and Mahalanobis rank
<figure><img src="image_7.png" /><figcaption>Fig. 4. Illustration of quartile of 1st-nearest distance score of documents grouped together by category of private datasets and RVL-CDIP [14] dataset.</figcaption></figure>

third and fourth on the private Small and Tiny datasets. On the Public RVL-CDIP dataset, Mahalanobis outperforms Cosine, although the gap between these two methods is not significant. With an FPR below 20 % in all three scenarios, both KNN and KNN* achieve similar TPRs, surpassing the other two remaining feature-based methods. Following them, Cosine ranks third, while Mahalanobis ranks last.

**Comparison by Precision and Recall:** Fig. 6 demonstrates that the feature-based out-of-distribution methods exhibit superior PRE performance compared to the two logits-based methods at the same REC.

At the same REC, KNN* consistently outperforms KNN in terms of the PRE metric when combined with five loss functions in all data scenarios. On the private datasets, KNN* achieves superior PRE compared to KNN, Mahalanobis, and Cosine at the same REC. In the REC range below 70 %, Mahalanobis ranks second, followed by KNN and Cosine. However, with REC above 70 %, this order changes to KNN in second place, followed by Cosine and Mahalanobis. On the public dataset, Mahalanobis outperforms the remaining methods when REC is less than 35 %, followed by KNN*, Cosine, and KNN, respectively. However, when
<table><caption>**Table 3**
Comparison of novelty detection results on the private small and private tiny datasets and RVL-CDIP [12,14] dataset. FPR is FPR@TPR95, the lower FPR value the model achieves, the better it is.</caption><thead><tr><th rowspan="2">Loss</th><th colspan="4">Private small</th><th colspan="4">Private tiny</th><th colspan="4">RVL-CDIP</th></tr><tr><th>FPR↓</th><th>AUC↑</th><th>F1↑</th><th>COV↑</th><th>FPR↓</th><th>AUC↑</th><th>F1↑</th><th>COV↑</th><th>FPR↓</th><th>AUC↑</th><th>F1↑</th><th>COV↑</th></tr></thead><tbody><tr><td colspan="13" style="text-align: left; font-weight: bold;">KNN*</td></tr><tr><td>Margin</td><td>10.00</td><td>98.55</td><td>97.20</td><td>94.01</td><td>05.91</td><td>98.36</td><td>96.82</td><td>81.41</td><td>50.73</td><td>91.73</td><td>89.32</td><td>82.60</td></tr><tr><td>Margin*</td><td>08.00</td><td>98.56</td><td>97.16</td><td>93.96</td><td>06.24</td><td>98.02</td><td><strong>96.82</strong></td><td><strong>81.43</strong></td><td>45.05</td><td>92.45</td><td>89.46</td><td>82.56</td></tr><tr><td>SCL</td><td>08.00</td><td>98.29</td><td>97.16</td><td>93.96</td><td>08.06</td><td>98.20</td><td>96.63</td><td>81.40</td><td>52.30</td><td>91.49</td><td>89.19</td><td>82.53</td></tr><tr><td>Weight</td><td>10.00</td><td>98.37</td><td>97.25</td><td>94.11</td><td>07.53</td><td>98.21</td><td>96.73</td><td>81.46</td><td>48.12</td><td>92.21</td><td>89.35</td><td>82.55</td></tr><tr><td>CE</td><td>02.00</td><td>98.44</td><td>97.20</td><td>93.97</td><td>07.20</td><td>97.99</td><td>96.74</td><td>81.43</td><td>45.84</td><td>92.56</td><td>89.37</td><td>82.49</td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">KNN</td></tr><tr><td>Margin</td><td>10.00</td><td>98.68</td><td>97.17</td><td>93.97</td><td><strong>05.91</strong></td><td><strong>98.50</strong></td><td>96.77</td><td>81.37</td><td>45.80</td><td>92.42</td><td>89.45</td><td>82.57</td></tr><tr><td>Margin*</td><td>06.00</td><td><strong>98.81</strong></td><td>97.13</td><td>93.92</td><td>06.02</td><td>98.29</td><td>96.80</td><td>81.40</td><td>39.87</td><td>93.19</td><td><strong>89.64</strong></td><td>82.56</td></tr><tr><td>SCL</td><td>08.00</td><td>98.41</td><td>97.10</td><td>93.91</td><td>07.63</td><td>98.29</td><td>96.63</td><td>81.37</td><td>48.77</td><td>92.17</td><td>89.48</td><td><strong>82.69</strong></td></tr><tr><td>Weight</td><td>10.00</td><td>98.58</td><td>97.24</td><td>94.14</td><td>07.31</td><td>98.42</td><td>96.70</td><td>81.43</td><td>43.16</td><td>92.85</td><td>89.57</td><td>82.60</td></tr><tr><td>CE</td><td><strong>02.00</strong></td><td>98.64</td><td>97.15</td><td>93.92</td><td>06.77</td><td>98.37</td><td>96.66</td><td>81.34</td><td>39.73</td><td><strong>93.44</strong></td><td>89.55</td><td>82.47</td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">Mahalanobis</td></tr><tr><td>Margin</td><td>12.00</td><td>98.07</td><td>97.00</td><td>93.82</td><td>08.06</td><td>98.17</td><td>96.49</td><td>81.28</td><td>43.62</td><td>92.19</td><td>89.32</td><td>82.38</td></tr><tr><td>Margin*</td><td>16.00</td><td>97.86</td><td>96.82</td><td>93.67</td><td>08.39</td><td>97.74</td><td>96.48</td><td>81.29</td><td><strong>35.58</strong></td><td>93.20</td><td>89.52</td><td>82.31</td></tr><tr><td>SCL</td><td>10.00</td><td>98.04</td><td>96.76</td><td>93.58</td><td>07.85</td><td>97.98</td><td>96.48</td><td>81.26</td><td>42.77</td><td>92.32</td><td>89.60</td><td>82.62</td></tr><tr><td>Weight</td><td>10.00</td><td>98.20</td><td>97.02</td><td>93.94</td><td>09.78</td><td>97.90</td><td>96.30</td><td>81.26</td><td>41.76</td><td>92.47</td><td>89.38</td><td>82.38</td></tr><tr><td>CE</td><td>10.00</td><td>98.12</td><td>96.85</td><td>93.67</td><td>10.75</td><td>97.67</td><td>96.21</td><td>81.23</td><td>38.87</td><td>93.05</td><td>89.36</td><td>82.27</td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">Cosine</td></tr><tr><td>Margin</td><td>10.00</td><td>98.36</td><td>97.10</td><td>93.91</td><td>07.31</td><td>98.19</td><td>96.58</td><td>81.31</td><td>45.98</td><td>92.20</td><td>89.38</td><td>82.51</td></tr><tr><td>Margin*</td><td>12.00</td><td>98.42</td><td>97.05</td><td>93.87</td><td>09.14</td><td>97.88</td><td>96.51</td><td>81.37</td><td>39.01</td><td>92.99</td><td>89.54</td><td>82.44</td></tr><tr><td>SCL</td><td>08.00</td><td>98.30</td><td>96.86</td><td>93.67</td><td>10.54</td><td>97.89</td><td>96.34</td><td>81.32</td><td>44.80</td><td>92.21</td><td>89.59</td><td>82.67</td></tr><tr><td>Weight</td><td>10.00</td><td>98.39</td><td>97.00</td><td>93.82</td><td>09.68</td><td>97.87</td><td>96.54</td><td>81.50</td><td>42.09</td><td>92.76</td><td>89.48</td><td>82.48</td></tr><tr><td>CE</td><td>10.00</td><td>98.45</td><td>96.94</td><td>93.75</td><td>11.83</td><td>97.81</td><td>96.27</td><td>81.35</td><td>38.94</td><td>93.22</td><td>89.50</td><td>82.40</td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">Energy</td></tr><tr><td>Margin</td><td>12.00</td><td>92.28</td><td>97.12</td><td>93.94</td><td>22.80</td><td>89.64</td><td>95.34</td><td>81.31</td><td>57.20</td><td>85.56</td><td>88.98</td><td>82.49</td></tr><tr><td>Margin*</td><td>12.00</td><td>93.34</td><td>97.11</td><td>93.92</td><td>17.10</td><td>94.78</td><td>95.77</td><td>81.29</td><td>54.56</td><td>87.28</td><td>89.09</td><td>82.51</td></tr><tr><td>SCL</td><td>18.00</td><td>92.20</td><td>96.82</td><td>93.67</td><td>20.97</td><td>93.00</td><td>95.45</td><td>81.28</td><td>58.02</td><td>76.78</td><td>88.94</td><td>82.48</td></tr><tr><td>Weight</td><td>14.00</td><td>94.96</td><td>97.19</td><td>94.09</td><td>19.14</td><td>93.36</td><td>95.72</td><td>81.43</td><td>53.59</td><td>87.85</td><td>89.09</td><td>82.48</td></tr><tr><td>CE</td><td>12.00</td><td>96.40</td><td>97.05</td><td>93.87</td><td>12.37</td><td>94.97</td><td>96.19</td><td>81.32</td><td>53.02</td><td>88.09</td><td>88.97</td><td>82.35</td></tr><tr><td colspan="13" style="text-align: left; font-weight: bold;">MSP</td></tr><tr><td>Margin</td><td>12.00</td><td>93.72</td><td>97.14</td><td>93.96</td><td>16.13</td><td>92.26</td><td>95.94</td><td>81.38</td><td>65.02</td><td>84.01</td><td>88.64</td><td>82.42</td></tr><tr><td>Margin*</td><td>12.00</td><td>94.33</td><td>97.12</td><td>93.94</td><td>15.48</td><td>95.72</td><td>95.98</td><td>81.35</td><td>61.81</td><td>86.54</td><td>88.80</td><td>82.46</td></tr><tr><td>SCL</td><td>20.00</td><td>93.84</td><td>96.95</td><td>93.80</td><td>15.48</td><td>93.19</td><td>95.96</td><td>81.35</td><td>58.74</td><td>75.69</td><td>88.97</td><td>82.53</td></tr><tr><td>Weight</td><td>12.00</td><td>94.43</td><td><strong>97.54</strong></td><td><strong>94.67</strong></td><td>18.82</td><td>93.90</td><td>95.82</td><td>81.46</td><td>64.56</td><td>87.24</td><td>88.65</td><td>82.41</td></tr><tr><td>CE</td><td>10.00</td><td>96.75</td><td>97.11</td><td>93.92</td><td>13.87</td><td>95.45</td><td>96.20</td><td>81.43</td><td>65.31</td><td>86.11</td><td>88.49</td><td>82.28</td></tr></tbody></table>

REC is higher, KNN* becomes the best method, followed by KNN, Cosine, and Mahalanobis as the least effective.

**Comparison of loss functions:** Fig. 7 illustrates that KNN* and KNN have the same trend when combined with five loss functions across all datasets; however, KNN* significantly improves precision efficiency at each recall compared to KNN in all scenarios. In general, Margin* outperforms other loss functions in most cases and is very stable. Specifically, when combined with KNN*, the Margin-based loss functions achieve the highest recall, followed by Weight, CE, and SCL on the Private Small dataset at a precision of 99.5 % or higher, respectively. The order remains the same at the same precision range on the Private Tiny dataset, the only change is that Margin achieves the highest recall when precision ranges from 99.5 % to 99.8 %. On the Public RVL-CDIP dataset at precision from 99.0 %, Margin achieves the highest recall, followed by SCL, CE, and lastly Margin*. However, with precision below 98.5 %, the order changes to Margin* being the best, followed by Margin and SCL, with CE being the last.

**Special highlights** Let's delve into the special highlights of the results, which offer crucial insights into the outcomes of the study. In real-practical applications, we need an algorithm to achieve high PRE to minimize error rate and also high COV to process as many documents as possible. These are the two most important indicators for applying the algorithm in practice. In Table 4, we compare loss functions and rejection methods at PRE equal to 99.5 %, 99.5 %, and 99.0 % for the Private Small, Private Tiny dataset, and Public RVL-CDIP, respectively. The table illustrates that the KNN-based rejection methods not only detect novel

documents but also have the capability to eliminate ambiguity in classification outcomes, consequently enhancing select accuracy and achieving the highest REC and COV at the same PRE on all datasets.

**On the Private Small dataset**, at the same PRE of 99.5 %, in terms of the rejection method, KNN* stands out as the most efficient, followed by KNN, MSP, Energy, Cosine, and finally, Mahalanobis. Regarding the loss function, Margin proves to be the most effective, followed by Weight, Margin*, CE, and SCL, respectively. Although the Mahalanobis-CE combination achieves the lowest FPR, KNN*-Margin proves to be the most effective, achieving the highest REC and COV at 97.34 % and 96.51 %, respectively. The next one is KNN-Margin with REC and COV slightly lower by only 0.17 %. Compared to KNN-based methods, both Mahalanobis and Cosine approaches, utilizing embedding features, achieve REC and COV effectiveness with a difference ranging from 1 % to 5 %, depending on the loss function.

**On the Private Tiny dataset**, at the same PRE of 99.5 %, overall, KNN is the most effective rejection method, and Margin* is the most efficient loss function. The difference between KNN* and KNN is not significant. Among the rejection methods, there is a notable distinction between approaches using embedding features and those using on logits information. Both logits-based methods (Energy and MSP) achieve very low REC and COV ranging from 0–40 % and 0–34 %, respectively, while the remaining the features-based methods achieve high REC and COV ranging from 76–89 % and 65–76 %, respectively. KNN-Margin* achieves the highest REC and COV at 89.49 % and 76.79 %, respectively. Slightly less but not significantly different is KNN*-Margin* with metrics of 89.47 %
<figure><img src="image_8.png" /><figcaption>Fig. 5. Comparison of AUC of novelty detection results of different OOD methods on the private small, private tiny and RVL-CDIP [12,14] datasets.</figcaption></figure>

and 76.77 %, respectively. In this scenario, we only use $D_{in}^{train}$ for a half of the categories for training (23/47), and the final $D_{test}^{test}$ has 14.19 % of $D_{ood}$ and 85.81 % of $D_{in}^{test}$. This means the maximum COV of end-to-end document classification is 85.81 %. Without novelty rejection, its PRE is also at most 85.81 %, which is too low for a real-world application.

However, when using our proposed rejection method, the framework increases PRE to 99.5 % (an increase of 13.69 %), while only decreasing COV by 9.02 %.

On the Public RVL-CDIP dataset, the study selected PRE level of 99.0 because the ACC of the models on $D_{in}^{test}$ of this dataset is only about 90 %.
<figure><img src="image_9.png" /><figcaption>Fig. 6. The curve of combination of precision and recall of known categories on the private small, private tiny and RVL-CDIP [14] datasets.</figcaption></figure>

Overall, features-based methods significantly outperform logsits-based methods by a large margin. These features-based rejection methods increase PRE by 10% from about 90% to 99% for all models of the five loss functions. This is a very important result because for real-practical application we need a near perfect PRE at about 99%, and an error rate of only 1%. Besides, among the features-based methods, KNN*

demonstrates the highest effectiveness compared to the other features-based methods. The KNN\*-Margin combination achieves the highest REC and COV at 53.82% and 50.07%, respectively. These results are much higher by over 3% compared to the remaining features-based methods (KNN, Mahalanobis, and Cosine) and far surpass by over 30% the two logsits-based methods (Energy and MSP).
<figure><img src="image_10.png" /><figcaption>Fig. 7. Comparison of AUC, precision, and recall of our end-to-end document classification of different loss functions when combining with KNN* and KNN rejection methods on the Private Small, Private Tiny datasets and RVL-CDIP [12,14] datasets.</figcaption></figure>

<caption>**Table 4**</caption>

Comparison of novelty detection results on the private small and private tiny datasets and RVL-CDIP [12,14] dataset at the same precision threshold. The precision threshold is 99.5 % for the private small and private tiny datasets, and is 99.0 % for the public RVL-CDIP dataset. The lower FPR value the model achieves, the better it is.

<table><thead><tr><th rowspan="2">Loss</th><th colspan="4">Private small</th><th colspan="4">Private tiny</th><th colspan="4">RVL-CDIP</th></tr><tr><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th></tr></thead><tbody><tr><td colspan="12" style="text-align: center;">KNN*@PRE99.5</td></tr><tr><td>Margin</td><td>12.00</td><td><strong>97.72</strong></td><td><strong>97.34</strong></td><td><strong>96.51</strong></td><td>02.26</td><td>89.51</td><td>89.44</td><td>76.74</td><td>00.82</td><td><strong>54.30</strong></td><td><strong>53.82</strong></td><td><strong>50.07</strong></td></tr><tr><td>Margin*</td><td>12.00</td><td>97.25</td><td>96.88</td><td>96.05</td><td>02.37</td><td>89.53</td><td>89.47</td><td>76.77</td><td>01.00</td><td>51.84</td><td>51.40</td><td>47.82</td></tr><tr><td>SCL</td><td>10.00</td><td>96.46</td><td>96.07</td><td>95.25</td><td>01.94</td><td>87.46</td><td>87.36</td><td>74.96</td><td>00.96</td><td>53.82</td><td>53.35</td><td>49.64</td></tr><tr><td>Weight</td><td>14.00</td><td>97.32</td><td>96.94</td><td>96.12</td><td>01.83</td><td>89.40</td><td>89.28</td><td>76.61</td><td>00.89</td><td>49.97</td><td>49.54</td><td>46.09</td></tr><tr><td>CE</td><td>12.00</td><td>97.22</td><td>96.84</td><td>96.02</td><td>02.26</td><td>88.90</td><td>88.83</td><td>76.22</td><td>00.68</td><td>51.42</td><td>50.95</td><td>47.41</td></tr><tr><td colspan="12" style="text-align: center;">KNN@PRE99.5</td></tr><tr><td>Margin</td><td>12.00</td><td>97.55</td><td>97.17</td><td>96.34</td><td>02.04</td><td>88.94</td><td>88.83</td><td>76.22</td><td>00.61</td><td>50.36</td><td>49.90</td><td>46.43</td></tr><tr><td>Margin*</td><td>12.00</td><td>97.17</td><td>96.79</td><td>95.97</td><td>02.15</td><td><strong>89.58</strong></td><td><strong>89.49</strong></td><td><strong>76.79</strong></td><td>00.82</td><td>49.12</td><td>48.69</td><td>45.30</td></tr><tr><td>SCL</td><td>08.00</td><td>95.66</td><td>95.26</td><td>94.45</td><td>01.72</td><td>86.93</td><td>86.79</td><td>74.47</td><td>00.75</td><td>50.35</td><td>49.91</td><td>46.43</td></tr><tr><td>Weight</td><td>12.00</td><td>97.15</td><td>96.77</td><td>95.95</td><td>01.94</td><td>89.54</td><td>89.42</td><td>76.73</td><td>00.71</td><td>45.49</td><td>45.09</td><td>41.95</td></tr><tr><td>CE</td><td>12.00</td><td>96.96</td><td>96.58</td><td>95.76</td><td>02.15</td><td>88.76</td><td>88.67</td><td>76.09</td><td>00.57</td><td>48.19</td><td>47.76</td><td>44.43</td></tr><tr><td colspan="12" style="text-align: center;">Mahalanobis@PRE99.5</td></tr><tr><td>Margin</td><td>12.00</td><td>94.94</td><td>94.58</td><td>93.77</td><td>01.72</td><td>83.43</td><td>83.30</td><td>71.48</td><td>01.04</td><td>50.60</td><td>50.17</td><td>46.68</td></tr><tr><td>Margin*</td><td>10.00</td><td>92.94</td><td>92.57</td><td>91.78</td><td>02.04</td><td>82.18</td><td>82.11</td><td>70.46</td><td>01.04</td><td>48.58</td><td>48.17</td><td>44.82</td></tr><tr><td>SCL</td><td>08.00</td><td>92.17</td><td>91.79</td><td>91.01</td><td>01.61</td><td>81.59</td><td>81.45</td><td>69.89</td><td>01.21</td><td>50.24</td><td>49.83</td><td>46.36</td></tr><tr><td>Weight</td><td>06.00</td><td>92.22</td><td>91.83</td><td>91.05</td><td>02.15</td><td>85.15</td><td>85.08</td><td>73.00</td><td>01.14</td><td>48.67</td><td>48.27</td><td>44.91</td></tr><tr><td>CE</td><td><strong>04.00</strong></td><td>92.96</td><td>92.53</td><td>91.74</td><td>01.94</td><td>79.58</td><td>79.51</td><td>68.23</td><td>01.00</td><td>47.32</td><td>46.92</td><td>43.66</td></tr><tr><td colspan="12" style="text-align: center;">Cosine@PRE99.5</td></tr><tr><td>Margin</td><td>12.00</td><td>96.52</td><td>96.14</td><td>95.32</td><td>01.72</td><td>83.91</td><td>83.78</td><td>71.89</td><td>00.89</td><td>51.50</td><td>51.05</td><td>47.50</td></tr><tr><td>Margin*</td><td>12.00</td><td>95.36</td><td>95.00</td><td>94.20</td><td>02.15</td><td>85.84</td><td>85.77</td><td>73.60</td><td>00.96</td><td>48.19</td><td>47.78</td><td>44.46</td></tr><tr><td>SCL</td><td>06.00</td><td>91.91</td><td>91.52</td><td>90.74</td><td>01.40</td><td>76.58</td><td>76.44</td><td>65.59</td><td>00.93</td><td>49.35</td><td>48.92</td><td>45.52</td></tr><tr><td>Weight</td><td>10.00</td><td>94.99</td><td>94.61</td><td>93.80</td><td>01.83</td><td>84.12</td><td>84.01</td><td>72.09</td><td>01.00</td><td>44.93</td><td>44.55</td><td>41.45</td></tr><tr><td>CE</td><td>06.00</td><td>94.49</td><td>94.08</td><td>93.28</td><td>01.94</td><td>81.84</td><td>81.75</td><td>70.15</td><td>00.64</td><td>45.16</td><td>44.75</td><td>41.64</td></tr><tr><td colspan="12" style="text-align: center;">Energy@PRE99.5</td></tr><tr><td>Margin</td><td>18.00</td><td>97.36</td><td>97.03</td><td>96.20</td><td>00.00</td><td>00.57</td><td>00.57</td><td>00.49</td><td>00.64</td><td>16.61</td><td>16.49</td><td>15.34</td></tr><tr><td>Margin*</td><td>18.00</td><td>96.31</td><td>95.98</td><td>95.17</td><td>00.65</td><td>22.57</td><td>22.57</td><td>19.37</td><td>00.57</td><td>19.85</td><td>19.70</td><td>18.33</td></tr><tr><td>SCL</td><td>08.00</td><td>86.56</td><td>86.20</td><td>85.46</td><td>00.43</td><td>20.20</td><td>20.18</td><td>17.32</td><td>00.43</td><td>05.88</td><td>05.86</td><td>05.45</td></tr><tr><td>Weight</td><td>14.00</td><td>96.45</td><td>96.09</td><td>95.27</td><td>00.11</td><td>03.77</td><td>03.77</td><td>03.24</td><td>00.54</td><td>12.42</td><td>12.34</td><td>11.48</td></tr><tr><td>CE</td><td>12.00</td><td>96.03</td><td>95.66</td><td>94.84</td><td>00.75</td><td>26.61</td><td>26.61</td><td>22.83</td><td>00.39</td><td>12.92</td><td>12.82</td><td>11.93</td></tr><tr><td colspan="12" style="text-align: center;">MSP@PRE99.5</td></tr><tr><td>Margin</td><td>20.00</td><td>97.46</td><td>97.15</td><td>96.32</td><td>00.00</td><td>00.00</td><td>00.00</td><td>00.00</td><td>00.00</td><td>00.02</td><td>00.02</td><td>00.01</td></tr><tr><td>Margin*</td><td>18.00</td><td>96.60</td><td>96.27</td><td>95.46</td><td>00.86</td><td>30.30</td><td>30.30</td><td>26.00</td><td>00.46</td><td>11.52</td><td>11.44</td><td>10.65</td></tr><tr><td>SCL</td><td>12.00</td><td>93.99</td><td>93.63</td><td>92.83</td><td>00.11</td><td>11.52</td><td>11.51</td><td>09.87</td><td>00.36</td><td>05.80</td><td>05.78</td><td>05.38</td></tr><tr><td>Weight</td><td>16.00</td><td>97.00</td><td>96.65</td><td>95.83</td><td>00.00</td><td>01.81</td><td>01.81</td><td>01.56</td><td>00.00</td><td>00.02</td><td>00.02</td><td>00.01</td></tr><tr><td>CE</td><td>16.00</td><td>96.52</td><td>96.17</td><td>95.35</td><td>01.18</td><td>40.55</td><td>40.55</td><td>34.79</td><td>00.00</td><td>00.01</td><td>00.01</td><td>00.01</td></tr></tbody></table>
<table><caption>**Table 5**
The KNN* rejection effectiveness of the loss functions is compared at the same precision thresholds. The thresholds are 99.8 %, 99.5 %, and 99.0 % on the private small dataset, 99.5 %, 99.0 %, and 98.5 % on the private tiny dataset, and 99.0 %, 98.5 %, and 98.0 % on the Public RVL-CDIP dataset.</caption><thead><tr><th rowspan="2">Loss</th><th colspan="4">Private small</th><th colspan="4">Private tiny</th><th colspan="4">RVL-CDIP</th></tr><tr><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th><th>FPR↓</th><th>TPR↑</th><th>REC↑</th><th>COV↑</th></tr></thead><tbody><tr><td></td><td colspan="4">@PRE99.8</td><td colspan="4">@PRE99.5</td><td colspan="4">@PRE99.0</td></tr><tr><td>Margin</td><td>02.00</td><td>91.36</td><td>91.21</td><td>90.43</td><td>02.26</td><td>89.51</td><td>89.44</td><td>76.74</td><td>00.82</td><td><strong>54.30</strong></td><td><strong>53.82</strong></td><td><strong>50.07</strong></td></tr><tr><td>Margin*</td><td>02.00</td><td><strong>92.70</strong></td><td><strong>92.55</strong></td><td><strong>91.76</strong></td><td>02.37</td><td><strong>89.53</strong></td><td><strong>89.47</strong></td><td><strong>76.77</strong></td><td>01.00</td><td>51.84</td><td>51.40</td><td>47.82</td></tr><tr><td>SCL</td><td><strong>00.00</strong></td><td>87.91</td><td>87.74</td><td>87.00</td><td><strong>01.94</strong></td><td>87.46</td><td>87.36</td><td>74.96</td><td>00.96</td><td>53.82</td><td>53.35</td><td>49.64</td></tr><tr><td>Weight</td><td>06.00</td><td>92.39</td><td>92.27</td><td>91.49</td><td>01.83</td><td>89.40</td><td>89.28</td><td>76.61</td><td>00.89</td><td>49.97</td><td>49.54</td><td>46.09</td></tr><tr><td>CE</td><td>02.00</td><td>92.07</td><td>91.91</td><td>91.13</td><td>02.26</td><td>88.90</td><td>88.83</td><td>76.22</td><td><strong>00.68</strong></td><td>51.42</td><td>50.95</td><td>47.41</td></tr><tr><td></td><td colspan="4">@PRE99.5</td><td colspan="4">@PRE99.0</td><td colspan="4">@PRE98.5</td></tr><tr><td>Margin</td><td>12.00</td><td><strong>97.72</strong></td><td><strong>97.34</strong></td><td><strong>96.51</strong></td><td>05.16</td><td><strong>94.15</strong></td><td><strong>94.06</strong></td><td><strong>80.71</strong></td><td>02.29</td><td>64.41</td><td>63.62</td><td>59.19</td></tr><tr><td>Margin*</td><td>12.00</td><td>97.25</td><td>96.88</td><td>96.05</td><td>05.27</td><td>93.46</td><td>93.38</td><td>80.13</td><td>02.04</td><td><strong>66.32</strong></td><td><strong>65.48</strong></td><td><strong>60.92</strong></td></tr><tr><td>SCL</td><td><strong>10.00</strong></td><td>96.46</td><td>96.07</td><td>95.25</td><td><strong>04.84</strong></td><td>92.83</td><td>92.71</td><td>79.55</td><td>02.18</td><td>64.36</td><td>63.56</td><td>59.14</td></tr><tr><td>Weight</td><td>14.00</td><td>97.32</td><td>96.94</td><td>96.12</td><td>05.27</td><td>94.04</td><td>93.97</td><td>80.63</td><td>01.89</td><td>62.83</td><td>62.03</td><td>57.71</td></tr><tr><td>CE</td><td>12.00</td><td>97.22</td><td>96.84</td><td>96.02</td><td>04.95</td><td>92.46</td><td>92.35</td><td>79.25</td><td><strong>01.79</strong></td><td>62.69</td><td>61.88</td><td>57.57</td></tr><tr><td></td><td colspan="4">@PRE99.0</td><td colspan="4">@PRE98.5</td><td colspan="4">@PRE98.0</td></tr><tr><td>Margin</td><td>32.00</td><td><strong>98.99</strong></td><td><strong>98.28</strong></td><td><strong>97.45</strong></td><td><strong>07.63</strong></td><td><strong>96.23</strong></td><td><strong>96.03</strong></td><td><strong>82.41</strong></td><td>03.97</td><td>70.64</td><td>69.52</td><td>64.68</td></tr><tr><td>Margin*</td><td>30.00</td><td>98.66</td><td>97.94</td><td>97.11</td><td>07.96</td><td>96.05</td><td>95.91</td><td>82.30</td><td>03.57</td><td><strong>71.80</strong></td><td><strong>70.63</strong></td><td><strong>65.72</strong></td></tr><tr><td>SCL</td><td><strong>24.00</strong></td><td>98.15</td><td>97.37</td><td>96.54</td><td>07.85</td><td>94.93</td><td>94.79</td><td>81.34</td><td>04.00</td><td>70.33</td><td>69.22</td><td>64.40</td></tr><tr><td>Weight</td><td>32.00</td><td>99.00</td><td>98.30</td><td>97.46</td><td>07.85</td><td>95.39</td><td>95.25</td><td>81.73</td><td>03.43</td><td>70.33</td><td>69.17</td><td>64.36</td></tr><tr><td>CE</td><td>38.00</td><td>98.56</td><td>97.91</td><td>97.07</td><td>07.96</td><td>95.61</td><td>95.48</td><td>81.93</td><td><strong>03.29</strong></td><td>69.95</td><td>68.80</td><td>64.01</td></tr></tbody></table>

**Special highlights of KNN***: Table 5 describes scenarios of error rates less than 2 % when using KNN* on the datasets. For the Private Small dataset, reducing the error rate by 0.3 % (increasing PRE from 99.5 % to 99.8 %) results in a 4.75 % decrease in COV, from 96.51 %. Conversely, increasing the error rate to 1 % (PRE = 99.0 %) leads to a 0.94 % increase in COV, but the FPR is very high at 32 %. This implies that, for this dataset, we should choose PRE threshold of 99.5 % to ensure a balance between FPR, PRE, and COV. Similarly, on the Private Tiny dataset, the most appropriate PRE threshold is 99.0 %, with a corresponding COV of 80.71 %, Compared to the model without a rejection module, it increases PRE by 13.19 % (85.81 % vs 99.0 %) and reduces COV by 5.1 % (85.81 % vs 80.71 %). Finally, for the Public RVL-CDIP dataset, PRE thresholds of 98.5 % and 98.0 % both yield very suitable COV and FPR, at 60.92 % & 2.04 % and 65.72 % & 3.57 %, respectively.

### 4.5. Discussion

This study has thoroughly evaluated a comprehensive algorithm for real-world document classification problem. The research has conducted a thorough assessment of numerous out-of-distribution methods and loss functions across three scenarios using both private and public datasets.

The research findings indicate that the contrastive loss function effectively compacts knowledge during training, consequently enhancing the performance of out-of-distribution detection. The last hidden layer's features contain more valuable information than logits of the output layer for novelty detection problem. All embedding feature-based methods demonstrate superior novelty detection performance compared to methods utilizing the logits information. Among them, the KNN-based methods not only emerge as the best novelty detection algorithm but also effectively eliminate ambiguous results, leading to a significant increase in PRE to nearly perfect levels, exceeding 99.50 % on the private datasets and achieving an approximately 9 % improvement on the public RVL-CDIP dataset. This result indicates that the correlation between a given document and its nearest neighbors from the training set provides valuable information for detecting novel documents and rejecting ambiguous documents. In terms of end-to-end document classification, KNN* stands out as the most efficient. KNN* removes ambiguous results when there is inconsistency between KNN output and logits output. Therefore, it improves precision and recall of known categories.

When combined with Margin-based loss functions, the KNN-based detection methods achieve the best performance in three scenarios. The

KNN*-Margin combination achieves the best results on the Private Small dataset, with FPR, PRE, REC and COV at 12.00 %, 99.50 %, 97.34 %, and 96.51 %, respectively. On the Private Tiny dataset, the KNN*-Margin combination performs the best metrics at 5.16 %, 99.00 %, 94.06 %, and 80.71 %, respectively. On public RVL-CDIP dataset, the KNN*-Margin* combination achieves the highest metrics at 2.04 %, 98.50 %, 65.48 %, and 60.92 %, respectively.

Finally, the KNN-Margin* combination achieves an AUC of 93.19 % and FPR@TPR95 of 39.87 %, representing a 0.89 % improvement in AUC score and a significant 6.23 % reduction in FPR score compared to the current state-of-the-art [12] on the public RVL-CDIP dataset.

### 5. Conclusion

This study proposes an end-to-end document classification framework with both novelty and ambiguity rejection. We propose a custom Margin-based contrastive loss function as a deep metric learning method to optimize neural networks and compress knowledge. This is the first study exploring the combination of text-based classifiers and KNN algorithm for not only novelty detection but also ambiguity rejection. Our study demonstrates the efficacy of embedding feature distances of the kth nearest neighbor for out-of-distribution detection on administrative documents. The most significant insights are that high-quality embedding features and a suitable distance measure are two indispensable key factors for novelty and ambiguity rejection. Indeed, when combined with custom contrastive loss, KNN improves the effectiveness of novelty detection and ambiguity rejection, achieving very high PRE, REC and COV levels that are entirely applicable in real-world scenarios. Our experiment illustrates that on the Private Small dataset, the KNN*-Margin combination achieves the highest effectiveness with PRE, REC and COV of 99.50 %, 97.34 % and 96.51 %, respectively. Meanwhile, on the Private Tiny dataset, KNN*-Margin outperforms with these metrics at 99.00 %, 94.06 % and 80.71 %, while utilizing only half of the categories in the training set and having over 14 % of the final $D^{test}$ as $D_{ood}$, meaning it only loses about 5 % of $D^{test}$ to increase PRE to 99.00 % or reduce the error to just 1 %. Last but not least, the proposed method is scalable. When evaluated on the public RVL-CDIP dataset with classifiers having only 90 % accuracy, it also surpasses other rejection methods, excelling not only in novelty rejection but especially in ambiguity rejection. The proposed method increases PRE to 99.0 %, a level that can be used in real applications, while maintaining COV at 50.07 %. It also achieves
state-of-the-art novelty detection results with an AUC of 93.19 % and an FPR@TPR95 of 39.87 %.

However, there are various aspects that could be further improved in our method. Firstly, novelty rejection methods currently use a global threshold that is not optimized for each category. Overall, the categories are optimized similarly, and the variance and standard deviation of these $k$th NN distances are not significantly different, as observed in public datasets. However, there may still be some categories with relatively large variances, as seen in the case of private datasets. Therefore, optimizing the threshold per category will lead to an increase in rejection effectiveness, although the algorithm and evaluation process will be more complex. Secondly, the margin-based contrastive loss function could be further fine-tuned for better knowledge compactness, requiring more time and GPU resources for a more detailed evaluation. Finally, the proposed algorithm may be applied to different datasets and neural network architectures, and further experimentation is needed in the future.

## CRediT authorship contribution statement

**Tri-Cong Pham:** Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Formal analysis, Data curation, Conceptualization. **Mickaël Coustaty:** Writing – review & editing, Supervision, Resources, Project administration, Methodology, Investigation, Funding acquisition, Conceptualization. **Antoine Doucet:** Writing – review & editing, Validation, Supervision, Methodology, Conceptualization. **Aurélie Joseph:** Writing – review & editing, Validation, Supervision, Methodology, Investigation, Data curation, Conceptualization. **Vincent Poulain d'Andecy:** Writing – review & editing, Validation, Supervision, Methodology, Funding acquisition, Data curation, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

This work was supported by the French government through the France Relance program and the ANR under grant number ANR-21-PRRD-0010-01, as well as by the YOOZ company. We also would like to thank Guénael Manic, Mohamed Saadi, Jonathan Ouellet and Jérôme Lacour from YOOZ for their support.

## Data availability

This study uses both public and private datasets. The public dataset is available at https://huggingface.co/datasets/aharley/rvlCDIP. The private dataset cannot be shared due to data privacy policies

## References

[1] J. Devlin, M.W. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171–4186.

[2] H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, D. Schwab, FlaubERT: unsupervised language model pre-training for French, in: Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC), 2020, pp. 2479–2490.

[3] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2012).

[4] Y. Huang, T. Lv, L. Cui, Y. Lu, F. Wei, LayoutLMv3: pre-training for document ai with unified text and image masking, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4083–4091, https://doi.org/10.1145/3503161.3548112.

[5] S. Appalaraju, B. Jasani, B.U. Kota, Y. Xie, R. Manmatha, DocFormer: end-to-end transformer for document understanding, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 973–983, https://doi.org/10.1109/ICCV48922.2021.00103.

[6] S. Bakkali, Z. Ming, M. Coustaty, M. Rusiñol, O.R. Terrades, Vlcdoc: vision-language contrastive pre-training model for cross-modal document classification, Pattern Recognit. 139 (2023) 109419, https://doi.org/10.1016/j.patcog.2023.109419.

[7] X. Yang, X. Yiheng, L. Tengchao, C. Lei, W. Furu, W. Guoxin, L. Yijuan, F. Dinei, Z. Cha, C. Wanxiang, Z. Min, Z. Lidong, LayoutLMv2: multi-modal pre-training for visually-rich document understanding, in: Proceedings of ACL-IJCNLP, 2021, pp. 2579–2591, https://doi.org/10.18653/v1/2021.acl-long.2021.

[8] P. Li, J. Gu, J. Kuen, V.I. Morariu, H. Zhao, R. Jain, V. Manjunath, H. Liu, SelfDoc: self-supervised document representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5648–5656 https://doi.org/10.1109/CVPR46437.2021.00560.

[9] H. Cevikalp, B. Uzun, Y. Salk, H. Saribas, O. Köpüklü, From anomaly detection to open set recognition: bridging the gap, Pattern Recognit. 138 (2023) 109385, https://doi.org/10.1016/j.patcog.2023.109385.

[10] H. Huang, Y. Wang, Q. Hu, M.-M. Cheng, Class-specific semantic reconstruction for open set recognition, IEEE Trans. Pattern Anal. Mach. Intell. 45 (2023) 4214–4228, https://doi.org/10.1109/TPAMI.2022.3200384.

[11] Y. Sun, Y. Ming, X. Zhu, Y. Li, Out-of-distribution detection with deep nearest neighbors, in: Proceedings of the 39th International Conference on Machine Learning (ICML) (2022).

[12] S. Larson, G. Lim, Y. Ai, D. Kuang, K. Leach, Evaluating out-of-distribution performance on document image classifiers, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks (2022).

[13] T.C. Pham, M. Coustaty, A. Joseph, V.P. d'Andecy, M. Visani, N. Sidere, Incremental learning and ambiguity rejection for document classification, in: Proceedings of the International Conference on Document Analysis and Recognition (ICDAR) (2023).

[14] A.W. Harley, A. Ufkes, K.G. Derpanis, Evaluation of deep convolutional nets for document image classification and retrieval, in: Proceedings of the International Conference on Document Analysis and Recognition (ICDAR) (2015) https://doi.org/10.1109/ICDAR.2015.7333910.

[15] S. Bakkali, Z. Ming, M. Coustaty, M. Rusiñol, EAML: ensemble self-attention-based mutual learning network for document image classification, Int. J. Doc. Anal. Recognit. (IJDAR) 24 (2021) 251–268, https://doi.org/10.1007/s10032-021-00378-0.

[16] I.S. Mahamoud, J. Voerman, M. Coustaty, P.A. Joseph, V.P. d'Andecy, J.M. Ogier, Multimodal attention-based learning for imbalanced corporate documents classification, in: Proceedings of the International Conference on Document Analysis and Recognition (ICDAR) (2021).

[17] Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, M. Zhou, LayoutLM: pre-training of text and layout for document image understanding, in: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 1192–1200, https://doi.org/10.1145/3394486.3403172.

[18] R. Tal, H. Yedid, Mean-shifted contrastive loss for anomaly detection, in: AAAI Conference on Artificial Intelligence (AAAI) (2023).

[19] L.A. Souto Arias, C.W. Oosterlee, P. Cirillo, Aida: analytic isolation and distance-based anomaly detection algorithm, Pattern Recognit. 141 (2023) 109607, https://doi.org/10.1016/j.patcog.2023.109607.

[20] J. Tu, H. Liu, C. Li, Weighted subspace anomaly detection in high-dimensional space, Pattern Recognit. 146 (2024) 110056, https://doi.org/10.1016/j.patcog.2023.110056.

[21] K. Lee, K. Lee, H. Lee, J. Shin, A simple unified framework for detecting out-of-distribution samples and adversarial attacks, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2018).

[22] L. Bergman, Y. Hoshen, Classification-based anomaly detection for general data, in: Proceedings of the International Conference for Learning Representations (ICLR) (2020).

[23] H. Cevikalp, B. Uzun, O. Köpüklü, G. Ozturk, Deep compact polyhedral conic classifier for open and closed set recognition, Pattern Recognit. 119 (2021) 108080, https://doi.org/10.1016/j.patcog.2021.108080.

[24] D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out-of-distribution examples in neural networks, in: Proceedings of the International Conference for Learning Representations (ICLR) (2017).

[25] L. Shu, H. Xu, B. Liu, Doc: deep open classification of text documents, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2017).

[26] H.M. Yang, X.Y. Zhang, F. Yin, Q. Yang, C.L. Liu, Convolutional prototype network for open set recognition, IEEE Trans. Pattern Anal. Mach. Intell. 44 (2022) 2358–2370, https://doi.org/10.1109/TPAMI.2020.3045079.

[27] W. Liu, X. Wang, J.D. Owens, Y. Li, Energy-based out-of-distribution detection, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2020).

[28] H. Cho, C. Park, J. Kang, K.M. Yoo, T. Kim, S. Goo Lee, Enhancing out-of-distribution detection in natural language understanding via implicit layer ensemble, in: the Findings of the Association for Computational Linguistics: EMNLP, 2022, pp. 783–798, https://doi.org/10.18653/v1/2022.findings-emnlp.55.

[29] Y. Wu, K. He, Y. Yan, Q. Gao, Z. Zeng, F. Zheng, L. Zhao, H. Jiang, W. Wu, W. Xu, Revisit overconfidence for OOD detection: reassigned contrastive learning with adaptive class-dependent threshold, in: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (2022) https://doi.org/10.18653/v1/2022.naACL-main.307.

[30] W. Zhou, F. Liu, M. Chen, Contrastive out-of-distribution detection for pretrained transformers, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2021) https://doi.org/10.18653/v1/2021.emnlp-main.84.
[31] B. Gunel, J. Du, A. Conneau, V. Stoyanov, Supervised contrastive learning for pre-trained language model fine-tuning, in: Proceedings of the International Conference for Learning Representations (ICLR) (2021).

[32] P. Liu, Z. Liu, Y. Lang, S. Liu, Q. Zhou, Q. Li, Deep metric learning assisted by intra-variance in a semi-supervised view of learning, Eng. Appl. Artif. Intell. 131 (2024) 107885, https://doi.org/10.1016/j.engappai.2024.107885.

[33] J. Park, J. Hong, J. Kwon, End-to-end metric learning from corrupted images using triplet dimensionality reduction loss, Expert Syst. Appl. 238 (2024) 122064, https://doi.org/10.1016/j.eswa.2023.122064.

[34] X. Dong, J. Shen, Triplet loss in siamese network for object tracking, in: Proceedings of the European Conference on Computer Vision (ECCV) (2018).

[35] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, D. Krishnan, Supervised contrastive learning, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2020).

[36] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: Proceedings of the 37th International Conference on Machine Learning (ICML) (2020).

[37] J. Bromley, I. Guyon, Y. LeCun, E. Sackinger, R. Shah, Signature verification using a "siamese" time delay neural network, in: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (1993).

[38] S. Chopra, R. Hadsell, Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2005).

[39] F. Schroff, D. Kalenichenko, J. Philbin, Facenet: a unified embedding for face recognition and clustering, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2015).

[40] H. Wang, J. Cao, Z.L. Shi, C.S. Leung, R. Feng, W. Cao, Y. He, Image classification on hypersphere loss, IEEE Trans. Ind. Inf. (2024) 1–11, https://doi.org/10.1109/TII.2023.3347759.

[41] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, H. Jégou, The faiss library (2024), https://github.com/facebookresearch/faiss.

## Author biography

<figure><img src="image_11.png" /></figure>

Tri-Cong Pham received his MSc degree in computer science from the Hanoi University of Science and Technology in 2013 and his PhD degree in computer science from the University of Science and Technology of Hanoi in 2021, respectively. He is currently a researcher at L3i Laboratory, La Rochelle University, France. He has published numerous research articles in the field of document analysis and medical image processing. His research interests include computer vision, document analysis, and artificial intelligence.

<figure><img src="image_12.png" /></figure>

Mickaël Coustaty received his MSc degree from La Rochelle University, after working on symbol recognition using structural signature. He received his PhD degree in computer science from the La Rochelle University in 2011. He has been an Associate Professor in computer science with the L3i Laboratory, La Rochelle University. He is also the Director of the IDEAS LabCom, a joint laboratory in partnership between the L3i laboratory and the Yooz company. His main research interests include multimodal, multilingual content analysis and recognition.

<figure><img src="image_13.png" /></figure>

Antoine Doucet received his PhD degree in computer science from the University in Helsinki, Finland, in 2005, and the French Research Supervision Habilitation (HDR) degree, in 2012. He has been a Full Professor in computer science with the L3i Laboratory, La Rochelle University, since 2014. He is also the Director of the ICT Department, University of Science and Technology of Hanoi. He leads the Research Group, La Rochelle, in document analysis, digital contents, and images.

<figure><img src="image_14.png" /></figure>

Aurëlie Joseph received her MSc degree in computer science in 2009, and her PhD degree in computer science in 2013 from the Paris 13 University, respectively. She is currently the Innovation Lab Team Leader of the Yooz company. Her main research interests include document analysis, document classification, information extraction, business intelligence. She has published numerous research articles in the field of document analysis in high-ranking international conferences and journals.

<figure><img src="image_15.png" /></figure>

Vincent Poulain d'Andecy received his PhD degree in computer science from the La Rochelle University in 2021. He is currently the manager of Research & Technologies department of the Yooz company. His research interests include document analysis, document classification, information extraction, text mining and intelligence automation. He has published numerous research articles in the field of document analysis in high-ranking international conferences and journals.