Out-of-Distribution Detection with Attention Head Masking for Multimodal
  Document Classification
Authors: name: Christos Constantinou; name: Georgios Ioannides; name: Aman Chadha; name: Aaron Elkins; name: Edwin Simpson
Published: 2024-08-20 23:30:00

Abstract: Detecting out-of-distribution (OOD) data is crucial in machine learning
applications to mitigate the risk of model overconfidence, thereby enhancing
the reliability and safety of deployed systems. The majority of existing OOD
detection methods predominantly address uni-modal inputs, such as images or
texts. In the context of multi-modal documents, there is a notable lack of
extensive research on the performance of these methods, which have primarily
been developed with a focus on computer vision tasks. We propose a novel
methodology termed as attention head masking (AHM) for multi-modal OOD tasks in
document classification systems. Our empirical results demonstrate that the
proposed AHM method outperforms all state-of-the-art approaches and
significantly decreases the false positive rate (FPR) compared to existing
solutions up to 7.5\%. This methodology generalizes well to multi-modal data,
such as documents, where visual and textual information are modeled under the
same Transformer architecture. To address the scarcity of high-quality publicly
available document datasets and encourage further research on OOD detection for
documents, we introduce FinanceDocs, a new document AI dataset. Our code and
dataset are publicly available.