SI-ViT: Shuffle instance-based Vision Transformer for pancreatic cancer ROSE image classification

Authors:
raw_author_name: Tianyi Zhang
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Youdan Feng
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Yu Zhao
raw_affiliation_string: Department of Pathology, Peking Union Medical College Hospital, Beijing, 100006, China
raw_author_name: Yanli Lei
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Nan Ying
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Fan Song
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Yufang He
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China
raw_author_name: Zhiling Yan
raw_affiliation_string: School of Biological Sciences, Nanyang Technological University, Singapore, 639798, Singapore
raw_author_name: Yunlu Feng
raw_affiliation_string: Department of Gastroenterology, Peking Union Medical College Hospital, Beijing, 100006, China
raw_author_name: Aiming Yang
raw_affiliation_string: Department of Gastroenterology, Peking Union Medical College Hospital, Beijing, 100006, China
raw_author_name: Guanglei Zhang
raw_affiliation_string: Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China

Abstract: The rapid on-site evaluation (ROSE) technique improves pancreatic cancer diagnosis by enabling immediate analysis of fast-stained cytopathological images. Automating ROSE classification could not only reduce the burden on pathologists but also broaden the application of this increasingly popular technique. However, this approach faces substantial challenges due to complex perturbations in color distribution, brightness, and contrast, which are influenced by various staining environments and devices. Additionally, the pronounced variability in cancerous patterns across samples further complicates classification, underscoring the difficulty in precisely identifying local cells and establishing their global relationships. To address these challenges, we propose an instance-aware approach that enhances the Vision Transformer with a novel shuffle instance strategy (SI-ViT). Our approach presents a shuffle step to generate bags of shuffled instances and corresponding bag-level soft-labels, allowing the model to understand relationships and distributions beyond the limited original distributions. Simultaneously, combined with an un-shuffle step, the traditional ViT can model the relationships corresponding to the sample labels. This dual-step approach helps the model to focus on inner-sample and cross-sample instance relationships, making it potent in extracting diverse image patterns and reducing complicated perturbations. Compared to state-of-the-art methods, significant improvements in ROSE classification have been achieved. Aiming for interpretability, equipped with instance shuffling, SI-ViT yields precise attention regions that identifying cancer and normal cells in various scenarios. Additionally, the approach shows excellent potential in pathological image analysis through generalization validation on other datasets. By proposing instance relationship modeling through shuffling, we introduce a new insight in pathological image analysis. The significant improvements in ROSE classification leads to protential AI-on-site applications in pancreatic cancer diagnosis. The code and results are publicly available at https://github.com/sagizty/MIL-SI.