PointNu-Net: Keypoint-Assisted Convolutional Neural Network for Simultaneous Multi-Tissue Histology Nuclei Segmentation and Classification

Authors:
raw_author_name: Kai Yao
raw_affiliation_string: Block Upsample Block Conv Block Upsample Block Conv Block; Kaizhu Huang is with Duke Kunshan University, Suzhou, Kunshan, 215316, China.; University of Liverpool and School of Advanced Technology, Xi'an Jiaotong-Liverpool University.; is with School of Computing, Edinburgh Napier University, Edinburgh, EH11 4BN, the United Kingdom.
raw_author_name: Kaizhu Huang
raw_affiliation_string: Block Upsample Block Conv Block Upsample Block Conv Block; Kaizhu Huang is with Duke Kunshan University, Suzhou, Kunshan, 215316, China.; University of Liverpool and School of Advanced Technology, Xi'an Jiaotong-Liverpool University.; is with School of Computing, Edinburgh Napier University, Edinburgh, EH11 4BN, the United Kingdom.
raw_author_name: Jie Sun
raw_affiliation_string: Block Upsample Block Conv Block Upsample Block Conv Block; Jie Sun is with School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, Jiangsu, 215000, China.; Kaizhu Huang is with Duke Kunshan University, Suzhou, Kunshan, 215316, China.; University of Liverpool and School of Advanced Technology, Xi'an Jiaotong-Liverpool University.; is with School of Computing, Edinburgh Napier University, Edinburgh, EH11 4BN, the United Kingdom.
raw_author_name: Amir Hussain
raw_affiliation_string: Block Upsample Block Conv Block Upsample Block Conv Block; Kaizhu Huang is with Duke Kunshan University, Suzhou, Kunshan, 215316, China.; University of Liverpool and School of Advanced Technology, Xi'an Jiaotong-Liverpool University.; is with School of Computing, Edinburgh Napier University, Edinburgh, EH11 4BN, the United Kingdom.

Abstract: Automatic nuclei segmentation and classification play a vital role in digital pathology.However, previous works are mostly built on data with limited diversity and small sizes, making the results questionable or misleading in actual downstream tasks.In this paper, we aim to build a reliable and robust method capable of dealing with data from the 'the clinical wild'.Specifically, we study and design a new method to simultaneously detect, segment, and classify nuclei from Haematoxylin and Eosin (H&E) stained histopathology data, and evaluate our approach using the recent largest dataset: PanNuke.We address the detection and classification of each nuclei as a novel semantic keypoint estimation problem to determine the center point of each nuclei.Next, the corresponding class-agnostic masks for nuclei center points are obtained using dynamic instance segmentation.Meanwhile, we proposed a novel Joint Pyramid Fusion Module (JPFM) to model the cross-scale dependencies, thus enhancing the local feature for better nuclei detection and classification.By decoupling two simultaneous challenging tasks and taking advantage of JPFM, our method can benefit from class-aware detection and class-agnostic segmentation, thus leading to a significant performance boost.We demonstrate the superior performance of our proposed approach for nuclei segmentation and classification across 19 different tissue types, delivering new benchmark results.Code available: https://github.com/Kaiseem/PointNu-Net