Exploring Forensic Dental Identification with Deep Learning

1 Introduction . Forensic identification targets to identify living or deceased persons by analyzing their trace evidences . The identification with dental data is particularly vital for the investigation of criminal scenes , accidents and mass disasters for at least two reasons [ 30 , 27 ] : ( i ) dental patterns can be highly identifying while their traces are widely archived than other methodologies e.g. , DNA profiling [ 4 , 45 , 59 ] . For instance , around 1.4 billion dental X-rays were performed in 2019 in the U.S.2 ; ( ii ) dental structures are more resistant to damages than other body tissues including bones [ 7 , 46 ] . In the current practice , forensic dental examinations are mostly done by the visual comparison of radiographs [ 46 , 45 ] between a target entity ( person ) and those from databases . Due to the lack of widely accepted automated solutions , such examination can not scale up to large databases , and its results are vulnerable to oversights and/or mistakes [ 19 , 39 ] . This work pioneers to study deep learning for forensic identification with dental panoramic radiographs — one of the most common dental traces nowadays [ 59 ] . ∗Corresponding authors 2https : //idataresearch.com/product/dental-imaging-market/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There have been a few attempts for forensic dental identification , most of which take the approaches of matching handcrafted feature [ 1 , 78 , 9 ] . According to our best knowledge , only one work utilizes off-the-shelf convolutional neural networks ( CNNs ) for learning discriminative representations of a dental radiograph for identification [ 39 ] . However , the work is preliminary since it is based on a small-scale dataset with both temporal and heterogeneous limitations — it is different from the applied situation where significant intra-entity variations in dental structures can exist because of external factors , e.g. , interventions , and internal factors , e.g. , decaying , over long scanning intervals [ 46 , 1 ] . Moreover , no architecture exploration was made for this unique identification task . We performed a formative study with two board-certified dentists , and outlined the distinct challenges of forensic dental identification : ( C1 ) the lack of paired dental radiographs for metric learning ; ( C2 ) sparseness of identification information in high-resolution radiographs ; and ( C3 ) dental variations can be heterogeneous but useful as inclusion or exclusion criteria for identifying . To study the task , we construct a comprehensive benchmark : it involves 583 persons , and comes with a challenging testing set spanning over 21.0±11.5 months of scanning intervals , covering a wide range of dental variations including orthodontics , tooth loss , and implanting/filling . Moreover , we propose the first optimized deep learning Forensic IDentification solution , named FoID . In specific , to take the advantage of overwhelmingly unpaired radiographs ( C1 ) , we develop a set of domain-specific augmentations ( DSA ) for effective self-supervised metric learning . To focus on meaningful areas , FoID incorporates hard attention extraction and alignment guided by semantic segmentation and anatomical registration ( C2 ) . Moreover , FoID introduces a novel transformer-based self-attention mechanism , as inspired by dentist ’ s practice , to dynamically reason the optimal aggregation of partial attentions for representation ( C3 ) . Overall , the main contributions of our work are three folds : • We give the first in-depth study of deep learning for dental forensic identification . The formative study with dentists outlines the unique challenges and offers insights into solutions . • We construct a comprehensive dental forensic benchmark , which covers a wide range of dental variations to reflect the realistic scenario of the task . • We present the first deep learning solution FoID , which achieves an mAP of 59.62 % , and largely outperforms the existing dental identification models and strong CNN baselines . Ablation studies show that the proposed DSA enables effective representation learning from unpaired data , boosting models by up to 12.33 % in terms of mAP ; Meanwhile , the proposed transformer-based self-attention outperforms state-of-the-art attention models by at least 4.82 % in terms of mAP . 2 Formative Study . Challenge 1 : paired scan instances are scarce With the high radiation of dental imaging , overwhelming cases from dental clinics contain only one scan instance per person . As such , it poses as a challenge for metric learning without positive instance pairs . To tackle it , we develop a set of domain-specific augmentations ( DSA ) by working with dentists to simulate potential dental anatomical variations . Different views of one entity are generated on the fly , which enables effective intra-/inter-reasoning in a self-supervised manner . Challenge 2 : identification information is sparse Forensic dentists mostly focus on several key anatomies on panoramic imaging for identification , since others are either non-discriminative or not well approximated on radiographs . As such , models should focus on such regions for image embedding to avoid over-fitting . Accordingly , as shown in Figure 1 , our method such incorporate domain knowledge by utilizing a hard attention mechanism on teeth , condyle ( Figure 1 ( a ) ) , angle of mandible ( Figure 1 ( b ) ) , and maxillary sinus ( Figure 1 ( c ) ) . A combination of semantic segmentation and registration is applied for efficient key-point localization . Challenge 3 : dental variations are heterogeneous but can be useful There can exist both local and global anatomical changes over two scanning . As demonstrated by a typical case in Figure 1 , two scans of a same entity can see anatomy loss ( Figure 1 ( d ) ) , artifact implanting ( Figure 1 ( e1 , e2 ) ) , structural changes ( Figure 1 ( f1 , f2 ) ) . In the forensic practice , dentists commonly look through all key anatomies in a scan , determine most discriminative features among them according to experience , and then apply those as indexes for matching . Our method follows the philosophy by introducing a transformer-based self-attention module , which dynamically reasons the relative importance of attentions among a set of them , for the effective representation learning of a radiograph . 3 Related Works . Person identification with deep learning Person identification tasks such as face recognition [ 56 , 52 , 43 , 47 , 8 ] and pedestrian re-identification [ 13 , 50 , 63 , 60 ] have been widely studied by the computer vision community . Current solutions either optimize a discriminant distance metric that takes a pair of instances [ 69 , 79 , 61 , 3 ] as input , or learn an identity-sensitive encoder that maps an input instance to representations [ 74 , 50 , 77 , 57 ] . Our method follows the later approach for its efficiency of inference within large-scale galleries . Attention mechanism has been widely explored for pedestrian re-identification to enforce models to focus on key body parts , such that representations are robust against background clutter , occlusion , and unconstrained human poses in surveillance images . For example , existing works [ 51 , 40 , 11 , 54 , 62 , 63 , 33 , 66 , 48 , 74 ] extract part-based features with either soft or hard attentions , guided by human pose estimations , and align/aggregate them for the final representation . Works [ 35 , 53 , 60 ] further propose to mine useful partial attentions from the end-to-end training . Our work follows the approach by introducing an attention mechanism , but differs in two ways . First , the key areas are partially localized with non-rigid image registration [ 5 ] , which effectively reduces the cost of manual annotations . Second , more importantly , we propose a novel self-attention module to dynamically reason the relative importance of each attention for the final representation by learning from training cases . Such self-attention is inspired by the practice of experts , where unique features of dental structures is searched , and used as matching criteria . Unsupervised person identification has also been recently discussed in pedestrian re-identification in order to train models with the widely available unlabeled surveillance videos . The current solutions can be categorized into clustering and fine-tuning [ 21 , 75 , 17 , 20 , 72 ] , negative instance pair mining [ 70 , 37 ] , and image translation [ 6 , 64 , 77 , 65 ] . However , the challenge of forensic dental identification is different from the above scenario : within a dental image database , each entity ( person ) has only one scan instance ( scan ) , and the method should learn representations from such set of instances . As such , we take the instance discriminative approach [ 10 , 44 , 21 ] , and generate different views of the same entity with the proposed domain-specific augmentations ( DSA ) for the contrastive intra-reasoning . Transformer for image retrieval Transformer has been recently applied to computer vision tasks [ 15 , 71 , 73 ] . Since our task is image retrieval in nature , we briefly review the related transformer works : Gkelios et al . [ 22 ] and El-Nouby et al . [ 16 ] directly replace CNNs with pure transformer models for image embedding ; He et al . [ 25 ] further incorporates encoding of meta information , ( e.g. , camera viewpoints ) , into the sequence of inputs ; Yang et al . [ 67 ] utilizes query and gallery images as transformer ’ s query and key inputs respectively for modeling spatial attention between them . Different from the above works , we propose to utilize a transformer for aggregating a set of model attentions : the self-attention characteristic of transformer enables the dynamic reasoning of attentions ’ importance for embedding by comparison , and its permutation-invariance [ 32 ] can relieve the attention misalignment caused by the imperfect key-point localization . 4 Methodologies . We present FoID by first describing its building components and then the multi-task learning strategy . As shown in Figure 2 , FoID maps each scan instance into an embedding . Given a query instance , we calculate the similarities between the query and all the gallery instances in the embedding space and return the ranked retrieval list . 4.1 Attention Localization & Alignment . Inspired by the dentists ’ practice , FoID employs a hard attention mechanism to focus on ( i ) teeth and ( ii ) landmarks of condyle , angle of mandible , and maxillary sinus to avoid over-fit to unrelated regions . For teeth localization , FoID formulates it as semantic segmentation , and applies a mask-RCNN [ 23 ] pre-trained on the existing panoramic segmentation dataset of UFBA_UESC [ 29 ] . Given an input instance I , a teeth segmentation map TH×W×32 is estimated ( Figure 2 ( a ) ) , where H and W are instance height and width , and a channel size is 32 for the number of teeth categories of an adult . The raw mask-RCNN outputs are filtered for the largest connected mask with a minimal size threshold for each tooth to denoise . For landmark localization , since the landmarks are largely separated and establish no accurate boundary , FoID employs a registration-based method to efficiently detect them from an atlas , with one pair of radiograph and its manual annotation with M=6 landmarks manually created ( Figure 2 ( b ) ) . For each input instance , the atlas is aligned to the instance with Syn registration [ 5 ] , with the annotation deformed accordingly for the landmark estimation ( Figure 2 ( c ) ) . Based on the teeth and landmark localization , the input is cropped to attention patches with a constant spatial dimension of h×w , and then sequentially piled as the attention stack Sh×w×38 for the representation representation . In the case of missing anatomies , zero-valued patches are correspondingly padded . Appendix includes detailed results on teeth segmentation , landmark localization , and the visualization of the annotated atlas .

Summary: The authors propose a framework for retrieval of dental panoramic X-Rays.  To find regions of interest they focus on teeth which are extracted by semantic segmentation using a pretrained mask-RCNN as well as relevant areas extracted based on landmarks for which it is necessary to register every image to an atlas in a deformable way. All those ROIs are extracted and augmented in a domain-specific way (fillings, tooth extraction, etc), which serves as input to a CNN which is trained with an objective function that consists of three loss terms, added in an unweighted manner: a contrastive loss, an ID loss and a deep supervision loss.  They use a publicly available dataset for training, but test the proposed method on data collected by the authors, s.t. each image in the test set is to be retrieved from the union of the test and training set.  An ablation study on different modules of the framework is performed to test their contributions to the overall improvement of the proposed method.