Exploring Forensic Dental Identification with Deep Learning

1 Introduction . Forensic identification targets to identify living or deceased persons by analyzing their trace evidences . The identification with dental data is particularly vital for the investigation of criminal scenes , accidents and mass disasters for at least two reasons [ 30 , 27 ] : ( i ) dental patterns can be highly identifying while their traces are widely archived than other methodologies e.g. , DNA profiling [ 4 , 45 , 59 ] . For instance , around 1.4 billion dental X-rays were performed in 2019 in the U.S.2 ; ( ii ) dental structures are more resistant to damages than other body tissues including bones [ 7 , 46 ] . In the current practice , forensic dental examinations are mostly done by the visual comparison of radiographs [ 46 , 45 ] between a target entity ( person ) and those from databases . Due to the lack of widely accepted automated solutions , such examination can not scale up to large databases , and its results are vulnerable to oversights and/or mistakes [ 19 , 39 ] . This work pioneers to study deep learning for forensic identification with dental panoramic radiographs — one of the most common dental traces nowadays [ 59 ] . ∗Corresponding authors 2https : //idataresearch.com/product/dental-imaging-market/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There have been a few attempts for forensic dental identification , most of which take the approaches of matching handcrafted feature [ 1 , 78 , 9 ] . According to our best knowledge , only one work utilizes off-the-shelf convolutional neural networks ( CNNs ) for learning discriminative representations of a dental radiograph for identification [ 39 ] . However , the work is preliminary since it is based on a small-scale dataset with both temporal and heterogeneous limitations — it is different from the applied situation where significant intra-entity variations in dental structures can exist because of external factors , e.g. , interventions , and internal factors , e.g. , decaying , over long scanning intervals [ 46 , 1 ] . Moreover , no architecture exploration was made for this unique identification task . We performed a formative study with two board-certified dentists , and outlined the distinct challenges of forensic dental identification : ( C1 ) the lack of paired dental radiographs for metric learning ; ( C2 ) sparseness of identification information in high-resolution radiographs ; and ( C3 ) dental variations can be heterogeneous but useful as inclusion or exclusion criteria for identifying . To study the task , we construct a comprehensive benchmark : it involves 583 persons , and comes with a challenging testing set spanning over 21.0±11.5 months of scanning intervals , covering a wide range of dental variations including orthodontics , tooth loss , and implanting/filling . Moreover , we propose the first optimized deep learning Forensic IDentification solution , named FoID . In specific , to take the advantage of overwhelmingly unpaired radiographs ( C1 ) , we develop a set of domain-specific augmentations ( DSA ) for effective self-supervised metric learning . To focus on meaningful areas , FoID incorporates hard attention extraction and alignment guided by semantic segmentation and anatomical registration ( C2 ) . Moreover , FoID introduces a novel transformer-based self-attention mechanism , as inspired by dentist ’ s practice , to dynamically reason the optimal aggregation of partial attentions for representation ( C3 ) . Overall , the main contributions of our work are three folds : • We give the first in-depth study of deep learning for dental forensic identification . The formative study with dentists outlines the unique challenges and offers insights into solutions . • We construct a comprehensive dental forensic benchmark , which covers a wide range of dental variations to reflect the realistic scenario of the task . • We present the first deep learning solution FoID , which achieves an mAP of 59.62 % , and largely outperforms the existing dental identification models and strong CNN baselines . Ablation studies show that the proposed DSA enables effective representation learning from unpaired data , boosting models by up to 12.33 % in terms of mAP ; Meanwhile , the proposed transformer-based self-attention outperforms state-of-the-art attention models by at least 4.82 % in terms of mAP . 2 Formative Study . Challenge 1 : paired scan instances are scarce With the high radiation of dental imaging , overwhelming cases from dental clinics contain only one scan instance per person . As such , it poses as a challenge for metric learning without positive instance pairs . To tackle it , we develop a set of domain-specific augmentations ( DSA ) by working with dentists to simulate potential dental anatomical variations . Different views of one entity are generated on the fly , which enables effective intra-/inter-reasoning in a self-supervised manner . Challenge 2 : identification information is sparse Forensic dentists mostly focus on several key anatomies on panoramic imaging for identification , since others are either non-discriminative or not well approximated on radiographs . As such , models should focus on such regions for image embedding to avoid over-fitting . Accordingly , as shown in Figure 1 , our method such incorporate domain knowledge by utilizing a hard attention mechanism on teeth , condyle ( Figure 1 ( a ) ) , angle of mandible ( Figure 1 ( b ) ) , and maxillary sinus ( Figure 1 ( c ) ) . A combination of semantic segmentation and registration is applied for efficient key-point localization . Challenge 3 : dental variations are heterogeneous but can be useful There can exist both local and global anatomical changes over two scanning . As demonstrated by a typical case in Figure 1 , two scans of a same entity can see anatomy loss ( Figure 1 ( d ) ) , artifact implanting ( Figure 1 ( e1 , e2 ) ) , structural changes ( Figure 1 ( f1 , f2 ) ) . In the forensic practice , dentists commonly look through all key anatomies in a scan , determine most discriminative features among them according to experience , and then apply those as indexes for matching . Our method follows the philosophy by introducing a transformer-based self-attention module , which dynamically reasons the relative importance of attentions among a set of them , for the effective representation learning of a radiograph . 3 Related Works . Person identification with deep learning Person identification tasks such as face recognition [ 56 , 52 , 43 , 47 , 8 ] and pedestrian re-identification [ 13 , 50 , 63 , 60 ] have been widely studied by the computer vision community . Current solutions either optimize a discriminant distance metric that takes a pair of instances [ 69 , 79 , 61 , 3 ] as input , or learn an identity-sensitive encoder that maps an input instance to representations [ 74 , 50 , 77 , 57 ] . Our method follows the later approach for its efficiency of inference within large-scale galleries . Attention mechanism has been widely explored for pedestrian re-identification to enforce models to focus on key body parts , such that representations are robust against background clutter , occlusion , and unconstrained human poses in surveillance images . For example , existing works [ 51 , 40 , 11 , 54 , 62 , 63 , 33 , 66 , 48 , 74 ] extract part-based features with either soft or hard attentions , guided by human pose estimations , and align/aggregate them for the final representation . Works [ 35 , 53 , 60 ] further propose to mine useful partial attentions from the end-to-end training . Our work follows the approach by introducing an attention mechanism , but differs in two ways . First , the key areas are partially localized with non-rigid image registration [ 5 ] , which effectively reduces the cost of manual annotations . Second , more importantly , we propose a novel self-attention module to dynamically reason the relative importance of each attention for the final representation by learning from training cases . Such self-attention is inspired by the practice of experts , where unique features of dental structures is searched , and used as matching criteria . Unsupervised person identification has also been recently discussed in pedestrian re-identification in order to train models with the widely available unlabeled surveillance videos . The current solutions can be categorized into clustering and fine-tuning [ 21 , 75 , 17 , 20 , 72 ] , negative instance pair mining [ 70 , 37 ] , and image translation [ 6 , 64 , 77 , 65 ] . However , the challenge of forensic dental identification is different from the above scenario : within a dental image database , each entity ( person ) has only one scan instance ( scan ) , and the method should learn representations from such set of instances . As such , we take the instance discriminative approach [ 10 , 44 , 21 ] , and generate different views of the same entity with the proposed domain-specific augmentations ( DSA ) for the contrastive intra-reasoning . Transformer for image retrieval Transformer has been recently applied to computer vision tasks [ 15 , 71 , 73 ] . Since our task is image retrieval in nature , we briefly review the related transformer works : Gkelios et al . [ 22 ] and El-Nouby et al . [ 16 ] directly replace CNNs with pure transformer models for image embedding ; He et al . [ 25 ] further incorporates encoding of meta information , ( e.g. , camera viewpoints ) , into the sequence of inputs ; Yang et al . [ 67 ] utilizes query and gallery images as transformer ’ s query and key inputs respectively for modeling spatial attention between them . Different from the above works , we propose to utilize a transformer for aggregating a set of model attentions : the self-attention characteristic of transformer enables the dynamic reasoning of attentions ’ importance for embedding by comparison , and its permutation-invariance [ 32 ] can relieve the attention misalignment caused by the imperfect key-point localization . 4 Methodologies . We present FoID by first describing its building components and then the multi-task learning strategy . As shown in Figure 2 , FoID maps each scan instance into an embedding . Given a query instance , we calculate the similarities between the query and all the gallery instances in the embedding space and return the ranked retrieval list . 4.1 Attention Localization & Alignment . Inspired by the dentists ’ practice , FoID employs a hard attention mechanism to focus on ( i ) teeth and ( ii ) landmarks of condyle , angle of mandible , and maxillary sinus to avoid over-fit to unrelated regions . For teeth localization , FoID formulates it as semantic segmentation , and applies a mask-RCNN [ 23 ] pre-trained on the existing panoramic segmentation dataset of UFBA_UESC [ 29 ] . Given an input instance I , a teeth segmentation map TH×W×32 is estimated ( Figure 2 ( a ) ) , where H and W are instance height and width , and a channel size is 32 for the number of teeth categories of an adult . The raw mask-RCNN outputs are filtered for the largest connected mask with a minimal size threshold for each tooth to denoise . For landmark localization , since the landmarks are largely separated and establish no accurate boundary , FoID employs a registration-based method to efficiently detect them from an atlas , with one pair of radiograph and its manual annotation with M=6 landmarks manually created ( Figure 2 ( b ) ) . For each input instance , the atlas is aligned to the instance with Syn registration [ 5 ] , with the annotation deformed accordingly for the landmark estimation ( Figure 2 ( c ) ) . Based on the teeth and landmark localization , the input is cropped to attention patches with a constant spatial dimension of h×w , and then sequentially piled as the attention stack Sh×w×38 for the representation representation . In the case of missing anatomies , zero-valued patches are correspondingly padded . Appendix includes detailed results on teeth segmentation , landmark localization , and the visualization of the annotated atlas .

Summary: The authors propose a new architecture FoID for forensic identification of individuals from dental imagery, a task often conducted by domain experts for identification at crime scenes/disaster sites. Given an image and a tooth segmentation map, they augment a straightforward CNN in four fashions: (i) hard attention on the regions of the scan where identifying information can be found, (ii) dental-specific augmentations (e.g. addition of a cavity) as opposed to general augmentations, (iii) the inclusion of three loss families (classification, contrastive, identification), and (iv) a transformer-based attention mechanism for focusing on unusual shapes of interest. In experiments, they outperform a series of benchmarks,  and ablations show the importance of each of the augmentations above. They include interpretations of the self-attention mechanism for explainability.