Exploring Forensic Dental Identification with Deep Learning
Authors: Yuan Liang
Weikun Han
Liang Qiu
Chen Wu
Yiting Shao
Kun Wang
Lei He
Venue: NeurIPS 2021 Poster (2021)
Keywords: person identification
medical imaging
forensics

Abstract: Dental forensic identification targets to identify persons with dental traces.
The task is vital for the investigation of criminal scenes and mass disasters because of the resistance of dental structures and the wide-existence of dental imaging. 
However, no widely accepted automated solution is available for this labour-costly task. 
In this work, we pioneer to study deep learning for dental forensic identification based on panoramic radiographs. 
We construct a comprehensive benchmark with various dental variations that can adequately reflect the difficulties of the task. 
By considering the task's unique challenges, we propose FoID, a deep learning method featured by: (\textit{i}) clinical-inspired attention localization, (\textit{ii}) domain-specific augmentations that enable instance discriminative learning, and (\textit{iii}) transformer-based self-attention mechanism that dynamically reasons the relative importance of attentions. 
We show that FoID can outperform traditional approaches by at least \textbf{22.98\%} in terms of Rank-1 accuracy, and outperform strong CNN baselines by at least \textbf{10.50\%} in terms of mean Average Precision (mAP). 
Moreover, extensive ablation studies verify the effectiveness of each building blocks of FoID. 
Our work can be a first step towards the automated system for forensic identification among large-scale multi-site databases. 
Also, the proposed techniques, \textit{e.g.}, self-attention mechanism, can also be meaningful for other identification tasks, \textit{e.g.}, pedestrian re-identification.
Related data and codes can be found at \href{https://github.com/liangyuandg/FoID}{https://github.com/liangyuandg/FoID}. 

Exploring Forensic Dental Identification with Deep Learning
Yuan Liang!?, Weikun Han', Liang Qiu', Chen Wu', Yiting Shao', Kun Wang', Lei He!"
University of California, Los Angeles
Topaz Labs
Liangyuandgeucla.edd
Abstract
Dental forensic identification targets to identify persons with dental traces. The task is vital for the investigation of criminal scenes and mass disasters because the resistance of dental structures and the wide-existence of dental imaging. However, widely accepted automated solution is available for this labour-costly task. In this work, we pioneer to study deep learning for dental forensic identifica- tion based on panoramic radiographs. We construct comprchensive benchmark with various dental variations that can adequately reflect the difficulties of the task. By considering the task's unique challenges, we propose FolD, deep learning method featured by: (i) clinical-inspired attention localization, (ii) domain-specific augmentations that enable instance discriminative learning, and (iii) transformer- based self-attention mechanism that dynamically reasons the relative importance of attentions. We show that FolD can outperform traditional approaches by least 22.98% in terms of Rank-1 accuracy, and outperform strong CNN baselines by least 10.50% in terms of mean Average Precision (mAP). Moreover, ex- tensive ablation studies verify the effectiveness of each building blocks of FolD. Our work can be first step towards the automated system for forensic identi- fication among large-scale multi-site databases. Also, the proposed techniques, e.g., self-attention mechanism, can also be meaningful for other identification tasks, e.g., pedestrian re-identification. Related data and codes can be found at httpss//gghub..co//mnngguummgg
Introduction
Forensic identification targets identify living or deceased persons by analyzing their trace evidences. The identification with dental data is particularly vital for the investigation of criminal scenes,
accidents and mass disasters for at least two reasons [30] 27]: (i) dental patterns can be highly identifying while their traces are widely archived than other methodologies e.g., DNA profiling For instance, around 1.4 billion dental X-rays were performed in 2019 in the U.S (ii) dental structures are more resistant to damages than other body tissues including bones 7 46] In the current practice, forensic dental examinations are mostly done by the visual comparison of radiographs [46] 45] between target entity (person) and those from databases. Due to the lack of widely accepted automated solutions, such examination cannot scale up to large databases, and its results are vulnerable to oversights and/or mistakes [19] (9). This work pioneers to study deep learning for forensic identification with dental panoramic radiographs - one of the most common dental traces nowadays [59].
"Corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: challenging case of dental identification with two dental radiographs captured from a same person. The case exists large anatomical variations including tooth reduction (d), the addition of braces and implants (el, e2), and the change of teeth alignment (fl, f2). the forensic practice, teeth, condyle (a), angle of mandible (b), and maxillary sinus (c) are the key anatomies for dentists.
There have been few attempts for forensic dental identification, most of which take the approaches of matching handcrafted feature [1 78 9 According to our best knowledge, only one work utilizes off-the-shelf convolutional neural networks (CNNs) for learning discriminative representations of a dental radiograph for identification [39]. However, the work is preliminary since it is based on a small-scale dataset with both temporal and heterogeneous limitations it is different from the applied situation where significant intra-entity variations in dental structures can exist because of external factors, e.g., interventions, and internal factors, e.g., decaying, over long scanning intervals [46 [. Moreover, architecture exploration was made for this unique identification task.
We performed formative study with board-certified dentists, and outlined the distinet challenges of forensic dental identification: (CI) the lack of paired dental radiographs for metric learning: (C2) sparseness identification information in high-resolution radiographs; and (C3) dental variations can be heterogeneous but useful as inclusion or exclusion criteria for identifying, To study the task, we construct comprehensive benchmark: it involves 583 persons, and comes with challenging testing set spanning over 21 11.5 months of scanning intervals, covering a wide range of dental variations including orthodontics, tooth loss, and implanting/iiling. Moreover, we propose the first optimized deep learning Forensic IDentification solution, named FolD. In specific, to take the advantage of overwhelmingly unpaired radiographs (CT), we develop a set of domain-specific augmentations (DSA) for effective self- self-supervised metric learning. To focus on meaningful areas, FolD incorporates hard attention extraction and alignment guided by semantic segmentation and anatomical registration (C2). Moreover, FolD introduces a novel transformer-ban based self-attention mechanism, as inspired by dentist's practice, to dynamically reason the optimal aggregation of partial attentions for representation (C3). Overall, the main contributions of our work are three folds:
We give the first in-depth study of deep learning for dental forensic identification. The formative study with dentists outlines the unique challenges and offers insights into solutions.
We construct a comprehensive dental forensic benchmark, which covers wide range of dental variations to reflect the realistic scenario of the task.
We present the first deep learning solution FolD, which achieves an mAP of 59.62%, and largely outperforms the existing dental identification models and strong CNN baselines. Ablation studies show that the proposed DSA enables effective representation learning from unpaired data, boosting models by up 12.33% in terms of mAP; Mcanwhile, the proposed transformer-baat based self-attention outperforms state-of-the- attention models by at least 4.82% in terms of mAP.
Formative Study
Challenge paired sean instances are scarce With the high radiation of dental imaging, over- whelming cases from dental clinics contain only one scan instance per person. As such, it poses as a challenge for metric learning without positive instance pairs. To tackle it, we develop a set of domain-specific augmentations (DSA) by working with dentists to simulate potential dental

---

anatomical variations. Different views of one entity are generated on the fly, which enables effective intra-/inter-reasonnng in self-supervised manner.
Challenge 2: identification information is sparse Forensic dentists mostly focus several key
anatomies on panoramic imaging for identification, since others are either non-discriminative or not well approximated on radiographs. As such, models should focus such regions for image embedding to avoid over-fitting. Accordingly, as shown in Figure I our method such incorporate domain knowledge by utilizing a hard attention mechanism on teeth, condyle (Figure [ angle of mandible (Figure [b)),, and maxillary sinus (Figure [()). A combination of semantic segmentation and registration is applied for efficient key-point localization.
Challenge 3: dental variations are heterogeneous but can be useful There can exist both local and global anatomical changes over two scanning. As demonstrated by a typical case in Figure T
two scans of a same entity can see anatomy loss (Figure[ artifact implanting F eel, e2)),
structural changes [Figure[[ft f2)). In the forensic practice, dentists commonly look through all key
anatomies in scan, determine most discriminative features among them according to experience, and then apply those as indexes for matching. Our method follows the philosophy by introducing transformer-basee self-attention module, which dynamically reasons the relative importance of attentions among a set of them, for the effective representation learning of a radiograph.
3 Related Works
Person identification with deep learning Person identification tasks such as face recogmition
63 60 have been widely studied by the computer vision community, Current solutions either optimize a discriminant distance metric that takes a pair of instances [69 79
56
52 43
47 g
and pedestrian re-identification 13 50
61
3 as input, or learn an
sensitive encoder that maps 50 77 (77. Our method follows the later approach for its
an input instance to representations 77]
efficiency of inference within large-scale galleries.
Attention mechanism has been widely explored for pedestrian re- re-identification to enforce models to focus on key body parts, such that representations are robust against background clutter, occlusion. and unconstrained human poses in surveillance images. For example, existing works [51] 40 [1
54
extract part-based features with either soft or hard attentions, guided by human pose estimations, and align/aggregate them for the final representation. Works [35 53 60
62
48 74
further propose to mine useful partial attentions from the end-to-end training. Our work follows the approach by introducing an attention mechanism, but differs in two ways. First, the key areas are partially localized with non-rigid image registration (5), which effectively reduces the cost of manual annotations. Second, more importantly, we propose novel self-attention module to dynamically reason the relative importance of each attention for the final representation by learning from training cases. Such self- attennteeoonn is inspired by the practice of experts, where unique features of dental structures is searched, and used as matching criteria.
Unsupervised person identification has also been recently discussed in pedestrian re- re-identification in order to train models with the widely available unlabeled surveillance videos. The current solutions can be categorized into clustering and fine-tuning [1 75 17
20 72), negative instance pair mining However, the challenge of forensic dental identification different from the above scenario: within dental image database, each entity (person) has only one scan instance (scan), and the method should learn representations from such set of instances. As such, we take the instance discriminative approach [10 44 211, and generate different views of the same entity with the proposed domain-specific augmentations (DSA) for the contrastive intra-reasoning.
70 37
and image translation (6 64 77 65)
Transformer for image retrieval Transformer has been recently applied computer vision tasks 15] 71 73 Since our task is image retrieval in nature, we briefly review the related transformer works: Gkelios et al. [22 and El-Nouby et al. [16] directly replace CNNs with pure transformer models for image embedding; He et al. [23] further incorporates encoding of meta information, (e.g., camera viewpoints), into the sequence of inputs; Yang el al. (67] utilizes query and gallery images as transformer's query and key inputs respectively for modeling spatial attention between them. Different from the above works, we propose to utilize transformer for aggregating a set of model attentions: the self-attention characteristic of transformer enables the dynamic reasoning of
3

---

Te
Attention Embeddings
Embedding
[clase]
Regressor 7id
ID Loss
Sogmentation
Stochastic Domain- Specifii Augmentions
CNN Encoder
Transformer Encoder
Non- Linesr Projectiion g
Linear Projection
Oijective Loss
Inoul Inslance
Representation
Alles with Landmarks
Deforment
Artertion Stack
Augmanted Attontion Statks
Deep Sup. Regressor T Zump)
Doep Sup.
Deformed Allas with Landmarks
Loss
Figure 2: Overall architecture of FolD, the key components of which consist of attention localization and alignment, domain-specific augmentations, and feature aggregation with self-attention The objective function includes three types of losses that are complementary for representation learning.
attentions' importance for embedding by comparison, and its permutation-invariane the attention misalignment caused by the imperfect key-point localization.
322 can relieve
4 Methodologies
We present FolD by first describing its building components and then the multi-task learning strategy. As shown in Figure FolD maps each sean instance into an embedding. Given a query instance, we calculate the similarities between the query and all the gallery instances in the embedding space and return the ranked retrieval list.
4.1 Attention Localization & Alignment
Inspired by the dentists' practice, FolD employs a hard attention mechanism to focus on (i) teeth and (ii) landmarks of condyle, angle mandible, and maxillary sinus to avoid over-fit unrelated regions. For teeth localization, FolD formulates it as semantic segmentation, and applies a mask-RCNT (23) pre-trained on the existing panoramic segmentation dataset of UFBA UESC [29]. Given an input instance I, teeth segmentation map H W x32 is estimated (Figure aa)),, where H and W are instance height and width, and channel size is 32 for the number of teeth categories an adult. The raw mask -CNN outputs filtered for the largest connected mask with minimal size threshold for each tooth to denoise. For landmark localization, since the landmarks are largely separated and establish no accurate boundary, FolD employs registration-based method to efficiently detect them from an atlas, with one pair of radiograph and its manual annotation with M=6 landmarks manually created (Figure[]b)) For each input instance, the atlas is aligned the instance with Syn registration [5). with the annotation deformed accordingly for the landmark estimation (Figure ()) Based on the teeth and landmark localization, the input is cropped to attention patches with constant spatial dimension of h x w. and then sequentially piled as the attention stack x for the representation representation. In the case of missing anatomies, vero-valued patches are correspondingly padded. Appendix includes detailed results on teeth segmentation, landmark localization, and the visualization of the annotated atlas.
4.2 Contrastive Learning with DSA
In order to build useful representations from overwhelming unpaired instances, FolD adapts self- supervised approach with the proposed domain augmentations (DSA). In specific, a set of structure-aware and roransic-realistic augmentations is stochastically applied to an instance to generate different views, which are in turn used for intra-reasoning, In the practice, given mini-batch of stacked attentions from entities B = Hi times DSA are repeatedly applied to
B. Denote sl(() as the i-th sample from augmented view of Sn an instance group ( s!/l) is

---

taken as positive pair, while 1 is taken as negative pair, where /n can be any index from 2I.,.... (1.2...,NV7Aff) (n)... The learning strategy does not relay on any paired instances or identification annotation, but can be easily extended when such data is available by including different augmented inputs belonging to the same identificatinn as positive pairs.
To achieve effective representation learning, augmentations that cover the potential anatomical variations as in clinics is the key. As showcased in Figure 2 the proposed DSA consists fours types of augmentations inspired by forensic observations, and is independently applied to each attention patch on the fly. (i) Random tooth reduction that aims to simulate tooth loss caused by injury or inventions (e.g.. Figure (e)). (ii) Random artifact addition that adds patches of flexible shapes to crown (Figure fe..)) or tooth regions (Figure 2 g)). Such augmentation aims to simulate the common artifacts of dental filling, implant, and brace. (iii) Random rigid patch transform within range of angles and displacements. The augmentation not only enforces view comparison as in the standard instance discrimination learning [10 44], but also simulates global teeth arrangement changes (e.g,, caused by orthodontics), since the transform performed for patches independently. (iv) Random color disturbance of contrast shifting and gaussian noise within designed ranges for simulating different scanning setup and machine noises. More details about the implementation of DSA are included in Appendix.
4.3
Sell-Attention for Feature Aggregation
To build the representation of an instance from its attention stack, FolD consists a CNN encoder f (Figure 2)) embed each attention patch, followed by a transformer-basee based self-attention module (Figure 20) for aggregating the attention embeddings. The encoder targets to extract salient features for an input patch with the existence of possible anatomical variations, and it can be adapted from any convolutional encoders. The output attention embeddings from the CNN encoder can be denoted as 238xC, where C' is the embedding length.
To aggregate the generated multiple embeddings of attentions, we propose to utilize a transformer to dynamically reason the importance of each attention among the set of them: dental structures that fall into the general distribution of populations should take less attentions, while the ones that are more identifying should take more attentions. More importantly, such importance of attentions should be relative based on the experience of forensic dentists: the more discriminating anatomies in an instance are more often selected as index for matching. This differs from the existing pedestrian re- identification works [1 33 74, where the attention to a body parts is modeled without conditioning on its comparison among all body parts. As shown in Figure 2 the sequence of attentions 238xC laf: 2 .... 2asl is first projected into a constant token dimension C' with linear encoder a, and then is fed into a two-layer BERT encoders (58 14] for unidirectional context learning. Note that
2C1
[15] is injected to attention embeddings for two reasons: (i) we assume that the semantic category of anatomies sets no prior for reasoning the importance, (ii) teeth can be often cis-classified due to their similarity in patterns (see Appendix for examples), which can in turn pose as noise when positions are modeled. The feature outputs from the last transformer layer are further encoded into reduced dimension C" as regularization with non-linear encoder B, and then concatenated as the final embedding Y cat(yf" . ..... Vas for the instance. Moreover, a learnable [class] token 25" is prepended to the transformer's 's input sequence, the embedding of which at the output y6 serves as a complementary instance embedding for the purpose of multi-task learning.
no positional information
4.4 Objective Function & Training
To learn discriminative representations, we adopt three complementary optimization tasks to train FolD; contrastive loss, an identification loss, and a deep supervision loss.
Contrastive loss Given a pair of instance embeddings {Y,, Y,J. the contrastive loss Loontrast is defined as: Leontrast = 11,=,, [Mpos - SY, Y,)I+ L1, (Y,, Y,) - mnngll+ where [a]+ equals mar(x, 0), and l; denotes the instance's s id category of representation (Y,) among the training set; Myos and mneg are similarity margins for positive and negative pairs; S(...) is the pre-defined similarity measure, and cosine distance is used this work. Online mining is applied to train on challenging positive (views of one instance) and negative (views of different instances) pairs.
5

---

Table 1: Demographic and dental variation information of the testing set. Scan interval is the average of largest time interval between any two radiographs of an entity. The numbers of entity presenting the dental variations and their proportions are shown.
Demographic
Variations
Age
Gender
Scan Interval
Orthodomiics
FillingImmplan
Brace
Tooth Loss
10~55
F:23: M:17
21.0=11,, 11.5 mo. 25 (62.5%)
10 (25.0%)
16 (40.0%
18 (45.0%
Identification loss An identification loss Lia is applied to all samples within mini-batch for the prediction of ID category from the transformer's [class] embedding output. specific, Lia- - C'E(so (ffazfrtaaar )), 1), where CE denotes cross entropy loss; D6 and and I are the sample's transformer's [class] output and id category label, respectively; rid regressor that transfer the embedding dimension from c' to the total training id count. The identification loss and contrastive loss are applied on different embeddings and have diverged objective, which can potentially serve as regularization for the transformer.
Deep supervision loss native for identification from given attention patch. Since the target diverges from the contrastive loss and identification loss that directly optimize on the aggregated embeddings, we incorporates a deep supervision signal for f. In specific, we formulate the deep supervision loss Tepup as a cross entropy loss: sup f (faup . ..... ()))) (). where >ssup is regressor that maps the concatenation of a sample's attention embeddings to vector, the dimension of which equals the total count of training instances.
The CNN encoder / is expected to extract salient features that are discrimi-
The final objective function for training FolD is the unweighted sum of the three types of loss: L Leontrastive Lid + Leup. During the training, the whole model can be optimized from end-to-end, while the transformer's non-class embeddings cat(y) 8/2 ..... y(=")) taken as the representation of an instance during the inference.
5 Benchmark Construction
Similar to the setup of existing image retrieval datasets 34 training set and testing set. The training set is derived from an existing DNS Panoramic dataset (49). which contains 543 panoramic radiographs and exist common dental variations of tooth reduction and artifact addition. Each of the radiographs is scanned from separate identity, which resembles the practical situation of dental forensic where there is a lack of paired data for metric learning. There are 271 images that overlap between this training set and the one used for teeth localization as aforementioned in Section 4.1 This setup simulates the potential forensic application: a subset of available data in the imaging repository can possibly be labeled for developing the teeth segmentation model. Moreover, the testing data of identification is unseen to the training of segmentation model, and thus would not artificially inflates the performance during the test.
76 36
our benchmark consists of a
The testing set contains 87 panoramic radiographs scanned with 40 identities, among which 33 identities hold 2 radiographs and 7 identities hold 3 radiographs. For covering comprehensive dental variations to make it sufficient for reflecting the difficulty of dental forensic, the testing set are selected from both endodontics and orthodontics departments. The method is distinct from previous dental forensic studies that either construct testing cases without controlling data variety [I8] I 26), or consider only a specific dental variation over short scanning interval (i.e, single tooth extraction over 64.0 days (39)). Table desscribs the profile of the testing set, and Figure shows one typical case. All the data has been anonymized. To evaluate a model, we apply an one-againtt-all strategy for the efficient use of data. In specific, one instance from the testing set is used as the query, while the union of training set and the rest of testing set is used as the gallery set. The average results among all testing instances is reported as the final accuracy measurement.

---

Table 2: Comparison to the existing dental forensic identification approaches and strong CNN baselines. AHI 26 is not designed to rank all gallery instances, and thus mAP cannot be measured.
Methods
Backbone
mAP
Rank-1
Rank-5
Rank- 10
Alll (26
0,2759
0,4483
0,5287
HIDPR 42
0,3018
0,2759
0,4138
0,4713
classification
Re
0,4912
0.4253
0.6437
0.7356
contrastive
ResNet-33
0,4072
0.3218
0,5977
0.6322
triplet
ResNet-34
0.4723
0.4253
0,6092
0.6667
reconsiruction
ResNet-34
0.3537
0.3333
0.4943
0.6092
classification
Inception-ResNe ResNett
0.3638
0,2644
0.5287
0.5862
contrustive
Inception-Rese ResNet
0,3854
0,3448
0,4943
0.6092
triplct
Inception-ResNe Restet
0,3609
0,2989
0,5172
0.6437
reconstruction
Inception-ResNc
0.2912
0.2299
0,5172
0.5862
FolD (ours)
ResNet-34
0,5465
0.4817
0.6897
0.7471
FoID (ours)
Inception-ResNel
0.5962
0.5057
0.7586
0.7816
6 Experimental Results
6.1 Implementation Details
We adopt two mostly used CNN architectures in re-identification works, ResNet-34 x and Inception- ResNet [55], for the CNN encoder f. Both models drop the final fully connected layer, and are initialized by pre-training on the ImageNet [12] and the face recognition dataset LFW by following 47] respectively. The output embedding of both encoders has 512 channels per input attention patch. By following [15, the single layer linear encoder a maps an attention emhedding to 128 channels; The transformer holds a hidden embedding dimension of 128 and multi-attention head number of 4; Meanwhile, the final encoder (3 consists a fully connected layer and batch normalization, mapping the transform's 's outputs to 13 channels per attention patch, and thus leading to dimension of 494 for the final instance embedding. The deep supervision regressor and [class] embedding regressor are both implemented with two fully connected layers, the output feature dimensions of which are 256 and the total count training instances respectively. Batch normalization and ReLU activation are applied in both regressors.
Regarding data pre-proese pre-processing, all the radiography instances are normalized to a pixel spacing of 0m1 x 0. 11mm, and are normalized to a same color space with histogram matching. The attention patch dimensions h and w are set as the minimal embodying size for all anatomies, which are 28. 16mm and 21 1.56mm respectively. Unless otherwise specified, the batch size is set to 6 and the augmentation repeat is set to 5 per instance for all the training. Unless otherwise specified, Adam optimizer is employed with an initial learning rate of le-4, and is reduced to a half every 30 epochs. All the methods are trained for 100 epochs. For the online hard pair mining, the positive and negative similarity margins are set to 0.6 and 0.4 respectively. All experiments are conducted three GeForce GTX 1080 Ti GPUs Cosine similarity is applied to rank the retrievals given a query instance. Rank-k k retrieval accuracy (k 5, 10)) and mean average precision (mAP) are used as evaluation metrics.
servers with
6.2 Comparison to Existing Approaches
We compare our method both: (i) key-point matching based handcrafted descriptors, and (ii) strong CNN baselines. For the key-point matching, it is the most studied approach for forensic dental identification. Two recent methods of AHI (26) and HIDPR [42 are included in comparison, which are based on the matching of Speeded Up Robust Features and tooth-aware appearance and positional similarity, respectively, Both methods have achieved satisfying identification accuracies with panoramic radiographs based on their own in-house datasets. For the CNN approach, few study has investigated it possibly due the challenge of learning from the overwhelming unpaired data. For fair comparison, we enable the proposed domain-speciff augmentations for all the training, and include four most commonly used representation learning methods as the strong baselines, the objective functions of which are id classification loss [56] (771, contrastive loss [52] 57 [3].
triplet loss [4 50 and reconstruction loss (31) As reported in a recent comparison study on pedestrian re-identification [4]. the above baselines can already achieve accuracy comparable to the -tate-of-the-arts. Note that all the CNN baselines take a whole radiograph as input without attention
mechanism, and apply the same backbone architectures of ResNet-34 and Inception Inception-ResNet ResNet as FolD.
7

---

Table 3: Performance of FolD across ages and genders measured in mAP, and the significance values of the observed difference between the corresponding groups.
Groups
mAP
95% C1
p-value
<=30
0,6322
0,5161-0.7483
Age
0.2703
>30
0,5161
0.3260-0,7062
Female
0.6353
0.5033-0.7674
Gender
0.2969
Male
0,5296
0.3806-0.6787
According to Table 2 our method significantly outperforms all the existing approaches, achieving up to 59.62% in terms of mAP and 50.57% in terms of rank-1 accuracy. Compared to the key-point matching approach, the improvement shows that the proposed augmentation strategy enables effective representation learning from unpaired data. Compared to CNN basclines, our method surpass by incorporating domain knowledge with attentions - the accuracy boost of at least 10.50% in terms of mAP indicates that the localized attention regions valid for identification, and can obviously reduce the over-fitting to the non-related areas. Examples that visualize the query images and corresponding FolD's 's retrievals can be found in Appendix.
As shown in Table 3] we also report the performance of our proposed FolD across two age groups (<=30 and 30) and two gender groups (female and male). Overall, according to the paired t- test, no significant difference in the model's 's performance (p-value > 0.1) is observed between the aforementioned groups. Mcanwhile, This can possibly be explained as artifacts, e.g., implants and fillings, could be more often introduced to the identities of this group, which makes the identification task more challenging.
can that FolD has a lower mAP on the group of age>30.,
6.3 Ablation Studies
Table 4: Ablation studies of proposed self- attention by comparing with the ofate-of-the-ar attention-basee identification methods and com- mon aggregation mechanisms.
Table 5: Ablation studies of the objective function. Our proposed function combining three comple- mentary types of losses leads to the overall highest accuracy as indicated by mAP.
Methods
Backhone
mAP
Rank-1
Rank-5
Rank-10
PDC 51
AM
0.4982
0.4556
0.4598
0.4253
0.5862
(1.6632
0.6182
D.6322
Lgontraat
Ledd
Laup | mAP Rank-1 Rank-5
Rank-10
0.4723
0.4368
0.5747
0.6092
ResNot-34 os backhone
Avg. Pool.
34
0.3707
0.5057
0,5977
04760
0.4253
0.6303
0.6811
Max Pool.
R4sNer:34
022119
0,2069
0.3993
0.4253
0.316
0,2874
0.4298
0,5434
PC
34
0,4000
0.2989
0.5517
0.6667
0.4598
34
0.4085
0.3563
0.5172
0.5632
0.4817
0.7471
Att
ResNer-34
0.4268
0.4025
0.4828
0.5000
FolD (ours)
34sNet-44
0.5465
0.4817
0,6897
0.7471
Inception ResNei backbone
Aug Pool.
Resteei
ResNet
0.3824
0,2989
0.5517
0.6092
Mat Pool.
Inception-Rede RedNei
0.2654
0.1954
0.3793
0.5172
0.2808
0.1609
0/4328
0.5517
FC
Inception ResNet
0.3007
0,2414
0.4138
0.5172
0.5194
0.4368
0.7356
0.8046
Inception ResNei
0,5172
0.5962
0,5057
0.7586
0,7816
Gated All
Inception-Roshh
0.3741
0.3103
0.5287
0.6322
FoID (ours)
Inception ResNet
0.5962
0,5057
0.7586
0,7816
Self-attention with transformer FolD incorporates a novel transformer- seff-attention mech- anism to reason the relative importance of attentions for the feature aggregating. In order to verify its effectiveness, we first compare with the state-of-the-art attention mechanisms introduced in pedestrian re-identification: PDC [51), PGFA 44), and Manes (60]. To adapt those methods to our task, the original human body attention localization is replaced by the dental one as beforehand described in subsection 4.1 For fair comparison, the same DSA and training strategy as FolD are applied. Table 4 shows that FolD can outperform all the methods, exceeding by up 9.80% and 8.04% in terms of mAP and rank-l accuracy respectively. Different from the observation in 40 600, FolD outperforms PGFA and Manes by applying hard attentions over soft attentions such mechanism provides (i) the accurate definition of key anatomies informed from the formative study, and (ii) the improved efficiency of training with patches cropped out of large unrelated regions.
More importantly, the results also prove that modeling self-attentions can potentially lead to better aggregation results. To verify, we ablate the effect of self-attention by replacing the transformer with the existing aggregation modules of average pooling (Avg Pool.), max pooling (Max Pool.), fully connected layers (FC), weighted attention (Att) (28), and gated attention (Gated att) [68 28).
8

---

0.0
0.5
10
4
6
2 b
3
d
5
Figure 3: Visualization of the dynamic self-attentions captured by the transformer for four typical instances. Attention rollout performed to infer the attentions on imput patches. View in color and zooming in for the best quality. More attention visualizations can be found in Appendix.
meanwhile fixing all other setups. As R by consistently achieving the highest accuracy on all metrics with both backbone CNNs, reasoning the relative importance of various attentions is important for representation learning, and the proposed transformer approach can fulfill the goal. More details on PDC, PGFA, Manes, and the adaptations we performed can be found in Appendix.
Self-attention visualization We further look into the self-attention reasoning by visualizing raw images and their corresponding self self-attention weight maps. In specific, we roll out the attention weights of the transformer encoder layers to capture the propagation of information from input tokens to the final representation [2]. In FolD, each input token is the embedding of cropped attention patch encoded by the upstream CNN encoder. Figure 3 demonstrates the visualization results of four typical cases. We can obviously see that, in each case, FolD can localize certain identifying anatomies with higher attentions than the others. Such pattern is clear to include tooth with a unique shape (Figure (a))), wise tooth (Figure 3 a2, b3)), tooth with distinct artifact (Figure FFigure3(c4.d55 ((,,,))) and condyle of uncommon structure (Figure d()).. The phenomenon proves that our method can dynamically compare between the captured attentions, and highlight discriminative ones to boost recognition performance. Moreover, it also shows that the approach has chance to improve the model explainability, which be vital when dentists deep learning predictions. Nonetheless, we should note that such attention reasoning results is learnt based on our specific training dataset, which can be limited in both size and variety when compared to real clinical database. As such, more robust attention reasoning can possibly be learnt when the method is applied large scale databases, and accordingly higher identification accuracy can also possibly be achieved. We provide visualization results for both success and failure cases of FolD, as well as more detailed analysis. in Appendix to help understand the capability of FolD.
making final forensic decisions based
Domain-specific augmentations We intro-
mean Average Recision
0.6
Besic augmentationns
Dommin-spactic sugmantations
duce the DSA to generate forensic-realistic views of an instance for the effective instance discriminative learning. To verify the effective- ness of DSA compared to basic augmentations, we perform the ablation study based on both FolD and the CNN baselines. We define set of basic augmentations (BA) by following existing self-supervised contrastive approaches 44 10
0.5
0.4
0.2
0.1
FoiD (ours) (mrap ReelVet)
FolD louns) (Rest Meeteet-- 34)
Incep Reodfer tmoplt))
lecap- Rertrke (conmaciho vi
Incep- Sccp--Restee Resskoet colveiiicaiio
FlesNet-3. 34
ReaNort-3 3d
(cord rantho)
FreeVee-3 3d
colmsificationn
(erplit))
which include random displacements, rotations, and gaussian noises. More details about BA can be found in Appendix. Figure shows that DSA consistently outperforms BA, achieving mAP improvements ranging from 3.28% to 15.32%.
Figure Ablation studies on the proposed domain- specific augmentations for the model training. Var- ious methods are included in comparison. set of basic augmentations is used as the baseline.
Objective function design Three types of losses are included in FolD to complement for the representation learning. In order to under- stand the effect of each loss type, an ablation study performed based on both CNN encoders of ResNet-34 and Inception-Re ResNet. Tables]indicaaes that contrastive loss and id classification losses alone can have inferior performance, while the combining of the two leads to largely improved accuracy in terms of mAP (ResNet-34 as backbone: 0.4760/0.3 110000.196--0.5533 as backbone: 0.4438/0. 2808 -0.88-00.3990 5394. Such phenomenon has been observed by several previous studies pedestrian re-identification, but the accuracy boosts in our work seem to be more significant
Inception- Res

---

(e.g., 0.817-+0.859 and 1706-+0.764 as in (8)). This can possibly due to that two losses in FolD optimize two less correlated tasks of different supports, which makes more effective for regularizing the transformer. Besides, plugging in the deep supervision signal for the CNN encoder can further increase the accuracy (ResNet-34 as backbone: 14394->0.54k Inception-ResMe as backbone: possibly by the improved back paok-propagation and the direct regularization of the CNN encoder.
7 Discussion and Conclusion
Dental forensic identification has always been an important part of criminal/disaster investigation, but challenging and abbuur-consuming by relying on expert's 's visual comparison. Many efforts have been made on systems for storing, accessing and comparing imaging data to benefit the identification, e.g., py The American Board of Forensic Odontology. Our study is the first deep learning approach, aiming to speed up identification in scale and increase the success rate of identifying mussing/uuideniifedwwaee persons, and its inspiring result can possibly lead it to work with the above existing systems.
Ethical Considerations Apart from the promising results achieved by our deep learning solution for dental forensics, its risk of misuse should be considered when deploying such technique. First, although the technique is designed to benefit criminal and disaster investigation, there is a chance for this technique to be used for surveillance. The forensic identification shares similar nature with pedestrian re-ideniffication - the techniques of which have drawn concerns to intentionally or unintentionally discriminate against marginalized groups. Meanwhile, the pedestrian one can cause identification leakage by one's 's daily activities captured from surveillance systems, and the forensic one can be by one's medical care experience. As such, medical practitioners should follow the existing medical data protection protocol to avoid the possible identification leakage. Second, forensic practitioners can over-relay on the deep learning results for identification. The proposed method can be limited by the quality of training data as well as the lack of using other medical records entries beyond imaging. As such, clarify that the method is designed to only provide assistance for dental forensics, rather than make conclusions. An appropriate application of our method can be using the method as initial recommendation engine for top ranked matches by searching from large-scale medical records among multi-site databases, followed by practitioners manually examining the recommended results for the final conclusions.
Known Limitations There exist two known gaps between our proposed method and the clinical practice of forensic dentists. First, this work only includes panoramic radiographs, since they embody more comprehensive information compared to partial screening, e.g., bitewing x-rays, and is more widely used than expensive 3D screening, e.g., Magnetic resonance imaging. However, in real practice, forensic dentists can take advantages of multi-modality scans, e.g., by matching certain structures between bitewing and panoramic radiographs. Second, dentists can also utilize medical records as priors for imaging matching. For example, if a medical record indicates an entity was receiving dental fillings, shape difference at certain location can thus be expected between the preoperative scan in the database and a query scan. As such, how to design deep learning methods to overcome the two gaps is open to explore.
Broad impact First, this work introduces a novel re-identification task to the machine learning community. Compared to the existing tasks, e.g., pedestrian re-identification, the forensic one differs several aspects as listed our formative study. As such, directly applying existing re-identification methods led to inferior results (-9.80% in mAP compared to FolD). Moreover, the needs for the multi-modality awareness and medical priors also make unique. Combining the potential social impacts, future research might find forensics interesting research topic. Second, we propose novel transformer- based self-attention module that enables better attentions aggregation, according to both quantitative and qualitative merits. Such method is also promising for the pedestrian re- identification task, where different attentions (body key-points, can be assigned with different weights for representation based on the relative importance of body parts under certain circumstances (e.g., occlusions). Third, our domain-specific augmentation and multi-task learning have considered domain knowledge of dental forensics. This would be applicable to radiological forensies in general.
/ntps://abfo.orr/wiii//
10

---

8 Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for the valuable discussions.
References
[1] Mohamed Abdel-Mottaleb, Omaima Nomir, Diaa Eldin Nassar, Gamal Fahmy, and Hany H
Ammar. Challenges developing an automated dental identification system. In 2003 46th Midwest Symposium Circuits and Systems, volume 1. pages 41 411-414. IEEE, 2003.
[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint SakYv:2005.00922.. 2020.
[3] Ejaz Ahmed, Michael Jones, and Tim K Marks. An improved deep learning architecture for person re-identification, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3908-3916, 2015.
[4] Javier Ata-Ali and Fadi Ata-Ali, Forensic dentistry in human identification: A review of the literature. Journal of clinical and experimental dentistry, 6(2):e162, 2014.
[5] Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffeo- morphic image registration with cross-correlations evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26-41, 2008
[6] Slawomir Bak, Peter Carr, and lean-Francois Lalonde. Domain adaptation through synthesis for unsupervised person re-identification. Proceedings of the European Conference on Computer Vision (ECCV), pages 189-205, 2018.
[7] Juliana Baranova, Dominik Büchner. Werner Götz, Margit Schulze, and Edda Tobiasch. Tooth formation: Are the hardest tissues of human body hard regenerate? International Journal of Molecular Sciences, 21 1):4031, 2020.
[8] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference automatic face & gesture recognition (FG 2018), pages 67-74. IEEE, 2018
[9] Hong Chen and Anil K Jain. Dental biometrics: Aligmment and matching of dental radiographs. IEEE transactions pattern analysis and machine intelligence, 27(8):1319) 1326, 2005.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[11] De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. Person re- identification by multi-channel basts-based cnn with improved triplet loss function. In Proceed- ings of the iEÉE conference on computer vision and pattern recognition, pages 1335-1344, 2016.
[12] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-22 255. leee, 2009.
[13] Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, and Jianbin Jiao. Image- image domain adaptation with preserved sell-similarity and re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 994-1003, 2018.
for person
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.8100000 04805, 2018.

---

[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[16] Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision trans- formers for image retrieval. arXiv preprint (.X.v:2212.055660 2021.
[17] Hehe Fan, Liang Zheng, Chenggang Yan, and Yi Yang. Unsupervised person re-identification: Clustering and fine-tuning, ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 14(4):1-18, 2018.
[18] Diane J Flint, Stephen Brent Dove, Paula C Brumit, Marea White, and David R Senn. Computer- aided dental identification: an objective method for assessment of radiographic image similarity. Journal forensic sciences, 54(1):177-184, 2009.
[19] Alexander S Forrest.
Collection and recording of radiological information for forensic purposes. Australian dental journal, 57:24-32, 2012.
[20] Yang Fu, Yunchao Wei, Guanshuo Wang, Yuqian Zhou, Honghui Shi, and Thomas S Huang. Self-similarity grouping: simple unsupervised cross domain adaptation approach for person re-identification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6112-1121, 2019.
[21] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, and Hongsheng Li. self-paced contrastive learning with hybrid memory for domain adaptive object re-id. arXiv preprint arXiv:2006.0271: 2020.
[22] Socratis Gkelios, Yiannis Boutalis, and Savvas A Chatzichristofis, Investigating the vision transformer model for image retrieval tasks. arXiv preprint arXiv:2101.03771, 2021.
[23] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[24] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. arXiv preprint arXiv:2102.04376 2021.
[26] Andreas Heinrich, Felix V Güttler, Sebastian Schenkl, Rebecca Wagner, and Ulf K-M Teich- gräber. Automatic human identification based on dental x-ray radiographs using computer vision. Scientific reports, 10(1):1-13, 2020.
[27] Anastasia Holobinko. Forensic human identification in the united states and canada: A review of the law, admissible techniques, and the legal implications of their application in forensic cases. Forensic Science International, 222(1-3):394-e1. 2012.
[28] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-basse deep multiple instance learning. In International conference machine learning, pages 2127-2136. PMLR, 2018
[29] Gil Jader, Jefferson Fontineli, Marco Ruiz, Kalyf Abdalla, Matheus Pithon, and Luciano Oliveira. Deep instance segmentation of teeth in panoramic x-ray images. In Conference on Graphics, Patterns and Images (SIBGRAPI), pages 400-407. IEEE, 2018
[30] Kahana and J Hiss. Identification of human remains: forensic radiology. Journal of clinical forensic medicine, 4(1):7-15, 1997.
[31] Daisuke Komura, Keisuke Fukuta, Ken Tominaga, Akihiro Kawabe, Hirotomo Koda, Ryohei Suzuki, Hiroki Konishi, Toshikazu Umezaki, Tatsuya Harada, and Shumpei Ishikawa. Luigi: Large-scale histopathological image retrieval system using deep texture representations. biorxiv, page 345785, 2018.
12

---

[32] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-basee based permutation-invann -nvariiant neural networks, In International Conference on Machine Learning, pages 3744-3753, PMLR, 2019.
[33] Dangwei Li, Xiaotang Chen, Zhang Zhang, and Kaiqi Huang. Learning deep context-aware features over body and latent parts for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 384-393, 2017.
[34] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 152-159 159, 2014.
[35] Hao Liu, Jiashi Feng, Meibin Qi, Jianguo Jiang, and Shuicheng Yan. End-to-end comparative attention networks for person re-identification. IEEE Transactions on Image Processing. 26(7):3492-3506, 2017.
[36] Hongye Liu, Yonghong Tian, Yaowei Yang, Lu Pang, and Tiejun Huang. Deep relative distance learning: Tell the difference between similar vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2167-2175, 2016.
[37] Zimo Liu, Dong Wang, and Huchuan Lu. Stepwise metric promotion for unsupervised video person re-identification. In Proceedings of the IEEE international conference computer vision, pages 2429-2438, 2017.
[38] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[39] Shinpei Matsuda, Takashi Miyamoto, Hitoshi Yoshimura, and Tatsuhito Hasegawa. Personal identification with orthopantomography using simple convolutional neural networks: prelimi- nary study, Scientific reports, 10(1):1-7, 2020. 2020.
[40] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages $42-551,2015
[41] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference Computer Vision, pages 681-699. Springer, 2020.
[42] Ayse Betul Oktay. Human identification with dental panoramic radiographic images. IET Biometrics, 7(4):349-333 2018
[43] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. 2015.
[44] Massimiliano Patacchiola and Amos Storkey, Self-supervised relational reasoning for represen- tation learning. In Advances in Neural Information Processing Systems, 2020.
[45] Jahagirdar B Pramod, Anand Marya, and Vidhii Sharma. Role of forensic odontologist in post mortem person identification. Dental research journal, 9(5::22,, 2012.
[46] IA Pretty and D Sweet. A look at forensic dentistry part 1: The role of teeth in the determination of human identity. British dental journal, 190(7):359-36) 366,, 2001.
[47] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 815-823, 2015.
[48] Jianlou Si, Honggang Zhang, Chun-Guang Li, Jason Kuen, Xiangfei Kong, Alex C Kot, and Gang Wang. Dual attention matching network for context-aware feature sequence based person re-identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5363-5372, 2018
[49] Bernardo Silva, Lais Pinheiro, Luciano Oliveira, and Matheus Pithon. A study on tooth segmentation and numbering using end-to-end deep neural networks. In Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2020.
13

---

[50] Chunfeng Song. Yan Huang. Wanli Ouyang, and Liang Wang. Mask- Mask-guider guided contrastive attention model for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1179-1888 2018.
[51] Chi Su, Jianing Li, Shiliang Zhang, Junliang Xing, Wen Gao, and Qi Tian. Pose-driven deep convolutional model for person re-identification. 2017 IEEE International Conference on Computer Vision (ICCV), pages 3980-3989, 2017.
[52] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predict- ing 10,000 classes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1891-1895 2014.
[53] Yifan Sun, Qin Xu, Yali Li, Chi Zhang, Yikang Li, Shengjin Wang, and Jian Sun. Perceive where to focus: Learning visibility-aware part-level features for partial person re-identification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 393-402, 2019.
[54] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and strong convolutional baseline). In Proceedings of the European conference on computer vision (ECCV), pages 480-496, 2018
[55] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception -4,t inception-resnet and the impact of residual connections learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[56] Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701-1708, 2014.
[57] Rahul Rama Varior, Bing Shuai, Jiwen Lu, Dong Xu, and Gang Wang. A siamese long short- term memory architecture for human re-identification. In European conference computer vision, pages 135-153. Springer, 2016.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706. 03762, 2017.
[59] Anoop K Verma, Sachil Kumar, Shiuli Rathore, and Abhishek Pandey. Role of dental expert in forensic odontology. National journal of maxillofacial
surgery, 5(1):2, 2014.
[60] Cheng Wang, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Manes: A multi- task attentional network with curriculum sampling for person re-identification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 365-381, 2018.
[61] Faqiang Wang, Wangmeng Zuo, Liang Lin, David Zhang, and Lei Zhang. Joint learning of single-image and cross-image representations for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1288-1296, 2016.
[62] Guan' an Wang, Shuo Yang, Huanyu Liu, Zhicheng Wang, Yang Yang, Shuliang Wang, Gang Yu, Erjin Zhou, and Jian Sun. High-order information matters: Learning relation and topology for occluded person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6449 6458, 2020.
[63] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. Learning discriminative features with multiple granularities for person re-identification. In Proceedings of the 26th ACM international conference on Multimedia, pages 274-282, 2018.
[64] Yanan Wang, Shengcai Liao, and Ling Shao. Surpassing real-world source training data: Random 3d characters for generalizable person re-identification. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3422-3430, 2020.
[65] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 79-88, 2018.
14

---

[66] Longhui Wei, Shiliang Zhang, Hantao Yao, Wen Gao, and Qi Tian. Glad: Global-local- aligmment descriptor for pedestrian retrieval. In Proceedings of the 25th ACM international conference on Multimedia, pages +20-428, 2017.
[67] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture trans- former network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5791-5800, 2020.
[68] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[69] Jinjie You, Ancong Wu, Xiang Li, and Wei-Shi Zheng. Top-push video-based person re- identification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recog- nition, pages 1345-1353, 2016.
[70] Hong-Xing Yu, Wei-Shi Zheng, Ancong Wu, Xiaowei Guo, Shaogang Gong, and Jian- Mtan-Huang Lai. Unsupervised person re-identification by soft multilabel learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2148-2157, 2019.
[71] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv: 201. 11966 2021.
[72] Kaiwei Zeng, Munan Ning, Yaohua Wang, and Yang Guo. Hierarchical clustering with hard- batch triplet loss for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13657-13665, 2020.
[73] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10076-10085, 2020.
[74] Liming Zhao, Xi Li, Yueting Zhuang, and Jingdong Wang. Deeply-leamedd part-aligned representations for person re-identification. In Proceedings of the IEEE international conference on computer vision, pages 3219-3228, 2017.
[75] Rui Zhao, Wanli Ouyang, and Xiaogang Wang. Unsupervised salience learning for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition,, pages 3586-3593, 2013.
[76] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re- iddetiffccaaion: A benchmark. In Proceedings of the IEEE international conference on computer vision, pages 1116 1124, 2015.
[77] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision, pages 3754-3762, 2017.
[78] Jindan Zhou and Mohamed Abdel-Mottaleh. A content-based system for human identification based bitewing dental x-ray images. Pattern Recognition, 38(1 (8(11):2132-2142 2005.
[79] Xiaoke Zhu, Xiao Yuan Jing, Xinge You, Xinyu Zhang, and Taiping Zhang Video-based person re-identification by simultancously learning intra-video and inter- video distance metrics. IEEE Transactions on Image Processing, 27(11):56835500 2018.
15


Reviews:
[{"review_id": "mpUXmLEyqW4", "reviewer": "Reviewer_tMZm", "summary": "The work proposes to use deep learning for forensic dental identification, i.e., identifying living or deceased persons from their panoramic dental radiographs. It outlines the main, domain-specific challenges: lack of paired radiographs for metric learning, lack of annotations (e.g., bounding boxes of salient anatomic parts used by dentists), and visual variability. The paper proposes the new task, describes a set of domain-specific augmentations (based on anatomical variation), and suggests strategies to adapt existing networks to this domain.\n\nBy introducing forensic dental identification as a new ML task, the authors point out a new problem that could invite other researchers to design algorithms for. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed an interesting and novel task, and designs a state of the art network to solve it. I found the introduction and related work easy to understand and very insightful. The results section, however, was difficult to follow and had quite a bit of grammar issues. Given that the architectural findings that the authors discovered, what implication does this approach have to forensic dental identification and machine learning in general? \n\nHere are some specific questions:\n\n- How important is it to use a panorama-retrained model?\n- Please add visualizations and sample annotations of regions used for hard attention\n- Please include a visualization of the proposed domain-specific augmentations\n- Include an analysis of false negatives, as this may be an important limitation of using this approach. How is the performance in comparison to identification performed by humans?\n\nI was not able to download the datasets, and do the authors plan to release it? I think this would be a critical task because the goal of the paper is to encourage more exploration in this specific domain.\n\nTypos/corrections:\n- Line 15: “effeteness” -> effectiveness\n- Line 47: “In specific,…” -> bad grammar\n- Line 147: “In specific” -> Specifically\n- Line 152: “does not relay” -> does not rely\n- Line 212: “we incorporates” -> we incorporate\n- Line 270: “few study” -> few studies\n- Line 312: “We can obviously see” -> We can see that. Why obvious?\n- Line 325: “effeteness” -> effectiveness\n\n\nEDIT (08/18): I have read the other reviews and authors' rebuttal. I thank the authors for clarifying my concerns. I second other reviewers' concern about adding additional discussion about ethics, limitations, and data description. Please see main comment. I do, however, think that this paper will be very valuable to the ML community as a practical application of current technologies to a new task.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the paper proposed a very novel ML application, it is also very limited to this application. This is an inherent limitation and while there is little the authors can do about this, they should discuss more specifically how their findings can affect other tasks such as person re-identification, data augmentation etc. \n\nI appreciate the thorough description of the limitations and broader impact. However, due to privacy concerns, how likely would the proposed system be used in practice? \n", "review_text": "", "extra_scores": {"ethical_concerns": "EDIT (08/18): see main review.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "m9B-5CP5if_", "reviewer": "Reviewer_kU4j", "summary": "The authors propose a new architecture FoID for forensic identification of individuals from dental imagery, a task often conducted by domain experts for identification at crime scenes/disaster sites. Given an image and a tooth segmentation map, they augment a straightforward CNN in four fashions: (i) hard attention on the regions of the scan where identifying information can be found, (ii) dental-specific augmentations (e.g. addition of a cavity) as opposed to general augmentations, (iii) the inclusion of three loss families (classification, contrastive, identification), and (iv) a transformer-based attention mechanism for focusing on unusual shapes of interest. In experiments, they outperform a series of benchmarks,  and ablations show the importance of each of the augmentations above. They include interpretations of the self-attention mechanism for explainability. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present an architecture that seems quite effective for forensic identification, on a relatively novel task for the machine learning community. No proof/theoretical claims are provided, and all empirical claims seem well-founded with fair and thorough baseline comparisons across multiple CNN architectures and ablations. The domain-specific augmentations are clever and thorough in creating effective data pairs. The provided attention maps further seem to indicate that the model is operating as expected. The paper is clearly written and easy to follow, and I appreciate the author's inclusion of code in their submission.\n\nHowever, there is admittedly limited novelty of the methods and conclusions from this work , which may not make it most suitable for the NeurIPS community. Going through the contributions:\n* Image augmentations have long been used as a way to augment data sets, and it seems expected that the more hand-crafted, tailored augmentations for the dental domain would outperform out-of-the-box standard approaches. \n* It is relatively standard to have multiple objective/objectives types in the machine learning community, and the authors themselves include the fact that the advantage of multiple objectives has already been shown in the closely related pedestrian re-identification domain. \n* While the transformer self-attention mechanism is in a slightly different position in the network than might be standard, there is now a wealth of work using transformer self-attention mechanisms (e.g. https://arxiv.org/pdf/2101.01169.pdf), including using them in combination with CNNs.  \nAs a result, it's not clear to me whether the paper necessarily provides generalizable conclusions beyond what's already existing in recent literature. \n\nAs a minor note to the author, you likely mean to use “effectiveness” over “effeteness”", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do think a little more emphasis should be placed on discussing the privacy risks of re-identification, particularly in settings outside of dentistry. While dental images may be locked within medical records, the abstract says the proposed techniques could be repurposed for pedestrian re-identification, and given there is currently a large debate on the ethics around person re-identification, particularly due to bias, I think it is important authors give the issue more consideration in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "As mentioned above, there are some ethical concerns with re-identification (particularly in non-dental settings they propose FoID could be extended to). While I don't think it necessarily requires an ethical review (though I'm not sure the line), I do think it requires at minimum further discussion from the authors. "}}, {"review_id": "Nvrp3d44Ms6", "reviewer": "Reviewer_zijN", "summary": "The authors propose a framework for retrieval of dental panoramic X-Rays. \nTo find regions of interest they focus on teeth which are extracted by semantic segmentation using a pretrained mask-RCNN as well as relevant areas extracted based on landmarks for which it is necessary to register every image to an atlas in a deformable way. All those ROIs are extracted and augmented in a domain-specific way (fillings, tooth extraction, etc), which serves as input to a CNN which is trained with an objective function that consists of three loss terms, added in an unweighted manner: a contrastive loss, an ID loss and a deep supervision loss.\n\nThey use a publicly available dataset for training, but test the proposed method on data collected by the authors, s.t. each image in the test set is to be retrieved from the union of the test and training set.\n\nAn ablation study on different modules of the framework is performed to test their contributions to the overall improvement of the proposed method.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The framework combines multiple state of the art methods in order to solve a specific and very challenging task.\nIt is interesting work in that there are several challenging subtasks that needed adaptation from their original publications to this kind of dental data which all come with their own challenges and hyper parameters (starting from the deformable registration to an atlas and the semantic segmentation to generate the input data).\nHowever I am missing a novelty which is transferrable to other kind of data than this explicit application. I believe that this work is very valuable and of significant importance in the field of digital dental identification, but I am not convinced that it entails enough novelty from an ML perspective that is of general purpose to other fields of application. (If the authors can point out a novelty which is of a broader interest apart from dental identification, then this will also have to be reflected in the broad societal impact.) As the framework consists of many subtasks, there is only little space to provide all details needed for reproducibility in a conference paper like this and I believe a journal which is in theme closer to the paper’s application would be a more suitable publishing channel.\n\nQuestions to address:\n\n\nIt is unclear to me how the baseline experiments were conducted which are presented in Table 2. What exactly was done for e.g. ResNet-34+reconstruction, or ResNet-34+triplet, etc.? What was the pipeline for one such experiment?\n\nI have some questions regarding the dataset. The paper says that UFBA_UESC was used to pretrain the mask-RCNN (line 132) for the semantic segmentation step to locate the teeth. Later, for the benchmark construction, the DNS Panoramic dataset is used as a training set. In [50], in which the DNS Panoramic dataset is introduced, it says that the original dataset, named “UFBA-UESC Dental Images data set” (published by Silva et al., ‘“Automatic segmenting teeth in x-ray images: Trends, a novel data set, benchmarking and future perspectives”, 2018) was modified by Jader et al. (“Deep instance segmentation of teeth in panoramic x-ray images”, 2018) to allow for instance segmentation, resulting in the dataset named “UFBA-UESC Dental Images Deep”. Silva et al. in [50] (“A study on tooth segmentation and numbering using end-to-end deep neural network”, 2020) then modified “UFBA-UESC Dental Images Deep” further, resulting in the “DNS (Detection, Numbering, and Segmentation) Panoramic Images”, which in turn was used as a training set in the proposed publication. So I wonder in what ways the UFBA_UESC dataset used for pretraining in the mask-RCNN step differs from the DNS dataset used for the overall training, and if the pretraining - in case of a big overlap of the two datasets - has much/any impact?\n\nWhat size are the images in pixels and how do they need to be resized to fit the CNN encoder which was pretrained on RGB ImageNet images?\nIs the atlas publicly available to which the images get registered?\n\nRegarding the addition of the three types of loss in the final objective function: are they all of the same magnitude, or is one of the loss terms dominating over the other two?\n\nIs the test set released?\n\nWhat is meant by online mining on line 201 in this context?\n\nMinor comments: formatting line 301;  typos or similar in lines: 143, 169, 177, 181, 184, 206, capitalization of embedding dim. line 207, 212, 224, 270/271, 282, 405,\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "8ZPjbHUVNzQ", "reviewer": "Reviewer_t8rG", "summary": "The paper presents an extensive study on applying deep learning (including novel techniques) to dental forensic identification.\n\n**Main Contributions:**\n\n- An in-depth discussion on the challenges of the task and limitations of past works;\n- Multiple techniques ranging from data augmentation to network architecture to tackle these challenges (with potential applications to other tasks);\n- A comprehensive dataset covering many dental variations, to reflect a realistic scenario;\n- An extensive and thoughtful ablation study that gives insights on the effectiveness and benefits of each major technical decision or proposed technique; and\n- SOTA results in the task by a large margin vs. sensible baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents extensive and thoughtful study on deep learning applications to dental forensic identification.\n\nFirst, it presents a discussion on limitations of past work and challenges of this task, encompassing both the available datasets and previously used models. This discussion is grounded on insights from dental professionals, addresses short-comings from techniques from related fields (e.g. person reid), and lays the foundation from the proposed techniques. Additionally, the ablation studies carefully confirm these claims, with the proposed solutions individually outperforming across the board.\n\nSecond, it demonstrates a novel dataset that addresses many of the limitations of past work, e.g. by including many dental variations and scenarios. Certain scenarios that are not covered in the dataset are discussed later in the paper.\n\nThird, it proposes a novel approach to leveraging deep learning to tackle this task. The techniques proposed range from data augmentation to network components. It cleverly combines approaches from diverse existing literature with original contributions, with justifications based on insights from dental professionals, and accompanying comprehensive ablation studies to validate their hypotheses.\n\nFinally, it presents a careful evaluation of the work, including comprehensive ablation studies. The proposed technique achieves SOTA results that strongly outperform sensible baselines presented by the authors. The ablation studies corroborate the choice of the major individual components of the technique (e.g. data augmentation, objective function).  \n\nThe writing is generally clear, and the code is available. The dataset is based on a publicly available source, and another anonymized source (I believe for the review process).\n\nAll in all, the paper presents a thoughtful discussion, novel methods (with potential applications to other tasks), and a careful and comprehensive evaluation.\n\n---\n\nMinor:\n\n- L301: missing/broken citations (\"[]\")\n- L15: typo \"effeteness\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper acknowledges and addresses the limitations of the proposed dataset, as well as the privacy risks of dental identification (which, like they discuss, are less risky vs. person reid and facial recognition). It might have been helpful to further discuss the potential impact of this technique in the context of law enforcement, which is prominent in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "Since some data sources are anonymized (maybe for the review process? see L227), we can't evaluate if there are any ethical concerns there. Considering this is medical data, this might be problematic. The paper does claim the data is anonymized."}}]

Author rebuttal:

Meta-review: This paper generated significant debate, both on technical contribution considerations (more general methodology) and ethics considerations. Thank you to the reviewers and authors for the detailed discussion. 

On the ethics considerations, the authors have responded saying they will include a more detailed discussion of potential misuses, and importantly, results across different subpopulations. The authors should include as many of these subgroups as possible (across gender, race and multiple age brackets) along with discussion on any places where the method performs below average.

On the technical considerations, a big debate was between whether the methods proposed are too specialized to this application. After careful review of the discussion, I think this paper can be viewed as an applied contribution, that should be of interest to others in the community who are studying this application.
Decision: Accept (Poster)