Neural Boneprint: Person Identification from Bones using Generative Contrastive Deep Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:07:16

Abstract: Forensic person identification is of paramount importance in accidents and criminal investigations. Existing methods based on soft tissue or DNA can be unavailable if the body is badly decomposed, white-ossified, or charred. However, bones last a long time. This raises a natural question: ***can we learn to identify a person using bone data?***
We present a novel feature of bones called ***Neural Boneprint*** for personal identification. In particular, we exploit the thoracic skeletal data including chest radiographs (CXRs) and computed tomography (CT) images enhanced by the volume rendering technique (VRT) as an example to explore the availability of the neural boneprint. We then represent the neural boneprint as a joint latent embedding of VRT images and CXRs through a bidirectional cross-modality translation and contrastive learning. Preliminary experimental results on real skeletal data demonstrate the effectiveness of the Neural Boneprint for identification. We hope that this approach will provide a promising alternative for challenging forensic cases where conventional methods are limited. The code will be available at ***.

Main paper: 
 motivation: 
 In this paper, we aim to explore the important but never exploited basic fact that bones generally persist for a very long time, either in the corpse or in the skeleton. We assume that there is an implicit feature boneprint in bones, similar to palmprints and fingerprints, which encodes identity information and is widely present in skeletons and skeletal data.It is worth noting that not each bone on the skeleton can be a suitable candidate for large-scale person identification. Those available bones, such as vertebrae and skulls [29, 42] , contain identifiable boundaries and rich morphology that varies from person to person. In brief, an efficient skeleton-based person identification requires the skeletal data (i) to contain identifiable boundaries and distinct morphology, and (ii) to be easily collected and organized in a largescale matching pool. While it is commonly acknowledged that human faces and skulls contain identifiable features [29] , we seek to investigate overlooked aspects of the skeletal structure that may also hold valuable identity information, and the thoracic skeleton serves as an apt example for this purpose. Thoracic skeletons, including ribs, vertebrae, and sternums [19, 25, 39, 56] , have been used as manual comparison materials to estimate sex and age due to complex morphology and distinct visual individual differences. Therefore, in this paper, we will explore the thoracic skeleton data to learn thoracic boneprints.Forensic pathologists are often required to preprocess computed tomography (CT) images by volume rendering technology (VRT) [16, 46] to obtain VRT images for analytical studies. For the corpse, chest VRT can obtain a clear thoracic skeleton to avoid the effects of soft tissue and organ decomposition on skeletal observation. However, it is impractical to construct an adequate pool from VRT images. That is, not everyone has had a CT scan because the risk of radiation-induced cancer is increased by the use of CT [4, 34] , resulting in a paucity of VRT images. As a result, it is unlikely that there is a pre-stored CT image to compare with a query VRT image of an unnamed corpse.Fortunately, the chest X-ray (CXR) is a regular and inexpensive part of the physical examination. In industrialized countries, there is an average acquisition of 238 erect-view CXRs per 1000 of the population annually [5, 35] . In 2006, approximately 129 million CXRs were obtained in the United States alone [33] . The CXR is often the first imaging study acquired and remains central to screening, diagnosis, and management of a broad range of conditions [40, 50] . After the COVID-19 pandemic, that number will only be higher. More importantly, CXR images usually contain all the skeletal elements of the chest and their identities. This shows that CXRs can naturally form a huge pool for the identification of thoracic skeletons. Simply using CXRs or synthetic CXRs of corpses may be proposed. However, for corpses that are not fully decomposed, the decomposition and expansion of organs and soft tissues can significantly impact observation and comparison. Furthermore, for skeletonized corpses, the simulated soft tissues and organs in synthetic CXRs can be misleading and thus unsuitable for analysis. The question then becomes, how can we exploit the vast amount of CXR data and introduce its rich identification information into the small VRT data? If we can learn boneprint from CXR and VRT images and how? We have visualized VRT images and CXRs from two individuals who possess similar overall thoracic skeletal morphology in Figure 4 . Although this similarity might obscure obvious differences at first glance, upon closer inspection, subtle variations in chest contours and rib boundary details become apparent. It is noteworthy that even within skeletons exhibiting overall similarity, these minute distinctions are present. In thoracic skeletons displaying significant morphological differences, the disparities would undoubtedly be more prominent.Challenges. To answer the above questions, several challenges related to the characteristics of skeletal data (VRT, CXR) need to be addressed. The main challenges can be categorized as follows:(1) Data Imbalance and 2 Shots. There is also a notable imbalance in the availability of image types. VRT images are relatively scarce and sparse in distribution, in contrast to CXRs, which are abundant and densely distributed. This disparity presents a significant challenge in data handling and analysis. Besides, we have only 2 shots in one category (a single CXR-VRT image pair per individual) in our dataset. This limitation exacerbates the difficulty in extracting reliable boneprint features. (2) Large Intra-Class Variation. The considerable modality gap between VRT images and CXRs results in significant intraclass variation. CXRs, in particular, exhibit complex overlaps of various anatomical structures, unlike VRT images that predominantly display skeletal features. Additionally, the varied postures during imaging lead to bone deformation, further complicating the analysis. (3) Small Inter-Class Difference. The subtlety of thoracic skeletal differences between individuals poses a formidable challenge. Unlike other identification tasks where inter-class features are distinct, the skeletal features in our case are less conspicuous, sometimes barely discernible to the human eye. This issue is compounded by the physiological similarities in rib numbers and orientations shared among humans, making differentiation based on these features challenging.Overcoming the above challenges requires sophisticated approaches to image processing, data analysis, and feature extraction to effectively learn and use boneprints for personal identification. In the next section, we will present a novel deep learning framework to solve these challenges and learn neural boneprints for indentification. 
neural boneprint (nbp): 
 We aim to use deep learning to learn the boneprints of thoracic skeletons, i.e. neural boneprint (NBP), from VRT images and CXRs for person identification. In particular, to overcome the aforementioned challenges, we propose three modules to learn NBP: In this paper, let 𝑥 and 𝑦 be the VRT image and CXR image, T 𝑥 𝑦 : 𝑥 → 𝑦 and T 𝑦𝑥 : 𝑦 → 𝑥 are two domain translators. The algorithm ultimately outputs a neural boneprint (NBP) feature through a fusion network F, represented as F = NBP(𝑥, 𝑦). In the below sections, we will provide a comprehensive description of our NBP algorithm, elucidating its workflow, proposed modules, and design details. The training detail, including the loss function, will also be discussed.The primary obstacle in converting between VRT images and CXRs is the significant differences in anatomical structures and the distortion of the thoracic skeleton [20] . This discrepancy creates a significant gap between the two modalities that cannot be effectively bridged by straightforward unidirectional translation [60] methods like [27] .From a probabilistic standpoint, focusing solely on 𝑝 (CXR|VRT) or 𝑝 (VRT|CXR) in a unidirectional approach hinders our ability to explore the intricate semantic relationships between VRT images and CXRs. To address this, we propose a bidirectional process, that is, simultaneously learning from VRT to CXRs and vice versa. By deliberately aligning their intermediate latent variables in terms of distribution, we establish a preliminary joint latent space between VRT and CXRs, denoted as 𝑝 (VRT, CXR). We argue that this shared latent space enhances the model's capacity to grasp the complex semantic interplay between VRT and CXRs during the cross-modal transformation, as shown in Figure 5 . Specifically, we employ two transformation networks to model the processes of 𝑝 (CXR|VRT) and 𝑝 (VRT|CXR) separately. We consider the latent variables in the networks' middle layer to follow a multivariate Gaussian distribution and compute the 2-Wasserstein distance 𝑊 to measure the divergence between them: In addition, we adopt an extra ℓ 2 loss penalty to maintain finegrained personally identifiable information. That is, the real VRT image is paired by identity with the CXR image at the semantic level, so the translated VRT image should be the same as the real VRT image due to the individual identity. In this manner, translated images can be viewed as sampling in the vicinity of the true image in manifold space [58] .EQUATIONAfter the cross-modality translation, we obtain a real and translated image pair for each individual. Due to the identifiable design, the translated distribution is already close enough to the real distribution in the identity manifold, so we also treat them as real.We employ a dual input reconstruction network based on contrastive learning to fuse distinguishable skeletal representations to a latent embedding F named neural boneprint. It maps each VRT-CXR image pair to a joint latent embedding in manifold space where the identity information is the primary constraint [14, 31] . It contains a VRT encoder-decoder module, a CXR encoder-decoder module, and a latent fusion module. Each encoder-decoder module is utilized to reconstruct real or translated images for learning the latent fine-grained skeletal representations and then fusing them as neural boneprints. We compute the mean squared error (MSE) between the reconstructed and original images in the pixel space to assist in learning the better fine-grained identical latent features. We also compute the contrastive loss [3, 10, 22] based on the fused embeddings of the real, translated, and the other's real image pairs for minimizing the intra-class distance and maximizing the inter-class distance. During the training phase, we jointly train encoder-decoder modules and the latent fusion module. In the application phase, we ignore decoders and only employ encoders and the latent fusion module with weights frozen.The main idea is that CXRs are widely available and identity-rich, while VRT images are common for skeletal remains. This process aims to create an identification function ID(𝑥) for efficient identification using a CXR database. Given a CXRs database, we process as following 3 steps:(1) Modality Unification: All CXRs images {𝑦 𝑖 } are first translated into VRT images {T 𝑦𝑥 (𝑦 𝑖 )} using a translator T 𝑦𝑥 . (2) Joint Embedding Generation: CXR-VRT pairs {(T 𝑦𝑥 (𝑦), 𝑦)} are mapped via F to joint latent embeddings (NBPs) that capture cross-modal relationships, incorporating identity information.(3) NBP-Bank Creation: The NBP-Bank, denoted as C, is constructed as a searchable table with embeddings as keys and identities as values:C ≜ {(Key = F(T 𝑦𝑥 (𝑦 𝑖 ), 𝑦 𝑖 ), Value = ID(𝑦 𝑖 ))} (2)As we presented above, the modality unification bridges the CXR and VRT domains, improving the accuracy of identification. The joint embeddings (NBP) can effectively represent cross-modal relationships and identity information. Finally, the constructed NBP bank will enable identification via matching, i.e.  
metrics: 
 Rank-𝑘. As a classical metric in face identification and person reidentification [52, 59] , Rank-𝑘 is suitable for our single-shot person identification. In addition, person identification generally involves ethical or legal issues in forensic science. It is also important to simplify the list of candidate victims [1] . Therefore, we propose to use the Percentile Ranking Rate as the metric.Percentile Ranking Rate. The percentile ranking rate aims to measure the ability of the model to hit targets in the top 𝒑 percent across databases of different scales. Similar to the Rank-𝒌 identification rate, we will compute the similarities between the query and the candidates one by one. Then sort the similarities from largest to smallest. If the correct pair is in the top 𝒑 percent, the match is considered successful, otherwise, the match fails. The ratio of successful pairs to all pairs is the percentile ranking rate.The VRT images in the test set of the first dataset are used as queries. We apply our approach to two NBP banks, which are organized from our dataset. One is extracted from the CXRs of the first dataset. To test robustness, we introduce the CXRs in the second dataset as distractors and contribute to the second NBP bank with the test set of the first dataset together. More experiments under large-scale databases are prepared in Supplementary.Alternative Comparison. Since this is the first comprehensive work in this new task, there is no comparable thoracic skeleton identification network. We employ several classical identification models for comparison. In Table 1 , we compare the performance of Ours and other alternatives in extracting neural skeletal features. The results of IResNet-18 and IResNet-50 [15] show that deeper networks did not lead to significant gains. We also compared with a classical cross-modality person re-identification method [38] .Cumulative Match Characteristic. We select the six methods with the best performance from all the experiments and plot the cumulative match characteristic curve [52] , as shown in Figure 6 . Ablation Study. We explore the ablation study on the extent of the introduction of CMT in Table 2 . It shows that only using one modality (whether CXR or VRT) cannot be effective. We also explore the ablation study of the positive sample strategies for contrastive learning in the neural boneprint extraction step, as shown in Table 3 . The Augmented one uses the augmentation strategy of SimCLR [10] without cross-modality translation results introduced. Besides, we explore the ablation study on the choice of the reconstruction loss functions with the CMT and the contrastive loss fixed, as shown in Table 4 . The suboptimal result means the L1 loss can capture fine-grained details than MSE in some specific samples, but MSE can get a better generalization performance for total samples.5DQNN 0DWFKLQJ5DWH 7ULSOHW 0RELOH)DFHQHW L5HV1HW 2XUVWe compared ours with some studies [12, 26] We visualize the top 5 samples and the last 5 samples of the identification results after introducing distractors, as shown in Figure 7 . With careful comparisons, it can be found that the top five image pairs are highly similar to the query image pairs while the last 5 have low similarity. What's more, the deformations of thoracic skeletons and complex overlaps between VRT images and CXRs make it extremely hard for humans to identify, but it can be done by the proposed approach.Some unidirectional [60] translation methods, such as Pix2Pix [27] , may be discussed as alternatives in our cross-modality translation step. With that in mind, we also visualize some VRT-CXR image pairs with their translated results by different methods, as shown in The translated ones by cycle method are highly similar to the real, both in the overall thoracic skeletal structure and in the boundaries and curvature of the ribs.rather than strictly pixel-level. Besides, the corresponding translated VRT images contain almost the same identifiable rib details and orientations compared with the real. Then we compare the visual performance of generating CXRs. As real VRT images do not contain those anatomical structures, some translated structures similar to organs and muscles may be created during the translation. Despite that, we can ignore the created structures because we only focus on the thoracic skeletons. 
related work: 
 Identification of human remains through antemortem and postmortem imaging materials [2] has been widely used.Manual Methods. For a long time, forensic odontologists have manually utilized dental radiographs taken of the victim before death and manually compared them to dental data from the remains to assist in the identification [43, 53] . Using head and neck CT and MR imaging to obtain identifying information, including dental findings, to identify individuals is also feasible [18] . Besides, it has been demonstrated that the identification of human remains using visual comparison could be performed by a forensic pathologist with CT experience through 20 cases with antemortem and postmortem CT images, but there is no consensus on the specific number of concordant traits required to establish an identity [2] . Study [51] investigated antemortem and postmortem radiographs of the claviculae and C3-T4 vertebrae to identify skeletons of missing U.S. soldiers from past military operations. The two-dimensional fusion of postmortem computed tomography and antemortem chest radiography makes human identification possible [48] . In summary, these methods not only rely on manual analysis by experienced forensic experts, but are also difficult to deploy on a large scale.Automatic Methods. Some automatic identification methods have been proposed recently. Study [26] utilized the CXR before death database to match with CT scan image after death. It obtained an accuracy of 63% within 27 subjects and an average ranking of 10 (total 27, 37.04%) based on the extraction and matching of two types of features, the Bag of Words (BoW) and the Histogram of Gradients (HOG). Study [12] utilized morphological erosion to extract rib boundaries and employed Discrete Fourier Transform (DFT) to extract features, which led to an accuracy of 74.07% within rank 
discussion: 
 We have shown how machine learning techniques can be used to extract neural boneprint from skeletal information that can be used for identification. We hope this will open up new avenues for identity verification and shed new light on the development and cross-fertilization of different disciplines, including forensics and machine learning. There are many open questions that we and the community need to work on together, and the quality of Neural Boneprint features can be improved from many perspectives. First, as Neural Boneprint features are extracted from VRT and CXR images, the quality of bone detection and imaging is particularly important, and we will investigate more effective imaging methods [6] [7] [8] to improve image quality. Second, although the NBPs extracted from VRT and CXR images in this paper can already be used for identity verification, a natural idea is to use more data modalities (e.g., MRI and PET, etc.) to extract NBPs that contain more information. Third, the cross-modality translation could also be a diffusion-style [23, 44] approach. The backbone of cross-modality fusion is also various. The localization of thoracic skeletons in original VRT images and CXRs could be deep learning approaches, such as the YOLO [41] family. Ultimately, we must prioritize stringent privacy protection and ethical review. Hospital data frequently contain highly sensitive personal information and health records of patients. Therefore, when collecting and utilizing such data, it is imperative to adhere strictly to applicable laws and regulations, ensuring comprehensive protection of patients' privacy rights and interests. To sum up, we start an initial discussion on the proposed task and look forward to more works on social equity and judicial justice based on artificial intelligence. of 27 subjects. However, the dataset scale of these works is very limited. The performance is not satisfactory and not convincing enough. Study[30] realizes the attention localization and alignment of teeth for person identification by semantic segmentation and creating an atlas with landmarks. However, the annotations of the segmentation and the landmarks require manual creation, which is costly both in time and labor. Besides, this method relies on the spatial arrangement of teeth, while skeletonized corpses usually face loss of teeth. Recently, there has been a burgeoning interest in re-identifying CXRs. Some studies[32,37,55] achieve patient re-identification through CXRs for proper archival purposes. Identifying solely through CXRs appears to be more straightforward for our task. However, due to the presence of real internal organs in CXRs, these studies are insufficient to demonstrate the capability for skeletal matching. For those not fully decomposed corpses, the decomposition and expansion of organs can greatly affect comparison. For those skeletonized corpses, the simulated soft tissues can be misleading. 
: 
 (DNA) [54] , faces [9, 14, 45, 57] , fingerprints [36, 49] , palmprints [28, 47, 61] , irises [11, 49, 62] , etc. However, in forensics, when a body/corpse is extensively decomposed, intentionally damaged, or burned, soft tissue markers such as facial features and fingerprints are often neither viable nor extractable for identification. In addition, DNA extraction from such remains is unfortunately extremely challenging due to the progressive degradation of DNA over time [17] . Beyond the technical difficulties, the identification process is constrained by financial costs, time requirements, and the extent of available DNA databases. The efficacy of DNA analysis heavily relies on the presence of the individual's DNA sequence, or that of close relatives, within these databases. Without prior DNA sequencing and storage, identification becomes significantly hindered, illustrating the limitations of current forensic methodologies in certain scenarios. The intractability and accessibility of biological evidence (e.g. DNA) to skeletons hinders modern forensic identification for skeletons.Fortunately, there is an important but seldom-exploited basic fact that bones generally persist for a very long time, either in the corpse or in the skeleton. Forensic studies have shown that manual identification of persons through comparison of some skeletal imaging materials between antemortem and postmortem is practical [2, 13, 18, 21, 42] . However, these analyses generally depend on the experience of forensic experts. This raises a natural question: can we learn to identify a person directly from skeletal data? To do this, two fundamental questions need to be answered. First, what skeletal data should we use; second, what kind of features should we learn for identification, and how?In this paper, we demonstrate that thoracic skeletal data, specifically CXRs and VRT images, are useful for learning to identify directly. We then introduce the Neural Boneprint (NBP), a joint latent embedding extracted through a bidirectional cross-modality translation and contrastive deep learning on CXRs and VRT images, complementing traditional biological metrics. Experimental results  
 
 Appendix:  


Decision: All three reviewers are positive about the submission, which addresses the challenging problem of human identification from bones. Some concerns were raised but mostly resolved by the rebuttal, and the final recommendations remain unanimously positive. We agree with the reviewers on the merit of the work and hence recommend its acceptance to ACMMM.

Reviews: review: paper_summary: paper_summary: The author of this article proposes the use of the Neural Boneprint (NBP) to address person identification challenges. The experimental process, designed by the author, is grounded in task scenarios and real-world conditions. Specifically, a deep learning framework is proposed to extract NBP from chest X-rays (CXR) and volume rendering technology (VRT). This framework employs techniques such as Cross-modality Translation and Cross-modality Fusion to mitigate issues related to data imbalance, small intra-class variation, and large inter-class differences. Moreover, CXR is utilized to construct an NBP Bank. Experimental results demonstrate that this method achieves outstanding performance, offering novel insights for the development and integration of different disciplines, including forensic science and machine learning.
strength_weakness: strength_weakness: In this paper, a deep learning framework is employed to effectively address challenges such as data imbalance, within-class variation, and between-class differences, thereby enabling the learning of Neural Boneprint (NBP) for person recognition. Experimental results based on real clinical data demonstrate the efficacy of NBP in person recognition, achieving a Rank-50 recognition accuracy of 84.79%, which is twice the performance of the latest existing methods.

1. The experimental design of the study is commendable, as it leverages a substantial amount of CXR data to create a recognition bank and utilizes VRT to design task scenarios for identity recognition in forensic contexts.
2. There exists a significant modal difference between CXR and VRT, coupled with data imbalance issues. The framework proposed in the paper employs Cross-modality Translation and a bidirectional process to address the complex semantic relationship issues between these modalities.
3. Multi-modal fusion is utilized, incorporating “Others’ Real images,” “Translated images,” and “Real images” to tackle the problems of large intra-class variation and small inter-class differences.

Overall, the proposed framework demonstrates significant advancements in person recognition, offering promising avenues for future research in the intersection of forensic science and machine learning.
limitations: limitations: 1. The paper mentions, "we believe that this shared latent space enhances the ability of the model to capture the complex semantic interaction between VRT and CXR during cross-modal transformation," but this claim lacks empirical support or detailed elaboration.

2. The paper does not provide an in-depth explanation of the method used to address intra-class differences. Given the considerable modal differences between VRT and CXR images, it is crucial to offer a detailed account of how the model manages these discrepancies.

3. The dataset comprises only healthy individuals and does not consider the potential partial loss of NBP information post-mortem. Consequently, the accuracy and robustness of NBP for forensic identification have not been fully evaluated or validated.

### Suggestions for Improvement

1. Provide a more detailed description of the implementation details of cross-modal transformation and fusion techniques.
2. Conduct research and experiments using more diverse datasets.
3. Include additional performance metrics for a more comprehensive evaluation of the model.
4. In Figure 3, the identifier for “Others’ Real images” is too similar to that for Real images.
5. The description in section “4.2 Metrics” only explains the rationale for choosing Rank-K and Percentile Ranking Rate. Providing additional descriptions would help readers better understand these metrics.
paper_summary: paper_summary: This paper focuses on developing a novel method for forensic person identification using skeletal data. It introduces the concept of Neural Boneprint, a biometric-like identifier extracted from skeletal images. Specifically, it focuses on the thoracic skeleton, using chest radiographs (CXRs) and volume rendering technique (VRT) images. It is an interesting work that might have a wide range of effects.
strength_weakness: strength_weakness: It is an interesting and novel work that might have a wide range of effects.

The motivation is clear , and the proposed includes cross-modality translation and contrastive learning is theoretically sound and technically correct.

Experimental results on real clinical data demonstrate the effectiveness of NBP in identifying people, achieving a Rank-50 identification accuracy of 84.79%.
limitations: limitations: It is an interesting and novel work, I am quite concerned about the robustness of this job. Will abnormal bone morphology affect the identification results? Additionally, considering that metals in clothing will affect X-ray imaging, will this affect the identification effect?

Could authors provide some conclusions about interpretability, such as the model's Region of Interest(RoI).

The reviewer is much concerned about the fairness of the model, especially for different ages and genders.
paper_summary: paper_summary: This paper has proposed the first method for medical imaging-based ReID using rib bones. To fully adopt widely available X-ray images, the authors use the modality translation technique.
strength_weakness: strength_weakness: 1. The use of bones for ReID is novel and interesting.
2. The idea of adopting modality translation to use widely available X-ray images as reference datasets is reasonable and interesting, especially considering the impact of the COVID-19 pandemic. 
3. The experimental performance seems to be promising.
limitations: limitations: 1. How would ReID be affected if parts of the rib bones were fractured, broken, or missing? The authors could design experiments where some bone areas in the query images are randomly masked to testify this. This would effectively validate the method’s robustness in more real-world applications, considering that bodies might have suffered violent trauma before death.
2. What impact would different backbone networks have? Since the authors are the first to work on bone-based ReID, I hope they can test various widespread baselines. This could benefit the community.
4. The technical contribution seems somewhat insufficient. Considering the prior knowledge of rib morphology might be a better approach.
5. Add common ReIDmetrics such as mAP, CMC, INP, etc.
6. In Table 1, have the comparison methods all used your proposed translator? If they perform direct cross-modality retrieval, it seems to be not fair. Please supplement with comparisons where they also use the translator.
7. In Table 5, show both the reproduced and claimed results, and mark them differently for distinction.