Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: null
review_content: I have read the rebuttal and other reviews. I agree that there are some ethical concerns with regard to not releasing the test dataset and associated patient demographics, as well as many concerns with potential misuse of the proposed technology for surveillance and similar tasks. As a result, authors promised to release a portion of the test set and detail the issues that were raised in the paper. 

In any case, these ethical concerns and limitations are inherent to most AI-based medical imaging technologies (and many of the general AI technologies too) today. On the other hand, the open discussion about these limitations that this paper has already elicited during the rebuttal period (and hopefully further at NeurIPS) would be of value to the ML community. I believe that the paper has a strong contribution being the first to propose to use NN for forensic dental identification. Given that neural networks are starting to be applied for forensic identification tasks, both the proposed approach and the clarifying discussion about limitations, ethics, and potential mis-use that the authors promised to add will be of great value to to this conference.

review_title: Post-rebuttal comments

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an extensive study on applying deep learning (including novel techniques) to dental forensic identification.

**Main Contributions:**

- An in-depth discussion on the challenges of the task and limitations of past works;
- Multiple techniques ranging from data augmentation to network architecture to tackle these challenges (with potential applications to other tasks);
- A comprehensive dataset covering many dental variations, to reflect a realistic scenario;
- An extensive and thoughtful ablation study that gives insights on the effectiveness and benefits of each major technical decision or proposed technique; and
- SOTA results in the task by a large margin vs. sensible baselines.
main_review: The paper presents extensive and thoughtful study on deep learning applications to dental forensic identification.

First, it presents a discussion on limitations of past work and challenges of this task, encompassing both the available datasets and previously used models. This discussion is grounded on insights from dental professionals, addresses short-comings from techniques from related fields (e.g. person reid), and lays the foundation from the proposed techniques. Additionally, the ablation studies carefully confirm these claims, with the proposed solutions individually outperforming across the board.

Second, it demonstrates a novel dataset that addresses many of the limitations of past work, e.g. by including many dental variations and scenarios. Certain scenarios that are not covered in the dataset are discussed later in the paper.

Third, it proposes a novel approach to leveraging deep learning to tackle this task. The techniques proposed range from data augmentation to network components. It cleverly combines approaches from diverse existing literature with original contributions, with justifications based on insights from dental professionals, and accompanying comprehensive ablation studies to validate their hypotheses.

Finally, it presents a careful evaluation of the work, including comprehensive ablation studies. The proposed technique achieves SOTA results that strongly outperform sensible baselines presented by the authors. The ablation studies corroborate the choice of the major individual components of the technique (e.g. data augmentation, objective function).  

The writing is generally clear, and the code is available. The dataset is based on a publicly available source, and another anonymized source (I believe for the review process).

All in all, the paper presents a thoughtful discussion, novel methods (with potential applications to other tasks), and a careful and comprehensive evaluation.

---

Minor:

- L301: missing/broken citations ("[]")
- L15: typo "effeteness"


limitations_and_societal_impact: The paper acknowledges and addresses the limitations of the proposed dataset, as well as the privacy risks of dental identification (which, like they discuss, are less risky vs. person reid and facial recognition). It might have been helpful to further discuss the potential impact of this technique in the context of law enforcement, which is prominent in this paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a new architecture FoID for forensic identification of individuals from dental imagery, a task often conducted by domain experts for identification at crime scenes/disaster sites. Given an image and a tooth segmentation map, they augment a straightforward CNN in four fashions: (i) hard attention on the regions of the scan where identifying information can be found, (ii) dental-specific augmentations (e.g. addition of a cavity) as opposed to general augmentations, (iii) the inclusion of three loss families (classification, contrastive, identification), and (iv) a transformer-based attention mechanism for focusing on unusual shapes of interest. In experiments, they outperform a series of benchmarks,  and ablations show the importance of each of the augmentations above. They include interpretations of the self-attention mechanism for explainability. 
main_review: The authors present an architecture that seems quite effective for forensic identification, on a relatively novel task for the machine learning community. No proof/theoretical claims are provided, and all empirical claims seem well-founded with fair and thorough baseline comparisons across multiple CNN architectures and ablations. The domain-specific augmentations are clever and thorough in creating effective data pairs. The provided attention maps further seem to indicate that the model is operating as expected. The paper is clearly written and easy to follow, and I appreciate the author's inclusion of code in their submission.

However, there is admittedly limited novelty of the methods and conclusions from this work , which may not make it most suitable for the NeurIPS community. Going through the contributions:
* Image augmentations have long been used as a way to augment data sets, and it seems expected that the more hand-crafted, tailored augmentations for the dental domain would outperform out-of-the-box standard approaches. 
* It is relatively standard to have multiple objective/objectives types in the machine learning community, and the authors themselves include the fact that the advantage of multiple objectives has already been shown in the closely related pedestrian re-identification domain. 
* While the transformer self-attention mechanism is in a slightly different position in the network than might be standard, there is now a wealth of work using transformer self-attention mechanisms (e.g. https://arxiv.org/pdf/2101.01169.pdf), including using them in combination with CNNs.  
As a result, it's not clear to me whether the paper necessarily provides generalizable conclusions beyond what's already existing in recent literature. 

As a minor note to the author, you likely mean to use “effectiveness” over “effeteness”
limitations_and_societal_impact: I do think a little more emphasis should be placed on discussing the privacy risks of re-identification, particularly in settings outside of dentistry. While dental images may be locked within medical records, the abstract says the proposed techniques could be repurposed for pedestrian re-identification, and given there is currently a large debate on the ethics around person re-identification, particularly due to bias, I think it is important authors give the issue more consideration in the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a framework for retrieval of dental panoramic X-Rays. 
To find regions of interest they focus on teeth which are extracted by semantic segmentation using a pretrained mask-RCNN as well as relevant areas extracted based on landmarks for which it is necessary to register every image to an atlas in a deformable way. All those ROIs are extracted and augmented in a domain-specific way (fillings, tooth extraction, etc), which serves as input to a CNN which is trained with an objective function that consists of three loss terms, added in an unweighted manner: a contrastive loss, an ID loss and a deep supervision loss.

They use a publicly available dataset for training, but test the proposed method on data collected by the authors, s.t. each image in the test set is to be retrieved from the union of the test and training set.

An ablation study on different modules of the framework is performed to test their contributions to the overall improvement of the proposed method.

main_review: The framework combines multiple state of the art methods in order to solve a specific and very challenging task.
It is interesting work in that there are several challenging subtasks that needed adaptation from their original publications to this kind of dental data which all come with their own challenges and hyper parameters (starting from the deformable registration to an atlas and the semantic segmentation to generate the input data).
However I am missing a novelty which is transferrable to other kind of data than this explicit application. I believe that this work is very valuable and of significant importance in the field of digital dental identification, but I am not convinced that it entails enough novelty from an ML perspective that is of general purpose to other fields of application. (If the authors can point out a novelty which is of a broader interest apart from dental identification, then this will also have to be reflected in the broad societal impact.) As the framework consists of many subtasks, there is only little space to provide all details needed for reproducibility in a conference paper like this and I believe a journal which is in theme closer to the paper’s application would be a more suitable publishing channel.

Questions to address:


It is unclear to me how the baseline experiments were conducted which are presented in Table 2. What exactly was done for e.g. ResNet-34+reconstruction, or ResNet-34+triplet, etc.? What was the pipeline for one such experiment?

I have some questions regarding the dataset. The paper says that UFBA_UESC was used to pretrain the mask-RCNN (line 132) for the semantic segmentation step to locate the teeth. Later, for the benchmark construction, the DNS Panoramic dataset is used as a training set. In [50], in which the DNS Panoramic dataset is introduced, it says that the original dataset, named “UFBA-UESC Dental Images data set” (published by Silva et al., ‘“Automatic segmenting teeth in x-ray images: Trends, a novel data set, benchmarking and future perspectives”, 2018) was modified by Jader et al. (“Deep instance segmentation of teeth in panoramic x-ray images”, 2018) to allow for instance segmentation, resulting in the dataset named “UFBA-UESC Dental Images Deep”. Silva et al. in [50] (“A study on tooth segmentation and numbering using end-to-end deep neural network”, 2020) then modified “UFBA-UESC Dental Images Deep” further, resulting in the “DNS (Detection, Numbering, and Segmentation) Panoramic Images”, which in turn was used as a training set in the proposed publication. So I wonder in what ways the UFBA_UESC dataset used for pretraining in the mask-RCNN step differs from the DNS dataset used for the overall training, and if the pretraining - in case of a big overlap of the two datasets - has much/any impact?

What size are the images in pixels and how do they need to be resized to fit the CNN encoder which was pretrained on RGB ImageNet images?
Is the atlas publicly available to which the images get registered?

Regarding the addition of the three types of loss in the final objective function: are they all of the same magnitude, or is one of the loss terms dominating over the other two?

Is the test set released?

What is meant by online mining on line 201 in this context?

Minor comments: formatting line 301;  typos or similar in lines: 143, 169, 177, 181, 184, 206, capitalization of embedding dim. line 207, 212, 224, 270/271, 282, 405,

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The work proposes to use deep learning for forensic dental identification, i.e., identifying living or deceased persons from their panoramic dental radiographs. It outlines the main, domain-specific challenges: lack of paired radiographs for metric learning, lack of annotations (e.g., bounding boxes of salient anatomic parts used by dentists), and visual variability. The paper proposes the new task, describes a set of domain-specific augmentations (based on anatomical variation), and suggests strategies to adapt existing networks to this domain.

By introducing forensic dental identification as a new ML task, the authors point out a new problem that could invite other researchers to design algorithms for. 
main_review: The paper proposed an interesting and novel task, and designs a state of the art network to solve it. I found the introduction and related work easy to understand and very insightful. The results section, however, was difficult to follow and had quite a bit of grammar issues. Given that the architectural findings that the authors discovered, what implication does this approach have to forensic dental identification and machine learning in general? 

Here are some specific questions:

- How important is it to use a panorama-retrained model?
- Please add visualizations and sample annotations of regions used for hard attention
- Please include a visualization of the proposed domain-specific augmentations
- Include an analysis of false negatives, as this may be an important limitation of using this approach. How is the performance in comparison to identification performed by humans?

I was not able to download the datasets, and do the authors plan to release it? I think this would be a critical task because the goal of the paper is to encourage more exploration in this specific domain.

Typos/corrections:
- Line 15: “effeteness” -> effectiveness
- Line 47: “In specific,…” -> bad grammar
- Line 147: “In specific” -> Specifically
- Line 152: “does not relay” -> does not rely
- Line 212: “we incorporates” -> we incorporate
- Line 270: “few study” -> few studies
- Line 312: “We can obviously see” -> We can see that. Why obvious?
- Line 325: “effeteness” -> effectiveness


EDIT (08/18): I have read the other reviews and authors' rebuttal. I thank the authors for clarifying my concerns. I second other reviewers' concern about adding additional discussion about ethics, limitations, and data description. Please see main comment. I do, however, think that this paper will be very valuable to the ML community as a practical application of current technologies to a new task.
limitations_and_societal_impact: While the paper proposed a very novel ML application, it is also very limited to this application. This is an inherent limitation and while there is little the authors can do about this, they should discuss more specifically how their findings can affect other tasks such as person re-identification, data augmentation etc. 

I appreciate the thorough description of the limitations and broader impact. However, due to privacy concerns, how likely would the proposed system be used in practice?
review_title: null

Meta-review: This paper generated significant debate, both on technical contribution considerations (more general methodology) and ethics considerations. Thank you to the reviewers and authors for the detailed discussion. 

On the ethics considerations, the authors have responded saying they will include a more detailed discussion of potential misuses, and importantly, results across different subpopulations. The authors should include as many of these subgroups as possible (across gender, race and multiple age brackets) along with discussion on any places where the method performs below average.

On the technical considerations, a big debate was between whether the methods proposed are too specialized to this application. After careful review of the discussion, I think this paper can be viewed as an applied contribution, that should be of interest to others in the community who are studying this application.
Decision: Accept (Poster)