Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a BERT-based model for solving the novel task of grounding sentences to 3D locations in a representation of a human male body. The authors induce training labels by automatically extracting a list of organs and their synonymous wordforms using the SciSpacy UmlsEntityLinker and extracting sentences containing at least one mention of an organ on their curated list. The ground-truth voxel coordinates are assigned using organ locations from a 3D atlas, and several BERT models are fine-tuned using a distance-based regression loss. The paper presents a novel task, is clearly written, and thoroughly explains their experimental choices and settings.

Issues:

1. One of the key premises of the paper is that physical proximity in the human body reflects functional and semantic relatedness to a significant degree. This statement needs more justification, especially since it is such an important part of the motivation for this work. Looking at a model of a human torso, it seems that the centroid of the left lung may be closer to the centroid of the stomach than to the centroid of the trachea. Similarly, the centroid of the stomach seems closer to that of the spleen and pancreas than of the large intestine.

2. It's not clear what is gained by performing this as a regression task to directly predict 3D coordinates instead of as a classification task that can then be transformed to 3D coordinates using the atlas (the ClassCenter model). The motivation seems to be that in cases when the classification model gets the wrong organ, the predicted 3D location is significantly worse than in the regression model (i.e. NVD-O for ClassCenter is higher). However, the classification model identifies the correct organ with much significantly higher frequency (i.e. IOR is higher). The overall effect is that even in the masked setting, the average per-sentence voxel distance is much lower for ClassCenter (i.e. taking the linear combination of NVD and NVD-O w.r.t. IOR).

3. The use case needs further justification. In what situations is specifying a position in a 3D atlas preferable to submitting a textual (or UMLS-keyword) query? Since the training data only uses the centroids of organs, is there a meaningful difference between querying for the centroid of the stomach vs. somewhere else within the organ? Are the relative positions of sentences embedded within the same organ semantically meaningful, or mostly a product of noise in the regression?

4. Representing abstracts as a point cloud composed of individual sentences seems like it would cause issues for the information retrieval task presented in section 7.2. What happens to the majority class of sentences that don't contain a mention of an organ, either implicitly or explicitly? Does the frequency of sentences containing an organ mention accurately represent the relevance of the document?
review_title: Novel task with under-developed motivation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes methods for learning a mapping between sentence vectors (as produced by the [CLS] token in BERT) and 3D points in the human body. The idea is that sentences that are about some anatomical site can be mapped to that position in the body for visualization. It is never completely clear from the paper why this needs to be learned, since the training pre-processing already is finding anatomical site mentions and essentially normalizing them to a set of body parts with known coordinates. The only thing I can figure is that they are using their pipeline as a way to generate labeled training data, and they intend for this to be applied to sentences without explicit anatomical site mentions. This is maybe supported by the fact that their training does include 50% masking of anatomical sites. In that case, there is a missing element here, which is that the vast majority of sentences in a document are not "about" any organ at all, so mapping them into organ space is meaningless. So for this algorithm to be applicable, one would first need to pre-process sentences somehow for whether they should go onto the organ mapping stage at all. And in any case, their example application presumably is just mapping the sentences that did have anatomical site terms. Overall, I think the logic of the approach needs to be spelled out more clearly. As it is, it feels like an interesting trick that one can learn this mapping but I just don't see why it's necessary.

Other issues:
- This mapping will surely generate "point clouds" around organs (i.e. sentences mentioning lung will be spatially distributed near the lungs rather than all mapping directly to the centroid of the lung). This may give an expert in anatomy some sense that these minor differences are anatomically meaningful where I believe it would be largely noise, and if some finer-grained meaning were there it would be impossible to distinguish from noise.
- Results tables are never referred to in the text
- There should be some discussion of what you "want" if there are multiple organ mentions. I certainly don't want some "average" position, but also don't want it to just show up in 1 position, it should probably show up in all 3 positions. But that is not a function anymore so not really something you can ask a neural network to do.
- When masking anatomical sites in the training data, do you ever use the same sentence in both the masked and unmasked form? I think it would be preferable with this dataset to randomly choose one form before training, rather than randomly choosing to mask each time an instance is loaded (which could result in different masking behavior for the same instance across epochs) because otherwise it could be memorized.
- I don't know if self supervised is the right term here. It's more like a clever way of generating supervised instances.
review_title: interesting exploration with unclear justification

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a self-supervised approach to ground sentences in COVID-19 articles to a 3D human atlas. This is the first study to use self-supervision to map sentences in medical articles to anatomical locations in the 3D atlas space. Unlike the prior works that utilized the atlas space, they formulated the grounding as a regression problem, thereby allowing a better idea about the semantic closeness of the sentences to the target body locations. The study has also proposed the potential use of this grounding approach for data visualization and document retrieval use cases.

Strengths:
-- The study employs a fully self-supervised technique for grounding sentences to a 3D human atlas. This is neat and has a lot of potential applications beyond COVID-19.
-- This work takes into account the prevalence of multiple organs in the same sentence while calculating the loss while training.

Major issues:
1. As discussed as one of the limitations that user study needs to be done to validate the query tools, as it is difficult to verify the actual performance of the document retrieval tool from the current version. Moreover, since the distance between the centroids of the point clouds is used to retrieve the articles relevant to a query, it is difficult to interpret whether this approach is useful as this does not take into account which of the sentences in the abstract are more close to the query. Some sentences in the article might just be providing background information and the corresponding organs might not be of interest to the user. To address this, it may be considered to have a way to identify the sentences of interest.

2. A brief discussion of interpreting the results mainly to indicate any potential relation between the impact of using the masked technique and the evaluation metrics used can be included.

3. Although the proposed method definitely obviates the expensive annotation part, it might be good to further compare the evaluations with a supervised model to validate that self-supervision performs better than a fully supervised approach. At the least, it would be good to compare the performance of these models with supervised models used on other datasets in the previous works for grounding medical text to 3D space.

Minor issues:
1. As already pointed out in the discussion that the models have only been trained on single sentences, it needs to be investigated whether a similar loss contribution strategy would work well when multiple sentences are included in the training process as organs appearing in different sentences might be very different from each other.
2. An evaluation of the methods to illustrate their performance on sentences having rare or infrequent organs may be reported.
3. How is the 8000 sentences mark determined for tackling the imbalance of different organs?
review_title: Interesting anatomical exploration of COVID articles using BERT-based grounding

Meta-review: Reviewers agreed there were a number of novel ideas here, though the motivation needs to be clarified in places. The authors mitigated the latter concern somewhat in their responses, but the case for the justification underlying their method could still be strengthened. In its present form, we think think the authors will benefit from presenting the work as an abstract at the meeting, and we hope that this meeting provides opportunity for them to receive additional feedback on the work. 
Decision: Accept (Abstract only)