Automatic Mapping of Anatomical Landmarks from Free-Text Using Large Language Models: Insights from Llama-2
Authors: Mohamad Abdi, Gerardo Hermosillo, Halid Ziya Yerebakan


Reviewer paper summaries: This paper explores the use of Llama-2 large language models (LLMs) to automate the mapping of anatomical landmarks from free-text radiology reports to spatial positions in medical images. The study demonstrates that Llama-2 models can linearly represent the spatial positions and sizes of anatomical landmarks with considerable robustness to different prompts. Through experiments, it was found that deeper layers of the models provide more accurate spatial representations. The results highlight the potential of LLMs to enhance the efficiency and accuracy of medical imaging workflows.
Summary of strengths: ``Novel application of LLMs``: The paper introduces a novel application of LLMs in automating the mapping of anatomical landmarks from free-text descriptions to spatial coordinates in images. This represents a significant advancement in leveraging LLMs for practical and impactful tasks in healthcare.

``Insightful empirical results``: The study provides strong empirical evidence that Llama-2 models can linearly represent anatomical landmarks' spatial positions and sizes. The results are robust across different model sizes and prompting strategies, highlighting the potential reliability of this application.
Summary of weaknesses: ``Vogue Task Definition and Limited Dataset``: The task remains vague to me; adding examples in the Introduction or Method sections would benefit the audience. The study uses a relatively small dataset of 117 anatomical landmarks, and it is unclear whether there is a vocabulary limit on possible anatomical landmarks or if just 117 landmarks were picked for this task. If it’s the latter, a broader set of landmarks could provide stronger evidence for the LLMs' spatial representation capabilities. Additionally, the preprocessing of these landmarks is not described in the paper.

``Lack of Strong Baseline Methods``: While the paper provides a baseline comparison using lexical similarity measures, it would benefit from a more detailed comparison with other existing automated methods. The probes generated from Llama-2 are then used as input for the linear regression model. More widely-used embeddings, such as Word2Vec, sentence embeddings, and BERT embeddings, could be used to encode anatomical landmark names and create stronger baselines. Additionally, I'm confused about the choice of using a non-linear (MLP) model for comparison, as the linear relationship between anatomical landmarks and their spatial coordinates seems intuitive. More work should spotlight the encoding part from my perspective.

``Underutilization of In-Context Learning``: The method underutilizes the few-shot learning capabilities of LLMs. Integrating known landmark coordinates into the prompts could provide additional context and potentially improve the accuracy of the predictions. If the work focuses on the power of LLMs, more advanced in-context learning methods could be considered for generating representative probes.
Comments, suggestions and typos: ``Suggestions``:

Even if the short paper is limited in 4 pages, authors can use Appendix to give more context.
For example, prompts, examples of anatomical landmarks could be listed in Appendix. 

``Small typos``:

Line 113: landmarks . -> landmarks.
What is n in line 146?
Add link refs for scikit-learn and pytorch libraries
Line 240: models -> models.
Overall assessment scores: 2.5