CLUE: A Clinical Language Understanding Evaluation for LLMs
Authors: Amin Dada, Osman Alperen Koraş, Marie Bauer, Amanda Butler Contreras, Kaleb E Smith, Constantin Marc Seibold, Jens Kleesiek


Reviewer paper summaries: CLUE is a new clinical benchmark composed of six tasks, which encompass: ICD-10 coding, information extraction, summarization, clinical reasoning and discharge letters simplification. They compared together 25 LLMs, selected for their diversity in terms of architecture, pre-training data, size and specialization. The overall accessement observe a degradation of performances for most of the biomedical-tuned models when applied on clinical tasks.
Summary of strengths: - The authors are conscious (L.144 and L.160) of the differences in term of semantic and lexicon of the clinical documents compared to biomedical or generalist ones.

- The evaluation metrics based on UMLS concepts is particularly well study for generative tasks such as "Summaries and MeDiSumQA" (L. 360) and offer a understanding of the error much more interesting than previous metrics like BLEU and ROUGE. The evaluation of the ICD-10 codes (L.367) is also well done since matching exactly the code will be difficult.

- The performance degradation analysis of the biomedical-tuned models is very interesting and show current limitations of the continual pretraining methods used by the community to adapt new models to the domains.

- The wide range of tasks is very interesting and cover the main type of tasks asked frequently by clinical practitioners.
Summary of weaknesses: - Its missing a link pointing where the dataset will be release.

- The authors underexplored the capabilities of the proposed evaluation setup used for generative tasks. I would be interesting to understand the gravity of the error made the models and maybe to compute a "harmfullness score" for each of them.

- The LongHealth task is very interesting but is under-explored. You are mentionning Levy et al. 2024, but you doesn't show how the LLMs used are affected with the length of your documents. Making the understanding of why it doesn't work very complicated, due to the lack of experiments on your data.

- Section 6.2 (L.559) doesn't any scientific value and require some experiments to get an idea is we can face similar issue here, since we doesn't know the type of data used by some models like Mistral 7B.

- You mentionned a comparison between BioMistral model based on "Mistral-7B-Instruct-v0.1" with in your case "Mistral-7B-Instruct-v0.2". Its not directly comparable.
Comments, suggestions and typos: - A lot of reference in the text to the appendix. You should remember that appendix aren't a continuation of the paper and that its not mandatory for reviewers to read them.

- The figure 3 is not meaningful enough to be part of the core of the paper.

- The concepts of Level 1 and 2 aren't clearly explained when you refer to them. It make the results complicated to understand.

- "wherein they repetitively produce the same sequences without reaching a conclusion" (L.505), its clearly the phenomenon of hallucination.
Overall assessment scores: 3.5