Effective Techniques for Non-English Radiology Report Classification: A Danish Case Study
Authors: Alice Schiavone, Lea Marie Pehrson, Silvia Ingala, Rasmus Bonnevie, Marco Fraccaro, Michael Bachmann Nielsen, Desmond Elliott


Reviewer paper summaries: They propose automatic annotation of chest X-rays written in Danish in a low-resource setting.
They used two labels: a large rule-based silver label and a small gold-standard human-annotated label.
The label is defined in a hierarchical tree structure, and the target task is a multi-class and multi-label classification for the X-ray report.
They used several BERT-based models with different training settings, such as using only rule-based labels, transfer learning, ensembles, etc.
Summary of strengths: The proposed methods and the experimental setting are practical and can be applied to similar low-resource task settings in different languages.
The experimental setting is well-designed, and the results and the analysis of the experiments are convincing.
Summary of weaknesses: The proposed methods are well-known approaches without significant contribution to the novelty.
Comments, suggestions and typos: How the constraints of the hierarchical label were checked for the results of inference?
In Figure 5,  it is better to change the marker size or use a different shape for RE.
Overall assessment scores: 3
Metareview score: 3
Metareview: The authors propose an automatic annotation system for Danish chest X-rays in a low-resource setting. They use a hierarchical label structure for multi-class and multi-label classification. They utilize two types of labels: a large rule-based silver label and a smaller gold-standard human-annotated label. Silver labels are produced by a previous system based on regular expression.
Several BERT-based models with different training strategies are considered during the evaluation. The models were evaluated on the gold test set, showing improved performance with Danish-tuned models and further enhancement with ensemble methods.
Summary of reasons to publish: - The paper analyzes radiology reports in Danish (a low-resource language), focusing on Clinical Concept Extraction. The work is generally well-written, with practical methods and a well-designed experimental setting that can be applied to similar low-resource tasks in other languages.
- The experiments are convincingly analyzed, demonstrating the feasibility of the approach.
- Using real-world EHR data adds significant value, especially given the scarcity of clinical NLP resources in Danish.

Overall, the paper presents an interesting contribution in a relevant domain in a low-resource language and task. However, it lacks novelty, and the dataset (one of the main contributions) cannot be released due to legal reasons.