A Survey of Evaluation Methods of Generated Medical Textual Reports
Year: 2023

Abstract:
Medical Report Generation (MRG) is a subtask of Natural Language Generation (NLG) and aims to present information from various sources in textual form and synthesize salient information, with the goal of reducing the time spent by domain experts in writing medical reports and providing support information for decision-making. Given the specificity of the medical domain, the evaluation of automatically generated medical reports is of paramount importance to the validity of these systems. Therefore, in this paper, we focus on the evaluation of automatically generated medical reports from the perspective of automatic and human evaluation. We present evaluation methods for general NLG evaluation and how they have been applied to domain-specific medical tasks. The study shows that MRG evaluation methods are very diverse, and that further work is needed to build shared evaluation methods. The state of the art also emphasizes that such an evaluation must be task specific and include human assessments, requesting the participation of experts in the field.

heading: 1 Introduction
text: Medical Report Generation (MRG)1 (Chen et al., 2022) is a subfield of Natural Language Generation (NLG) and aims to present information from various sources in textual form to synthesize salient information, with the goal of reducing the time spent by domain experts in writing medical reports and providing supporting information for decisionmaking.
MRG includes all systems used to generate medical documentation, such as the generation of radiology reports (Chen et al., 2021), discharge summaries or SOAP notes (Krishna et al., 2021), etc.
The evaluation of automatically generated texts is important for the validity of the systems, especially in the era of ChatGPT and its increased
1It is also called ‘Clinical Note Generation’.
usage in medical domain (Ma et al., 2023). However, it has often been reported that the two main approaches of NLG evaluation – human-based and automatic-based – need to be improved (Reiter and Belz, 2009; van der Lee et al., 2019). On the one hand, the use of automatic metrics for system quality assessment has been criticized for two main points: they are uninterpretable and do not correlate well with human evaluations (Qader et al., 2018; van der Lee et al., 2019). On the other hand, the deployment of human evaluation is sometimes too complex. Indeed, crowdsourcing solutions are not always reliable or adequate while using experts, when available, can lead to high costs. Furthermore, there is a lack of unified framework/criteria (van der Lee et al., 2019).
Although there have been many surveys on NLG evaluation (Gkatzia and Mahamood, 2015; Amidei et al., 2018; van der Lee et al., 2019; Sai et al., 2022), there is no systematic study on report generation evaluation in the medical domain. It is worth mentioning Messina et al. (2021), which has done survey work in the area of automatic report generation from medical images, including an analysis of evaluation methods.
In this paper, we focus on MRG tasks and their evaluations. We include more than 20 papers in this study, classified into two broad categories: text-totext and data-to-text. We summarize the evaluation methods currently in use and make recommendations for future evaluation of MRG systems.

heading: 2 Medical Report Generation

heading: 2.1 Paper Search and Selection
text: To perform a literature review of MRG evaluation, we first followed the paper list introduced in a survey paper on dialogue summarization (Feng et al., 2022). We then extended our search to search engine such as Google Scholar. Papers reviewed in this study were primarily from the major NLP con-
447
ferences, including ACL, EMNLP, NAACL, etc. In addition, we also included some articles from journal and domain-specific workshops, including NLPMC, ClinicalNLP, etc. 2

heading: 2.2 MRG Systems and Applications
text: Since the seminal work of Kukich (1983), there have been several kinds of medical report systems and applications, such as the generation of psychiatric case notes (Kazi and Kahanda, 2019), the generation of consultation notes from transcripts (Papadopoulos Korfiatis et al., 2022), the generation of radiology reports (Chen et al., 2021), nurse-patient summaries (Liu et al., 2019), counseling (conversation) summarization (Srivastava et al., 2022), discharge summaries or clinical notes (Krishna et al., 2021), and even data augmentation for other medical tasks (Kocabiyikoglu et al., 2021).
Joshi et al. (2020) provided a general definition of a medical report in the case of medical dialogue summarization: “the medical report captures and summarizes the important parts of the medical conversation necessary for clinical decision-making and subsequent follow-ups.”
Despite the diversity of their tasks, structures and audiences, the main characteristics of MRG remain similar, namely the use of the documentation and the subsequent use of the diagnosis, which can also be used for administration and by institutions, subsequently referenced by clinicians and retained by patients. The main objectives of such systems in clinical practice are to reduce the time spent by clinicians on manual writing and facilitate medical decision-making.

heading: 2.3 Main NLG approaches to MRG
text: According to the different types of input sources, MRG can be divided into two categories: 1) Textto-text, e.g. summarizing medical dialogues; 2) Data-to-text, e.g. automatically generating reports from medical images or other data. After classifying the different approaches according to the input, we then further categorized different works in the literature according to their tasks, with examples in Table 1.

heading: 2.3.1 Text-to-text
text: There are three main tasks in the Text-toText category: 1) summarizing medical dia-
2The list of papers about MRG is available at https://github.com/yongxin2020/ Medical-Report-Generation-Papers
logues/conversations, including spoken conversations and online medical conversations; 2) summarizing hospital stays/hospitalizations; and 3) summarizing medical reports, where the original reports may come from different domains, such as radiology reports or general clinical reports.
The most common work in text-based MRG is that of summarizing medical conversations, where the input source can be either transcripts of clinician-patient spoken conversations (Kazi and Kahanda, 2019; Enarvi et al., 2020; Liu et al., 2019; Krishna et al., 2021; Zhang et al., 2021; Molenaar et al., 2020; Srivastava et al., 2022; Lacson et al., 2006; Moramarco et al., 2022; Yim and Yetisgen, 2021); or online medical conversations (Chintagunta et al., 2021; Nair et al., 2021; Joshi et al., 2020; Song et al., 2020; Chen et al., 2022).
Regarding the summarization of hospital stays, some work (Di Eugenio et al., 2014; Acharya et al., 2016) used both physician discharge notes (free text) and the structured nursing documentation (such as nursing plans of care) to generate a unique summary. Others generated summaries from longform hospital admissions (Adams et al., 2023).
Moreover, work has also been carried out on summarizing medical reports. For example, Moramarco et al. (2021) used the MTSamples dataset to fill automatically the ‘description field’ of a medical report based on the information present in the overall report. In addition, radiology report summarization (Zhang et al., 2020; Karn et al., 2022) is intended to produce a concise and easily comprehensible ‘IMPRESSIONS’ section from the rest of the radiology report. The ‘IMPRESSIONS’ section of a radiology report is considered a summary of the radiologist’s reasoning and conclusions, which helps the referring physician confirm or exclude certain diagnoses (Karn et al., 2022).

heading: 2.3.2 Data-to-text
text: As presented in Table 1, there are different tasks in the Data-to-Text category: 1) generation of reports from medical images, such as radiology images, brain image data.; 2) generation of text summaries from intensive care data; and 3) generation of medical reports from multimodal inputs; 4) other applications such as the generation of tailored smoking cessation letters based on responses to a smoking questionnaire (Reiter et al., 2003).
In order to meet the growing demand of imagebased diagnosis from patients using artificial in-
telligence and applying image captioning to the medical field, radiology report generation is the subject of continuous work and growing interest from researchers, which aims to describe radiology images with professional quality reports (Chen et al., 2020, 2021; Lovelace and Mortazavi, 2020; Nooralahzadeh et al., 2021; Yan et al., 2021; Qin and Song, 2022; Miura et al., 2021; Liu et al., 2021). Such research has also been applied to generation of clinician reports from brain imaging data (Jordan et al., 2014).
Besides images, the summarization of physiological data as also been the subject of research. In the BabyTalk project, Portet et al. (2009) presented a prototype that generates textual summaries of about 45 minutes of continuous physiological signals and discrete events. Their evaluation with physicians showed that text summaries could be an effective decision-support aids for clinicians.
To cope with the high workload due to the time required for proper documentation, Maas et al. (2021) presented a real-time automated report of the interaction between care provider and patient, taking multimodal inputs that include audio, video, and sensor data from medical consultations, and in-
troducing knowledge graphs – the Patient Medical Graph. They used speech and action recognition technology to first transform multimodal inputs into text before formally representing them and generating reports.

heading: 3 Evaluation in NLG
text: In this section, we will briefly introduce automatic evaluation in NLG and then will look at human evaluation with some current practices.

heading: 3.1 Automatic Evaluation
text: Automatic evaluation is popular because it is cheap and fast and it is widely used in benchmarking activity and for system development.
There is a wide range of automatic evaluation metrics used in NLG (Sai et al., 2022) and we will restrict to the two most popular: 1) the corpusbased metrics and 2) the trainable metrics. The corpus-based metrics rely on a set of reference texts (i.e. gold standard outputs) to which system outputs are compared. For instance, it can be based on n-grams, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004); or on edit distance: WER (Woodard and Nelson, 1982), TER (Snover et al.,
2006), etc. Most automatic metrics require gold references, but these are not always available. Reference-less metrics, where neural models are trained to predict human ratings from texts (e.g. regression models trained on ratings data), are getting more and more attention recently. For example, BLEURT (Sellam et al., 2020) is a learned evaluation metric for English to predict human judgments. It relies on the BERT model using unsupervised techniques with millions of synthetic examples. There are also metrics based on question-answer pairs on a given source document (Scialom et al., 2021; Rebuffel et al., 2021), for example QuestEval (Scialom et al., 2021) uses pre-trained models to assess if two different inputs contain the same information. Note that QuestEval can also be used with references.
To summarize, ROUGE scores assess the similarity between candidates and references based on the overlap of unigrams, bigrams, and the longest common sequence, likewise for BLEU; while BLEU focuses on precision, ROUGE focuses on recall. BERTScore evaluates the similarity between candidates and references at token level, using contextual embeddings from BERT, while QuestEval assesses whether a summary contains all the relevant information from its source document and BLEURT attempts to model human judgments.
However, automatic evaluation metrics have their limitations and do not sufficiently reflect human judgments of system performance (Novikova et al., 2017).

heading: 3.2 Human Evaluation
text: Human evaluation is considered the most informative form of evaluation of NLG systems, but it can be expensive and time-consuming since qualified human evaluators have to be recruited. Hence, human evaluation is difficult to scale up unless using crowd sourcing approaches but these are difficult to apply in medicine for expertise and privacy reasons.
There are several commonly used methods for human evaluation, including the Likert scale scoring and pairwise comparison for general text generation, as well as Pyramid and binary factuality evaluation specifically designed for summarization (Gao et al., 2023). Some other methods consist in evaluating how much information can be extracted back from the text in a formal form (A. Baez Miranda et al., 2015).
It has been argued that human evaluation approaches are difficult to compare (van der Lee et al., 2019; Belz et al., 2020) since different tasks and criteria are used (with different names). Furthermore, only a small number of papers provide full details of human evaluation experiments (Belz et al., 2020). Howcroft et al. (2020) concluded that due to a pervasive lack of clarity in reports and extreme diversity in approaches, human evaluation in NLG presents as extremely confused in 2020, and that the field is in urgent need of standard methods and terminology.
In addition, van der Lee et al. (2019) provided an overview of best practices in human evaluation of automatically generated text based on papers published at INLG (N=51) and ACL (N=38) in 2018, and released a list of best practices on 7 different topics: general, criteria, sampling, annotation, measurement, design and statistics.

heading: 4 Evaluation for Text-based Medical Report Generation Systems
text: In the following subsections, we summarize the automatic measures and human evaluation used in the literature in Table 2.

heading: 4.1 Automatic Metrics
text: We divide automatic metrics into two categories: text quality and medical concept correctness, for medical correctness there are two subcategories: those based on reports and those for auxiliary or intermediate tasks.

heading: 4.1.1 Text Quality
text: For automatic text quality assessment, there are word-overlap-based metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002), embedding-based metrics such as BERTScore (Zhang* et al., 2020); learned evaluation metrics like BLEURT (Sellam et al., 2020); and evaluation metrics which rely on question answering like QuestEval (Scialom et al., 2021).
ROUGE (Lin, 2004) has been widely used in MRG tasks: some papers reported only ROUGE-L F1 score (Joshi et al., 2020; Enarvi et al., 2020; Nair et al., 2021), while some others reported ROUGE1, ROUGE-2, and ROUGE-L scores (Song et al., 2020; Yim and Yetisgen, 2021). Yim and Yetisgen (2021) reported BLEU (Papineni et al., 2002) in addition to ROUGE performances across different note sections.
Commonly used automated metrics, such as ROUGE and BLEU, have their limitations and are known to correlate poorly with human evaluations
(van der Lee et al., 2019). Therefore, other measures such as QuestEval (Scialom et al., 2021) and BLEURT (Sellam et al., 2020) which can correlate
better with human judgements are used, Srivastava et al. (2022) used these two scores in addition to ROUGE. In addition to ROUGE and BLEURT, Ben Abacha et al. (2023) also reported BERTScore.

heading: 4.1.2 Medical Correctness: Report-based
text: In the study by Joshi et al. (2020) two measures are defined: Medical Concept Coverage and Negation Correctness. The former captures the coverage of medical terms in the predicted summaries to the gold standard reference, while the latter identifies the negated status of medical concepts. In the healthcare domain, it is crucial to ensure highquality results in terms of accurate usage of medical terms and capturing negation.
The evaluation of Concept involves using specific and in-house extractors and Named Entity Recognition (NER) models. They refer to domainspecific knowledge and compare the match of extracted concepts to standardized health and biomedical vocabularies, such as the Unified Medical Language Systems (UMLS). Several studies have utilized concept correctness measures, such as F1score, precision, recall, and false positives, at various levels of granularity, including the report level and section level.
For instance, Joshi et al. (2020) used an inhouse medical entity extractor to match concepts in the summary to UMLS, and they used Negex (Harkema et al., 2009) to determine negated concepts. Medical concepts in the predicted summary that were not present in the original conversation would be false positives, and vice versa for false negatives. Among the concepts present in the predicted summary, they assessed precision and recall to see whether the predicted negation was accurate for the decoded concepts and computed a Negation F1. The set of automatic metrics proposed was then used in several works (Chintagunta et al., 2021; Nair et al., 2021).
If in-house entity extractor to match concepts in the summary to UMLS have been frequent (Soldaini and Goharian, 2016; Joshi et al., 2020; Zhang et al., 2021), entity extraction using machine learning has appeared recently, which is even more specific to the task. For instance, Enarvi et al. (2020) employed a machine learning-based clinical fact extractor to measure factual correctness by extracting medical facts from both the predicted reports and the ground-truth reports, such as conditions and medications, as well as their attributes such as body part, severity, or dosage, then calculat-
ing the F1 score from these two sets. To compute Concept-F1, Chen et al. (2022) used the medical entity extractor – BERT-CRF (Devlin et al., 2019) trained on their NER task to match entities in the predicted summary to the reference summary.
Similarly, Ben Abacha et al. (2023) used "factbased metrics (Fact Scores)", which is a machine learning-based medical fact extraction system. The Fact Score metric measures the F1-score of medically relevant facts extraction, is used to assess the factual consistency of the generated summaries. The Fact-based metrics consist of two variants: Fact-Core, which relies on the extraction of seven core fact attributes, and Fact-Full, which combines these core facts and five additional attributes.
In addition, there is also work combining the two approaches to extract concepts: Zhang et al. (2021) extracted medical relevant concepts via one of two systems: their in-house rule-based system and quickUMLS (Soldaini and Goharian, 2016) – a Python implementation of UMLS. Their rulebased system was found to be effective in capturing symptom-related findings in clinical reports, and quickUMLS is capable of extracting a wide scope of medical findings such as symptoms, diseases, medication and procedures.
Moreover, Molenaar et al. (2020) measured the quality of the dialogue summarization pipeline for healthcare reporting by establishing the number of items included in the generated and gold standards, using precision, recall and false positives (FPs) as metrics. They followed the SOEP/SOAP format – Subjective (S), Objective (O), Evaluation (E) / Assessment (A) and Plan (P) – commonly used by general practitioners in the Netherlands. It appears that they manually calculated the number of items included in each section of the SOEP format for the eight reports generated.
However, concept-based evaluation can have its own limitations, particularly with regard to false positives errors, Zhang et al. (2021) employed filtering rules to attempt to mitigate this issue.
Additionally, Chen et al. (2022) reported Regexbased Diagnostic Accuracy (RD-Acc), which measures the model’s ability to diagnose the disease. Their reference reports written by annotators contain six parts, RD-Acc is calculated using the regexbased approach based on the diagnosis part. They calculated for what percentage of the generated reports, the content of their diagnosis part contains the actual disease text or key concepts.

heading: 4.1.3 Medical Correctness: Auxiliary or Intermediate Tasks
text: Another subcategory of automatic measures of concept correctness is those that evaluate auxiliary or intermediate tasks, there are two types: classification of electronic health record (EHR) categories and utterances classification.
To generate case notes from digital transcripts of doctor-patient conversations, Kazi and Kahanda (2019) divided the task into two subtasks: (1) predict semantic topics for segments of the transcripts (EHR categories) and then (2) generate a more formal version of the text that goes into the corresponding section of the EHR form. They used the AUROC (Area Under the ROC Curve) scale (Bewick et al., 2005) to assess their first task of predicting EHR categories, which could be any of the following: Client Details, Chief Complaint, Family History, Social History, Medical History and Others. Correct prediction of EHR categories could be useful for subsequent formal text generation.
For utterances classification, there are different types of classification such as classifying noteworthy utterance (Krishna et al., 2021), label prediction for medical conversation utterances (Song et al., 2020), and dialogue turn classification (Lacson et al., 2006).
In detail, Krishna et al. (2021) evaluated the multi-label classification of noteworthy utterances that are relevant to each summary section before clustering related utterances and generating one summary sentence per cluster. Their modular summarization technique outperforms its purely abstractive counterpart, producing much more factual and coherent sentences. Besides, Song et al. (2020) first identified two types of utterances (problem statements and treatment recommendations) and then generated summaries, they showed that for the particular dataset used, high-quality summaries can be generated by extracting these two types of utterances. Thus, in addition to reporting ROUGE scores, they also reported the precision, recall, and F scores of the predicted labels for utterances of medical conversations, compared to the standard labels. In addition, Lacson et al. (2006) also measured precision, recall, and F measure of dialogue turns classification.

heading: 4.2 Human Evaluation
text: The differences between intrinsic and extrinsic evaluation lie in the fact that the former aims to evaluate
the properties of the system’s output (Graham et al., 2018; Ji et al., 2022), while the latter aims to examine the extent to which the system accomplishes the overarching task for which it was developed. Of the 19 articles reviewed on text-based MRG, 9 included human evaluation (47%), and only 3 of them (16%) included extrinsic evaluation.

heading: 4.2.1 Intrinsic Approaches
text: The intrinsic human evaluation of generated reports comprises two categories as for automated metrics: text quality and medical correctness.
Text quality is important in MRG as in general NLG output. For text quality, a wide variety of properties can be considered and various linguistic parameters can be used, e.g. relevance, consistency, fluency, coherence, missing, hallucination, repetition and contraction. As an example, Srivastava et al. (2022) used four standard linguistic parameters: relevance (selection of relevant content), consistency (factual alignment between the summary and the source), fluency (linguistic quality of each sentence), and coherence (structure and organization of summary). In addition to these commonly used and well-studied criteria, the evaluation of MRG also concludes other medical correctness criteria, such as factually correct and medically relevant information (Joshi et al., 2020; Chintagunta et al., 2021), which are specific to MRG tasks. As another example, Ben Abacha et al. (2023) performed expert-based manual evaluation using NLG criteria such as Fluency and Non-redundancy, and medical criteria such as Critical Omissions, Hallucinations, Correct Facts, Incorrect Facts based on fact extraction.
Furthermore, depending on whether evaluators assess the output directly or by comparing different texts, intrinsic human evaluation can be classified into direct and relative evaluation. As for the articles involving human evaluation, they all used at least direct evaluation, i.e. the evaluators judged the generated texts directly on a defined scale. Some authors also performed relative evaluation in addition to direct evaluation: Joshi et al. (2020) and Chintagunta et al. (2021) performed a comparison task in which, given two summaries generated by different models and the associated dialogue, annotators had to choose which summary was better, they could also choose "both" and "none". Yim and Yetisgen (2021) ranked the four systems against each other, with 1 being the best, in addition to evaluating each system independently with a score
from 1 to 5 for the categories relevancy, factual accuracy, writing-style, completeness, and overall.
In general, MRG outputs are evaluated at the report level, however, depending on the design of the model, some are additionally evaluated at the sentence/section/part level. For example, Krishna et al. (2021) divided SOAP notes into several subsections: Family Medical History, Past Surgical History, Chief Complaint, etc. Therefore, they evaluated the generated SOAP notes in two ways: 1) SOAP note sentence level and 2) SOAP note level.

heading: 4.2.2 Extrinsic Approaches
text: As for extrinsic human evaluation: to evaluate generated summaries, a team of mental health experts used clinical acceptability framework (Sekhon et al., 2017), which includes six parameters: affective attitude, burden, ethicality, coherence, opportunity costs, and perceived effectiveness (Srivastava et al., 2022). In addition, to perform a task-based evaluation and measure the usefulness of summaries for preserving important information in the medical setting, Lacson et al. (2006) asked physicians and nurses to create a list of key questions based on topics that commonly arise between hemodialysis patients and caregivers, and then asked five physicians to answer each of the six "yes/no" questions using each of 40 dialogues. Furthermore, in a study evaluating the correlation between human evaluation and automatic metrics in consultation note generation, Moramarco et al. (2022) asked 5 clinicians to post-edit generated notes and extract all errors.

heading: 4.2.3 Presence of Domain Experts
text: Most of the articles we reviewed that included a human evaluation involved domain experts, such as doctors serving patients on their telehealth platform (Chintagunta et al., 2021; Joshi et al., 2020), five licensed physicians (Lacson et al., 2006), three general practice physicians (Moramarco et al., 2021), an annotator with a medical degree (Yim and Yetisgen, 2021), etc. Sometimes, the expertise of the annotators is not specified, e.g. "trained human annotators" (Krishna et al., 2021).
We also note that of the 9 articles including human evaluation, 5 of them reported Inter-Evaluator Agreement: three of the medical dialogue summarization articles (Zhang et al., 2021; Moramarco et al., 2022; Ben Abacha et al., 2023), and two medical (report) summarization articles (Moramarco et al., 2021; Karn et al., 2022). It would be prefer-
able to indicate Inter-Evaluator Agreement in the presence of several annotators.

heading: 5 Conclusion
text: Automating medical report generation can save time for experts and provide crucial information for decision-making. However, the evaluation process is necessary for validation and adoption of MRG systems in the real world. Due to the specificity of domain-specific NLG tasks like MRG, their evaluation requires more investigation and subtlety.
MRG evaluation shares similarities with general NLG evaluation, but it differs in its focus on domain knowledge and task-specific concerns, especially in the assessment of conceptual accuracy of medical concepts. However, the question of which medical facts to pay attention to (correlation, consensus, etc) is an open question, requiring close collaboration with experts in the field.
In addition, the evaluation of MRG systems requires both intrinsic and extrinsic evaluation. Intrinsic evaluation focuses on properties of the system’s output, while extrinsic evaluation involves professional experts in the design acceptability process, developing a list of key questions, and post-editing. Future research should prioritize extrinsic evaluation, particularly in scenarios where references are unavailable, and developing efficient, medical task-specific automated measures.
Limitations
In this work, we studied only the evaluation of textual medical report generation from both automatic and human evaluation perspectives, but we did not study the evaluation of data-to-text medical report generation, which has its own specificities.

heading: Acknowledgements
text: This research was carried out as part of the THERADIA project (funded by BPI) and was partially supported by MIAI@Grenoble-Alpes (ANR-19-P3IA0003) and the COST action Multi3Generation (CA18231). We would also like to thank the anonymous reviewers for their insightful comments.

heading: A Appendix
text: A.1 Review Structure We reviewed the MRG articles following the criteria described in van der Lee et al. (2019), the attributes examined are presented in Table 3.
A.2 Resources for Text-based MRG We identified six datasets used in text-based MRG and summarized them in Table 4. Song et al. (2020) collected medical conversations from online platforms. MTSamples (Moramarco et al., 2021) were also collected from a community platform website. Srivastava et al. (2022) extended data from the publicly available counseling conversation dataset - HOPE (Malhotra et al., 2022), which takes place between therapist and patient. We observed that mock conversations were mentioned several times: Kazi et al. (2020) used transcripts from two different sources to generate audio recordings of simulated doctor-patient conversations, Papadopoulos Korfiatis et al. (2022) simulated primary care consultations. Recently, Ben Abacha et al. (2023) introduced a new collection of simulated doctor-patient conversations from publicly available clinical notes and corresponding clinical summaries.

heading: Future Work
text: Future research should prioritize extrinsic evaluation, particularly in scenarios where references are unavailable, and developing efficient, medical task-specific automated measures. Limitations
In this work, we studied only the evaluation of textual medical report generation from both automatic and human evaluation perspectives, but we did not study the evaluation of data-to-text medical report generation, which has its own specificities