Contrastive Learning Penalized Cross-Entropy with Diversity Contrastive Search Decoding for Diagnostic Report Generation of Reduced Token Repetition  
[PERSON]  
1 State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China; [EMAIL_ADDRESS]  
[PERSON]  
2 School of Information and Communication Engineering, Communication University of China, Beijing 100024, China; [EMAIL_ADDRESS]  
[PERSON]  
2 School of Information and Communication Engineering, Communication University of China, Beijing 100024, China; [EMAIL_ADDRESS]  
[PERSON]  
1 State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China; [EMAIL_ADDRESS]  
Footnote 1: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Footnote 2: [[https://www.mdpi.com/journals/pdf/12/](https://www.mdpi.com/journals/pdf/12/)]([https://www.mdpi.com/journals/pdf/12/](https://www.mdpi.com/journals/pdf/12/))  
and semantics and utilizes natural language generation techniques to create a new and comprehensive text summary [2]. As a specific application of text summarization, diagnosis report generation (DRG) aims to make summaries and generate diagnostic reports according to the text description of medical imaging findings. It is part of the medical report generation [3] task which concentrates on using deep learning networks for generating diagnosis reports in terms of the medical image input. Clinically, medical imaging description and disease diagnosis are predominant in radiologists' daily works. These works are vitally important yet tedious and time-consuming. Accurate DRG from medical imaging description in an automated manner can decrease clinicians' workload dramatically, subsequently improving their routine efficiency. However, medical imaging diagnostic reports involve specific field vocabulary, complex organization structure and detailed visual description [4]. Due to the professionalism in disease diagnosis, treatment planning and therapeutic delivery, higher demands are required for the DRG quality, including precise comprehension of medical terminology, content understanding and reasoning capabilities, coherent diagnosis and ambiguity avoidance. Abstractive approaches for high-quality DRG have been developed. Traditional methods mainly rely on statistics and shallow learning, such as using maximum entropy models to predict words or constructing feature engineering and classifiers to generate key sentences. These methods might be unable to handle large-scale text document inputs [5]. On the other hand, significant progress has been made in the field of natural language processing (NLP) by using deep learning networks [6], such as recurrent neural networks (RNNs) [7] and long short-term memory (LSTM) networks [8]. One milestone comes from the attention mechanisms of Transformers [9] that build encoder-decoder-based sequence-to-sequence models, and essential messages in the input text are concentrated on. Later, as a cost-effective approach, a great deal of attention has been paid to pre-trained large language models (LLMs), such as bidirectional encoder representations from Transformers (BERT) [10] and generative pre-trained Transformer (GPT) [11]. Through pre-training on large-scale corpora, LLMs can effectively improve the performance in massive downstream tasks, including but not limited to clinical notes summarization [12], biomedical natural language tasks [13] and text-to-image generation [14], and LLMs outperform medical experts in clinical text summarization [15] that could help clinicians to focus more on patient care. Unfortunately, when transferring a pre-trained LLM to a specific application, semantic inconsistency of sentence embeddings has been massively witnessed from dull repetitions and undesirable text generation.