# PneumoNet: Artificial Intelligence Assistance for Pneumonia Detection on X-Rays
## 3 Pneumonet Framework
### XAI Module
Figure 5: The PneumNet framework works inside the AIHealth [37], providing the prediction of pneumonia for the uploaded X-ray and the generated medical report via LLM. The top image shows the report; the bottom one illustrates the chatbot functionality. Going into more detail on the employment of GPT-Neo in the Medical Report Module, GPT-Neo generates comprehensive medical reports using four structured prompt templates, ensuring consistent and detailed outputs. The templates are: (1) Introduction: \"Generate an introductory paragraph exceeding 15 lines for a medical report dated May 01, 2025. The diagnosis from PneumNet.onnx is [prediction] with confidence [confidence:.2f] and class probabilities {probabilities}, based on chest X-ray analysis using LIME, Grad-CAM, and LungFocusXAI (yellow: least used, orange: medium used, red: most used, red in lungs); (2) Findings: \"Generate a findings paragraph exceeding 15 lines for a chest X-ray with {prediction} diagnosis from PneumNet.onnx, confidence [confidence:.2f], and probabilities {probabilities}. Incorporate this XAI analysis: '{vlm_response}'. Include vital signs: [as above]. Explain how PneumNet and XAI methods identified features, their correlation with vital signs, and support for the diagnosis.\"; (3) Clinical Interpretation: \"Generate a clinical interpretation paragraph exceeding 15 lines for a {prediction} diagnosis from PneumNet.onnx, confidence [confidence:.2f], probabilities {probabilities}. Include XAI analysis: '{vlm_response}' and vital signs: [as above]. Discuss implications, next steps, and how XAI and vital signs inform treatment.\"; (4) Conclusion: \"Generate a concluding paragraph exceeding 15 lines for a {prediction} diagnosis from PneumNet.onnx, confidence {confidence:.2f], probabilities [probabilities}. Include XAI analysis: '{vlm_response}' and vital signs: [as above]. Summarise PneumNet's process, XAI significance, vital signs' impact, and follow-up steps.\" These prompts integrate prediction outcomes, confidence scores, class probabilities, XAI metrics (e.g., LIME superpixel importance, Grad-CAM heatmaps), and vital signs to produce cohesive narratives. The prompts incorporate a visual language model response for XAI analysis, detailing lung features (e.g., opacities, consolidations) and their alignment with the diagnosis. The code's error handling ensures robustness, logging failures and providing fallback responses if GPT-Neo fails. By explicitly defining these templates, the MRM ensures transparency in report generation, enabling reproducible outputs that combine technical precision (e.g., 0.85 confidence for \"Pneumonia Positive\") with clinical relevance (e.g., vital signs correlation). Future enhancements could refine prompts to include additional clinical metadata or fine-tune GPT-Neo on medical corpora, further improving the MRM's diagnostic utility and interpretability within the PneumNet framework. Finally, it is crucial to emphasise that the top image in Figure 5 also demonstrates that the X-ray image results show (or do not show) symptoms of COVID-19; this particular result is a part of a prior work published in [7], and the explanation of the COVID-19 classification method is the focus of this publication but can be consulted in the reference mentioned. Also, out of the focus of this publication is the detailed explanation of all the functionalities of AIHealth UI; for that, see [7]. Finally, it is important to stress that in [7], no functionality of automatic report or use of a chatbot has been implemented or made available. Algorithm 2 describes a web-based app for pneumonia detection and medical report generation. Once the classification is complete, the system generates a medical report using the GPT-Neo model (as mentioned before). The report is tailored to explain the implications of the detected condition in clear and concise language. This enhances the interpretability of the results, making it easier for users to understand the medical significance of the findings.