Hugging Face's logo

ECOFRI
/
CXR-LLAVA-v2

Feature Extraction
Transformers
PyTorch
Safetensors
CXR-LLAVA
custom_code
Model card Files Community
2
CXR-LLAVA-v2
File size: 3,364 Bytes 
88475f0
 
 
80f8630
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
---
license: cc-by-nc-4.0
---

# CXR LLaVA
https://github.com/ECOFRI/CXR_LLaVA


### Multimodal Large Language Model Fine-Tuned for Chest X-ray Images
CXR LLaVA is an innovative open-source, multimodal large language model specifically designed for generating radiologic reports from chest X-ray images.

-   **Arxiv Preprint Paper**: Explore the detailed scientific background of CXR LLaVA on [Arxiv](https://arxiv.org/abs/2310.18341).
-   **Demo Website**: Experience the model in action at [Radiologist App](https://radiologist.app/cxr-llava).




|Version| Input CXR resolution | Channels | Vision Encoder | Base LLM | Weight 
|--|--|--|--|--|--|
| v1.0 | 512x512 | RGB|RN50|LLAMA2-13B-CHAT|Deprecated
|v2.0 (Latest)|512x512|Grayscale|ViT-L/16|LLAMA2-7B-CHAT|[Link](https://huggingface.co/ECOFRI/CXR-LLAVA-v2)


## Usage Guide
### Importing Packages

    from transformers import AutoModel
    from PIL import Image
### Prepare CXR
    
-   Ensure you have an CXR image file ready, such as 'img.jpg'.
-   Use the following code to load the image

    cxr_image = Image.open(os.path.join(os.path.dirname(__file__), "IMG", "img.jpg"))

### Load model
Loading the CXR-LLAVA model is straightforward and can be done in one line of code.


      model = AutoModel.from_pretrained("ECOFRI/CXR-LLAVA-v2", trust_remote_code=True)
      model = model.to("cuda")


### Generating Radiologic Reports

To write a radiologic report of a chest radiograph:



      response = model.write_radiologic_report(cxr_image)

 > The radiologic report reveals a large consolidation in the right upper lobe of the lungs. There is no evidence of pleural effusion or pneumothorax. The cardiac and mediastinal contours are normal. 


### Differential Diagnosis
For differential diagnosis:


    model.write_differential_diagnosis(cxr_image)

> Possible differential diagnoses for this patient include pneumonia,tuberculosis, lung abscess, or a neoplastic process such as lung cancer.

### Question Answering
To ask a question:

      question = "What is true meaning of consolidation?"
      response = model.ask_question(question=question, image=cxr_image)

> Consolidation refers to the filling of the airspaces in the lungs with fluid, pus, blood, cells or other substances, resulting in a region of lung tissue that has become dense and solid rather than containing air.

## Custom Prompt
For custom interactions:

    img = Image.open(os.path.join(os.path.dirname(__file__), "IMG", "img.jpg"))
    chat = [
        {"role": "system",
         "content": "You are a helpful radiologist. Try to interpret chest x ray image and answer to the question that user provides."},
        {"role": "user",
         "content": "<image>\nWrite a radiologic report on the given chest radiograph, including information about atelectasis, cardiomegaly, consolidation, pulmonary edema, pleural effusion, and pneumothorax.\n"}
    ]
    response = model.generate_cxr_repsonse(chat=chat,pil_image=img, temperature=0, top_p=1)



## License Information

CXR LLaVA is available under a Creative Commons NonCommercial License. 

Users must obtain the LLAMA-2 license prior to use. More details can be found [here](https://ai.meta.com/resources/models-and-libraries/llama-downloads/).


Lastly, we extend our heartfelt thanks to all the contributors of the [LLaVA project](https://llava-vl.github.io/).