---
license: mit
---

# Prompt2MedImage - Diffusion for Medical Images

Prompt2MedImage is a latent text to image diffusion model that has been fine-tuned on medical images from ROCO dataset. 

The weights here are itended to be used with the 🧨Diffusers library. 

This model was trained using Amazon SageMaker and the Hugging Face Deep Learning container.

## Model Details
- **Developed by:** Nihir Chadderwala
- **Model type:** Diffusion based text to medical image generation model
- **Language:** English
- **License:** MiT
- **Model Description:** This latent text to image diffusion model can be used to generate high quality medical images based on text prompts. It uses a fixed, pretrained text encoder ([CLIP ViT-L/14](https://arxiv.org/abs/2103.00020)) as suggested in the [Imagen paper](https://arxiv.org/abs/2205.11487). 


## License

This model is open access and available to all, with a Do What the F*ck You want to public license further specifying rights and usage.

- You can't use the model to deliberately produce nor share illegal or harmful outputs or content.
- The author claims no rights on the outputs you generate, you are free to use them and are accountable for their use.
- You may re-distribute the weights and use the model commercially and/or as a service.


## Run using PyTorch

```bash
pip install diffusers transformers
```

Running pipeline with default PNDM scheduler:

```python
import torch
from diffusers import StableDiffusionPipeline

model_id = "Prompt2MedImage"
device = "cuda"

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to(device)

prompt = "Showing the subtrochanteric fracture in the porotic bone."
image = pipe(prompt).images[0]  
    
image.save("porotic_bone_fracture.png")
``` 

## Citation

```
O. Pelka, S. Koitka, J. Rückert, F. Nensa, C.M. Friedrich,
"Radiology Objects in COntext (ROCO): A Multimodal Image Dataset".
MICCAI Workshop on Large-scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS) 2018, September 16, 2018, Granada, Spain. Lecture Notes on Computer Science (LNCS), vol. 11043, pp. 180-189, Springer Cham, 2018.
doi: 10.1007/978-3-030-01364-6_20
```