Model Card for Model ID
LLaVA-LLaMA 3 8B Fine-tuned Adapter
This repository contains the LoRA adapter weights and processor files for fine-tuning of LLaVA LLaMA 3 8B on the ROCO_v2 dataset for dual tasks:
- Medical Image Concept Detection
- Medical Image Captioning
The adapter allows applying the fine-tuning over the existing base model for reproducibility and lightweight sharing of modifications.
tags:
- medical
- vision-language
- image-captioning
- concept-detection
- llava
- llama3
- peft
- lora
Model Description
This repository contains the LoRA fine-tuned adapter for LLaVA LLaMA 3 8B applied to our dataset.
The base model used is:
xtuner/llava-llama-3-8b-v1_1-transformers
You must load this base model and apply this adapter to reproduce our results.
Usage example
from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel from huggingface_hub import snapshot_download
Download adapter + processor
adapter_path = snapshot_download(repo_id="JoVal26/ja-med-clef-model")
Load base model
model_base = AutoModelForCausalLM.from_pretrained("xtuner/llava-llama-3-8b-v1_1-transformers")
Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(f"{adapter_path}/final_processor")
Apply LoRA adapter
model = PeftModel.from_pretrained(model_base, f"{adapter_path}/final_lora_adapter_explicit")
Intended Use
This model is designed for use in clinical vision-language research and evaluation challenges with the following subtasks:
Concept Detection Task
Identify presence and location of relevant clinical concepts from medical images.
It serves as a building block for scene understanding and supports downstream image retrieval and clinical decision support.
Evaluation metrics: precision, recall, F1 (set coverage metrics).
Caption Prediction Task
Generate coherent medical captions describing an image, focusing on the interplay of visible elements and detected concepts for full-scene interpretation.
Evaluation metrics: BLEU, CIDEr, METEOR, ROUGE, and clinical concept coverage.
Example inference
inputs = tokenizer("Describe the clinical findings in this radiology image:", return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
This model is for research purposes only. It has been fine-tuned on an extended version of ROCO_v2 and may not generalize to other medical imaging datasets or modalities. Clinical use without additional validation is NOT recommended.
Citation
This fine-tuned adapter repo: JoVal26/ja-med-clef-model
License
This work is released under CC BY-NC 4.0 license. For non-commercial research use only.
- Developed by: JoVal26
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: Visual
- Language(s) (NLP): English
- License: CC BY-NC 4.0
- Finetuned from model [optional]: xtuner/llava-llama-3-8b-v1_1-transformers
Model Sources [optional]
Fine-tuned model from xtuner/llava-llama-3-8b-v1_1-transformers.
- Repository: JoVal26/ja-med-clef-model
- Paper [optional]: https://www.dei.unipd.it/~faggioli/temp/clef2025/paper_179.pdf
- Demo: https://github.com/Jangulo7/med_explain_ja
Usage and License Restrictions
- This adapter is shared for research and academic use via the Hugging Face Hub only.
- Downloading, redistribution, or commercial use is NOT permitted.
- Please use the model via the Hugging Face inference endpoints or via
transformers' direct loading (no local weight download). - For commercial licensing or dataset/model access, please contact the author.
License: CC BY-NC 4.0 (non-commercial use only)
Citation [optional]
BibTeX:
How to Cite
If you use this adapter in academic work or publications, please cite:
BibTeX
@misc{JoVal26-ja-med-clef-model,
title = {LLaVA-LLaMA 3 8B Fine-tuned Adapter},
author = {JoVal26},
year = {2025},
howpublished = {\url{https://huggingface.co/JoVal26/ja-med-clef-model}},
note = {Fine-tuned adapter for LLaVA-LLaMA 3 8B for medical vision-language tasks.}
}
- Downloads last month
- 2