Model Card: T5-Base Fine-Tuned on PubMedQA for Long-Form Medical Question Answering

Model Overview

Model Name: google-t5/t5-base fine-tuned on PubMedQA
Model Type: Text-to-Text Transformer (T5)
Task: Long-form Question Answering (QA) in the medical domain
Framework: Hugging Face Transformers + PEFT (Parameter-Efficient Fine-Tuning)
Precision: bf16 (optimized for RTX 3060)
Dataset: PubMedQA

Model Description

This model is a fine-tuned version of T5-Base on PubMedQA, a dataset of biomedical research questions. The model generates long, structured answers to medical questions by utilizing contextual information.

Use Case

Medical research assistants
AI-powered healthcare chatbots
Automated document processing (summarization of research papers)
Clinical decision support

Training Details

Base Model: google-t5/t5-base
Training Method: Fine-tuned using PEFT & LoRA for memory-efficient training
Precision: bf16 for optimized GPU utilization (NVIDIA RTX 3060)
Batch Size: 8
Epochs: 3
Optimizer: AdamW
Learning Rate: 3e-5
Loss Function: Cross-Entropy Loss

Dataset Details

PubMedQA is a dataset consisting of medical questions derived from PubMed articles. The model has been trained on the long-answer subset of PubMedQA, which provides detailed explanations based on medical literature.

Preprocessing Steps:

Tokenized using T5 tokenizer (max input length: 512, max output length: 256)
Padded/truncated for uniform input size
Used bf16 mixed precision to optimize performance

How to Use

Loading the Model

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = "your-huggingface-username/t5-pubmedqa"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Generating Answers

question = "Does aspirin help with heart disease?"
context = "Aspirin has been studied for its effects on cardiovascular disease prevention. It reduces the risk of heart attacks by preventing blood clots, but it may increase the risk of bleeding."

input_text = f"question: {question} context: {context}"
inputs = tokenizer(input_text, return_tensors="pt", padding=True, truncation=True, max_length=512)

output = model.generate(**inputs, max_length=256, min_length=100, num_beams=5)
answer = tokenizer.decode(output[0], skip_special_tokens=True)
print(answer)

Limitations & Considerations

Requires Context: The model performs best when provided with relevant medical context.
Not a Replacement for Medical Professionals: This model is not a diagnostic tool and should not replace expert medical advice.
Potential Hallucination Issues: While it generates structured responses, some hallucinations may occur for ambiguous questions.
Context Retrieval Needed for Open-Domain QA: To be used as a chatbot, the model must be integrated with retrieval-based methods (e.g., FAISS, PubMed API).

Deployment

This model can be used in production for medical question answering applications, including:

Chatbots (integrate with FAISS or Pinecone for knowledge retrieval)
Automated Medical Document Processing (summarization, answering queries from PDFs)
Healthcare Assistants (providing structured answers based on research papers)

Citation

If you use this model, please cite:

@article{pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin, Di and colleagues},
  year={2019},
  journal={arXiv preprint arXiv:1909.06146}
}

Acknowledgments

This fine-tuned model is built on T5-Base by Google and trained on PubMedQA. Special thanks to the Hugging Face community for providing powerful NLP tools.

Model Maintainer: @your-huggingface-username
License: MIT
Model Link: Hugging Face Model Hub