Model Card for gemma-2-2b-scientific-summarizer

This is a Parameter-Efficient Fine-Tuning (PEFT) LoRA adapter for google/gemma-2-2b optimized for scientific lay summarization and key-point extraction.

Model Details

Model Description

• Developed by: coder1969
• Model type: PEFT (LoRA) adapter for Causal Language Modeling
• Language(s) (NLP): English
• License: Apache-2.0
• Finetuned from model: google/gemma-2-2b

Uses

Direct Use

This model is directly intended for taking scientific abstract/literature context and generating structured lay summaries or key points.

Out-of-Scope Use

• Clinical or medical diagnostics without peer review.
• Automated code generation or general chatbot applications.

Bias, Risks, and Limitations

Users should be aware that language models can hallucinate or omit critical details from context. Outputs should be verified against original sources.

How to Get Started with the Model

Use the code below to load the base model and apply the fine-tuned adapter weights:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_name = "google/gemma-2-2b"
adapter_model_name = "coder1969/gemma-2-2b-scientific-summarizer"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Apply adapters
model = PeftModel.from_pretrained(model, adapter_model_name)

# Inference Example
prompt = "Document:\nTopic: quantum machine learning\n\nRelevant Literature:\n[Insert relevant abstracts or papers here]\n\nSummary:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

Finetuned on the tomasg25/scientific_lay_summarisation dataset (subset: plos), containing pairs of scientific articles and summaries.

Training Hyperparameters

• LoRA Config:
  • Rank (r): 8
  • Alpha: 16
  • Dropout: 0.1
  • Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Optimization parameters:
  • Learning Rate: 2e-05
  • Batch Size: 8
  • Gradient Accumulation Steps: 4
  • Epochs: 3
  • Weight Decay: 0.01
  • Precision: Mixed precision (FP16)

Framework versions

• PEFT 0.19.1
• Transformers 4.40.0+
• PyTorch 2.0+