JamesZar/OliveGemma-3B

OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet

OliveGemma is a google/paligemma2-3b-pt-448 model fine-tuned (LoRA) for food recognition and grounded food reasoning over a unified Greek/Mediterranean food corpus (MedGR + ODIN + Vippstar).

You read full model report here OliveGemma

License: these are Gemma derivative weights. Use is governed by the Gemma Terms of Use. By using this model you accept those terms.

Model Details

• Developed by:

  • Dimitrios I. Zaridis, PhD, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece
  • Traianos Tsiokris, MSc, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece
  • Vasileios C. Pezoulas, PhD, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece
  • Daphni Plati, MSc, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece
  • Eugenia Mylona, Unit of Medical Technology & Intelligent Information Systems Greece & Department of Medical Physics, School of Medicine, University of Patras, Greece
  • Eleni Georga, PhD, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece
  • Antonis Sakellarios, Dept. of Mechanical and Aeronautics Engineering, University of Patras, Greece
  • Dimitrios I. Fotiadios, Professor of Biomedical Technology, Unit of Medical Technology & Intelligent Information Systems, University of Ioannina, Greece

• Model type: Vision-Language Model (PaliGemma-2-3B, LoRA fused into base weights), Language Model Head

• Task: image-text-to-text (visual question answering for food images, ingredients etc.)

• Language(s): English

• License: Gemma (inherited from the base model)

• Finetuned from: google/paligemma2-3b-pt-448

What it does

Given a single food photo, OliveGemma answers free-text questions about the dish:

• dish_name — "What is the name of this dish?" → canonical dish name.
• likely_ingredients — "What ingredients are typically found in this food?" → list split into visible-typical vs commonly-present-but-not-visually-confirmable.
• class_boundary — "How is this dish different from ?" → short discriminative explanation.

How to Get Started

import torch
from PIL import Image
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

REPO = "JamesZar/OliveGemma-3B"

processor = AutoProcessor.from_pretrained(REPO)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, device_map="auto"
).eval()

image = Image.open("dish.jpg").convert("RGB")
question = "What is the name of this dish?"

# IMPORTANT: PaliGemma prompt format used in training:
prompt = f"<image>answer en {question}\n"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
in_len = inputs["input_ids"].shape[-1]
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.decode(out[0][in_len:], skip_special_tokens=True).strip())

Prompt format matters: the model was trained with the prefix <image>answer en {question}\n. Greedy decoding (do_sample=False) is the evaluated configuration. Merge was done in bf16, so outputs match the adapter-on-base path exactly.

Uses & Out-of-Scope Use

Use OliveGemma for dish_name, likely_ingredients, and class_boundary.

• Not a medical, nutritional, or allergen-safety tool. Ingredient lists are typical-recipe priors, not verified contents — do not use for allergy or clinical decisions.
• Cuisines/dishes outside the Greek/Mediterranean-centric training corpus are unreliable.
• The visible_ingredients and visual_evidence question types did not converge (see Limitations) — do not use them.

Bias, Risks, and Limitations

• Domain skew toward MedGR (Greek/Mediterranean); degrades on under-represented cuisines.
• Two question types effectively failed on the held-out set: visible_ingredients top-1 ≈ 0.011, visual_evidence top-1 ≈ 0.000. These open-vocabulary attributes were under-represented (~5.7K QA pairs each vs ~30K for headline attributes) and scored under strict exact match.
• Ingredient hallucination: lists reflect recipe priors for the recognised dish and may include ingredients not actually present in the photo.

Training Details

Single-stage SFT (LoRA) over the merged corpus of three datasets, fold 3 of a group-aware unified 3-fold scheme.

Source	Dataset	Approx. QA pairs
medgr	MedGR (Greek/Mediterranean)	~111K
vippstar	Vippstar	~46K
odin	ODIN	~6.7K

The figure illustrates the overall workflow followed in this study. Initially, the MedGR, ODIN, and VIPPSTAR datasets were collected, harmonized, and structured to support a unified experimental pipeline. A 3 fold cross validation scheme was then applied while preserving the original class imbalance and ensuring sufficient representation of all food categories across the three folds. For the deep learning (DL) baseline experiments, four established image classification architectures pretrained on the Food-101 dataset were selected to leverage prior domain knowledge of food images. Each model was finetuned and evaluated independently on every fold.

The proposed OliveGemma model was fine-tuned using question-answer (QA) pairs generated from the annotations already available in the employed datasets together with knowledge distilled from Anthropic Claude 4.6 Opus. Approximately 61,071 QA pairs were generated for each fold. The training corpus was designed to capture multiple aspects of food understanding, including food recognition, visual reasoning between visually similar dishes, likely ingredients, visual evidence supporting the predicted class, food groups, cuisine identification, preparation steps, and uncertainty-aware responses. The latter encourages the model to explicitly acknowledge information that cannot be reliably inferred from a single image, reducing overconfident hallucinations during inference.

To compare OliveGemma against frontier vision-language models, OpenAI GPT-5.4 Mini, Google Gemini Flash 3 and Gemini Flash 3.5, and Anthropic Claude Haiku 4.6 were evaluated. To ensure a consistent evaluation protocol, all models were prompted using the same system prompt, instructing them to act as food recognition experts. Furthermore, the complete canonical class vocabulary was provided, and each model was constrained to return exactly one food category from the predefined label set, thereby isolating the effect of model capability from differences in prompting or output formatting.

OliveGemma training & evaluation workflow.

Hyperparameters: LoRA r=16, α=32, dropout=0.05; targets q,k,v,o,gate,up,down_proj; bf16 mixed precision; grad checkpointing; effective batch 32 (4×8); AdamW + cosine, LR 1e-4, warmup 100, weight decay 0.01; max length 1024; vision encoder frozen 1500 steps then unfrozen at 0.1× LR; 6 epochs (early stopping, best @ step 4800); seed 42. Hardware: single NVIDIA A40 (48 GB), ~48 h.

Splits use StratifiedGroupKFold keyed on <source>/<dish> with leakage-safe group keys; a constant holdout_val slice for early stopping and a separate holdout_test slice for final unseen evaluation. dish_name answers were homogenised to one canonical string per image before training/eval.

Evaluation (held-out holdout_test)

Attribute	Metric	Score	n
dish_name	top-1 exact	0.943	2982
dish_name	top-3 exact	0.964	2982
dish_name	top-5 exact	0.967	2982
dish_name	mean Jaccard	0.949	2982
dish_name	macro-F1	0.856	2982
likely_ingredients	top-1 exact	0.928	3003
class_boundary	top-1 exact	0.828	3003
visible_ingredients	top-1 exact	0.011	536
visual_evidence	top-1 exact	0.000	536

Greedy decoding, max 64 new tokens; top-3/top-5 via beam search.

Technical Specifications

PaliGemma-2-3B (SigLIP-So400m vision encoder + Gemma-2 2B LM); LoRA fused into attention + MLP projections, saved in bf16. Objective: causal LM over the answer suffix, conditioned on image + prompt.

Citation

OliveGemma

@misc{zaridis2026olivegemma3billionvisual,
      title={OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet}, 
      author={Dimitrios I. Zaridis and Traianos Tsiokris and Vasileios C. Pezoulas and Daphni Plati and Eugenia Mylona and Eleni Georga and Nikos Tsiknakis and Antonis Sakellarios and Dimitrios I. Fotiadis},
      year={2026},
      eprint={2608.03428},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.03428}, 
}

Model Card Contact

dimzaridis@gmail.com