WasteAssistant-InstructBLIP-Qwen2-VL

WasteAssistant-InstructBLIP-Qwen2-VL is an InstructBLIP-based visual question-answering checkpoint fine-tuned for image-grounded waste identification and waste-management questions framed around Indian waste-management practices.

In this training variant, Qwen2-VL-7B was used to generate candidate/reference answers for the training pipeline. Qwen2-VL-7B is the answer-generation model; the model served from this repository is the fine-tuned InstructBLIP checkpoint.

• Base model: Salesforce/instructblip-flan-t5-xl
• Dataset: KhushKataruka/WasteVQA
• Research paper: arXiv:2607.10610
• Project repository: WasteAssistant on GitHub

Overview

Given an image and a natural-language question, the model generates an answer based on visual information and patterns learned during fine-tuning. It is intended for research and prototyping, not as a definitive source of legal or safety advice.

Question types represented in the evaluation split

The supplied evaluation split contains 2,784 image-question-answer records, 2,070 unique image paths, 15 question templates, and 21 waste labels. The categories and question templates below are based on the supplied test(1).json split. They describe evaluation coverage and do not guarantee accuracy for every task.

1. Waste identification and material properties

• “What specific type of waste item do you see in this image?”
• “What materials is this waste object primarily made of?”
• “What visible characteristics like labels, shape, texture, color, and size help identify this item?”

2. Visible contamination

• “Are there any visible signs of contamination such as food residue, liquids, stains, or dirt on this object?”

3. Waste categories and SWM classification

• “Which main SWM 2016 waste category does this item belong to - Dry Waste, Wet Waste, Sanitary Waste, or Domestic Hazardous Waste?”
• “How would you classify this item as biodegradable, non-biodegradable, or hazardous according to SWM Rules 2016?”
• “Does this item fall under any special sub-categories like E-waste, Composite Waste, or Inert Waste and why?”

4. Preparation and special handling

• “What preprocessing steps like cleaning, drying, or wrapping are required before disposing of this item?”
• “Are there any special handling procedures needed for this item including separate collection methods?”

5. Recycling and compostability

• “How recyclable is this item through standard municipal recycling systems in India?”
• “What is the compostability potential of this item at household level and what is your reasoning?”

6. Bin selection

• “What colored bin should this waste be disposed in according to SWM Rules and what is the reasoning?”

7. Hazard level and environmental impact

• “On a scale of 1 to 10, what is the hazard level of this item and what hazard category does it belong to?”
• “What environmental impacts would occur if this item were disposed of improperly?”

8. Decomposition estimates

• “How long would this item take to decompose in a typical Indian landfill and what factors affect this?”

Waste labels in the supplied split

Cans, CardBoard, Chips Packets, CofeeCup, Construction scrap, Electrical Cables, Electronic chips, Electronics, Empty Medicine Packets, Glass, Gloves, Mask, Medicines, Metal, Paper, Plastic Bottles, Plastic Materials, Polythene, Syringes, WetWaste, Wire

These are dataset annotation labels. Their presence does not mean that all classes are equally represented or that the model performs equally well for every class.

Training-answer generation

The Qwen-based variant used Qwen2-VL-7B to generate candidate answers for the training pipeline. Model-generated answers should be reviewed and filtered before being treated as reference annotations or used for evaluation. The answer-generation model (Qwen2-VL-7B) and the checkpoint hosted here (InstructBLIP) are separate components of the pipeline.

Installation

pip install torch transformers pillow accelerate sentencepiece

For GPU inference, install the PyTorch build suitable for your CUDA environment. InstructBLIP with the Flan-T5-XL backbone can require substantial memory; CPU inference may be slow.

Inference example

This example assumes that this repository contains the fine-tuned InstructBLIP checkpoint and all required configuration, tokenizer, and processor files.

import torch
from PIL import Image
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration

model_id = "KhushKataruka/WasteAssistant-InstructBLIP-Qwen2-VL"

processor = InstructBlipProcessor.from_pretrained(model_id)
model = InstructBlipForConditionalGeneration.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

image = Image.open("waste_image.jpg").convert("RGB")
question = "What specific type of waste item do you see in this image?"

inputs = processor(images=image, text=question, return_tensors="pt")
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=80)

print(processor.batch_decode(output, skip_special_tokens=True)[0].strip())

If loading fails, verify that the repository contains both model weight shards and their index file (if sharded), config.json, generation configuration, image processor files, tokenizer files, SentencePiece model, and the qformer_tokenizer/ files if saved separately during training. Keep files consistent with the checkpoint's saved architecture.

Intended use

• Research on visual question answering for waste identification and segregation.
• Prototyping image-grounded responses about materials, waste categories, recycling, compostability, handling, and disposal.
• Comparing InstructBLIP variants trained with different answer-generation pipelines.

Limitations and responsible use

• Question coverage in the supplied split does not prove that every question type is answered accurately.
• Model-generated reference answers may be incomplete, inconsistent, or incorrect. Evaluate against a reviewed test set before reporting metrics.
• The model may hallucinate material or safety properties that cannot be established from an image alone.
• Visible appearance cannot confirm hidden contamination, chemical composition, infection risk, or whether an item is safe to handle.
• Bin colours, collection practices, and legal requirements can vary by local authority. Do not treat outputs as authoritative legal guidance.
• Hazard ratings and decomposition estimates are approximate and may not be inferable from images alone.
• Do not rely on this model alone for medical waste, sharps, hazardous substances, or other safety-critical disposal decisions.

Citation

If you use this checkpoint, cite the associated research paper and dataset. Verify the paper's official title and author metadata before publication.

@article{wasteassistant2026,
  title = {WasteAssistant},
  journal = {arXiv preprint arXiv:2607.10610},
  year = {2026},
  url = {https://arxiv.org/abs/2607.10610}
}

License

This repository declares the MIT license. Before public release, confirm that the checkpoint, training code, and image/annotation sources can legally be redistributed under the selected license. Base models and source datasets may have separate terms.