WasteAssistant-BLIP-LLaVA

WasteAssistant-BLIP-LLaVA is a fine-tuned BLIP model developed as part of the WasteAssistant research project. It performs image-based visual question answering (VQA) for waste identification, classification, segregation, and disposal-related queries.

The model takes an image of waste and a natural-language question as input and generates a text response based on the visual question-answering task it was fine-tuned for.

Research Paper

This model is associated with the WasteAssistant research paper.

• Paper: WasteAssistant — arXiv:2607.10610
• Hugging Face Papers: https://huggingface.co/papers/2607.10610

Please refer to the paper for details about the research methodology and the WasteAssistant system.

Dataset

• Dataset: KhushKataruka/WasteVQA

The dataset contains waste-related images paired with natural-language questions and corresponding answers. It supports training and evaluation of visual question-answering models for waste-management applications.

Model Details

• Model name: WasteAssistant-BLIP-LLaVA
• Architecture: BLIP for visual question answering
• Base model: Salesforce/blip-vqa-base
• Fine-tuning: WasteAssistant training pipeline
• Ground-truth generation: LLaVA
• Task: Image-based waste-related question answering
• Framework: Hugging Face Transformers
• Weights format: Safetensors

In this experimental variant, BLIP is the fine-tuned VQA model, while LLaVA was used to generate the ground-truth answers for the corresponding training pipeline.

What Questions Can This Model Answer?

The model was trained for waste-related questions spanning the following 15 question types, represented in the WasteAssistant test data.

1. Waste Identification

Identify the waste visible in an image and describe its apparent form.

Example questions:

• What type of waste is represented in the image?
• What visible material properties indicate the waste classification?

2. Waste Classification

Understand the broad waste classification associated with an item.

Example questions:

• Is this waste biodegradable, non-biodegradable, or hazardous?
• Which category does this waste belong to according to Indian Solid Waste Management Rules 2016?

3. Recycling and Reuse

Answer questions about recycling possibilities and potential reuse.

Example questions:

• Is this item recyclable through municipal systems in India?
• Can this item be practically reused or upcycled before disposal?

4. Composting

Address whether an item is suitable for household-level composting.

Example question:

• Can this waste be composted at household level under SWM Rules?

5. Waste Segregation and Bin Selection

Provide waste-segregation guidance in the context of the question and the referenced rules.

Example questions:

• Which colored bin should be used to dispose of this waste according to SWM Rules?
• Does this item require special handling or segregation according to SWM guidelines?

6. Disposal Rules and Compliance

Answer questions about improper disposal and the Solid Waste Management Rules, 2016.

Example question:

• Does improper disposal of this item violate SWM Rules 2016?

7. Environmental Impact

Describe potential environmental consequences associated with improper disposal.

Example question:

• What environmental impact may result from improper disposal of this waste?

8. Decomposition Time

Provide an estimated decomposition timeframe for waste in landfill conditions.

Example question:

• What is the estimated decomposition time for this waste in landfill conditions?

9. Disposal Preparation

Describe basic preparation steps before disposal or recycling.

Example question:

• What preprocessing steps should be taken before disposing of this waste?

10. Hazard Assessment

Respond to questions about the apparent hazard level of an item.

Example question:

• Rate the hazard level of this item on a scale of 1–10, where 1 is safe and 10 is extremely hazardous.

11. Municipal Collection Systems

Identify the relevant type of municipal authority or collection system in the Indian context.

Example question:

• Which municipal authority or collection system handles this waste type in India?

These are the question types represented in the supplied test data; their inclusion does not guarantee that every generated answer will be correct.

Waste Categories Represented

The supplied test file contains examples from the following 21 waste labels:

• Cans
• CardBoard
• Chips Packets
• CofeeCup
• Construction scrap
• Electrical Cables
• Electronic chips
• Electronics
• Empty Medicine Packets
• Glass
• Gloves
• Mask
• Medicines
• Metal
• Paper
• Plastic Bottles
• Plastic Materials
• Polythene
• Syringes
• WetWaste
• Wire

The model's performance on an individual item depends on the image, the question, and the patterns learned during fine-tuning. The listed categories describe the supplied test data and should not be interpreted as a guarantee of exhaustive waste-category coverage.

Loading the Model

Install the required libraries:

pip install torch transformers safetensors pillow

Load the model and processor:

import torch
from PIL import Image
from transformers import BlipProcessor, BlipForQuestionAnswering

model_id = "KhushKataruka/WasteAssistant-BLIP-LLaVA"

processor = BlipProcessor.from_pretrained(model_id)
model = BlipForQuestionAnswering.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

image = Image.open("waste_image.jpg").convert("RGB")
question = "What type of waste is represented in the image?"

inputs = processor(
    images=image,
    text=question,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=50)

answer = processor.decode(
    output[0],
    skip_special_tokens=True
)

print(answer)

Replace waste_image.jpg with the path to your image and change question to ask another waste-related question.

Compatibility note: This example assumes the saved checkpoint is compatible with BlipForQuestionAnswering and BlipProcessor. Verify the loading classes against the original training code before using the example in production.

Repository Contents

The repository includes the fine-tuned model weights and supporting configuration and tokenizer files.

• model.safetensors — fine-tuned model weights.
• config.json — model architecture configuration.
• generation_config.json — generation settings.
• preprocessor_config.json — image preprocessing configuration.
• special_tokens_map.json — special-token definitions.
• tokenizer.json — tokenizer configuration and vocabulary.
• tokenizer_config.json — tokenizer settings.
• vocab.txt — tokenizer vocabulary.

Limitations

• Generated answers may be incorrect, incomplete, or inconsistent with the image.
• Performance may vary with image quality, lighting, occlusion, unusual waste items, and images containing multiple objects.
• Estimated decomposition times and hazard assessments should not be treated as authoritative measurements.
• Waste-segregation and disposal recommendations should be checked against applicable local regulations and municipal guidance.
• LLaVA-generated ground-truth answers may introduce errors or biases into the fine-tuning data.
• The model should not replace expert assessment for hazardous, medical, pharmaceutical, electronic, or other regulated waste.

License

This repository is marked as MIT licensed. Review the licensing terms of the base model, dataset, and other third-party components before redistribution or commercial use.

Citation

If you use this model in your research, please cite the associated paper.

Paper: https://arxiv.org/abs/2607.10610

@article{wasteassistant2026,
  title = {WasteAssistant},
  journal = {arXiv preprint arXiv:2607.10610},
  year = {2026},
  url = {https://arxiv.org/abs/2607.10610}
}

Verify the official title and author information against the published paper and update the BibTeX metadata as needed.

Acknowledgements

This work builds on the BLIP visual-question-answering architecture and uses LLaVA-generated ground-truth answers as part of the WasteAssistant training pipeline.

• BLIP VQA base model
• WasteVQA dataset
• WasteAssistant research paper