VishavGupta01/MRBEAN_Vision

MRBEAN Vision Engine

This repository contains the offline multi-task computer vision model for Project MRBEAN (Mobile Resilient Broadcast for Emergency Ad-hoc Networks).

Currently in its initial development phase, this lightweight model is designed to run entirely offline on Android edge devices to analyze live camera feeds, identifying both the type of disaster and the severity of the damage simultaneously.

---

Dataset

Link: https://crisisnlp.qcri.org/medic/#:~:text=CrisisNLP,The%20dataset%20contains%2071%2C198%20images.

---

Model Architecture

The vision engine utilizes a shared MobileNetV3 (Large) backbone, splitting into two distinct dense classification heads:

• Backbone: MobileNetV3 (Pre-trained on ImageNet, fine-tuned on the MEDIC dataset)
• Head 1 (Disaster Type): 6 Output Classes
• Head 2 (Damage Severity): 3 Output Classes

---

Supported Classification Classes

Disaster Types (Head 1)

• 0 = Earthquake
• 1 = Fire
• 2 = Flood
• 3 = Hurricane
• 4 = Landslide
• 5 = Not a Disaster

Damage Severity (Head 2)

• 0 = Little or None
• 1 = Mild
• 2 = Severe

---

Exported Model Formats

To support cross-platform development and strict mobile deployment, the model has been exported into the following formats:

• best_multitask_vision_model.pth: The raw PyTorch weights for continued training or server-side inference.
• mrbean_tf_model/mrbean_vision_float32.tflite (12.7 MB): Standard TensorFlow Lite model for high-accuracy Android deployment.
• mrbean_tf_model/mrbean_vision_float16.tflite (6.39 MB): Half-precision compressed model to minimize the Flutter application bundle size.

---

Input Specifications

The model expects images preprocessed to match the PyTorch standard:

• Resolution: 224x224 pixels
• Channels: 3 (RGB)
• Normalization: ImageNet standard (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
• TFLite Input Shape: [1, 224, 224, 3] (NHWC format)

---

Python Testing (TFLite)

To test the mobile-ready .tflite files on a PC before integrating them into the Flutter frontend, use the following snippet. Note: Testing the float16 model on a PC requires bypassing the XNNPACK engine.

import os
os.environ["TF_LITE_DISABLE_XNNPACK"] = "1" 

import numpy as np
import tensorflow as tf
from PIL import Image

# 1. Load Model
model_path = r"mrbean_tf_model\mrbean_vision_float16.tflite"
interpreter = tf.lite.Interpreter(model_path=model_path)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()[0]
output_details = interpreter.get_output_details()

# 2. Preprocess Image
def preprocess_image(image_path, expected_shape):
    img = Image.open(image_path).convert('RGB').resize((224, 224))
    img_array = np.array(img, dtype=np.float32) / 255.0
    
    mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
    std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
    img_array = (img_array - mean) / std
    
    if expected_shape[1] == 3: # NCHW fallback
        img_array = np.transpose(img_array, (2, 0, 1))
        
    return np.expand_dims(img_array, axis=0)

# 3. Inference
input_data = preprocess_image("test_image.jpg", input_details['shape']).astype(input_details['dtype'])
interpreter.set_tensor(input_details['index'], input_data)
interpreter.invoke()

# 4. Extract Results
out_0 = interpreter.get_tensor(output_details[0]['index'])[0]
out_1 = interpreter.get_tensor(output_details[1]['index'])[0]

pred_disaster = np.argmax(out_0) if len(out_0) == 6 else np.argmax(out_1)
pred_severity = np.argmax(out_1) if len(out_0) == 6 else np.argmax(out_0)

print(f"Disaster Class ID: {pred_disaster}")
print(f"Severity Class ID: {pred_severity}")