Repository: GoogleCloudPlatform/generative-ai
Pull request #1931: feat: vision assistant using Gemini Pro
Author: login: iprajwaal
Created: 2025-04-03T14:36:56Z | Merged: 2025-04-03T18:52:16Z

# Description

Thank you for opening a Pull Request!
Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Follow the [`CONTRIBUTING` Guide](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/CONTRIBUTING.md).
- [x] You are listed as the author in your notebook or README file.
  - [ ] Your account is listed in [`CODEOWNERS`](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/.github/CODEOWNERS) for the file(s).
- [x] Make your Pull Request title in the <https://www.conventionalcommits.org/> specification.
- [x] Ensure the tests and linter pass (Run `nox -s format` from the repository root to format).
- [x] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕



Diff:
diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt
index e557ee11431..2589b49572f 100644
--- a/.github/actions/spelling/allow.txt
+++ b/.github/actions/spelling/allow.txt
@@ -802,6 +802,7 @@ imagefont
 imageno
 imagetext
 imdb
+imencode
 importmap
 imread
 imshow
diff --git a/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb b/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
new file mode 100644
index 00000000000..f457c4f72bf
--- /dev/null
+++ b/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
@@ -0,0 +1,1162 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {
+        "id": "f705f4be70e9"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2025 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e47ccf0ace61"
+      },
+      "source": [
+        "# Enhanced Vision Assistant with Gemini\n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "\n",
+        "<div style=\"clear: both;\"></div>\n",
+        "\n",
+        "<b>Share to:</b>\n",
+        "\n",
+        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+        "</a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5a832eb3d892"
+      },
+      "source": [
+        "| | |\n",
+        "|-|-|\n",
+        "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "79329fa09037"
+      },
+      "source": [
+        "### Overview\n",
+        "\n",
+        "![Tech](https://storage.googleapis.com/github-repo/generative-ai/gemini/sample-apps/vision-assistant/Tech.jpg)\n",
+        "\n",
+        "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using Gen AI with Gemini alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+        "\n",
+        "### Demo Video \n",
+        "\n",
+        "[YouTube Video](https://youtu.be/Jpili5kx3hA)\n",
+        "\n",
+        "### Objectives\n",
+        "\n",
+        "In this notebook, you will build a complete vision assistant system that:\n",
+        "\n",
+        "* Detects objects in real-time using Google Cloud Vision API\n",
+        "* Analyzes scenes with Gemini to identify hazards and safe paths\n",
+        "* Generates natural language navigation guidance prioritized by urgency\n",
+        "* Delivers audio instructions through Google Text-to-Speech\n",
+        "* Implements a full navigation pipeline with object tracking and context awareness\n",
+        "\n",
+        "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem.\n",
+        "\n",
+        "![Architecture](https://storage.googleapis.com/github-repo/generative-ai/gemini/sample-apps/vision-assistant/architecture.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "862632423257"
+      },
+      "source": [
+        "### Features\n",
+        "\n",
+        "- **Real-time object detection** using Google Cloud Vision API\n",
+        "- **Intelligent scene analysis** with Google Gen AI and Gemini\n",
+        "- **Natural language navigation guidance** prioritized by urgency\n",
+        "- **Audio feedback** through Google Text-to-Speech\n",
+        "- **Obstacle tracking** and movement prediction\n",
+        "- **Dynamic priority system** for focusing on the most relevant hazards\n",
+        "\n",
+        "### How It Works\n",
+        "\n",
+        "1. The camera captures the user's environment in real-time.\n",
+        "2. Cloud Vision API detects objects and their positions.\n",
+        "3. The system estimates depth and analyzes potential hazards.\n",
+        "4. Gemini processes the scene and generates natural language guidance.\n",
+        "5. Audio instructions are delivered through text-to-speech, prioritized by urgency."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f0316df526f8"
+      },
+      "source": [
+        "## Get started"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ba5a5d175314"
+      },
+      "source": [
+        "#### Install Google Gen AI SDK for Python, and required dependencies for the project"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "018ac3f38d64"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade --quiet google-genai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "94c93de4621b"
+      },
+      "source": [
+        "#### Authenticate your notebook environment (Colab only)\n",
+        "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {
+        "id": "2082bbcb771c"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "if \"google.colab\" in sys.modules:\n",
+        "    from google.colab import auth\n",
+        "\n",
+        "    auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1e6301eae57"
+      },
+      "source": [
+        "### Importing the necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 19,
+      "metadata": {
+        "id": "788feebbbf09"
+      },
+      "outputs": [],
+      "source": [
+        "from dataclasses import dataclass\n",
+        "from enum import Enum\n",
+        "import os\n",
+        "import queue\n",
+        "import threading\n",
+        "import time\n",
+        "\n",
+        "import cv2\n",
+        "from google import genai\n",
+        "from google.cloud import texttospeech, vision_v1\n",
+        "import numpy as np\n",
+        "import pygame"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "97b83b55360b"
+      },
+      "source": [
+        "### Set Google Cloud Project and Create Client\n",
+        "\n",
+        "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+        "\n",
+        "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+        "2. Enable the required APIs:\n",
+        "   - [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)  \n",
+        "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+        "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+        "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+        "\n",
+        "The code below configures your project settings and initializes the Gen AI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+        "\n",
+        "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "714c768d628a"
+      },
+      "source": [
+        "#### Set your Google Cloud project details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "metadata": {
+        "id": "d37a589ae408"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+        "\n",
+        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e85a4c270774"
+      },
+      "source": [
+        "#### Set up credentials (you can also use application default credentials)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {
+        "id": "f2ea5b356262"
+      },
+      "outputs": [],
+      "source": [
+        "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+        "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+        "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ec8a2b80cfc4"
+      },
+      "source": [
+        "#### Initialize the Gen AI client"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {
+        "id": "149ebb514c54"
+      },
+      "outputs": [],
+      "source": [
+        "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b55d9449233a"
+      },
+      "source": [
+        "### Define Navigation and Object Detection Classes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 9,
+      "metadata": {
+        "id": "2b4859bc7c93"
+      },
+      "outputs": [],
+      "source": [
+        "# Define priority levels for navigation\n",
+        "\n",
+        "\n",
+        "class NavigationPriority(Enum):\n",
+        "    CRITICAL = 1  # Immediate collision risk\n",
+        "    HIGH = 2  # Hazardous objects or close obstacles\n",
+        "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+        "    LOW = 4  # Background information\n",
+        "    IGNORE = 5  # Not worth mentioning"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 10,
+      "metadata": {
+        "id": "99b1c793d19c"
+      },
+      "outputs": [],
+      "source": [
+        "@dataclass\n",
+        "class DetectedObject:\n",
+        "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+        "\n",
+        "    name: str\n",
+        "    bbox: list[vision_v1.NormalizedVertex]\n",
+        "    confidence: float\n",
+        "    depth_estimate: float\n",
+        "    priority: NavigationPriority\n",
+        "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+        "    last_seen: float = time.time()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d343df995d83"
+      },
+      "source": [
+        "### Movement Tracking"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {
+        "id": "8517b9dfb4fa"
+      },
+      "outputs": [],
+      "source": [
+        "class MovementTracker:\n",
+        "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.movement_history = []\n",
+        "        self.last_position = None\n",
+        "\n",
+        "    def update_position(self, position):\n",
+        "        if self.last_position:\n",
+        "            movement = {\n",
+        "                \"from\": self.last_position,\n",
+        "                \"to\": position,\n",
+        "                \"timestamp\": time.time(),\n",
+        "            }\n",
+        "            self.movement_history.append(movement)\n",
+        "        self.last_position = position\n",
+        "\n",
+        "    def get_recent_movement(self):\n",
+        "        # Return last 5 movements\n",
+        "        return self.movement_history[-5:] if self.movement_history else []"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f838da6be93c"
+      },
+      "source": [
+        "### Navigation Context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "metadata": {
+        "id": "51e16167d4d2"
+      },
+      "outputs": [],
+      "source": [
+        "class NavigationContext:\n",
+        "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+        "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+        "        self.safe_paths: list[tuple[float, float]] = []\n",
+        "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+        "        self.last_guidance_time = 0\n",
+        "        self.environment_type = \"unknown\"\n",
+        "\n",
+        "    def update(self, new_objects: list[DetectedObject]):\n",
+        "        current_time = time.time()\n",
+        "        # Update known obstacles with new information\n",
+        "        for obj in new_objects:\n",
+        "            if obj.name in self.known_obstacles:\n",
+        "                old_obj = self.known_obstacles[obj.name]\n",
+        "                # Calculate object movement\n",
+        "                old_center = self.get_object_center(old_obj.bbox)\n",
+        "                new_center = self.get_object_center(obj.bbox)\n",
+        "                trajectory = (\n",
+        "                    new_center[0] - old_center[0],\n",
+        "                    new_center[1] - old_center[1],\n",
+        "                )\n",
+        "                obj.trajectory = trajectory\n",
+        "            self.known_obstacles[obj.name] = obj\n",
+        "\n",
+        "        # Clean up old obstacles\n",
+        "        self.known_obstacles = {\n",
+        "            k: v\n",
+        "            for k, v in self.known_obstacles.items()\n",
+        "            if current_time - v.last_seen < 10\n",
+        "        }  # 10 second timeout\n",
+        "\n",
+        "        # Update danger zones based on current obstacles\n",
+        "        self.update_danger_zones()\n",
+        "\n",
+        "    def get_object_center(self, bbox):\n",
+        "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+        "\n",
+        "    def update_danger_zones(self):\n",
+        "        self.danger_zones = []\n",
+        "        for obj in self.known_obstacles.values():\n",
+        "            center = self.get_object_center(obj.bbox)\n",
+        "            # Calculate danger zone radius based on object size and type\n",
+        "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+        "            radius = size * (\n",
+        "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+        "            )\n",
+        "            self.danger_zones.append((center[0], center[1], radius))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "118df7f082e5"
+      },
+      "source": [
+        "### Agent Mind - Scene Analysis Logic"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "metadata": {
+        "id": "b96ce7de52d7"
+      },
+      "outputs": [],
+      "source": [
+        "class AgentMind:\n",
+        "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.context = NavigationContext()\n",
+        "        self.hazard_weights = {\n",
+        "            \"stairs\": 1.0,\n",
+        "            \"hole\": 1.0,\n",
+        "            \"edge\": 1.0,\n",
+        "            \"glass\": 0.9,\n",
+        "            \"knife\": 0.9,\n",
+        "            \"fire\": 1.0,\n",
+        "            \"chair\": 0.7,\n",
+        "            \"table\": 0.7,\n",
+        "            \"person\": 0.6,\n",
+        "            \"wall\": 0.8,\n",
+        "            \"door\": 0.6,\n",
+        "            \"furniture\": 0.7,\n",
+        "        }\n",
+        "\n",
+        "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+        "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+        "        analysis = {\n",
+        "            \"immediate_threats\": [],\n",
+        "            \"potential_hazards\": [],\n",
+        "            \"safe_paths\": [],\n",
+        "            \"guidance_priority\": NavigationPriority.LOW,\n",
+        "            \"recommended_action\": None,\n",
+        "        }\n",
+        "\n",
+        "        # Update navigation context\n",
+        "        self.context.update(objects)\n",
+        "\n",
+        "        # Identify immediate threats\n",
+        "        for obj in objects:\n",
+        "            if self.is_immediate_threat(obj):\n",
+        "                analysis[\"immediate_threats\"].append(obj)\n",
+        "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+        "\n",
+        "        # Find safe paths\n",
+        "        safe_paths = self.identify_safe_paths(objects)\n",
+        "        if safe_paths:\n",
+        "            analysis[\"safe_paths\"] = safe_paths\n",
+        "\n",
+        "        # Determine recommended action\n",
+        "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+        "\n",
+        "        return analysis\n",
+        "\n",
+        "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+        "        bbox = obj.bbox\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        width = bbox[2].x - bbox[0].x\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        conditions = [\n",
+        "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+        "            obj.name.lower() in self.hazard_weights\n",
+        "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+        "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+        "            any(\n",
+        "                self.is_moving_towards_user(o) for o in [obj]\n",
+        "            ),  # Object moving towards user\n",
+        "        ]\n",
+        "\n",
+        "        return any(conditions)\n",
+        "\n",
+        "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+        "        if not obj.trajectory:\n",
+        "            return False\n",
+        "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+        "\n",
+        "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+        "        \"\"\"Identify safe navigation paths\"\"\"\n",
+        "        safe_paths = []\n",
+        "        sectors = {\n",
+        "            \"left\": {\"clear\": True, \"score\": 0},\n",
+        "            \"center\": {\"clear\": True, \"score\": 0},\n",
+        "            \"right\": {\"clear\": True, \"score\": 0},\n",
+        "        }\n",
+        "\n",
+        "        for obj in objects:\n",
+        "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+        "            if center_x < 0.33:\n",
+        "                sectors[\"left\"][\"clear\"] = False\n",
+        "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            elif center_x < 0.66:\n",
+        "                sectors[\"center\"][\"clear\"] = False\n",
+        "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            else:\n",
+        "                sectors[\"right\"][\"clear\"] = False\n",
+        "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "\n",
+        "        for sector, data in sectors.items():\n",
+        "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+        "                safe_paths.append(sector)\n",
+        "\n",
+        "        return safe_paths\n",
+        "\n",
+        "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+        "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+        "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+        "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+        "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+        "        return size * hazard_weight * depth_factor\n",
+        "\n",
+        "    def determine_best_action(self, analysis: dict) -> str:\n",
+        "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+        "        if analysis[\"immediate_threats\"]:\n",
+        "            threat = analysis[\"immediate_threats\"][0]\n",
+        "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+        "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+        "\n",
+        "        if analysis[\"safe_paths\"]:\n",
+        "            if \"center\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Proceed straight ahead carefully\"\n",
+        "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly left and proceed\"\n",
+        "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly right and proceed\"\n",
+        "\n",
+        "        return \"Stop and wait for assistance\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4b601d9f3bad"
+      },
+      "source": [
+        "### Enhanced Vision Assistant - Main Application"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b739686ae411"
+      },
+      "outputs": [],
+      "source": [
+        "class EnhancedVisionAssistant:\n",
+        "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+        "\n",
+        "    def __init__(self, project_id, location, credentials_path=None):\n",
+        "        # Initialize pygame mixer for audio\n",
+        "        pygame.mixer.init(buffer=512)\n",
+        "\n",
+        "        # Configuration\n",
+        "        self.PROJECT_ID = project_id\n",
+        "        self.REGION = location\n",
+        "        self.CREDENTIALS_PATH = credentials_path\n",
+        "\n",
+        "        # Initialize state variables\n",
+        "        self.running = False\n",
+        "        self.is_speaking = False\n",
+        "        self.cap = None\n",
+        "        self.audio_thread = None\n",
+        "        self.previous_guidance = None\n",
+        "        self.movement_tracker = MovementTracker()\n",
+        "\n",
+        "        # Initialize queues\n",
+        "        self.audio_queue = queue.PriorityQueue()\n",
+        "\n",
+        "        # Set credentials if provided\n",
+        "        if self.CREDENTIALS_PATH:\n",
+        "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+        "\n",
+        "        # Initialize Google clients\n",
+        "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+        "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+        "\n",
+        "        # Initialize Gen AI client\n",
+        "        self.genai_client = genai.Client(\n",
+        "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+        "        )\n",
+        "\n",
+        "        # Initialize the agent mind\n",
+        "        self.agent = AgentMind()\n",
+        "\n",
+        "        # Configuration parameters\n",
+        "        self.DETECTION_INTERVAL = 3  # seconds\n",
+        "        self.MIN_OBJECT_SIZE = 0.1\n",
+        "\n",
+        "        print(f\"Enhanced Vision Assistant initialized with Gemini model\")\n",
+        "\n",
+        "    def detect_objects_with_depth(self, frame):\n",
+        "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+        "        try:\n",
+        "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+        "            content = buffer.tobytes()\n",
+        "\n",
+        "            image = vision_v1.Image(content=content)\n",
+        "            features = [\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+        "            ]\n",
+        "\n",
+        "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+        "            response = self.vision_client.annotate_image(request=request)\n",
+        "\n",
+        "            detected_objects = []\n",
+        "            for obj in response.localized_object_annotations:\n",
+        "                bbox = obj.bounding_poly.normalized_vertices\n",
+        "                height = bbox[2].y - bbox[0].y\n",
+        "                width = bbox[2].x - bbox[0].x\n",
+        "\n",
+        "                if height * width > self.MIN_OBJECT_SIZE:\n",
+        "                    depth_estimate = 1 / (height * width)\n",
+        "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+        "\n",
+        "                    detected_obj = DetectedObject(\n",
+        "                        name=obj.name,\n",
+        "                        bbox=bbox,\n",
+        "                        confidence=obj.score,\n",
+        "                        depth_estimate=depth_estimate,\n",
+        "                        priority=priority,\n",
+        "                    )\n",
+        "                    detected_objects.append(detected_obj)\n",
+        "\n",
+        "            return detected_objects\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Object detection error: {e}\")\n",
+        "            return []\n",
+        "\n",
+        "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+        "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+        "        if self.agent.is_immediate_threat(\n",
+        "            DetectedObject(\n",
+        "                name=obj.name,\n",
+        "                bbox=bbox,\n",
+        "                confidence=obj.score,\n",
+        "                depth_estimate=depth_estimate,\n",
+        "            )\n",
+        "        ):\n",
+        "            return NavigationPriority.CRITICAL\n",
+        "\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+        "            return NavigationPriority.HIGH\n",
+        "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+        "            return NavigationPriority.MEDIUM\n",
+        "        elif height > 0.1:\n",
+        "            return NavigationPriority.LOW\n",
+        "        return NavigationPriority.IGNORE\n",
+        "\n",
+        "    def generate_smart_guidance(self, objects):\n",
+        "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+        "        try:\n",
+        "            if not objects:\n",
+        "                return None\n",
+        "\n",
+        "            # Generate scene description\n",
+        "            scene_description = []\n",
+        "            for obj in objects:\n",
+        "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "                scene_description.append(description)\n",
+        "\n",
+        "            # Update context\n",
+        "            context = {\n",
+        "                \"previous_guidance\": self.previous_guidance,\n",
+        "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+        "                \"environment_type\": \"indoor\",  # This could be detected\n",
+        "                \"current_speed\": \"normal\",\n",
+        "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+        "                \"safe_zones\": [],\n",
+        "                \"last_guidance_time\": time.time(),\n",
+        "            }\n",
+        "\n",
+        "            # Generate and send prompt to Gen AI model\n",
+        "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+        "\n",
+        "            # Use the client's generate_content method directly\n",
+        "            response = self.genai_client.generate_content(\n",
+        "                model=\"gemini-2.5-pro\", contents=prompt\n",
+        "            )\n",
+        "\n",
+        "            # Store guidance for future context\n",
+        "            self.previous_guidance = response.text\n",
+        "            return self.previous_guidance\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Guidance generation error: {e}\")\n",
+        "            return self.generate_fallback_guidance(objects)\n",
+        "\n",
+        "    def generate_enhanced_object_description(self, obj, bbox):\n",
+        "        \"\"\"Generate detailed object description\"\"\"\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "        position = self.calculate_relative_position(center_x)\n",
+        "        distance = self.estimate_distance(bbox[2].y - bbox[0].y, bbox[2].x - bbox[0].x)\n",
+        "\n",
+        "        description = f\"{obj.name} {position} at {distance}\"\n",
+        "        return description\n",
+        "\n",
+        "    def calculate_relative_position(self, center_x):\n",
+        "        \"\"\"Calculate relative position of object\"\"\"\n",
+        "        if center_x < 0.2:\n",
+        "            return \"far left\"\n",
+        "        elif center_x < 0.4:\n",
+        "            return \"to your left\"\n",
+        "        elif center_x < 0.6:\n",
+        "            return \"directly ahead\"\n",
+        "        elif center_x < 0.8:\n",
+        "            return \"to your right\"\n",
+        "        else:\n",
+        "            return \"far right\"\n",
+        "\n",
+        "    def estimate_distance(self, height, width):\n",
+        "        \"\"\"Estimate distance using object dimensions\"\"\"\n",
+        "        # Note: These thresholds should be calibrated for your specific camera\n",
+        "        # and environment setup. Consider implementing a more sophisticated\n",
+        "        # depth estimation technique for better accuracy.\n",
+        "        area = height * width\n",
+        "\n",
+        "        # Using the camera-specific calibrated thresholds from configuration\n",
+        "        if area > self.config.DISTANCE_THRESHOLD_VERY_CLOSE:\n",
+        "            return \"very close\"\n",
+        "        elif area > self.config.DISTANCE_THRESHOLD_CLOSE:\n",
+        "            return \"close\"\n",
+        "        elif area > self.config.DISTANCE_THRESHOLD_MODERATE:\n",
+        "            return \"moderate distance\"\n",
+        "        else:\n",
+        "            return \"far ahead\"\n",
+        "\n",
+        "    def generate_agent_prompt(self, scene_description, context):\n",
+        "        \"\"\"Generate a sophisticated prompt for the AI agent\"\"\"\n",
+        "        prompt = f\"\"\"You are an intelligent navigation assistant for a visually impaired person. Your role is to be their eyes and ensure their safety while helping them navigate their environment. Think carefully through each step before providing guidance.\n",
+        "\n",
+        "            Current Scene Analysis:\n",
+        "            1. Detected Objects: {', '.join(scene_description)}\n",
+        "            2. Previous Context: {context.get('previous_guidance', 'No previous guidance')}\n",
+        "            3. Movement History: {context.get('movement_history', 'Starting navigation')}\n",
+        "            4. Environment Type: {context.get('environment_type', 'Unknown')}\n",
+        "\n",
+        "            Step-by-step Thinking Process:\n",
+        "            1. First, analyze immediate safety threats:\n",
+        "            - Are there any obstacles in imminent collision path?\n",
+        "            - Are there any hazardous objects nearby?\n",
+        "            - Is there any moving object approaching?\n",
+        "\n",
+        "            2. Then, evaluate navigation options:\n",
+        "            - Which paths are completely clear?\n",
+        "            - What is the safest direction to move?\n",
+        "            - Are there any stable objects that could serve as landmarks?\n",
+        "\n",
+        "            3. Consider environmental context:\n",
+        "            - Is this an indoor or outdoor space?\n",
+        "            - Are there any recognizable features for orientation?\n",
+        "            - Are there any potential changes in elevation (steps, curbs)?\n",
+        "\n",
+        "            4. Think about user comfort:\n",
+        "            - How can they move most confidently?\n",
+        "            - What landmarks can help them maintain orientation?\n",
+        "            - How urgent is the guidance needed?\n",
+        "\n",
+        "            Based on this analysis, provide:\n",
+        "            1. A primary safety alert if needed (most urgent threats)\n",
+        "            2. Clear directional guidance (where to move)\n",
+        "            3. Contextual information (what to expect in that direction)\n",
+        "\n",
+        "            Requirements for your response:\n",
+        "            - Use natural, conversational language suitable for text-to-speech\n",
+        "            - Be concise but thorough (maximum 3 sentences)\n",
+        "            - Start with any urgent warnings\n",
+        "            - Use clear spatial references (left, right, ahead, behind)\n",
+        "            - Mention distances in practical terms (very close, nearby, ahead)\n",
+        "            - If relevant, include time-sensitive information (approaching object, changing conditions)\n",
+        "\n",
+        "            Additional Context:\n",
+        "            - Person's current speed: {context.get('current_speed', 'unknown')}\n",
+        "            - Recent obstacles: {context.get('recent_obstacles', [])}\n",
+        "            - Known safe zones: {context.get('safe_zones', [])}\n",
+        "            - Last guidance timestamp: {context.get('last_guidance_time', 'initial')}\n",
+        "\n",
+        "            Give your response in a clear, calming voice that prioritizes safety while maintaining user confidence.\n",
+        "            \"\"\"\n",
+        "        return prompt\n",
+        "\n",
+        "    def generate_fallback_guidance(self, objects):\n",
+        "        \"\"\"Generate basic guidance when AI model fails\"\"\"\n",
+        "        if not objects:\n",
+        "            return \"Path appears clear, proceed with caution.\"\n",
+        "\n",
+        "        warnings = []\n",
+        "        for obj in objects:\n",
+        "            description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "            warnings.append(description)\n",
+        "\n",
+        "        return \". \".join(warnings) + \". Proceed with caution.\"\n",
+        "\n",
+        "    def process_audio_queue(self):\n",
+        "        \"\"\"Process audio queue with priority handling\"\"\"\n",
+        "        while self.running:\n",
+        "            try:\n",
+        "                if not self.is_speaking and not self.audio_queue.empty():\n",
+        "                    priority, text = self.audio_queue.get(timeout=1)\n",
+        "                    if text:  # Only speak if there's actual guidance\n",
+        "                        self.speak(text, priority)\n",
+        "                time.sleep(0.1)\n",
+        "            except queue.Empty:\n",
+        "                continue\n",
+        "            except Exception as e:\n",
+        "                print(f\"Audio processing error: {e}\")\n",
+        "\n",
+        "    def speak(self, text, priority=5):\n",
+        "        \"\"\"Enhanced text-to-speech with priority handling\"\"\"\n",
+        "        try:\n",
+        "            self.is_speaking = True\n",
+        "\n",
+        "            # Configure voice based on priority\n",
+        "            speaking_rate = 1.4 if priority <= 2 else 1.2\n",
+        "\n",
+        "            input_text = texttospeech.SynthesisInput(text=text)\n",
+        "            voice = texttospeech.VoiceSelectionParams(\n",
+        "                language_code=\"en-US\",\n",
+        "                name=\"en-US-Standard-F\",\n",
+        "                ssml_gender=texttospeech.SsmlVoiceGender.FEMALE,\n",
+        "            )\n",
+        "            audio_config = texttospeech.AudioConfig(\n",
+        "                audio_encoding=texttospeech.AudioEncoding.MP3,\n",
+        "                speaking_rate=speaking_rate,\n",
+        "                pitch=1 if priority > 2 else 1.2,\n",
+        "            )\n",
+        "\n",
+        "            response = self.speech_client.synthesize_speech(\n",
+        "                input=input_text, voice=voice, audio_config=audio_config\n",
+        "            )\n",
+        "\n",
+        "            # Use a temporary file with priority in name\n",
+        "            temp_file = f\"temp_audio_p{priority}_{time.time()}.mp3\"\n",
+        "            with open(temp_file, \"wb\") as out:\n",
+        "                out.write(response.audio_content)\n",
+        "\n",
+        "            pygame.mixer.music.load(temp_file)\n",
+        "            pygame.mixer.music.play()\n",
+        "\n",
+        "            while pygame.mixer.music.get_busy() and self.running:\n",
+        "                time.sleep(0.1)\n",
+        "\n",
+        "            if os.path.exists(temp_file):\n",
+        "                os.remove(temp_file)\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Speech synthesis error: {e}\")\n",
+        "        finally:\n",
+        "            self.is_speaking = False\n",
+        "\n",
+        "    def display_debug_frame(self, frame, objects):\n",
+        "        \"\"\"Display frame with object annotations for debugging\"\"\"\n",
+        "        if objects:\n",
+        "            for obj in objects:\n",
+        "                bbox = obj.bbox\n",
+        "                h, w = frame.shape[:2]\n",
+        "                pts = np.array([[int(v.x * w), int(v.y * h)] for v in bbox], np.int32)\n",
+        "\n",
+        "                # Color based on priority\n",
+        "                color = (0, 255, 0)  # Default color\n",
+        "                if obj.priority == NavigationPriority.CRITICAL:\n",
+        "                    color = (0, 0, 255)  # Red for critical\n",
+        "                elif obj.priority == NavigationPriority.HIGH:\n",
+        "                    color = (0, 165, 255)  # Orange for high priority\n",
+        "\n",
+        "                cv2.polylines(frame, [pts], True, color, 2)\n",
+        "                cv2.putText(\n",
+        "                    frame,\n",
+        "                    f\"{obj.name} (P:{obj.priority.value})\",\n",
+        "                    (int(bbox[0].x * w), int(bbox[0].y * h) - 10),\n",
+        "                    cv2.FONT_HERSHEY_SIMPLEX,\n",
+        "                    0.5,\n",
+        "                    color,\n",
+        "                    2,\n",
+        "                )\n",
+        "\n",
+        "        cv2.imshow(\"Enhanced Vision Assistant View\", frame)\n",
+        "\n",
+        "    def start(self):\n",
+        "        \"\"\"Start the enhanced vision assistant\"\"\"\n",
+        "        try:\n",
+        "            print(\"Starting Enhanced Vision Assistant...\")\n",
+        "            self.running = True\n",
+        "\n",
+        "            # Start audio processing thread\n",
+        "            self.audio_thread = threading.Thread(target=self.process_audio_queue)\n",
+        "            self.audio_thread.daemon = True\n",
+        "            self.audio_thread.start()\n",
+        "\n",
+        "            # Initialize camera with proper error handling\n",
+        "            try:\n",
+        "                self.cap = cv2.VideoCapture(0)\n",
+        "                if not self.cap.isOpened():\n",
+        "                    raise Exception(\"Could not open camera\")\n",
+        "            except Exception as e:\n",
+        "                print(f\"Error opening camera: {e}\")\n",
+        "                self.running = False  # Stop the assistant if camera fails\n",
+        "                return  # Exit the method early\n",
+        "\n",
+        "            last_detection_time = 0\n",
+        "\n",
+        "            print(\"Press 'q' to quit\")\n",
+        "\n",
+        "            while self.running:\n",
+        "                try:\n",
+        "                    ret, frame = self.cap.read()\n",
+        "                    if not ret:\n",
+        "                        break\n",
+        "\n",
+        "                    current_time = time.time()\n",
+        "\n",
+        "                    # Perform object detection at intervals\n",
+        "                    if current_time - last_detection_time >= self.DETECTION_INTERVAL:\n",
+        "                        objects = self.detect_objects_with_depth(frame)\n",
+        "\n",
+        "                        if objects:\n",
+        "                            guidance = self.generate_smart_guidance(objects)\n",
+        "                            if guidance:  # Only queue if there's meaningful guidance\n",
+        "                                # Determine priority based on scene analysis\n",
+        "                                analysis = self.agent.analyze_scene(objects)\n",
+        "                                priority = analysis[\"guidance_priority\"].value\n",
+        "                                self.audio_queue.put((priority, guidance))\n",
+        "\n",
+        "                        last_detection_time = current_time\n",
+        "\n",
+        "                    # Display frame with annotations (debug view)\n",
+        "                    self.display_debug_frame(frame, objects)\n",
+        "\n",
+        "                    if cv2.waitKey(1) & 0xFF == ord(\"q\"):\n",
+        "                        break\n",
+        "                except Exception as e:\n",
+        "                    print(f\"Error in frame processing: {e}\")\n",
+        "                    break  # Exit loop on error\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Error in main loop: {e}\")\n",
+        "        finally:\n",
+        "            self.stop()\n",
+        "\n",
+        "    def stop(self):\n",
+        "        \"\"\"Stop the assistant and cleanup\"\"\"\n",
+        "        print(\"Stopping Enhanced Vision Assistant...\")\n",
+        "        self.running = False\n",
+        "\n",
+        "        # Clear audio queue\n",
+        "        while not self.audio_queue.empty():\n",
+        "            try:\n",
+        "                self.audio_queue.get_nowait()\n",
+        "            except queue.Empty:\n",
+        "                break\n",
+        "\n",
+        "        # Cleanup resources\n",
+        "        pygame.mixer.music.stop()\n",
+        "        pygame.mixer.quit()\n",
+        "\n",
+        "        if self.cap is not None:\n",
+        "            self.cap.release()\n",
+        "\n",
+        "        cv2.destroyAllWindows()\n",
+        "\n",
+        "        if self.audio_thread is not None and self.audio_thread.is_alive():\n",
+        "            self.audio_thread.join(timeout=2)\n",
+        "\n",
+        "        # Clean up temporary files\n",
+        "        for file in os.listdir():\n",
+        "            if file.startswith(\"temp_audio_\") and file.endswith(\".mp3\"):\n",
+        "                try:\n",
+        "                    os.remove(file)\n",
+        "                except:\n",
+        "                    pass\n",
+        "\n",
+        "        print(\"Enhanced Vision Assistant stopped.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a75803098d6f"
+      },
+      "source": [
+        "### Run the Enhanced Vision Assistant\n",
+        "\n",
+        "The following code will initialize and run the Enhanced Vision Assistant. \n",
+        "\n",
+        "> **Note**: You will need a camera connected to your system and appropriate permissions to access it.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 30,
+      "metadata": {
+        "id": "c2a458836f49"
+      },
+      "outputs": [],
+      "source": [
+        "def main():\n",
+        "    # Initialize the assistant with your project details\n",
+        "    assistant = EnhancedVisionAssistant(\n",
+        "        project_id=PROJECT_ID, location=LOCATION, credentials_path=CREDENTIALS_PATH\n",
+        "    )\n",
+        "\n",
+        "    try:\n",
+        "        # Start the assistant\n",
+        "        assistant.start()\n",
+        "    except KeyboardInterrupt:\n",
+        "        print(\"\\nKeyboard interrupt received\")\n",
+        "    finally:\n",
+        "        # Ensure proper cleanup\n",
+        "        assistant.stop()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4c8d35b1a06b"
+      },
+      "source": [
+        "### Execute the Application\n",
+        "\n",
+        "Run the following cell to start the Enhanced Vision Assistant. Press 'q' to quit the application."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 32,
+      "metadata": {
+        "id": "afd7d74043d1"
+      },
+      "outputs": [],
+      "source": [
+        "if __name__ == \"__main__\":\n",
+        "    main()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "271834603285"
+      },
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "Congratulations! You've successfully built a complete Enhanced Vision Assistant using Google Cloud's AI capabilities!\n",
+        "\n",
+        "This notebook demonstrated how to build an enhanced vision assistant that uses Google Cloud APIs:\n",
+        "* Google Cloud Vision API for object detection\n",
+        "* Google Gen AI SDK with Gemini for intelligent scene guidance\n",
+        "* Google Cloud Text-to-Speech for audio feedback\n",
+        "\n",
+        "The system provides real-time navigation assistance for visually impaired users by:\n",
+        "1. Detecting objects in the environment\n",
+        "2. Analyzing their position, size, and potential danger\n",
+        "3. Using Gemini to generate natural language guidance\n",
+        "4. Providing audio feedback in a prioritized manner\n",
+        "\n",
+        "This approach showcases how multiple Google Cloud services can be combined to create assistive technology applications with generative AI at their core.\n",
+        "\n",
+        "To explore more applications of Google's generative AI technologies, visit the [Google AI documentation](https://ai.google.dev/) and [Google Cloud Vertex AI resources](https://cloud.google.com/vertex-ai)."
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "enhanced_vision_assistant.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}


Reviews:
user: login: gemini-code-assist[bot]
body: Hello @iprajwaal, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

## Summary of Changes
This pull request introduces an Enhanced Vision Assistant, designed to aid visually impaired individuals in navigating their surroundings. It leverages Google Cloud's Vision API, GenAI with Gemini Pro, and Text-to-Speech capabilities to provide real-time environmental awareness through audio guidance. The core component is a Jupyter notebook (`enhanced_vision_assistant.ipynb`) containing the application logic, along with a README providing setup and usage instructions. A `.gitignore` file is added to exclude unnecessary files from version control, and a `requirements.txt` file lists the project's dependencies.

### Highlights
* **Core Functionality**: The `enhanced_vision_assistant.ipynb` notebook implements real-time object detection, scene analysis, and audio feedback to assist visually impaired users.
* **Dependency Management**: The `requirements.txt` file ensures that all necessary Python packages are installed for the project to run correctly.
* **Documentation**: The `README.md` provides a comprehensive guide on setting up, using, and customizing the Enhanced Vision Assistant.

### Changelog
<details>
<summary>Click here to see the changelog</summary>

* **vision/use-cases/vision-assistant/.gitignore**
  * Added a `.gitignore` file to exclude unnecessary files such as byte-compiled files, secret keys, and environment-specific files from version control.
* **vision/use-cases/vision-assistant/README.md**
  * Created a `README.md` file providing an overview of the Enhanced Vision Assistant, including features, how it works, prerequisites, installation instructions, usage guidelines, customization options, and troubleshooting tips.
* **vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb**
  * Implemented the core logic of the Enhanced Vision Assistant in a Jupyter notebook.
  * Utilizes Google Cloud Vision API for real-time object detection.
  * Integrates Gemini Pro for intelligent scene analysis and hazard identification.
  * Provides natural language navigation guidance via Google Text-to-Speech.
  * Includes classes for navigation context, movement tracking, and agent decision-making.
  * Offers a debugging view with object annotations on the camera feed.
  * Allows customization of hazard weights, detection intervals, and voice settings.
* **vision/use-cases/vision-assistant/requirements.txt**
  * Added a `requirements.txt` file listing all the necessary Python packages for the project, including `google-generativeai`, `opencv-python`, `pygame`, `numpy`, `scipy`, `google-cloud-vision`, and `google-cloud-texttospeech`.

</details>


<details>

<summary><b>Using Gemini Code Assist</b></summary>
<br>

The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.

<b>Invoking Gemini</b>

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands.

Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments.
Help | `/gemini help` | Displays a list of available commands.

<b>Customization</b>

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).

<b>Limitations & Feedback</b>

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments to provide feedback.

</details>

<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>


[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/review-github-code). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).

---

> A screen's dark, a world unseen,
> AI's light, a guiding sheen.
> With sight restored,
> A path explored,
> Gemini's voice, a helpful queen.


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T14:37:05Z
user: login: gemini-code-assist[bot]
body: ## Code Review
This pull request introduces an enhanced vision assistant using Google Cloud services. The application aims to help visually impaired individuals navigate their surroundings using real-time object detection, scene analysis, and audio guidance. The addition of a .gitignore file and a README.md file is a good start. However, the core logic is within the Jupyter notebook, which requires a thorough review to ensure correctness, efficiency, and maintainability.

### Summary of Findings
* **Missing Error Handling in `EnhancedVisionAssistant.start()`**: The `EnhancedVisionAssistant.start()` method lacks comprehensive error handling for camera initialization and the main loop, potentially leading to unhandled exceptions and application crashes.  Specifically, the `cv2.VideoCapture(0)` call and the loop reading frames should be wrapped in try-except blocks to gracefully handle camera access issues or unexpected errors during frame processing. This is a critical issue as it directly impacts the reliability of the application.
* **Inconsistent Use of `NavigationPriority` in `EnhancedVisionAssistant`**: The `EnhancedVisionAssistant` class exhibits inconsistent usage of the `NavigationPriority` enum, particularly in the `calculate_priority` method.  The method creates a `DetectedObject` with `NavigationPriority.LOW` and passes it to `self.agent.is_immediate_threat()`, which seems redundant as the priority is recalculated later. This inconsistency can lead to confusion and potential errors in priority assignment. This is a high severity issue as it can lead to incorrect guidance being provided to the user.
* **Hardcoded Values in Distance Estimation**: The `estimate_distance` method in `EnhancedVisionAssistant` uses hardcoded area thresholds (0.5, 0.3, 0.1) to estimate distance, which may not generalize well across different camera setups and environments.  Consider calibrating these values or using a more sophisticated depth estimation technique for improved accuracy. This is a medium severity issue as it affects the accuracy of the guidance provided.

### Merge Readiness
The pull request introduces important functionality for visually impaired users. However, several critical and high severity issues need to be addressed before merging. Specifically, the error handling in `EnhancedVisionAssistant.start()` needs to be improved, and the inconsistent use of `NavigationPriority` needs to be resolved. I am unable to approve this pull request, and recommend that it not be merged until these issues are addressed and further reviewed by other team members.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T14:37:21Z
user: login: holtskinner
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T16:58:21Z
user: login: gemini-code-assist[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T16:58:39Z
user: login: iprajwaal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T17:35:41Z
user: login: holtskinner
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T17:44:27Z
user: login: holtskinner
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T17:47:01Z
user: login: holtskinner
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T17:47:43Z
user: login: holtskinner
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T17:48:04Z
user: login: iprajwaal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T17:55:59Z
user: login: holtskinner
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-03T18:48:50Z

Review comments:
diff_hunk: @@ -0,0 +1,1159 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f705f4be70e9"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2025 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e47ccf0ace61"
+      },
+      "source": [
+        "# Enhanced Vision Assistant with Gemini Pro\n",
+        "\n",
+        "![Tech](./assets/Tech.jpg) \n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "\n",
+        "\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "\n",
+        "<div style=\"clear: both;\"></div>\n",
+        "\n",
+        "<b>Share to:</b>\n",
+        "\n",
+        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+        "</a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5a832eb3d892"
+      },
+      "source": [
+        "| | |\n",
+        "|-|-|\n",
+        "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "79329fa09037"
+      },
+      "source": [
+        "### Overview\n",
+        "\n",
+        "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using GenAI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+        "\n",
+        "### Objectives\n",
+        "\n",
+        "In this notebook, you will build a complete vision assistant system that:\n",
+        "\n",
+        "* Detects objects in real-time using Google Cloud Vision API\n",
+        "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+        "* Generates natural language navigation guidance prioritized by urgency\n",
+        "* Delivers audio instructions through Google Text-to-Speech\n",
+        "* Implements a full navigation pipeline with object tracking and context awareness\n",
+        "\n",
+        "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4c599b3db438"
+      },
+      "source": [
+        "## Solution Architecture"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "af8b43769d47"
+      },
+      "source": [
+        "![architecture](./assets/architecture.png) "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f0316df526f8"
+      },
+      "source": [
+        "## Get started"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ba5a5d175314"
+      },
+      "source": [
+        "#### Install Google Gen AI SDK for Python"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7021022e7b5f"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade --quiet google-genai"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "94c93de4621b"
+      },
+      "source": [
+        "#### Authenticate your notebook environment (Colab only)\n",
+        "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "911453311a5d"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "if \"google.colab\" in sys.modules:\n",
+        "    from google.colab import auth\n",
+        "\n",
+        "    auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ecee90c4d7f4"
+      },
+      "source": [
+        "### Setup \n",
+        "- Install required dependencies for the project"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f2a4f8987a72"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1e6301eae57"
+      },
+      "source": [
+        "### Importing the necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c95691bc96ce"
+      },
+      "outputs": [],
+      "source": [
+        "from dataclasses import dataclass\n",
+        "from enum import Enum\n",
+        "import os\n",
+        "import queue\n",
+        "import threading\n",
+        "import time\n",
+        "\n",
+        "import cv2\n",
+        "from google import genai\n",
+        "from google.cloud import texttospeech, vision_v1\n",
+        "import numpy as np\n",
+        "import pygame"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "97b83b55360b"
+      },
+      "source": [
+        "### Set Google Cloud Project and Create Client\n",
+        "\n",
+        "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+        "\n",
+        "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+        "2. Enable the required APIs:\n",
+        "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+        "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+        "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+        "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+        "\n",
+        "The code below configures your project settings and initializes the GenAI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+        "\n",
+        "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "714c768d628a"
+      },
+      "source": [
+        "#### Set your Google Cloud project details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f46f1119c789"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+        "\n",
+        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e85a4c270774"
+      },
+      "source": [
+        "#### Set up credentials (you can also use application default credentials)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3431ec5cf238"
+      },
+      "outputs": [],
+      "source": [
+        "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+        "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+        "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ec8a2b80cfc4"
+      },
+      "source": [
+        "#### Initialize the GenAI client"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "58ead05a5856"
+      },
+      "outputs": [],
+      "source": [
+        "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b55d9449233a"
+      },
+      "source": [
+        "### Define Navigation and Object Detection Classes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "04661205bc93"
+      },
+      "outputs": [],
+      "source": [
+        "# Define priority levels for navigation\n",
+        "\n",
+        "\n",
+        "class NavigationPriority(Enum):\n",
+        "    CRITICAL = 1  # Immediate collision risk\n",
+        "    HIGH = 2  # Hazardous objects or close obstacles\n",
+        "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+        "    LOW = 4  # Background information\n",
+        "    IGNORE = 5  # Not worth mentioning"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "06f08dabaae1"
+      },
+      "outputs": [],
+      "source": [
+        "@dataclass\n",
+        "class DetectedObject:\n",
+        "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+        "\n",
+        "    name: str\n",
+        "    bbox: list[vision_v1.NormalizedVertex]\n",
+        "    confidence: float\n",
+        "    depth_estimate: float\n",
+        "    priority: NavigationPriority\n",
+        "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+        "    last_seen: float = time.time()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d343df995d83"
+      },
+      "source": [
+        "### Movement Tracking"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "29db370b3335"
+      },
+      "outputs": [],
+      "source": [
+        "class MovementTracker:\n",
+        "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.movement_history = []\n",
+        "        self.last_position = None\n",
+        "\n",
+        "    def update_position(self, position):\n",
+        "        if self.last_position:\n",
+        "            movement = {\n",
+        "                \"from\": self.last_position,\n",
+        "                \"to\": position,\n",
+        "                \"timestamp\": time.time(),\n",
+        "            }\n",
+        "            self.movement_history.append(movement)\n",
+        "        self.last_position = position\n",
+        "\n",
+        "    def get_recent_movement(self):\n",
+        "        # Return last 5 movements\n",
+        "        return self.movement_history[-5:] if self.movement_history else []"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f838da6be93c"
+      },
+      "source": [
+        "### Navigation Context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "14a5eeed4642"
+      },
+      "outputs": [],
+      "source": [
+        "class NavigationContext:\n",
+        "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+        "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+        "        self.safe_paths: list[tuple[float, float]] = []\n",
+        "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+        "        self.last_guidance_time = 0\n",
+        "        self.environment_type = \"unknown\"\n",
+        "\n",
+        "    def update(self, new_objects: list[DetectedObject]):\n",
+        "        current_time = time.time()\n",
+        "        # Update known obstacles with new information\n",
+        "        for obj in new_objects:\n",
+        "            if obj.name in self.known_obstacles:\n",
+        "                old_obj = self.known_obstacles[obj.name]\n",
+        "                # Calculate object movement\n",
+        "                old_center = self.get_object_center(old_obj.bbox)\n",
+        "                new_center = self.get_object_center(obj.bbox)\n",
+        "                trajectory = (\n",
+        "                    new_center[0] - old_center[0],\n",
+        "                    new_center[1] - old_center[1],\n",
+        "                )\n",
+        "                obj.trajectory = trajectory\n",
+        "            self.known_obstacles[obj.name] = obj\n",
+        "\n",
+        "        # Clean up old obstacles\n",
+        "        self.known_obstacles = {\n",
+        "            k: v\n",
+        "            for k, v in self.known_obstacles.items()\n",
+        "            if current_time - v.last_seen < 10\n",
+        "        }  # 10 second timeout\n",
+        "\n",
+        "        # Update danger zones based on current obstacles\n",
+        "        self.update_danger_zones()\n",
+        "\n",
+        "    def get_object_center(self, bbox):\n",
+        "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+        "\n",
+        "    def update_danger_zones(self):\n",
+        "        self.danger_zones = []\n",
+        "        for obj in self.known_obstacles.values():\n",
+        "            center = self.get_object_center(obj.bbox)\n",
+        "            # Calculate danger zone radius based on object size and type\n",
+        "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+        "            radius = size * (\n",
+        "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+        "            )\n",
+        "            self.danger_zones.append((center[0], center[1], radius))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "118df7f082e5"
+      },
+      "source": [
+        "### Agent Mind - Scene Analysis Logic"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c9e38629070c"
+      },
+      "outputs": [],
+      "source": [
+        "class AgentMind:\n",
+        "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.context = NavigationContext()\n",
+        "        self.hazard_weights = {\n",
+        "            \"stairs\": 1.0,\n",
+        "            \"hole\": 1.0,\n",
+        "            \"edge\": 1.0,\n",
+        "            \"glass\": 0.9,\n",
+        "            \"knife\": 0.9,\n",
+        "            \"fire\": 1.0,\n",
+        "            \"chair\": 0.7,\n",
+        "            \"table\": 0.7,\n",
+        "            \"person\": 0.6,\n",
+        "            \"wall\": 0.8,\n",
+        "            \"door\": 0.6,\n",
+        "            \"furniture\": 0.7,\n",
+        "        }\n",
+        "\n",
+        "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+        "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+        "        analysis = {\n",
+        "            \"immediate_threats\": [],\n",
+        "            \"potential_hazards\": [],\n",
+        "            \"safe_paths\": [],\n",
+        "            \"guidance_priority\": NavigationPriority.LOW,\n",
+        "            \"recommended_action\": None,\n",
+        "        }\n",
+        "\n",
+        "        # Update navigation context\n",
+        "        self.context.update(objects)\n",
+        "\n",
+        "        # Identify immediate threats\n",
+        "        for obj in objects:\n",
+        "            if self.is_immediate_threat(obj):\n",
+        "                analysis[\"immediate_threats\"].append(obj)\n",
+        "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+        "\n",
+        "        # Find safe paths\n",
+        "        safe_paths = self.identify_safe_paths(objects)\n",
+        "        if safe_paths:\n",
+        "            analysis[\"safe_paths\"] = safe_paths\n",
+        "\n",
+        "        # Determine recommended action\n",
+        "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+        "\n",
+        "        return analysis\n",
+        "\n",
+        "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+        "        bbox = obj.bbox\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        width = bbox[2].x - bbox[0].x\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        conditions = [\n",
+        "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+        "            obj.name.lower() in self.hazard_weights\n",
+        "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+        "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+        "            any(\n",
+        "                self.is_moving_towards_user(o) for o in [obj]\n",
+        "            ),  # Object moving towards user\n",
+        "        ]\n",
+        "\n",
+        "        return any(conditions)\n",
+        "\n",
+        "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+        "        if not obj.trajectory:\n",
+        "            return False\n",
+        "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+        "\n",
+        "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+        "        \"\"\"Identify safe navigation paths\"\"\"\n",
+        "        safe_paths = []\n",
+        "        sectors = {\n",
+        "            \"left\": {\"clear\": True, \"score\": 0},\n",
+        "            \"center\": {\"clear\": True, \"score\": 0},\n",
+        "            \"right\": {\"clear\": True, \"score\": 0},\n",
+        "        }\n",
+        "\n",
+        "        for obj in objects:\n",
+        "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+        "            if center_x < 0.33:\n",
+        "                sectors[\"left\"][\"clear\"] = False\n",
+        "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            elif center_x < 0.66:\n",
+        "                sectors[\"center\"][\"clear\"] = False\n",
+        "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            else:\n",
+        "                sectors[\"right\"][\"clear\"] = False\n",
+        "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "\n",
+        "        for sector, data in sectors.items():\n",
+        "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+        "                safe_paths.append(sector)\n",
+        "\n",
+        "        return safe_paths\n",
+        "\n",
+        "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+        "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+        "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+        "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+        "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+        "        return size * hazard_weight * depth_factor\n",
+        "\n",
+        "    def determine_best_action(self, analysis: dict) -> str:\n",
+        "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+        "        if analysis[\"immediate_threats\"]:\n",
+        "            threat = analysis[\"immediate_threats\"][0]\n",
+        "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+        "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+        "\n",
+        "        if analysis[\"safe_paths\"]:\n",
+        "            if \"center\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Proceed straight ahead carefully\"\n",
+        "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly left and proceed\"\n",
+        "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly right and proceed\"\n",
+        "\n",
+        "        return \"Stop and wait for assistance\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4b601d9f3bad"
+      },
+      "source": [
+        "### Enhanced Vision Assistant - Main Application"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b739686ae411"
+      },
+      "outputs": [],
+      "source": [
+        "class EnhancedVisionAssistant:\n",
+        "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+        "\n",
+        "    def __init__(self, project_id, location, credentials_path=None):\n",
+        "        # Initialize pygame mixer for audio\n",
+        "        pygame.mixer.init(buffer=512)\n",
+        "\n",
+        "        # Configuration\n",
+        "        self.PROJECT_ID = project_id\n",
+        "        self.REGION = location\n",
+        "        self.CREDENTIALS_PATH = credentials_path\n",
+        "\n",
+        "        # Initialize state variables\n",
+        "        self.running = False\n",
+        "        self.is_speaking = False\n",
+        "        self.cap = None\n",
+        "        self.audio_thread = None\n",
+        "        self.previous_guidance = None\n",
+        "        self.movement_tracker = MovementTracker()\n",
+        "\n",
+        "        # Initialize queues\n",
+        "        self.audio_queue = queue.PriorityQueue()\n",
+        "\n",
+        "        # Set credentials if provided\n",
+        "        if self.CREDENTIALS_PATH:\n",
+        "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+        "\n",
+        "        # Initialize Google clients\n",
+        "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+        "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+        "\n",
+        "        # Initialize GenAI client\n",
+        "        self.genai_client = genai.Client(\n",
+        "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+        "        )\n",
+        "\n",
+        "        # Initialize the agent mind\n",
+        "        self.agent = AgentMind()\n",
+        "\n",
+        "        # Configuration parameters\n",
+        "        self.DETECTION_INTERVAL = 3  # seconds\n",
+        "        self.MIN_OBJECT_SIZE = 0.1\n",
+        "\n",
+        "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+        "\n",
+        "    def detect_objects_with_depth(self, frame):\n",
+        "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+        "        try:\n",
+        "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+        "            content = buffer.tobytes()\n",
+        "\n",
+        "            image = vision_v1.Image(content=content)\n",
+        "            features = [\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+        "            ]\n",
+        "\n",
+        "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+        "            response = self.vision_client.annotate_image(request=request)\n",
+        "\n",
+        "            detected_objects = []\n",
+        "            for obj in response.localized_object_annotations:\n",
+        "                bbox = obj.bounding_poly.normalized_vertices\n",
+        "                height = bbox[2].y - bbox[0].y\n",
+        "                width = bbox[2].x - bbox[0].x\n",
+        "\n",
+        "                if height * width > self.MIN_OBJECT_SIZE:\n",
+        "                    depth_estimate = 1 / (height * width)\n",
+        "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+        "\n",
+        "                    detected_obj = DetectedObject(\n",
+        "                        name=obj.name,\n",
+        "                        bbox=bbox,\n",
+        "                        confidence=obj.score,\n",
+        "                        depth_estimate=depth_estimate,\n",
+        "                        priority=priority,\n",
+        "                    )\n",
+        "                    detected_objects.append(detected_obj)\n",
+        "\n",
+        "            return detected_objects\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Object detection error: {e}\")\n",
+        "            return []\n",
+        "\n",
+        "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+        "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+        "        if self.agent.is_immediate_threat(\n",
+        "            DetectedObject(\n",
+        "                name=obj.name,\n",
+        "                bbox=bbox,\n",
+        "                confidence=obj.score,\n",
+        "                depth_estimate=depth_estimate,\n",
+        "                priority=NavigationPriority.LOW,\n",
+        "            )\n",
+        "        ):\n",
+        "            return NavigationPriority.CRITICAL\n",
+        "\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+        "            return NavigationPriority.HIGH\n",
+        "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+        "            return NavigationPriority.MEDIUM\n",
+        "        elif height > 0.1:\n",
+        "            return NavigationPriority.LOW\n",
+        "        return NavigationPriority.IGNORE\n",
+        "\n",
+        "    def generate_smart_guidance(self, objects):\n",
+        "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+        "        try:\n",
+        "            if not objects:\n",
+        "                return None\n",
+        "\n",
+        "            # Generate scene description\n",
+        "            scene_description = []\n",
+        "            for obj in objects:\n",
+        "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "                scene_description.append(description)\n",
+        "\n",
+        "            # Update context\n",
+        "            context = {\n",
+        "                \"previous_guidance\": self.previous_guidance,\n",
+        "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+        "                \"environment_type\": \"indoor\",  # This could be detected\n",
+        "                \"current_speed\": \"normal\",\n",
+        "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+        "                \"safe_zones\": [],\n",
+        "                \"last_guidance_time\": time.time(),\n",
+        "            }\n",
+        "\n",
+        "            # Generate and send prompt to GenAI model\n",
+        "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+        "\n",
+        "            # Use the client's generate_content method directly\n",
+        "            response = self.genai_client.generate_content(\n",
+        "                model=\"gemini-pro\", contents=prompt\n",
+        "            )\n",
+        "\n",
+        "            # Store guidance for future context\n",
+        "            self.previous_guidance = response.text\n",
+        "            return self.previous_guidance\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Guidance generation error: {e}\")\n",
+        "            return self.generate_fallback_guidance(objects)\n",
+        "\n",
+        "    def generate_enhanced_object_description(self, obj, bbox):\n",
+        "        \"\"\"Generate detailed object description\"\"\"\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "        position = self.calculate_relative_position(center_x)\n",
+        "        distance = self.estimate_distance(bbox[2].y - bbox[0].y, bbox[2].x - bbox[0].x)\n",
+        "\n",
+        "        description = f\"{obj.name} {position} at {distance}\"\n",
+        "        return description\n",
+        "\n",
+        "    def calculate_relative_position(self, center_x):\n",
+        "        \"\"\"Calculate relative position of object\"\"\"\n",
+        "        if center_x < 0.2:\n",
+        "            return \"far left\"\n",
+        "        elif center_x < 0.4:\n",
+        "            return \"to your left\"\n",
+        "        elif center_x < 0.6:\n",
+        "            return \"directly ahead\"\n",
+        "        elif center_x < 0.8:\n",
+        "            return \"to your right\"\n",
+        "        else:\n",
+        "            return \"far right\"\n",
+        "\n",
+        "    def estimate_distance(self, height, width):\n",
+        "        \"\"\"Estimate distance using object dimensions\"\"\"\n",
+        "        area = height * width\n",
+        "        if area > 0.5:\n",
+        "            return \"very close\"\n",
+        "        elif area > 0.3:\n",
+        "            return \"close\"\n",
+        "        elif area > 0.1:\n",
+        "            return \"moderate distance\"\n",
+        "        else:\n",
+        "            return \"far ahead\"\n",
+        "\n",
+        "    def generate_agent_prompt(self, scene_description, context):\n",
+        "        \"\"\"Generate a sophisticated prompt for the AI agent\"\"\"\n",
+        "        prompt = f\"\"\"You are an intelligent navigation assistant for a visually impaired person. Your role is to be their eyes and ensure their safety while helping them navigate their environment. Think carefully through each step before providing guidance.\n",
+        "\n",
+        "            Current Scene Analysis:\n",
+        "            1. Detected Objects: {', '.join(scene_description)}\n",
+        "            2. Previous Context: {context.get('previous_guidance', 'No previous guidance')}\n",
+        "            3. Movement History: {context.get('movement_history', 'Starting navigation')}\n",
+        "            4. Environment Type: {context.get('environment_type', 'Unknown')}\n",
+        "\n",
+        "            Step-by-step Thinking Process:\n",
+        "            1. First, analyze immediate safety threats:\n",
+        "            - Are there any obstacles in imminent collision path?\n",
+        "            - Are there any hazardous objects nearby?\n",
+        "            - Is there any moving object approaching?\n",
+        "\n",
+        "            2. Then, evaluate navigation options:\n",
+        "            - Which paths are completely clear?\n",
+        "            - What is the safest direction to move?\n",
+        "            - Are there any stable objects that could serve as landmarks?\n",
+        "\n",
+        "            3. Consider environmental context:\n",
+        "            - Is this an indoor or outdoor space?\n",
+        "            - Are there any recognizable features for orientation?\n",
+        "            - Are there any potential changes in elevation (steps, curbs)?\n",
+        "\n",
+        "            4. Think about user comfort:\n",
+        "            - How can they move most confidently?\n",
+        "            - What landmarks can help them maintain orientation?\n",
+        "            - How urgent is the guidance needed?\n",
+        "\n",
+        "            Based on this analysis, provide:\n",
+        "            1. A primary safety alert if needed (most urgent threats)\n",
+        "            2. Clear directional guidance (where to move)\n",
+        "            3. Contextual information (what to expect in that direction)\n",
+        "\n",
+        "            Requirements for your response:\n",
+        "            - Use natural, conversational language suitable for text-to-speech\n",
+        "            - Be concise but thorough (maximum 3 sentences)\n",
+        "            - Start with any urgent warnings\n",
+        "            - Use clear spatial references (left, right, ahead, behind)\n",
+        "            - Mention distances in practical terms (very close, nearby, ahead)\n",
+        "            - If relevant, include time-sensitive information (approaching object, changing conditions)\n",
+        "\n",
+        "            Additional Context:\n",
+        "            - Person's current speed: {context.get('current_speed', 'unknown')}\n",
+        "            - Recent obstacles: {context.get('recent_obstacles', [])}\n",
+        "            - Known safe zones: {context.get('safe_zones', [])}\n",
+        "            - Last guidance timestamp: {context.get('last_guidance_time', 'initial')}\n",
+        "\n",
+        "            Give your response in a clear, calming voice that prioritizes safety while maintaining user confidence.\n",
+        "            \"\"\"\n",
+        "        return prompt\n",
+        "\n",
+        "    def generate_fallback_guidance(self, objects):\n",
+        "        \"\"\"Generate basic guidance when AI model fails\"\"\"\n",
+        "        if not objects:\n",
+        "            return \"Path appears clear, proceed with caution.\"\n",
+        "\n",
+        "        warnings = []\n",
+        "        for obj in objects:\n",
+        "            description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "            warnings.append(description)\n",
+        "\n",
+        "        return \". \".join(warnings) + \". Proceed with caution.\"\n",
+        "\n",
+        "    def process_audio_queue(self):\n",
+        "        \"\"\"Process audio queue with priority handling\"\"\"\n",
+        "        while self.running:\n",
+        "            try:\n",
+        "                if not self.is_speaking and not self.audio_queue.empty():\n",
+        "                    priority, text = self.audio_queue.get(timeout=1)\n",
+        "                    if text:  # Only speak if there's actual guidance\n",
+        "                        self.speak(text, priority)\n",
+        "                time.sleep(0.1)\n",
+        "            except queue.Empty:\n",
+        "                continue\n",
+        "            except Exception as e:\n",
+        "                print(f\"Audio processing error: {e}\")\n",
+        "\n",
+        "    def speak(self, text, priority=5):\n",
+        "        \"\"\"Enhanced text-to-speech with priority handling\"\"\"\n",
+        "        try:\n",
+        "            self.is_speaking = True\n",
+        "\n",
+        "            # Configure voice based on priority\n",
+        "            speaking_rate = 1.4 if priority <= 2 else 1.2\n",
+        "\n",
+        "            input_text = texttospeech.SynthesisInput(text=text)\n",
+        "            voice = texttospeech.VoiceSelectionParams(\n",
+        "                language_code=\"en-US\",\n",
+        "                name=\"en-US-Standard-F\",\n",
+        "                ssml_gender=texttospeech.SsmlVoiceGender.FEMALE,\n",
+        "            )\n",
+        "            audio_config = texttospeech.AudioConfig(\n",
+        "                audio_encoding=texttospeech.AudioEncoding.MP3,\n",
+        "                speaking_rate=speaking_rate,\n",
+        "                pitch=1 if priority > 2 else 1.2,\n",
+        "            )\n",
+        "\n",
+        "            response = self.speech_client.synthesize_speech(\n",
+        "                input=input_text, voice=voice, audio_config=audio_config\n",
+        "            )\n",
+        "\n",
+        "            # Use a temporary file with priority in name\n",
+        "            temp_file = f\"temp_audio_p{priority}_{time.time()}.mp3\"\n",
+        "            with open(temp_file, \"wb\") as out:\n",
+        "                out.write(response.audio_content)\n",
+        "\n",
+        "            pygame.mixer.music.load(temp_file)\n",
+        "            pygame.mixer.music.play()\n",
+        "\n",
+        "            while pygame.mixer.music.get_busy() and self.running:\n",
+        "                time.sleep(0.1)\n",
+        "\n",
+        "            if os.path.exists(temp_file):\n",
+        "                os.remove(temp_file)\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Speech synthesis error: {e}\")\n",
+        "        finally:\n",
+        "            self.is_speaking = False\n",
+        "\n",
+        "    def display_debug_frame(self, frame, objects):\n",
+        "        \"\"\"Display frame with object annotations for debugging\"\"\"\n",
+        "        if objects:\n",
+        "            for obj in objects:\n",
+        "                bbox = obj.bbox\n",
+        "                h, w = frame.shape[:2]\n",
+        "                pts = np.array([[int(v.x * w), int(v.y * h)] for v in bbox], np.int32)\n",
+        "\n",
+        "                # Color based on priority\n",
+        "                color = (0, 255, 0)  # Default color\n",
+        "                if obj.priority == NavigationPriority.CRITICAL:\n",
+        "                    color = (0, 0, 255)  # Red for critical\n",
+        "                elif obj.priority == NavigationPriority.HIGH:\n",
+        "                    color = (0, 165, 255)  # Orange for high priority\n",
+        "\n",
+        "                cv2.polylines(frame, [pts], True, color, 2)\n",
+        "                cv2.putText(\n",
+        "                    frame,\n",
+        "                    f\"{obj.name} (P:{obj.priority.value})\",\n",
+        "                    (int(bbox[0].x * w), int(bbox[0].y * h) - 10),\n",
+        "                    cv2.FONT_HERSHEY_SIMPLEX,\n",
+        "                    0.5,\n",
+        "                    color,\n",
+        "                    2,\n",
+        "                )\n",
+        "\n",
+        "        cv2.imshow(\"Enhanced Vision Assistant View\", frame)\n",
+        "\n",
+        "    def start(self):\n",
+        "        \"\"\"Start the enhanced vision assistant\"\"\"\n",
+        "        try:\n",
+        "            print(\"Starting Enhanced Vision Assistant...\")\n",
+        "            self.running = True\n",
+        "\n",
+        "            # Start audio processing thread\n",
+        "            self.audio_thread = threading.Thread(target=self.process_audio_queue)\n",
+        "            self.audio_thread.daemon = True\n",
+        "            self.audio_thread.start()\n",
+        "\n",
+        "            # Initialize camera\n",
+        "            self.cap = cv2.VideoCapture(0)\n",
+        "            if not self.cap.isOpened():\n",
+        "                raise Exception(\"Could not open camera\")\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: gemini-code-assist[bot]
body: ![critical](https://img.shields.io/badge/priority-critical-red)

The camera initialization lacks proper error handling. If `cv2.VideoCapture(0)` fails, the application will crash. Wrap this in a `try...except` block to handle potential camera access issues.

```
            try:
                self.cap = cv2.VideoCapture(0)
                if not self.cap.isOpened():
                    raise Exception("Could not open camera")
            except Exception as e:
                print(f"Error opening camera: {e}")
                self.running = False # Stop the assistant if camera fails
```
created_at: 2025-04-03T14:37:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,1159 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f705f4be70e9"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2025 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e47ccf0ace61"
+      },
+      "source": [
+        "# Enhanced Vision Assistant with Gemini Pro\n",
+        "\n",
+        "![Tech](./assets/Tech.jpg) \n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "\n",
+        "\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "\n",
+        "<div style=\"clear: both;\"></div>\n",
+        "\n",
+        "<b>Share to:</b>\n",
+        "\n",
+        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+        "</a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5a832eb3d892"
+      },
+      "source": [
+        "| | |\n",
+        "|-|-|\n",
+        "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "79329fa09037"
+      },
+      "source": [
+        "### Overview\n",
+        "\n",
+        "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using GenAI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+        "\n",
+        "### Objectives\n",
+        "\n",
+        "In this notebook, you will build a complete vision assistant system that:\n",
+        "\n",
+        "* Detects objects in real-time using Google Cloud Vision API\n",
+        "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+        "* Generates natural language navigation guidance prioritized by urgency\n",
+        "* Delivers audio instructions through Google Text-to-Speech\n",
+        "* Implements a full navigation pipeline with object tracking and context awareness\n",
+        "\n",
+        "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4c599b3db438"
+      },
+      "source": [
+        "## Solution Architecture"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "af8b43769d47"
+      },
+      "source": [
+        "![architecture](./assets/architecture.png) "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f0316df526f8"
+      },
+      "source": [
+        "## Get started"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ba5a5d175314"
+      },
+      "source": [
+        "#### Install Google Gen AI SDK for Python"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7021022e7b5f"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade --quiet google-genai"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "94c93de4621b"
+      },
+      "source": [
+        "#### Authenticate your notebook environment (Colab only)\n",
+        "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "911453311a5d"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "if \"google.colab\" in sys.modules:\n",
+        "    from google.colab import auth\n",
+        "\n",
+        "    auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ecee90c4d7f4"
+      },
+      "source": [
+        "### Setup \n",
+        "- Install required dependencies for the project"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f2a4f8987a72"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1e6301eae57"
+      },
+      "source": [
+        "### Importing the necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c95691bc96ce"
+      },
+      "outputs": [],
+      "source": [
+        "from dataclasses import dataclass\n",
+        "from enum import Enum\n",
+        "import os\n",
+        "import queue\n",
+        "import threading\n",
+        "import time\n",
+        "\n",
+        "import cv2\n",
+        "from google import genai\n",
+        "from google.cloud import texttospeech, vision_v1\n",
+        "import numpy as np\n",
+        "import pygame"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "97b83b55360b"
+      },
+      "source": [
+        "### Set Google Cloud Project and Create Client\n",
+        "\n",
+        "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+        "\n",
+        "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+        "2. Enable the required APIs:\n",
+        "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+        "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+        "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+        "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+        "\n",
+        "The code below configures your project settings and initializes the GenAI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+        "\n",
+        "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "714c768d628a"
+      },
+      "source": [
+        "#### Set your Google Cloud project details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f46f1119c789"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+        "\n",
+        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e85a4c270774"
+      },
+      "source": [
+        "#### Set up credentials (you can also use application default credentials)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3431ec5cf238"
+      },
+      "outputs": [],
+      "source": [
+        "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+        "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+        "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ec8a2b80cfc4"
+      },
+      "source": [
+        "#### Initialize the GenAI client"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "58ead05a5856"
+      },
+      "outputs": [],
+      "source": [
+        "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b55d9449233a"
+      },
+      "source": [
+        "### Define Navigation and Object Detection Classes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "04661205bc93"
+      },
+      "outputs": [],
+      "source": [
+        "# Define priority levels for navigation\n",
+        "\n",
+        "\n",
+        "class NavigationPriority(Enum):\n",
+        "    CRITICAL = 1  # Immediate collision risk\n",
+        "    HIGH = 2  # Hazardous objects or close obstacles\n",
+        "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+        "    LOW = 4  # Background information\n",
+        "    IGNORE = 5  # Not worth mentioning"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "06f08dabaae1"
+      },
+      "outputs": [],
+      "source": [
+        "@dataclass\n",
+        "class DetectedObject:\n",
+        "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+        "\n",
+        "    name: str\n",
+        "    bbox: list[vision_v1.NormalizedVertex]\n",
+        "    confidence: float\n",
+        "    depth_estimate: float\n",
+        "    priority: NavigationPriority\n",
+        "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+        "    last_seen: float = time.time()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d343df995d83"
+      },
+      "source": [
+        "### Movement Tracking"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "29db370b3335"
+      },
+      "outputs": [],
+      "source": [
+        "class MovementTracker:\n",
+        "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.movement_history = []\n",
+        "        self.last_position = None\n",
+        "\n",
+        "    def update_position(self, position):\n",
+        "        if self.last_position:\n",
+        "            movement = {\n",
+        "                \"from\": self.last_position,\n",
+        "                \"to\": position,\n",
+        "                \"timestamp\": time.time(),\n",
+        "            }\n",
+        "            self.movement_history.append(movement)\n",
+        "        self.last_position = position\n",
+        "\n",
+        "    def get_recent_movement(self):\n",
+        "        # Return last 5 movements\n",
+        "        return self.movement_history[-5:] if self.movement_history else []"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f838da6be93c"
+      },
+      "source": [
+        "### Navigation Context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "14a5eeed4642"
+      },
+      "outputs": [],
+      "source": [
+        "class NavigationContext:\n",
+        "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+        "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+        "        self.safe_paths: list[tuple[float, float]] = []\n",
+        "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+        "        self.last_guidance_time = 0\n",
+        "        self.environment_type = \"unknown\"\n",
+        "\n",
+        "    def update(self, new_objects: list[DetectedObject]):\n",
+        "        current_time = time.time()\n",
+        "        # Update known obstacles with new information\n",
+        "        for obj in new_objects:\n",
+        "            if obj.name in self.known_obstacles:\n",
+        "                old_obj = self.known_obstacles[obj.name]\n",
+        "                # Calculate object movement\n",
+        "                old_center = self.get_object_center(old_obj.bbox)\n",
+        "                new_center = self.get_object_center(obj.bbox)\n",
+        "                trajectory = (\n",
+        "                    new_center[0] - old_center[0],\n",
+        "                    new_center[1] - old_center[1],\n",
+        "                )\n",
+        "                obj.trajectory = trajectory\n",
+        "            self.known_obstacles[obj.name] = obj\n",
+        "\n",
+        "        # Clean up old obstacles\n",
+        "        self.known_obstacles = {\n",
+        "            k: v\n",
+        "            for k, v in self.known_obstacles.items()\n",
+        "            if current_time - v.last_seen < 10\n",
+        "        }  # 10 second timeout\n",
+        "\n",
+        "        # Update danger zones based on current obstacles\n",
+        "        self.update_danger_zones()\n",
+        "\n",
+        "    def get_object_center(self, bbox):\n",
+        "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+        "\n",
+        "    def update_danger_zones(self):\n",
+        "        self.danger_zones = []\n",
+        "        for obj in self.known_obstacles.values():\n",
+        "            center = self.get_object_center(obj.bbox)\n",
+        "            # Calculate danger zone radius based on object size and type\n",
+        "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+        "            radius = size * (\n",
+        "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+        "            )\n",
+        "            self.danger_zones.append((center[0], center[1], radius))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "118df7f082e5"
+      },
+      "source": [
+        "### Agent Mind - Scene Analysis Logic"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c9e38629070c"
+      },
+      "outputs": [],
+      "source": [
+        "class AgentMind:\n",
+        "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.context = NavigationContext()\n",
+        "        self.hazard_weights = {\n",
+        "            \"stairs\": 1.0,\n",
+        "            \"hole\": 1.0,\n",
+        "            \"edge\": 1.0,\n",
+        "            \"glass\": 0.9,\n",
+        "            \"knife\": 0.9,\n",
+        "            \"fire\": 1.0,\n",
+        "            \"chair\": 0.7,\n",
+        "            \"table\": 0.7,\n",
+        "            \"person\": 0.6,\n",
+        "            \"wall\": 0.8,\n",
+        "            \"door\": 0.6,\n",
+        "            \"furniture\": 0.7,\n",
+        "        }\n",
+        "\n",
+        "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+        "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+        "        analysis = {\n",
+        "            \"immediate_threats\": [],\n",
+        "            \"potential_hazards\": [],\n",
+        "            \"safe_paths\": [],\n",
+        "            \"guidance_priority\": NavigationPriority.LOW,\n",
+        "            \"recommended_action\": None,\n",
+        "        }\n",
+        "\n",
+        "        # Update navigation context\n",
+        "        self.context.update(objects)\n",
+        "\n",
+        "        # Identify immediate threats\n",
+        "        for obj in objects:\n",
+        "            if self.is_immediate_threat(obj):\n",
+        "                analysis[\"immediate_threats\"].append(obj)\n",
+        "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+        "\n",
+        "        # Find safe paths\n",
+        "        safe_paths = self.identify_safe_paths(objects)\n",
+        "        if safe_paths:\n",
+        "            analysis[\"safe_paths\"] = safe_paths\n",
+        "\n",
+        "        # Determine recommended action\n",
+        "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+        "\n",
+        "        return analysis\n",
+        "\n",
+        "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+        "        bbox = obj.bbox\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        width = bbox[2].x - bbox[0].x\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        conditions = [\n",
+        "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+        "            obj.name.lower() in self.hazard_weights\n",
+        "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+        "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+        "            any(\n",
+        "                self.is_moving_towards_user(o) for o in [obj]\n",
+        "            ),  # Object moving towards user\n",
+        "        ]\n",
+        "\n",
+        "        return any(conditions)\n",
+        "\n",
+        "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+        "        if not obj.trajectory:\n",
+        "            return False\n",
+        "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+        "\n",
+        "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+        "        \"\"\"Identify safe navigation paths\"\"\"\n",
+        "        safe_paths = []\n",
+        "        sectors = {\n",
+        "            \"left\": {\"clear\": True, \"score\": 0},\n",
+        "            \"center\": {\"clear\": True, \"score\": 0},\n",
+        "            \"right\": {\"clear\": True, \"score\": 0},\n",
+        "        }\n",
+        "\n",
+        "        for obj in objects:\n",
+        "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+        "            if center_x < 0.33:\n",
+        "                sectors[\"left\"][\"clear\"] = False\n",
+        "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            elif center_x < 0.66:\n",
+        "                sectors[\"center\"][\"clear\"] = False\n",
+        "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            else:\n",
+        "                sectors[\"right\"][\"clear\"] = False\n",
+        "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "\n",
+        "        for sector, data in sectors.items():\n",
+        "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+        "                safe_paths.append(sector)\n",
+        "\n",
+        "        return safe_paths\n",
+        "\n",
+        "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+        "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+        "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+        "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+        "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+        "        return size * hazard_weight * depth_factor\n",
+        "\n",
+        "    def determine_best_action(self, analysis: dict) -> str:\n",
+        "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+        "        if analysis[\"immediate_threats\"]:\n",
+        "            threat = analysis[\"immediate_threats\"][0]\n",
+        "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+        "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+        "\n",
+        "        if analysis[\"safe_paths\"]:\n",
+        "            if \"center\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Proceed straight ahead carefully\"\n",
+        "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly left and proceed\"\n",
+        "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly right and proceed\"\n",
+        "\n",
+        "        return \"Stop and wait for assistance\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4b601d9f3bad"
+      },
+      "source": [
+        "### Enhanced Vision Assistant - Main Application"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b739686ae411"
+      },
+      "outputs": [],
+      "source": [
+        "class EnhancedVisionAssistant:\n",
+        "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+        "\n",
+        "    def __init__(self, project_id, location, credentials_path=None):\n",
+        "        # Initialize pygame mixer for audio\n",
+        "        pygame.mixer.init(buffer=512)\n",
+        "\n",
+        "        # Configuration\n",
+        "        self.PROJECT_ID = project_id\n",
+        "        self.REGION = location\n",
+        "        self.CREDENTIALS_PATH = credentials_path\n",
+        "\n",
+        "        # Initialize state variables\n",
+        "        self.running = False\n",
+        "        self.is_speaking = False\n",
+        "        self.cap = None\n",
+        "        self.audio_thread = None\n",
+        "        self.previous_guidance = None\n",
+        "        self.movement_tracker = MovementTracker()\n",
+        "\n",
+        "        # Initialize queues\n",
+        "        self.audio_queue = queue.PriorityQueue()\n",
+        "\n",
+        "        # Set credentials if provided\n",
+        "        if self.CREDENTIALS_PATH:\n",
+        "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+        "\n",
+        "        # Initialize Google clients\n",
+        "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+        "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+        "\n",
+        "        # Initialize GenAI client\n",
+        "        self.genai_client = genai.Client(\n",
+        "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+        "        )\n",
+        "\n",
+        "        # Initialize the agent mind\n",
+        "        self.agent = AgentMind()\n",
+        "\n",
+        "        # Configuration parameters\n",
+        "        self.DETECTION_INTERVAL = 3  # seconds\n",
+        "        self.MIN_OBJECT_SIZE = 0.1\n",
+        "\n",
+        "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+        "\n",
+        "    def detect_objects_with_depth(self, frame):\n",
+        "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+        "        try:\n",
+        "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+        "            content = buffer.tobytes()\n",
+        "\n",
+        "            image = vision_v1.Image(content=content)\n",
+        "            features = [\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+        "            ]\n",
+        "\n",
+        "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+        "            response = self.vision_client.annotate_image(request=request)\n",
+        "\n",
+        "            detected_objects = []\n",
+        "            for obj in response.localized_object_annotations:\n",
+        "                bbox = obj.bounding_poly.normalized_vertices\n",
+        "                height = bbox[2].y - bbox[0].y\n",
+        "                width = bbox[2].x - bbox[0].x\n",
+        "\n",
+        "                if height * width > self.MIN_OBJECT_SIZE:\n",
+        "                    depth_estimate = 1 / (height * width)\n",
+        "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+        "\n",
+        "                    detected_obj = DetectedObject(\n",
+        "                        name=obj.name,\n",
+        "                        bbox=bbox,\n",
+        "                        confidence=obj.score,\n",
+        "                        depth_estimate=depth_estimate,\n",
+        "                        priority=priority,\n",
+        "                    )\n",
+        "                    detected_objects.append(detected_obj)\n",
+        "\n",
+        "            return detected_objects\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Object detection error: {e}\")\n",
+        "            return []\n",
+        "\n",
+        "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+        "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+        "        if self.agent.is_immediate_threat(\n",
+        "            DetectedObject(\n",
+        "                name=obj.name,\n",
+        "                bbox=bbox,\n",
+        "                confidence=obj.score,\n",
+        "                depth_estimate=depth_estimate,\n",
+        "                priority=NavigationPriority.LOW,\n",
+        "            )\n",
+        "        ):\n",
+        "            return NavigationPriority.CRITICAL\n",
+        "\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+        "            return NavigationPriority.HIGH\n",
+        "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+        "            return NavigationPriority.MEDIUM\n",
+        "        elif height > 0.1:\n",
+        "            return NavigationPriority.LOW\n",
+        "        return NavigationPriority.IGNORE\n",
+        "\n",
+        "    def generate_smart_guidance(self, objects):\n",
+        "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+        "        try:\n",
+        "            if not objects:\n",
+        "                return None\n",
+        "\n",
+        "            # Generate scene description\n",
+        "            scene_description = []\n",
+        "            for obj in objects:\n",
+        "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "                scene_description.append(description)\n",
+        "\n",
+        "            # Update context\n",
+        "            context = {\n",
+        "                \"previous_guidance\": self.previous_guidance,\n",
+        "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+        "                \"environment_type\": \"indoor\",  # This could be detected\n",
+        "                \"current_speed\": \"normal\",\n",
+        "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+        "                \"safe_zones\": [],\n",
+        "                \"last_guidance_time\": time.time(),\n",
+        "            }\n",
+        "\n",
+        "            # Generate and send prompt to GenAI model\n",
+        "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+        "\n",
+        "            # Use the client's generate_content method directly\n",
+        "            response = self.genai_client.generate_content(\n",
+        "                model=\"gemini-pro\", contents=prompt\n",
+        "            )\n",
+        "\n",
+        "            # Store guidance for future context\n",
+        "            self.previous_guidance = response.text\n",
+        "            return self.previous_guidance\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Guidance generation error: {e}\")\n",
+        "            return self.generate_fallback_guidance(objects)\n",
+        "\n",
+        "    def generate_enhanced_object_description(self, obj, bbox):\n",
+        "        \"\"\"Generate detailed object description\"\"\"\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "        position = self.calculate_relative_position(center_x)\n",
+        "        distance = self.estimate_distance(bbox[2].y - bbox[0].y, bbox[2].x - bbox[0].x)\n",
+        "\n",
+        "        description = f\"{obj.name} {position} at {distance}\"\n",
+        "        return description\n",
+        "\n",
+        "    def calculate_relative_position(self, center_x):\n",
+        "        \"\"\"Calculate relative position of object\"\"\"\n",
+        "        if center_x < 0.2:\n",
+        "            return \"far left\"\n",
+        "        elif center_x < 0.4:\n",
+        "            return \"to your left\"\n",
+        "        elif center_x < 0.6:\n",
+        "            return \"directly ahead\"\n",
+        "        elif center_x < 0.8:\n",
+        "            return \"to your right\"\n",
+        "        else:\n",
+        "            return \"far right\"\n",
+        "\n",
+        "    def estimate_distance(self, height, width):\n",
+        "        \"\"\"Estimate distance using object dimensions\"\"\"\n",
+        "        area = height * width\n",
+        "        if area > 0.5:\n",
+        "            return \"very close\"\n",
+        "        elif area > 0.3:\n",
+        "            return \"close\"\n",
+        "        elif area > 0.1:\n",
+        "            return \"moderate distance\"\n",
+        "        else:\n",
+        "            return \"far ahead\"\n",
+        "\n",
+        "    def generate_agent_prompt(self, scene_description, context):\n",
+        "        \"\"\"Generate a sophisticated prompt for the AI agent\"\"\"\n",
+        "        prompt = f\"\"\"You are an intelligent navigation assistant for a visually impaired person. Your role is to be their eyes and ensure their safety while helping them navigate their environment. Think carefully through each step before providing guidance.\n",
+        "\n",
+        "            Current Scene Analysis:\n",
+        "            1. Detected Objects: {', '.join(scene_description)}\n",
+        "            2. Previous Context: {context.get('previous_guidance', 'No previous guidance')}\n",
+        "            3. Movement History: {context.get('movement_history', 'Starting navigation')}\n",
+        "            4. Environment Type: {context.get('environment_type', 'Unknown')}\n",
+        "\n",
+        "            Step-by-step Thinking Process:\n",
+        "            1. First, analyze immediate safety threats:\n",
+        "            - Are there any obstacles in imminent collision path?\n",
+        "            - Are there any hazardous objects nearby?\n",
+        "            - Is there any moving object approaching?\n",
+        "\n",
+        "            2. Then, evaluate navigation options:\n",
+        "            - Which paths are completely clear?\n",
+        "            - What is the safest direction to move?\n",
+        "            - Are there any stable objects that could serve as landmarks?\n",
+        "\n",
+        "            3. Consider environmental context:\n",
+        "            - Is this an indoor or outdoor space?\n",
+        "            - Are there any recognizable features for orientation?\n",
+        "            - Are there any potential changes in elevation (steps, curbs)?\n",
+        "\n",
+        "            4. Think about user comfort:\n",
+        "            - How can they move most confidently?\n",
+        "            - What landmarks can help them maintain orientation?\n",
+        "            - How urgent is the guidance needed?\n",
+        "\n",
+        "            Based on this analysis, provide:\n",
+        "            1. A primary safety alert if needed (most urgent threats)\n",
+        "            2. Clear directional guidance (where to move)\n",
+        "            3. Contextual information (what to expect in that direction)\n",
+        "\n",
+        "            Requirements for your response:\n",
+        "            - Use natural, conversational language suitable for text-to-speech\n",
+        "            - Be concise but thorough (maximum 3 sentences)\n",
+        "            - Start with any urgent warnings\n",
+        "            - Use clear spatial references (left, right, ahead, behind)\n",
+        "            - Mention distances in practical terms (very close, nearby, ahead)\n",
+        "            - If relevant, include time-sensitive information (approaching object, changing conditions)\n",
+        "\n",
+        "            Additional Context:\n",
+        "            - Person's current speed: {context.get('current_speed', 'unknown')}\n",
+        "            - Recent obstacles: {context.get('recent_obstacles', [])}\n",
+        "            - Known safe zones: {context.get('safe_zones', [])}\n",
+        "            - Last guidance timestamp: {context.get('last_guidance_time', 'initial')}\n",
+        "\n",
+        "            Give your response in a clear, calming voice that prioritizes safety while maintaining user confidence.\n",
+        "            \"\"\"\n",
+        "        return prompt\n",
+        "\n",
+        "    def generate_fallback_guidance(self, objects):\n",
+        "        \"\"\"Generate basic guidance when AI model fails\"\"\"\n",
+        "        if not objects:\n",
+        "            return \"Path appears clear, proceed with caution.\"\n",
+        "\n",
+        "        warnings = []\n",
+        "        for obj in objects:\n",
+        "            description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "            warnings.append(description)\n",
+        "\n",
+        "        return \". \".join(warnings) + \". Proceed with caution.\"\n",
+        "\n",
+        "    def process_audio_queue(self):\n",
+        "        \"\"\"Process audio queue with priority handling\"\"\"\n",
+        "        while self.running:\n",
+        "            try:\n",
+        "                if not self.is_speaking and not self.audio_queue.empty():\n",
+        "                    priority, text = self.audio_queue.get(timeout=1)\n",
+        "                    if text:  # Only speak if there's actual guidance\n",
+        "                        self.speak(text, priority)\n",
+        "                time.sleep(0.1)\n",
+        "            except queue.Empty:\n",
+        "                continue\n",
+        "            except Exception as e:\n",
+        "                print(f\"Audio processing error: {e}\")\n",
+        "\n",
+        "    def speak(self, text, priority=5):\n",
+        "        \"\"\"Enhanced text-to-speech with priority handling\"\"\"\n",
+        "        try:\n",
+        "            self.is_speaking = True\n",
+        "\n",
+        "            # Configure voice based on priority\n",
+        "            speaking_rate = 1.4 if priority <= 2 else 1.2\n",
+        "\n",
+        "            input_text = texttospeech.SynthesisInput(text=text)\n",
+        "            voice = texttospeech.VoiceSelectionParams(\n",
+        "                language_code=\"en-US\",\n",
+        "                name=\"en-US-Standard-F\",\n",
+        "                ssml_gender=texttospeech.SsmlVoiceGender.FEMALE,\n",
+        "            )\n",
+        "            audio_config = texttospeech.AudioConfig(\n",
+        "                audio_encoding=texttospeech.AudioEncoding.MP3,\n",
+        "                speaking_rate=speaking_rate,\n",
+        "                pitch=1 if priority > 2 else 1.2,\n",
+        "            )\n",
+        "\n",
+        "            response = self.speech_client.synthesize_speech(\n",
+        "                input=input_text, voice=voice, audio_config=audio_config\n",
+        "            )\n",
+        "\n",
+        "            # Use a temporary file with priority in name\n",
+        "            temp_file = f\"temp_audio_p{priority}_{time.time()}.mp3\"\n",
+        "            with open(temp_file, \"wb\") as out:\n",
+        "                out.write(response.audio_content)\n",
+        "\n",
+        "            pygame.mixer.music.load(temp_file)\n",
+        "            pygame.mixer.music.play()\n",
+        "\n",
+        "            while pygame.mixer.music.get_busy() and self.running:\n",
+        "                time.sleep(0.1)\n",
+        "\n",
+        "            if os.path.exists(temp_file):\n",
+        "                os.remove(temp_file)\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Speech synthesis error: {e}\")\n",
+        "        finally:\n",
+        "            self.is_speaking = False\n",
+        "\n",
+        "    def display_debug_frame(self, frame, objects):\n",
+        "        \"\"\"Display frame with object annotations for debugging\"\"\"\n",
+        "        if objects:\n",
+        "            for obj in objects:\n",
+        "                bbox = obj.bbox\n",
+        "                h, w = frame.shape[:2]\n",
+        "                pts = np.array([[int(v.x * w), int(v.y * h)] for v in bbox], np.int32)\n",
+        "\n",
+        "                # Color based on priority\n",
+        "                color = (0, 255, 0)  # Default color\n",
+        "                if obj.priority == NavigationPriority.CRITICAL:\n",
+        "                    color = (0, 0, 255)  # Red for critical\n",
+        "                elif obj.priority == NavigationPriority.HIGH:\n",
+        "                    color = (0, 165, 255)  # Orange for high priority\n",
+        "\n",
+        "                cv2.polylines(frame, [pts], True, color, 2)\n",
+        "                cv2.putText(\n",
+        "                    frame,\n",
+        "                    f\"{obj.name} (P:{obj.priority.value})\",\n",
+        "                    (int(bbox[0].x * w), int(bbox[0].y * h) - 10),\n",
+        "                    cv2.FONT_HERSHEY_SIMPLEX,\n",
+        "                    0.5,\n",
+        "                    color,\n",
+        "                    2,\n",
+        "                )\n",
+        "\n",
+        "        cv2.imshow(\"Enhanced Vision Assistant View\", frame)\n",
+        "\n",
+        "    def start(self):\n",
+        "        \"\"\"Start the enhanced vision assistant\"\"\"\n",
+        "        try:\n",
+        "            print(\"Starting Enhanced Vision Assistant...\")\n",
+        "            self.running = True\n",
+        "\n",
+        "            # Start audio processing thread\n",
+        "            self.audio_thread = threading.Thread(target=self.process_audio_queue)\n",
+        "            self.audio_thread.daemon = True\n",
+        "            self.audio_thread.start()\n",
+        "\n",
+        "            # Initialize camera\n",
+        "            self.cap = cv2.VideoCapture(0)\n",
+        "            if not self.cap.isOpened():\n",
+        "                raise Exception(\"Could not open camera\")\n",
+        "\n",
+        "            last_detection_time = 0\n",
+        "\n",
+        "            print(\"Press 'q' to quit\")\n",
+        "\n",
+        "            while self.running:\n",
+        "                ret, frame = self.cap.read()\n",
+        "                if not ret:\n",
+        "                    break\n",
+        "\n",
+        "                current_time = time.time()\n",
+        "\n",
+        "                # Perform object detection at intervals\n",
+        "                if current_time - last_detection_time >= self.DETECTION_INTERVAL:\n",
+        "                    objects = self.detect_objects_with_depth(frame)\n",
+        "\n",
+        "                    if objects:\n",
+        "                        guidance = self.generate_smart_guidance(objects)\n",
+        "                        if guidance:  # Only queue if there's meaningful guidance\n",
+        "                            # Determine priority based on scene analysis\n",
+        "                            analysis = self.agent.analyze_scene(objects)\n",
+        "                            priority = analysis[\"guidance_priority\"].value\n",
+        "                            self.audio_queue.put((priority, guidance))\n",
+        "\n",
+        "                    last_detection_time = current_time\n",
+        "\n",
+        "                # Display frame with annotations (debug view)\n",
+        "                self.display_debug_frame(frame, objects)\n",
+        "\n",
+        "                if cv2.waitKey(1) & 0xFF == ord(\"q\"):\n",
+        "                    break\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Error in main loop: {e}\")\n",
+        "        finally:\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: gemini-code-assist[bot]
body: ![critical](https://img.shields.io/badge/priority-critical-red)

The main loop in `EnhancedVisionAssistant.start()` should also have a `try...except` block to catch any unexpected errors during frame processing. This will prevent the application from crashing if an error occurs during object detection or guidance generation.

```
                try:
                    ret, frame = self.cap.read()
                    if not ret:
                        break

                    current_time = time.time()

                    # Perform object detection at intervals
                    if current_time - last_detection_time >= self.DETECTION_INTERVAL:
                        objects = self.detect_objects_with_depth(frame)

                        if objects:
                            guidance = self.generate_smart_guidance(objects)
                            if guidance:  # Only queue if there's meaningful guidance
                                # Determine priority based on scene analysis
                                analysis = self.agent.analyze_scene(objects)
                                priority = analysis["guidance_priority"].value
                                self.audio_queue.put((priority, guidance))

                        last_detection_time = current_time

                    # Display frame with annotations (debug view)
                    self.display_debug_frame(frame, objects)

                    if cv2.waitKey(1) & 0xFF == ord("q"):
                        break
                except Exception as e:
                    print(f"Error in main loop: {e}")
                    break # Exit loop on error
```
created_at: 2025-04-03T14:37:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,1159 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f705f4be70e9"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2025 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e47ccf0ace61"
+      },
+      "source": [
+        "# Enhanced Vision Assistant with Gemini Pro\n",
+        "\n",
+        "![Tech](./assets/Tech.jpg) \n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "\n",
+        "\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "\n",
+        "<div style=\"clear: both;\"></div>\n",
+        "\n",
+        "<b>Share to:</b>\n",
+        "\n",
+        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+        "</a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5a832eb3d892"
+      },
+      "source": [
+        "| | |\n",
+        "|-|-|\n",
+        "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "79329fa09037"
+      },
+      "source": [
+        "### Overview\n",
+        "\n",
+        "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using GenAI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+        "\n",
+        "### Objectives\n",
+        "\n",
+        "In this notebook, you will build a complete vision assistant system that:\n",
+        "\n",
+        "* Detects objects in real-time using Google Cloud Vision API\n",
+        "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+        "* Generates natural language navigation guidance prioritized by urgency\n",
+        "* Delivers audio instructions through Google Text-to-Speech\n",
+        "* Implements a full navigation pipeline with object tracking and context awareness\n",
+        "\n",
+        "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4c599b3db438"
+      },
+      "source": [
+        "## Solution Architecture"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "af8b43769d47"
+      },
+      "source": [
+        "![architecture](./assets/architecture.png) "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f0316df526f8"
+      },
+      "source": [
+        "## Get started"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ba5a5d175314"
+      },
+      "source": [
+        "#### Install Google Gen AI SDK for Python"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7021022e7b5f"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade --quiet google-genai"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "94c93de4621b"
+      },
+      "source": [
+        "#### Authenticate your notebook environment (Colab only)\n",
+        "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "911453311a5d"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "if \"google.colab\" in sys.modules:\n",
+        "    from google.colab import auth\n",
+        "\n",
+        "    auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ecee90c4d7f4"
+      },
+      "source": [
+        "### Setup \n",
+        "- Install required dependencies for the project"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f2a4f8987a72"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1e6301eae57"
+      },
+      "source": [
+        "### Importing the necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c95691bc96ce"
+      },
+      "outputs": [],
+      "source": [
+        "from dataclasses import dataclass\n",
+        "from enum import Enum\n",
+        "import os\n",
+        "import queue\n",
+        "import threading\n",
+        "import time\n",
+        "\n",
+        "import cv2\n",
+        "from google import genai\n",
+        "from google.cloud import texttospeech, vision_v1\n",
+        "import numpy as np\n",
+        "import pygame"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "97b83b55360b"
+      },
+      "source": [
+        "### Set Google Cloud Project and Create Client\n",
+        "\n",
+        "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+        "\n",
+        "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+        "2. Enable the required APIs:\n",
+        "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+        "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+        "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+        "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+        "\n",
+        "The code below configures your project settings and initializes the GenAI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+        "\n",
+        "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "714c768d628a"
+      },
+      "source": [
+        "#### Set your Google Cloud project details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f46f1119c789"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+        "\n",
+        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e85a4c270774"
+      },
+      "source": [
+        "#### Set up credentials (you can also use application default credentials)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3431ec5cf238"
+      },
+      "outputs": [],
+      "source": [
+        "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+        "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+        "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ec8a2b80cfc4"
+      },
+      "source": [
+        "#### Initialize the GenAI client"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "58ead05a5856"
+      },
+      "outputs": [],
+      "source": [
+        "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b55d9449233a"
+      },
+      "source": [
+        "### Define Navigation and Object Detection Classes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "04661205bc93"
+      },
+      "outputs": [],
+      "source": [
+        "# Define priority levels for navigation\n",
+        "\n",
+        "\n",
+        "class NavigationPriority(Enum):\n",
+        "    CRITICAL = 1  # Immediate collision risk\n",
+        "    HIGH = 2  # Hazardous objects or close obstacles\n",
+        "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+        "    LOW = 4  # Background information\n",
+        "    IGNORE = 5  # Not worth mentioning"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "06f08dabaae1"
+      },
+      "outputs": [],
+      "source": [
+        "@dataclass\n",
+        "class DetectedObject:\n",
+        "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+        "\n",
+        "    name: str\n",
+        "    bbox: list[vision_v1.NormalizedVertex]\n",
+        "    confidence: float\n",
+        "    depth_estimate: float\n",
+        "    priority: NavigationPriority\n",
+        "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+        "    last_seen: float = time.time()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d343df995d83"
+      },
+      "source": [
+        "### Movement Tracking"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "29db370b3335"
+      },
+      "outputs": [],
+      "source": [
+        "class MovementTracker:\n",
+        "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.movement_history = []\n",
+        "        self.last_position = None\n",
+        "\n",
+        "    def update_position(self, position):\n",
+        "        if self.last_position:\n",
+        "            movement = {\n",
+        "                \"from\": self.last_position,\n",
+        "                \"to\": position,\n",
+        "                \"timestamp\": time.time(),\n",
+        "            }\n",
+        "            self.movement_history.append(movement)\n",
+        "        self.last_position = position\n",
+        "\n",
+        "    def get_recent_movement(self):\n",
+        "        # Return last 5 movements\n",
+        "        return self.movement_history[-5:] if self.movement_history else []"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f838da6be93c"
+      },
+      "source": [
+        "### Navigation Context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "14a5eeed4642"
+      },
+      "outputs": [],
+      "source": [
+        "class NavigationContext:\n",
+        "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+        "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+        "        self.safe_paths: list[tuple[float, float]] = []\n",
+        "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+        "        self.last_guidance_time = 0\n",
+        "        self.environment_type = \"unknown\"\n",
+        "\n",
+        "    def update(self, new_objects: list[DetectedObject]):\n",
+        "        current_time = time.time()\n",
+        "        # Update known obstacles with new information\n",
+        "        for obj in new_objects:\n",
+        "            if obj.name in self.known_obstacles:\n",
+        "                old_obj = self.known_obstacles[obj.name]\n",
+        "                # Calculate object movement\n",
+        "                old_center = self.get_object_center(old_obj.bbox)\n",
+        "                new_center = self.get_object_center(obj.bbox)\n",
+        "                trajectory = (\n",
+        "                    new_center[0] - old_center[0],\n",
+        "                    new_center[1] - old_center[1],\n",
+        "                )\n",
+        "                obj.trajectory = trajectory\n",
+        "            self.known_obstacles[obj.name] = obj\n",
+        "\n",
+        "        # Clean up old obstacles\n",
+        "        self.known_obstacles = {\n",
+        "            k: v\n",
+        "            for k, v in self.known_obstacles.items()\n",
+        "            if current_time - v.last_seen < 10\n",
+        "        }  # 10 second timeout\n",
+        "\n",
+        "        # Update danger zones based on current obstacles\n",
+        "        self.update_danger_zones()\n",
+        "\n",
+        "    def get_object_center(self, bbox):\n",
+        "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+        "\n",
+        "    def update_danger_zones(self):\n",
+        "        self.danger_zones = []\n",
+        "        for obj in self.known_obstacles.values():\n",
+        "            center = self.get_object_center(obj.bbox)\n",
+        "            # Calculate danger zone radius based on object size and type\n",
+        "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+        "            radius = size * (\n",
+        "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+        "            )\n",
+        "            self.danger_zones.append((center[0], center[1], radius))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "118df7f082e5"
+      },
+      "source": [
+        "### Agent Mind - Scene Analysis Logic"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c9e38629070c"
+      },
+      "outputs": [],
+      "source": [
+        "class AgentMind:\n",
+        "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.context = NavigationContext()\n",
+        "        self.hazard_weights = {\n",
+        "            \"stairs\": 1.0,\n",
+        "            \"hole\": 1.0,\n",
+        "            \"edge\": 1.0,\n",
+        "            \"glass\": 0.9,\n",
+        "            \"knife\": 0.9,\n",
+        "            \"fire\": 1.0,\n",
+        "            \"chair\": 0.7,\n",
+        "            \"table\": 0.7,\n",
+        "            \"person\": 0.6,\n",
+        "            \"wall\": 0.8,\n",
+        "            \"door\": 0.6,\n",
+        "            \"furniture\": 0.7,\n",
+        "        }\n",
+        "\n",
+        "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+        "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+        "        analysis = {\n",
+        "            \"immediate_threats\": [],\n",
+        "            \"potential_hazards\": [],\n",
+        "            \"safe_paths\": [],\n",
+        "            \"guidance_priority\": NavigationPriority.LOW,\n",
+        "            \"recommended_action\": None,\n",
+        "        }\n",
+        "\n",
+        "        # Update navigation context\n",
+        "        self.context.update(objects)\n",
+        "\n",
+        "        # Identify immediate threats\n",
+        "        for obj in objects:\n",
+        "            if self.is_immediate_threat(obj):\n",
+        "                analysis[\"immediate_threats\"].append(obj)\n",
+        "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+        "\n",
+        "        # Find safe paths\n",
+        "        safe_paths = self.identify_safe_paths(objects)\n",
+        "        if safe_paths:\n",
+        "            analysis[\"safe_paths\"] = safe_paths\n",
+        "\n",
+        "        # Determine recommended action\n",
+        "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+        "\n",
+        "        return analysis\n",
+        "\n",
+        "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+        "        bbox = obj.bbox\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        width = bbox[2].x - bbox[0].x\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        conditions = [\n",
+        "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+        "            obj.name.lower() in self.hazard_weights\n",
+        "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+        "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+        "            any(\n",
+        "                self.is_moving_towards_user(o) for o in [obj]\n",
+        "            ),  # Object moving towards user\n",
+        "        ]\n",
+        "\n",
+        "        return any(conditions)\n",
+        "\n",
+        "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+        "        if not obj.trajectory:\n",
+        "            return False\n",
+        "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+        "\n",
+        "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+        "        \"\"\"Identify safe navigation paths\"\"\"\n",
+        "        safe_paths = []\n",
+        "        sectors = {\n",
+        "            \"left\": {\"clear\": True, \"score\": 0},\n",
+        "            \"center\": {\"clear\": True, \"score\": 0},\n",
+        "            \"right\": {\"clear\": True, \"score\": 0},\n",
+        "        }\n",
+        "\n",
+        "        for obj in objects:\n",
+        "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+        "            if center_x < 0.33:\n",
+        "                sectors[\"left\"][\"clear\"] = False\n",
+        "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            elif center_x < 0.66:\n",
+        "                sectors[\"center\"][\"clear\"] = False\n",
+        "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            else:\n",
+        "                sectors[\"right\"][\"clear\"] = False\n",
+        "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "\n",
+        "        for sector, data in sectors.items():\n",
+        "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+        "                safe_paths.append(sector)\n",
+        "\n",
+        "        return safe_paths\n",
+        "\n",
+        "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+        "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+        "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+        "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+        "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+        "        return size * hazard_weight * depth_factor\n",
+        "\n",
+        "    def determine_best_action(self, analysis: dict) -> str:\n",
+        "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+        "        if analysis[\"immediate_threats\"]:\n",
+        "            threat = analysis[\"immediate_threats\"][0]\n",
+        "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+        "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+        "\n",
+        "        if analysis[\"safe_paths\"]:\n",
+        "            if \"center\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Proceed straight ahead carefully\"\n",
+        "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly left and proceed\"\n",
+        "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly right and proceed\"\n",
+        "\n",
+        "        return \"Stop and wait for assistance\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4b601d9f3bad"
+      },
+      "source": [
+        "### Enhanced Vision Assistant - Main Application"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b739686ae411"
+      },
+      "outputs": [],
+      "source": [
+        "class EnhancedVisionAssistant:\n",
+        "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+        "\n",
+        "    def __init__(self, project_id, location, credentials_path=None):\n",
+        "        # Initialize pygame mixer for audio\n",
+        "        pygame.mixer.init(buffer=512)\n",
+        "\n",
+        "        # Configuration\n",
+        "        self.PROJECT_ID = project_id\n",
+        "        self.REGION = location\n",
+        "        self.CREDENTIALS_PATH = credentials_path\n",
+        "\n",
+        "        # Initialize state variables\n",
+        "        self.running = False\n",
+        "        self.is_speaking = False\n",
+        "        self.cap = None\n",
+        "        self.audio_thread = None\n",
+        "        self.previous_guidance = None\n",
+        "        self.movement_tracker = MovementTracker()\n",
+        "\n",
+        "        # Initialize queues\n",
+        "        self.audio_queue = queue.PriorityQueue()\n",
+        "\n",
+        "        # Set credentials if provided\n",
+        "        if self.CREDENTIALS_PATH:\n",
+        "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+        "\n",
+        "        # Initialize Google clients\n",
+        "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+        "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+        "\n",
+        "        # Initialize GenAI client\n",
+        "        self.genai_client = genai.Client(\n",
+        "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+        "        )\n",
+        "\n",
+        "        # Initialize the agent mind\n",
+        "        self.agent = AgentMind()\n",
+        "\n",
+        "        # Configuration parameters\n",
+        "        self.DETECTION_INTERVAL = 3  # seconds\n",
+        "        self.MIN_OBJECT_SIZE = 0.1\n",
+        "\n",
+        "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+        "\n",
+        "    def detect_objects_with_depth(self, frame):\n",
+        "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+        "        try:\n",
+        "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+        "            content = buffer.tobytes()\n",
+        "\n",
+        "            image = vision_v1.Image(content=content)\n",
+        "            features = [\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+        "            ]\n",
+        "\n",
+        "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+        "            response = self.vision_client.annotate_image(request=request)\n",
+        "\n",
+        "            detected_objects = []\n",
+        "            for obj in response.localized_object_annotations:\n",
+        "                bbox = obj.bounding_poly.normalized_vertices\n",
+        "                height = bbox[2].y - bbox[0].y\n",
+        "                width = bbox[2].x - bbox[0].x\n",
+        "\n",
+        "                if height * width > self.MIN_OBJECT_SIZE:\n",
+        "                    depth_estimate = 1 / (height * width)\n",
+        "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+        "\n",
+        "                    detected_obj = DetectedObject(\n",
+        "                        name=obj.name,\n",
+        "                        bbox=bbox,\n",
+        "                        confidence=obj.score,\n",
+        "                        depth_estimate=depth_estimate,\n",
+        "                        priority=priority,\n",
+        "                    )\n",
+        "                    detected_objects.append(detected_obj)\n",
+        "\n",
+        "            return detected_objects\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Object detection error: {e}\")\n",
+        "            return []\n",
+        "\n",
+        "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+        "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+        "        if self.agent.is_immediate_threat(\n",
+        "            DetectedObject(\n",
+        "                name=obj.name,\n",
+        "                bbox=bbox,\n",
+        "                confidence=obj.score,\n",
+        "                depth_estimate=depth_estimate,\n",
+        "                priority=NavigationPriority.LOW,\n",
+        "            )\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: gemini-code-assist[bot]
body: ![high](https://img.shields.io/badge/priority-high-orange)

You are creating a `DetectedObject` with `NavigationPriority.LOW` and passing it to `self.agent.is_immediate_threat()`. This seems redundant, as the priority is recalculated later. Consider removing the initial priority assignment or ensuring it aligns with the subsequent calculation.

```
        if self.agent.is_immediate_threat(
            DetectedObject(
                name=obj.name,
                bbox=bbox,
                confidence=obj.score,
                depth_estimate=depth_estimate
            )
        ):
```
created_at: 2025-04-03T14:37:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,1159 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f705f4be70e9"
+      },
+      "outputs": [],
+      "source": [
+        "# Copyright 2025 Google LLC\n",
+        "#\n",
+        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e47ccf0ace61"
+      },
+      "source": [
+        "# Enhanced Vision Assistant with Gemini Pro\n",
+        "\n",
+        "![Tech](./assets/Tech.jpg) \n",
+        "\n",
+        "<table align=\"left\">\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+        "    </a>\n",
+        "  </td>\n",
+        "\n",
+        "\n",
+        "  <td style=\"text-align: center\">\n",
+        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+        "    </a>\n",
+        "  </td>\n",
+        "</table>\n",
+        "\n",
+        "<div style=\"clear: both;\"></div>\n",
+        "\n",
+        "<b>Share to:</b>\n",
+        "\n",
+        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+        "</a>\n",
+        "\n",
+        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+        "</a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5a832eb3d892"
+      },
+      "source": [
+        "| | |\n",
+        "|-|-|\n",
+        "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "79329fa09037"
+      },
+      "source": [
+        "### Overview\n",
+        "\n",
+        "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using GenAI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+        "\n",
+        "### Objectives\n",
+        "\n",
+        "In this notebook, you will build a complete vision assistant system that:\n",
+        "\n",
+        "* Detects objects in real-time using Google Cloud Vision API\n",
+        "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+        "* Generates natural language navigation guidance prioritized by urgency\n",
+        "* Delivers audio instructions through Google Text-to-Speech\n",
+        "* Implements a full navigation pipeline with object tracking and context awareness\n",
+        "\n",
+        "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4c599b3db438"
+      },
+      "source": [
+        "## Solution Architecture"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "af8b43769d47"
+      },
+      "source": [
+        "![architecture](./assets/architecture.png) "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f0316df526f8"
+      },
+      "source": [
+        "## Get started"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ba5a5d175314"
+      },
+      "source": [
+        "#### Install Google Gen AI SDK for Python"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7021022e7b5f"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade --quiet google-genai"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "94c93de4621b"
+      },
+      "source": [
+        "#### Authenticate your notebook environment (Colab only)\n",
+        "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "911453311a5d"
+      },
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "\n",
+        "if \"google.colab\" in sys.modules:\n",
+        "    from google.colab import auth\n",
+        "\n",
+        "    auth.authenticate_user()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ecee90c4d7f4"
+      },
+      "source": [
+        "### Setup \n",
+        "- Install required dependencies for the project"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f2a4f8987a72"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b1e6301eae57"
+      },
+      "source": [
+        "### Importing the necessary libraries"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c95691bc96ce"
+      },
+      "outputs": [],
+      "source": [
+        "from dataclasses import dataclass\n",
+        "from enum import Enum\n",
+        "import os\n",
+        "import queue\n",
+        "import threading\n",
+        "import time\n",
+        "\n",
+        "import cv2\n",
+        "from google import genai\n",
+        "from google.cloud import texttospeech, vision_v1\n",
+        "import numpy as np\n",
+        "import pygame"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "97b83b55360b"
+      },
+      "source": [
+        "### Set Google Cloud Project and Create Client\n",
+        "\n",
+        "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+        "\n",
+        "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+        "2. Enable the required APIs:\n",
+        "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+        "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+        "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+        "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+        "\n",
+        "The code below configures your project settings and initializes the GenAI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+        "\n",
+        "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "714c768d628a"
+      },
+      "source": [
+        "#### Set your Google Cloud project details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "f46f1119c789"
+      },
+      "outputs": [],
+      "source": [
+        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+        "\n",
+        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e85a4c270774"
+      },
+      "source": [
+        "#### Set up credentials (you can also use application default credentials)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3431ec5cf238"
+      },
+      "outputs": [],
+      "source": [
+        "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+        "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+        "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ec8a2b80cfc4"
+      },
+      "source": [
+        "#### Initialize the GenAI client"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "58ead05a5856"
+      },
+      "outputs": [],
+      "source": [
+        "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "b55d9449233a"
+      },
+      "source": [
+        "### Define Navigation and Object Detection Classes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "04661205bc93"
+      },
+      "outputs": [],
+      "source": [
+        "# Define priority levels for navigation\n",
+        "\n",
+        "\n",
+        "class NavigationPriority(Enum):\n",
+        "    CRITICAL = 1  # Immediate collision risk\n",
+        "    HIGH = 2  # Hazardous objects or close obstacles\n",
+        "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+        "    LOW = 4  # Background information\n",
+        "    IGNORE = 5  # Not worth mentioning"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "06f08dabaae1"
+      },
+      "outputs": [],
+      "source": [
+        "@dataclass\n",
+        "class DetectedObject:\n",
+        "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+        "\n",
+        "    name: str\n",
+        "    bbox: list[vision_v1.NormalizedVertex]\n",
+        "    confidence: float\n",
+        "    depth_estimate: float\n",
+        "    priority: NavigationPriority\n",
+        "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+        "    last_seen: float = time.time()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d343df995d83"
+      },
+      "source": [
+        "### Movement Tracking"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "29db370b3335"
+      },
+      "outputs": [],
+      "source": [
+        "class MovementTracker:\n",
+        "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.movement_history = []\n",
+        "        self.last_position = None\n",
+        "\n",
+        "    def update_position(self, position):\n",
+        "        if self.last_position:\n",
+        "            movement = {\n",
+        "                \"from\": self.last_position,\n",
+        "                \"to\": position,\n",
+        "                \"timestamp\": time.time(),\n",
+        "            }\n",
+        "            self.movement_history.append(movement)\n",
+        "        self.last_position = position\n",
+        "\n",
+        "    def get_recent_movement(self):\n",
+        "        # Return last 5 movements\n",
+        "        return self.movement_history[-5:] if self.movement_history else []"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f838da6be93c"
+      },
+      "source": [
+        "### Navigation Context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "14a5eeed4642"
+      },
+      "outputs": [],
+      "source": [
+        "class NavigationContext:\n",
+        "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+        "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+        "        self.safe_paths: list[tuple[float, float]] = []\n",
+        "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+        "        self.last_guidance_time = 0\n",
+        "        self.environment_type = \"unknown\"\n",
+        "\n",
+        "    def update(self, new_objects: list[DetectedObject]):\n",
+        "        current_time = time.time()\n",
+        "        # Update known obstacles with new information\n",
+        "        for obj in new_objects:\n",
+        "            if obj.name in self.known_obstacles:\n",
+        "                old_obj = self.known_obstacles[obj.name]\n",
+        "                # Calculate object movement\n",
+        "                old_center = self.get_object_center(old_obj.bbox)\n",
+        "                new_center = self.get_object_center(obj.bbox)\n",
+        "                trajectory = (\n",
+        "                    new_center[0] - old_center[0],\n",
+        "                    new_center[1] - old_center[1],\n",
+        "                )\n",
+        "                obj.trajectory = trajectory\n",
+        "            self.known_obstacles[obj.name] = obj\n",
+        "\n",
+        "        # Clean up old obstacles\n",
+        "        self.known_obstacles = {\n",
+        "            k: v\n",
+        "            for k, v in self.known_obstacles.items()\n",
+        "            if current_time - v.last_seen < 10\n",
+        "        }  # 10 second timeout\n",
+        "\n",
+        "        # Update danger zones based on current obstacles\n",
+        "        self.update_danger_zones()\n",
+        "\n",
+        "    def get_object_center(self, bbox):\n",
+        "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+        "\n",
+        "    def update_danger_zones(self):\n",
+        "        self.danger_zones = []\n",
+        "        for obj in self.known_obstacles.values():\n",
+        "            center = self.get_object_center(obj.bbox)\n",
+        "            # Calculate danger zone radius based on object size and type\n",
+        "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+        "            radius = size * (\n",
+        "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+        "            )\n",
+        "            self.danger_zones.append((center[0], center[1], radius))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "118df7f082e5"
+      },
+      "source": [
+        "### Agent Mind - Scene Analysis Logic"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "c9e38629070c"
+      },
+      "outputs": [],
+      "source": [
+        "class AgentMind:\n",
+        "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+        "\n",
+        "    def __init__(self):\n",
+        "        self.context = NavigationContext()\n",
+        "        self.hazard_weights = {\n",
+        "            \"stairs\": 1.0,\n",
+        "            \"hole\": 1.0,\n",
+        "            \"edge\": 1.0,\n",
+        "            \"glass\": 0.9,\n",
+        "            \"knife\": 0.9,\n",
+        "            \"fire\": 1.0,\n",
+        "            \"chair\": 0.7,\n",
+        "            \"table\": 0.7,\n",
+        "            \"person\": 0.6,\n",
+        "            \"wall\": 0.8,\n",
+        "            \"door\": 0.6,\n",
+        "            \"furniture\": 0.7,\n",
+        "        }\n",
+        "\n",
+        "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+        "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+        "        analysis = {\n",
+        "            \"immediate_threats\": [],\n",
+        "            \"potential_hazards\": [],\n",
+        "            \"safe_paths\": [],\n",
+        "            \"guidance_priority\": NavigationPriority.LOW,\n",
+        "            \"recommended_action\": None,\n",
+        "        }\n",
+        "\n",
+        "        # Update navigation context\n",
+        "        self.context.update(objects)\n",
+        "\n",
+        "        # Identify immediate threats\n",
+        "        for obj in objects:\n",
+        "            if self.is_immediate_threat(obj):\n",
+        "                analysis[\"immediate_threats\"].append(obj)\n",
+        "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+        "\n",
+        "        # Find safe paths\n",
+        "        safe_paths = self.identify_safe_paths(objects)\n",
+        "        if safe_paths:\n",
+        "            analysis[\"safe_paths\"] = safe_paths\n",
+        "\n",
+        "        # Determine recommended action\n",
+        "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+        "\n",
+        "        return analysis\n",
+        "\n",
+        "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+        "        bbox = obj.bbox\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        width = bbox[2].x - bbox[0].x\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        conditions = [\n",
+        "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+        "            obj.name.lower() in self.hazard_weights\n",
+        "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+        "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+        "            any(\n",
+        "                self.is_moving_towards_user(o) for o in [obj]\n",
+        "            ),  # Object moving towards user\n",
+        "        ]\n",
+        "\n",
+        "        return any(conditions)\n",
+        "\n",
+        "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+        "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+        "        if not obj.trajectory:\n",
+        "            return False\n",
+        "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+        "\n",
+        "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+        "        \"\"\"Identify safe navigation paths\"\"\"\n",
+        "        safe_paths = []\n",
+        "        sectors = {\n",
+        "            \"left\": {\"clear\": True, \"score\": 0},\n",
+        "            \"center\": {\"clear\": True, \"score\": 0},\n",
+        "            \"right\": {\"clear\": True, \"score\": 0},\n",
+        "        }\n",
+        "\n",
+        "        for obj in objects:\n",
+        "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+        "            if center_x < 0.33:\n",
+        "                sectors[\"left\"][\"clear\"] = False\n",
+        "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            elif center_x < 0.66:\n",
+        "                sectors[\"center\"][\"clear\"] = False\n",
+        "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "            else:\n",
+        "                sectors[\"right\"][\"clear\"] = False\n",
+        "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+        "\n",
+        "        for sector, data in sectors.items():\n",
+        "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+        "                safe_paths.append(sector)\n",
+        "\n",
+        "        return safe_paths\n",
+        "\n",
+        "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+        "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+        "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+        "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+        "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+        "        return size * hazard_weight * depth_factor\n",
+        "\n",
+        "    def determine_best_action(self, analysis: dict) -> str:\n",
+        "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+        "        if analysis[\"immediate_threats\"]:\n",
+        "            threat = analysis[\"immediate_threats\"][0]\n",
+        "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+        "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+        "\n",
+        "        if analysis[\"safe_paths\"]:\n",
+        "            if \"center\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Proceed straight ahead carefully\"\n",
+        "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly left and proceed\"\n",
+        "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+        "                return \"Turn slightly right and proceed\"\n",
+        "\n",
+        "        return \"Stop and wait for assistance\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4b601d9f3bad"
+      },
+      "source": [
+        "### Enhanced Vision Assistant - Main Application"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b739686ae411"
+      },
+      "outputs": [],
+      "source": [
+        "class EnhancedVisionAssistant:\n",
+        "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+        "\n",
+        "    def __init__(self, project_id, location, credentials_path=None):\n",
+        "        # Initialize pygame mixer for audio\n",
+        "        pygame.mixer.init(buffer=512)\n",
+        "\n",
+        "        # Configuration\n",
+        "        self.PROJECT_ID = project_id\n",
+        "        self.REGION = location\n",
+        "        self.CREDENTIALS_PATH = credentials_path\n",
+        "\n",
+        "        # Initialize state variables\n",
+        "        self.running = False\n",
+        "        self.is_speaking = False\n",
+        "        self.cap = None\n",
+        "        self.audio_thread = None\n",
+        "        self.previous_guidance = None\n",
+        "        self.movement_tracker = MovementTracker()\n",
+        "\n",
+        "        # Initialize queues\n",
+        "        self.audio_queue = queue.PriorityQueue()\n",
+        "\n",
+        "        # Set credentials if provided\n",
+        "        if self.CREDENTIALS_PATH:\n",
+        "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+        "\n",
+        "        # Initialize Google clients\n",
+        "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+        "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+        "\n",
+        "        # Initialize GenAI client\n",
+        "        self.genai_client = genai.Client(\n",
+        "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+        "        )\n",
+        "\n",
+        "        # Initialize the agent mind\n",
+        "        self.agent = AgentMind()\n",
+        "\n",
+        "        # Configuration parameters\n",
+        "        self.DETECTION_INTERVAL = 3  # seconds\n",
+        "        self.MIN_OBJECT_SIZE = 0.1\n",
+        "\n",
+        "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+        "\n",
+        "    def detect_objects_with_depth(self, frame):\n",
+        "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+        "        try:\n",
+        "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+        "            content = buffer.tobytes()\n",
+        "\n",
+        "            image = vision_v1.Image(content=content)\n",
+        "            features = [\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+        "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+        "            ]\n",
+        "\n",
+        "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+        "            response = self.vision_client.annotate_image(request=request)\n",
+        "\n",
+        "            detected_objects = []\n",
+        "            for obj in response.localized_object_annotations:\n",
+        "                bbox = obj.bounding_poly.normalized_vertices\n",
+        "                height = bbox[2].y - bbox[0].y\n",
+        "                width = bbox[2].x - bbox[0].x\n",
+        "\n",
+        "                if height * width > self.MIN_OBJECT_SIZE:\n",
+        "                    depth_estimate = 1 / (height * width)\n",
+        "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+        "\n",
+        "                    detected_obj = DetectedObject(\n",
+        "                        name=obj.name,\n",
+        "                        bbox=bbox,\n",
+        "                        confidence=obj.score,\n",
+        "                        depth_estimate=depth_estimate,\n",
+        "                        priority=priority,\n",
+        "                    )\n",
+        "                    detected_objects.append(detected_obj)\n",
+        "\n",
+        "            return detected_objects\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Object detection error: {e}\")\n",
+        "            return []\n",
+        "\n",
+        "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+        "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+        "        if self.agent.is_immediate_threat(\n",
+        "            DetectedObject(\n",
+        "                name=obj.name,\n",
+        "                bbox=bbox,\n",
+        "                confidence=obj.score,\n",
+        "                depth_estimate=depth_estimate,\n",
+        "                priority=NavigationPriority.LOW,\n",
+        "            )\n",
+        "        ):\n",
+        "            return NavigationPriority.CRITICAL\n",
+        "\n",
+        "        height = bbox[2].y - bbox[0].y\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "\n",
+        "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+        "            return NavigationPriority.HIGH\n",
+        "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+        "            return NavigationPriority.MEDIUM\n",
+        "        elif height > 0.1:\n",
+        "            return NavigationPriority.LOW\n",
+        "        return NavigationPriority.IGNORE\n",
+        "\n",
+        "    def generate_smart_guidance(self, objects):\n",
+        "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+        "        try:\n",
+        "            if not objects:\n",
+        "                return None\n",
+        "\n",
+        "            # Generate scene description\n",
+        "            scene_description = []\n",
+        "            for obj in objects:\n",
+        "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+        "                scene_description.append(description)\n",
+        "\n",
+        "            # Update context\n",
+        "            context = {\n",
+        "                \"previous_guidance\": self.previous_guidance,\n",
+        "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+        "                \"environment_type\": \"indoor\",  # This could be detected\n",
+        "                \"current_speed\": \"normal\",\n",
+        "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+        "                \"safe_zones\": [],\n",
+        "                \"last_guidance_time\": time.time(),\n",
+        "            }\n",
+        "\n",
+        "            # Generate and send prompt to GenAI model\n",
+        "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+        "\n",
+        "            # Use the client's generate_content method directly\n",
+        "            response = self.genai_client.generate_content(\n",
+        "                model=\"gemini-pro\", contents=prompt\n",
+        "            )\n",
+        "\n",
+        "            # Store guidance for future context\n",
+        "            self.previous_guidance = response.text\n",
+        "            return self.previous_guidance\n",
+        "\n",
+        "        except Exception as e:\n",
+        "            print(f\"Guidance generation error: {e}\")\n",
+        "            return self.generate_fallback_guidance(objects)\n",
+        "\n",
+        "    def generate_enhanced_object_description(self, obj, bbox):\n",
+        "        \"\"\"Generate detailed object description\"\"\"\n",
+        "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+        "        position = self.calculate_relative_position(center_x)\n",
+        "        distance = self.estimate_distance(bbox[2].y - bbox[0].y, bbox[2].x - bbox[0].x)\n",
+        "\n",
+        "        description = f\"{obj.name} {position} at {distance}\"\n",
+        "        return description\n",
+        "\n",
+        "    def calculate_relative_position(self, center_x):\n",
+        "        \"\"\"Calculate relative position of object\"\"\"\n",
+        "        if center_x < 0.2:\n",
+        "            return \"far left\"\n",
+        "        elif center_x < 0.4:\n",
+        "            return \"to your left\"\n",
+        "        elif center_x < 0.6:\n",
+        "            return \"directly ahead\"\n",
+        "        elif center_x < 0.8:\n",
+        "            return \"to your right\"\n",
+        "        else:\n",
+        "            return \"far right\"\n",
+        "\n",
+        "    def estimate_distance(self, height, width):\n",
+        "        \"\"\"Estimate distance using object dimensions\"\"\"\n",
+        "        area = height * width\n",
+        "        if area > 0.5:\n",
+        "            return \"very close\"\n",
+        "        elif area > 0.3:\n",
+        "            return \"close\"\n",
+        "        elif area > 0.1:\n",
+        "            return \"moderate distance\"\n",
+        "        else:\n",
+        "            return \"far ahead\"\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

The `estimate_distance` method uses hardcoded area thresholds. These values may not be appropriate for all camera setups and environments. Consider calibrating these values or using a more sophisticated depth estimation technique.

```
        area = height * width
        if area > 0.5:
            return "very close"
        elif area > 0.3:
            return "close"
        elif area > 0.1:
            return "moderate distance"
        else:
            return "far ahead"
```
created_at: 2025-04-03T14:37:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,105 @@
+# Enhanced Vision Assistant
+
+## Overview
+
+The Enhanced Vision Assistant is an AI-powered solution designed to help visually impaired individuals navigate their surroundings safely. By combining Google Cloud's Vision API, Gen AI with Gemini Pro, and Text-to-Speech capabilities, this application provides real-time environmental awareness through audio guidance.
+
+|           |                                         |
+| --------- | --------------------------------------- |
+| Author(s) | [Prajwal](https://github.com/iprajwaal) |
+
+## 🎥 Demo Video
+
+[YouTube Video](https://youtu.be/Jpili5kx3hA)
+
+## Features
+
+- **Real-time object detection** using Google Cloud Vision API
+- **Intelligent scene analysis** with Google Gen AI and Gemini Pro
+- **Natural language navigation guidance** prioritized by urgency
+- **Audio feedback** through Google Text-to-Speech
+- **Obstacle tracking** and movement prediction
+- **Dynamic priority system** for focusing on the most relevant hazards
+
+## How It Works
+
+1. The camera captures the user's environment in real-time.
+2. Cloud Vision API detects objects and their positions.
+3. The system estimates depth and analyzes potential hazards.
+4. Gemini Pro processes the scene and generates natural language guidance.
+5. Audio instructions are delivered through text-to-speech, prioritized by urgency.
+
+## Prerequisites
+
+- Google Cloud account with billing enabled.
+- The following APIs enabled:
+  - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)
path: vision/use-cases/vision-assistant/README.md
user: login: holtskinner
body: Change this to use the Vertex AI API
created_at: 2025-04-03T16:51:00Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,105 @@
+# Enhanced Vision Assistant
+
+## Overview
+
+The Enhanced Vision Assistant is an AI-powered solution designed to help visually impaired individuals navigate their surroundings safely. By combining Google Cloud's Vision API, Gen AI with Gemini Pro, and Text-to-Speech capabilities, this application provides real-time environmental awareness through audio guidance.
+
+|           |                                         |
+| --------- | --------------------------------------- |
+| Author(s) | [Prajwal](https://github.com/iprajwaal) |
+
+## 🎥 Demo Video
+
+[YouTube Video](https://youtu.be/Jpili5kx3hA)
+
+## Features
+
+- **Real-time object detection** using Google Cloud Vision API
+- **Intelligent scene analysis** with Google Gen AI and Gemini Pro
+- **Natural language navigation guidance** prioritized by urgency
+- **Audio feedback** through Google Text-to-Speech
+- **Obstacle tracking** and movement prediction
+- **Dynamic priority system** for focusing on the most relevant hazards
+
+## How It Works
+
+1. The camera captures the user's environment in real-time.
+2. Cloud Vision API detects objects and their positions.
+3. The system estimates depth and analyzes potential hazards.
+4. Gemini Pro processes the scene and generates natural language guidance.
+5. Audio instructions are delivered through text-to-speech, prioritized by urgency.
+
+## Prerequisites
+
+- Google Cloud account with billing enabled.
+- The following APIs enabled:
+  - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)
+  - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)
+  - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)
+- Python 3.8 or higher.
+- Webcam or camera device.
+- Speaker for audio output.
+
+## Installation
+
+1. Clone the repository:
+
+   ```bash
+   git clone https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai
+   cd vision/use-cases/vision-assistant
+   ```
+
+2. Install dependencies:
+
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+3. Set up your Google Cloud credentials:
+
+   - Create a service account with access to Vision API, Text-to-Speech API, and Gen AI API.
+   - Download the service account key as JSON.
+   - Set the environment variable:
+
+     ```bash
+     export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your-service-account-key.json"
+     ```
path: vision/use-cases/vision-assistant/README.md
user: login: holtskinner
body: ```suggestion
   ```bash
   export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your-service-account-key.json"
   ```

```
created_at: 2025-04-03T16:51:16Z
author_association: COLLABORATOR
path: vision/use-cases/vision-assistant/assets/architecture.png
user: login: holtskinner
body: Change references for "Gemini Pro" and "Gemini Pro Vision" to just say "Gemini" It makes it difficult when the model updates. Do this throughout.
created_at: 2025-04-03T16:52:13Z
author_association: COLLABORATOR
path: vision/use-cases/vision-assistant/assets/architecture.png
user: login: holtskinner
body: Once the images are updated, I'll upload them to Google Cloud Storage, so they won't stay in the repo, you can link them with public urls.
created_at: 2025-04-03T16:52:53Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,1124 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright 2025 Google LLC\n",
+    "#\n",
+    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+    "# you may not use this file except in compliance with the License.\n",
+    "# You may obtain a copy of the License at\n",
+    "#\n",
+    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+    "#\n",
+    "# Unless required by applicable law or agreed to in writing, software\n",
+    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+    "# See the License for the specific language governing permissions and\n",
+    "# limitations under the License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e47ccf0ace61"
+   },
+   "source": [
+    "# Enhanced Vision Assistant with Gemini Pro\n",
+    "\n",
+    "![Tech](./assets/Tech.jpg) \n",
+    "\n",
+    "<table align=\"left\">\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+    "    </a>\n",
+    "  </td>\n",
+    "\n",
+    "\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+    "    </a>\n",
+    "  </td>\n",
+    "</table>\n",
+    "\n",
+    "<div style=\"clear: both;\"></div>\n",
+    "\n",
+    "<b>Share to:</b>\n",
+    "\n",
+    "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+    "</a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "5a832eb3d892"
+   },
+   "source": [
+    "| | |\n",
+    "|-|-|\n",
+    "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "79329fa09037"
+   },
+   "source": [
+    "### Overview\n",
+    "\n",
+    "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using Gen AI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+    "\n",
+    "### Objectives\n",
+    "\n",
+    "In this notebook, you will build a complete vision assistant system that:\n",
+    "\n",
+    "* Detects objects in real-time using Google Cloud Vision API\n",
+    "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+    "* Generates natural language navigation guidance prioritized by urgency\n",
+    "* Delivers audio instructions through Google Text-to-Speech\n",
+    "* Implements a full navigation pipeline with object tracking and context awareness\n",
+    "\n",
+    "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f0316df526f8"
+   },
+   "source": [
+    "## Get started"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ba5a5d175314"
+   },
+   "source": [
+    "#### Install Google Gen AI SDK for Python"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade --quiet google-genai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "94c93de4621b"
+   },
+   "source": [
+    "#### Authenticate your notebook environment (Colab only)\n",
+    "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "\n",
+    "if \"google.colab\" in sys.modules:\n",
+    "    from google.colab import auth\n",
+    "\n",
+    "    auth.authenticate_user()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ecee90c4d7f4"
+   },
+   "source": [
+    "### Setup \n",
+    "- Install required dependencies for the project"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: holtskinner
body: Add these installs to the same place you install google-genai and remove google-generativeai
created_at: 2025-04-03T16:53:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,1124 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright 2025 Google LLC\n",
+    "#\n",
+    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+    "# you may not use this file except in compliance with the License.\n",
+    "# You may obtain a copy of the License at\n",
+    "#\n",
+    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+    "#\n",
+    "# Unless required by applicable law or agreed to in writing, software\n",
+    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+    "# See the License for the specific language governing permissions and\n",
+    "# limitations under the License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e47ccf0ace61"
+   },
+   "source": [
+    "# Enhanced Vision Assistant with Gemini Pro\n",
+    "\n",
+    "![Tech](./assets/Tech.jpg) \n",
+    "\n",
+    "<table align=\"left\">\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+    "    </a>\n",
+    "  </td>\n",
+    "\n",
+    "\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+    "    </a>\n",
+    "  </td>\n",
+    "</table>\n",
+    "\n",
+    "<div style=\"clear: both;\"></div>\n",
+    "\n",
+    "<b>Share to:</b>\n",
+    "\n",
+    "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+    "</a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "5a832eb3d892"
+   },
+   "source": [
+    "| | |\n",
+    "|-|-|\n",
+    "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "79329fa09037"
+   },
+   "source": [
+    "### Overview\n",
+    "\n",
+    "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using Gen AI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+    "\n",
+    "### Objectives\n",
+    "\n",
+    "In this notebook, you will build a complete vision assistant system that:\n",
+    "\n",
+    "* Detects objects in real-time using Google Cloud Vision API\n",
+    "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+    "* Generates natural language navigation guidance prioritized by urgency\n",
+    "* Delivers audio instructions through Google Text-to-Speech\n",
+    "* Implements a full navigation pipeline with object tracking and context awareness\n",
+    "\n",
+    "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f0316df526f8"
+   },
+   "source": [
+    "## Get started"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ba5a5d175314"
+   },
+   "source": [
+    "#### Install Google Gen AI SDK for Python"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade --quiet google-genai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "94c93de4621b"
+   },
+   "source": [
+    "#### Authenticate your notebook environment (Colab only)\n",
+    "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "\n",
+    "if \"google.colab\" in sys.modules:\n",
+    "    from google.colab import auth\n",
+    "\n",
+    "    auth.authenticate_user()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ecee90c4d7f4"
+   },
+   "source": [
+    "### Setup \n",
+    "- Install required dependencies for the project"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b1e6301eae57"
+   },
+   "source": [
+    "### Importing the necessary libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from dataclasses import dataclass\n",
+    "from enum import Enum\n",
+    "import os\n",
+    "import queue\n",
+    "import threading\n",
+    "import time\n",
+    "\n",
+    "import cv2\n",
+    "from google import genai\n",
+    "from google.cloud import texttospeech, vision_v1\n",
+    "import numpy as np\n",
+    "import pygame"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "97b83b55360b"
+   },
+   "source": [
+    "### Set Google Cloud Project and Create Client\n",
+    "\n",
+    "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+    "\n",
+    "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+    "2. Enable the required APIs:\n",
+    "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: holtskinner
body: Change this to use Vertex AI
created_at: 2025-04-03T16:54:08Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,1124 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright 2025 Google LLC\n",
+    "#\n",
+    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+    "# you may not use this file except in compliance with the License.\n",
+    "# You may obtain a copy of the License at\n",
+    "#\n",
+    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+    "#\n",
+    "# Unless required by applicable law or agreed to in writing, software\n",
+    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+    "# See the License for the specific language governing permissions and\n",
+    "# limitations under the License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e47ccf0ace61"
+   },
+   "source": [
+    "# Enhanced Vision Assistant with Gemini Pro\n",
+    "\n",
+    "![Tech](./assets/Tech.jpg) \n",
+    "\n",
+    "<table align=\"left\">\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+    "    </a>\n",
+    "  </td>\n",
+    "\n",
+    "\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+    "    </a>\n",
+    "  </td>\n",
+    "</table>\n",
+    "\n",
+    "<div style=\"clear: both;\"></div>\n",
+    "\n",
+    "<b>Share to:</b>\n",
+    "\n",
+    "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+    "</a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "5a832eb3d892"
+   },
+   "source": [
+    "| | |\n",
+    "|-|-|\n",
+    "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "79329fa09037"
+   },
+   "source": [
+    "### Overview\n",
+    "\n",
+    "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using Gen AI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+    "\n",
+    "### Objectives\n",
+    "\n",
+    "In this notebook, you will build a complete vision assistant system that:\n",
+    "\n",
+    "* Detects objects in real-time using Google Cloud Vision API\n",
+    "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+    "* Generates natural language navigation guidance prioritized by urgency\n",
+    "* Delivers audio instructions through Google Text-to-Speech\n",
+    "* Implements a full navigation pipeline with object tracking and context awareness\n",
+    "\n",
+    "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f0316df526f8"
+   },
+   "source": [
+    "## Get started"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ba5a5d175314"
+   },
+   "source": [
+    "#### Install Google Gen AI SDK for Python"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade --quiet google-genai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "94c93de4621b"
+   },
+   "source": [
+    "#### Authenticate your notebook environment (Colab only)\n",
+    "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "\n",
+    "if \"google.colab\" in sys.modules:\n",
+    "    from google.colab import auth\n",
+    "\n",
+    "    auth.authenticate_user()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ecee90c4d7f4"
+   },
+   "source": [
+    "### Setup \n",
+    "- Install required dependencies for the project"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b1e6301eae57"
+   },
+   "source": [
+    "### Importing the necessary libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from dataclasses import dataclass\n",
+    "from enum import Enum\n",
+    "import os\n",
+    "import queue\n",
+    "import threading\n",
+    "import time\n",
+    "\n",
+    "import cv2\n",
+    "from google import genai\n",
+    "from google.cloud import texttospeech, vision_v1\n",
+    "import numpy as np\n",
+    "import pygame"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "97b83b55360b"
+   },
+   "source": [
+    "### Set Google Cloud Project and Create Client\n",
+    "\n",
+    "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+    "\n",
+    "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+    "2. Enable the required APIs:\n",
+    "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+    "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+    "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+    "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+    "\n",
+    "The code below configures your project settings and initializes the Gen AI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+    "\n",
+    "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "714c768d628a"
+   },
+   "source": [
+    "#### Set your Google Cloud project details"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+    "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+    "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+    "\n",
+    "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e85a4c270774"
+   },
+   "source": [
+    "#### Set up credentials (you can also use application default credentials)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+    "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+    "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ec8a2b80cfc4"
+   },
+   "source": [
+    "#### Initialize the Gen AI client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b55d9449233a"
+   },
+   "source": [
+    "### Define Navigation and Object Detection Classes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define priority levels for navigation\n",
+    "\n",
+    "\n",
+    "class NavigationPriority(Enum):\n",
+    "    CRITICAL = 1  # Immediate collision risk\n",
+    "    HIGH = 2  # Hazardous objects or close obstacles\n",
+    "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+    "    LOW = 4  # Background information\n",
+    "    IGNORE = 5  # Not worth mentioning"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@dataclass\n",
+    "class DetectedObject:\n",
+    "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+    "\n",
+    "    name: str\n",
+    "    bbox: list[vision_v1.NormalizedVertex]\n",
+    "    confidence: float\n",
+    "    depth_estimate: float\n",
+    "    priority: NavigationPriority\n",
+    "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+    "    last_seen: float = time.time()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "d343df995d83"
+   },
+   "source": [
+    "### Movement Tracking"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class MovementTracker:\n",
+    "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.movement_history = []\n",
+    "        self.last_position = None\n",
+    "\n",
+    "    def update_position(self, position):\n",
+    "        if self.last_position:\n",
+    "            movement = {\n",
+    "                \"from\": self.last_position,\n",
+    "                \"to\": position,\n",
+    "                \"timestamp\": time.time(),\n",
+    "            }\n",
+    "            self.movement_history.append(movement)\n",
+    "        self.last_position = position\n",
+    "\n",
+    "    def get_recent_movement(self):\n",
+    "        # Return last 5 movements\n",
+    "        return self.movement_history[-5:] if self.movement_history else []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f838da6be93c"
+   },
+   "source": [
+    "### Navigation Context"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class NavigationContext:\n",
+    "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+    "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+    "        self.safe_paths: list[tuple[float, float]] = []\n",
+    "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+    "        self.last_guidance_time = 0\n",
+    "        self.environment_type = \"unknown\"\n",
+    "\n",
+    "    def update(self, new_objects: list[DetectedObject]):\n",
+    "        current_time = time.time()\n",
+    "        # Update known obstacles with new information\n",
+    "        for obj in new_objects:\n",
+    "            if obj.name in self.known_obstacles:\n",
+    "                old_obj = self.known_obstacles[obj.name]\n",
+    "                # Calculate object movement\n",
+    "                old_center = self.get_object_center(old_obj.bbox)\n",
+    "                new_center = self.get_object_center(obj.bbox)\n",
+    "                trajectory = (\n",
+    "                    new_center[0] - old_center[0],\n",
+    "                    new_center[1] - old_center[1],\n",
+    "                )\n",
+    "                obj.trajectory = trajectory\n",
+    "            self.known_obstacles[obj.name] = obj\n",
+    "\n",
+    "        # Clean up old obstacles\n",
+    "        self.known_obstacles = {\n",
+    "            k: v\n",
+    "            for k, v in self.known_obstacles.items()\n",
+    "            if current_time - v.last_seen < 10\n",
+    "        }  # 10 second timeout\n",
+    "\n",
+    "        # Update danger zones based on current obstacles\n",
+    "        self.update_danger_zones()\n",
+    "\n",
+    "    def get_object_center(self, bbox):\n",
+    "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+    "\n",
+    "    def update_danger_zones(self):\n",
+    "        self.danger_zones = []\n",
+    "        for obj in self.known_obstacles.values():\n",
+    "            center = self.get_object_center(obj.bbox)\n",
+    "            # Calculate danger zone radius based on object size and type\n",
+    "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+    "            radius = size * (\n",
+    "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+    "            )\n",
+    "            self.danger_zones.append((center[0], center[1], radius))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "118df7f082e5"
+   },
+   "source": [
+    "### Agent Mind - Scene Analysis Logic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class AgentMind:\n",
+    "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.context = NavigationContext()\n",
+    "        self.hazard_weights = {\n",
+    "            \"stairs\": 1.0,\n",
+    "            \"hole\": 1.0,\n",
+    "            \"edge\": 1.0,\n",
+    "            \"glass\": 0.9,\n",
+    "            \"knife\": 0.9,\n",
+    "            \"fire\": 1.0,\n",
+    "            \"chair\": 0.7,\n",
+    "            \"table\": 0.7,\n",
+    "            \"person\": 0.6,\n",
+    "            \"wall\": 0.8,\n",
+    "            \"door\": 0.6,\n",
+    "            \"furniture\": 0.7,\n",
+    "        }\n",
+    "\n",
+    "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+    "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+    "        analysis = {\n",
+    "            \"immediate_threats\": [],\n",
+    "            \"potential_hazards\": [],\n",
+    "            \"safe_paths\": [],\n",
+    "            \"guidance_priority\": NavigationPriority.LOW,\n",
+    "            \"recommended_action\": None,\n",
+    "        }\n",
+    "\n",
+    "        # Update navigation context\n",
+    "        self.context.update(objects)\n",
+    "\n",
+    "        # Identify immediate threats\n",
+    "        for obj in objects:\n",
+    "            if self.is_immediate_threat(obj):\n",
+    "                analysis[\"immediate_threats\"].append(obj)\n",
+    "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+    "\n",
+    "        # Find safe paths\n",
+    "        safe_paths = self.identify_safe_paths(objects)\n",
+    "        if safe_paths:\n",
+    "            analysis[\"safe_paths\"] = safe_paths\n",
+    "\n",
+    "        # Determine recommended action\n",
+    "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+    "\n",
+    "        return analysis\n",
+    "\n",
+    "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+    "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+    "        bbox = obj.bbox\n",
+    "        height = bbox[2].y - bbox[0].y\n",
+    "        width = bbox[2].x - bbox[0].x\n",
+    "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+    "\n",
+    "        conditions = [\n",
+    "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+    "            obj.name.lower() in self.hazard_weights\n",
+    "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+    "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+    "            any(\n",
+    "                self.is_moving_towards_user(o) for o in [obj]\n",
+    "            ),  # Object moving towards user\n",
+    "        ]\n",
+    "\n",
+    "        return any(conditions)\n",
+    "\n",
+    "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+    "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+    "        if not obj.trajectory:\n",
+    "            return False\n",
+    "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+    "\n",
+    "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+    "        \"\"\"Identify safe navigation paths\"\"\"\n",
+    "        safe_paths = []\n",
+    "        sectors = {\n",
+    "            \"left\": {\"clear\": True, \"score\": 0},\n",
+    "            \"center\": {\"clear\": True, \"score\": 0},\n",
+    "            \"right\": {\"clear\": True, \"score\": 0},\n",
+    "        }\n",
+    "\n",
+    "        for obj in objects:\n",
+    "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+    "            if center_x < 0.33:\n",
+    "                sectors[\"left\"][\"clear\"] = False\n",
+    "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "            elif center_x < 0.66:\n",
+    "                sectors[\"center\"][\"clear\"] = False\n",
+    "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "            else:\n",
+    "                sectors[\"right\"][\"clear\"] = False\n",
+    "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "\n",
+    "        for sector, data in sectors.items():\n",
+    "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+    "                safe_paths.append(sector)\n",
+    "\n",
+    "        return safe_paths\n",
+    "\n",
+    "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+    "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+    "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+    "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+    "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+    "        return size * hazard_weight * depth_factor\n",
+    "\n",
+    "    def determine_best_action(self, analysis: dict) -> str:\n",
+    "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+    "        if analysis[\"immediate_threats\"]:\n",
+    "            threat = analysis[\"immediate_threats\"][0]\n",
+    "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+    "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+    "\n",
+    "        if analysis[\"safe_paths\"]:\n",
+    "            if \"center\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Proceed straight ahead carefully\"\n",
+    "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Turn slightly left and proceed\"\n",
+    "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Turn slightly right and proceed\"\n",
+    "\n",
+    "        return \"Stop and wait for assistance\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4b601d9f3bad"
+   },
+   "source": [
+    "### Enhanced Vision Assistant - Main Application"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "b739686ae411"
+   },
+   "outputs": [],
+   "source": [
+    "class EnhancedVisionAssistant:\n",
+    "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+    "\n",
+    "    def __init__(self, project_id, location, credentials_path=None):\n",
+    "        # Initialize pygame mixer for audio\n",
+    "        pygame.mixer.init(buffer=512)\n",
+    "\n",
+    "        # Configuration\n",
+    "        self.PROJECT_ID = project_id\n",
+    "        self.REGION = location\n",
+    "        self.CREDENTIALS_PATH = credentials_path\n",
+    "\n",
+    "        # Initialize state variables\n",
+    "        self.running = False\n",
+    "        self.is_speaking = False\n",
+    "        self.cap = None\n",
+    "        self.audio_thread = None\n",
+    "        self.previous_guidance = None\n",
+    "        self.movement_tracker = MovementTracker()\n",
+    "\n",
+    "        # Initialize queues\n",
+    "        self.audio_queue = queue.PriorityQueue()\n",
+    "\n",
+    "        # Set credentials if provided\n",
+    "        if self.CREDENTIALS_PATH:\n",
+    "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+    "\n",
+    "        # Initialize Google clients\n",
+    "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+    "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+    "\n",
+    "        # Initialize Gen AI client\n",
+    "        self.genai_client = genai.Client(\n",
+    "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+    "        )\n",
+    "\n",
+    "        # Initialize the agent mind\n",
+    "        self.agent = AgentMind()\n",
+    "\n",
+    "        # Configuration parameters\n",
+    "        self.DETECTION_INTERVAL = 3  # seconds\n",
+    "        self.MIN_OBJECT_SIZE = 0.1\n",
+    "\n",
+    "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+    "\n",
+    "    def detect_objects_with_depth(self, frame):\n",
+    "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+    "        try:\n",
+    "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+    "            content = buffer.tobytes()\n",
+    "\n",
+    "            image = vision_v1.Image(content=content)\n",
+    "            features = [\n",
+    "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+    "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+    "            ]\n",
+    "\n",
+    "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+    "            response = self.vision_client.annotate_image(request=request)\n",
+    "\n",
+    "            detected_objects = []\n",
+    "            for obj in response.localized_object_annotations:\n",
+    "                bbox = obj.bounding_poly.normalized_vertices\n",
+    "                height = bbox[2].y - bbox[0].y\n",
+    "                width = bbox[2].x - bbox[0].x\n",
+    "\n",
+    "                if height * width > self.MIN_OBJECT_SIZE:\n",
+    "                    depth_estimate = 1 / (height * width)\n",
+    "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+    "\n",
+    "                    detected_obj = DetectedObject(\n",
+    "                        name=obj.name,\n",
+    "                        bbox=bbox,\n",
+    "                        confidence=obj.score,\n",
+    "                        depth_estimate=depth_estimate,\n",
+    "                        priority=priority,\n",
+    "                    )\n",
+    "                    detected_objects.append(detected_obj)\n",
+    "\n",
+    "            return detected_objects\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            print(f\"Object detection error: {e}\")\n",
+    "            return []\n",
+    "\n",
+    "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+    "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+    "        if self.agent.is_immediate_threat(\n",
+    "            DetectedObject(\n",
+    "                name=obj.name,\n",
+    "                bbox=bbox,\n",
+    "                confidence=obj.score,\n",
+    "                depth_estimate=depth_estimate,\n",
+    "            )\n",
+    "        ):\n",
+    "            return NavigationPriority.CRITICAL\n",
+    "\n",
+    "        height = bbox[2].y - bbox[0].y\n",
+    "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+    "\n",
+    "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+    "            return NavigationPriority.HIGH\n",
+    "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+    "            return NavigationPriority.MEDIUM\n",
+    "        elif height > 0.1:\n",
+    "            return NavigationPriority.LOW\n",
+    "        return NavigationPriority.IGNORE\n",
+    "\n",
+    "    def generate_smart_guidance(self, objects):\n",
+    "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+    "        try:\n",
+    "            if not objects:\n",
+    "                return None\n",
+    "\n",
+    "            # Generate scene description\n",
+    "            scene_description = []\n",
+    "            for obj in objects:\n",
+    "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+    "                scene_description.append(description)\n",
+    "\n",
+    "            # Update context\n",
+    "            context = {\n",
+    "                \"previous_guidance\": self.previous_guidance,\n",
+    "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+    "                \"environment_type\": \"indoor\",  # This could be detected\n",
+    "                \"current_speed\": \"normal\",\n",
+    "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+    "                \"safe_zones\": [],\n",
+    "                \"last_guidance_time\": time.time(),\n",
+    "            }\n",
+    "\n",
+    "            # Generate and send prompt to Gen AI model\n",
+    "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+    "\n",
+    "            # Use the client's generate_content method directly\n",
+    "            response = self.genai_client.generate_content(\n",
+    "                model=\"gemini-pro\", contents=prompt\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: holtskinner
body: Update this to use Gemini 2.5 Pro, Gemini 2.0 Flash, or Gemini 2.0 Flash-Lite
created_at: 2025-04-03T16:57:18Z
author_association: COLLABORATOR
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: holtskinner
body: Can you move this to the Gemini directory? The vision directory is more for Imagen/Veo.

I'm also not sure that a Notebook is the best way to show this demo, it seems very cool, but I think it would work better as a deployed application, not a notebook tutorial.


created_at: 2025-04-03T16:58:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,1124 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Copyright 2025 Google LLC\n",
+    "#\n",
+    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+    "# you may not use this file except in compliance with the License.\n",
+    "# You may obtain a copy of the License at\n",
+    "#\n",
+    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
+    "#\n",
+    "# Unless required by applicable law or agreed to in writing, software\n",
+    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+    "# See the License for the specific language governing permissions and\n",
+    "# limitations under the License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e47ccf0ace61"
+   },
+   "source": [
+    "# Enhanced Vision Assistant with Gemini Pro\n",
+    "\n",
+    "![Tech](./assets/Tech.jpg) \n",
+    "\n",
+    "<table align=\"left\">\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
+    "    </a>\n",
+    "  </td>\n",
+    "\n",
+    "\n",
+    "  <td style=\"text-align: center\">\n",
+    "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\">\n",
+    "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
+    "    </a>\n",
+    "  </td>\n",
+    "</table>\n",
+    "\n",
+    "<div style=\"clear: both;\"></div>\n",
+    "\n",
+    "<b>Share to:</b>\n",
+    "\n",
+    "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
+    "</a>\n",
+    "\n",
+    "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb\" target=\"_blank\">\n",
+    "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
+    "</a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "5a832eb3d892"
+   },
+   "source": [
+    "| | |\n",
+    "|-|-|\n",
+    "|Author(s) | [Prajwal](https://github.com/iprajwaal) |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "79329fa09037"
+   },
+   "source": [
+    "### Overview\n",
+    "\n",
+    "The Enhanced Vision Assistant shows how Google Cloud services can be combined to create assistive technology for visually impaired users. This solution delivers real-time environmental awareness through computer vision and AI-powered guidance. Using Gen AI with Gemini Pro alongside Cloud Vision and Text-to-Speech APIs, the system transforms visual information into helpful audio navigation cues.\n",
+    "\n",
+    "### Objectives\n",
+    "\n",
+    "In this notebook, you will build a complete vision assistant system that:\n",
+    "\n",
+    "* Detects objects in real-time using Google Cloud Vision API\n",
+    "* Analyzes scenes with Gemini Pro to identify hazards and safe paths\n",
+    "* Generates natural language navigation guidance prioritized by urgency\n",
+    "* Delivers audio instructions through Google Text-to-Speech\n",
+    "* Implements a full navigation pipeline with object tracking and context awareness\n",
+    "\n",
+    "This cookbook demonstrates practical AI application development for accessibility technology using Google Cloud's API ecosystem."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f0316df526f8"
+   },
+   "source": [
+    "## Get started"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ba5a5d175314"
+   },
+   "source": [
+    "#### Install Google Gen AI SDK for Python"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade --quiet google-genai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "94c93de4621b"
+   },
+   "source": [
+    "#### Authenticate your notebook environment (Colab only)\n",
+    "##### If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "\n",
+    "if \"google.colab\" in sys.modules:\n",
+    "    from google.colab import auth\n",
+    "\n",
+    "    auth.authenticate_user()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ecee90c4d7f4"
+   },
+   "source": [
+    "### Setup \n",
+    "- Install required dependencies for the project"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --upgrade google-generativeai opencv-python pygame numpy scipy google-cloud-vision google-cloud-texttospeech "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b1e6301eae57"
+   },
+   "source": [
+    "### Importing the necessary libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from dataclasses import dataclass\n",
+    "from enum import Enum\n",
+    "import os\n",
+    "import queue\n",
+    "import threading\n",
+    "import time\n",
+    "\n",
+    "import cv2\n",
+    "from google import genai\n",
+    "from google.cloud import texttospeech, vision_v1\n",
+    "import numpy as np\n",
+    "import pygame"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "97b83b55360b"
+   },
+   "source": [
+    "### Set Google Cloud Project and Create Client\n",
+    "\n",
+    "Before running this notebook, you'll need a Google Cloud project with the necessary APIs enabled. If you haven't already set up your environment:\n",
+    "\n",
+    "1. [Create or select a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager)\n",
+    "2. Enable the required APIs:\n",
+    "   - [Google AI (Generative AI) API](https://console.cloud.google.com/flows/enableapi?apiid=generativelanguage.googleapis.com)\n",
+    "   - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)\n",
+    "   - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)\n",
+    "3. [Set up authentication](https://cloud.google.com/docs/authentication/getting-started)\n",
+    "\n",
+    "The code below configures your project settings and initializes the Gen AI client that will be used throughout this notebook. Simply update the project ID or use environment variables if running in a [managed environment](https://cloud.google.com/vertex-ai/docs/workbench/managed/introduction).\n",
+    "\n",
+    "For more detailed setup instructions, see [Google Cloud's documentation on setting up a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "714c768d628a"
+   },
+   "source": [
+    "#### Set your Google Cloud project details"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
+    "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
+    "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
+    "\n",
+    "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "e85a4c270774"
+   },
+   "source": [
+    "#### Set up credentials (you can also use application default credentials)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "CREDENTIALS_PATH = \"[path-to-your-credentials]\"  # @param {type: \"string\", placeholder: \"[path-to-your-credentials]\", isTemplate: true}\n",
+    "if CREDENTIALS_PATH and CREDENTIALS_PATH != \"[path-to-your-credentials]\":\n",
+    "    os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = CREDENTIALS_PATH"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ec8a2b80cfc4"
+   },
+   "source": [
+    "#### Initialize the Gen AI client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "genai_client = genai.Client(project=PROJECT_ID, location=LOCATION, vertexai=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b55d9449233a"
+   },
+   "source": [
+    "### Define Navigation and Object Detection Classes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define priority levels for navigation\n",
+    "\n",
+    "\n",
+    "class NavigationPriority(Enum):\n",
+    "    CRITICAL = 1  # Immediate collision risk\n",
+    "    HIGH = 2  # Hazardous objects or close obstacles\n",
+    "    MEDIUM = 3  # Notable obstacles at moderate distance\n",
+    "    LOW = 4  # Background information\n",
+    "    IGNORE = 5  # Not worth mentioning"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@dataclass\n",
+    "class DetectedObject:\n",
+    "    \"\"\"Class for detected objects with navigation-relevant attributes\"\"\"\n",
+    "\n",
+    "    name: str\n",
+    "    bbox: list[vision_v1.NormalizedVertex]\n",
+    "    confidence: float\n",
+    "    depth_estimate: float\n",
+    "    priority: NavigationPriority\n",
+    "    trajectory: tuple[float, float] = (0, 0)  # x, y movement\n",
+    "    last_seen: float = time.time()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "d343df995d83"
+   },
+   "source": [
+    "### Movement Tracking"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class MovementTracker:\n",
+    "    \"\"\"Tracks movement history to provide context for navigation decisions\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.movement_history = []\n",
+    "        self.last_position = None\n",
+    "\n",
+    "    def update_position(self, position):\n",
+    "        if self.last_position:\n",
+    "            movement = {\n",
+    "                \"from\": self.last_position,\n",
+    "                \"to\": position,\n",
+    "                \"timestamp\": time.time(),\n",
+    "            }\n",
+    "            self.movement_history.append(movement)\n",
+    "        self.last_position = position\n",
+    "\n",
+    "    def get_recent_movement(self):\n",
+    "        # Return last 5 movements\n",
+    "        return self.movement_history[-5:] if self.movement_history else []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "f838da6be93c"
+   },
+   "source": [
+    "### Navigation Context"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class NavigationContext:\n",
+    "    \"\"\"Maintains context about navigation environment and obstacles\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.known_obstacles: dict[str, DetectedObject] = {}\n",
+    "        self.danger_zones: list[tuple[float, float, float]] = []  # x, y, radius\n",
+    "        self.safe_paths: list[tuple[float, float]] = []\n",
+    "        self.user_movement = {\"speed\": 0, \"direction\": 0}\n",
+    "        self.last_guidance_time = 0\n",
+    "        self.environment_type = \"unknown\"\n",
+    "\n",
+    "    def update(self, new_objects: list[DetectedObject]):\n",
+    "        current_time = time.time()\n",
+    "        # Update known obstacles with new information\n",
+    "        for obj in new_objects:\n",
+    "            if obj.name in self.known_obstacles:\n",
+    "                old_obj = self.known_obstacles[obj.name]\n",
+    "                # Calculate object movement\n",
+    "                old_center = self.get_object_center(old_obj.bbox)\n",
+    "                new_center = self.get_object_center(obj.bbox)\n",
+    "                trajectory = (\n",
+    "                    new_center[0] - old_center[0],\n",
+    "                    new_center[1] - old_center[1],\n",
+    "                )\n",
+    "                obj.trajectory = trajectory\n",
+    "            self.known_obstacles[obj.name] = obj\n",
+    "\n",
+    "        # Clean up old obstacles\n",
+    "        self.known_obstacles = {\n",
+    "            k: v\n",
+    "            for k, v in self.known_obstacles.items()\n",
+    "            if current_time - v.last_seen < 10\n",
+    "        }  # 10 second timeout\n",
+    "\n",
+    "        # Update danger zones based on current obstacles\n",
+    "        self.update_danger_zones()\n",
+    "\n",
+    "    def get_object_center(self, bbox):\n",
+    "        return ((bbox[0].x + bbox[2].x) / 2, (bbox[0].y + bbox[2].y) / 2)\n",
+    "\n",
+    "    def update_danger_zones(self):\n",
+    "        self.danger_zones = []\n",
+    "        for obj in self.known_obstacles.values():\n",
+    "            center = self.get_object_center(obj.bbox)\n",
+    "            # Calculate danger zone radius based on object size and type\n",
+    "            size = (obj.bbox[2].x - obj.bbox[0].x) * (obj.bbox[2].y - obj.bbox[0].y)\n",
+    "            radius = size * (\n",
+    "                1.5 if obj.priority == NavigationPriority.CRITICAL else 1.0\n",
+    "            )\n",
+    "            self.danger_zones.append((center[0], center[1], radius))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "118df7f082e5"
+   },
+   "source": [
+    "### Agent Mind - Scene Analysis Logic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class AgentMind:\n",
+    "    \"\"\"Core scene analysis and decision making engine\"\"\"\n",
+    "\n",
+    "    def __init__(self):\n",
+    "        self.context = NavigationContext()\n",
+    "        self.hazard_weights = {\n",
+    "            \"stairs\": 1.0,\n",
+    "            \"hole\": 1.0,\n",
+    "            \"edge\": 1.0,\n",
+    "            \"glass\": 0.9,\n",
+    "            \"knife\": 0.9,\n",
+    "            \"fire\": 1.0,\n",
+    "            \"chair\": 0.7,\n",
+    "            \"table\": 0.7,\n",
+    "            \"person\": 0.6,\n",
+    "            \"wall\": 0.8,\n",
+    "            \"door\": 0.6,\n",
+    "            \"furniture\": 0.7,\n",
+    "        }\n",
+    "\n",
+    "    def analyze_scene(self, objects: list[DetectedObject]) -> dict:\n",
+    "        \"\"\"Analyze the scene and make intelligent decisions about navigation\"\"\"\n",
+    "        analysis = {\n",
+    "            \"immediate_threats\": [],\n",
+    "            \"potential_hazards\": [],\n",
+    "            \"safe_paths\": [],\n",
+    "            \"guidance_priority\": NavigationPriority.LOW,\n",
+    "            \"recommended_action\": None,\n",
+    "        }\n",
+    "\n",
+    "        # Update navigation context\n",
+    "        self.context.update(objects)\n",
+    "\n",
+    "        # Identify immediate threats\n",
+    "        for obj in objects:\n",
+    "            if self.is_immediate_threat(obj):\n",
+    "                analysis[\"immediate_threats\"].append(obj)\n",
+    "                analysis[\"guidance_priority\"] = NavigationPriority.CRITICAL\n",
+    "\n",
+    "        # Find safe paths\n",
+    "        safe_paths = self.identify_safe_paths(objects)\n",
+    "        if safe_paths:\n",
+    "            analysis[\"safe_paths\"] = safe_paths\n",
+    "\n",
+    "        # Determine recommended action\n",
+    "        analysis[\"recommended_action\"] = self.determine_best_action(analysis)\n",
+    "\n",
+    "        return analysis\n",
+    "\n",
+    "    def is_immediate_threat(self, obj: DetectedObject) -> bool:\n",
+    "        \"\"\"Determine if an object poses an immediate threat\"\"\"\n",
+    "        bbox = obj.bbox\n",
+    "        height = bbox[2].y - bbox[0].y\n",
+    "        width = bbox[2].x - bbox[0].x\n",
+    "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+    "\n",
+    "        conditions = [\n",
+    "            height > 0.4 and 0.3 < center_x < 0.7,  # Large object directly ahead\n",
+    "            obj.name.lower() in self.hazard_weights\n",
+    "            and self.hazard_weights[obj.name.lower()] > 0.8,\n",
+    "            obj.depth_estimate < 1.5 and width > 0.3,  # Very close wide object\n",
+    "            any(\n",
+    "                self.is_moving_towards_user(o) for o in [obj]\n",
+    "            ),  # Object moving towards user\n",
+    "        ]\n",
+    "\n",
+    "        return any(conditions)\n",
+    "\n",
+    "    def is_moving_towards_user(self, obj: DetectedObject) -> bool:\n",
+    "        \"\"\"Check if object is moving towards the user\"\"\"\n",
+    "        if not obj.trajectory:\n",
+    "            return False\n",
+    "        return obj.trajectory[1] > 0.1  # Threshold for significant movement\n",
+    "\n",
+    "    def identify_safe_paths(self, objects: list[DetectedObject]) -> list[str]:\n",
+    "        \"\"\"Identify safe navigation paths\"\"\"\n",
+    "        safe_paths = []\n",
+    "        sectors = {\n",
+    "            \"left\": {\"clear\": True, \"score\": 0},\n",
+    "            \"center\": {\"clear\": True, \"score\": 0},\n",
+    "            \"right\": {\"clear\": True, \"score\": 0},\n",
+    "        }\n",
+    "\n",
+    "        for obj in objects:\n",
+    "            center_x = (obj.bbox[0].x + obj.bbox[2].x) / 2\n",
+    "            if center_x < 0.33:\n",
+    "                sectors[\"left\"][\"clear\"] = False\n",
+    "                sectors[\"left\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "            elif center_x < 0.66:\n",
+    "                sectors[\"center\"][\"clear\"] = False\n",
+    "                sectors[\"center\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "            else:\n",
+    "                sectors[\"right\"][\"clear\"] = False\n",
+    "                sectors[\"right\"][\"score\"] += self.calculate_obstacle_score(obj)\n",
+    "\n",
+    "        for sector, data in sectors.items():\n",
+    "            if data[\"clear\"] or data[\"score\"] < 0.5:\n",
+    "                safe_paths.append(sector)\n",
+    "\n",
+    "        return safe_paths\n",
+    "\n",
+    "    def calculate_obstacle_score(self, obj: DetectedObject) -> float:\n",
+    "        \"\"\"Calculate how much an obstacle blocks a path\"\"\"\n",
+    "        size = (obj.bbox[2].y - obj.bbox[0].y) * (obj.bbox[2].x - obj.bbox[0].x)\n",
+    "        hazard_weight = self.hazard_weights.get(obj.name.lower(), 0.5)\n",
+    "        depth_factor = 1 / max(obj.depth_estimate, 0.1)\n",
+    "        return size * hazard_weight * depth_factor\n",
+    "\n",
+    "    def determine_best_action(self, analysis: dict) -> str:\n",
+    "        \"\"\"Determine the best action based on scene analysis\"\"\"\n",
+    "        if analysis[\"immediate_threats\"]:\n",
+    "            threat = analysis[\"immediate_threats\"][0]\n",
+    "            center_x = (threat.bbox[0].x + threat.bbox[2].x) / 2\n",
+    "            return \"Stop and step right\" if center_x < 0.5 else \"Stop and step left\"\n",
+    "\n",
+    "        if analysis[\"safe_paths\"]:\n",
+    "            if \"center\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Proceed straight ahead carefully\"\n",
+    "            elif \"left\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Turn slightly left and proceed\"\n",
+    "            elif \"right\" in analysis[\"safe_paths\"]:\n",
+    "                return \"Turn slightly right and proceed\"\n",
+    "\n",
+    "        return \"Stop and wait for assistance\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4b601d9f3bad"
+   },
+   "source": [
+    "### Enhanced Vision Assistant - Main Application"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "id": "b739686ae411"
+   },
+   "outputs": [],
+   "source": [
+    "class EnhancedVisionAssistant:\n",
+    "    \"\"\"Main application class combining vision, AI, and audio guidance\"\"\"\n",
+    "\n",
+    "    def __init__(self, project_id, location, credentials_path=None):\n",
+    "        # Initialize pygame mixer for audio\n",
+    "        pygame.mixer.init(buffer=512)\n",
+    "\n",
+    "        # Configuration\n",
+    "        self.PROJECT_ID = project_id\n",
+    "        self.REGION = location\n",
+    "        self.CREDENTIALS_PATH = credentials_path\n",
+    "\n",
+    "        # Initialize state variables\n",
+    "        self.running = False\n",
+    "        self.is_speaking = False\n",
+    "        self.cap = None\n",
+    "        self.audio_thread = None\n",
+    "        self.previous_guidance = None\n",
+    "        self.movement_tracker = MovementTracker()\n",
+    "\n",
+    "        # Initialize queues\n",
+    "        self.audio_queue = queue.PriorityQueue()\n",
+    "\n",
+    "        # Set credentials if provided\n",
+    "        if self.CREDENTIALS_PATH:\n",
+    "            os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = self.CREDENTIALS_PATH\n",
+    "\n",
+    "        # Initialize Google clients\n",
+    "        self.vision_client = vision_v1.ImageAnnotatorClient()\n",
+    "        self.speech_client = texttospeech.TextToSpeechClient()\n",
+    "\n",
+    "        # Initialize Gen AI client\n",
+    "        self.genai_client = genai.Client(\n",
+    "            project=self.PROJECT_ID, location=self.REGION, vertexai=True\n",
+    "        )\n",
+    "\n",
+    "        # Initialize the agent mind\n",
+    "        self.agent = AgentMind()\n",
+    "\n",
+    "        # Configuration parameters\n",
+    "        self.DETECTION_INTERVAL = 3  # seconds\n",
+    "        self.MIN_OBJECT_SIZE = 0.1\n",
+    "\n",
+    "        print(f\"Enhanced Vision Assistant initialized with Gemini Pro model\")\n",
+    "\n",
+    "    def detect_objects_with_depth(self, frame):\n",
+    "        \"\"\"Detect objects and estimate their depth\"\"\"\n",
+    "        try:\n",
+    "            success, buffer = cv2.imencode(\".jpg\", frame)\n",
+    "            content = buffer.tobytes()\n",
+    "\n",
+    "            image = vision_v1.Image(content=content)\n",
+    "            features = [\n",
+    "                vision_v1.Feature(type=vision_v1.Feature.Type.OBJECT_LOCALIZATION),\n",
+    "                vision_v1.Feature(type=vision_v1.Feature.Type.LABEL_DETECTION),\n",
+    "            ]\n",
+    "\n",
+    "            request = vision_v1.AnnotateImageRequest(image=image, features=features)\n",
+    "            response = self.vision_client.annotate_image(request=request)\n",
+    "\n",
+    "            detected_objects = []\n",
+    "            for obj in response.localized_object_annotations:\n",
+    "                bbox = obj.bounding_poly.normalized_vertices\n",
+    "                height = bbox[2].y - bbox[0].y\n",
+    "                width = bbox[2].x - bbox[0].x\n",
+    "\n",
+    "                if height * width > self.MIN_OBJECT_SIZE:\n",
+    "                    depth_estimate = 1 / (height * width)\n",
+    "                    priority = self.calculate_priority(obj, bbox, depth_estimate)\n",
+    "\n",
+    "                    detected_obj = DetectedObject(\n",
+    "                        name=obj.name,\n",
+    "                        bbox=bbox,\n",
+    "                        confidence=obj.score,\n",
+    "                        depth_estimate=depth_estimate,\n",
+    "                        priority=priority,\n",
+    "                    )\n",
+    "                    detected_objects.append(detected_obj)\n",
+    "\n",
+    "            return detected_objects\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            print(f\"Object detection error: {e}\")\n",
+    "            return []\n",
+    "\n",
+    "    def calculate_priority(self, obj, bbox, depth_estimate):\n",
+    "        \"\"\"Calculate priority based on object properties\"\"\"\n",
+    "        if self.agent.is_immediate_threat(\n",
+    "            DetectedObject(\n",
+    "                name=obj.name,\n",
+    "                bbox=bbox,\n",
+    "                confidence=obj.score,\n",
+    "                depth_estimate=depth_estimate,\n",
+    "            )\n",
+    "        ):\n",
+    "            return NavigationPriority.CRITICAL\n",
+    "\n",
+    "        height = bbox[2].y - bbox[0].y\n",
+    "        center_x = (bbox[0].x + bbox[2].x) / 2\n",
+    "\n",
+    "        if height > 0.4 or obj.name.lower() in self.agent.hazard_weights:\n",
+    "            return NavigationPriority.HIGH\n",
+    "        elif 0.3 < center_x < 0.7 and height > 0.2:\n",
+    "            return NavigationPriority.MEDIUM\n",
+    "        elif height > 0.1:\n",
+    "            return NavigationPriority.LOW\n",
+    "        return NavigationPriority.IGNORE\n",
+    "\n",
+    "    def generate_smart_guidance(self, objects):\n",
+    "        \"\"\"Generate intelligent guidance using scene analysis\"\"\"\n",
+    "        try:\n",
+    "            if not objects:\n",
+    "                return None\n",
+    "\n",
+    "            # Generate scene description\n",
+    "            scene_description = []\n",
+    "            for obj in objects:\n",
+    "                description = self.generate_enhanced_object_description(obj, obj.bbox)\n",
+    "                scene_description.append(description)\n",
+    "\n",
+    "            # Update context\n",
+    "            context = {\n",
+    "                \"previous_guidance\": self.previous_guidance,\n",
+    "                \"movement_history\": self.movement_tracker.get_recent_movement(),\n",
+    "                \"environment_type\": \"indoor\",  # This could be detected\n",
+    "                \"current_speed\": \"normal\",\n",
+    "                \"recent_obstacles\": [obj.name for obj in objects],\n",
+    "                \"safe_zones\": [],\n",
+    "                \"last_guidance_time\": time.time(),\n",
+    "            }\n",
+    "\n",
+    "            # Generate and send prompt to Gen AI model\n",
+    "            prompt = self.generate_agent_prompt(scene_description, context)\n",
+    "\n",
+    "            # Use the client's generate_content method directly\n",
+    "            response = self.genai_client.generate_content(\n",
+    "                model=\"gemini-pro\", contents=prompt\n",
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: gemini-code-assist[bot]
body: Thanks for the suggestion! I'll update the notebook to use a more recent Gemini model. I'll check for the availability of Gemini 2.5 Pro, Gemini 2.0 Flash, or Gemini 2.0 Flash-Lite and update the `model` parameter accordingly in `self.genai_client.generate_content` on line 732.

```suggestion
            response = self.genai_client.generate_content(
                model="gemini-2.5-pro", contents=prompt
            )
```
created_at: 2025-04-03T16:58:39Z
author_association: CONTRIBUTOR
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: iprajwaal
body: Sure! I’ll move it to the right directory—thanks for pointing that out.  

As for the second point, I did some research and also spoke with a few folks in the GenAI space. The best fit for this would actually be an on-device deployment, like wearable glasses (think Google Lens). Unfortunately, due to resource constraints, I couldn’t implement that part yet. I’d love to collaborate if anyone’s interested in the future!  

Honestly, I’m really excited to take this project further!
created_at: 2025-04-03T17:35:40Z
author_association: CONTRIBUTOR
path: vision/use-cases/vision-assistant/enhanced_vision_assistant.ipynb
user: login: holtskinner
body: Ok, that makes some sense. We can keep the notebook for now
created_at: 2025-04-03T17:44:27Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,105 @@
+# Enhanced Vision Assistant
+
+## Overview
+
+The Enhanced Vision Assistant is an AI-powered solution designed to help visually impaired individuals navigate their surroundings safely. By combining Google Cloud's Vision API, Gen AI with Gemini, and Text-to-Speech capabilities, this application provides real-time environmental awareness through audio guidance.
+
+|           |                                         |
+| --------- | --------------------------------------- |
+| Author(s) | [Prajwal](https://github.com/iprajwaal) |
+
+## 🎥 Demo Video
+
+[YouTube Video](https://youtu.be/Jpili5kx3hA)
+
+## Features
+
+- **Real-time object detection** using Google Cloud Vision API
+- **Intelligent scene analysis** with Google Gen AI and Gemini
+- **Natural language navigation guidance** prioritized by urgency
+- **Audio feedback** through Google Text-to-Speech
+- **Obstacle tracking** and movement prediction
+- **Dynamic priority system** for focusing on the most relevant hazards
+
+## How It Works
+
+1. The camera captures the user's environment in real-time.
+2. Cloud Vision API detects objects and their positions.
+3. The system estimates depth and analyzes potential hazards.
+4. Gemini processes the scene and generates natural language guidance.
+5. Audio instructions are delivered through text-to-speech, prioritized by urgency.
+
+## Prerequisites
+
+- Google Cloud account with billing enabled.
+- The following APIs enabled:
+  - [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)
+  - [Cloud Vision API](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com)
+  - [Cloud Text-to-Speech API](https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com)
+- Python 3.8 or higher.
+- Webcam or camera device.
+- Speaker for audio output.
+
+## Installation
+
+1. Clone the repository:
+
+   ```bash
+   git clone https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai
path: gemini/use-cases/vision-assistant/README.md
user: login: holtskinner
body: I don't think this is the correct link.
created_at: 2025-04-03T17:44:58Z
author_association: COLLABORATOR
path: gemini/use-cases/vision-assistant/requirements.txt
user: login: holtskinner
body: Can you move these requirements into the notebook itself? I suspect most of them aren't actually needed explicitly.
created_at: 2025-04-03T17:46:28Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,72 @@
+annotated-types==0.7.0
+anyio==4.9.0
+appnope==0.1.4
+asttokens==3.0.0
+cachetools==5.5.2
+certifi==2025.1.31
+charset-normalizer==3.4.1
+comm==0.2.2
+debugpy==1.8.13
+decorator==5.2.1
+executing==2.2.0
+google-ai-generativelanguage==0.6.15
+google-api-core==2.24.2
+google-api-python-client==2.166.0
+google-auth==2.38.0
+google-auth-httplib2==0.2.0
+google-cloud-texttospeech==2.25.1
+google-cloud-vision==3.10.1
+google-genai==1.9.0
+google-generativeai==0.8.4
path: gemini/use-cases/vision-assistant/requirements.txt
user: login: holtskinner
body: ```suggestion
```
created_at: 2025-04-03T17:46:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,72 @@
+annotated-types==0.7.0
+anyio==4.9.0
+appnope==0.1.4
+asttokens==3.0.0
+cachetools==5.5.2
+certifi==2025.1.31
+charset-normalizer==3.4.1
+comm==0.2.2
+debugpy==1.8.13
+decorator==5.2.1
+executing==2.2.0
+google-ai-generativelanguage==0.6.15
path: gemini/use-cases/vision-assistant/requirements.txt
user: login: holtskinner
body: ```suggestion
```
created_at: 2025-04-03T17:46:49Z
author_association: COLLABORATOR
path: gemini/use-cases/vision-assistant/.gitignore
user: login: holtskinner
body: I think you can remove this file or add the missing sections to the main gitignore
created_at: 2025-04-03T17:47:43Z
author_association: COLLABORATOR
path: gemini/use-cases/vision-assistant/README.md
user: login: holtskinner
body: Since this is just a notebook, you can put all of the README content in the notebook itself.
created_at: 2025-04-03T17:48:04Z
author_association: COLLABORATOR
path: gemini/use-cases/vision-assistant/README.md
user: login: iprajwaal
body: So I will just keep the notebook and remove the rest of the files. Will it work?
created_at: 2025-04-03T17:55:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: google-cla[bot]
created_at: 2025-04-03T14:37:00Z
body: Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this [failed invocation](https://github.com/GoogleCloudPlatform/generative-ai/pull/1931/checks?check_run_id=39926103720) of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.
author_association: NONE
user: login: holtskinner
created_at: 2025-04-03T15:24:53Z
body: @iprajwaal Please resolve Test errors
author_association: COLLABORATOR
user: login: iprajwaal
created_at: 2025-04-03T16:49:42Z
body: @holtskinner looks perfectly set now! I am looking forward to contributing to more unique cookbooks in the coming days!
author_association: CONTRIBUTOR