Title: People Counter on OAK - PyImageSearch

URL Source: https://pyimagesearch.com/2023/08/21/people-counter-on-oak/

Published Time: 2023-08-21T13:00:00+00:00

Markdown Content:
* * *

* * *

**Table of Contents**
---------------------

*   [People Counter on OAK](#h2BPTitle)

*   [Introduction](#h3Introduction)
*   [Configuring Your Development Environment](#h3Development)
*   [Need Help Configuring Your Development Environment?](#h3Help)
*   [Project Structure](#h3Structure)
*   [What Is a Script Node in DepthAI?](#h3Script)
*   [Configuring the Prerequisites](#h3Configuring)
*   [Defining the Utilities](#h3Defining)

*   [Setting Up Imports](#h4Imports)
*   [Video Detection Pipeline](#h4VideoDetectionPipeline)
*   [Tracker Pipeline](#h4TrackerPipeline)
*   [Helper Functions](#h4Helper)
*   [Object Tracker Logic](#h4ObjectTrackerLogic)

*   [People Counting: Python Driver Script](#h3Driver)

*   [Results](#h4Results)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, you will learn how to construct a people-counting system on an OAK device in conjunction with DepthAI and Python. This people-counting application aims to tally the number of individuals moving “up” or “down” within a video sequence.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20368'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/08/oak-people-counter-pyimagesearch-scaled.webp)

In counting people, our first step involves detecting individuals in a specific frame, followed by the implementation of tracking on the OAK device. For the detection phase, we will harness the capabilities of a pre-trained YOLOv8 nano model and employ a tracker within the device.

Upon the conclusion of this tutorial, you will understand the procedures involved in developing and deploying a people-counting application on the OAK platform.

This lesson is the 3rd in our series on **OAK 102**:

1.  [_Training the YOLOv8 Object Detector for OAK-D_](https://pyimg.co/9qcei)
2.  [_Hand Gesture Recognition with YOLOv8 on OAK-D in Near Real-Time_](https://pyimg.co/92by6)
3.  [**_People Counter on OAK_**](https://pyimg.co/pi5v4) **(this tutorial)**

**To learn how to implement and run people counting on OAK,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**People Counter on OAK**](#TOC)
---------------------------------

* * *

### [**Introduction**](#TOCh3Introduction)

People counting is a cutting-edge application within computer vision, focusing on accurately determining the number of individuals in a particular area or moving in specific directions, such as “entering” or “exiting.” It is immensely useful across various fields (e.g., retail analytics, smart building management, and public safety).

While counting people, techniques like object detection and tracking are deployed to scrutinize and make sense of the visuals of human movement within a frame or sequence of frames. Notably, this application is greatly enhanced by utilizing edge devices like OAK, which combines advanced computer vision and artificial intelligence (AI) capabilities.

Imagine being able to analyze foot traffic in a retail store to optimize product placement or efficiently managing building occupancy for energy savings and safety — the applications are manifold.

**Figure 1** illustrates an example frame from a video sequence where individuals are accurately counted as they move in different directions. The annotations displaying the individual counts of people moving “up” or “down” are shown in real-time.

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20665%20500'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/FO1cu6ZxJNyv3d60D90XkC0_iq5ByF1Mug51tdHPJZZ4hoMo2cjOWA1Gh8oEAoSPXrduZuERiBZ9mPKB5cSUj0nEf1Fjqly1zFsJSArpDb29KznZp9X2cBH1aN3dc0Cu1YhgZwkznL1wWA01d-u9Sds)

**Figure 1:** Sample frame from a video sequence with people counting annotations (source: image by the author).

For those enthusiastic about exploring the potential of OAK devices and the array of computer vision applications they can facilitate, we suggest browsing through the [Introduction to OpenCV AI Kit (OAK)](https://pyimagesearch.com/2022/11/28/introduction-to-opencv-ai-kit-oak/) tutorial on PyImageSearch, particularly the [Applications on OAK](https://pyimagesearch.com/2022/11/28/introduction-to-opencv-ai-kit-oak/#h3Applications) section.

If you’ve been following our OAK series, you may recall that in [our previous post](https://pyimg.co/92by6), we delved into gesture recognition and how it can be applied to identify and interpret human gestures by deploying on an OAK device. Today, we are shifting gears to explore another fascinating application: people counting.

Let’s not delay any further but dive straight into this engaging tutorial to unravel the procedure for building, deploying, and running a people counting application on the OAK platform. But before that, let’s quickly discuss the topics we will cover to implement people counting on OAK.

In the first segment of this tutorial, we will address the essential Python packages necessary for building our people counter on OAK. The primary focus will be on DepthAI and OpenCV, as they are the cornerstone libraries for this application.

Next, we’ll set up the foundational elements for today’s project. Among these, a crucial component of DepthAI, known as the Script node, will be discussed. The Script node is instrumental in enabling the implementation and execution of custom tracking logic directly on the OAK device.

Subsequently, we will lay out the utilities required for the smooth operation of the people counting application on OAK. This includes the construction of video detection and tracker pipeline, as well as the creation of helper functions. Moreover, we will establish the object tracker logic, designed to operate within the Script node.

As we approach the culmination of this tutorial, we will define the primary Python driver script that integrates all the utilities and logic that have been developed. This script will be the driving force behind the successful execution of the application.

Lastly, we will evaluate the fruits of our labor by analyzing the results of employing the people counting application on OAK to real video footage. Through this, you will witness firsthand the effectiveness and practicality of the people counter in action.

* * *

### [**Configuring Your Development Environment**](#TOCh3Development)

To follow this guide, you need to have `depthai` and `opencv` libraries installed on your system.

Luckily, all these libraries are pip-installable:

$ pip install depthai==2.21.2.0
$ pip install opencv-python==4.6.0

**If you need help configuring your development environment for OpenCV, we _highly recommend_ that you read our** [**_pip install OpenCV_ guide**](https://pyimagesearch.com/2018/09/19/pip-install-opencv/) — it will have you up and running in minutes.

* * *

### [**Need Help Configuring Your Development Environment?**](#TOCh3Help)

[![Image 4: Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join PyImageSearch University — you’ll be up and running with this tutorial in minutes.](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 2:** Need help configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code on your Windows, macOS, or Linux system** **_now_****?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Project Structure**](#TOCh3Structure)

We first need to review our project directory structure.

Start by accessing this tutorial’s **_“Downloads”_** section to retrieve the source code and example images.

From there, take a look at the directory structure:

$ tree .
.
├── main.py
├── output
│   └── tracking\_result\_long.mp4
├── pyimagesearch
│   ├── \_\_init\_\_.py
│   ├── config.py
│   ├── track.py
│   └── utils.py
├── videos
│   ├── example\_01.mp4
│   └── example\_02.mp4
└── yolov8-model
    └── yolov8n
        ├── yolov8n.blob
        └── yolov8n.json

6 directories, 10 files

In the `pyimagesearch` directory, we have the following:

*   `__init__.py`: A special Python file that allows the directory to be treated as a package
*   `config.py`: The configuration file for the object tracking task
*   `utils.py`: The utilities for running the people counter on OAK (e.g., creating video and tracker pipelines and a few other helper functions)
*   `track.py`: A script that keeps track of the movement of objects across the screen in four directions and communicates with OAK to send information on the count data

In the core directory, we have the following:

*   `main.py`: The main Python driver script to run people counting on OAK leveraging the pipelines and other utilities from `utils.py`
*   `yolov8-model`: Houses YOLOv8 variants model files
*   `yolov8n`: A subdirectory inside the `yolov8-model` directory
    *   `yolov8n.blob`: A file inside the `yolov8n` directory containing the weights of the YOLOv8 model.
    *   `yolov8n.json`: A JSON file inside the `yolov8n` directory containing the configuration or metadata of the YOLOv8 model.
*   `videos`: Contains a few test video files, which the `main.py` script will use to run people counting on OAK
*   `output`: Houses people counting result video files

* * *

### [**What Is a Script Node in DepthAI?**](#TOCh3Script)

In today’s tutorial, we will use the Script node to execute people counting on the OAK device. Up to this point in our OAK series, we haven’t delved into the Script node, so let’s take a moment to understand its functionality.

In the context of the DepthAI library, a Script node (**Figure 3**) is a special node within the DepthAI pipeline that allows you to execute custom Python scripts onboard the device. It’s part of the pipeline that can be used for additional processing using Python code.

If you’re not acquainted with DepthAI pipelines, then be sure to look at [this tutorial](https://pyimagesearch.com/2022/12/19/oak-d-understanding-and-running-neural-network-inference-with-depthai-api/).

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20315'%3E%3C/svg%3E)](https://lh5.googleusercontent.com/6f4PDktURUr99leGVELoufpM0iLJA5nIXDg8bwFG-fMCRI1GS1S04M5e94TsF_f6QWQIQ9y2QatDluJI1k_od6Df5KEt3_1PeTUeu6oTOYk4xMt1nox4dakwRvUS5zfxDE1dM7HPxqs-_qKlDL_dpUM)

**Figure 3:** Script Node in DepthAI ([source](https://docs.luxonis.com/projects/api/en/latest/components/nodes/script/)).

Following are some of the key points about the Script node:

*   **Custom Processing:** It is often used for custom data processing or algorithms not covered by the built-in nodes. For example, a Script node could perform mathematical operations, data filtering, or custom logic.

*   **Python Code:** The Script node runs Python code. You can create your own custom Python script and load it into the Script node. For instance, in this tutorial, we will write a custom Python script named `track.py`, which will then be loaded into a Script node.

*   **Interfacing with Other Nodes:** The Script node can take input from other pipeline nodes and provide output to other nodes. For example, in this tutorial, we would `link` the `ObjectTracker` node output with the Script node.

The Script node in DepthAI is very useful for incorporating custom logic and processing within the DepthAI pipeline, leveraging the flexibility of Python scripting. However, as the scripts run on the device itself, it’s important to be mindful of the resource constraints and optimize the scripts accordingly.

If you are still interested in learning more about the Script node, you can check the Luxonis documentation [here](https://docs.luxonis.com/projects/api/en/latest/components/nodes/script/).

* * *

### [**Configuring the Prerequisites**](#TOCh3Configuring)

Before we start our implementation, let’s review our project’s configuration. For that, we will move on to the `config.py` script located in the `pyimagesearch` directory.

The `config.py` script sets up the necessary paths for the YOLOv8n models, their configurations, test video, and output directories for the resulting video. It also defines the camera preview dimensions and tracking threshold.

\# import necessary packages
import os

# path to the model, test data directory, and results
YOLOV8N\_MODEL = os.path.join(
    "yolov8-model", "yolov8n", "yolov8n.blob"
)  # path to yolov8 model blob
YOLOV8N\_CONFIG = os.path.join(
    "yolov8-model", "yolov8n", "yolov8n.json"
)  # path to yolov8 model configuration

We start by importing the `os` module on **Line 2**.

On **Lines 4-10**, we set up two variables, `YOLOV8N_MODEL` and `YOLOV8N_CONFIG`, to store the file paths of the YOLOv8 model’s blob file and configuration file, respectively. To accomplish this, we leverage `os.path.join` to construct the file paths.

INPUT\_VIDEO\_LONG = os.path.join("videos", "example\_02.mp4")  # path to long input video
INPUT\_VIDEO\_SHORT = os.path.join(
    "videos", "example\_01.mp4"
)  # path to short input video
OUTPUT\_VIDEO\_LONG = os.path.join(
    "output", "tracking\_result\_long.mp4"
)  # path to long output video
OUTPUT\_VIDEO\_SHORT = os.path.join(
    "output", "tracking\_result\_short.mp4"
)  # path to short output video

On **Lines 12-21**, we set the input and output video paths for the short and long video files:

*   `INPUT_VIDEO_LONG`: set to the file path of a longer input video
*   `INPUT_VIDEO_SHORT`: set to the file path of a shorter input video
*   `OUTPUT_VIDEO_LONG`: set to the file path where the inference result of the longer input video will be saved
*   `OUTPUT_VIDEO_SHORT`: set to the file path where the inference result of the shorter input video will be saved

\# minimum distance the person has to move (across the x/y axis) to be considered a real movement
DISTANCE\_THRESHOLD = 0.25

# define camera preview dimensions same as yolov8 model input size
CAMERA\_PREV\_DIM = (640, 640)

On **Line 24**, `DISTANCE_THRESHOLD` is set to `0.25`. This constant represents the minimum distance an object (a person) must move across the `x` or `y` axis to be considered as having moved.

This would help filter out small movements that should not be considered actual.

Lastly, the `CAMERA_PREV_DIM` is assigned the tuple `(640, 640)`, which has been utilized in some of our earlier tutorials in the OAK series, so we assume you are already familiar with it.

* * *

### [**Defining the Utilities**](#TOCh3Defining)

Now that the configuration is defined, we can outline the utilities needed for creating video detection and object tracker pipelines with DepthAI, as well as some helper functions. The `utils.py` script defines several functions:

*   Assists in creating a pipeline for object detection in videos using the YOLOv8 model
*   Sets up a pipeline for tracking people by taking the output from the detection pipeline and adding a Script node for the tracking logic
*   Defines a handful of helper functions for loading configuration files, annotating camera frames, and normalizing predictions

* * *

#### [**Setting Up Imports**](#TOCh4Imports)

\# import the necessary packages
import json
import logging
from pathlib import Path

import cv2
import depthai as dai
import numpy as np

from pyimagesearch import config

# set up logging configuration
logging.basicConfig(
    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)

On **Lines 2-10**, necessary packages and modules are imported:

*   `json`: to work with JSON data
*   `logging`: module to log messages
*   `Path` from `pathlib`: for handling file paths in an object-oriented way
*   `cv2`: for reading videos, processing images, and annotating frames
*   `depthai`: is the most important module for working on interacting with OAK and building pipelines for it
*   `numpy`: for array related operations
*   custom `config`: module from `pyimagesearch` for importing configuration constants

Logging is configured on **Lines 13-15** using `logging.basicConfig` to display logs when required. The parameters such as logging `level` is set to `INFO`, meaning log messages with severity `INFO` and higher will be displayed. And the `format` of the log messages to indicate the timestamp, log level, and the actual log message.

* * *

#### [**Video Detection Pipeline**](#TOCh4VideoDetectionPipeline)

def video\_detection\_pipeline(config\_path, model\_path):
    # initialize a depthai pipeline
    pipeline = dai.Pipeline()
    logging.info("Initialized DepthAI pipeline")

    # load model config file and fetch nn\_config parameters
    configPath = Path(config\_path)
    model\_config = load\_config(configPath)
    nnConfig = model\_config.get("nn\_config", {})

On **Line 18**, we define the `video_detection_pipeline` function which

*   initializes a DepthAI pipeline on **Line 20**
*   loads some configuration parameters using the `load_config` method for a YOLOv8n model and fetches `nn_config` from the JSON configuration file on **Lines 24-26**

    # using nnConfig extract metadata like classes, iou and confidence threshold, number of coordinates
    metadata = nnConfig.get("NN\_specific\_metadata", {})
    classes = metadata.get("classes", {})
    coordinates = metadata.get("coordinates", {})
    anchors = metadata.get("anchors", {})
    anchorMasks = metadata.get("anchor\_masks", {})
    iouThreshold = metadata.get("iou\_threshold", {})
    confidenceThreshold = metadata.get("confidence\_threshold", {})

In this continuation of the `video_detection_pipeline` function, the code snippet above retrieves `metadata` and other configuration parameters from the `nnConfig` dictionary. This dictionary holds configurations specific to the neural network, established on **Lines 30-35**.

The following shell block demonstrates what the `nn_config` looks like and what values it holds. From `nn_config`, we extract the key `NN_specific_metadata`, which contains the configuration necessary for object detection, and straightforwardly assigns each of these parameters to their respective variables (e.g., `classes`, `coordinates`, etc.).

"nn\_config": {
        "output\_format": "detection",
        "NN\_family": "YOLO",
        "input\_size": "640x640",
        "NN\_specific\_metadata": {
            "classes": 80,
            "coordinates": 4,
            "anchors": \[\],
            "anchor\_masks": {},
            "iou\_threshold": 0.5,
            "confidence\_threshold": 0.5
        }

For instance, in the shell block mentioned above, the parameter `classes` is set to `80` because the pre-trained YOLOv8 model we are using was trained on the [MS COCO dataset](https://public.roboflow.com/object-detection/microsoft-coco-subset), which comprises 80 classes.

How would you like immediate access to 120K+ images curated and labeled with object bounding boxes to train, explore, and experiment with … for free. Head over to [Roboflow](https://universe.roboflow.com/isl/az-6mqow?ref=pyimagesearch) and get a free account to grab these hand gesture images.

    # configure inputs for depthai pipeline
    # since this pipeline is dealing with images an XLinkIn node is created
    videoIN = pipeline.createXLinkIn()
    # create a Yolo detection node
    detectionNetwork = pipeline.create(dai.node.YoloDetectionNetwork)
    # create a XLinkOut node for fetching the neural network outputs to host
    nnOut = pipeline.create(dai.node.XLinkOut)

    # set stream names used in queue to fetch data when the pipeline is started
    nnOut.setStreamName("nn")
    videoIN.setStreamName("video\_input")

On **Line 39**, an `XLinkIn` node is instantiated within the pipeline and is assigned to the variable `videoIN`. This node is the conduit for data transmission from the host system (e.g., a computer executing the script) into the DepthAI pipeline. Typically, it is employed for transmitting images or video frames into the pipeline for further processing. In this particular scenario, it is utilized for sending video frames.

Subsequently, on **Line 41**, a node named `YoloDetectionNetwork` is instantiated within the pipeline. This node executes the YOLO object detection algorithm on the inputted video frames.

On **Line 43**, an `XLinkOut` node is instantiated within the pipeline. The variable `nnOut` will transmit the data processed by the neural network from the DepthAI pipeline back to the host system.

On **Lines 46 and 47**, the stream names for the `nnOut` and `videoIN` nodes are configured as `nn` and `video_input`, respectively. These names facilitate retrieving data from these particular output streams at the host end.

    # network specific settings - parameters read from config file
    # confidence and iou threshold, classes, coordinates are set
    # most important the model .blob file is used to load weights
    detectionNetwork.setConfidenceThreshold(confidenceThreshold)
    detectionNetwork.setNumClasses(classes)
    detectionNetwork.setCoordinateSize(coordinates)
    detectionNetwork.setAnchors(anchors)
    detectionNetwork.setAnchorMasks(anchorMasks)
    detectionNetwork.setIouThreshold(iouThreshold)
    detectionNetwork.setBlobPath(model\_path)
    detectionNetwork.setNumInferenceThreads(2)
    detectionNetwork.input.setBlocking(False)

The code on **Lines 52-60** configures the YOLO detection network node within the DepthAI pipeline with various parameters such as:

*   Setting up the confidence threshold for the detection. Detections with a confidence score below this threshold will be discarded.
*   The number of object classes the YOLO network has been trained to detect.
*   The bounding box coordinate size (`xmin`, `ymin`, `xmax`, `ymax`).
*   Sets the anchor boxes, which are used in YOLO for predicting the bounding boxes of objects.
*   Sets the anchor masks. These are used in YOLO to divide the prediction layer outputs among multiple scales.
*   Sets the Intersection over Union (IoU) threshold for non-maximum suppression. This is used to determine which boxes should be kept vs discarded.
*   The path to the model file (in blob format). This file contains the weights and architecture of the trained model.
*   The number of threads to be used for running inferences. This can be tuned for performance.
*   The input to non-blocking mode. In non-blocking mode, if the input queue is full, the node will not wait for it to free up space and will continue processing. This can be useful to keep data flowing through the pipeline without stalling.

It’s important to note that both the anchors and anchor masks are left empty following the `nn_config` dictionary that was previously observed.

    # linking the nodes - image node output is linked to detection node
    # detection network node output is linked to XLinkOut input
    videoIN.out.link(detectionNetwork.input)
    detectionNetwork.out.link(nnOut.input)

    # return the pipeline to the calling function
    logging.info("DepthAI pipeline created")
    return detectionNetwork, pipeline

In the final part of the `video_detection_pipeline` method, we establish the data flow within the DepthAI pipeline by linking the nodes in the correct order.

On **Line 64**, the output of the XLinkIn node (`videoIN`) is linked to the input of the YOLO detection network node (`detectionNetwork`). The XLinkIn node receives video frames from the host into the pipeline. This line essentially sends the image to the detection network for inference.

Then, the output of the YOLO detection network node (`detectionNetwork`) is linked to the input of the XLinkOut node (`nnOut`). This line returns the processed output (detections) to the host.

Finally, on **Lines 68 and 69**, a log message is printed to indicate that the pipeline has been successfully created, and the function returns `detectionNetwork` and `pipeline` to the calling function.

In this instance, in conjunction with the `pipeline` object, the `detectionNetwork` node is also returned because it will be connected to the object tracker node within the `object_tracker_pipeline` method. This connection is essential for further processing and tracking of the detected objects.

* * *

#### [**Tracker Pipeline**](#TOCh4TrackerPipeline)

def object\_tracker\_pipeline(pipeline, detectionNetwork):
    # create and configure the object tracker
    objectTracker = pipeline.create(dai.node.ObjectTracker)
    objectTracker.setDetectionLabelsToTrack(\[0\])  # Track people
    objectTracker.setTrackerType(dai.TrackerType.ZERO\_TERM\_COLOR\_HISTOGRAM)
    objectTracker.setTrackerIdAssignmentPolicy(
        dai.TrackerIdAssignmentPolicy.SMALLEST\_ID
    )

The function `object_tracker_pipeline` on **Line 72** takes two arguments: `pipeline` (the DepthAI pipeline) and `detectionNetwork` (the YOLO detection network node).

Within this function, on **Lines 74-79**, an object tracker node is created and configured to track people (or objects with label `0`) using a color histogram-based tracking algorithm and a specific ID assignment policy:

*   `objectTracker = pipeline.create(dai.node.ObjectTracker)`: Creates an object tracker node within the DepthAI pipeline.
*   `objectTracker.setDetectionLabelsToTrack([0])`: Configures the object tracker to only track objects with label `0`. In object detection models, different object classes have various labels (e.g., `person` might be labeled as `0`, `car` as `1`, etc.), so this line configures the tracker to track persons.
*   `objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)`: Sets the type of tracker to use. Here, `ZERO_TERM_COLOR_HISTOGRAM` is chosen, which is a specific algorithm for object tracking based on color histograms.
*   `objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.SMALLEST_ID)`: Sets the ID assignment policy for the tracker. In this case, it assigns the smallest available ID to the new objects being tracked.

    # link detection networks outputs to the object tracker
    detectionNetwork.passthrough.link(objectTracker.inputTrackerFrame)
    detectionNetwork.passthrough.link(objectTracker.inputDetectionFrame)
    detectionNetwork.out.link(objectTracker.inputDetections)

Here, we connect the detection network’s output (both the raw frames and the processed detections) to the object tracker node. This allows the object tracker to use both the raw video frames and the detection results to track objects in a video stream.

Let’s dissect each of these lines to comprehend their respective functions:

*   `detectionNetwork.passthrough.link(objectTracker.inputTrackerFrame)`: This links the raw frames (unprocessed by the neural network) from the detection network to the input tracker frame of the object tracker. This means that the object tracker will have access to the original video frames.

*   `detectionNetwork.passthrough.link(objectTracker.inputDetectionFrame)`: This links the same raw frames from the detection network to the input detection frame of the object tracker. Often, object trackers use additional information from the original image to improve tracking accuracy.

*   `detectionNetwork.out.link(objectTracker.inputDetections)`: This links the processed output from the detection network (bounding boxes, class labels, etc.) to the input detections of the object tracker. This means the object tracker will use the output of the detection network to keep track of the objects as they move across frames.

    script = pipeline.create(dai.node.Script)
    objectTracker.out.link(script.inputs\["tracklets"\])

    with open("pyimagesearch/track.py", "r") as f:
        s = f.read()
        s = s.replace("THRESH\_DIST\_DELTA", str(config.DISTANCE\_THRESHOLD))
        script.setScript(s)

This is where things become more interesting as we introduce a Script node into the DepthAI pipeline and link the object tracker’s output to the Script node’s input. Additionally, we load a Python script from a file, modify it, and set it as the script for the Script node. Let’s understand it step-by-step:

On **Line 86**, a Script node is created in the DepthAI pipeline. Then in the next line, the output of the `objectTracker` node (which contains tracking information of objects) is linked to the input of the Script node. The name `tracklets` refers to the specific input in the Script node receiving the tracking information.

Then on **Lines 89-92**,

*   Open a file named `script.py` located in the `pyimagesearch` directory for reading and read the entire content of the file into a string variable `s`.
*   We replace the placeholder text `THRESH_DIST_DELTA` within the script with the actual distance threshold value from the configuration (`config.DISTANCE_THRESHOLD`). It parameterizes the script with a specific threshold value.
*   Finally, set the modified script as the script for the Script node within the DepthAI pipeline.

    # send tracklets to the host
    trackerOut = pipeline.create(dai.node.XLinkOut)
    trackerOut.setStreamName("out")
    script.outputs\["out"\].link(trackerOut.input)

Next, we add an XLinkOut node to the DepthAI pipeline to send the `tracklets` processed by the Script node back to the host.

**Lines 95 and 96** collectively instantiate an XLinkOut node within the DepthAI pipeline and designate the stream name for this XLinkOut node as `out`. This node and naming are crucial as it facilitates identifying and retrieving this specific data stream on the host system.

**Line 97** links the output named `out` of the Script node to the input of the XLinkOut node. This effectively means the processed tracklets from the Script node will be sent back to the host through the XLinkOut node.

    # send RGB preview frames to the host
    xlinkOut = pipeline.create(dai.node.XLinkOut)
    xlinkOut.setStreamName("preview")
    objectTracker.passthroughTrackerFrame.link(xlinkOut.input)

    logging.info("Object Tracker pipeline created")
    return pipeline

In the concluding segment of the tracker function, an additional XLinkOut node is integrated into the DepthAI pipeline. However, on this occasion, it is employed to transmit the RGB preview frames, which have undergone processing by the object tracker, back to the host system.

Specifically, on **Lines 100 and 101**, an XLinkOut node is instantiated, and the stream name for this XLinkOut node is assigned as `preview`.

On **Line 102**, `objectTracker.passthroughTrackerFrame.link(xlinkOut.input)` establishes a connection between the passthrough tracker frame of the Object Tracker node and the input of the XLinkOut node. This effectively signifies that the RGB preview frames processed by the Object Tracker node will be relayed back to the host via the XLinkOut node.

Finally, **Line 105** ensures the pipeline is returned to the invoked function.

* * *

#### [**Helper Functions**](#TOCh4Helper)

def frameNorm(frame, bbox):
    # nn data, being the bounding box locations, are in <0..1> range
    # normalized them with frame width/height
    normVals = np.full(len(bbox), frame.shape\[0\])
    normVals\[::2\] = frame.shape\[1\]
    return (np.clip(np.array(bbox), 0, 1) \* normVals).astype(int)


def load\_config(config\_path):
    # open the config file and load using json module
    with config\_path.open() as f:
        config = json.load(f)
        logging.info("Config loaded from file")
        return config

The above code defines two utility functions, `frameNorm` and `load_config`, which convert neural network output to usable bounding box values and load configuration settings from a file, respectively.

On **Lines 108-113**, the `frameNorm` is defined which accepts two arguments:

*   `frame`: A frame from a video or image
*   `bbox`: A bounding box coordinate in normalized format (ranging from `0` to `1`)

The `frameNorm` function converts the normalized coordinates of the bounding box to actual pixel values based on the frame’s dimensions. This is useful for mapping the bounding box correctly on the original image. The denormalized coordinates are returned to the calling function for annotating the frame with bounding boxes.

Then, on **Lines 116-121**, we define the `load_config` function, which accepts a `config_path`: Path to a configuration file. This function opens a JSON configuration file, loads its contents into a Python dictionary, and returns it to the calling function.

def to\_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
    # resize and rearrange dimensions
    return cv2.resize(arr, shape).transpose(2, 0, 1).flatten()


def annotateFrame(frame, detections):
    # loops over all detections in a given frame
    # annotates the frame with bounding box on the object
    color = (0, 0, 255)
    for detection in detections:
        bbox = frameNorm(
            frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax)
        )
        cv2.rectangle(frame, (bbox\[0\], bbox\[1\]), (bbox\[2\], bbox\[3\]), color, 2)
    # logging.info(f"Annotated frame with {len(detections)} detections")
    return frame

We define two more utility functions, `to_planar` and `annotateFrame`, which are used for resizing images and annotating images with bounding boxes, respectively.

On **Lines 124-126**, the `to_planar` function is defined as accepting an array representing an image and a tuple representing the target shape to resize the image to. The function returns an n-dimensional array.

The `to_planar` function:

*   resizes the input array (image) to the target shape specified by the `shape` parameter
*   transposes the dimensions so that the channel comes first, followed by height and width
*   flattens the array and returns it

Next, on **Lines 129-139**, the `annotateFrame` method is defined that accepts two parameters:

*   `frame`: A frame from a video or an image that is to be annotated
*   `detections`: A list of detections, where each detection has the properties `xmin`, `ymin`, `xmax`, and `ymax` representing the coordinates of the bounding box

This function iterates through all detections in the provided `detections` list. For each detection, it:

*   normalizes the bounding box coordinates to pixel values using the previously mentioned `frameNorm` function
*   draws a rectangle on the input frame using the normalized bounding box coordinates

Finally, after looping over all the detections, the function returns the annotated frame.

class TextHelper:
    def \_\_init\_\_(self) -> None:
        self.bg\_color = (0, 0, 0)
        self.color = (255, 255, 255)
        self.text\_type = cv2.FONT\_HERSHEY\_SIMPLEX
        self.line\_type = cv2.LINE\_AA

    def putText(self, frame, text, coords):
        cv2.putText(
            frame, text, coords, self.text\_type, 1, self.bg\_color, 6, self.line\_type
        )
        cv2.putText(
            frame, text, coords, self.text\_type, 1, self.color, 2, self.line\_type
        )

In the aforementioned code, a `TextHelper` class is defined that facilitates the addition of text annotations to frames (images). This class is meticulously crafted to append text along with a background outline to enhance visibility. Here’s a succinct explanation of the class and its methods:

The `TextHelper` class is particularly useful for incorporating text annotations in images, which is especially crucial in scenarios such as object detection, where clear visibility of text against the background is essential. The code is largely self-explanatory, so you are expected to grasp the underlying logic of the operations within the `TextHelper` class.

def update(previewQ, outQ, detectionNN, text, frame, counters):
    inDet = detectionNN.get()

    if previewQ.has():
        frame = previewQ.get().getCvFrame()
        if inDet is not None and frame is not None:
            detections = inDet.detections
            annotateFrame(frame, detections)

    if outQ.has():
        jsonText = str(outQ.get().getData(), "utf-8")
        logging.debug(f"Received json text: {jsonText}")
        counters = json.loads(jsonText)

    if counters is not None:
        text.putText(frame, f"Up: {counters\['up'\]}, Down: {counters\['down'\]}", (30, 30))

    logging.info("Frame and counters updated")
    return frame, counters

The `update` function is the last in the `utils.py` script and also a very important function that will help us in the Python driver script `main.py`. It is part of an object detection and tracking pipeline, where the objects detected and tracked in video frames are processed, and the frames are annotated with relevant information.

Let’s first break down the function parameters:

*   `previewQ`: A queue that holds the preview (raw) frames
*   `outQ`: A queue that has tracking information
*   `detectionNN`: The detection queue that receives YOLO detection output from the device
*   `text`: An instance of the `TextHelper` class that helps in annotating the frames with text
*   `frame`: The current frame being processed
*   `counters`: A dictionary that keeps count of certain attributes (e.g., the number of objects moving up or down).

On **Line 159**, we fetch the detection results from the `detectionNN` queue.

Then, on **Lines 161-165**, we:

*   check if there are frames available in the `previewQ`
    *   retrieve the next frame from the queue and convert it to a format suitable for processing by OpenCV
    *   if `inDet` (YOLO detection) is not `None` and the frame is not `None`
        *   extracts detections from the input
        *   calls the `annotateFrame` function to annotate the frame with bounding boxes around the detected objects

On **Lines 167-170**,

*   check if there is tracking data is available in `outQ`
    *   retrieve the output data from the queue, convert it to a string encoded in UTF-8
    *   deserialize the JSON string into a Python dictionary (`counters`).

If `counters` is not `None` (this means that data was successfully retrieved from `outQ`):

*   We use the `TextHelper` instance to add text to the frame, displaying the counters for `up` and `down` (**Lines 172 and 173**).

Finally, the updated frame and counters dictionary are returned to the calling function.

* * *

#### [**Object Tracker Logic**](#TOCh4ObjectTrackerLogic)

\# import necessary packages
import json

# initialize the data dictionary and the counter dictionary.
# the counter dictionary tracks the number of times objects
# move up, or down across the screen.
data = {}
counter = {
    "up": 0,
    "down": 0,
}

We start by setting up some initial variables that will be used later in the script for object tracking in the video, where the directions of object movements will be counted.

On **Line 2,** we import the `json` module. This module allows you to work with JSON data (e.g., you can read JSON data from a file or write Python data structures to a file in JSON format).

An empty dictionary named `data` is initialized on **Line 7**. This `data` dictionary will be used later to store some data.

A dictionary named `counter` is initialized with keys `up`, `down` on **Lines 8-11**, and all the values are set to `0`. The `counter` dictionary will be used to keep track of the number of times objects move in the specified directions (`up`, `down`) across the frame.

\# function to send the current state of the counter as json data
def send():
    # initialize a buffer with a size of 50
    b = Buffer(50)
    # set the buffer's data to be the counter dictionary
    # converted to json and then encoded as bytes
    b.setData(json.dumps(counter).encode("utf-8"))
    # send the data via the output port of the node
    node.io\["out"\].send(b)

The above code snippet defines a function named `send`. The purpose of this function is to send the current state of the `counter` dictionary as JSON data through an output port of a node.

Here’s how the function accomplishes this:

**Line 17** creates a buffer named `b` with a size of `50`. In DepthAI, the term `Buffer` generally refers to a temporary storage area for data used to manage the data flow between the host (e.g., a PC) and the DepthAI device (e.g., OAK). When data is being processed or transferred at different rates, buffers help handle this discrepancy efficiently. In our case, the buffer `b` will hold the data that needs to be sent.

**Line 20** sets the data in buffer `b`. It takes the `counter` dictionary, converts it to a JSON-formatted string using `json.dumps(counter)`, and then encodes this string into bytes using UTF-8 encoding with `.encode("utf-8")`. The resulting bytes are set as the data for the buffer.

**Line 22** sends the data through a node’s output port, accessing the output port named “out” of a node defined in the `utils` module (**Line 96)**. The `.send(b)` function sends the data in object `b` through the output port. This is transmitting the data to the next stage in a pipeline.

\# function to handle when a tracklet (a tracked object)
# is removed and update the counter accordingly
def tracklet\_removed(tracklet, coords2):
    # get the initial coordinates of the tracklet
    coords1 = tracklet\["coords"\]
    # calculate the difference in x and y coordinates
    deltaX = coords2\[0\] - coords1\[0\]
    deltaY = coords2\[1\] - coords1\[1\]

    # check if the object moved significantly along the y-axis
    if abs(deltaY) > abs(deltaX) and abs(deltaY) > THRESH\_DIST\_DELTA:
        # if so, update the counter for the appropriate direction
        # and send the updated data
        direction = "up" if 0 > deltaY else "down"
        counter\[direction\] += 1
        send()

Next, we define a function named `tracklet_removed`. This function is designed to handle situations where a tracklet (a tracked object) is removed and updates a counter according to the direction in which the object moved. This function takes two parameters: `tracklet` and `coords2`.

On **Line 29**, we extract the initial coordinates of the tracklet from the `tracklet` dictionary using the key `coords` and assign it to the variable `coords1`.

Then, on **Lines 31 and 32**, calculate the difference in the `x` and `y` coordinates between the initial position (`coords1`) and the final position (`coords2`) of the tracklet.

The `if` condition on **Line 35** checks if the object moved significantly along the `y`\-axis. It does this by comparing the absolute value of `deltaY` with the absolute value of `deltaX` and checking if the absolute value of `deltaY` is greater than a threshold (`THRESH_DIST_DELTA`).

If this condition is `True`, the object moves vertically. On **Line 38**, the variable `direction` is set to `up` if `deltaY` is negative (meaning it moved up) or `down` if `deltaY` is positive (meaning it moved down).

On **Lines 39 and 40**, the `counter` dictionary is updated to increment the count for the appropriate direction by `1`, and the `send()` function is called to send the updated counter data.

\# function to compute the centroid of a region of interest (roi)
def get\_centroid(roi):
    # calculate the centroid coordinates by averaging
    # the top-left and bottom-right coordinates
    x1 = roi.topLeft().x
    y1 = roi.topLeft().y
    x2 = roi.bottomRight().x
    y2 = roi.bottomRight().y
    return (x2 - x1) / 2 + x1, (y2 - y1) / 2 + y1

Then the function called `get_centroid` is defined, which calculates the centroid of a region of interest (roi) in a frame. The centroid is the point equidistant from all the shape boundaries. Basically, the arithmetic mean position of all the points in the shape.

The function takes a single parameter, `roi`, representing the region of interest. The `roi` parameter is expected to have methods `topLeft()` and `bottomRight()`, which return the coordinates of the top-left and bottom-right corners of the region, respectively.

**Lines 47-50** retrieve:

*   The `x` coordinate of the top-left corner of the region of interest and assigns it to variable `x1`.
*   The `y` coordinate of the top-left corner of the region of interest and assigns it to variable `y1`.
*   The `x` coordinate of the bottom-right corner of the region of interest and assigns it to variable `x2`.
*   The `y` coordinate of the bottom-right corner of the region of interest and assigns it to variable `y2`.

Finally, on **Line 51**, we compute the average of the `x`\-coordinates (`x1` and `x2`) and the `y`\-coordinates (`y1` and `y2`) to find the centroid and return the centroid coordinates as a tuple.

\# send initial counter data (all zeros)
send()

# main loop for processing tracklets
while True:
    # get the current set of tracklets
    tracklets = node.io\["tracklets"\].get()

    for t in tracklets.tracklets:
        # handle different tracking statuses
        if t.status == Tracklet.TrackingStatus.NEW:
            # if the tracklet is new, initialize a new dictionary
            # for it in the data dictionary
            data\[str(t.id)\] = {}
            # store the centroid of the tracklet's region of interest
            data\[str(t.id)\]\["coords"\] = get\_centroid(t.roi)
        elif t.status == Tracklet.TrackingStatus.TRACKED:
            # if the tracklet is currently being tracked, reset its "lost"
            # counter in the data dictionary
            data\[str(t.id)\]\["lostCnt"\] = 0

Now starts our main loop for processing the tracklets, but before that, we call the previously defined `send` function that sends the initial counter data (which should be all zeros at the beginning) as JSON data.

**Line 58** starts an infinite loop, typical for video or stream processing, as the code needs to keep processing the frames as they come in.

**Line 60** gets the current set of tracklets from the node’s output. A tracklet here refers to a set of observations of an object’s state being tracked. This line implies that there’s an ongoing tracking process, and it retrieves the tracking data.

**Line 62** starts iterating through each tracklet in the set of tracklets.

On **Line 64**, we check if the status of the tracklet is `NEW`. If a tracklet is new, it means that it has been newly identified in the frame and was not there in the previous frame(s):

*   Initialize a new dictionary for this tracklet in the `data` dictionary using the tracklet’s ID as the key (**Line 67**)
*   Compute the centroid of the tracklet’s region of interest using the `get_centroid` function defined earlier and store it in the data dictionary under the key `coords` (**Line 69**)

If the status of the tracklet is `TRACKED` (**Line 70**), this means that the tracklet is currently being actively tracked across frames:

*   Reset the `lost` counter for this tracklet in the data dictionary. The `lost` counter represents how often the tracklet has been lost during tracking. Resetting it to `0` indicates that the tracklet is currently being successfully tracked.

        elif t.status == Tracklet.TrackingStatus.LOST:
            # if the tracklet is lost, increment the "lostCnt" counter for the
            # tracklet to keep track of the number of frames it has been lost
            data\[str(t.id)\]\["lostCnt"\] += 1
            # if tracklet has been lost for more than 10 frames, remove it
            if 10 < data\[str(t.id)\]\["lostCnt"\] and "lost" not in data\[str(t.id)\]:
                # call the tracklet\_removed function to handle the removal
                # of the lost tracklet
                tracklet\_removed(data\[str(t.id)\], get\_centroid(t.roi))
                # mark the tracklet as lost by setting the "lost" flag to True
                data\[str(t.id)\]\["lost"\] = True
        elif (t.status == Tracklet.TrackingStatus.REMOVED) and "lost" not in data\[
            str(t.id)
        \]:
            # if the tracklet is removed and not marked as lost,
            # call the tracklet\_removed function to handle the removal of the tracklet
            tracklet\_removed(data\[str(t.id)\], get\_centroid(t.roi))

The above part of the code continues handling the different states of the tracklets:

Next, on **Line 74**, we check if the tracklet is `LOST`. This generally means that the object being tracked is not visible or not detectable in the current frame:

*   Increment the `lostCnt` counter for the tracklet to keep track of the number of frames it has lost (**Line 77**).
*   If the tracklet has been lost for more than 10 frames and not yet marked as lost, then the code considers this tracklet as lost (**Line 79**).
    *   The `tracklet_removed` function finds the object’s direction based on its initial and last coordinates (**Line 82**).
    *   Mark the tracklet as lost by setting the `"lost"` key in the data dictionary to `True` (**Line 84**).

**Lines 85-87** are the final condition when a tracklet is removed. A tracklet can be removed for various reasons (e.g., if the object has left the scene):

*   Similar to the previous block, the `tracklet_removed` function calculates the changes in the object’s `x` and `y` coordinates and updates the movement direction counters accordingly.

* * *

### [**People Counting: Python Driver Script**](#TOCh3Driver)

With the configurations and utilities implemented, we can finally get into the code walkthrough of people counting with OAK.

\# import the necessary packages
import logging
from pathlib import Path
from time import monotonic

import cv2
import depthai as dai

from pyimagesearch import config, utils

# set up logging configuration
# level: the root logger will delegate an event to all the handlers
# if the event’s level is greater than or equal to the handler’s level.
# format: handlers use this format for the emitted log message.
logging.basicConfig(
    level=logging.INFO, format="%(asctime)s - " "%(levelname)s - %(message)s"
)

We start by importing the necessary packages on **Lines 2-9**. We import `logging`, `cv2`, `depthai`, and custom modules `config` and `utils` from `pyimagesearch`.

On **Lines 15-17**, we set up the logging system, which will be used throughout the code to log messages for information.

\# assign the path of the video to be processed
videoPath = config.INPUT\_VIDEO\_LONG

# set the video codec to use with video writer. MJPG is a format that uses motion JPEG
fourcc = cv2.VideoWriter\_fourcc(\*"MJPG")
# create video writer object with parameters: output video path,
# video codec, frame rate of output video, and dimensions of video frame
out = cv2.VideoWriter(config.OUTPUT\_VIDEO\_LONG, fourcc, 20.0, config.CAMERA\_PREV\_DIM)
logging.info("video writer initialized.")  # logging the information

On **Line 20**, we assign the path of the input video where we want to run people counting.

The path is retrieved from the `config.INPUT_VIDEO_LONG` file.

Next, on **Lines 23 and 26**, we set the video codec to use with the video writer and initialize the video writer, which we will use to store the inference results in a video format. The `cv2.VideoWriter` accepts the path to store the inference video, video codec, output video frame rate, and video frame dimensions.

**Line 27** logs a message indicating that the video writer has been initialized.

\# create video pipeline with the specified yolo configuration and model
detectionNetwork, pipeline = utils.video\_detection\_pipeline(
    config.YOLOV8N\_CONFIG, config.YOLOV8N\_MODEL
)

# add object tracker to the pipeline
pipeline = utils.object\_tracker\_pipeline(pipeline, detectionNetwork)
logging.info("detection and tracking pipeline created.")  # logging the information

In the above code snippet, we set up the video detection pipeline defined in `utils.py` by passing in the YOLOv8 configuration and model files.

Subsequently, we add the object tracker to the pipeline by passing the `pipeline` object and `detectionNetwork` node.

\# pipeline defined, now the device is connected to. HIGH means we choose high speed USB
with dai.Device(pipeline, dai.UsbSpeed.HIGH) as device:
    logging.info("connected to the device.")  # logging the information

    # get the output and input queues of the device
    previewQ = device.getOutputQueue("preview")
    outQ = device.getOutputQueue("out")
    detectionNN = device.getOutputQueue("nn")
    videoQ = device.getInputQueue("video\_input")

    counters = None  # initialize counters
    frame = None  # initialize frame
    text = utils.TextHelper()  # create a TextHelper object

    # capture the video from the specified path
    cap = cv2.VideoCapture(str(Path(videoPath).resolve().absolute()))

The above code establishes a connection to a device with the defined pipeline and high-speed USB communication. Various queues are then set up for data handling, and video capture is initiated.

**Line 39** creates a context for connecting to the device using the previously defined pipeline and specifying high-speed USB communication through `dai.UsbSpeed.HIGH`. The `with` statement ensures the device is properly opened and closed, managing resources efficiently.

On **Lines 43-46**, several queues are defined:

*   `preview`: An output queue retrieved from the device and assigned to the variable `previewQ`. This queue will handle preview (raw) frames from the processing pipeline.
*   `out`: An output queue retrieved and assigned to the variable `outQ`. This queue holds the tracking outputs.
*   `nn`: Another output queue fetched and assigned to the variable `detectionNN`. It’s used for neural network detection outputs.
*   `video_input`: An input queue retrieved from the device and assigned to `videoQ`. This queue feeds video frames from the host to the OAK device.

On **Lines 48-50**, we initialize

*   `counters`: set to `None` and used later for counting objects.
*   `frame`: set it to `None`, which will store the video’s individual frames.
*   `text`: an instance of the `TextHelper` class (defined earlier in the code in the `utils` module) for adding text annotations to the frames.

We initialize video capture by creating a `cv2.VideoCapture` object using OpenCV on **Line 53**. The video file’s path is converted to an absolute path and passed as an argument. The `VideoCapture` object is assigned to the variable `cap`, which will read frames from the video.

while cap.isOpened():  # as long as the video is playing
        read\_correctly, video\_frame = cap.read()  # read a frame from the video
        if not read\_correctly:  # if frame not read correctly, break the loop
            break
        logging.info("frame read successfully.")  # logging the information

        img = dai.ImgFrame()  # create an ImgFrame object
        # reshapes the video frame to 640x640 and send the reshaped frame to the img object
        img.setData(utils.to\_planar(video\_frame, config.CAMERA\_PREV\_DIM))
        img.setType(dai.RawImgFrame.Type.BGR888p)  # set the type of img
        img.setTimestamp(monotonic())  # set the timestamp of img
        img.setWidth(config.CAMERA\_PREV\_DIM\[0\])  # set the width of img
        img.setHeight(config.CAMERA\_PREV\_DIM\[1\])  # set the height of img
        videoQ.send(img)  # send img to the video input queue
        logging.info("frame sent to video input queue.")  # logging the information

In this code block, we run a loop that reads frames from an opened video file, processes them, and sends them into a DepthAI pipeline for object detection and tracking.

On **Line 55**, a `while` loop runs as long as the video file is open (`cap.isOpened()`).

Then read a single frame from the video on **Line 56**. `read_correctly` is a boolean indicating whether the frame was read properly, and `video_frame` is the actual frame if it was read correctly.

If the frame is not read correctly, the loop is broken, and the processing ends (**Lines 57 and 58**). This also usually indicates the end of the video file.

On **Lines 61-68**,

*   We initiate an `ImgFrame` object to forward the frame to the OAK device.
*   The video frame (**Line 63**) is first transformed into a format suitable for neural network processing using the `to_planar` function from the `utils` module. This function resizes the frame to the desired dimensions (`640x640`) and rearranges its data into a planar format.
*   We then specify the image type as BGR (Blue Green Red) in a planar format using `setType`.
*   The current monotonic time is set as the timestamp for the frame with `setTimestamp`.
*   The dimensions of the image, both width and height, are explicitly set using `setWidth` and `setHeight`.
*   Finally, the prepared frame is sent to the OAK device via an input queue named `videoQ` for further processing.

On **Line 69**, we log information that the frame was sent to the video input queue.

        # update the frame and counters with the detected objects and texts
        frame, counters = utils.update(
            previewQ, outQ, detectionNN, text, frame, counters
        )
        # if the frame is successfully updated
        if frame is not None:
            out.write(frame)  # write the frame to the output video
            cv2.imshow("frame", frame)  # show the frame
            logging.info(
                "frame updated with detections and text."
            )  # logging the information

        # if the 'q' key is pressed, break from the loop
        if cv2.waitKey(1) == ord("q"):
            break

    logging.info("End of the video")  # logging the information that the video ended

out.release()  # release the VideoWriter object
logging.info("video writer released.")  # logging the information

The above code block continues the while loop that reads frames from a video file, and processes and displays them.

We have reached the concluding stage of annotating and displaying the results, having completed the majority of the intensive tasks such as:

*   Setting up both the detection and tracking pipelines
*   Defining the input and output queues
*   Reading video frames
*   Sending frames to the device
*   Defining the `update()` method in the `utils` module

On **Lines 72-74**, we call the `update` function (defined in the `utils` module), which updates the frame with object detections and text overlays like the person `up` and `down` count. The `counters` contain the statistics or counts of the person `up` and `down`.

Next, on **Lines 76-81**, we check if the frame is not empty (meaning that it has been successfully updated with detections and counter information)

*   Write the processed frame to the output video file using the `VideoWriter` object `out`.
*   Display the processed frame in a window titled `frame`.

Finally, on **Lines 84-90**, we break out of the loop (if the `q` key is pressed), log that the frame has been updated with detections and text, and release the `VideoWriter` resource.

* * *

#### [**Results**](#TOCh4Results)

A full video of the demo can be seen below:

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOCh2Summary)
----------------------------

In today’s tutorial, we took a deep dive into building a people counting system on the OAK device, using key Python packages: DepthAI and OpenCV.

We began by discussing the essential libraries required and proceeded to set up the prerequisites for the project. A notable feature we explored was the Script node within DepthAI, which is pivotal in executing custom tracking logic on the OAK device.

We then defined the utilities crucial for operating the people counting application, including the video detection and tracker pipeline, along with auxiliary helper functions. We also developed object tracker logic tailored to function within the Script node.

As a final step, we brought together all the elements by creating a Python driver script that harmonized the utilities and logic for the effective functioning of the application. To wrap up, we assessed the results of the people counting system by applying it to real video data, showcasing the practicality and efficiency of the solution developed on the OAK platform.

We hope you found immense value and insights in today’s tutorial on crafting a sophisticated people counting system using DepthAI and OpenCV on the OAK platform!

* * *

### [**Citation Information**](#TOCh3Citation)

**Sharma, A.** “People Counter on OAK,” _PyImageSearch_, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, and R. Raha, eds., 2023, [https://pyimg.co/pi5v4](https://pyimg.co/pi5v4)

@incollection{Sharma\_2023\_PeopleCounter,
  author = {Aditya Sharma},
  title = {People Counter on {OAK}},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha},
  year = {2023},
  url = {https://pyimg.co/pi5v4},
}

* * *

![Image 6: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!