Build a Realtime Object Detection Web App in 30 Minutes | by Erdem Isbilen | Towards Data Science

Tensorflow.js is an open-source library enabling us to define, train and run machine learning models in the browser, using Javascript. I will use the Tensorflow.js framework in Angular to build a Web App that detects multiple objects on a webcam video feed.
First, we have to select the pre-trained model which we are going to use for object detection. Tensorflow.js provides several pre-trained models for classification, pose estimation, speech recognition and object detection purposes. Check out all the Tensoflow.js pre-trained models for more information.
COCO-SSD model, which is a pre-trained object detection model that aims to localize and identify multiple objects in an image, is the one we will use for object detection.
The original ssd_mobilenet_v2_coco model size is 187.8 MB and can be downloaded from the TensorFlow model zoo. Compared to the original model, the Tensorflow.js version of the model is very lightweight and optimized for browser execution.
The default object detection model for Tensorflow.js COCO-SSD is ‘lite_mobilenet_v2’ which is very small in size, under 1MB, and fastest in inference speed. If you would like better classification accuracy you can use ‘mobilenet_v2’, in this case, the size of the model increases to 75 MB, which is not suitable for the web-browser experience.
‘model.detect’ takes image or video inputs directly from HTML, so you do not need to convert the inputs into tensors before using them. It returns an array of classes, probability scores as well as bounding box coordinates for the detected objects.
model.detect( img: tf.Tensor3D | ImageData | HTMLImageElement |   HTMLCanvasElement | HTMLVideoElement, maxDetectionSize: number)[{ bbox: [x, y, width, height], class: "person", score: 0.8380282521247864}, { bbox: [x, y, width, height], class: "kite", score: 0.74644153267145157}]
After we all clear about the model, it is time to use the Angular command-line interface to initialize the Angular web application.
npm install -g @angular/cling new TFJS-ObjectDetectioncd TFJS-ObjectDetection
Then I will use the NMP package manager to load Tensorflow.js and COCO-SSD library.
TFJS-ObjectDetection npm install @tensorflow/tfjs --saveTFJS-ObjectDetection npm install @tensorflow-models/coco-ssd --save
It is all set now. So we can start coding. I will start by importing the COCO-SSD model in ‘app.component.ts’.
import { Component, OnInit } from '@angular/core';//import COCO-SSD model as cocoSSDimport * as cocoSSD from '@tensorflow-models/coco-ssd';
Then, I will start webcam feed with the following code.
webcam_init(){  this.video = <HTMLVideoElement> document.getElementById("vid");  navigator.mediaDevices  .getUserMedia({  audio: false,  video: {facingMode: "user",}  })  .then(stream => {  this.video.srcObject = stream;  this.video.onloadedmetadata = () => {  this.video.play();};  });}
We need another function to load the COCO-SSD model which also calls ‘detectFrame’ function to make the prediction using images from the webcam feed.
public async predictWithCocoModel(){  const model = await cocoSSD.load('lite_mobilenet_v2');  this.detectFrame(this.video,model);  }
‘detectFrame’ function uses requestAnimationFrame to loop the prediction over and over by making sure that the video feed is as much smooth as possible.
detectFrame = (video, model) => {  model.detect(video).then(predictions => {  this.renderPredictions(predictions);  requestAnimationFrame(() => {  this.detectFrame(video, model);});  });}
Meantime, the ‘renderPredictions’ function draws the bounding boxes and class names into the screen for detected objects.
renderPredictions = predictions => {const canvas = <HTMLCanvasElement> document.getElementById     ("canvas");    const ctx = canvas.getContext("2d");    canvas.width  = 300;  canvas.height = 300;  ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);      // Fonts  const font = "16px sans-serif";  ctx.font = font;  ctx.textBaseline = "top";  ctx.drawImage(this.video,0,0,300,300);    predictions.forEach(prediction => {        // Bounding boxes's coordinates and sizes   const x = prediction.bbox[0];   const y = prediction.bbox[1];   const width = prediction.bbox[2];   const height = prediction.bbox[3];// Bounding box style   ctx.strokeStyle = "#00FFFF";   ctx.lineWidth = 2;// Draw the bounding   ctx.strokeRect(x, y, width, height);        // Label background   ctx.fillStyle = "#00FFFF";   const textWidth = ctx.measureText(prediction.class).width;   const textHeight = parseInt(font, 10); // base 10   ctx.fillRect(x, y, textWidth + 4, textHeight + 4);   });      predictions.forEach(prediction => {    // Write prediction class names    const x = prediction.bbox[0];    const y = prediction.bbox[1];      ctx.fillStyle = "#000000";    ctx.fillText(prediction.class, x, y);});      };
All required functions are ready now to perform the object detection on the browser.
We just need to call ‘webcam_init’ and ‘predictWithCocoModel’ on ‘ngOnInit’ to initialize the app on the start.
ngOnInit(){  this.webcam_init();  this.predictWithCocoModel();}
One last step remaining is to modify ‘app.component.html’ to include <video> and <canvas> HTML elements which are required for the above functions to work.
<div style="text-align:center">  <h1>Tensorflow.js Real Time Object Detection</h1>  <video hidden id="vid" width="300" height="300"></video>  <canvas id="canvas"></canvas></div>
Visit my GitHub repository for the full code of this project.
Visit the live demo application to see the codes in action. The application runs in the Google Chrome browser without any issue. If you use any other browser, make sure the browser you use supports ‘requestAnimationFrame’.