Crop and Identify Specific Objects in an Image or Video in the Browser using Machine Learning | by Neeraj Krishna | Towards Data Science

Tensorflow.js allows us to train models and run inference entirely client-side in the browser. This unlocks a whole bunch of new opportunities!
In this article, we will build a tool which allows the user to crop a specific part of an Image, and run inference on the cropped Image. We will use an Image Classification model which helps us to classify the Image into a specific category, but in theory you can use any model.
Live demo of the application shown in the above gif. Instructions to use are mentioned in the Note below.
Github Repository
To use the live tool click the Live demo link, this opens up the application. Next upload any video clip of your choice. The Screen will still remain blank at this time. To play the video click on the play button, the video will start playing. Next, click pause button on the interested instance of the video. This will initialize cropper on the frame. Select the interested area to crop and click the crop button. This pops up a modal showing the cropped image. Click the make prediction button to run inference on the cropped Image. The results will be displayed in the Modal.The tool shown in the demo is an expanded version of the tool we will be building in this article. You can check the github repo for the source code.
To use the live tool click the Live demo link, this opens up the application. Next upload any video clip of your choice. The Screen will still remain blank at this time. To play the video click on the play button, the video will start playing. Next, click pause button on the interested instance of the video. This will initialize cropper on the frame. Select the interested area to crop and click the crop button. This pops up a modal showing the cropped image. Click the make prediction button to run inference on the cropped Image. The results will be displayed in the Modal.
The tool shown in the demo is an expanded version of the tool we will be building in this article. You can check the github repo for the source code.
We will create a simple directory structure shown below.
|-- index.html|-- index.js|-- style.css|-- |-- Images|   `-- image.jpg
In the head tag, we import the css stylesheet and certain libraries using cdn to help us out. The Libraries we’re going to use are:
Cropper.js: A Javascript Image Cropping Library.
Tensorflow.js: A Library for Machine Learning in JavaScript.
In the body we have three containers. The first container is for the image we will be testing on. The Second Container is where we will have our buttons, the first button is to initialize the cropping tool on the image and the second button is to crop the selected portion and run inference on the cropped portion of the image. The third container is empty, but we will populate it with the results dynamically using JavaScript once we run the inference. At the end we import cropper.js library and our index.js script.
Next we add some css to restrict the image within a boundary and to make everything look good.
After the html and css is done, the page will look something like this:
Now this is the most important part. Let’s understand what the code does.
First, we grab all the elements so that we can manipulate them. This includes the image, the buttons etc.
Next, we run the loadModel function. This function downloads the MobileNet Pretrained Model which has been trained on a large dataset with thousands of categories. MobileNet refers to a specific architecture of the neural network.
We add event listeners to the two buttons so they can respond on click.
When we click the first button — “Initialize Cropper”, we run the function initializeCropper. This function creates a new Instance of Cropper on our image, and also empties our result container.
The second button — “Crop Image and Make Prediction” runs the predict function. This function takes the cropped image data which we get from cropper and feeds it to our MobileNet model. The model then outputs the class or category the cropped image belongs to and the confidence probability. We then create an img element and a h1 element dynamically and append them to our result container to display the results.
The flow is:
Click on ‘Initialize Cropper’ buttonClick on ‘Crop Image and Make Prediction’Results will be displayed
Click on ‘Initialize Cropper’ button
Click on ‘Crop Image and Make Prediction’
Results will be displayed
In the above example, I’ve cropped the right polar bear and ran inference on the cropped image. As we can see, the model has correctly predicted it’s class with pretty high confidence.
In this article we’ve built a simple tool that allows the user to crop a specific part of an Image and run an image classification model on the cropped image.
Hope you’ve enjoyed it! Even though this is a basic example, it can be expanded to build awesome things. One example is to use this on videos as I’ve shown in the demo (github).
Client-Side Machine Learning has opened up a lot of possibilites and I encourage you to go and build awesome stuff.
“If you don’t make stuff, There is no stuff” — Elon Musk
If you have any feedback or want to get in touch in general, please drop me a note on ms.neerajkrishna@gmail.com