How to turn your webcam into a Motion Detect controller | by Benson Ruan | Towards Data Science

Have you ever played Xbox games with a motion sensor? It feel so cool to just swing your arms to hit the tennis ball in the video game. Motion detection is the technique that make this magic happened, it can detect a change in the position of an object relative to its surroundings.
In this article, let’s have some fun. I am going to build a Virtual Drums Playground which will let you play the drums by just waving your hands in the air. And it don’t need any motion sensor, all it need is to turn on the webcam on your PC or Mobile.
Here it is the demo of the Virtual Drums Playground. As you can see, the drum and cymbal is played when it detects significant movement in that area.
Try it yourself in the link below :
bensonruan.com
Did you have fun and kept playing the drums like a rock star? Are you curious about how this magic happened? I was first inspired by the xylophone version of this, and then built my drum version above. Follow me below for the journey of implementing this Virtual Drum Playground.
To stream your webcam into the browser, I utilize the JavaScript library navigator.mediaDevices.getUserMedia. To find out more details about that, please refer to my previous blog :
medium.com
Overlay the webcam with the images of drums and cymbals. This is to set the target areas in the webcam, so that whenever in those areas detected a significant movement, it triggers the effect of playing the gear.
HTML
JavaScript
Store the location of the gears by their (x, y, width, height), please note that the width and height need to be calculated by the ratio between the full screen camera size and the blended canvas size.
For the purpose of playing the drums and cymbals, we need to load the audio sound track files for those music instruments.
Here is the most important part that doing the motion detection. It is using a technique call Blend Mode Difference.
First, let me explain the concept of Blend Mode Difference. Imagine you take one picture, then another one after half second, if there is nothing moved in that half second, those two pictures would be exactly the same. When you subtract the pixel values of those 2 pictures, it would be all 0, the blended images would be completely black. If there is movement in certain areas, the blended image would high light the differences.
For more details, please refer to the article below:
www.adobe.com
We are repeatedly calling below functions for each frame of the webcam video for motion detection.
function update() {  drawVideo();  blend();  checkAreas();  window.requestAnimationFrame(update);}
Within the drums and cymbals’ areas, if it detects a significant difference between frames, it indicate that there is movement in that area. Then we play the effect and sound of the corresponding gear.
while (i < (blendedData.data.length * 0.25)) {  average += (blendedData.data[i*4] + blendedData.data[i*4+1] + blendedData.data[i*4+2]) / 3;  ++i;}average = Math.round(average / (blendedData.data.length * 0.25));if (average > 20) { //over a threshold, means significant movement  playHover(drum);          }
That’s pretty much for the code. Let’s enjoy and play some crazy drums!
You can download the complete code of the above demo in the link below:
github.com
Nowadays, motion detection had been applied in many areas, e.g. Security alarms, Automatic gates... Some works with sensors, others might be based on computer vision object detection. After all, I hope you find this Virtual Drum Playground funny, and piqued your interest in the latest technology.
Thank you for reading. If you like this article, feel free to share on LinkedIn or Twitter. Let me know in the comment if you have any questions. Follow me on GitHub and Linkedin.