Revolutionizing Music Interaction: HTML & JavaScript Finger-Detection for Chord Generation
Channel: BioniChaos
Date: 2024-02-05

In this video, I dive into an innovative project where I use HTML and JavaScript to create a unique musical experience. The focus is on developing a feature that utilizes the camera to detect the number of fingers shown on screen. This detection then triggers the playing of multiple notes simultaneously, essentially creating chords. I discuss using MediaPipe for finger counting and hand landmark detection, which is pivotal for this feature. I also delve into the challenges of integrating this with Tone.js for sound generation, maintaining a clean and intuitive user interface, and tweaking the JavaScript code to enhance the user experience. Additionally, I explore the potential of GitHub Copilot in coding and troubleshoot various issues such as volume control and coordinate display. Join me as I navigate these technical intricacies and share insights into making music interaction more dynamic and intuitive.

The tools I develop are available on https://bionichaos.com 
You can support my work on https://patreon.com/bionichaos

00:00 - Introduction to the Project: HTML & JavaScript for Music Interaction
00:08 - Concept of Finger Detection for Playing Chords
00:29 - Exploring Chord Structures and Note Matching
00:36 - MediaPipe Integration for Finger Counting and Tracking
01:08 - Mapping Extended Fingers to Corresponding Notes
01:44 - User Interface Design: Keeping the Screen Clean
02:01 - Coding Approach: Direct Coding vs GitHub Copilot
02:12 - Detailed JavaScript Code Walkthrough
02:23 - Limiting Scope to Face and Hands Detection
02:39 - Discussing GitHub Copilot's Role and Code Generation
03:01 - Debugging and Code Integration Challenges
03:31 - Feature Enhancement: Displaying Coordinates Without Camera Footage
03:59 - Issues with Second Camera and Landmark Display
04:17 - Code Review: Functionality and Structure
04:44 - User Feedback: Volume Control and Display Improvements
05:02 - Enhancing Face Coordinate Display
05:13 - Managing Sound Volume in JavaScript
05:53 - Testing and Debugging Live Code
06:45 - Resolving Issues with Tone Synth Initialization
07:06 - Adding Dynamic Loudness Control Based on Finger Position
07:35 - Browser Console Error Troubleshooting
07:56 - Clarifying Tone Poly Synth Initialization
08:35 - Comparing Two Versions of the Tool
09:05 - Volume Control and Note Length Variation
09:21 - Evaluating the Improved Tool vs. Initial Version
09:41 - Future Developments: Chord Formation Based on Finger Extension
09:53 - Conclusion and Musical Outro

#MusicInteraction
#JavaScriptMusic
#HTML5Audio
#MediaPipeHandTracking
#ToneJs
#FingerDetection
#ChordGeneration
#UserInterfaceDesign
#GitHubCopilotCoding
#BiomedicalEngineering
#InteractiveMusicApp
#WebDevelopmentMusic
#CreativeCoding
#DigitalMusicCreation