tags
</Instructions Structure>

<Instructions>
Your task is to generate a set of instructions for building a demo similar to Google's Gemini demo, which can respond to real-time video and audio input using GPT-4, Whisper for speech recognition, and a text-to-speech model.

First, read through the following video transcript carefully:

<Video Transcript>

Title: Real Gemini demo? Rebuild with GPT4V + Whisper + TTS

Channel name: AI Jason

Video length: PT31M38S

View count: 16927

Like count: 808

Publish date: 2023-12-19
Captions:
which path is safer for the little bird to go pass one or pass two path one is safer for the little bird to go as it avoids the cat path two leads directly to the cat which could be dangerous for the bird what should be the next shape in this sequence the next shape in the sequence should be a hexagon which book is standard for me to read if I want to learn AI if you want to learn about AI . the coming Wave by Mustafa suan would be the more appropriate Choice it seems to be focused on the future of AI and its implications which would be relevant to your interests in artificial intelligence Google had their Infamous demo of Gemini where Gemini is responding to realtime video almost instantly this really blow everyone's mind but later it turned out that this demo is heavily edited it cut off all the latencies and behind the scenes are feeding a lot of different prompts into Gemini to get the popular response back and on the other hand even though Gemini . Altra is not available yet people start doing a lot of tests taking example that Google test with Gemini Ultra with GPT 4V in ter of the GPT 4B is able to do almost all the tasks that the Gina outra claim to be very good at from screenshot of a mass problem to understanding image of natural PL as well as object detection and gbt 4V was able to solve most of the example provided by Google except some more hardcore Mass questions so people's expectation to Google's Gemini model really changed overnight but . think we should still give a lot of credit to Google's Gemini demo because it really showcase a lot of interesting applications we have real native multimodal model they can understand all sorts of different data the big question I have back mind was can we actually rebuild this gini demo with GP for and what would be the results we can get today cuz theoretically the way it works is pretty straightforward with Gemini it basically feed both video and audio data to Gemini model and it will be able to speed back because it has that audio generation B . build him but we can totally build the same experience with GPD 4V plus whisper and text to speech model we will need to turn the video into different screenshots that we can pass on to GPD 4V and then we can also turn users voice into a text prompt with whisper transcri ability you can feed both transcript and screenshots to GPD 4V and utilize tactive speech model to let AI speak back but on the other hand there are also some challenges that I didn't know how to solve yet for example IDE one experience where the user . don't need to interact with any UI they literally just point to camera and talk to Ai and to achieve this kind of handsfree ux we need a way to actually detect when users stop talking so that we can send those requests to open AI so how can we actually detect users stop talking it is problem that I'm not too sure how to solve and on the other hand how can we stream the live video effectively we can take multiple different screenshots does it actually deliver the performance that we want to be able to answer user question instruction and most importantly how big the lat . y actually will be when we build this system with multiple different models working together luckily I found one project done by Julian deuca where he basically Implement Gemini demo with almost the same structure that I was thinking about and the result is quite stunning from his demo can you tell me in which direction I'm moving my hand you are moving your hand to the left . tell me what you see you are placing a white paper on a wooden surface with both hands and then removing them what about now you're drawing a circle in the center of the white paper . and now what do you think I'm drawing do you recognize it you're drawing a snowman with three circles two eyes a nose a mouth and buttons . can you tell me under which cup the bow is the ball is under the center cup can you tell me if I'm doing well with this game . yes you're doing well you're successfully matching the puzzle pieces to their correct positions on the board now can you tell me where I should put this one place the red star-shaped piece in the empty space on the board to the left of the yellow star-shaped piece . all right and now I have this blue one where should I put it place the blue circular piece in the empty space on the board to the right of the green circular piece suggest a game we can play together with these objects let's play a memory game I will describe a pattern and you will have to recreate it from memory . for example I might say place a red token next to a green token and then put a yellow token to the right of the green one you would then arrange the tokens accordingly if you're ready we can start with the first pattern yes I'm ready can you also count the points during the game please great here's your first pattern to recreate from memory place . a blue token above a green token put a red token to the left of the green token position a yellow token to the right of the green token arrange the tokens as described and once you're done I'll score your attempt if you get it right you'll earn a point let's start okay I'm done you've successfully recreated the pattern as . described you've earned a point would you like to try another pattern or do you have a different game in mind and in his example he figured out some simple but effective way to solve the problem that I had before for example for how to detect users stop talking he found a library that we can use right away CU silence a wear recorder it is open- Source library that can automatically detect if the user stop talking for a certain time and inal streaming in the live video he figured out a way to stitching together all the different screenshots into a grill which from . experience seem to communicate time sequence better to GPD 4V I think it also reduce latency because we are now sending 60 different image to gbt 4V everything I wrapped under a gr that can be sent and the latency does seem to be too bad it does have like three to four seconds latency but it's already better than I thought and he open source this demo called GPD video to Showcase how he build this demo I tried out and really works it is great example about how can we build this multimodal application so I'm going to give you a step-by-step breakdown of . of how you can build a similar Gemini demo by yourself with GPD 4V plus whisper plus text to speech and versal Ai and silence aware recorder so let's get it but before we get into the implementation details I want to give big thanks to the sponsor of this video hpot hpot is one of the leading CRM that invest heavily in AI this year they embed generative AI in almost every part of their platform to help you scale and grow your business and this one free tool called campaign assistant that I found very useful for . creating marketing assets at scale to get started I can click on this button it will ask me to choose the type of a. the user stop talking we want to generate image Gade to capture what has been happening for the past few seconds and also using whisper to automatically transcript the voice into a text promt so we can send both the text prompt and image to GPD 4V for you to get a response and first thing we're going to do is generate this image grade so we're need to move to top and Define a few different functions . so our first it create a transparent pixel which we will be using as divider to generate image grade and first function is get image Dimensions based on the image source and here you will see something called Promise so this is basically a way in JS to do things a sync and next we'll add a second function called base 64 to2 block so this function basically converts the image data into a format that we can manipulate and send to server it will be used in the next function we're going to Define called upload image to free image host so this basically . will upload the image screenshots we take into a free image hosting service called temp files so temp file is a free temporary file storage we will get Blob information from the image PL it to Temp file and get a URL back last but not least we will convert all those imag into image grade so it will get all the image and also the number of columns ways the image quality that we defined above here we get dimension of the image calculate the SP ratio and calculate number of . rows that we should have prepare the image for the mergin and in the end merge all the image together into one grid image those are all the function that we need to generate the image grade and then we need to trigger those functions so I move down to the UNS speech function which will be called when the user stopped talking and triggering that function so first they will set a state phase to uploading video captures which is defined by Max screenshots we defined above then generate image grade and upload the image grade to the free hosting service and set . State image grade URL to be the URL it return and in the end set the state to be processing completion and we will also need to add a new use f to save screenshots every few frames so we Define capture frame here if the video is actually recording then we'll proceed to GA the image ways data we defined above GA image data form the video recorder push that new image to screenshots reference and we will also set an interval which is how often this image should be captured and that's pretty much it .
</Video Transcript>

In the <scratchpad> section below, generate a step-by-step plan for how you would approach building this demo, utilizing the tools mentioned (GPT-4, Whisper, TTS). Consider the key components needed, potential challenges, and how you would solve them.

<scratchpad>
[Your step-by-step plan for building the Gemini demo goes here]
</scratchpad>

Now, based on your plan and the information provided in the video transcript, write detailed instructions for implementing this demo in the <Instructions> section below. Make sure to refer to relevant details from the transcript where applicable.

<Instructions>
[Your detailed instructions for building the Gemini demo go here]