Capture, Processing, And Assembly Of Immersive Experience

A headgear for capturing an immersive experience includes two cameras and two microphone pairs, each with a low volume microphone and a high volume microphone. A global positioning system (GPS) tracking device, accelerometer, and gyroscope are also provided. The data recorded from the cameras, microphones, GPS tracking device, accelerometer, and gyroscope are used to record several feeds and assemble said feeds into a single multimedia clip. Multimedia clips can be combined by time and/or region, forming an aggregate clip for a district or city. The component multimedia clips are linked to each other, allowing a user to experience the aggregate multimedia clip at their direction. Scenarios and advertisements can be embedded into the aggregate multimedia clip, allowing the immersive experience to be used for education, policy analysis, entertainment, and other similar tasks. Information from the GPS tracking device, accelerometer, and gyroscope are used to synchronize audio-video feeds and insert scenarios.

DETAIL DESCRIPTIONS OF THE INVENTION

The present invention is an immersive capture system that enables the recording, processing, and assembly of an immersive multimedia experience. The immersive capture system comprises a headgear1, a video system2, a binaural audio system3, a portable power source4, a global positioning system (GPS) tracking device4, a chipset6, an accelerometer7, a gyroscope8, and a compass9. The portable power source4provides the energy to run the video system2, the binaural audio system3, the portable power source4, the GPS tracking device4, and the chipset6. The video system2is mounted to the front of the headset, while the binaural audio system3is mounted on each side of the headset. The GPS tracking device4and the chipset6are housed within the structure of the headgear1.

The headgear1, as illustrated inFIG. 1-FIG.3, comprises a front section11, a rear section12, a camera mount14, a chinstrap13, and a rear shelf15. The front section11and rear section12form opposite parts of the headgear1, with the camera mount14being adjacent to the front section11. The chinstrap13is connected across the headgear1, to a left side and a right side. The camera mount14itself comprises a vertical track141, a cart142, a horizontal track143, a first housing socket144, and a second housing socket145while the chinstrap13comprises a buckle131. The vertical track141is connected to the front section11of the headgear1, securing the camera mount14to the headgear1. The rear shelf15is connected to the rear section12of the headgear1, opposite the camera mount14.

Engaged with the vertical track141is the cart142, which is capable of sliding up and down along the vertical track141. The interaction of the cart142and the vertical track141allow the video system2to be adjusted in the vertical direction. Centrally connected to the cart142, on the opposite side of the vertical track141, is the horizontal track143. The horizontal track143is oriented to be perpendicular to the vertical track141, and serves as a mounting point for the first housing socket144and the second housing socket145. The first housing socket144and the second housing socket145are slidably engaged with the horizontal track143, enabling lateral motion along the horizontal track143. The buckle131of the chinstrap13is centrally positioned; by engaging or disengaging the buckle131the headgear1may be secured to or released from a user's chin. The components of the headgear1serve to secure the headgear1to a user's chin as well as provide vertical and horizontal motion for a connected video system2. These arrangements are visible inFIG. 1andFIG. 2.

The rear shelf15comprises a wall151, a floor152, and a plurality of inputs153. The wall151is connected to the rear section12, securing the rear shelf15to the rear section12. Connected perpendicular to the wall151is the floor152, forming a support for accessory devices which may be connected to the headgear1. These accessory devices, which can enable a number of functions such as additional audio capabilities, removable storage, and auxiliary batteries, interface with the headgear1through the plurality of inputs153. For example, digital audio recording could be secured and connected to the headgear1by means of the rear shelf15. The plurality of inputs153are positioned on the wall151, providing a proximal connection point for any accessories that are secured to the floor152. The rear shelf15is visible inFIG. 3.

The video system2comprises a first camera21, a second camera22, a data storage device23, and a wireless transmitter24. The first camera21and the second camera22are secured to the camera mount14while the data storage device23and the wireless transmitter24are housed within the headgear1. The first camera21is placed in the first housing socket144while the second camera22is placed in the second housing socket145. The first camera21and second camera22are provided with two degrees of freedom in the rotational element; that is, they are capable of tilting and pivoting in relation to the first housing socket144and the second housing socket145. The tilt capability allows for looking up and down, while the pivot capability is allows for convergence (e.g. refocusing as an object moves closer). Potentially, a swivel capability can be added, enabling looking left and right. With the swivel function added, the present invention would then have 3 degrees of freedom in the rotational element. In combination with the first vertical track141and the second vertical track141, the first camera21and the second camera22are provided with two translational degrees of freedom and two rotational degrees of freedom.

To power the present invention, the portable power source4is installed in the headgear1. The portable power source4is connected to multiple components of the present invention, supplying power to the video system2, the binaural audio system3, the GPS tracking device4, the chipset6, the accelerometer7, the gyroscope8, the compass9, and the plurality of inputs153. To enable communication and data transfer between the systems of the headgear1, the chipset6is included as a central hub. The chipset6is electronically coupled to the first camera21, the second camera22, the data storage device23, the wireless transmitter24, the high volume microphone34, the low volume microphone33, the GPS tracking device4, the accelerometer7, the gyroscope8, and the compass9. The chipset6enables the various components of the headgear1to communicate with each other, e.g. saving video recorded by the first camera21and the second camera22to the data storage device23.

While the cameras and microphones allow the headgear1to record video and audio, the data storage device23, the wireless transmitter24, GPS tracking device4, accelerometer7, gyroscope8, and compass9provide additional capabilities that are especially helpful when processing the recorded video and audio. The GPS tracking device4continuously records position coordinates including altitude during filming, allowing the recorded video to be matched to a geographic location. The accelerometer7measures the pitch of the headgear1, allowing rapid changes in tilt to be accommodated for by the first camera21and the second camera22. Similarly, the gyroscope8tracks orientation of the headgear1, enabling camera shake and similar undesired effects to be smoothed out during video processing. The compass9tracks the directional heading of the headgear1, which can be added to the recorded video. The data storage device23provides local storage, allowing the headgear1to add the recorded media to the data storage device23. The wireless transmitter24allows for remote viewing of the live feeds generated from the video system2, audio system, and accessory devices like the GPS tracking device4, accelerometer7, gyroscope8, and compass9. For example, a producer in a studio room could view the real time video and audio being captured by the present invention, along with the positional, motion, orientation, and directional data being recorded by the accessory devices. Potentially, the live-streamed data could be saved to a remote location, providing a producer or other viewer with a local copy separate from that saved to the data storage device23. These connections between the chipset6, portable power source4, and other components of the present invention are shown inFIG. 4. The headgear1of the present invention is designed to allow for the capture of an immersive experience, as well as the subsequent processing and assembly of the immersive experience. Creating the immersive experience requires two core processes, with the material for said processes being generated by the headgear1of the present invention.

The first process is the capture and processing of the immersive experience. The ability to capture the experience is provided by the headgear1. The first camera21and the second camera22are used to generate a first video feed and a second video feed. The binaural audio system3generates four separate feeds; a first high volume audio feed from a first high volume microphone34, a first low volume audio feed from a first low volume microphone33, a second high volume audio feed from a second high volume microphone34, and a second low volume audio feed from a second low volume microphone33. In addition to generating these video and audio feeds, a timestamp is embedded throughout the entirety of each feed; a first video timestamp for the first video feed, a second video timestamp for the second video feed, a first high volume timestamp for the first high volume audio feed, a first low volume timestamp for the first low volume audio feed, a second high volume timestamp for the second high volume audio feed, and a second low volume timestamp for the second low volume audio feed. These generated feeds and timestamps are saved to the local data storage device23, allowing them to be accessed for later steps in the editing process.

In addition to the audio-visual feeds, data streams are provided by the GPS tracking device4, the accelerometer7, and the gyroscope8. More specifically, the GPS tracking device4generates a location data stream, the accelerometer7generates a motion data stream, and the gyroscope8generates an orientation data stream. A directional facing data stream is also generated by the compass9. Along with the audio-visual feeds, these data streams are saved to the data storage device23. The audio-video feeds and the data streams provided the starting material for the immersive experience, which must next be processed and finally assembled into the desired finished product. The motion data stream from the accelerometer7can be used to make real time adjustments to the first camera21and the second camera22during filming; movements such as tilting of the headgear1can be accommodated for with corresponding changes to the tilt of the first camera21and the second camera22.

Once the raw data has been recorded by the headgear1, the processing stage can begin. To do this, the audio-visual feeds first need to be combined into tracks. The first video feed and the second video feed are merged to create a stereoscopic three-dimensional track, the first high volume audio feed and first low volume audio feed are merged to create a first audio track, and the second high volume feed, and second low volume audio feed are merged to create a second audio track. To ensure that the feeds of the combined components are synchronized, each feed is matched to its partner using the embedded timestamp. For example, the first video timestamp and the second video timestamp are referenced when creating the stereoscopic three-dimensional video track. The result is that the first video feed and second video feed are completely synchronized with each other, starting and stopping at the same time to provide a smooth stereoscopic three-dimensional video track. In the same way, the first high volume timestamp and first low volume timestamp are used to synchronize the first high volume audio feed and first low volume audio feed, thus generating the first audio track. Likewise, the second audio track is generated by using the second high volume timestamp and second low volume timestamp to synchronize the second high volume audio feed to the second low volume audio feed.

After synchronizing the generated feeds to create the stereoscopic three-dimensional video track, first audio track, and second audio track, the three tracks are saved together as a multimedia clip, added to the local data storage device23. The same process that is used to create the tracks is used to create the multimedia clip. By matching the timestamps of the component tracks, the multimedia clip is assembled such that the stereoscopic three-dimensional video track, first audio track, and second audio track are in sync with each other. Since synchronizing by timestamps is not perfect, it is necessary to fine-tune the synchronization. To fine tune the synchronization of the first audio track and the second audio track to the stereoscopic three-dimensional video track, waveforms generated by the video system2and audio system are used. As the first camera21and second camera22generate the first video feed and the second video feed, a first video waveform and second video waveform are also produced. Similarly, a first high volume waveform, first low volume waveform, second high volume waveform, and second low volume waveform are produced along with the first high volume audio feed, first low volume audio feed, second high volume audio feed, and second low volume audio feed, respectively. After synchronizing the timestamps of the audio-video feeds, the waveforms of each feed can be matched to each other to produce an exact match.

To provide various information about the multimedia clip, a metadata tag is inserted into the multimedia clip. The metadata tag is formed from the location data stream, the orientation data stream, and the motion data stream. As a result, information regarding the coordinates, camera orientation, and camera motions is readily available; this is especially important in the processing of the multimedia clip, where such information is used to make adjustments and corrections to imperfections or errors in the multimedia clip.

With the multimedia clip having been fully prepared, processing of the multimedia clip can proceed. Processing of the multimedia clip involves numerous adjustments, corrections, and accommodations to the multimedia clip. By referencing the orientation data stream and the motion data stream (accessed through the metadata tag), the stereoscopic three-dimension video track can be stabilized, eliminating a plurality of jitters that would otherwise detract from the viewing experience. The multimedia clip is reviewed for other unwanted events, which are removed or adjusted through editing. Processing can also be used to add elements to the multimedia clip, or alter existing element for artistic purposes. Processing the multimedia clip may involve a number of steps, such as rotoscoping, color correction, adjusting convergence, focus, fisheye, and zoom, as well as depth mapping.

Once the multimedia clip has been captured and processed, the assembly of the immersive experience can begin. Creating the immersive experience requires a plurality of immersive capture systems, such as the headgear1, generating a plurality of multimedia clips. The first step in assembling the immersive experience is grouping the plurality of multimedia clips into geographic group of multimedia clips, wherein each geographic group of multimedia clip is formed by a minimum of two multimedia clips. The creation of the geographic group of multimedia clips is done by comparing the metadata tag of each of the plurality of multimedia clips. By looking at coordinates in the location data stream from each metadata tag, multimedia clips which are from proximal locations (as indicated by their respective location data streams) can be combined in the geographic group of multimedia clips. Alternatively, instead of having the present invention automatically link together multimedia clips in a geographic region, a producer can create a playlist of multimedia clips. For example, in order to create a fine dining tour of Paris, a producer might create a playlist of video clips that explore a number of restaurants in Paris, even if the sequential videos are not immediately geographically adjacent. Creating a playlist is not limited to a producer; a user or anyone with access to a plurality of multimedia clips can selectively arrange, remove, and otherwise edit them to create a playlist.

Potentially, the geographic group of multimedia clips can be grouped by time as well as or instead of location. For example, by accessing the timestamps from the metadata tags, time variants of geographic groups of multimedia clips can be formed. Multimedia clips generated in one year, e.g. 2000, would be arranged into one geographic group of multimedia clips for a region. Multimedia clips for that same region, but with different capture dates such as 2010, are then grouped into a separate geographic group of multimedia clips for the region. Alternatively, a group of multimedia clips can be combined to form highlights of a 2010 multimedia experience, for example. By providing multimedia clips that are sorted not only by location, but also time, the present invention allows a user to experience different eras as well as different locations.

To improve the immersive experience, an accessory data stream can be incorporated as part of the metadata tag. This accessory data stream serves as a connection point for accessory devices. These accessory devices are added in order to provide accessory feedback to the multimedia clip. While the audio-video feeds address the senses of sight and sound, the accessory devices can target the other senses. For example, by including a vibrating device the sensation of touch can be incorporated into the immersive experience. As an example, a vibrating chair can be fed the information from the accessory data stream, shaking to add a tactile element to the accessory data stream. More specifically, if an earthquake occurs as part of the immersive experience, the vibrating chair will shake as the earthquake unfolds. Resultantly, a user will not only see and hear the effects of the earthquake, but feel them as well.

Subsequently, the individual clips from the geographic group of multimedia clips are linked to each other. This is done by comparing the location data stream of each of the component multimedia clips. When a reference multimedia clip and an adjacent multimedia clip have location data streams that show shared position coordinates, the reference multimedia clip and the adjacent multimedia clip can be linked together at or near those shared position coordinates. This linking of the reference multimedia clip and the adjacent multimedia clip effectively acts as an aggregate multimedia clip. For example, if the reference multimedia clip ends at an intersection and the adjacent multimedia clip begins at that same intersection, the two multimedia clips are linked. Since two adjacent multimedia clips will usually not have an exact match in shared position coordinates, a tolerance level is set to determine how close to an exact match the shared position coordinates need to be in order to link the two multimedia clips. This creates the aggregate multimedia clip, which switches from playing the reference multimedia clip to the adjacent multimedia clip at the junction, essentially acting as a single larger multimedia clip. These adjacent multimedia clips act as linked multimedia clips with regards to the reference multimedia clip. Ideally, the transition between the reference multimedia clip and adjacent multimedia clip is seamless, occurring exactly at the shared position coordinates. Potentially, a reference multimedia clip can be linked to multiple adjacent multimedia clips. For example, if the reference multimedia clip ends at a four-way intersection there are three potential adjacent multimedia clips to link to. These adjacent multimedia clips correspond to continuing straight, turning left, or turning right at the intersection. Provided enough initial multimedia clips, large areas such as districts and cities can be represented through the geographical group of multimedia clips. The plurality of multimedia clips from the geographical group is saved together to a central storage location as an aggregate multimedia clip Likewise, the metadata tags of the plurality of multimedia clips are combined to provide an aggregate metadata tag for the aggregate multimedia clip. This aggregate multimedia clip is named to indicate the captured geographical region. By looking through the metadata tags and location data streams of the component multimedia clips, the position coordinates can be converted to the appropriate geographic location, which is then used as the name for the aggregate multimedia clip. For example, given the coordinates of 48.87, 2.34 indicates that the aggregate metadata clip should be named Paris.

Beyond simply linking adjacent multimedia clips together to form the aggregate metadata clip, the present invention provides an element of user interaction. Defining a junction of shared position coordinates where a reference multimedia clip is linked to an adjacent multimedia clip, the present invention allows a user to make a choice at each junction. When a user watching the aggregate multimedia clips reaches a junction, they are prompted to choose a linked multimedia clip to continue to. The user interaction is enabled by an integrated user interface, which can be implemented in various designs depending on the playback machine of choice. Returning to the earlier example, at a four way intersection a user may choose between continuing straight, turning left, or turning right, with each choice linking the user to the appropriate linked multimedia clip. Provided a large enough database, this allows a user to take an immersive tour of a city, even entering individual stores or buildings when the relevant multimedia clips are available.

Expanding upon the user interaction, scenarios can be presented throughout the aggregate multimedia clip by utilizing a plurality of audio-visual cues. There are two variations; position-based audio-visual cues and temporal-based audio-visual cues. The position-based cues are tied to predetermined physical positions, a set of coordinates from the location data stream of the aggregate metadata tag. Parallel to the position-based cues, the temporal-based cues are tied to a specific timestamp from the aggregate metadata tag. This enables a specific position-based cue or a specific temporal-based cue to be triggered at a specific location or time in the aggregate multimedia clip. For example, a specific positioned-based cue can be triggered at a crosswalk while a specific temporal-based cue can be triggered two minutes into the aggregate multimedia clip. Once these audio-visual cues are triggered, a user is prompted for a response, which is saved to the central data storage. This allows a user's response to various scenarios and challenges to be presented in an immersive environment and saved for later analysis.

Further utilization of the aggregate metadata tag includes inserting a plurality of location-based advertisements throughout the aggregate multimedia clip. This is accomplished by providing a plurality of different advertisement coordinates, themselves selected from the coordinates provided in the location data stream. Specific location-based advertisements are then linked to specific advertisement coordinates. These location-based advertisements are triggered when the coordinates from the location data stream match the specific advertisement coordinates. This allows relevant advertising to be inserted into the aggregate multimedia clip. For example, a user may be previewing a trip to Paris by viewing an aggregate multimedia clip for the Paris region. During the aggregate multimedia clip, a local bakery is passed, triggering an advertisement for that specific bakery. This could influence the user to go to said bakery during a physical visit to Paris. This is just one example of many potential advertisements.

The immersive experience provided by the present invention can be used in a number of applications. For example, the aggregate multimedia clip can be used to provide an immersive experience from another person's point of view; a user could experience great cities, famous landscapes, and different cultures through the aggregate multimedia clip. Alternatively, in combination with the audio-visual cues, the aggregate multimedia clip can be used for more interactive tasks such as policy analysis or education. For example, by creating a specific plurality of audio-visual cues, a police department can see if their officer's responses are in line with official police policy. Or, in the education sector, the audio-visual cues can guide a student through the proper procedure in an experiment, testing their ability to follow guidelines without using up valuable physical lab space and materials. The educational application of the present invention is especially relevant to blended learning, where an immersive experience can be used to replicate real-time, hands-on, or practical experience, which are often requirements for completing a class. Traditionally, these requirements require an investment in time and money, either on the part of the instructor or the students (e.g. travelling to a lab for hands on experience). The immersive experience provided by the present invention is capable of imitating these practical experiences, thus improving the blended learning approach. These are just a few examples of applications that utilize the present invention; additional ways of integrating the aggregate multimedia clip and audio-visual cues can be implemented in sectors such as entertainment, healthcare, education, and business.

The immersive experience is designed to provide a stereoscopic three-dimensional experience; however, users may choose to use traditional two-dimensional displays as a more cost effective way of experiencing the aggregate multimedia clip. In the ideal situation, a user will be able to experience the aggregate multimedia clip through a closed environment, such as three-dimensional goggles with integrated headphones.