Patent ID: 12190916

DETAILED DESCRIPTION

Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the invention is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.

For example, the techniques of the present invention will be described in the context of particular audio segments and components. However, it should be noted that the techniques of the present invention can apply to one or more of any variety of different audio segments and components. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. Particular example embodiments of the present invention may be implemented without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.

Various techniques and mechanisms of the present invention will sometimes be described in singular form for clarity. However, it should be noted that some embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. For example, a system uses a processor in a variety of contexts. However, it will be appreciated that a system can use multiple processors while remaining within the scope of the present invention unless otherwise noted. Furthermore, the techniques and mechanisms of the present invention will sometimes describe a connection between two entities. It should be noted that a connection between two entities does not necessarily mean a direct, unimpeded connection, as a variety of other entities may reside between the two entities. For example, a processor may be connected to memory, but it will be appreciated that a variety of bridges and controllers may reside between the processor and memory. Consequently, a connection does not necessarily mean a direct, unimpeded connection unless otherwise noted.

Various three-dimensional (3D) media formats have developed with advances in technology, such as multi-view interactive media representations. These multi-view interactive digital media representations include formats such as surround views, multiview images, and 3D data formats. In these multi-view interactive digital media representations, a user can control how to view the image data. For instance, the user can navigate around various objects and select a viewpoint from which to view the image data.

Because users can navigate around various objects within multi-view interactive digital media representation, one problem is how to include audio information in this viewing process. Although image data and audio information may be recorded simultaneously, a user may choose to view the images in a different order than they were acquired during the recording process. Because a user may navigate through the images in the captured multi-view interactive digital media representation in any order, the displayed visual representation of the scene may not be synchronized with playback of the recorded audio. Various embodiments described herein relate to improved mechanisms and processes for integrating audio into a multi-view interactive digital media representation.

As described above, a multi-view interactive digital media representation can take numerous forms within the scope of this disclosure. For instance, a multi-view interactive digital media representation may include a surround view, multi-view image, or three dimensional model. Surround views are described in more detail with regard to U.S. patent application Ser. No. 14/530,669 by Holzer et al., filed on Oct. 31, 2014, titled “Analysis and Manipulation of Images and Video for Generation of Surround Views,” which is incorporated by reference herein in its entirety and for all purposes. According to various embodiments described therein, a surround view provides a user with the ability to control the viewpoint of the visual information displayed on a screen. In addition, a surround view presents a user with an interactive and immersive active viewing experience.

According to various embodiments, the data used to generate a surround view can come from a variety of sources. In particular, data such as, but not limited to, two-dimensional (2D) images can be used to generate a surround view. These 2D images can include color image data streams such as multiple image sequences, video data, etc., or multiple images in any of various formats for images, depending on the application. Another source of data that can be used to generate a surround view includes location information. This location information can be obtained from sources such as accelerometers, gyroscopes, magnetometers, GPS, WiFi, IMU-like systems (Inertial Measurement Unit systems), and the like. Yet another source of data that can be used to generate a surround view can include depth images. These depth images can include depth, 3D, or disparity image data streams, and the like, and can be captured by devices such as, but not limited to, stereo cameras, time-of-flight cameras, three-dimensional cameras, three-dimensional capture devices, a combination of devices, a combination of multidimensional capture devices, and the like.

According to one example, gathered data can be fused together. In some embodiments, a surround view can be generated by a combination of data that includes both 2D images and location information, without any depth images provided. In other embodiments, depth images and location information can be used together. Various combinations of image data can be used with location information, depending on the application and available data.

In the present example, the data that has been fused together is then used for content modeling and context modeling. According to various examples, the subject matter featured in the images can be separated into content and context. The content can be delineated as the object of interest and the context can be delineated as the scenery surrounding the object of interest. According to various embodiments, the content can be a three-dimensional model, depicting an object of interest, although the content can be a two-dimensional model in some embodiments. Furthermore, in some embodiments, the context can be a two-dimensional model depicting the scenery surrounding the object of interest. Although in many examples the context can provide two-dimensional views of the scenery surrounding the object of interest, the context can also include three-dimensional aspects in some embodiments. For instance, the context can be depicted as a “flat” image along a cylindrical “canvas,” such that the “flat” image appears on the surface of a cylinder. In addition, some examples may include three-dimensional context models, such as when some objects are identified in the surrounding scenery as three-dimensional objects. In various embodiments, the models provided by content modeling and context modeling can be generated by combining the image and location information data.

According to various embodiments, context and content of a surround view are determined based on a specified object of interest. In some examples, an object of interest is automatically chosen based on processing of the image and location information data. For instance, if a dominant object is detected in a series of images, this object can be selected as the content. In other examples, a user specified target can be chosen. It should be noted, however, that a surround view can be generated without a user specified target in some applications.

According to various embodiments, one or more enhancement algorithms can be applied. In particular example embodiments, various algorithms can be employed during capture of surround view data, regardless of the type of capture mode employed. These algorithms can be used to enhance the user experience. For instance, automatic frame selection, stabilization, view interpolation, filters, and/or compression can be used during capture of surround view data. In some examples, these enhancement algorithms can be applied to image data after acquisition of the data. In other examples, these enhancement algorithms can be applied to image data during capture of surround view data.

Although various embodiments described herein may include references to surround views, other types of multi-view interactive digital media representations are also intended to be included. For instance, representations such as a multi-view image, three dimensional model, or other formats can be integrated with audio data. For instance, a multi-view image or three dimensional model may include navigation capabilities, views of the subject matter from various viewpoints, etc. In these representations, content and context need not necessarily be separated.

With reference toFIGS.1A-1B, shown is an example of a navigation through a multi-view interactive digital media representation on a mobile device. Specifically, a mobile device100is shown at two different navigation points. InFIG.1Aa view of a multi-view interactive digital media representation at a first position102is shown. A car is featured as content106, and some trees are shown as context104. In some embodiments, the multi-view interactive digital media representation can be played automatically from this position102if the user presses the autoplay button112. Such automatic play may include a predetermined navigation through the multi-view interactive digital media representation. In other embodiments, the autoplay button112may play an audio track while the user navigates manually through the multi-view interactive digital media representation. Additional buttons or navigation tools can also be included.

In the present example, various navigations are available to the user. For instance, the user can browse through the multi-view interactive digital media representation by swiping around the trees using navigation102. This may involve a rotation around the trees to reach a new viewpoint behind the car. The user can also browse through the multi-view interactive digital media representation by dragging the car to the left of the screen using navigation108. In this case, the car would move to the left and the scenery around the car would also shift relative to the car. Yet another example of browsing includes the user swiping in the direction of navigation110to move the viewpoint of the scene. The car would then be viewed at a different angle and the scenery surrounding this viewpoint would also shift. Endless possibilities for navigating through the multi-view interactive digital media representation are possible.

In the present example, pressing the autoplay button112shows the car moving from right to left in the scene and the scenery moving relative to the car.FIG.1Bshows a view at second position116once the scene has progressed114in the automatic playback. As shown, the content106has shifted relative to the context104. Although the present example shows the visual playback of the scene based on the user's navigation through the multi-view interactive digital media representation, the audio data accompanying this visual playback will be described in more detail below with particular reference toFIGS.3A-3C. It should be noted that audio data may be attached to either position of an object or the position of a capture device. Video object position refers to position of an object such as a thing or entity or person in a surround view. Video capture position refers to the position of the capture device. Both can be used for triggering playback and/or positioning the audio data. Position may also include different angles, perspectives, geographic locations, etc. According to various embodiments, a surround view of a piano may be depicted where the keys of the piano are the dominant elements and an audio file is attached to any piano key location. Therefore, someone can play a piano by touching the corresponding keys and navigating through the piano keys in the surround view.

In other examples, a panoramic surround view includes a car that is driving by. In the background there are trees with chirping birds and the ocean with waves crashing on the shore. The audio data that is recorded is decomposed into the sound of the car, the sound of the birds, and the sound of the waves and the audio files are attached to the locations of those elements in the visual data. In still other examples, a surround view of a person includes a person making a face at a specific camera position within the surround view. An audio file is automatically played when that camera position is reached while navigating through the surround view.

In the present example, a particular multi-view interactive digital media representation is depicted with a car as content106and the trees as context104. With reference toFIG.2, shown is a depiction of content and context in the multi-view interactive digital media representation described with regard toFIGS.1A-1B. In particular,FIG.2illustrates one example of separation of content and context in a scene200. In the present example, the viewer212is shown viewing or capturing images of an object of interest, such as the car shown.

According to various embodiments, the digital visual data included in a scene200can be, semantically and/or practically, separated into content206and context210, especially in the implementation of surround views. According to particular embodiments, content206can include the object(s), person(s), or scene(s) of interest while the context210represents the remaining elements of the scene surrounding the content206. In the present example, the object202is a car. This object202constitutes the content206of the scene200. The trees in the scenery208constitute the context. In some examples, a surround view may represent the content206as three-dimensional data, and the context210as a two-dimensional panoramic background. In other examples, a surround view may represent both the content206and context210as two-dimensional panoramic scenes. In yet other examples, content206and context210may include three-dimensional components or aspects. In particular embodiments, the way that the surround view depicts content206and context210depends on the capture mode used to acquire the images.

In some examples, such as but not limited to: recordings of objects, persons, or parts of objects or persons, where only the object, person, or parts of them are visible, recordings of large flat areas, and recordings of scenes where the data captured appears to be at infinity (i.e., there are no subjects close to the camera), the content206and the context210may be the same. In these examples, the surround view produced may have some characteristics that are similar to other types of digital media such as panoramas. However, according to various embodiments, surround views include additional features that distinguish them from these existing types of digital media. For instance, a surround view can represent moving data. Additionally, a surround view is not limited to a specific cylindrical, spherical or translational movement. Various motions can be used to capture image data with a camera or other capture device. Furthermore, unlike a stitched panorama, a surround view can display different sides of the same object.

With reference toFIG.3A, shown is an example of a process for integrating audio with a multi-view interactive digital media representation. In particular, the process300includes retrieving a multi-view interactive digital media representation at301. As described above, the multi-view interactive digital media representation can include formats such as a surround view, multi-view image, or three dimensional model. In some examples, the multi-view interactive digital media representation includes numerous images fused together into content and context models, where the content model includes an object and the context model includes scenery surrounding the object.

In the present example, the process300continues by retrieving audio data to be integrated into the multi-view interactive digital media representation at303. The audio data can be obtained in a variety of ways depending on the application or desired effect. For instance, an audio stream can be recorded together (i.e., at the same time) with the recording of the visual data included in the multi-view interactive digital media representation. In another example, an audio stream can be recorded separately from the visual data. One or more audio recordings can be created to use with the visual data. In other examples, pre-recorded audio files can be used. For instance, the user may have recorded this audio data at an earlier time or may use one or more existing audio files from the Internet or other sources. Some examples of audio files that may be used include musical recordings, sound effects, ambient noise or sounds, voice recordings, etc. A variety of effects can be applied in the processing step. Examples of effects include changing the pitch or introducing an echo effect.

In the present example, the process300further includes processing the audio data at305. In particular, after an audio file has been recorded or selected it has to be processed in order to be integrated into the format of the multi-view interactive digital media representation. Several options for processing are possible, one or more of which can be combined in some examples. In one example, the recorded/selected audio file is directly used without processing. In another example, the recorded/selected audio file is decomposed into different components. For instance, voices are separated from background sounds, and different sound sources are separated (e.g. cars, ocean, birds, talking). This decomposition can be implemented in a variety of ways. One way includes using independent component analysis. Once the audio file is decomposed into different components, the separate audio streams are then either presented to the user for further selection and positioning or automatically assigned to locations in the multi-view interactive digital media representation corresponding to where they originated (i.e. locations of the original audio if the audio was recorded with the video). More details relating to the positioning of audio streams within the multi-view interactive digital media representation are discussed below with regard to audio playback at307. Additionally, a particular example of processing audio data is described in conjunction withFIG.3B.

Once the audio data is processed, the audio data is then played in coordination with the multi-view interactive digital media representation at307. The playback of the audio data can be done in several ways. In one example, the audio data is played once as soon as the multi-view interactive digital media representation is loaded and displayed. In some instances, the audio data is played while the user navigates through the multi-view interactive digital media representation and the audio data is played at original speed during this navigation, independent of navigation direction or speed. In other instances, the audio data is played while the multi-view interactive digital media representation follows a predetermined auto play sequence. In yet other instances, audio and image data are initially played (once or multiple times) without user interaction available. Once this initial play is over, the user can manually navigate through the multi-view interactive digital media representation. In another example, the audio data is played repeatedly as soon as surround view is loaded and displayed. In some instances, the audio data is played repeatedly whether the user navigates the multi-view interactive digital media representation or the multi-view interactive digital media representation plays through an automatic playback sequence.

In some examples, the audio data is played in conjunction with navigation through the multi-view interactive digital media representation. For instance, navigating in one direction plays the audio forward and navigating in the other direction plays the audio backwards. In some instances, the speed of audio playback corresponds to navigation speed.

Another option for audio playback includes playing the audio data when a “Play” button is pressed or otherwise selected. For instance, as shown inFIGS.1A-1B, an autoplay button112can be located in a predefined position in a user interface and selection of this button can begin audio playback. In another instance, a play button can be positioned in a manner to follow the corresponding image data. In this instance, the play button can either be visible or invisible to the user. According to various embodiments, a play button can have standard shape (e.g., triangular) or follow the shape of a specific object/region in the scene (e.g. a bird, a car, etc.).

In other embodiments, audio playback is based on navigation through the multi-view interactive digital media representation. In one example, audio is played when a certain position/frame of the multi-view interactive digital media representation is reached during navigation, whether manual or automatic. In some examples, audio data is associated with specific positions or frames in the multi-view interactive digital media representation. The closer the user navigates towards these positions or frames, the louder the corresponding audio plays. In particular examples, audio data can be “attached” to a 3D location in a scene and the volume of the audio playback depends on the distance and orientation of the current view point of the visualization. Specifically, the sound volume of an audio playback increases if the navigation location approaches the 3D location of the audio data. Similarly, the volume increases if the viewing direction of the navigation is oriented towards the location of the audio data and the volume decreases if the viewing direction of the navigation is oriented away from the location of the audio data.

Referring toFIG.4, shown is an example representation of a multi-view interactive digital media representation where the navigation location varies with respect to different objects in the scene400. The object402moving in direction404may have associated sounds such as car revving and motor sounds. At navigation location408, the viewpoint is directed towards the object402and the sound of the object402would be louder than when the navigation location is at any of the other locations410,412, or414as shown. When the navigation location414is selected, the sounds associated with the scenery406may be more audible, such as trees swaying in the wind. From this location, sounds from both the scenery406and the object402might be audible. In contrast, at navigation location410, the scenery406sounds may not be audible and the sounds of the object402may be played at a lower volume because the location410is further away from the object402.

Referring back toFIG.3A, in some embodiments, the volume, pitch, and/or playback speed of audio data at307is dependent upon user input. For instance, swiping speed on a touch screen or by a mouseover or force applied to a force-sensitive touch pad can affect volume, pitch, and/or playback speed of audio data in some examples. In addition, rotational velocity, as measured by an inertial sensor, of a mobile device can affect volume, pitch, and/or playback speed of audio data in some examples.

Another option for audio playback is to play background music (e.g. a song) or sound effects while the user navigates through a multi-view interactive digital media representation. In some embodiments, the type or style of the background music can be correlated with the occurrence and strength of visual filters. For instance, detection of a beat in the music increases the strength of a filter or a different filter is applied if the style or type of the music changes. In some examples, the filters can be applied to correspond to the chosen background music. In other examples, the background music can be chosen based on any filters or effects included in the visual file. For instance, certain filters that make the visuals dark and murky may cause selection of darker songs or sound effects.

In some examples, audio playback in307can also occur during automatic playback of the multi-view interactive digital media representation. For instance, automatic playback may be initiated by user action such as selection of an autoplay button, as shown inFIGS.1A-1B, automatically without user action, or in a video export. In some embodiments, one or more multi-view interactive digital media representations can be exported and synchronized with background music. Specifically, certain types of music can be correlated with certain types of multi-view interactive digital media representations. For instance, calm music can be correlated with scenic multi-view interactive digital media representations. In addition, the playback speed of a multi-view interactive digital media representation can be correlated with the type of music. In particular, calm music can be combined with slow playback of a multi-view interactive digital media representation.

In some embodiments, a switch between multi-view interactive digital media representations or a switch in the type of visualization or playback effects within a certain multi-view interactive digital media representation can happen when the style of the music changes or in synchronization with the occurrence of certain instruments or beats. In some examples, a change in the visualization direction within a multi-view interactive digital media representation occurs in synchronization with certain instruments or beats. In other examples, a change in the playback speed of the visual data may occur if the type of music changes or in synchronization with a certain instrument or beat. In yet other examples, the occurrence and strength of visual filters can correspond to the type of music and the occurrence of certain instruments. For instance, a beat increases the strength of a filter or a different filter is applied if the style or type of the music changes. In some embodiments, the user can define where and when the multi-view interactive digital media representations are switched and/or how the playback changes during automatic playback.

In particular embodiments, a combination of automatic playback and interactive navigation is possible. For example, a specific song can be played in the background and the current visible multi-view interactive digital media representation is switched at certain locations within the song, but the user is able to interactively navigate through the currently visible multi-view interactive digital media representation. In a converse example, a song can be played in the background and the user can switch between different multi-view interactive digital media representations, but the multi-view interactive digital media representations are autoplayed.

Although the above example describes various embodiments relating to integrating audio with a multi-view interactive digital media representation,FIGS.3B and3Cinclude examples of particular embodiments of methods for processing audio data and playing the audio data in coordination with a multi-view interactive digital media representation. With reference toFIG.3B, shown is an example of a particular method for processing an audio file to be integrated with a multi-view interactive digital media representation as referred to in step305ofFIG.3A. In the present example, processing the audio data305includes segmenting the audio data into a first segment and a second segment at311. According to various embodiments, the first segment and second segment can be segmented from the audio data using independent component analysis. In the present example, the first segment includes a first set of sound sources and the second segment includes a second set of sound sources. For instance, the first set of sound sources may include voices and the second set of sound sources may include background noises. In another instance, the first segment may include sounds occurring near a first position in the multi-view interactive digital media representation, and the second segment may include sounds occurring near a second position in the multi-view interactive digital media representation.

Next, the first segment is associated with a first position in the multi-view interactive digital media representation at313and the second segment is associated with a second position in the multi-view interactive digital media representation at315. In some instances, the first position is the same as the second position. In these cases, the first segment and second segment will overlap when played. In some examples, the first position and second position are located in separate places, but at least a portion of the first segment and second segment may overlap when played.

With reference toFIG.3C, shown is a particular example of a process for playing an audio file integrated with a multi-view interactive digital media representation. According to various embodiments, the audio data is played back based on a user's navigation through the multi-view interactive digital media representation. In some examples, the first segment is played when the first position in the multi-view interactive digital media representation is played and the second segment is played when the second position in multi-view interactive digital media representation is played. In particular, playing the audio data307includes receiving a request to play a multi-view interactive digital media representation from the first position at321. In response, the first segment of audio data is played when the multi-view interactive digital media representation is played from the first position at323. Next, a request is received to play the multi-view interactive digital media representation from the second position at325. In response, the second segment of audio data is played when the multi-view interactive digital media representation is played from the second position at327.

According to various embodiments, the requests to navigate to the first position and second position can be made in numerous ways. In one example, a user's navigation through the multi-view interactive digital media representation includes selecting automatic playback, where automatic playback triggers play of a predetermined navigation through the multi-view interactive digital media representation. This predetermined navigation includes a set sequence of views and audio data that includes navigation to the first position and the second position. In another example, a user's navigation through the multi-view interactive digital media representation includes navigating to the first position and navigating to the second position. As described above with regard toFIG.1A, the user can navigate to these positions through user input such as swiping a screen, manipulating a mouse or cursor, or tilting or otherwise moving a mobile device. In some embodiments, the first segment plays when the user navigates to the first position, and the volume of the first segment decreases as the user navigates away from the first position. Similarly, the volume of the first segment increases as the user navigates towards the first position.

Various computing devices can implement the methods described herein. For instance, a mobile device, computer system, etc. can be used to display a multi-view interactive digital media representation and the associated audio media. With reference toFIG.5, shown is a particular example of a computer system that can be used to implement particular examples of the present invention. For instance, the computer system500can be used to display a multi-view interactive digital media representation and the associated audio data according to various embodiments described above. In addition, the computer system500shown can represent a computing system on a mobile device. According to particular example embodiments, a system500suitable for implementing particular embodiments of the present invention includes a processor501, a memory503, an interface511, and a bus515(e.g., a PCI bus). The interface511may include separate input and output interfaces, or may be a unified interface supporting both operations. When acting under the control of appropriate software or firmware, the processor501is responsible for such tasks such as optimization. Various specially configured devices can also be used in place of a processor501or in addition to processor501. The complete implementation can also be done in custom hardware. The interface511is typically configured to send and receive data packets or data segments over a network. Particular examples of interfaces the device supports include Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like.

In addition, various very high-speed interfaces may be provided such as fast Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces and the like. Generally, these interfaces may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some instances, volatile RAM. The independent processors may control such communications intensive tasks as packet switching, media control and management.

According to particular example embodiments, the system500uses memory503to store data and program instructions and maintain a local side cache. The program instructions may control the operation of an operating system and/or one or more applications, for example. The memory or memories may also be configured to store received metadata and batch requested metadata.

Because such information and program instructions may be employed to implement the systems/methods described herein, the present invention relates to tangible, machine readable media that include program instructions, state information, etc. for performing various operations described herein. Examples of machine-readable media include hard disks, floppy disks, magnetic tape, optical media such as CD-ROM disks and DVDs; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and programmable read-only memory devices (PROMs). Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.

Although many of the components and processes are described above in the singular for convenience, it will be appreciated by one of skill in the art that multiple components and repeated processes can also be used to practice the techniques of the present disclosure.

While the present disclosure has been particularly shown and described with reference to specific embodiments thereof, it will be understood by those skilled in the art that changes in the form and details of the disclosed embodiments may be made without departing from the spirit or scope of the invention. It is therefore intended that the invention be interpreted to include all variations and equivalents that fall within the true spirit and scope of the present invention.