Interactive virtual reality broadcast systems and methods

A method for generating a live VR broadcast of a three-dimensional (3D) virtual reality (VR) performance environment is provided. The 3D VR performance environment includes 3D VR avatars corresponding to performers performing on a performance area of a concert space, with the performers being separated by a predetermined distance selected to prevent transmission of airborne pathogens between the performers while performing. The 3D VR avatars are displayed on a VR performance area representing the performance area. The facial expressions and body movements of the 3D VR avatars are synchronized with audio data of the performers. The 3D VR avatars are displayed within the VR performance area at relative locations that appear separated by less than the predetermined distance. Audience devices provide audience feedback data that is outputted in real time to the performers.

FIELD

The present disclosure relates to interactive virtual reality broadcast systems and methods and, more particularly, to interactive virtual reality broadcast systems and methods that generate a live virtual reality broadcast of three-dimensional virtual reality avatars performing with instruments in a virtual reality performance area.

BACKGROUND

Live musical performances, such as live orchestral performances at a concert hall, have long been an exciting and enjoyable experience for both the audience members watching and listening to the musical performance as well as the musicians that are performing for the audience. During a live musical performance, the audience members and the musicians often share a special bond resulting from the interaction between the musicians creating the live music and the audience members viscerally reacting as they hear the music and watch the performance. Musicians often feed off of the energy of the audience and the audience members are able to participate in the creation of the music being performed as they watch and react to the facial expressions and body movements of the performers. This interaction between the musicians and the audience makes the live musical performance an engaging and memorable experience for both the musicians and performers and the audience members.

During an epidemic or global pandemic, however, health concerns make traditional live musical performances nearly impossible. For example, to prevent airborne pathogens from being transmitted, individual audience members and musicians must maintain social distance from all other audience members and musicians within the concert hall. Given the layout of most concert halls, it is logistically nearly impossible for musicians and audience members to be located within the same indoor concert hall and replicate the same levels of interaction and engagement while being socially distanced.

Traditional broadcast systems can be used to broadcast a live performance from a concert hall to remote audience members. The musicians at such a live performance, however, would need to be socially distanced from each other. Such traditional broadcast systems, however, do not provide audience member feedback to the musicians and do not result in the same levels of interaction and engagement that were enjoyed by musicians and audience members during traditional live musical performances.

SUMMARY

The present disclosure provides a method that includes arranging a plurality of performers at a plurality of performers locations on a performance area of a concert space such that the performers are separated by at least a predetermined distance, the predetermined distance being selected to prevent transmission of airborne pathogens between the plurality of performers while the plurality of performers are performing at the concert space. The method further includes receiving, with a virtual reality (VR) system controller, facial expression data from a plurality of facial expression sensors and body movement data from a plurality of body movement sensors, each facial expression sensor and each body movement sensor being associated with a performer of the plurality of performers, with the facial expression data from each facial expression sensor capturing facial expressions of the associated performer and the body movement data from each body movement sensor capturing body movements of the associated performer. The method further includes receiving, with the VR system controller, audio data from at least one microphone located in the concert space. The method further includes retrieving, with the VR system controller, a plurality of previously stored three-dimensional (3D) VR avatars associated with the plurality of performers and a previously stored VR performance area corresponding to and representing the performance area of the concert space from a data storage. The method further includes generating, with the VR system controller, a 3D VR performance environment showing the plurality of 3D VR avatars performing within the VR performance area and with each 3D VR avatar being manipulated to include the facial expressions and the body movements of the associated performer based on the facial expression data from the associated facial expression sensor for the associated performer and the body movement data from the associated body movement sensor for the associated performer. The method further includes generating, with the VR system controller, a live VR broadcast of the 3D VR performance environment including the plurality of 3D VR avatars displayed on the VR performance area and including the audio data such that the facial expressions and the body movements of the plurality of 3D VR avatars are synchronized with the audio data, the 3D VR avatars being displayed within the VR performance area at relative locations that appear separated by less than the predetermined distance. The method further includes transmitting, with the VR system controller, the live VR broadcast to a plurality of VR audience devices, each VR audience device outputting the live VR broadcast to an associated audience member and including at least one of an audience camera and an audience microphone that captures audience feedback data of the associated audience member. The method further includes receiving, with the VR system controller, the audience feedback data from the plurality of VR audience devices, including at least one of audience audio data and audience video data. The method further includes outputting, with the VR system controller, the audience feedback data, including at least one of the audience audio data and the audience video data, in real time to the plurality of performers while the plurality of performers are performing at the concert space.

In other features, the plurality of VR audience devices includes at least one VR headset having a stereoscopic head-mounted display and at least one tracking sensor that generates at least one of head motion data and eye motion data. The VR headset is configured to display the live VR broadcast to the associated audience member from a viewpoint determined based on at least one of the head motion data and the eye motion data.

In other features, the plurality of VR audience devices includes a computing device with a two-dimensional display. The computing device includes at least one of a tablet, a smartphone, a laptop, and a desktop computer. The computing device is configured to display the live VR broadcast on the two-dimensional display to the associated audience member from a viewpoint determined based on user input received by the computing device.

In other features, the plurality of VR audience devices are configured to enable the associated audience member to manipulate the live VR broadcast and zoom in on a selected performer of the plurality of performers, and to display a closer 3D representation of the 3D VR avatar associated with the selected performer.

In other features, the at least one microphone includes a plurality of microphones located at a plurality of difference audience member locations in an audience area of the concert space and the audio data includes a plurality of audio streams generated by the plurality of microphones. The method can further include generating, with the VR system controller, the live VR broadcast for a particular VR audience device of the plurality of VR audience devices with audio data corresponding to a particular audio stream of the plurality of audio streams.

In other features, the method can also include receiving, with the VR system controller, user input from the particular VR audience device indicating a different location and generating the live VR broadcast for the particular VR audience device with audio data corresponding to a different audio stream of the plurality of audio streams in response to receiving the user input indicating the change of location, the different audio stream being associated with a microphone at the different location.

In other features, the VR performance area includes a VR microphone stand and at least two of the 3D VR avatars are displayed within the VR performance area together singing into the VR microphone stand during at least a portion of the live VR broadcast.

In other features, the VR performance area includes a designated location and at least two of the 3D VR avatars are displayed within the VR performance area together located at the designated location of the VR performance area during at least a portion of the live VR broadcast.

In other features, outputting, with the VR system controller, the audience feedback data in real time to the plurality of performers includes outputting the audience feedback data to a plurality of augmented reality (AR) glasses worn by the plurality of performers.

In other features, the plurality of body movement sensors are accelerometers, each accelerometer being attached to the associated performer.

In other features, the plurality of body movement sensors are LIDAR sensors, each LIDAR sensor being directed to the associated performer.

In other features, the plurality of body movement sensors are video cameras, each tracking a plurality of passive optical system markers located on the associated performer.

In other features, the performance area of the concert space includes at least two tiers with at least one performer of the plurality of performers located on each of the at least two tiers, the at least two tiers are not shown in the VR performance area, and the plurality of 3D VR avatars are displayed on a single level of the VR performance area.

In other features, a floor of an upper tier of the at least two tiers is porous and the performance area of the concert space includes air handling equipment and at least one air duct that directs air flow from the air handling equipment through the floor of the upper tier and around the at least one performer located on the upper tier.

In other features, the method further includes noise filtering, with the VR system controller, the audio data to remove noise generated by the air handling equipment from the audio data.

In other features, the method further includes receiving, with the VR system controller, at least one of conductor audio data and conductor video data from a conductor device that includes at least one of a conductor body movement sensor that captures body movement instructions of a conductor of the plurality of performers and a conductor microphone that captures voice instructions of the conductor. The method also includes outputting, with the VR system controller, the at least one of the conductor audio data and the conductor video data in real time to the plurality of performers while the plurality of performers are performing at the concert space.

The present disclosure also provides a method that includes arranging a plurality of performers at a plurality of performers locations on a performance area of a concert space such that the performers are separated by at least a predetermined distance, the predetermined distance being selected to prevent transmission of airborne pathogens between the plurality of performers while the plurality of performers are performing at the concert space. The method also includes receiving, with a virtual reality (VR) system controller, facial expression data from a plurality of facial expression sensors and body movement data from a plurality of body movement sensors, each facial expression sensor and each body movement sensor being associated with a performer of the plurality of performers, with the facial expression data from each facial expression sensor capturing facial expressions of the associated performer and the body movement data from each body movement sensor capturing body movements of the associated performer. The method also includes receiving, with the VR system controller, audio data from at least one microphone located in the concert space. The method also includes retrieving, with the VR system controller, a plurality of previously stored three-dimensional (3D) VR avatars associated with the plurality of performers from a data storage. The method also includes generating, with the VR system controller, a 3D VR performance environment showing the plurality of 3D VR avatars performing with each 3D VR avatar being manipulated to include the facial expressions and the body movements of the associated performer based on the facial expression data from the associated facial expression sensor for the associated performer and the body movement data from the associated body movement sensor for the associated performer. The method also includes generating, with the VR system controller, a live VR broadcast of the 3D VR performance environment including the plurality of 3D VR avatars and including the audio data such that the facial expressions and the body movements of the plurality of 3D VR avatars are synchronized with the audio data, the 3D VR avatars being displayed at relative locations that appear separated by less than the predetermined distance. The method also includes transmitting, with the VR system controller, the live VR broadcast to a plurality of audience devices. Each audience device of the plurality of audience devices includes an audience device environment camera, a display, and at least one speaker, each audience device being configured to (i) generate image data representing an environment of the audience device with the audience device environment camera; (ii) generate live augmented reality (AR) image data that overlays the plurality of 3D VR avatars onto the image data representing the environment of the audience device; (iii) output the live AR image data on the display such that the plurality of 3D VR avatars appear to be performing within the environment of the audience device; and (iv) output the audio data of the live VR broadcast with the at least one speaker.

In other features, each audience device is further configured to output the live AR image data on the display such that the plurality of 3D VR avatars appear stationary within the environment of the audience device while the audience device is moved around the environment of the audience device.

In other features, each audience device is further configured to receive user input to select a particular 3D VR avatar and to output the live AR image data on the display such that only the particular 3D VR avatar is displayed within the environment of the audience device and the particular 3D VR avatar appears larger within the environment.

In other features, each audience device is further configured to receive user input selecting and rotating a particular 3D VR avatar and to output the live AR image data on the display such that the particular 3D VR avatar is rotated within the environment of the audience device based on the received user input.

In other features, each audience device includes an audience camera and an audience microphone that captures audience feedback data of the associated audience member. The method can further include receiving, with the VR system controller, the audience feedback data from the plurality of audience devices, including audience audio data and audience video data, and outputting, with the VR system controller, the audience feedback data, including the audience audio data and the audience video data, in real time to the plurality of performers while the plurality of performers are performing at the concert space.

In other features, the plurality of body movement sensors are accelerometers, each accelerometer being attached to the associated performer.

In other features, the plurality of body movement sensors are LIDAR sensors, each LIDAR sensor being directed to the associated performer.

In other features, the plurality of body movement sensors are video cameras, each tracking a plurality of passive optical system markers located on the associated performer.

In other features, the plurality of facial expression cameras are video cameras.

The present disclosure also provides a method that includes arranging a plurality of performers at a plurality of performers locations on a performance area of a concert space such that the performers are separated by at least a predetermined distance, the predetermined distance being selected to prevent transmission of airborne pathogens between the plurality of performers while the plurality of performers are performing at the concert space. The method further includes receiving, with a virtual reality (VR) system controller, facial expression data from a plurality of facial expression sensors and body movement data from a plurality of body movement sensors, each facial expression sensor and each body movement sensor being associated with a performer of the plurality of performers, with the facial expression data from each facial expression sensor capturing facial expressions of the associated performer and the body movement data from each body movement sensor capturing body movements of the associated performer. The method further includes receiving, with the VR system controller, audio data from at least one microphone located in the concert space. The method further includes retrieving, with the VR system controller, a plurality of previously stored three-dimensional (3D) VR avatars associated with the plurality of performers from a data storage. The method further includes generating, with the VR system controller, a 3D VR performance environment showing the plurality of 3D VR avatars associated with the plurality of performers with each 3D VR avatar being manipulated to include the facial expressions and the body movements of the associated performer, based on the facial expression data from the associated facial expression sensor for the associated performer and the body movement data from the associated body movement sensor for the associated performer. The method further includes generating, with the VR system controller, a live VR broadcast of the 3D VR performance environment including the plurality of 3D VR avatars and including the audio data such that the facial expressions and the body movements of the plurality of 3D VR avatars are synchronized with the audio data, the 3D VR avatars being displayed at relative locations that appear separated by less than the predetermined distance. The method further includes transmitting, with the VR system controller, the live VR broadcast to a plurality of audience devices. Each audience device of the plurality of audience devices is configured to generate and display live holographic data such that the plurality of 3D VR avatars appear to be performing as holographs within an environment of the audience device.

In other features, each audience device includes a holographic screen and the live holographic data is displayed via the holographic screen.

In other features, each audience device includes a volumetric display and the live holographic data is displayed via the volumetric display.

In other features, the plurality of body movement sensors are accelerometers, each accelerometer being attached to the associated performer.

In other features, the plurality of body movement sensors are LIDAR sensors, each LIDAR sensor being directed to the associated performer.

In other features, the plurality of body movement sensors are video cameras, each tracking a plurality of passive optical system markers located on the associated performer.

DETAILED DESCRIPTION

The present disclosure provides interactive virtual reality broadcast systems and methods that include arranging performers at socially distanced performer locations on a performance area, such as a stage of a concert space. In other words, the performers are located on the stage at locations that are separated by a predetermined distance, such as six feet, selected to prevent the transmission of airborne pathogens between the performers while performing on stage. As discussed in detail below, each of the performers has an associated facial expression sensor, such as a camera, directed towards a face of the performer to capture the facial expressions of the performer while the performer is performing. For example, the performer may be a musician and the facial expression sensor may capture the facial expressions of the musician while the musician is playing music. Each of the performers also has an associated body movement camera that captures the body movements of the performer. Microphones are placed in the concert space, such as in an audience area of the concert space, in the performance area, or on or near instruments being played by the performers, and capture audio data generated by the performers, such as the music being played by the musicians.

A virtual reality system controller receives video data from the performer cameras and the body movement cameras and audio data from the microphones and generates a three-dimensional virtual reality avatar for each of the performers performing on the stage. For example, the avatars are generated and manipulated to exhibit body movements and facial expressions that correspond to the performer associated with the avatar. The virtual reality system controller then synchronizes the audio data and the video data such that the facial expressions and body movements of the avatars are synchronized with the audio data. The virtual reality system controller then generates a live virtual broadcast of a virtual reality performance that displays the avatars performing on a virtual reality performance area and outputs audio data synchronized with the movements of the avatars such that the avatars are shown playing the music of the audio data. While the live performers are socially distanced on the actual stage where they are performing, the virtual reality avatars can be shown close together, and not socially distanced, at locations in the virtual performance area corresponding to where the performers would normally be located during a traditional live performance.

As discussed in further detail below, the live virtual reality broadcast can be transmitted to virtual reality audience devices, such as virtual reality headsets or other audience devices, such as tablets, mobile devices, or other computing devices. The virtual reality audience devices can include video cameras and microphones to transmit audience feedback in the form of video data of the associated audience member's face and audio data, such as cheering, whistling, etc., generated by the associated audience member. The audience feedback data can be transmitted back to the virtual reality system controller, which can then output the audience feedback data to augmented reality devices, such as augmented reality glasses, being worn by and/or associated with the performers so that the performers can receive audience feedback in real time during the performance.

In this way, the interactive virtual reality broadcast systems and methods of the present disclosure allow performers to be socially distanced on stage, while generating and transmitting a virtual reality environment, including avatars of the performers synced with the music being played by the performers, to audience members using virtual reality devices. In addition, the interactive virtual reality broadcast systems and methods of the present disclosure receive and transmit audience member feedback, such as audience member video and audio data, to augmented reality devices of the performers so the performers can receive live audience feedback during their performance. In this way, the interactive virtual reality broadcast systems and methods of the present disclosure are able to generate similar levels of interaction and engagement that would previously have been enjoyed by musicians and audience members during traditional live musical performances, while maintaining social distance between all participants to prevent the transmission of airborne pathogens. Additional details and features are discussed below with reference to the figures.

With reference toFIG. 1, an interactive virtual reality (VR) broadcast system10is shown and includes a VR system controller12and a data storage13. The VR system controller12receives input from one or more microphones14, one or more facial expression sensors16, and one or more body movement sensors18. The VR system controller12generates output to one or more audience devices22that may include, for example, a camera24and/or a microphone26. The VR system controller12also generates output to one or more performer augmented reality (AR) devices28. As discussed in further detail below, the data storage13stores data including, for example, a previously generated three-dimensional (3D) VR model representing an associated concert hall space and 3D. The 3D VR model of the associated concert hall space can include image data and corresponding coordinate data so that the entire concert hall space can be replicated in the 3D VR environment, as discussed in further detail below. The data storage13also stores data representing previously generated 3D VR avatars for each of the performers within the concert hall space. As discussed in further detail below, 3D VR avatars corresponding to each of the performers can be generated to look like the actual performers, including similar body types, similar face shapes, similar eye color, similar hair color, similar hair style, etc. The data storage13also stores a library of 3D VR instruments that includes different types and styles of instruments that can be played by the 3D VR avatars in the 3D VR environment. The VR system controller12can retrieve the data for the 3D VR model of the concert hall, the data for the 3D VR avatars, and the data for the particular 3D VR instruments to be played by the 3D VR avatars, from the data storage13, place the 3D VR avatars with the 3D VR instruments in the 3D VR model of the concert hall to generate a 3D VR environment. The VR system controller12can then manipulate the 3D VR avatars based on facial expression data received from the facial expression sensors16and body movement data from the body movement sensors18, and synchronize the body movements and facial expressions of the 3D VR avatars with the audio data from the microphones14to generate and output a 3D VR audio/visual representation of the performers performing and playing their instruments within the concert hall space.

With additional reference toFIG. 2andFIG. 3, the facial expression sensors16can each include a video camera pointed at a face of a performer40. The facial expression sensors16, for example, can generate facial expression data that captures facial expressions of the associated performer40. As shown inFIG. 2, the performers40are arranged at performer locations on a performance area42, such as a stage, of a concert space44, such as a concert hall. WhileFIG. 2shows four performers40playing different instruments as an example, any number of performers40playing any number of instruments can be used. As shown inFIG. 2, the performers40are located at locations separated by a distance46that is larger than a predetermined distance. For example, the predetermined distance can be selected to be sufficiently large to prevent the transmission of airborne pathogens, such as viruses, between the performers40while the performers40are performing at the concert space44.

The body movement sensors18can be video cameras that perform motion capture by tracking the locations of passive optical system markers48place at various locations on the performers40. For example, the passive optical system markers48can be placed on the fingers, chest, shoulders, elbows, hips, knees, toes, etc., of the performer40, as shown inFIG. 3. The video camera of the body movement sensors18can then generate body movement data that captures body movements of the associated performer40based on the motion capture of the passive optical system markers48and transmit the body movement data to the VR system controller12.

Additionally or alternatively, the body movement sensors18can be accelerometers. In such case, the passive optical system markers48are replaced with individual accelerometers that generate body movement data that captures body movements of the performer40as the performer40moves and that communicate the body movement data to the VR system controller12. For example, the accelerometers can be located at the fingers, chest, shoulders, elbows, hips, knees, toes, etc., of the performer40. Additionally or alternatively, the body movement sensors18can be LIDAR sensors. In such case, the LIDAR sensors can be pointed at the performer40to generate body movement data that captures body movements of the associated performer40based on the movement of the performer40and to communicate the body movement data to the VR system controller12. In the event LIDAR sensors are used as the body movement sensors18, the passive optical system markers48can be omitted. Additionally or alternatively, any combination of video cameras with passive optical system markers, accelerometers, and/or LIDAR sensors can be used together. For example, video cameras and passive optical system markers can be used for motion capture for visible portions of the performers body that can be easily tracked with motion capture. In addition, accelerometers and/or LIDAR sensors can be used to capture other areas of the performers body, such as individual fingers. In this way, any combination of video cameras with passive optical system markers, accelerometers, and/or LIDAR sensors can be used to body movements of the associated performer40and generate body movement data indicating the captured body movements of the associated performer40.

The facial expression data from the facial expression sensors16and the body movement data from the body movement sensors18is received by the visual input synchronization module30of the VR system controller12. The visual input synchronization module30synchronizes the facial expression data and the body movement data to generate synchronized visual data. For example, the visual input synchronization module30can use time markers associated with, or embedded within, the facial expression data and the body movement data to synchronize both sets of data together and generate the synchronized visual data.

As mentioned above, the data storage13stores a library of 3D VR instruments, each having corresponding image data and 3D coordinate data representing the instrument. The library of 3D VR instruments can include any number of standard instruments corresponding to a standard shape, size, and color of the instrument, such as a standard violin, a standard cello, etc. The library of 3D VR instruments can also include a specialized instrument having a non-standard shape, size or color of the instrument. In the event a particular performer is generally known for playing a specialized instrument, a 3D VR representation of that performer's specialized instrument can be stored in the library of 3D VR instruments. As an example, violinist Laurie Anderson plays an electric violin that has a distinctive non-traditional body shape and the tuning pegs located on the body of the instrument instead of at the end of the neck of the violin. In this example, if Laurie Anderson were playing at a concert hall, the 3D VR avatar60corresponding to Laurie Anderson could be shown playing a 3D VR non-traditionally shaped violin that looks like the non-traditional violin that Laurie Anderson plays in real life. Other violinists at the performance, however, could be shown playing the standard violin from the library of 3D VR instruments. The library of 3D VR instruments can also include different styles or shapes of the same type of instrument. For example, the library of 3D VR instruments can include a number of different types of guitars, including guitars with a Stratocaster shape, guitars with a Les Paul shape, etc. In such case, a particular performer could choose which type of 3D VR guitar they would like their corresponding 3D VR avatar60to be playing during the performance. In addition, a performer could switch instruments in real life during the performance and a corresponding switch could be made in the VR environment.

FIG. 3shows a real-world performer40on the left box61playing a violin and the resulting 3D VR avatar60representing the same performer40on the right box63. As shown inFIG. 3, the 3D VR avatar60is generated and shown playing the same 3D VR instrument, i.e., the violin, as the associated performer40upon which the 3D VR avatar60is based. Generation of the 3D VR avatars60is discussed in further detail below with reference toFIG. 19.

The visual VR generation module32generates 3D VR avatars60for each of the performers40, as shown inFIG. 4. In addition, the visual VR generation module32retrieves the 3D VR model of the concert space44and generates a VR environment showing the 3D VR avatars60playing their 3D VR instruments in a VR performance area62that is a VR representation of the real-world performance area42and a VR concert space64that is a VR representation of the real-world concert space44. In other words, the real-world performers40are shown inFIG. 2performing on the real-world performance area42of the real-world concert space44while the associated 3D VR avatars60for each of the performers40are shown performing on the VR performance area62of the VR concert space64inFIG. 4. As noted above, the real-world performers40are each separated from each other by a distance46that is larger than a predetermined distance selected to be sufficiently large to prevent the transmission of airborne pathogens, such as viruses. In the VR concert space64, however, the 3D VR avatars60are generated to appear on the VR performance area62at relative locations that appear in the VR environment to be separated by less than the predetermined distance. In other words, the 3D VR avatars60are generated to appear at locations the performers would normally be located at during a traditional live performance without social distancing and without the health concerns associated with an epidemic or global pandemic.

The entire VR environment, including the 3D VR avatars60, the VR performance area62, and the VR concert space64, is generated by the visual VR generation module32of the VR system controller12and outputted to the audio/visual synchronization module34of the VR system controller12.FIG. 4also includes an overhead view68of the VR concert space64showing a viewpoint position69of the viewer. As shown inFIG. 4, the viewpoint is located at a location in the center of the VR concert space64in a left-to-right direction, a number of rows back from the VR performance area62. WhileFIG. 4shows the overhead view68presented on a tablet device as an example to show the overhead view68, the audience member can use any type of audience device22to select the location for the audience member's viewpoint of the performance within the 3D VR environment.

The audio/visual synchronization module34receives the entire VR environment, including the 3D VR avatars60, the VR performance area62, and the VR concert space64, as well as the audio data from the one or more microphones14. As shown inFIG. 2, the one or more microphones14can be located at various locations within an audience area70throughout the concert space44. InFIG. 2, for example, four microphones14are shown at different locations in the audience area70located near the front of the performance area42. A microphone14is also shown on the actual performance area42between the performers40. Additional microphones14can be provided at locations throughout the concert space44. Additional microphones14could also be located on the actual performance area42near individual performers40. Additional microphones14could also be placed near the back of the concert space44, in any balconies of the concert space44, etc. Alternatively, microphones14could be placed in each seat or in alternating seats, such as every other seat or row, of the audience area70. As discussed in further detail, the audience devices22can be configured to generate audio output that corresponds to a microphone location that is associated with a closest position to where the audience member's viewpoint is located within the VR environment. For example, if an audience member is viewing the VR performance from a location that is to the front right of the VR concert space64, then the associated audio output for the audience device of that particular audience member can be configured to correspond to the audio data generated by a microphone that is nearest to that location with the real-world concert space44.

The audio/visual synchronization module34synchronizes the audio data from the one or more microphones14with the entire VR environment, including the 3D VR avatars60, the VR performance area62, and the VR concert space64generated by the visual VR generation module32, such that the facial expressions and body movements of the 3D VR avatars60are synchronized with the audio data. For example, the audio/visual synchronization module34can use time markers associated with, or embedded within, the facial expression data, the body movement data, and the audio data to synchronize the audio data with the facial expression and body movement data. The audio/visual synchronization module34outputs the synchronized audio/visual VR environment data to the VR/AR output module36.

The VR/AR output module36of the VR system controller12outputs the synchronized audio/visual VR environment data, including the synchronized facial expression data, body movement data, and audio data, to one or more audience devices22.

With reference toFIG. 5, the audience device22can include a VR headset worn and used by an audience member80. The VR headset71can include a stereoscopic head-mounted display, headphone speakers to output audio data, at least one head motion tracking sensor that generates head motion data, and at least one eye tracking sensor that generates eye motion data. As shown inFIG. 5, the entire VR environment, including the 3D VR avatars60, the VR performance area62, and the VR concert space64are displayed to the audience member80via the stereoscopic head-mounted display and the synchronized audio data is outputted to the audience member via the headphone speakers. The at least one head motion tracking sensor and the at least one eye tracking sensor generate head motion data and eye motion data that indicates the directional viewpoint of the audience member's eyes so that the audience member80can look around the VR concert space64by moving the VR headset71up and down and rotating VR headset71left and right.

The viewpoint of the audience member80within the VR environment can be set based on a predetermined position within the VR concert space64. For example, the audience member80can purchase a “seat” within the VR concert space64and view the VR performance from that purchased seat location. In such case, the synchronized audio data outputted to the audience member could be selected to correspond to a microphone14within the audience area70of the concert space44that is closest to a seat location corresponding to the purchased seat location within the VR concert space64. In addition, the audience member80can move from one location to another within the VR concert space. For example, audience members80can be allowed to freely move from location to location within the VR concert space64. Alternatively, audience members80can purchase “seats” within the VR concert space64and pay additional money to upgrade their “seat” to a closer location within the VR concert space64. The seat locations within the VR concert space64can correspond to seat locations within the actual real-world concert space44. Additionally or alternatively, audience members80can be allowed to move anywhere within the VR concert space64, including to a location on the VR performance area62, i.e., to a location on stage next to or near a 3D VR avatar60on the VR performance area62.

With reference toFIG. 6, a viewpoint of the audience member80has moved from the viewpoint location shown inFIG. 5to a viewpoint to the right of the stage and zoomed in, as compared with the viewpoint location shown inFIG. 4andFIG. 5. With reference to the overhead view68, the viewpoint position69of the viewer has moved to the lower right of the concert space64to a location in first row of the 3D VR environment. In such case, the synchronized audio data outputted to the audience member70can likewise be moved to correspond to a microphone14within the audience area70of the concert space44that is closest to a seat location corresponding to the purchased seat location within the VR concert space64. As noted above with respect toFIG. 4, whileFIG. 6shows the overhead view68presented on a tablet device as an example to show the overhead view68, the audience member can use any type of audience device22to select the location for the audience member's viewpoint of the performance within the 3D VR environment. As further noted above, both the visual and the audio feed outputted to the particular audience member is updated and adjusted based on selected location for the viewpoint to provide a VR experience that would correspond to the selected location to replicate the visual and audio experience the audience member would have seen and heard if the audience member were physically present within the concert space44at a physical location corresponding to the selected location for the viewpoint.

From any location within the VR concert space64, the audience member80can utilize a binocular feature to zoom in and view a closer 3D VR representation of any focal point within the VR concert space64. For example, an audience member80can use the binocular feature to zoom in on a particular 3D VR avatar60. More specifically, an audience member80can use the binocular feature to zoom in on a particular portion of a particular 3D VR avatar, such as the 3D VR avatar's hands or on the 3D VR avatar's face. In this way, the audience member80can focus and zoom in to watch the 3D VR avatar's hands playing the associated virtual instrument. Additionally or alternatively, the audience member80can focus and zoom in to watch the 3D VR avatar's face and see the facial expressions. In each case, motions of the 3D VR avatar60correspond to those of the corresponding performer40. In other words, the motion of the 3D VR avatar's hands corresponds to the motion of the hands of the corresponding performer40as captured by the associated body movement sensor18. Similarly, the facial expressions of the 3D VR avatar's face correspond to the facial expressions of the corresponding performer40as captured by the associated facial expression sensor16.

While the audience device22can be a VR headset71, as shown inFIG. 5, the audience device22can alternatively be a computing device having a two-dimensional display, such as a tablet, a smartphone, a laptop, a desktop computer, or any other computing device with a two-dimensional display. With reference toFIG. 7, the audience device22can be a handheld tablet90. In the event the audience device22has a two-dimensional display, the VR environment can be collapsed from a 3D environment to a flat two-dimensional (2D) environment. In such case, the VR/AR output module36outputs the synchronized audio/visual VR environment data to the audience device22and the audience device displays the VR environment data in a 2D format. Additionally or alternatively, the audience device22can communicate with the VR/AR output module36to indicate to the VR/AR output module36the type of display utilized by the audience device22. In such case, in the event the audience device22indicates that it includes a 2D display, the VR/AR output module36can output the VR environment data in the 2D format. In this way, the amount of data communicated over a network to the audience device22can be reduced as compared with the amount of VR environment data communicated in a 3D format to the audience device22.

The audience devices22can be configured with appropriate input devices to capture audience feedback data, such as audience audio data and audience video data. For example, with reference toFIG. 1, the audience devices22can include one or more cameras24and a microphone26. The one or more cameras24can, for example, capture video data that includes facial expressions of the associated audience member80. The microphone26can capture audio data that includes audible noises, such as hand claps, whistles, cheering, etc., generated by the associated audience member80.

The audience feedback data from the audience devices22is fed back to the user feedback module38of the VR system controller12. The user feedback module38receives and synchronizes the audience feedback data from each of the audience devices22and outputs the synchronized audience feedback data to the VR/AR output module36.

The VR/AR output module36can output the audience feedback data to the audience devices22and/or performer augmented reality devices28. For example, the VR/AR output module36can output the audience feedback data to the audience devices22so that audience members80can hear the audible reactions of other audience members80watching the live VR broadcast performance and/or look around to see the real time facial expressions of other audience members80watching the live VR broadcast performance.

In addition, the performers40can be wearing performer AR devices28. The performer augmented reality devices28, for example, can include AR glasses, as shown inFIG. 2andFIG. 3. The AR glasses can be configured to display visual data to display the facial expressions of one or more audience members80during the live VR broadcast performance. For example, the VR/AR output module36can output the visual data to a display of the AR glasses such that the visual audience feedback data is displayed as an overlay visible to the performer40. Additionally, AR glasses can include integrated speakers to output audio data. Alternatively, the augmented reality devices28can additionally include separate headphones. In each case, the VR/AR output module36can output the audio audience feedback data to the speakers of the AR glasses or to the separate headphones so that the performers40can hear the audible reactions of the audience members80watching the live VR broadcast performance. Additionally or alternatively, the VR/AR output module36can output the visual audience feedback data to a display physically located within the concert space. For example, the screen can be located toward the front of the performance area42such that the performers40can look up to see the facial expressions of audience members80displayed on the screen, while also listening to the audio data including the audible reactions of the audience members during the live VR broadcast performance. Additionally or alternatively, the audio data including the audible reactions of the audience members could be output with speakers into the performance area42such that the performers can hear the audio audience feedback data. In such case, the microphones14within the performance area42could receive the audio feedback data, along with the audio data of the performers40, so that a combination of the audio audience feedback data with the audio data of the performance by the performers40is captured by the microphones14and then output by the VR/AR output module36to the audience devices22. In this way, audience members80can hear the audible audience reactions as part of the virtual performance. Additionally or alternatively, audio and visual data from other performers40and/or audio and visual data from a conductor of the performers40and/or a producer of the live VR performance can be outputted to the performer AR devices28.

In other features, and in an alternative embodiment, instead of displaying the 3D VR avatars60of the performers40within the VR concert space64, the 3D VR avatars60can be displayed in an AR environment by overlaying the 3D VR avatars60onto image data representing a real-world environment of the audience device22. For example, as shown inFIG. 8, the audience device22is a tablet90that utilizes the camera24of the audience device22to generate image data of the environment of the audience device22. In the example ofFIG. 7, the audience device22generates image data of a living room at a location of the audience device. The audience device22can then display an AR environment100whereby the 3D VR avatars60are overlaid on the image data of the environment such that the 3D VR avatars60appear to be performing in the living room at the location of the audience device, as shown inFIG. 8. In this embodiment, the VR/AR output module36can communicate data representing the 3D VR avatars60along with corresponding synchronized audio data to the audience device22. The audience device can then receive the image data of the environment of the audience device22from the audience device camera24and generate the image of the AR environment by overlaying the 3D VR avatars60onto the image data of the environment of the audience device22.

With reference toFIG. 9, the audience member80can utilize a lasso zoom feature to zoom in and/or enlarge a particular 3D VR avatar60within the AR environment100. For example, the audience member80can utilize a touchscreen of the tablet90to draw a circular outline110around a particular 3D VR avatar60, as shown on the left side112ofFIG. 9. Once the particular 3D VR avatar60is selected with the lasso zoom feature by drawing the circular outline110, the audience device22can reconfigure the display of the AR environment100such that the selected 3D VR avatar60is enlarged to appear larger within the AR environment as compared with the previous display, as shown on the right side114ofFIG. 9.

With reference toFIG. 10, the audience device22can be moved around the environment of the audience device22, (for example, around the living room shown inFIG. 10) to view one or more of the 3D VR avatars60from different perspectives and angles. For example, as shown on the left side116ofFIG. 10, the particular 3D VR avatar60is shown from a first angle with the audience member's perspective being located to the right-hand side of the 3D VR avatar60. As shown on the right side118ofFIG. 10, the audience device22has been move towards the right so that the audience member's perspective is located more towards the center and left-hand side of the 3D VR avatar60.

Additionally or alternatively, with reference toFIG. 11, instead of rotating the audience device22around the environment to view the particular 3D VR avatar60, the particular 3D VR avatar60can be selected and rotated within the AR environment100. For example, on the left side120ofFIG. 11, the particular 3D VR avatar60is shown facing a first direction within the AR environment100. On the right side122ofFIG. 11, the particular 3D VR avatar60has been selected and rotated around so that the particular 3D VR avatar60is facing a second direction rotated from the first direction.

While a single 3D VR avatar60is shown inFIG. 10andFIG. 11, the same functionality can be implemented and performed with multiple 3D VR avatars60within the AR environment100.

In other features, and in another alternative embodiment, instead of displaying the 3D VR avatars60on a computing device having a 2D display, the audience device can instead be a holographic display system whereby the 3D VR avatars60are displayed as holograms by outputting live holographic data within the environment of the audience device. For example, with reference toFIG. 12, a holographic screen150is shown whereby a holographic display device is utilized to display the 3D VR avatars60as holograms via live holographic data on the holographic screen150. As in the preceding embodiments, the audience device22also includes a speaker to output audio data synchronized with the body movement data and facial expression data so that the 3D VR avatars60displayed as holograms on the holographic screen150are shown performing in synchronization with the audio data.

In addition, instead of a holographic screen, the audience device can includes a volumetric display such that the live holographic data is displayed utilizing the volumetric display. The volumetric display, for example, can display the live holographic data utilizing particles suspended in the air of the environment and projecting the live holographic data onto the suspended particles.

With reference toFIG. 13, the interactive virtual reality broadcast systems and methods of the present disclosure can include a backstage mode whereby an audience member80can engage with the 3D VR avatar60of a particular performer40to interact, ask questions, and carry on conversations with the performer. As shown inFIG. 13, an audience member80is viewing a VR environment whereby the 3D VR avatar60of a particular performer40is engaged in a question and answer discussion with a number of virtual audience members160.

With reference toFIG. 14, another example of real-world performers40performing on a performance area42of a concert space44is shown. As with previous examples, the performers40are located at locations separated by a distance46that is larger than a predetermined distance. For example, the predetermined distance can be selected to be sufficiently large to prevent the transmission of airborne pathogens, such as viruses, between the performers40while the performers40are performing at the concert space44. As shown inFIG. 14, two of the performers40are performing in front of microphones with microphone stands190with a predetermined location200between them. With reference toFIG. 15, 3D VR avatars60are shown performing on a VR performance area62within a VR concert space64. Similar to previous examples, an audience member80is viewing the VR performance using an audience device22, such as a VR headset71. In the VR environment shown inFIG. 15, instead of the performers40performing at separate microphone stands, the interactive VR broadcast systems and methods of the present disclosure display the 3D VR avatars60performing and singing at a single microphone stand located in the virtual environment at a location corresponding to the predetermined location200in the real world. In other words, while the real-world performers40are singing and performing at different microphone stands in the real-world performance area42of the concert space44that are socially distanced, in the virtual environment the corresponding 3D VR avatars60are displayed as singing and performing at the same microphone stand at the same location. The 3D VR avatars60could be moved within the virtual environment at particular times, such as when the two performers40sing harmonies together during the performance, to appear as singing from the same microphone stand, while being moved apart during other times of the performance.

In other features, as shown inFIG. 16, the performance area42of the concert space44can include multiple tiers, including a first tier250and a second tier252located above the first tier250. The multiple tiers250,252can allow for additional locations for the performers to be located at locations that are separated by a distance that is larger than the predetermined distance. For example, a first group of performers300can be located on the first tier250while a second group of performers302can be located on the second tier252. All of the performers40located in both groups are located at locations that are separated by a distance from other performers40in the same group and other performers40in the other group that is larger than the predetermined distance.

In addition, in the example ofFIG. 16, the second tier304can be composed of a porous material that allows air to pass through the second tier304. The concert space44can include air handling equipment that include, for example, at least one air duct306that directs air flow, including potentially or partially laminar air flow, from the air handling equipment through the porous floor of the second tier304, past the performers40in the second group of performers302and into an exhaust air duct308. In this way, the laminar air flow around the performers40can provide additional protection against airborne pathogens that may be present in the performance area42. As shown inFIG. 16, the performers40located on the second tier can include performers40playing woodwind and brass instruments that require air to be blown from the performer40.

With reference toFIG. 17, in addition to the air handling equipment shown inFIG. 16, the concert space44can also include an additional air duct320that directs air flow toward the performers40in the first group of performers300and upwards through the porous floor of the second tier304. WhileFIG. 16andFIG. 17illustrate particular examples of the air handling equipment utilizing the at least one air duct306, the at least one additional air duct320, and the at least one exhaust air duct308, any number of intake and exhaust air ducts can be used in any configuration that provides air flow between and around the performers40to prevent the transmission of airborne pathogens between the performers while performing on stage, in accordance with the present disclosure. In this way, the use of porous stage floors with air handling equipment that provides airflow between and around the performers40provides further isolation and separation between the performers40such that any airborne pathogens from a particular performer will be captured by the air flow of the air handling equipment and directed into the at least one exhaust air duct308where the exhaust air can be vented outdoors. The VR system controller12can perform noise cancellation or filtering to remove any hum or noise generated by the air handling equipment from the audio data picked up by the microphones14so that the audio feed of the live VR broadcast does not include any noise from the air handling equipment.

With reference toFIG. 18, an interactive VR broadcast method400in accordance with the present disclosure is shown. The method400can be performed by the VR system controller12, including one or more of the individual modules30,32,34,36, and38of the VR system controller12. While a specific example sequence of steps of the method is shown inFIG. 18, particular steps of the method400can alternatively be performed in a different order and/or in parallel with other steps.

The method400starts at402. At404, the VR system controller12creates a 3D model of the concert hall, i.e., the concert space44, including the performance area42. For example, the VR system controller can receive image data and LIDAR data for the interior of the concert space44. For example, the image data can be generated by a 3D 360° camera that captures image data of the interior of the concert space44from all angles. The LIDAR data can be generated by a LIDAR unit that captures dimensional data, such as X, Y, Z coordinate data, mapping the physical characteristics, surfaces, and features of the interior of the concert space44in a 3D coordinate space. In this way, each physical point or area of the physical concert space44will have (a) associated image data generated by the 3D 360° corresponding to how the physical point or area looks within the concert space44, and (b) associated coordinate data, represented, for example, in an X, Y, Z coordinate space, corresponding to the location of the physical point or area within the concert space44. The VR system controller12can then generate 3D model of the concert hall, i.e., the concert space44, based on the image data and the coordinate data for each physical point or area within the concert space. In other words, the VR system controller12can create a mapping of the image data to the associated coordinate data for each point or area within the concert space44. The 3D 360° camera and the LIDAR unit can be positioned at a central location within the concert space44to capture the image data and the LIDAR data. Alternatively, the 360° camera and the LIDAR unit can capture the image or video data from multiple positions within the interior of the concert space44and the resulting image data and LIDAR data captured from the different positions can be stitched together. At405, the VR system controller12stores the generated 3D VR model of the concert space44, including the image data and the associated coordinate data for each point or area within the concert space44, in the data storage13.

At406, the VR system controller12creates 3D VR avatars of the performers40. For example, prior to any VR performance, each performance can undergo a 3D VR avatar creation process to generate a 3D VR avatar corresponding to the performer. For example, the VR system controller12can receive image data of the performer40from one or more cameras and/or LIDAR units and can generate a corresponding 3D VR avatar60, based on the image data and/or the LIDAR data, that represents the performer40. Alternatively, third party specialty companies can be used to generate the 3D VR avatars of the performers. For example, the Black Hangar Studios based company Metapixel, located at Lasham Airfield, Lasham, Alton Hampshire, GU34 5SS, United Kingdom, is a specialty company that utilizes150high resolution portable full-body scanners to capture image data of a person and create a corresponding 3D VR avatar of the person. Specifically, the company utilizes control software to enable a capture rate of 5 shots per second at 36 megapixel RAW. The company can produce a normal resolution full body scan at a rate of one every ten minutes, per station, or one every 1.5 hours at higher resolution. In this way, the 3D VR avatar of the performer will look nearly identical to the actual performer, with the same body type, face shape, eye color, hair color, hair style, etc. Additionally, or alternatively, a camera can be used to capture an image of the performer's face. The system can then use the image data of the performer's face on a default or generic avatar, which can then be edited to correspond to the actual performer's body by, for example, receiving data input indicating whether the performer is taller or shorter than the default avatar and whether the performer is skinnier or wider than the default avatar. At407, the VR system controller12stores the 3D VR avatars of the performers in the data storage13.

At408, the VR system controller12generates 3D VR instruments. The 3D VR instruments can be generated with image data from cameras and/or LIDAR data from LIDAR units, similar to the 3D VR avatars generated for the performers, as discussed above. In this case, however, a standard style instruments, such as a standard style violin, can be scanned with the cameras and/or the LIDAR unit and a standard 3D VR violin can be generated based on the captured image data. That same standard 3D VR violin can then be used for any performer that utilizes a standard style violin. If a specialty or non-traditionally shaped instrument is to be used by a performer, the instrument could be scanned and a non-traditional 3D VR instrument could be generated based on the resulting image and/or LIDAR data. In addition, multiple 3D VR instruments having different types or shapes could be generated. At409, the VR system controller12stores the 3D VR instruments in the instrument library in the data storage13.

At410, the VR system controller12generates an initial rendering of the 3D VR avatars60placed precisely within the VR concert space64holding their corresponding 3D VR instruments. Once the performance by the performers begins, at411, the VR system controller12receives facial expression data from the facial expression sensors, as described above. At412, the VR system controller12receives body movement data412from the body movement sensors18, as described above. At414, the VR system controller414and, more specifically, the visual input synchronization module30of the VR system controller, synchronizes the facial expression data with the body movement data and manipulates the 3D VR avatars60with real-time measurements from the facial expression sensors16and the body movement sensors18to generate visual VR environment data.

At416, the VR system controller receives audio data from the one or more microphones14. At418, the VR system controller and, more specifically, the audio/visual synchronization module34of the VR system controller synchronizes the audio data from the one or more microphones14with the visual VR environment data, as discussed above. For example, the VR system controller can utilize time markers embedded within the audio data and within the visual VR environment data, for synchronization to generate synchronized audio and video data. At420, the VR system controller12generates the 3D VR environment including 3D VR environment data that includes the synchronized audio/visual data showing the 3D VR avatars60playing their 3D VR instruments within the 3D VR concert space44in synchronization with the audio data so that the 3D VR performers visually appear to be playing the music of the audio data in the 3D VR environment.

At422, the VR system controller12receives user input data from the audience devices22for controlling, for example, a viewpoint location and position within the 3D VR environment and, at424, the VR system controller12outputs the audio/visual VR data to the audience devices22. In this way, the VR system controller12can transmit only the data needed for the particular audience device22to display the visual data of the VR environment from that particular viewpoint location within the 3D VR environment. Alternatively, the VR system controller12can transmit the entire 3D VR environment, including all of the VR visual data for the entire VR environment, to the audience devices22so that the audience devices22can then receive the user input for controlling the viewpoint location and position within the 3D VR environment. Additionally, the VR system controller12can initially transmit the VR data for the VR concert space64and VR performance area62, along with data for representing the 3D VR avatars60to the audience devices22. The VR system controller12can then subsequently transmit a stream of data to the audience devices22that includes the real-time audio data and the real-time body movement and facial expression for the 3D VR avatars60. In such case, the audience devices22can initially construct and display the VR environment with the 3D VR avatars60and can then update the VR environment based on the real-time body movement and facial expression data for the 3D VR avatars60while outputting synchronized audio data and while displaying the VR environment from the viewing location and position indicated by the user input received from the associated audience member. In other words, a central processing unit (CPU) and/or a graphics processing unit (GPU) of the audience device22can process and manipulate the VR environment based on input received by the audience device22to present the VR environment from a viewpoint position or location and/or zoom level indicated by user input received by the audience device22. In this way, user input received by the audience device22indicating a viewpoint position or location need not be transmitted back to the VR system controller12. Rather, the audience device22can utilize the data received from the VR system controller, including the VR data for the VR concert space64and VR performance area62, the data representing the 3D VR avatars60, the real-time audio data, and the real-time body movement and facial expression data for the 3D VR avatars60, to manipulate and present the VR environment to the user via the visual and audio output devices of the audience devices22based on the viewpoint position or location and/or zoom level indicated by the user input received by the audience device. Additionally or alternatively the VR system controller12can utilize a separate VR distribution server that is in communication with the VR system controller12and the VR audience devices22. The VR distribution server, for example, can receive the VR environment data from the VR system controller12and can distribute the VR environment data to the audience devices22, as needed.

After outputting the audio/visual VR data to audience devices22, the VR system controller12then loops back to410and receives additional facial expression data410and additional body movement data at412, etc. In this way, the VR system controller12continues to execute the method400until the VR performance has concluded and/or the VR broadcast system10is shut down.

With reference toFIG. 20, a method500for generating AR data and VR data based on audience member feedback in accordance with the present disclosure is shown. The method500can be performed by the VR system controller12, including one or more of the individual modules30,32,34,36, and38of the VR system controller12. While a specific example sequence of steps of the method is shown inFIG. 20, particular steps of the method500can alternatively be performed in a different order and/or in parallel with other steps.

The method starts at502. At504, the VR system controller12receives audience member feedback from audience devices22. For example, the audience member feedback can include audio data including audible reactions, such as cheering, whistling, etc., of the audience members during the VR performance captured by microphones26of the audience devices22. Additionally, the audience member feedback can include visual data including facial expressions of the audience members captured by cameras24of the audience's devices22.

At505, the VR system controller12evaluates the audience feedback and filters out any audience feedback deemed to be inappropriate, including, for example, any abusive or vulgar language. The VR system controller12can utilize speech recognition algorithms, for example, to recognize any audible words in the audience feedback data. The recognized audible words can then be checked against a list of inappropriate words or phrases. Any words or phrases deemed in appropriate can be excised and blocked from the audio data. In the event a particular audience microphone receives inappropriate audience feedback more than a predetermined threshold number of times, the audience feedback from that particular audience microphone can be blocked for the remainder of the performance.

At506, the VR system controller12generates AR data and/or VR data based on the audience member feedback data. At508, the VR system controller12outputs AR data based on the audience member feedback data to the performer AR devices28. At510, the VR system controller12outputs VR data to audience members via the audience devices22. The AR and VR data can be outputted by the VR/AR output module36of the VR system controller12to the performer AR devices28and to the audience devices22. For example, as discussed above, the VR/AR output module36can output the audience feedback data to the audience devices22so that audience members80can hear the audible reactions of other audience members80watching the live VR broadcast performance and/or look around to see the real time facial expressions of other audience members80watching the live VR broadcast performance. For further example, as discussed above, the performers40can be wearing performer AR devices28, such as the AR glasses shown inFIG. 2andFIG. 3. The AR glasses can display visual data to display the facial expressions of one or more audience members80during the live VR broadcast performance. Additionally, AR glasses can include integrated speakers to output audio data based on the audience member feedback data. Alternatively, the augmented reality devices28can additionally include separate headphones. In each case, the VR/AR output module36can output the audio audience feedback data to the speakers of the AR glasses or to the separate headphones so that the performers40can hear the audible reactions of the audience members80watching the live VR broadcast performance. Additionally or alternatively, the VR/AR output module36can output the visual audience feedback data to a display physically located within the concert space44. For example, the screen could be located toward a front of the performance area42such that the performers40could look up to see the facial expressions of audience members80displayed on the screen, while also listening to the audio data including the audible reactions of the audience members during the live VR broadcast performance.

In addition to the performer AR devices28and the audience devices22, the VR broadcast system10can also include a separate conductor device to capture the movements, instructions, and commands of a conductor of the performers40. For example, a video camera and a microphone can be used to capture the movements, instructions, and commands of the conductor and the resulting video and audio data can be transmitted to the performer AR devices28so that the performer AR devices28can view and follow the movements, instructions, and commands of the conductor.

In this way, the interactive VR broadcast systems and methods of the present disclosure allow performers to be socially distanced on stage, while generating and transmitting a VR environment, including avatars of the performers synced with the music being played by the performers, to audience members using virtual reality devices. In addition, the interactive VR broadcast systems and methods of the present disclosure receive and transmit audience member feedback, such as audience member video and audio data, to AR devices of the performers so the performers can receive live audience feedback during their performance and to the VR devices of other audience members. In this way, the interactive VR broadcast systems and methods of the present disclosure are able to generate similar levels of interaction and engagement that would previously have been enjoyed by musicians and audience members during traditional live musical performances, while maintaining social distance between all participants which prevents the transmission of airborne pathogens. In addition, the interactive VR broadcast systems and methods of the present disclosure can facilitate the broadcast of live performances to a geographically dispersed audience, including to any areas of the world that have a sufficient data connection to connect to and stream VR data from the VR broadcast system. Additionally, the interactive VR broadcast systems and methods of the present disclosure can facilitate the broadcast of live performances to audience members who are not able to leave their location and travel to the physical concert hall where the performance is located.

Spatial and functional relationships between elements (for example, between modules) are described using various terms, including “connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship encompasses a direct relationship where no other intervening elements are present between the first and second elements, and also an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.

As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR. For example, the phrase at least one of A, B, and C should be construed to include any one of: (i) A alone; (ii) B alone; (iii) C alone; (iv) A and B together; (v) A and C together; (vi) B and C together; (vii) A, B, and C together. The phrase at least one of A, B, and C should not be construed to mean “at least one of A, at least one of B, and at least one of C.”

The module or controller may include one or more interface circuits. In some examples, the interface circuit(s) may implement wired or wireless interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2016 (also known as the WIFI wireless networking standard) and IEEE Standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of a WPAN are IEEE Standard 802.15.4 (including the ZIGBEE standard from the ZigBee Alliance) and, from the Bluetooth Special Interest Group (SIG), the BLUETOOTH wireless networking standard (including Core Specification versions 3.0, 4.0, 4.1, 4.2, 5.0, and 5.1 from the Bluetooth SIG).