Patent ID: 12262145

DETAILED DESCRIPTION

FIG.1is a logical diagram of a distributed audio system100that enables a remote studio110to interface with a venue system121. Generally, a mixer provides audio data from the remote studio100for a remote participant105and routes some instantiation of that audio data to the venue system121in order to create some audio presence of the remote participant105in the venue.

The remote studio110may include a remote participant105that desires to participate in a venue event in a more immersive manner. As shown, the remote participant105is using a ticketing application on their wireless device115to connect provide information to systems support the venue. As shown, the remote studio110optionally includes a video camera118, and a display120. The video camera118can be used to provide video imagery into the venue, and the display120can be used to render the broadcast of the event (e.g., sporting event, concert, a speech, a play, etc.). The video camera118may additionally or alternatively be used to capture the orientation and/or motion of the participant105. For example, the video camera118may be a motion capture camera (e.g., having multiple lenses, an array of lenses, multiple separate camera components, etc.). As shown, the wireless device115interfaces with a home router116, which provides wireless and wired connectivity for the home. The home router116may be a 5G capable device. That is, the home router116may be able to, for example, connect to the internet through a 5G mobile network.

As shown, the display120also interfaces with the home router116, which may interface through one or more intermediary systems (e.g., a set top box that is not shown) in order to provide the video rendering of the event. In one example, the wireless device115may separately provide audio and/or a virtual experience to provide an enhanced experience. For example, the wireless device115may provide local audio for the remote participant105, which is then processed and routed into the venue. The camera118may also be integrated into the wireless device115. That is, for example, a front-facing camera of the wireless device115may be used to capture video data of the remote participant in order to provide video imagery into the venue. Similarly, a display of the wireless device115can be used in place of (or in addition to) the display120. The display of the wireless device115can be used to, for example, render the broadcast of the event (e.g., sporting event, concert, etc.).

The wireless device115may be a mobile device, such as a smart phone, a laptop computer, a tablet, a virtual reality headset, a virtual reality headset with an integrated smart phone, etc. The wireless device115may be a 5G capable device, e.g., having a 5G capable integrated antenna array and modem and that is able to connect to a 5G mobile network. Additionally or alternatively, the wireless device115may be capable of taking advantage of a 5G mobile network (e.g., obtain some of the latency benefits provided by a 5G mobile network) indirectly using the home router116in cases where the home router116is a 5G capable device that can connect to a 5G mobile network.

In another configuration, the wireless device115works with an A/V receiver, local computer, audio support system (e.g., a “sound bar”), to provide an enhanced experience. For example, a local computer may use microphones embedded in the wireless device115, display120, and sound bar in order to provide a superior audio signal that better isolates and filters the source audio of the remote participant105. For example, a receiver may implement one or more DSPs (Digital Signal Processors) in order to filter out playback of the venue noise being rendered by a sound bar. Thus, the audio data sent to the mixer would represent a superior quality audio source suitable for mixing and rendering in the venue.

The remote studio110interfaces with network125through a mixer (not shown). The network125may be a cellular network, such as a 5G mobile network. Once audio data from the user has been processed, a venue signal can be played into the venue121. A shown, the venue features a basketball venue. Behind the scorer's table, there are three audio systems,130,135, and140, that are used to provide some instantiation of the audio data from the remote participant105. In one configuration, each of audio systems130,135, and140represents audio unique to one remote participant. For example, each of the audio systems130,135, and140correspond to a particular seat in the venue. That is, each of the audio systems130,135, and140may be arranged in or in the vicinity of a particular seat of the venue. In other configuration, the audio for audio system130may represent audio data from a logical arrangement of fans assigned to particular seats (or section). The mixer may attenuate and combine audio data from different users and send a combined signal to one or more of the speaker systems.

In some configurations, the audio systems cannot be located in the exact location associated with a user's assigned seat. Notwithstanding these differences, the signal may be transformed using, for example, a Fast Fourier Transform, that accounts for the actual location(s) of one or more audio systems used to render the audio signal in the venue. Thus, athletes may perceive audio for a particular user as originating from the second rote midcourt even though the speaker is located closer to the baseline.

The venue may include anyone of various types of locations or properties where an event may be hosted. For example, the venue may be a concert hall, a stadium, an arena, a pavilion, a conference center, a community center, a club, a bar, a restaurant, an art gallery, etc. Similarly, the events that the participant105may reserve a virtual seat for can include a variety of different events (e.g., performances), such as sporting events, gaming events such as card games, concerts, plays, operas, speeches such as political speeches, charity events, magic shows, talent shows, etc.

FIG.2is an example user interface200showing a virtual seat application. The UI200may be presented on the wireless device115shown inFIG.1. The UI200includes an event identifier210, a fan preference identifier220, a virtual seat identifier230, an audio control240, and a seating arrangement250. Generally, the virtual seat application is an application resident on a computer application, such as a wireless phone, that transports remote information (e.g., remote audio) to a venue using a mixer. The virtual seat application may require a registration and login process to ensure that a user is authenticated, and to manage the allocation of a limited number of virtual seats. For example, it may result in a bad experience or reproduction if too many users are allowed to virtually occupy the same virtual seat. A remote participant will want to know that their voice (and antics) may be uniquely identifiable for the participants. Thus, a system administrator may limit the density of fans and/or the number of active participants for a given region or location.

The event identifier210allows the user to select from one of several events (e.g., sports games, concerts, speeches, charity events, etc.). A system administrator may limit the remote participant to those that are licensed ticket holders, or the system administrator may allow others to purchase a virtual set for an event. For example, a system administrator may be trying to maintain social distancing guidelines and limit the virtual ticket applications to those that were otherwise set to attend the event. In some cases, a system administrator may allow family members, or a social network of a designated number of individuals to have a threshold amount of “audio presence” in a designated geographically-limited “virtual” area of the venue, such as if five people were occupying the space of three seats.

The fan preference identifier220may allow a user to declare their team preference. With their team preference, a broadcaster or the team itself may favor certain audio feeds while discouraging or diminishing the impact of adverse fans.

The virtual seat identifier230allows the user to perceive their virtual presence in the venue. This ability to secure a virtual seat identifier may secure a stronger presence as it relates to any audio data being broadcast from that location. For example, the user with a closer seat may receive a higher volume setting in the venue signal than those that are more distant from the floor (or area of interest). The virtual seat identifier also may be used to identify which speaker systems are used to broadcast the user's event (e.g., game as shown).

Audio control240is used to provide a degree of control as to whether their audio is broadcast. For example, a user may elect to conduct other business during the event (e.g., during the game) and not wish to share audio from these other conversations (e.g., a parent dealing with disciplinary issues). Similarly, the user may selectively allow whether they want to hear their neighbors, i.e., audio from other remote participants that are siting proximate to the remote participant in question. Some neighbors can be quite entertaining while others may be more annoying. The ability to block or attenuate this volume may allow the user to carve out their own experience.

Seating arrangement250may allow a user to design their own audio (and video) experience. For example, as shown the user is sitting in a second row behind three celebrities. The user is sitting between two friends, John Smith, and Paul Ross. In addition, two brothers and cousins are sitting behind the user. Seating arrangement may feature drag-and-drop functionality so that a remote participant (e.g., a user) can readily control which audio is locally played in the user's room. The arrangement of users also may be used to specify a directional control to reflect the virtual seat assignment. That is, users sitting to the left are perceived as speaking from the left hand side.

FIG.3is an example diagram300showing multiple seat configurations for a distributed audio system. The diagram300includes a first seat configuration310and a second seat configuration320. In configuration310, every seat has an audio system. For convenience of attending fans, (or to better mimic actual virtual fans), each virtual seat application may be linked to a particular audio system. The audio systems may represent a powered speaker temporarily moved into an assigned seat. Alternatively, each seat may include an audio system with appropriate safety settings. During some events, the audio systems can be used to provide guest amenities (e.g., a concierge service). When the seat is not occupied during an event, the power settings may be elevated so that a remote participant can enthusiastically support their team. Configuration320illustrates a setting where some seats have their own audio system, while other seats are forced to share an audio system. The mixer may modify the venue signal so that the audio is perceived as originating from one of the empty seats.

FIG.4is an example user interface400that illustrates how a user may adjust their virtual experience. For example, through the UI400a user may modify and adjust their social network in order to create their desired event experience (e.g., game day experience). The UI400may be displayed on the wireless device115shown inFIG.1. As shown, the user first identifies the event (e.g., game), the seat, and the price of attending the event. The user has elected to participate alongside an identified media profile, which in this case is CELEBRITY1. As a result of this option, the user may be allowed to hear commentary and enthusiasm from the identified celebrity. In one setting, the user is limited to listen-only mode so that the user cannot interfere with the celebrity's enjoyment of the event. In another, perhaps premium setting, the user is afforded a capability to interact with the celebrity by allowing the user to ask questions (or share questions via chat). This may be a premium feature for those users that paid for this capability.

The user has assembled a network of friends that are invited to participate in the event. In one configuration, the user purchases the right for friends to participate with the user. In another configuration, the user designating friends merely invites the other users to purchase a virtual seat alongside the inviting user.

For example, the user may allow their audio to be sourced anonymously. In another setting, the user allows their social media profile to be shared. In still another setting, the user may be alerted when their participation in the virtual rendering of their event is recognized. In this manner, users can build their own following through enthusiasm and insight while also sharing the experience with friends.

FIG.5is an example user interface500for an administrator of a distributed audio system. As shown the user (e.g., an administrator) can identify through the UI500which audio systems are used at which locations. Power settings can be specified, monitored, and managed so that the venue experience remains safe and/or enjoyable for participants. Thus, a user exerting high volume settings at home may be individually limited from raising the venue volume to unsafe levels. The power settings may be set at a specified decibel level and/or watt setting.

The administrator also is configured to selectively invoke spacing for underlying speakers. For example, the administrator may have access to speakers at designated distances (e.g., 1M, 3M, 5M, 10M, common speakers, and existing speaker systems). A 1M spacing may ensure that a dense configuration of speakers is employed to broadcast audio from remote participants. Common audio settings may transition to an expert or self-diagnostic setting that does not require an administrator to have a detailed understanding of the speaker system topology.

The administrator may limit the number of participants that are allowed to participate, that is, provide audio to the venue. The administrator may specify the number, which in turn propagates a listing of which seats are available. If the administrator reduces the number of seats from 5,000 virtual participants to 2,500 participants, a master ticketing application may eliminate every other seat, or eliminate 2,500 of the most distant seats.

The administrator also provides visibility into which celebrities are active and also identify which “casual” or non-celebrity fans are active. The user then can select a “showcase” button to commit more prominence for the showcased profile. For example, if the administrator selects the showcase button for CELEBRITY1, the audio for the announcers may be subdued while the audio levels for the CELEBRITY may be increased. Similarly, the administrator may elect to showcase active fans during breaks in the action through this same control. “Top” designation may be determined by the number of followers during particular event, the number of endorsements of the commentary, the number of times at which the mixer determines that the remote participant is active and/or the ambient volume level for the remote participant due to the remote participant's own cheering.

FIG.6is a diagram that illustrates an example mixer600having multiple sources. Mixer600includes audio data for a remote participants610. Generally, these are users using a virtual ticket selector application that have opted to route their audio into the venue. Ambient audio source620provides the ambient audio for the venue. This may include audio from microphones located throughout a venue. Professional audio/video source630may include audio and/or visual material that represents video and commentary that represents the primary broadcast materials.

Within mixer600, computational mixer640algorithmically mixes the different sources of data to provide the desired experience. For example, the computational mixer640may determine that 70% of the existing audio should remain unchanged. 30% of the audio should be added from the remote audio participants that are within three seats of the user's selected seat. The seat directly next to the identified seat may receive a weighting (mix) of 15), while those seats that are within two seats may receive a weighing or mix of (10), and those that are three seats away receive a mix of 5. This may include power allocations of 15%, 10%, and 5%. This can then be routed to the audio system of the remote participant in question.

As shown by multiple lines of communication, a venue signal is routed to the venue audio/video system650so that it can be rendered in the venue. In one configuration, the mixer combines several different audio sources at equal levels and routes them to a designated audio system in the venue audio system. In another configuration, a time-based scheme is used so that different users receive prioritized rendering. This may increase the intelligibility of the underlying audio in the venue so that users are not contending with one another to be understood. For example, the computational mixer640may use one serve process to integrate 10 sources for one speaker. Nine of the sources may be combined for 15% of the audio energy. The computational mixer may alternate between the 10 audio sources (or alternate between those active audio sources) to provide 85% of the available audio energy to that prioritized audio resource. Users may receive an alert in their virtual ticket application to perceive when their audio data is receiving prioritization.

Generally, the computational mixer640may be implemented on an array or collection of one or more processors configured to receive a stream of audio data from remote participants (e.g., TCP/IP audio data). The computational mixer640may provide each audio stream a threshold commitment computational processing. As the number of streams grows larger, the computational mixer640may spawn additional processes and/or access additional processes in order to ensure that the audio data is being processed in an appropriate manner.

The computational mixer640may include an audio mixer and/or a video mixer. Accordingly, in addition to receiving a stream of audio data from remote participants (e.g., TCP/IP audio data), the computational mixer640may optionally receive a stream of video data from the remote participants (e.g., from a front facing camera of the wireless device115shown inFIG.1). The computational mixer640may also receive other data, such as sensor data indicating motion of and/or an orientation of the participants (e.g., looking up, looking down, looking left, looking right, standing up, sitting down, hands up, hands down, thumbs up, etc.). The sensor data may be obtained from one more or motion capture devices, such as a motion capture camera (e.g., the camera118) in the studio110, accelerometers in the wireless device115, one or more IMUs in the wireless device115, a front facing camera of the wireless device115, etc.

The computational mixer640may be able to manipulate audio and/or video data in real-time or substantially real-time (e.g., near real-time). For example, the computational mixer640may be able to manipulate audio and/or video data in real-time or substantially real-time to provide a virtual reality or augmented reality to event participants. That is, the computational mixer640may be able to, for example, synchronize live video of various remote participants and live video of the event (e.g., obtained from a professional stream of the event, or from a camera setup at or near a particular seat of the venue that the corresponding participant has virtually reserved) with a real-time or substantially real-time video game engine to provide the virtual reality or augmented reality. Specifically, the computational mixer640may interface audio transmissions of the participants, video transmissions of the participants, and/or other audio and/or video transmissions (e.g., video and/or audio data of the event such as the professional audio/visual source630and/or the ambient audio source620) with virtual reality graphics from a game engine, and proceed to composite the transmissions together in real-time or substantially real-time. The resulting composite may then be fed to the participants.

The composite generated may be particular to a specific participant, or a specific group of participants (e.g., those in the same section, part of the same group, etc.). For example, the composite generated may include a composite video stream that includes a model for the participant and models for neighboring participants (e.g., participants that have reserved virtual seats that neighbor the participant's virtual seat).

In providing the virtual reality or augmented reality environment, the computational mixer640may generate 3D models for various participants and/or objects in real-time or substantially real-time. That is, the computational mixer640may generate a model for a particular participant on the fly, e.g., using one or more video transmissions from the participant. Alternatively, the computational mixer640may update existing 3D models for various participants and/or objects in real-time or substantially real-time. For example, the computational mixer640may update the facial expressions for an existing model for a particular participant on the fly, e.g., using a video stream from a front-facing camera of the wireless device115. The computational mixer640may also update the motions and/or orientations of the model, e.g., using sensor data from one or more motion capture devices. The existing model for the participant105may be generated, for example, by the venue audio/video system650and/or by the computational mixer640using an obtained image of the participant105(e.g., from a social network that the participant105belongs to, from a profile image of the participant105that they provided when registering to use the virtual seat application shown inFIG.2,FIG.4, andFIG.11).

In providing the virtual reality or augmented reality environment, the computational mixer640may align the various 3D models with the environment. For example, the computational mixer640may align each model of each participant with a particular seat in the venue.

In some cases, the computational mixer640may be part of cloud computing system and/or controlled through a cloud computing system.

In some cases, the computational mixer640is part of the venue audio/video system650.

As an example, the computational mixer640may mix a section of virtual seats to provide 20% of the venue signal that will be sent to a designated speaker. The most active remote participant then may be allocated 80% of the venue signal. Designation as the most active remote participant may be identified based on the most frequent audio transmission level detected on a microphone, for example, or based on greatest detected volume.

In some cases, the computational mixer640generates multiple different venue signals, e.g., for the particular speaker or group of speakers that are meant to output the corresponding signal. For example, the computational mixer640may mix the audio/video data differently for the particular seat and/or section of seats (e.g., based on the location of a user's their virtual seat in the venue, based on the location of a user's virtual seat with respect to other users, based on the user paying to hear or communicate with celebrities, etc.). Accordingly, the computational mixer640may generate multiple, different venue signals that are streamed/transmitted to the audio/video system650.

In response to receiving the venue signal, the venue audio/video system650can output the venue signal over one or more speakers in the venue. For example, the audio/video system650can output the venue signal over one or more of the multiple seat speakers shown inFIG.3that each correspond to a particular seat in the venue. The venue signal output by the computational mixer640may be streamed/transmitted over a 5G mobile network.

In some cases, the venue signal is transmitted to and/or accessed by the wireless device115. For example, the venue signal can be received at the wireless device115over a 5G network. The wireless device115may then output video data using a display of the device, and/or audio data using a speaker of the device (or an audio device, such as a speaker or headphones, connected to the device over a wired connection such as an auxiliary port, a Bluetooth connection, etc.).

In some cases, the audio/video system650outputs a signal (e.g., an audio/video signal). This signal may be transmitted to and/or accessed by the wireless device115.

FIG.7is a flow chart of the process by which a system enables an audio system to provide audio from distributed sources into a venue. Initially, the system establishes a link that includes audio data between a remote participant and an audio mixer (710). This may include, for example, establishing a TCP/IP link (or similar network protocol link) with audio data from a wireless device and a server being used to mix data for the user's home. Similarly, this may include, for example establishing a low latency network protocol link with audio data and/or video data from a wireless device and a server used to mix data for the user's devices (e.g., virtual reality headset). The system references a profile for the remote participant (720). The profile may include a user identity, events that the user has registered for (e.g., reserved a virtual seat for), a virtual seat identifier and/or an indication of how the audio data and/or video data should be modified or mixed before sending on as a venue signal.

Based on the profile for the remote participant and using the audio mixer, a venue signal is generated that is related to the audio data (730). For example, the audio mixer may mix a section of virtual seats to provide 20% of the venue signal that will be sent to a designated speaker. The most active remote participant then may be allocated 80% of the venue signal. Designation as the most active remote participant may be identified based on the most frequent audio transmission level detected on a microphone, for example, or based on greatest detected volume.

In some cases multiple different venue signals are generated by the computational mixer640, e.g., for the particular speaker or group of speakers that are meant to output the venue signal. For example, the computational mixer640may mix the audio/video data differently for the particular seat and/or section of seats (e.g., based on the location of a user's virtual seat in the venue, based on the location of a user's virtual seat with respect to other users, based on the user paying to hear or communicate with celebrities, etc.). Accordingly, the mixer640may generate multiple, different venue signals that are streamed/transmitted to the audio/video system650.

A venue sound system to access the venue signal (740). For example, the generated venue signal can be transmitted to the venue sounds system, such as the audio/video system650shown inFIG.6. In response to receiving the venue signal, the venue audio/video system650can output the venue signal over one or more speakers in the venue. For example, the audio/video system650can output the venue signal over one or more of the multiple seat speakers shown inFIG.3that each correspond to a particular seat in the venue. The venue signal output by the computational mixer640may be streamed/transmitted over a 5G mobile network.

In some cases, the venue signal is transmitted to and/or accessed by a user device. For example, the venue signal can be received at a user device. The user device may then output a video data using a display of the device, and/or audio data using a speaker of the device (or an audio device, such as a speaker or headphones, connected to the device over a wired connection such as an auxiliary port, a Bluetooth connection, etc.). The user device may receive/access the venue signal over a 5G mobile network, which can provide low latency and other transmission benefits that would allow the user to seamlessly enjoy the virtual experience (e.g., watch the event, interact with virtual seat neighbors, hear celebrities, interact with celebrities, etc.) in substantially real-time with no or substantially no perceivable lag (e.g., a latency less than 50 ms).

FIG.8is a diagram showing an example venue configuration800illustrating how a mixer may be used to provide a visual rendering of the remote participants at an event. For example, as shown, the configuration800illustrates how a mixer (e.g., the computational mixer640shown inFIG.6) may be used to provide a visual rending of the remote participants behind a scorer's table810in a basketball game. As noted above with respect toFIG.1, a remote participant may have the ability to source video data from a camera located on a wireless device (e.g., the wireless device115), embedded in a display or laptop, a web camera, or perhaps in someone household appliance (e.g., a Portal or other intelligent appliance).

The venue configuration800reveals displays820and830designed to provide the players with a visual representation of the remote participation. Each of the displays is associated with an audio system. Together, a visual and audio wall of participation may be rendered that enhances the player experience.

As shown, display820features John Smith and Paul Ross. Display830features CELEBRITY1 and the user's brother. The seating configurations shown previously may be configured to drive a multi-unit display to present richer visual imagery to the players.

For example, the user may allow their audio to be sourced anonymously. In another setting, the user allows their social media profile to be shared. In still another setting, the user may be alerted when their participation in the virtual rendering of their event is recognized. In this manner, users can build their own following through enthusiasm and insight while also sharing the experience with friends.

FIGS.9A-9Care example diagrams illustrating a distributed audio/video system900. The distributed audio/video system900may include a venue audio/video system950and a 5G device915which can communicate over a 5G network925either directly (e.g., using a 5G capable integrated antenna array and modem) or indirectly (e.g., through the router116that may be a 5G capable device that can connect to the internet using the 5G network925). The 5G network925may be a 5G cellular network. The distributed audio/video system900may further include a computational mixer, such as the computational mixer640, to combine the audio and/or video data from multiple sources, including audio and/or video data from multiple participants that have reserved virtual seats in the venue, to generate one or more venue output signals. These one or more venue output signals may be fed back to the participants, and may include multiple different signals that are fed to different participants depending on their specific virtual seat, the section of a venue where the virtual seat is located, and/or the participant's audio settings or added features for the event (e.g., receiving audio data of coaches, celebrities, performers, etc.). The audio and/or video data fed back to the participants may be used to generate an augmented reality on an interface of the 5G device915that the participants can interact with to, for example, provide an in-person like experience during the event.

In some cases, the venue audio/video system950is the venue system121shown inFIG.1. Similarly, the venue audio/video system950may be the venue audio/video system650shown inFIG.6.

The 5G device915may be a mobile device, such as a smart phone, a laptop computer, a tablet, a virtual reality headset, etc. The 5G device915may be a 5G capable device, e.g., having a 5G capable integrated antenna array and modem and that is able to connect to the 5G mobile network925. Additionally or alternatively, the 5G device915may be capable of taking advantage of a 5G mobile network (e.g., obtain some of the latency benefits provided by a 5G mobile network) indirectly using the home router116in cases where the home router116is a 5G capable device that can connect to the 5G mobile network925.

In some cases, the 5G device915is the wireless device115shown inFIG.1.

FIG.9Ashows an example logical diagram of the distributed audio/video system900A. As illustrated, the 5G device915may be able to communicate with the venue audio/video system950over the 5G network925directly or indirectly through the router116. In using the 5G network925to communicate with the venue audio/video system950, low latency (e.g., less than 50 ms, less than 30 ms, less than 20 ms, less than 10 ms, less than 4 ms, less than 2 ms, or less than or equal to 1 ms) or very low latency (e.g., less than 10 ms, less than 4 ms, less than 2 ms, or less than or equal to 1 ms) and other transmission benefits can be achieved that would allow the participant105to seamlessly enjoy the virtual experience (e.g., watch the event, interact with virtual seat neighbors, hear celebrities, interact with celebrities, etc.) in substantially real-time (e.g., near real-time) with no, or substantially no, perceivable lag. Moreover, in leveraging the transmission speeds of the 5G network other benefits can be afforded. For example, the distributed audio/video system900A can take advantage of the 5G network925in order to generate and/or update a virtual reality or augmented reality on the fly with no, or substantially no, perceivable lag.

As an example, the venue audio/video system950may include a computational mixer (e.g., the computational mixer640shown inFIG.6). Using the computational mixer (e.g., which may include an audio mixer and/or a video mixer), the venue audio/video system950can generate a composite using multiple data transmissions from remote participants, such as video transmission from the camera118and/or the wireless device115, audio transmission from a the wireless device115, IMU transmission from the wireless device115, motion capture data from the camera118, etc. Specifically, the venue audio/video system950may receive a stream of audio data and/or video data from the remote participants. The venue audio/video system950may also receive other data, such as sensor data indicating motion of and/or an orientation of the remote participants (e.g., sensor data that indicates whether the remote participants are looking up, looking down, looking left, looking right, standing up, sitting down, have their hands up, have their hands down, have their thumbs up, have their thumbs down, etc.). The sensor data may be obtained from one more or motion capture devices, such as a motion capture camera (e.g., the camera118) in the studio110, accelerometers in the 5G device915, one or more IMUs in the 5G device915, a front or back facing camera of the 5G device915, etc.

The venue audio/video system950may be able to manipulate audio and/or video data in real-time or substantially real-time (e.g., near real-time), e.g., using the computational mixer. For example, the venue audio/video system950may be able to manipulate audio and/or video data in real-time or substantially real-time to provide a virtual reality or augmented reality to remote participants. Specifically, the venue audio/video system950may, for example, synchronize live video of various remote participants and live video of the event (e.g., obtained from a professional stream of the event, or from a camera setup at or near a particular seat of the venue that the corresponding participant has virtually reserved) with a real-time or substantially real-time video game engine to provide the virtual reality or augmented reality. In doing this, the venue audio/video system950(e.g., a computational mixer of the venue audio/video system950) may interface audio transmissions of the participants, video transmissions of the participants, and/or other audio and/or video transmissions (e.g., video and/or audio data of the event, and/or the ambient audio generated by output of speakers placed in the venue such as by the speakers that are placed in the seats of the venue as demonstrated inFIG.3) with virtual reality graphics from a game engine, and proceed to composite the transmissions together in real-time or substantially real-time. The resulting composite may then be fed to the participants.

The composite generated may be particular to a specific participant, or a specific group of participants (e.g., those in the same section, part of the same group, etc.). For example, the composite generated may include a composite video stream that includes a model for the participant105and models for the participant105's neighboring participants (e.g., participants that have reserved virtual seats that neighbor the participant105's virtual seat).

In providing the virtual reality or augmented reality environment, the venue audio/video system950(e.g., using the computational mixer) may generate 3D models for various participants and/or objects in real-time or substantially real-time. That is, for example, the venue audio/video system950may generate a model for the participant105on the fly (e.g., using one or more video transmissions from the 5G device915and/or from the camera118). Alternatively, the venue audio/video system950may update existing 3D models for various participants and/or objects in real-time or substantially real-time. For example, the venue audio/video system950(e.g., using a computational mixer) may update the facial expressions for an existing model for the participant105on the fly, e.g., using a video stream from a front-facing camera of the wireless device115and/or the camera118. The venue audio/video system950may also update the motions and/or orientations of the model, e.g., using sensor data from one or more motion capture devices. The existing model for the participant105may be generated, for example, by the venue audio/video system950using an obtained image of the participant105(e.g., from a social network that the participant105belongs to, from a profile image of the participant105that they provided when registering to use the virtual seat application shown inFIG.2,FIG.4, andFIG.11).

In providing the virtual reality or augmented reality environment, the venue audio/video system950(e.g., using a computational mixer) may align the various 3D models with the environment (e.g., the venue, particular section of the venue, particular seat of the venue, particular group of seats of the venue, etc.). For example, the venue audio/video system950may align each model of each participant with a particular seat in the venue.

As illustrated inFIG.9A, the resulting composite generated by the venue audio/video system950may be transmitted to the 5G device915over the 5G network925, resulting in the participant105being presented the interface910aon a display of the 5G device915. As shown, the interface910ais an augmented reality environment that includes video data of the actual venue where the corresponding event is taking place. As an example, the video data may be generated from a professional stream of the event and/or from a camera that is setup at or near the seat in the venue that the participant105reserved. The video data of the venue (e.g., from the perspective of the seat of the venue that the participant105reserved) may be augmented with 3D models of other remote participants that have reserved seats neighboring the participant105's seat. For example, the interface910aincludes a first model912afor a remote participant John Smith, a second model914for a remote participant Celebrity 1, a third model916for a remote participant Celebrity 2, a fourth model918afor a remote Celebrity 3, and a fifth model920for a remote participant Paul Ross.

The 3D models provided in the composite (e.g., the augmented reality environment) may include the real-time or substantially real-time audio, motions, and/or appearances of the remote participants. For example, as shown, the model912ashows that the remote participant John Smith is currently standing with one arm raised and yelling “Go Team A!” The orientation of the model912aand the audio projected from the location of the model912amay match the real-time or substantially real-time orientation and/or audio of the remote participant John Smith. For example, the audio/video system950may use motion capture data to determine that the remote participant John Smith is current standing with one armed raised and may use audio transmission from John Smith to determine that he is currently yelling “Go Team A!” Similarly, the audio/video system950may use audio data captured from the remote participant Celebrity 3 to include audio of “Defense!” to be included in the audio that the participant105hears and to assign the source of that specific audio to a location where the model918ais located. As will be discussed in more detail below, the general audio settings960for the participant105indicate that the audio power contribution of the remote participant John Smith to the audio component of the composite is currently set to 15%.

In some cases, the 3D models include a depiction of a name or other identifier for the remote participants. The name may be an actual name for the participants, a first name for the participants (e.g., for privacy purposes), or a username or ID number for the participants (e.g., for privacy purposes). The participants themselves may be able to select whether they want their full name or only their first name displayed. Similarly, a participant may be able to select an option to not have their name displayed, in which case a username or other identifier may be assigned to the corresponding model. The name or identifier may be displayed as virtual text floating above the corresponding model, appearing on the corresponding model, or on a seat associated with the corresponding model. In some case, e.g., for sporting events or political speeches, in place of or in addition to the identifier, the models include an indication of the team or political party that the participant is supporting. This indication may be in the form of a jersey placed on the corresponding model (e.g., the model912amay be generated to depict wearing a Team A jersey based on the statements by the remote participant John Smith, on a section of the venue that the corresponding seat is located in (e.g., home vs away section), based on a preference indicated by the remote participant John Smith when they reserved a ticket for their seat, etc.).

The general audio settings960depict the audio settings that are being used to generate the audio that is transmitted to the participant105(e.g., transmitted to the 5G device915and outputted through one or more speakers of the 5G device915). That is, for example, the general audio settings960represent how a computational mixer (e.g., of the audio/video system950) is combing different sources to generate the audio (and/or video) component of the composite sent to the 5G device915of the participant105. The general audio settings960may represent default settings that may be modified, e.g., by the participant105as described in more detail below with respect toFIG.9B. As shown, the audio component of the composite provided to the participant105is made up of 30% participant audio data (e.g., from other remote participants), 5% from ambient noise (e.g., from microphones located throughout a venue), and 65% from professional audiovisual contribution (e.g., audio and/or visual material that represents video and commentary that represents the primary broadcast materials, such as the audio of announcers, players, coaches, performers, etc.). Accordingly, most of what the participant105is hearing is coming from the professional audiovisual sources, followed by the sounds of other remote participants, and finally by the ambient noise in the venue. In more detail, the participant contribution is broken down further by seat distance from a seat of the participant105. For example, remote participants that are one virtual seat away from the participant105contribute 15% to the audio component of the composite (e.g., such as the remote participants John Smith, Paul Ross, and Celebrity 2), whereas remote participants that are two virtual seats away contribute 10% (e.g., such as the remote participants Celebrity 1, and Celebrity 3) and remote participants that are three virtual seats away contribute only 5%. The general audio settings960may provide that remote participants that more than three virtual seats away from the participant105's seat may be effectively muted with 0% contribution (though they may have an insignificant contribution through the ambient noise contribution and/or the professional audiovisual contribution).

The general audio settings960may be stored as part of a profile for the participant105, e.g., on the venue audio/video system950.

FIG.9Bshows an example logical diagram of the distributed audio/video system900B. As illustrated, in place of the general audio settings960inFIG.9A, advanced audio settings962are used.

The advanced audio settings962may allow the participant105(or an administrator) greater control over the audio that they receive (e.g., from a computation mixer of the audio/video system950). As illustrated, the advanced audio settings962may allow the participant105(or an administrator) to control the audio contribution based on categories of other remote participants in addition to the virtual distance of those remote participants. For example, the participant105may be able to specify settings based on whether the remote participant is unknown, a friend, or a celebrity. Similarly, the participant105may be able to adjust audio settings based on whether a speaker is a performer, a coach, an announcer, etc. (e.g., as they may each have unique audiovisual sources). As shown, the participant105has configured the advanced audio settings962such that that the contributions of the celebrities are weighted more heavily than other remote participants.

As depicted in the interface910bof the 5G device915, the effect that the advanced audio settings962have on the participant105's virtual experience include the audio of “Defense!” from the Celebrity 3 can be heard much loader and/or more clearly. This is because the advanced audio settings962provide that the contribution for the remote participant Celebrity 3 is 20% based on the remote participant Celebrity 3 being a celebrity and being two seats away from the seat of the participant105.

In some cases, the advanced audio settings962may allow the participant105(or an administrator) to change the audio contribution for particular remote participants. For example, if the participant105does not want to hear anything from the remote participant John Smith, the participant105may be able to adjust the audio settings for the computation mixer of the audio/video system950to mute the remote participant John Smith. This option or similar options may be presented to the participant105in an audio setting interface on the display of the 5G device915, e.g., as part of the virtual seat application.

The advanced audio settings962may be stored as part of a profile for the participant105, e.g., on the venue audio/video system950.

FIG.9Cis a diagram that illustrates an example distributed audio/video system900C using a 5G network. The distributed audio/video system900C includes the venue audio/video system950, the 5G device915of the remote participant105, and 5G devices935of other remote participants. The venue audio/video system950, the 5G device915, and 5G devices935may communicate over the 5G network925. The venue audio/video system950includes a server952and multiple audio systems954. The server952may store profiles for the remote participants (e.g., may create a profile for a participant when they register to reserve a virtual seat). The audio systems954may be, for example, the audio systems130,135, and140shown inFIG.1. The audio systems954may include, for example, speakers that are assigned to individual seats in the venue, speakers that are assigned to a section of seats in the venue, speakers that are assigned to particular persons (e.g., announcers, coaches, performers, etc.). The server952may communicate with the audio systems954through a wired or a wireless connection. For example, the server952may communicate with the audio systems954through the 5G network925.

FIG.9Calso illustrates a flow of data, shown as stages (A) to (D), with each representing a step in an example process. Stages (A) to (D) may occur in the illustrated sequence, or in a sequence that is different from the illustrated sequence. For example, some of the stages may occur concurrently. The stages (A) to (D) also correspond with a timeline980showing approximate times of different events occurring with respect to the flow of data.

In stage (A), the server952of the venue audio/video system950outputs audio/video data970. The audio/video data may include, for example, video data showing that a player 1 in a game being played in the venue just stole the basketball. The audio/video data970may be transmitted over the 5G network to the 5G device915and the 5G devices935. The transmission of the audio/video data970corresponds to a first time982—of, for example, 0 ms—when the server952transmits the audio/video data970and a second time984—of, for example, approximately 0.5 ms—when audio/video data970is received at the 5G device915and the 5G devices935.

In stage (B), one or more of the 5G devices535transmit audio data972to the server952. The audio data972may include, for example, the audio data of “Go Team A!” generated by a microphone of a 5G device belonging to the remote participant John Smith shown inFIGS.9A-9B. The audio data972may be generated by the remote participant John Smith in response to seeing the player steal the basketball as shown in the audio/video data970. The transmission of the audio data972may correspond to a third time986—of, for example, approximately 0.5 ms plus the reaction time of the remote participant John Smith—when the audio data972is sent by one of the 5G devices935to the server952and a fourth time988—of, for example, approximately 1 ms plus the reaction time of the remote participant John Smith—when the audio data972is received by the server952.

In stage (C), the server952receives the audio data972. The server952may use the audio data972along with other sources of audio data and/or video data to generate, for example, a composite audiovisual signal for the 5G device915and/or for one or more of the audio systems954. The server952may do this in real-time or substantial real-time such that there is little if any significant delay. That is, the server952may use a computational mixer to combine the different and audio sources for the participant105in accordance with their audio settings (e.g., the general audio settings960shown inFIG.9Aor the advanced audio settings962shown inFIG.9B) without significant processing time.

In stage (D), the server952sends audio/video data974to the 5G device915. As shown, the audio/video data974may include the “Go Team A!” audio data of the audio data972but with its contribution (e.g., contribution to an audio component of a composite venue audiovisual signal) reduced to 15% by the server952. The transmission of the audio/video data974may correspond to the fourth time988—of, for example, approximately 1 ms plus the reaction time of the remote participant John Smith—when the audio data974is sent by the sever to the 5G device915and a fifth time990—of, for example, approximately 1.5 ms plus the reaction time of the remote participant John Smith—when the audio data974is received by the server952over the 5G network925.

As depicted, streaming the audio and/or video data of an event to multiple remote participants can be achieved with very low latency through the use of a 5G network. Accordingly, the distributed audio/video system900C can provide a seamless virtual experience such that various remote participants can interact with each other and can react to the event itself without or substantially without any perceivable lag (e.g., of the game or of the reactions of other remote participants).

The times982-990in the timeline980may represent ideal times that can be achieved only using a 5G network. A 5G network may achieve times different than what is depicted in the example ofFIG.9C, however the times that can be achieved using a 5G network are significantly faster than what is possible using other cellular networks (e.g., 3G and 4G), and, thus, can be used to provide an environment with substantially reduced latency and an overall better virtual experience for the participants. In addition, although not shown, there may be processing times by the various devices of the distributed audio/video system900C in extracting, loading, presenting, mixing, and/or packaging data (e.g., audio data, video data, etc.).

AlthoughFIGS.9A-9Care described above with respect to 5G mobile networks, different low latency technologies or technology standards may be used in place of a 5G mobile network. Such technologies or technology standards may still be in development but can be, upon release, implemented with the other disclosed teaching to realize the benefits described herein. For example, instead of using the 5G network925, the distributed audio/video system900may use a 6G cellular over which a 6G enabled wireless device of the participant105can communicate with the venue audio/video system950.

FIGS.10A-10Bare example diagrams illustrating a distributed audio/video system1000. The distributed audio/video system1000may include the venue audio/video system950and the 5G device915which can communicate over the 5G network925either directly (e.g., using a 5G capable integrated antenna array and modem) or indirectly (e.g., through the router116that may be a 5G capable device that can connect to the internet using the 5G network925). The 5G network925may be a 5G cellular network. The distributed audio/video system1000may further include a computational mixer, such as the computational mixer640shown inFIG.6, to combine the audio and/or video data from multiple sources, including audio and/or video data from multiple participants that have reserved virtual seats in the venue, to generate one or more venue output signals. These one or more venue output signals may be fed back to the participants, and may include multiple different signals that are fed to different participants depending on their specific virtual seat, the section of a venue where the virtual seat is located, and/or the participant's audio settings or added features for the event (e.g., receiving audio data of coaches, celebrities, performers, etc.). The audio and/or video data fed back to the participants may be used to generate an augmented reality on an interface of the 5G device915that the participants can interact with to, for example, provide an in-person like experience during the event.

In some cases, the distributed audio/video system1000is the distributed audio/video system900shown inFIGS.9A-9C.

FIG.10Ais example logical diagrams of a distributed audio/video system1000. As illustrated, the 5G device915may be able to communicate with the venue audio/video system950over the 5G network925directly or indirectly through the router116. In using the 5G network925to communicate with the venue audio/video system950, low latency (e.g., less than 50 ms, less than 30 ms, less than 20 ms, less than 10 ms, less than 4 ms, less than 2 ms, or less than or equal to 1 ms) or very low latency (e.g., less than 10 ms, less than 4 ms, less than 2 ms, or less than or equal to 1 ms) and other transmission benefits can be achieved that would allow the participant105to seamlessly enjoy the virtual experience (e.g., watch the event, interact with virtual seat neighbors, hear celebrities, interact with celebrities, etc.) in substantially real-time (e.g., near real-time) with no, or substantially no, perceivable lag. Moreover, in leveraging the transmission speeds of the 5G network other benefits can be afforded. For example, the distributed audio/video system900A can take advantage of the 5G network925in order to generate and/or update a virtual reality or augmented reality on the fly with no, or substantially no, perceivable lag.

As illustrated inFIG.10A, remote participants may be able to view and/or modify celebrity options1002. The celebrity options1002may have default values that can be customized for the particular remote participant (e.g., by the remote participants or based on information in a user profile for the remote participant). The celebrity options1002may be particular to a particular event, such that the participant105is presented the celebrity options1002after reserving a seat that is in the vicinity (e.g., within three seats of) of one or more celebrities. As shown, the celebrity options1002include an option to enable or disable (e.g., mute) individual celebrities having reserved seats neighboring the seat reserved by the participant105. The celebrity options1002may also include an option to enable or disable interactivity between particular celebrities. Such an option may only be available to the participant105if, for example, a corresponding celebrity has agreed to interactivity (e.g., which would allow audio data generated by the participant105to be transmitted to the celebrity or transmitted to the celebrity with a contribution that is higher than what is typical per the audio settings of the celebrity, and/or higher than what is typical for contribution of other remote participants in the vicinity). Similarly, such an option may only be available to the participant105if they reserved such an option when they reserved their virtual seat.

The celebrity options1002provide that the audio data that the participant105receives from the venue audio/video system950does include the audio of all celebrities in the virtual vicinity of the participant. That is, for example, the audio component of a composite audiovisual signal outputted by a computation mixer of the venue audio/video system950will include detected audio generated by the remote participant Celebrity 1, the remote participant Celebrity 2, and the remote participant Celebrity 3. The celebrity options1002further provide that the participant105can currently interact with the remote participant Celebrity 3, but not the other two celebrities.

An example interaction between the participant105and the remote participant Celebrity 3 is depicted in the interface1010of the 5G device915. Here, the participant105says “Hey Celebrity 3!” The audio/video system950may receive this audio data and send it (e.g., mixed along with other audiovisual data, but with a higher contribution per the celebrity options, such as 30% contribution to Celebrity 3 instead of a typical 10% for other nearby remote participants) to the remote participant Celebrity 3. Having received and heard this audio, Celebrity 3 responds to the participant105, stating “Hey Bob!” Due to the low latency provided by the 5G network925, this conversation between the participant105and Celebrity 3 can happen in real-time or substantially real-time.

The participant105may be able to update the celebrity options1002using the 5G device915. For example, the participant105may access a celebrity options interface on the virtual seat application running on the wireless device (e.g., as depicted inFIG.2).

The celebrity options1002may be stored as part of a profile for the participant105, e.g., on the venue audio/video system950.

FIG.10Ba diagram that illustrates an example distributed audio/video system1000B using a 5G network. The distributed audio/video system1000B includes the venue audio/video system950, the 5G device915of the remote participant105, and a celebrity 5G device1015of other remote participants. The venue audio/video system950, the 5G device915, and celebrity 5G device1015may communicate over the 5G network925. The venue audio/video system950includes a server952and multiple audio systems954. The audio systems954may be, for example, the audio systems130,135, and140shown inFIG.1. The audio systems954may include, for example, speakers that are assigned to individual seats in the venue, speakers that are assigned to a section of seats in the venue, speakers that are assigned to particular persons (e.g., announcers, coaches, performers, etc.). The server952may communicate with the audio systems954through a wired or a wireless connection. For example, the server952may communicate with the audio systems954through the 5G network925.

FIG.10Balso illustrates a flow of data, shown as stages (A) to (F), with each representing a step in an example process. Stages (A) to (F) may occur in the illustrated sequence, or in a sequence that is different from the illustrated sequence. For example, some of the stages may occur concurrently. The stages (A) to (F) also correspond with a timeline1020showing approximate times of different events occurring with respect to the flow of data.

In stage (A), the 5G device915outputs detected audio data1040and transmits it to the server952of the venue audio/video system950over the 5G network925. As shown, the audio data1040includes the participant105's greeting “Hey Celebrity 3.” The transmission of the audio data1040corresponds to a first time1022—of, for example, 0 ms—when the 5G device915sends the audio data1040to the server952.

In stage (B), the server952receives the audio data1040. The reception of the audio data1040corresponds to a second time1022—of, for example, approximately 0.5 ms—when the server952receives the audio data1040. After receiving the audio data1040, the server952may use the audio data1040along with other sources of audio data and/or video data to generate, for example, a composite audiovisual signal for the celebrity 5G device1015and/or for one or more of the audio systems954. The server952may do this in real-time or substantial real-time such that there is little if any significant delay. That is, the server952may use a computational mixer to combine the different and audio sources for the Celebrity 3 in accordance with their audio settings (e.g., the general audio settings960shown inFIG.9Aor the advanced audio settings962shown inFIG.9B) and the celebrity options1002of the participant105without significant processing time. In some cases, such as depicted, there may be a slight delay in order to mix the multiple audio and/or video sources.

In stage (C), the server transmits audio/video data1042to the celebrity 5G device1015over the 5G network925. The transmission of the audio/video data1042may correspond to a third time1026—of, for example, approximately 1 ms—when the server952sends the audio/video data1042to the celebrity 5G device1015. In this example, there was a slight delay of approximately 0.5 ms on the part of the server952to generate the audio/video data1042from the audio data1040(e.g., and from other sources). As shown, the audio/video data1042includes the audio data1040with a contribution to the Celebrity 3's total audio signal of 15%.

In stage (D), the celebrity 5G device1015transmits audio data1044to the server952over the 5G network925. The audio data1044includes a statement “Hey Bob!” based on sound detected by the celebrity 5G device1015of the Celebrity 3 responding to the participant105's greeting, “Hey Celebrity 3.” The transmission of the audio data1044may correspond to a fourth time1028—of, for example, approximately 1 ms plus Celebrity 3's reaction time—when the celebrity 5G device1015send the audio data1044to the server952.

In stage (E), the server952receives the audio data1044from the celebrity 5G device1015. The reception of the audio data1044corresponds to a fifth time1030—of, for example, approximately 1.5 ms plus the reaction time of Celebrity 3—when the server952receives the audio data1044. After receiving the audio data1044, the server952may use the audio data1044along with other sources of audio data and/or video data to generate, for example, a composite audiovisual signal for the 5G device915and/or for one or more of the audio systems954. The server952may do this in real-time or substantial real-time such that there is little if any significant delay. That is, the server952may use a computational mixer to combine the different and audio sources for the participant105in accordance with their audio settings (e.g., the general audio settings960shown inFIG.9Aor the advanced audio settings962shown inFIG.9B) without significant processing time. As shown, the audio settings for the 5G device915provide that the audio corresponding to Celebrity 3 is to make up 20% of the audio component of the composite audiovisual signal provided to the 5G device915.

In stage (F), the server952transmits audio/video data1046to the 5G device915over the 5G network925. The audio/video data1046include the audio response from Celebrity 3 of “Hey Bob!” The contribution of the Celebrity 3's audio to the total audio signal for the participant105is 20%. The transmission of the audio/video data1046corresponds to the fifth time1030—of, for example, approximately 1.5 ms plus the reaction time of Celebrity 3—when the server952sends the audio/video data1046to the 5G device915and to a sixth time1032—of, for example, approximately 2 ms plus the reaction time of Celebrity 3—when the 5G device915receives the audio/video data1046.

As depicted, streaming the audio and/or video data of an event to multiple remote participants while allowing communication between the remote participants can be achieved with very low latency through the use of a 5G network. Accordingly, the distributed audio/video system1000B can provide a seamless virtual experience such that various remote participants can interact with each other and can react to the event itself without or substantially without any perceivable lag (e.g., of the game, in their conversations, in other participants' reactions to events, etc.).

The times1022-990in the timeline980may represent ideal times that can be achieved only using a 5G network. A 5G network may achieve times different than what is depicted in the example ofFIG.9C, however the times that can be achieved using a 5G network are significantly faster than what is possible using other cellular networks (e.g., 3G and 4G), and, thus, can be used to provide an environment with substantially reduced latency and an overall better virtual experience for the participants. In addition, although not shown, there may be processing times by the various devices of the distributed audio/video system900C in extracting, loading, presenting, mixing, and/or packaging data (e.g., audio data, video data, etc.).

The augmented virtual environment (or entirely virtual environment) depicted in the interfaces910aofFIG.9A,910bofFIG.9B, and1010ofFIG.10Amay allow the participant105to see different portions of the environment by, for example, changing their own orientation and/or performing actions. For example, the participant105may be shown the venue from a different perspective if they stand up. Similarly, the participant105may be able to see and/or interact with additional, different remote participants if they turn their head to the left or right.

AlthoughFIGS.10A-10Bare described above with respect to 5G mobile networks, different low latency technologies or technology standards may be used in place of a 5G mobile network. Such technologies or technology standards may still be in development but can be, upon release, implemented with the other disclosed teaching to realize the benefits described herein. For example, instead of using the 5G network925, the distributed audio/video system1000may use a 6G cellular over which a 6G enabled wireless device of the participant105can communicate with the venue audio/video system950.

FIG.11is an example user interface1100for reserving virtual seats. The UI1100may be presented to the participant105on the 5G device915. The UI1100may be presented as part of a virtual seat or a virtual ticket mobile application running on the 5G device.

As shown, the UI1100may allow the participant105to select a particular event that will be performed at a venue, such as a sports game or a concert. The participant105may, for example, use a dropdown menu to see upcoming events that will be performed in the venue (e.g., in the next week, next month, next six months, next year, etc.).

The UI1100may also present a representation1102of the venue. The representation1102may include, for example, depictions of the different seating sections for the venue. In some cases, the sections are dependent on the event selected by the participant. A section1104(Section101) has been highlighted, e.g., due to the section1104being a section with one or more recommended virtual seats, being the section with the highest recommended virtual seat, and/or being selected by the participant105. For example, after selecting the section1104, the participant105may be presented a list1106of recommended seats in the selected section. As another example, the list1106of recommended seats may include the seats with the highest recommendations throughout the venue.

The recommended seats in the list1106may be selected by the 5G device915and/or the server952shown inFIG.9CandFIG.10B. In generating the list1106, the 5G device915and/or the server952may generate a score the virtual seats (e.g., for each available virtual seat that has yet to be reserved). The score may be based on, for example, a view quality of the venue from the corresponding seat, an audio quality of the venue from the corresponding seat, other audio factor such as friends having reserved neighboring seats or celebrities having reserved neighboring seats. As an example, a video score may be generated and an audio score may be generated. These two scores may be used to generate an overall score for the seat (e.g., by averaging the scores, or applying particular weights to each of the scores). As an example, the server952may determine that seat 5 in row 2 of section101has a score of 95 based on having a video quality score of 95 (e.g., section is close to middle of the court and seat is the second row which is close to the court) and having an audio score of 95 (e.g., based on a friend being one seat away and having three celebrities in neighboring seats).

The list1106may be particular to the participant105or may be made for all participants. As an example, the server952may generate the list1106using information stored in a profile of the participant105. The profile information may include, for example, a virtual seat reservation history for the participant105. As an example, the server952may use the participant105's virtual seat reservation history to determine that they only ever reserve seats in section101,103or105. In response to this determination, the server952may increase the score of virtual seats that are found available in those particular sections.

Reserving a virtual seat can include preventing other persons from reserving the same virtual seat. Accordingly, the audiovisual perspective of the virtual or augmented environment provided to a participant may be unique to them. In some cases, some virtual seats may optionally be made available to multiple participants (e.g., if an event is overbooked).

FIG.12is a flowchart of an example process1200of using a distributed audio/video system. The process1200can be performed by one or more computers or computing devices as discussed above. For example, the process1200can be performed by the wireless device115shown inFIG.1, venue audio/video system121shown inFIG.1, the computational mixer640shown inFIG.6, the 5G device915shown inFIGS.9A-9C and10A-10B, or the venue audio/video system950shown inFIGS.9A-9C and10A-10B.

The process1200includes establishing a link that includes at least one of audio data and video data between a wireless device of a remote participant and a computational mixer (1210). For example, with respect toFIGS.9A-9C, a link may be established between the 5G device915of the remote participant105and the server952of the venue audio/video system950. The server952and/or the venue audio/video system950may include a computational mixer (e.g., that includes an audio mixer and/or video mixer), such as the computational mixer640shown inFIG.6. The computational mixer may interface audio transmissions of the remote participants, video transmissions of the remote participants, and/or other audio and/or video transmissions (e.g., video and/or audio data of the event such as the professional audio/visual source630and/or the ambient audio source620shown inFIG.6) with virtual reality graphics from a game engine, and proceed to composite the transmissions together in real-time or substantially real-time. The resulting composite may then be fed to the participant.

The process1200includes referencing a profile of the remote participant (1220). For example, the server952may access a profile stored for the participant105. The profile may include, for example, a virtual seat reservation history for the participant, audio settings for the participant, celebrity settings for the participant, virtual/augmented environment presentation settings, etc.

The process1200includes generating a venue signal that is related to the at least one audio data and video data (1230). The venue signal may be a composite signal that is generated from multiple different sources. For example, the mixer640may combine multiple audio streams from remote participants, ambient audio data, and professional audio data in accordance with the audio settings in the profile of the participant105to generate a composite audio signal (or an audio component to a composite audiovisual signal). The venue signal may also include video data from various sources, such as the audio sources mentioned. The venue signal may be particular to a particular participant or a particular group of participants (e.g., all having reserved virtual seats in the same section). Alternatively, the venue signal may be, for example, an audio and/or video signal that is to be outputted in the venue, in a particular section of the venue (e.g., using one or more speakers and/or displays in the venue), or using particular devices (e.g., particular speakers and/or displays) in the venue.

The process1200includes transmitting the venue signal (1240). For example, the server952may transmit the venue signal (e.g., the composite audiovisual signal) to the 5G device915of the participant105. The 5G device915may use the venue signal to output audio and/or video data of a virtual environment corresponding to the seat in the venue that the participant105reserved. For example, the 5G device915may generate the interface910ashown inFIG.9Ausing the venue signal. As another example, the venue signal may be transmitted to the audio systems954of the venue audio/video system950. The audio systems954or a specific system in the audio systems954may output the audio signal using one or more speakers (e.g., a set of speakers for a particular section of a venue) and/or displays.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Although a few implementations have been described in detail above, other modifications are possible. Moreover, other mechanisms for performing the systems and methods described in this document may be used. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.