Patent Description:
Technological advances gave rise to our modern information age and facilitated an explosion of easily and readily accessible content, particularly streaming content (e.g., multimedia content) over multi-user platforms. These multi-user platforms, in turn, support an ever-increasing consumer base of spectators and players who engage or otherwise participate in online gameplay. Because of this popularity, there is substantial interest in improving spectator experiences.

However, creating spectator experiences similar to spectating conventional sporting events present new challenges due to the inherently online nature of such gameplay. For example, while traditional sporting events typically involve a physical meeting of teams on a shared field and broadcast licensing between specified parties, online gameplay is often hosted over multi-user platforms, which allow players (and spectators) from anywhere in the world to connect and interact within a virtual environment of a game. In addition, online gameplay often involves more nuanced and complex media content (as compared to conventional sporting events), including a large number of viewing angles, different point systems, graphical renderings, and the like. Further, current distribution models to broadcast online game play in conjunction with spectator-generated content (e.g., commentary content) often require license agreements, which hinder real-time commentary of live online gameplay.

Therefore, there is a need in the art for improved spectator experiences, including integrating commentary and gameplay over a multi-user platform. <CIT> shows annotation of video game playback.

<NUM> The invention is defined by claims <NUM>, <NUM> and <NUM>.

The embodiments herein may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numerals indicate identical or functionally similar elements.

Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the scope of the appended claims.

As used herein, the term "user" refers to a user of an electronic device(s) and can include participants or players, as well as non-participants or spectators. Actions performed by a user in the context of computer software shall be considered to be actions taken by a user to provide an input to the electronic device(s) to cause the electronic device to perform the steps embodied in computer software. The terms "stream" or "media stream" are synonymous and generally refer to data or content associated with an online game or an online game session.

As discussed in herein, the subject disclosure generally relates to online gameplay hosted by multi-user platforms and improving spectator experiences. In particular, the techniques disclosed herein integrate and synchronize commentary content an gameplay content and broadcast such content to subscribers of the multi-user platform.

Referring to the figures, <FIG> illustrates a schematic diagram <NUM> of an example communication network <NUM> (e.g., the Internet). Communication network <NUM> is shown for purposes of illustration and represents various types of networks, ranging from local area networks (LANs) to wide area networks (WANs). LANs typically connect the nodes over dedicated private communications links located in the same general physical location, such as a building or campus. WANs, on the other hand, typically connect geographically dispersed nodes over long-distance communications links, such as common carrier telephone lines, optical lightpaths, synchronous optical networks (SONET), synchronous digital hierarchy (SDH) links, or Powerline Communications (PLC) such as IEEE <NUM>, IEEE P1901. <NUM>, and others.

Communication network <NUM> includes a geographically distributed collection of devices or nodes <NUM>, interconnected by communication links <NUM> for exchanging data such as data packets <NUM> and for transporting data to end nodes or client devices <NUM> through a multi-user platform <NUM>. In particular, multi-user platform <NUM> distributes or broadcasts multi-media content (e.g., audio content, visual content, textual content, etc.) to end nodes or client devices <NUM>. Client devices <NUM> include personal computing devices, online game systems, laptops, tablets, mobile devices, or other devices as is appreciated by those skilled in the art. Notably, one client device <NUM> represents a network game system, which includes a game console, peripheral devices, and display hardware. Operatively, a user can subscribe client device <NUM> to multi-user platform <NUM> and play, spectate, or otherwise access online media content hosted by multi-user platform <NUM>.

Further, communication links <NUM> represent wired links or shared media links (e.g., wireless links, PLC links, etc.) where certain devices, such as, e.g., routers, servers, switches, sensors, computers, etc., may be in communication with other devices, based on distance, signal strength, current operational status, location, etc. Those skilled in the art will understand that any number of nodes, devices, links, etc. may be used in the communication network, and that the view shown herein is for simplicity.

Data packets <NUM> such as network traffic/messages are exchanged between devices over and within communication network <NUM> using predefined network communication protocols such as certain known wired protocols, wireless protocols (e.g., IEEE Std. <NUM>, WiFi, Bluetooth®, etc.), PLC protocols, or other shared-media protocols where appropriate. In this context, a protocol consists of a set of rules defining how the devices or nodes interact with each other.

<FIG> illustrates a block diagram of an example network device <NUM> that represents multi-user platform <NUM> (or components thereof). Device <NUM> includes one or more network interfaces <NUM>, a user input interface <NUM>, at least one processor <NUM>, and a memory <NUM> interconnected by a system bus <NUM>.

Network interface(s) <NUM> contain the mechanical, electrical, and signaling circuitry for communicating data over links coupled to one or more of the networks shown in schematic diagram <NUM>. Network interfaces <NUM> are configured to transmit and/or receive data using a variety of different communication protocols, as will be understood by those skilled in the art.

User input interfaces <NUM> may be inclusive of any variety of user interface known in the art for receiving different types of user input, including at least handheld controllers, portable controllers, keyboards, keypads, touchscreens, cameras, game peripherals and accessories, etc. Some interfaces <NUM> may be specific to virtual reality (VR) environments. Virtual reality (VR) interface(s) <NUM> provide interactive graphical interfaces to solicit and receive user input corresponding to gameplay content in a VR environment. For example, VR interface <NUM> may include any number of menus, boxes, buttons, editor interfaces, drawing tools, playback tools, selectable elements, graphical icons, and the like. These graphical interfaces can be manipulated by a user to provide commentary for a game session as discussed in greater detail below.

Memory <NUM> comprises a plurality of storage locations that are addressable by processor <NUM> for storing software programs and data structures associated with the embodiments described herein.

Processor <NUM> may comprise necessary elements or logic adapted to execute the software programs and manipulate data structures <NUM>. An operating system <NUM>, portions of which are typically resident in memory <NUM> and executed by processor <NUM>, functionally organizes the device by, inter alia, invoking operations in support of software processes and/or services executing on the device. These software processes and/or services may comprise an illustrative "commentary" process/service <NUM>. Note that while processes/services <NUM> are shown in centralized memory <NUM>, these processes/services may be configured to operate in a distributed communication network.

It will be apparent to those skilled in the art that other processor and memory types, including various computer-readable media, may be used to store and execute program instructions pertaining to the techniques described herein. Also, while the description illustrates various processes, it is expressly contemplated that various processes may be embodied as modules configured to operate in accordance with the techniques herein (e.g., according to the functionality of a similar process). Further, while the processes have been shown separately, those skilled in the art will appreciate that processes may be routines or modules within other processes. For example, processor <NUM> can include one or more programmable processors, e.g., microprocessors or microcontrollers, or fixed-logic processors. In the case of a programmable processor, any associated memory, e.g., memory <NUM>, may be any type of tangible processor readable memory, e.g., random access, read-only, etc., that is encoded with or stores instructions that can implement program modules, e.g., a module having commentary process <NUM> encoded thereon. Processor <NUM> can also include a fixed-logic processing device, such as an application specific integrated circuit (ASIC) or a digital signal processor that is configured with firmware comprised of instructions or logic that can cause the processor to perform the functions described herein. Thus, program modules may be encoded in one or more tangible computer readable storage media for execution, such as with fixed logic or programmable logic, e.g., software/computer instructions executed by a processor, and any processor may be a programmable processor, programmable digital logic, e.g., field programmable gate array, or an ASIC that comprises fixed digital logic, or a combination thereof. In general, any process logic may be embodied in a processor or computer readable medium that is encoded with instructions for execution by the processor that, when executed by the processor, are operable to cause the processor to perform the functions described herein.

<FIG> illustrates a schematic diagram <NUM> representing gameplay content for a game session. The game session may be hosted by multi-user platform <NUM> (discussed above) and accessible by any number of subscribers (e.g., players, spectators, etc.). As illustrated, the gameplay content relates to a car racing tournament, which can be viewed by a number of cameras - camera 305c, camera 310c, camera 315c, camera 320c, and etc. Cameras 305c, 310c, 315c, and 320c are shown for purposes of illustration and may (or may not) be shown during the game session. Importantly, cameras 305c, 310c, 315c, and 320c represent different media streams (e.g., various Points of View (POV) or viewing angles, etc.) of gameplay and can be viewed by subscribers to multi-user platform <NUM>. The different media streams are assigned or mapped to respective display screens <NUM>, <NUM>, <NUM>, and <NUM> in a digital or VR environment (discussed in greater detail below).

<FIG> illustrates a schematic diagram <NUM> of an exemplary VR environment, particularly showing a simulated commentary studio from a perspective of a user <NUM>. User <NUM> experiences and interacts with the VR environment using a headset <NUM> and a controller <NUM>. In operation, headset <NUM> and/or controller <NUM> may be wirelessly connected to additional components such as a network game system (discussed above) or, alternatively, headset <NUM> and controller <NUM> may be independently coupled to multi-user platform <NUM> over a network (e.g., network <NUM>).

Headset <NUM> simulates the VR environment (e.g., the commentary studio) and displays or projects graphical elements to user <NUM>, tracks eye movements, and measures biometric data, and the like. Controller <NUM>, similar to headset <NUM>, facilitates user interaction with and within the VR environment and is operable to, for example, detect, track, or otherwise monitor movement and biometric information, communicate data signals with headset <NUM> and the network game console, and provide feedback (e.g., tactile, audible, etc.) to a user <NUM>. In this fashion, headset <NUM> and/or controller <NUM> can include any number of sensors, gyros, radios, processors, touch detectors, transmitters, receivers, feedback circuitry, and the like. Headset <NUM> and controller <NUM> (and any other supporting VR equipment such as network game system) cooperate to provide an immersive and interactive VR environment to user <NUM>.

The VR environment shown here can include includes interactive graphical representation of video editing tools as well as a number of display screens showing gameplay content for game session <NUM>. As mentioned above, the display screens, including display screen <NUM>, display screen <NUM>, display screen <NUM>, and display screen <NUM>, show different media streams, each corresponding to a respective POV or viewing angle that corresponds to camera 305c, camera 310c, camera 315c, camera 320c, respectively.

<FIG> illustrates a schematic diagram <NUM> of the commentary studio of <FIG>, further showing a display screen <NUM> selected as an active display screen. In some embodiments, user <NUM> moves controller <NUM> to indicate a selection operation. The network game system detects a change in controller orientation, direction, acceleration, or velocity, determines a corresponding path <NUM>, and projects path <NUM> to headset <NUM>. Path <NUM> intersects with display screen <NUM> and results in selection of display screen <NUM> (e.g., the graphical representation corresponding to display screen <NUM>) as an active display screen or an active representation of gameplay content. Notably, in some embodiments, user <NUM> may provide additional inputs to select display screen <NUM> as the active display screen (e.g., button press, eye movement, eye hovering, etc.). As used herein, the terms "active display screen", "active media stream", and/or active representation" may be used interchangeable to refer to a selected display screen and/or media content corresponding to the selected display screen. In this fashion, selecting display screen <NUM> as the active display screen sets the media stream mapped to display screen <NUM> as an active media stream for a time period (e.g., when the user <NUM> provides commentary content corresponding to the active media stream).

In other embodiments, display screen <NUM> may be selected as the active display screen based on milestones (e.g., transitions or events) in the game session. For example, games can include milestones such as transitions to new levels, discovering new inventory, defeating bosses, interactions between players (e.g., defeats, victories, etc.), a number of points achieved, a likelihood of an event occurring, and the like. Some of these milestones may be set by a game design in advance, while others may be determined based on game statistics derived from iterations of the gameplay. These milestones may be used as a trigger to automatically select and set certain display screens as the active display screen when, for example, the display screen shows players in relative proximity to a milestone. The player's proximity or approach to the gameplay milestone can be determined based on character locations in the game environment (e.g., on a world map), proximity between character locations and the milestone location, players in relative proximity to each other (e.g., in a tournament style game), and the like. In this fashion, the character locations may be tracked during the course of the game session and trigger selection of a display screen as the active display screen.

Still referring to <FIG>, user <NUM> operatively provides commentary content such as audio commentary, textual commentary, and/or visual commentary, for the gameplay in the game session. The VR environment - here, the commentary studio - provides an immersive experience to solicit commentary content and associate/map portions of the commentary content to the media stream displayed by the active display screen. The portions of the commentary content is further associated and synchronized with the media stream for the active display screen, and broadcast to subscribers of the multi-user platform. In addition to the display screens (which show various POVs/viewing angles of gameplay), the VR environment also provides a variety of editing interfaces or tools. User <NUM> can view various aspects of the gameplay over the plurality of display screens, select active display screens/active media streams for periods of time, edit portions of active media streams (corresponding to respective active display screens), and generally provide commentary content about the gameplay using controller <NUM>, headset <NUM>, or other input devices.

<FIG> illustrates a schematic diagram <NUM> of a commentary module <NUM>. Commentary module includes an active screen module <NUM>, a commentary module <NUM>, and an integration module <NUM>. In operation, commentary module <NUM> receives commentary content during a time period - e.g., commentary content 165a, 615b, 615c, 615d, and so on. Active screen module <NUM> monitors and identifies portions of active media streams for the same time period to create corresponding media content - e.g., media content 610a, media content 610b, media content 610c, media content 610d, and so on. Integration module <NUM> maps or associates the active media content with the commentary content based on the time period to synchronize the commentary content with the active media content. That is, user <NUM> selects an active display screen in the VR environment and provides commentary content corresponding to the active media stream associated with the active display screen. Commentary module <NUM>, including sub-modules active screen module <NUM>, commentary module <NUM>, and integration module <NUM>, collectively cooperate to receive the commentary content and synchronize the commentary content with portions of active media content (for an active media stream) to create synchronized content. The synchronized content is further passed to a broadcast module <NUM> for subsequent broadcast transmission to one or more subscribers to the multi-user platform. The synchronized content may include a commentary channel, which can provide commentary content and corresponding media content.

Commentary module <NUM> may represent components or modules of device <NUM> (and/or multi-user platform <NUM>). For example, commentary module <NUM> may perform operations embodied by commentary process/services <NUM> to provide an immersive VR environment, intuitively present media streams (or portions thereof) as well as editing interfaces/tools, etc., map media content with commentary content, and synchronize such content for subsequent broadcast.

<FIG>, <FIG> illustrate schematic diagrams of the VR environment, particularly showing an editing interface <NUM>. The editing interface provides various editing tools that allow user <NUM> to generally edit (e.g., mark-up, sketch over, rewind, set playback options, overlay gameplay statistics, predict player or gameplay behaviors, etc.) portions of media streams displayed by an active display screen to create modified media content. For example, <FIG> shows a drawing tools interface <NUM> that allows a user to create paths, shapes, symbols, and the like, as an overlay to the portion of media stream displayed by the active display screen. Here, the "paths" tool is selected and user <NUM> creates a route or path highlighting a potential move by player <NUM> (P3) to pass player <NUM> (P2). <FIG> shows a playback tools interface <NUM>, which manipulates playback time of the portion of the media stream - e.g., rewind, fast forward, slow motion, stop, replay, etc..

<FIG> shows a gameplay stats interface <NUM> that allows user <NUM> to select and overlay various gameplay statistics over the portion of the media stream. These gameplay stats can include any number of statistics such as play specific statistics, level statistics, match-up statistics between one or more players, and the like. <FIG> shows a predictive model interface <NUM> that allows a user to select and overlay predictive behaviors over the portion of the media stream. The predictive model may be determined based on prior iterations of gameplay for a specific player, for a particular game, and so on.

It is appreciated that the editing interface <NUM> shown by 7A, 7B, 7C, and 7D provides a variety of editing options to user <NUM> and that the illustrated options or tools are provided for purposes of illustration, not limitation. Editing interface <NUM> can include (or exclude) any number of editing tools as desired. Editing interface <NUM> provides intuitive tools that can enhance or supplement commentary content. In this fashion, a user can manipulate, enhance, or otherwise modify portions of media streams to create modified media content. This modified content may be included as part of the commentary content, which can be broadcast to subscribers of the multi-user platform, discussed above.

<FIG> illustrates an example simplified procedure <NUM> for providing commentary related to gameplay, particularly from the perspective of device <NUM>, commentary module <NUM> and/or multi-user platform <NUM> (or components thereof). For purposes of discussion below, reference is made to a multi-user platform.

Procedure <NUM> begins at step <NUM> and continues to steps <NUM>, where the multi-user platform receives gameplay content for a game session. For example, the multi-user platform can host and provide game content to its subscribers. The subscribers interact with the game content and generate gameplay content for a game session. The multi-user platform receives such gameplay content and can further provide a virtual reality (VR) environment to a user. The VR environment can, for example, include the above discussed commentary studio with graphical representations of media streams related to the gameplay, as shown in step <NUM>. For example, a media stream for a particular POV/viewing angle may be represented by a graphical representation of a display screen (e.g., display screen <NUM>, <NUM>, <NUM>, <NUM>, etc.) In addition, the VR environment can provide editing interfaces that allow a user to manipulate media content shown by the display screens. These editing interfaces can overlay various types of graphics, information, playback options, statistics, and the like.

As discussed above, the VR environment provides an immersive experience to solicit user commentary related to gameplay content - here, the user can select a particular display screen as an active display screen, provide commentary content regarding portions of a media stream displayed by the active representation/active display screen. This commentary content can include audio content, visual content, textual content, and the like. In this fashion, the commentary content can include audio commentary from the user as well as a graphics, playback options, gameplay statistics, overlays, and so on. Notably, in some embodiments, the multi-user platform may automatically set certain display screens as active based on gameplay milestones (discussed above).

The multi-user platform further synchronizes, at step <NUM>, the commentary content received in a time period with portions of a media stream that correspond to an active display screen for the time period to create synchronized content. For example, the multi-user platform determines a time period for portions of the commentary content and selects a graphical representation in the VR environment as an active representation (e.g., an active display screen) for the time period. The multi-user platform further parses portion of a media stream associated with the active representation for the time period and associates or maps the portion of the media stream with the commentary content. Notably, in some embodiments, the commentary content includes gameplay statistics (gameplay stats <NUM>), playback modifications (playback tools <NUM>), graphical overlays (e.g., drawing tools <NUM>, predictive models <NUM>), and the like.

The multi-user platform broadcasts, at step <NUM>, the synchronized content to its subscribers. In some embodiments, the synchronized content may be broadcast in conjunction with real-time gameplay on a delay, or a time after the game session ends.

Procedure <NUM> subsequently ends at step <NUM>, but may begin again at step <NUM> where the multi-user platform receives gameplay content for the game session. Collectively, the steps in procedure <NUM> describe a process to provide commentary content in conjunction with gameplay content. It should be noted that certain steps within procedures <NUM> may be optional, and further, the steps shown in <FIG> are merely examples for illustration. Certain other steps may be included or excluded as desired. Further, while a particular order of the steps is shown and executed from the perspective of a particular device or system, this ordering is merely illustrative, and any suitable arrangement of the steps and/or any number of systems, platforms, or devices may be utilized without departing from the scope of the embodiments herein.

The techniques described herein, therefore, provide interactive commentary processes that combine an immersive simulated VR environment with gameplay content for a game session. These interactive commentary processes define simple and intuitive techniques to enhance spectator participation as well as spectator enjoyment.

While there have been shown and described illustrative embodiments of the commentary processes for VR environments (e.g., a commentary studio), it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the embodiments herein. For example, the embodiments and certain functionality have been shown and described herein with relation to certain systems, platforms, hardware, devices, and modules. However, the embodiments in their broader sense are not as limited, and may, in fact, be employed in non-VR environments as well as employed by any combination of the devices or components discussed herein.

Claim 1:
A method for providing commentary related to gameplay, the method comprising:
receiving (<NUM>) gameplay content for a game session hosted by a multi-user platform (<NUM>), wherein the gameplay content includes a plurality of different media streams that show one or more views of the game session;
generating (<NUM>) a graphical representation (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) for each of the media streams in a virtual reality (VR) environment;
generating an editor interface in the VR environment, the editor interface including one or more tools selectable to modify one or more portions of the plurality of media streams to create a modified media stream;
receiving commentary content that includes the modified media stream created via the editor interface;
designating an identified graphical representation as an active graphical representation (<NUM>) in the VR environment for a time period, wherein the active graphical representation is associated with an identified one of the media streams;
synchronizing (<NUM>) a portion of the commentary content with the identified media stream associated with the active graphical representation by associating the commentary content with the active graphical representation based on the time period to create synchronized content; and
broadcasting (<NUM>) the synchronized content via a gameplay channel accessible to one or more subscribers connected to the multi-user platform.