Abstract:
Methods of providing in-game virtual split screens with peer-to-peer video conferencing are described for use in online gaming, for instance. In one approach, a live video stream, a live audio stream, and a player viewpoint are sent from a first computer system for receipt by a second computer system. The first and second computer systems concurrently execute the online game. The live video stream, the in-game video stream, and locally rendered in-game content are combined to create a composite video stream, and the live audio stream, the in-game audio stream, and locally generated in-game audio are combined to create a composite audio stream. The composite video stream and the composite audio stream are operable to be presented at the second computer system in real-time.

Description:
FIELD 
       [0001]    Embodiments of the present invention generally relate to the field of multiplayer gaming. More specifically, embodiments of the present invention relate to systems and methods for providing in-game screen sharing and video communication using virtual split screens for multiplayer gaming. 
       BACKGROUND 
       [0002]    There is a growing need, in the field of computer gaming, to share multimedia content and communications in real-time with other connected (e.g., online) players without interrupting in-game activity. To this end, some multiplayer games provide in-game text or voice chat to enable real-time communications between players as an alternative to external devices or third-party chat applications. However, providing concurrent video and audio communications in-game in real-time is more complicated and requires more bandwidth and processing capabilities than text or audio communications alone. 
         [0003]    In addition to providing real-time video communications captured using a camera and audio communications captured using a microphone, there is a growing need to share activity and events that occur in-game with other players in real-time. For example, cooperative (co-op) first-person shooting games (FPS) and massively multiplayer online games (MMOs) may organize players into squads or teams that share a common objective. In these situations, it is often desirable for team members to share information with one another, including real-time visual representations of the game world from another player&#39;s perspective. However, this ability is not often supported by game developers due to the complexity of implementing such a system on a wide scale. 
       SUMMARY 
       [0004]    A method and apparatus for providing built-in support for virtual split screens with peer-to-peer video is disclosed herein. Embodiments of the present invention enable a user to share real-time audio and video communications and player viewpoint data with other players during multiplayer gaming. 
         [0005]    According to one embodiment, a method of sharing player viewpoint data and audio-video communication with remote players of an online game is disclosed. The method includes executing the online game using a processor and a memory of a computer system, capturing a live video stream using a camera coupled to the computer system, capturing a live audio stream using a microphone coupled to the computer system, and generating player viewpoint data of the online game during execution, where the player viewpoint data comprises an in-game video stream and an in-game audio stream. The live video stream, the live audio stream, and the player viewpoint data are sent to a second computer system using a computer network, where the second computer system is operable to concurrently execute the online game. The live video stream and the in-game video stream are combined to create a composite video stream, the live audio stream and the in-game audio stream are combined to create a composite audio stream, and the composite video stream and the composite audio stream are rendered at the second computer system alongside in-game content generated by the second computer system. 
         [0006]    According to another embodiment, an apparatus for sharing player viewpoint data and audio-video communication with remote players of an online game is described. The apparatus includes a processor and memory configured to execute instructions of an online game, where the instructions, when executed, generate local in-game audio data and local in-game video data, a camera for capturing a local real-time video stream, a microphone for capturing a local real-time audio stream, and a network streamer for sending the local in-game audio data, the local in-game video data, the local real-time video stream, and the local real-time audio stream for reception by a host using a computer network and receiving data from a second computer system using the computer network. The apparatus further includes a video compositor for combining remote in-game video data, remote real-time video data, and local in-game video data to create a composite video stream, and an audio compositor for combining remote in-game audio data, remote real-time audio data, and local in-game audio data to create a composite audio stream. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention: 
           [0008]      FIG. 1A  is a block diagram of an exemplary computer system for sending real-time communications and player viewpoint data according to embodiments of the present invention. 
           [0009]      FIG. 1B  is a block diagram of an exemplary computer system for receiving and presenting real-time communications and player viewpoint data according to embodiments of the present invention. 
           [0010]      FIG. 2A  is a diagram of an exemplary display screen for computer gaming without virtual split screens. 
           [0011]      FIG. 2B  is a diagram of an exemplary display screen with virtual split screens for providing real-time communication and player viewpoint data according to embodiments of the present invention. 
           [0012]      FIG. 2C  is a diagram of an exemplary display screen with virtual pop-up windows for providing real-time communication and player viewpoint data according to embodiments of the present invention. 
           [0013]      FIG. 2D  is a diagram of an exemplary display screen with re-sized virtual split screens for providing real-time communication and player viewpoint data in a compact form according to embodiments of the present invention. 
           [0014]      FIG. 2E  is a diagram of an exemplary display screen with virtual split screens operating in various modes for providing real-time communication and player viewpoint data according to embodiments of the present invention. 
           [0015]      FIG. 2F  is a diagram of an exemplary display screen with virtual split screens for providing real-time communications and player viewpoint data according to embodiments of the present invention 
           [0016]      FIG. 3A  is a diagram of an exemplary network architecture for providing real-time communication and player viewpoint data using a view server and a peer-to-peer network topology according to embodiments of the present invention. 
           [0017]      FIG. 3B  is a diagram of an exemplary network architecture for providing real-time communication and player viewpoint data using a star network topology with a dedicated view server according to embodiments of the present invention. 
           [0018]      FIG. 4  is a flowchart depicting an exemplary sequence of computer implemented steps for providing in-game screen sharing and audio-video communication using virtual split screens for multiplayer gaming according to embodiments of the present invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    Reference will now be made in detail to several embodiments. While the subject matter will be described in conjunction with the alternative embodiments, it will be understood that they are not intended to limit the claimed subject matter to these embodiments. On the contrary, the claimed subject matter is intended to cover alternative, modifications, and equivalents, which may be included within the spirit and scope of the claimed subject matter as defined by the appended claims. 
         [0020]    Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. However, it will be recognized by one skilled in the art that embodiments may be practiced without these specific details or with equivalents thereof. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects and features of the subject matter. 
         [0021]    Portions of the detailed description that follows are presented and discussed in terms of a method. Although steps and sequencing thereof are disclosed in a figure herein (e.g.,  FIG. 4 ) describing the operations of this method, such steps and sequencing are exemplary. Embodiments are well suited to performing various other steps or variations of the steps recited in the flowchart of the figure herein, and in a sequence other than that depicted and described herein. 
         [0022]    Some portions of the detailed description are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer-executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. 
         [0023]    It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout, discussions utilizing terms such as “accessing,” “writing,” “including,” “storing,” “transmitting,” “traversing,” “associating,” “identifying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
       In-Game Virtual Split Screen With Peer-To-Peer Video Conferencing 
       [0024]    Embodiments of the present invention are drawn to exemplary computing devices generally having a network interface component, a video camera, and a microphone. The following discussion describes one such exemplary computing device. 
         [0025]    In the example of  FIG. 1A , the exemplary computer system  101  includes a central processing unit (CPU)  113 , memory  114 , and network streamer  115  for sending real-time audio and video communications and player viewpoint data to other players during multiplayer gaming. In the context of the present invention, “real-time” refers to a near-real-time or perceptually real-time constraint or delay. System  101  is associated with an exemplary user, “Player  1 ”. CPU  113  runs software applications such as games (e.g., game  104 ) and optionally an operating system. Memory  114  may comprise random access memory and/or read-only memory for storing applications and other data. A network streamer  115  allows computer system  101  to communicate with screen sharing server  103  used for sharing real-time communications and player viewpoint data with other on-line players. The network streamer  115  is coupled to network interface  116  to communicate with other computer systems, networks, or devices via an electronic communications network, including wired and/or wireless communication and including an Intranet or the Internet. 
         [0026]    In the embodiment of  FIG. 1A , microphone  105  and camera  106  are used to capture real-time audio and video data. Microphone  105  captures audio (e.g., voice chat) from the payer, and camera  106  captures video of the player. Audio data captured by microphone  105  is optionally encoded and/or compressed by audio encoder  109 , and video data captured by camera  106  is optionally encoded and/or compressed by video encoder  110 . Audio encoder  109  and video encoder  110  pass the encoded and/or compressed data to network streamer  115 . 
         [0027]    System  101  of  FIG. 1A  further comprises 3D driver  107  and audio driver  108 . 3D driver  107  interfaces with CPU  113  to generate graphics data suitable for display on a monitor or television, for example. 3D driver  107  accepts commands and data from CPU  113  and generates signals with suitable voltage, current, timing and/or demultiplexing to generate formatted graphics for display. In one illustrative example, during execution of game  104  by CPU  113 , 3D driver  107  receives instructions comprising graphics data from CPU  113  and translates the instructions to be encoded by video encoder  111 . According to some embodiments, 3D driver  107  receives graphics data from an operating system of computer system  101  and translates the instructions for display. 
         [0028]    Audio driver  108  receives audio data from CPU  113  and generates signals with suitable voltage, current, timing and/or demultiplexing to generate formatted audio for playback. In on illustrative example, during execution of game  104  by CPU  113 , audio driver  108  receives instructions comprising audio data from CPU  113  executing game  104  and translates the instructions to be encoded by audio encoder  112 . According to some embodiments, audio driver  108  receives audio data from an operating system of computer system  101  and converts the audio to a format suitable for playback. 
         [0029]    Network streamer  115  receives data from video encoders  110  and  111  and audio encoders  109  and  111  and prepares the data for transmission to another system or device using network interface  116 . Network interface  116  comprises an Ethernet or wireless network adapter for sending and receiving data over a computer network. Network streamer  115  receives audio and video data captured using microphone  105  and camera  106 , respectively. Network streamer  115  also receives in-game graphics and audio content from video encoder  111  and audio encoder  112  for transmission. 
         [0030]    Network streamer  115  prepares the in-game and real-world multimedia data for sharing with other connected devices, and may be configured to share the data over a network connection (e.g., network interface  116 ) using a peer-to-peer or star network topology. An exemplary peer-to-peer network topology for sharing in-game and real-world audio and video data is described in  FIG. 3A , and an exemplary star network topology for sharing in-game and real-world audio and video data is described in  FIG. 3B . Network streamer  115  may transmit the multimedia data using the WebRTC standard, for example. WebRTC is an API definition that supports light-weight applications for voice calling, video chat, and P2P file sharing and does not require internal or external plugins. 
         [0031]    Network interface  116  comprises a hardware network interface or wireless interface for connecting to a local area network, Intranet, or the Internet. Network interface  116  is used by network streamer  115  to transmit audio and video data to screen sharing server  103 . Network interface  116  is also sends data (e.g., game-state, player input, etc.) from game  104  to game server  102 , and also to receive data from game server  102  to be used by game  104  during execution. 
         [0032]      FIG. 1B  is a block diagram of an exemplary computer system  150  for receiving and reproducing real-time communications and in-game data (e.g., player viewpoint data) according to embodiments of the present invention. System  150  is associated with an exemplary user, Player  2 . In the example of  FIG. 1B , the exemplary computer system  150  includes a central processing unit (CPU)  164 , memory  163 , and network streamer  161  for sending real-time audio and video communications and player viewpoint data during multiplayer gaming. CPU  164  runs software applications such as games (e.g., game  104 ) and optionally an operating system. Memory  114  may comprise random access memory and/or read-only memory for storing applications and other data. 
         [0033]    A network streamer  161  enables computer system  150  to communicate with game server  102  and screen sharing server  103  for sharing real-time communications and player viewpoint data with other on-line players. Network streamer  161  is coupled to network interface  162  to communicate with other computer systems, networks, or devices via an electronic communications network, including wired and/or wireless communication and including a local area network, Intranet or the Internet. Real-time communication (e.g., video and/or voice chat) data received by network streamer  161  from screen sharing server  103  is passed to video decoder  157  and audio decoder  159  for processing (e.g., decoding, decompressing, etc.). Network streamer  161  also receives real-time game data from screen sharing server  103  comprising audio and/or video data based on in-game assets. The game data comprising video data is sent to video decoder  158  for processing, and the game data comprising audio data is sent to audio decoder  160 . The processed video data from video decoders  157  and  158  is sent to video composer  155 , and the processed audio from audio decoders  159  and  160  is sent to audio composer  156 . According to some embodiments, video compositor  155  and audio compositor  156  are a software component of an operating system. According to other embodiments, video compositor  155  and audio compositor  156  are hardware components located on a video card, sound card, processor, or motherboard, for example. 
         [0034]    Still with regard to  FIG. 1B , the processed video data from video decoders  157  and  158  is received by video compositor  155 . Video compositor  155  also receives local game data comprising video/graphics data for rendering in-game assets from 3D driver  154 . During execution of game  151  by CPU  164 , 3D driver  154  receives instructions comprising graphics data from CPU  164  executing game  151  and translates the instructions for display. Translated instructions are sent to video compositor  155  for generating virtual split screens comprising in-game and real-world (e.g., video chat) content for presentation on display  152 . Display  152  comprises a monitor, television, touchscreen, or projector, for example. 
         [0035]    Processed audio data from audio decoders  159  and  160  is received by audio compositor  156 . Audio compositor  156  also receives local game data comprising audio data based on in-game assets from audio driver  163 . Audio driver  163  receives audio data from CPU  113  and generates signals with suitable voltage, current, timing and/or demultiplexing to generate formatted audio for playback. During execution of game  151  by CPU  164 , audio driver  163  receives instructions comprising audio data from CPU  164  and translates the instructions for playback. The translated instructs are then sent to audio compositor  156  to combine in-game and real-world audio content for playback using speaker  153 . Speaker  153  comprises a multi-channel surround sound system, TV speakers, or headphones, for example. 
         [0036]    While the embodiment of  FIG. 1A  depicts an exemplary system for sending player viewpoint and communication data and the embodiment of  FIG. 1B  depicts an exemplary system for receiving player viewpoint and communication data, according to some embodiments, the systems are combined into a single system that is capable of both sending and receiving player viewpoint and communication data. 
       Displaying Virtual Split Screens Using Audio and Video Compositors 
       [0037]    With regard to  FIG. 2A , an exemplary display screen  200 A for computer gaming without virtual split screens is depicted according to embodiments of the present invention. As depicted, display screen  200 A displays game window  201 A running in a full-screen mode without split screens or pop-up windows. 
         [0038]    With regard to  FIG. 2B , an exemplary display screen  200 B with split screen window  202  for providing real-time communications and player viewpoint data is depicted according to embodiments of the present invention. Game window  201 B is displayed taking up a left portion of the display screen  200 B. Game window  201 B is resized so that split screen window  202  is displayed without obscuring or obstructing view of any in-game content. According to some embodiments, game window  201 B is zoomed to preserve the native aspect ratio and prevent distortion. In the example of  FIG. 2B , split screen window  202  comprises three virtual split screens. The split screens display a player viewpoint component (e.g., player viewpoint  203 ) comprising real-time, in-game content. The split screens also comprise a communication window (e.g., communications window  204 ) for displaying real-time video communications captured by a camera, with complementary audio captured using a microphone and played back through a speaker system. The communication windows can be disabled so that only player viewpoint data is displayed. According to some embodiments, the communication window portion of a split screen is only displayed while the associated user is actively communicating (e.g., speaking into a microphone). 
         [0039]    With regard to  FIG. 2C , an exemplary display screen  200 C with pop-up windows  205  and  207  for providing real-time communications and player viewpoint data is depicted according to embodiments of the present invention. In the example of  FIG. 2C , game window  201 C is not resized and some portions of the screen are obstructed by the pop-up windows. The pop-up windows may be resized to occupy more or less of the display screen. Similarly, the transparency/opaqueness of the windows may be adjusted to allow the player to view screen content behind the pop-up windows. The pop-up windows display a player viewpoint component (e.g., player viewpoint  206 A and  208 A) comprising real-time, in-game content. The pop-up windows also display a communication window (e.g., communication windows  206 B and  208 B) for displaying real-time audio and video communications captured by a camera and microphone. According to some embodiments, the pop-up windows are only displayed while the associated user is actively communicating (e.g., speaking into a microphone). In other embodiments, the pop-up windows are displayed only when the associated player is actively participating in the game and will not be displayed when the player is away (e.g., inactive) or disconnected. 
         [0040]    With regard to  FIG. 2D , a diagram of an exemplary display screen  200 D with compact virtual split screens for providing real-time communications and player viewpoint data is depicted according to embodiments of the present invention. In the example of  FIG. 2D , split screen window  209  has been resized to consume less screen space in comparison to the example of  FIG. 2B . Player viewpoint  210  and communication window  211  are displayed in a smaller form and game window  201 D is extended to fill more screen space. 
         [0041]    With regard to  FIG. 2E , an exemplary display screen  200 E with virtual split screens operating in various modes for providing real-time communications and/or player viewpoint data is depicted according to embodiments of the present invention. In the example of  FIG. 2E , split screen window  212  comprises three split screen windows  213 - 215 . Split screen window  213  has been configured to present player viewpoint and communications data from exemplary user “Player  2 ” and waits for Player  2  to join the game. When Player  2  participates in the game as an active player, split screen window  213  presents Player  2 &#39;s real-time communications and/or player viewpoint data. Split screen window  214  is configured to display both player viewpoint data as well as real-time communications from exemplary user “Player  3 ”. Split screen window  215  is operating in communication mode to display only communications data (e.g., video chat) from exemplary user “Player  4 ”. The display mode (e.g., communication mode, viewpoint mode, or both) of each split screen window may be changed at any time. 
         [0042]    With regard to  FIG. 2F , an exemplary display screen  200 F with virtual split screens for providing real-time communications and player viewpoint data is depicted according to embodiments of the present invention. In the example of  FIG. 2F , display screen  200 F is displaying a full-screen virtual split screen window comprising three player viewpoint windows and three communication windows. Communication windows  216 - 218  presents video chat from on-line players, and player viewpoint windows  219 - 221  display the associated in-game player viewpoint. According to some embodiments, the center portion of screen  200 F (e.g., communication window  217  and player viewpoint window  220 ) displays the activity of a local user. 
         [0043]    With regard to  FIG. 3A , a diagram of an exemplary network architecture for providing real-time communications and player viewpoint data using a peer-to-peer network is depicted according to embodiments of the present invention. Client systems  302 - 307  run games  308 - 313  and communicate game and communication (e.g., video chat) data to other peers using network  310 . Network  310  comprises a local area network, intranet, and/or the Internet. In the example of  FIG. 3A , Network  310  comprises a peer-to-peer network with nodes/clients  302 - 307 . Each client is connected to one or more peers to send and receive viewpoint and communication data. For example, client system  302  running game  308  generates player viewpoint data and captures player communication data using a camera and a microphone. The data is processed by a network streamer and communicated over network  301  to client/peer  305 . Client  305  receives the player viewpoint data and communication data and presents the viewpoint and communication data to the user using a display and speaker system. Client  305  sends the viewpoint data and communication data to client  303 , and so on, until all active peers have received the viewpoint data and communication data generated by clients  302 - 307 . 
         [0044]    With regard to  FIG. 3B , a diagram of an exemplary network architecture for providing real-time communications and player viewpoint data using a dedicated view server and a star network topology is depicted according to embodiments of the present invention. Client systems  316 - 325  run games  321 - 324  and communicate viewpoint and communication data to view server  326  using network  315 . Network  315  comprises a local area network, intranet, and/or the Internet. In the example of  FIG. 3B , Network  315  comprises a network having a star topology with clients  316 - 325  connected to a central node/hub  326 . The clients can send and receive viewpoint and communication data generated by other players using view server  326 . For example, client system  316  running game  321  and client  230  running game  325  generate player viewpoint data and capture player communication data using cameras and microphones. The data is processed by a network streamer and communicated over network  315  to view server  326 . Client  318  receives the player viewpoint data and player communication data from view server  326  using network  315 . The player viewpoint data is presented on a display connected to client  318  during execution of game  323  alongside in-game content rendered locally. The communication data is presented in a communication window on the display and voice audio is played through speakers connected to client  318 . 
         [0045]    With regard to  FIG. 4 , a flowchart depicting an exemplary sequence of computer implemented steps  400  for providing in-game screen sharing and audio-video communication using virtual split screens for multiplayer gaming is depicted according to embodiments of the present invention. At step  401 , a computer system running a copy of an online game generates player viewpoint data and captures player communication data using a camera and a microphone. At step  402 , the data is processed by a network streamer. At step  403 , the processed data is communicated over a computer network to a second computer system (e.g., a peer of view server). At step  404 , the client system receives the player viewpoint data and communication data and combines the data with locally rendered in-game content. At step  405 , the combined data is presented at the client system. 
         [0046]    Embodiments of the present invention are thus described. While the present invention has been described in particular embodiments, it should be appreciated that the present invention should not be construed as limited by such embodiments, but rather construed according to the following claims.