MULTITASKING EXPERIENCES WITH INTERACTIVE PICTURE-IN-PICTURE

A user interface (“UI”) includes a personalized home screen that can be brought up at any time from any experience provided by applications, games, movies, television, and other content that is available on a computing platform such as a multimedia console using a single button press on a controller, using a “home” gesture, or using a “home” voice command. The home screen features a number of dynamically maintained visual objects called tiles that represent the experiences available on the console. An application can be “snapped” to the application that fills the PIP so that the snapped application renders into a separate window that is placed next to the UI for the filled application. The user interface is further adapted so that the user can quickly and easily switch focus between the tiles in the home screen and resume an experience in full screen.

BACKGROUND

Computing platforms typically support a user interface (“UI) that enables users to interact with a platform. Computing platforms such as multimedia consoles have evolved to include more features and capabilities and provide access to an ever increasing array of entertainment, information, and communication options. As a result, there exists a need for UIs that provide a full set of features and capabilities while still being easy to use that enable users to get the most out of their computing platforms while maintaining a satisfying and rich user experience.

SUMMARY

A user interface (“UI”) includes a personalized home screen that can be brought up at any time from any experience provided by applications, games, movies, television, and other content that is available on a computing platform such as a multimedia console using a single button press on a controller, using a “home” gesture, or using a “home” voice command. The personalized home screen features a number of visual objects called tiles that represent the experiences available on the console. The tiles are dynamically maintained on the personalized home screen as their underlying applications run. Within the larger UI, one of the tiles on the personalized home screen is configured as a picture-in-picture (“PIP”) display that can be filled by the graphical output of an application that is currently running. Other tiles show shortcuts to the most recently used and favorite applications. An application can be “snapped” to the application that fills the PIP so that the snapped application renders into a separate window that is placed next to the UI for the filled application. That way, the user can readily engage in multitasking experiences with the snapped and filled applications both in the personalized home screen and in full screen. The user interface is further adapted so that the user can quickly and easily switch focus between the tiles in the personalized home screen and resume an experience in full screen.

DETAILED DESCRIPTION

FIG. 1shows an illustrative computing environment100in which the present multitasking experiences with interactive picture-in-picture (“PIP”) may be implemented. An entertainment service102can typically expose applications (“apps”)104, games106, and media content108such as television shows and movies to a user110of a multimedia console112over a network such as the Internet114. Other providers103may also be in the environment100that can provide various other services such as communication services, financial services, travel services, news and information services, etc.

Local content116, including apps, games, and/or media content may also be utilized and/or consumed in order to provide a particular user experience in the environment100. As shown inFIG. 1, the user is playing a particular game title118. The game118may execute locally on the multimedia console112, be hosted remotely by the entertainment service102, or use a combination of local and remote execution in some cases using local or networked content/apps/games as needed. The game118may also be one in which multiple other players120with other computing devices can participate.

The user110can typically interact with the multimedia console112using a variety of different interface devices including a camera system122that can be used to sense visual commands, motions, and gestures, and a headset124or other type of microphone or audio capture device. In some cases a microphone and camera can be combined into a single device. The user may also utilize a controller126(shown in enlarged view in the lower left ofFIG. 1) to interact with the multimedia console112. The controller126may include a variety of physical controls including joysticks128and130, a directional pad (“D-pad”)132, “A,” “B,” “X,” and “Y” buttons134,136,138, and140respectively, menu button142and view button144. A centrally-located button, referred to in this application as the center button146, is also provided. One or more triggers and/or bumpers (not shown) may also be incorporated into the controller126. The user110can also typically interact with a user interface148that is shown on a display device150such as a television or monitor.

It is emphasized that the number of controls utilized and the features and functionalities supported by the controls in the controller126can vary from what is shown inFIG. 1according to the needs of a particular implementation. In addition in the description that follows various button presses and control manipulations are described. It is noted that those actions are intended to be illustrative. For example, the user may actuate a particular button or control in order to prompt a system operating on the multimedia console112to perform a particular function or task. It will be appreciated that the particular mapping of controls to functions can vary from that described below according the needs of a particular implementation. As used here, the term “system” encompasses the various software (including the software operating system (“OS”)), hardware, and firmware components that are instantiated on the multimedia console and its peripheral devices in support of various user experiences that are provided by the console.

A home app152executes on the multimedia console112in this illustrative example. As shown inFIG. 2, the home app152is configured to provide a variety of user experiences205when operating as a part of the system running on the multimedia console112. Some of the experiences may execute simultaneously to implement multitasking in some cases. The user experiences205include a personalized home screen210that is shown on the UI148(FIG. 1). The personalized home screen210can be arranged to support a number of graphic tiles including a resume tile215that employs one or more interactive PIPs. Shortcuts to most recently used apps/games220can also be included in the personalized home screen210as tiles. Pins225are tiles that are user-selectable for inclusion in the personalized home screen and can generally represent, for example, shortcuts to apps/games that the user particularly likes and/or uses the most often.

The tiles in the personalized home screen can be configured as “live” tiles in some implementations so that they show or represent activity of any underlying running app/game right on the personalized home screen. For example, a news app could display news headlines, sports scores, and the like from the personalized home screen. In some cases, the tiles can be configured to enable some interactivity with the underlying application through user interaction with the tile itself. For example, a tile could be configured to expose user-accessible controls to change tracks or playlists in an underlying music app that is running on the multimedia console.

The user experiences205further include a one button to home experience230in which the user110can push the center button146(FIG. 1) on the controller to launch or return to the personalized home screen at anytime during a session on the multimedia console112. The user can also make a particular “home” gesture that is captured by the camera system122and associated gesture recognition system that is implemented on the multimedia console112, or speak a voice command such as “home” or its non-English language equivalents, or other words or word combinations to launch or return to the personalized home screen. Simple and fast switching of focus235from one app/game to another is also supported. In-experience contextual menus240may be invoked by the user110in order to launch menus that are specifically configured for a user experience provided by the context of a given app or game. Some apps/games may be snapped to the PIP (indicated by reference numeral245) in the personalized home screen. Various notifications250may also be supported in the personalized home screen.

Each of the user experiences205are described in more detail below.

FIG. 3shows a screenshot300of a UI that includes an illustrative personalized home screen305. It is emphasized that the particular personalized home screen shown in this example is intended to be illustrative and the home screens utilized in various implementations of the present multitasking experiences with interactive PIP can vary from what is shown by content, format, and layout according to particular needs. In addition, in the screenshot300and those that follow, the UIs shown have been simplified for clarity of exposition as black and white line drawings.

The personalized home screen305inFIG. 3shows that the user110is currently consuming a movie that is displayed in the large resume tile302that is implemented as an interactive PIP within the larger UI. As the movie continues, the resume tile302is continuously refreshed so that the user110can watch the movie, use transport controls such as fast ahead, pause, audio mute, bring up menus and go to full screen, etc., all while interacting with the rest of the objects that are provided on the personalized home screen305. In some implementations, more than one PIP can be utilized and a given PIP can be implemented to be within another PIP (i.e., a PIP within a PIP) and the user110can interact with and/or control each of the various PIPs and their experiences from the personalized home screen. In addition, a given PIP does not have to be mapped on a one-to-one basis to an app/game so that it can be used to support multiple running applications. In some implementations, instead of representing an experience from a running app/game a PIP can be configured to represent a link for a launching an app/game. For example, upon start up of the multimedia console112, a PIP could be used to display a launch tile for the last app/game that was running in the resume tile302before the console was powered down.

Below the resume tile302is a row of four tiles, in this illustrative example, that represent the most recently used apps/games, referred to here as the MRU tiles304. The particular apps/games that are included in MRU tiles304can be expected to change over time as the user110launches and closes apps/games during the course of a session. The MRU tiles304can also be used in some implementations to represent either or both links for launching their respective underlying apps/games and live, currently executing applications that are running in the background (an example of an application that is running in the background while shown in the MRU tiles304is provided below in the text accompanyingFIG. 21). When an MRU tile is representing a currently executing app/game in some cases, the user can interact with the tile to control the experience such a bringing up an in-experience menu, pause/resume a game or movie, or invoke some other feature. In addition, the MRU tiles themselves can be configured to support one or more PIPs into which running apps/games may render experiences.

Next to the MRU tiles304, in this illustrative example, are several rows of tiles which comprise pins306. The pins306can represent the apps/games that the user110likes the most and/or uses the most frequently and be used as launch tiles. Typically, the system and/or home app is configured to enable the user110to pick which apps/games are included as pins on the personalized home screen305. Alternatively, the system or home app may automatically populate some or all of the pins for the user110. For example, the system/home app may apply various rules or heuristics to determine which apps/games are included as pins or analyze usage statistics, user-expressed preferences, user behaviors, or the like when populating tiles in the pins306on the personalized home page305. When a pin is selected and activated by the user, its underlying app or game can be launched. In addition, in some implementations one or more of the pins can be configured to represent a currently executing app/game with user controllability (e.g., experience control, menus, etc.) and/or implement one or more PIPs, as with the MRU tiles described above. Other apps, games, and other content can typically be browsed, selected, and launched from the menu bar308that is located above the resume tile302.

In this illustrative example, the user110has employed the controller126(FIG. 1), for example by manipulating the D-pad or joystick, to select a game tile310from among the pins306(the selection indicator is a rectangle with a thick border in this example). By activating a button on the controller (e.g., the “A” button134), the game is launched (here, a boxing game) which then completely fills the UI supported by the multimedia console as shown in the screenshot400inFIG. 4. In some implementations, the game will restart from the point where the user was last playing. However, it is expected that the restart behavior can vary by application and it is typically up to the app/game developer to decide how a particular app/game will operate when launched as a pin. For example, some apps/games may restart from the user's last point of interaction, while others will start anew or at another point.

FIG. 5is a screenshot500that shows an example of an in-experience contextual menu505that may be brought up by the user110during gameplay of the boxing game shown in screenshot400inFIG. 4. For example, the user110can press the menu button142on the controller126in order to invoke the menu505. Calling up an in-experience menu in this particular example will pause the gameplay and give the user110several menu choices, as shown. Here, the user110has selected the “restart” option with the controller and could restart the boxing game by pressing the “A” button134, for example. As with the pin described above, the choices, content, and behaviors provided by a particular in-experience menu are typically matters of design choice for the app/game developer. Thus, for example, some multiplayer games might not support a pause feature through the in-experience menu.

If the user110wishes to go back to the personalized home screen from the filled game screen shown inFIG. 5, then the user can press the center button146, for example, on the controller126. As noted above, the center button146can be configured to bring the user back to the personalized home screen from any experience on the multimedia console's UI at anytime.FIG. 6is a screenshot600that shows the UI after returning to the personalized home screen305from the filled game screen inFIG. 5. Now the large resume tile302shows the game experience (with the displayed in-experience menu) that the user110just left. The previous occupant of the resume tile, the movie, moves down to the first position (i.e., the leftmost position as indicated by reference numeral605) in the MRU tiles304. In this particular example, the in-experience menu is up and the gameplay is paused. Typically, if the user had not paused the gameplay by calling up the in-experience menu before going back to the personalized home screen305, the game would continue to run as it would in full screen, and the resume tile would display the ongoing gameplay. However, the particular behavior exhibited when running in the resume tile302can vary by application in accordance with the developer's design choices.

As shown inFIG. 6, the user110has employed the controller126to move the selection indicator to the resume tile302. When the user presses the “A” button134, for example on the controller126, the boxing game will then resume and fill the UI completely as shown in the screenshot700inFIG. 7. The particular behavior of an app/game when resuming is again a matter of design choice for the developer and can vary. In this example, the boxing game has closed its in-experience menu and resumes gameplay.

FIG. 8is a screenshot800that shows a “snap app” button805being selected by the user110(FIG. 1) using the controller126. The snap app button805enables the user to launch an app/game that provides an experience that executes simultaneously with whatever is running in the resume tile302. That experience renders into a window that is located, i.e., “snapped” to the side of the filled app's UI so that the user can multitask by interacting with both the filled app and the snapped app. The term “filled app” refers to an app/game that is configured to be capable of rendering into substantially the full extent of the UI. It is anticipated that some apps/games will only be configured to run as filled apps, others will only be configured to run as snapped apps, while yet other apps/games will be configured to run as either snapped or filled apps.

When the user110selects the snap app button805and presses the “A” button134, for example, on the controller126, a snap app menu905opens on the UI as shown in the screenshot900inFIG. 9. The snap app menu905lists various snappable experiences provided by apps and games in a scrollable filmstrip910, as shown. In this illustrative example, the user has selected an icon for an app915titled “Halo Waypoint.” App915is an example of a “hub app” that functions as a central location for content and information related to a particular gaming experience. When the user presses the “A” button134, for example, on the controller126the app915launches on the UI as a snapped experience, as shown in the screenshot1000inFIG. 10. Here, the snapped app915provides various player statistics/information as well as several selectable buttons (e.g., “launch atlas,” “store,” and “games”) that enable the user110to navigate to various experiences that will render into the snapped app UI window.

FIG. 10shows the experience from the snapped app915being rendered into a window that is snapped to left side of the filled app which, in the example, is the boxing game310. In alternative implementations, the snapped app can be located elsewhere in the UI. Both the snapped app915and the filled app run at the same time so that the gameplay in the boxing game continues while the hub app provides its own interactive experiences for the user. As with the in-experience menus for filled apps, if the user presses the menu button142on the controller126, an in-experience menu (not shown inFIG. 10) having particular context for the snapped app915is brought up.

If the user110uses the center button146to go back to the personalized home screen at this point, then the resume tile302is broken down into two smaller sub-tiles1105and1110as shown in the screenshot1100inFIG. 11. By providing the two sub-tiles in the resume tile, the user110can simply and quickly switch focus between the experiences provided by the snapped hub app and the boxing game. An app/game that has focus is typically the target of user inputs from the system such as controller events and events associated with motion-capture/gesture-recognition and voice commands.

As shown inFIG. 11, the user110has employed the controller126to move the selection indicator to the larger sub-tile1110in the resume tile302. When the user presses the “A” button134, for example, on the controller126the boxing game310will then resume as shown in the screenshot1200inFIG. 12and receive focus. In addition to focus switching from the personalized home screen, the system and/or home app152may be configured so that center button146may be double tapped to switch focus between a filled and snapped app without first going to the personalized home screen.

As shown in the screenshot1300inFIG. 13when the waypoint app915is snapped in the personalized home screen, the snap app button805(FIG. 8) is replaced with a “close snap” button1305. Here, the user110has employed the controller126to move the selection indicator to the close snap button1305. When the user presses the “A” button134, for example, on the controller126the snapped app915closes and the experience is removed from the resume tile as shown in the screenshot1400inFIG. 14. It is noted that the term “closes” can mean different things for different apps/games depending on developer design choices. In some cases, an app/game may still run in the background even though it is not actively rendering into the UI as a snapped or filled app in the personalized home screen. An example of this background behavior is presented below in the text accompanyingFIG. 21below.

As shown inFIG. 14, the previously snapped app915moves to the first position in the row of MRU tiles304below the resume tile302. A decoration1405(shown in an enlarged view) is provided in the first MRU tile to indicate the visual state that the app will take when resumed. In this case, the app915will resume as a snapped app when re-launched (e.g., when the user110selects the MRU tile and presses the “A” button134, for example, on the controller126) as shown in the screenshot1500inFIG. 15.

FIG. 15also shows an illustrative example of a notification1505being displayed on the UI for the boxing game310. In some implementations, a notification can provide some initial information about it. In this example, the notification1505shows that it is an incoming call from a caller on a VoIP (Voice over Internet Protocol) service for example supplied by a provider103(FIG. 1). The user110can interact with the notification1505by pressing the center button146, for example, on the controller126. Instead of going to the personalized home page as usual, here the center button press will take the user110to a notification center1605as shown in the screenshot1600inFIG. 16. The notification center1605shows a number of options (indicated by reference numeral1610) for the active notification1615as well as enables the user110to get information about notifications1620that may have been missed. Responding to a notification can take the user to either a filled or a snapped experience according to app/game design.

In this example, the user110has positioned the selection indicator and pressed the “A” button134, for example, on the controller126in order to answer the call. This action invokes a link to start up a VoIP app1705which replaces the boxing game on the UI as shown in the screenshot1700inFIG. 17. The user110can then participate in the call through interaction with the VoIP app1705.

If the user presses the center button146, for example, on the controller126, the personalized home screen is again brought up on the UI as shown in screenshot1800inFIG. 18. As noted above, when the user presses the menu button142, for example, on the controller126a menu will be displayed. The menu shown will depend on which tile the user happens to be on in the UI. In this example, the VoIP app1705is selected so an in-experience menu1905that has context for the VoIP app is displayed on the UI, as shown in the screenshot1900inFIG. 19.

In this example, the in-experience menu1905provides several ways to interact with the call including muting, ending, snapping, pinning, and getting call details. As shown, the user110has selected “snap.” When the “A” button, for example, is pressed, the VoIP app1705is presented at the side of the UI (in this example, on the left side) as a snapped app that renders its experience into a smaller PIP2005and the previous application that was running, the boxing game310becomes the filled app as shown in the screenshot2000inFIG. 20. It is noted that the option to snap an app is typically only provided as a menu choice when an app is designed to support both snapped and filled configurations.

If the user110uses the center button146to go back to the personalized home screen at this point, and chooses to close the snapped app using the close snap button1305as described above, then the snapped VoIP app1705closes and moves to the first position in row of MRU tiles304below the resume tile302. This behavior is shown in the screenshot2100inFIG. 21. As noted above, the term “closes” can mean different things for different apps/games. In this example, the VoIP app1705continues to run after being closed and will continue to transmit video and audio even though the VoIP app is no longer part of a visible experience on the UI. Other apps/games having communication features may be designed to exhibit similar background operation behaviors in some cases. A decoration2105(shown in an enlarged view) is provided in the first MRU the to indicate that the VoIP app1705is still active.

The system can be configured to inform currently running apps/games as to which of the various PIPs are being utilized to a support their experiences when the personalized home screen is being displayed. That way each running app/game has the option to adjust its experience depending on which PIP it is rendering into. For example as shown inFIG. 21, the VoIP app1705can be configured as a launch tile when placed in the row of MRU tiles304. However, the VoIP app could also be configured to render into another PIP (not shown) that is supported on the MRU tile in some implementations. The VoIP app could, for example, show live video in the tile (that is scaled but otherwise the same as when rendered into the resume tile), a timer showing the elapsed time of the call, an invocable interactive menu, and the like.

While some apps and games may render their normal experience into all tiles the same way, other may change the way they render their experiences based on the size, location, number of PIPs currently being displayed, and/or other criteria being utilized on a given personalized home screen. For example, if rendering into a relatively small PIP on a live tile (e.g., on an MRU tile or a pin), an application may chose to simplify or modify what is rendered compared to what it may render when it has a larger PIP to work with such as with the resume tile302or with a PIP in a snapped experience. Alternatively, the application could choose to render something that is different from its normal output such as advertising, an attract screen that is designed to catch a user's attention, or other objects if, for example, a tile is not particularly appropriate or suited to normal output, or as a result of a developer's design choice.

FIG. 22shows an illustrative layered software architecture2200that may be used to implement various aspects of the present multitasking experiences with interactive PIP. The software architecture2200may be adapted for use by the system running on the multimedia console112(FIG. 1) and similar systems operating on other computing platforms and devices. A XAML-based UI layer2205provides a library of XAML (eXtensible Application Markup Language) methods and routines that can be accessed by the home app152when the personalized home screen is drawn into the UI. The system further exposes an API (application programming interface) layer2210that includes an animation API2215that enables apps/games to be rendered during the transitions to and from the personalized home screen. Other system components2220may also be utilized in some implementations to facilitate the various features and functionalities described here. For example, it will be appreciated that the XAML-based UI layer can be built on top of various sub-systems in the layer2220such as the DirectX render system.

FIG. 23is a flowchart of an illustrative method2300by which various aspects of the present multitasking experiences with interactive PIP may be implemented. Unless specifically stated, the methods or steps shown in the flowchart and described below are not constrained to a particular order or sequence. In addition, some of the methods or steps thereof can occur or be performed concurrently and not all the methods or steps have to be performed in a given implementation depending on the requirements of such implementation. Some methods or steps may also be optionally utilized.

At step2305, the user110(FIG. 1) clicks the center button146on the controller126, or alternatively, makes a gesture that is recognized by a gesture recognition system on the multimedia console102as a “home” gesture, speaks a voice command such as “home” to create a user input event that instructs the system to open the user's personalized home screen. At step2310, the system will open the personalized home page in response to the user input event. The system exposes the state of the system including all running apps/games to the home app152(FIG. 1) at step2315. At step2320, the XAML-based UI library is exposed so that the home app152can draw using Direct Composition (“DComp”) surfaces provided by the apps/games. The system further exposes the animation API2215(FIG. 22) to the home app so it may render app/game transitions to and from the personalized home screen at step2325. Thus, the home app can redirect the video output from the running apps/games and manipulate it as needed to fit the spaces on the personalized home screen. For example, such manipulation can including scaling, sizing, repositioning, etc.

FIG. 24is an illustrative functional block diagram of the multimedia console112shown inFIG. 1. The multimedia console112has a central processing unit (CPU)2401having a level 1 cache2402, a level 2 cache2404, and a Flash ROM (Read Only Memory)2406. The level 1 cache2402and the level 2 cache2404temporarily store data and hence reduce the number of memory access cycles, thereby improving processing speed and throughput. The CPU2401may be configured with more than one core, and thus, additional level 1 and level 2 caches2402and2404. The Flash ROM2406may store executable code that is loaded during an initial phase of a boot process when the multimedia console112is powered ON.

A graphics processing unit (GPU)2408and a video encoder/video codec (coder/decoder)2414form a video processing pipeline for high speed and high resolution graphics processing. Data is carried from the GPU2408to the video encoder/video codec2414via a bus. The video processing pipeline outputs data to an A/V (audio/video) port2440for transmission to a television or other display. A memory controller2410is connected to the GPU2408to facilitate processor access to various types of memory2412, such as, but not limited to, a RAM.

The multimedia console112includes an I/O controller2420, a system management controller2422, an audio processing unit2423, a network interface controller2424, a first USB (Universal Serial Bus) host controller2426, a second USB controller2428, and a front panel I/O subassembly2430that are preferably implemented on a module2418. The USB controllers2426and2428serve as hosts for peripheral controllers2442(1) and2442(2), a wireless adapter2448, and an external memory device2446(e.g., Flash memory, external CD/DVD ROM drive, removable media, etc.). The network interface controller2424and/or wireless adapter2448provide access to a network (e.g., the Internet, home network, etc.) and may be any of a wide variety of various wired or wireless adapter components including an Ethernet card, a modem, a Bluetooth module, a cable modem, or the like.

System memory2443is provided to store application data that is loaded during the boot process. A media drive2444is provided and may comprise a DVD/CD drive, hard drive, or other removable media drive, etc. The media drive2444may be internal or external to the multimedia console112. Application data may be accessed via the media drive2444for execution, playback, etc. by the multimedia console112. The media drive2444is connected to the I/O controller 2420 via a bus, such as a Serial ATA bus or other high speed connection (e.g., IEEE 1394).

The system management controller2422provides a variety of service functions related to assuring availability of the multimedia console112. The audio processing unit2423and an audio codec2432form a corresponding audio processing pipeline with high fidelity and stereo processing. Audio data is carried between the audio processing unit2423and the audio codec2432via a communication link. The audio processing pipeline outputs data to the A/V port2440for reproduction by an external audio player or device having audio capabilities.

The front panel I/O subassembly2430supports the functionality of the power button2450and the eject button2452, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of the multimedia console112. A system power supply module2436provides power to the components of the multimedia console112. A fan2438cools the circuitry within the multimedia console112.

The CPU2401, GPU2408, memory controller2410, and various other components within the multimedia console112are interconnected via one or more buses, including serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include a Peripheral Component Interconnects (PCI) bus, PCI-Express bus, etc.

When the multimedia console112is powered ON, application data may be loaded from the system memory2443into memory2412and/or caches2402and2404and executed on the CPU2401. The application may present a graphical user interface that provides a consistent user experience when navigating to different media types available on the multimedia console112. In operation, applications and/or other media contained within the media drive2444may be launched or played from the media drive2444to provide additional functionalities to the multimedia console112.

The multimedia console112may be operated as a standalone system by simply connecting the system to a television or other display. In this standalone mode, the multimedia console112allows one or more users to interact with the system, watch movies, or listen to music. However, with the integration of broadband connectivity made available through the network interface controller2424or the wireless adapter2448, the multimedia console112may further be operated as a participant in a larger network community.

When the multimedia console112is powered ON, a set amount of hardware resources are reserved for system use by the multimedia console operating system. These resources may include a reservation of memory (e.g., 16 MB), CPU and GPU cycles (e.g., 5%), networking bandwidth (e.g., 8 kbps), etc. Because these resources are reserved at system boot time, the reserved resources do not exist from the application's view.

In particular, the memory reservation preferably is large enough to contain the launch kernel, concurrent system applications, and drivers. The CPU reservation is preferably constant such that if the reserved CPU usage is not used by the system applications, an idle thread will consume any unused cycles.

With regard to the GPU reservation, lightweight messages generated by the system applications (e.g., pop-ups) are displayed by using a GPU interrupt to schedule code to render pop-ups into an overlay. The amount of memory needed for an overlay depends on the overlay area size and the overlay preferably scales with screen resolution. Where a full user interface is used by the concurrent system application, it is preferable to use a resolution independent of application resolution. A scaler may be used to set this resolution such that the need to change frequency and cause a TV re-sync is eliminated.

After the multimedia console112boots and system resources are reserved, concurrent system applications execute to provide system functionalities. The system functionalities are encapsulated in a set of system applications that execute within the reserved system resources described above. The operating system kernel identifies threads that are system application threads versus gaming application threads. The system applications are preferably scheduled to run on the CPU2401at predetermined times and intervals in order to provide a consistent system resource view to the application. The scheduling is to minimize cache disruption for the gaming application running on the console.

When a concurrent system application requires audio, audio processing is scheduled asynchronously to the gaming application due to time sensitivity. A multimedia console application manager (described below) controls the gaming application audio level (e.g., mute, attenuate) when system applications are active.

Input devices (e.g., controllers2442(1) and2442(2)) are shared by gaming applications and system applications. The input devices are not reserved resources, but are to be switched between system applications and the gaming application such that each will have a focus of the device. The application manager preferably controls the switching of input stream, without knowledge of the gaming application's knowledge and a driver maintains state information regarding focus switches.

FIG. 25is a simplified block diagram of an illustrative computer system2500such as a PC, client device, or server with which the present multitasking experiences with interactive PIP may be implemented. Computer system2500includes a processing unit2505, a system memory2511, and a system bus2514that couples various system components including the system memory2511to the processing unit2505. The system bus2514may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory2511includes read only memory (“ROM”)2517and random access memory (“RAM”)2521. A basic input/output system (“BIOS”)2525, containing the basic routines that help to transfer information between elements within the computer system2500, such as during startup, is stored in ROM2517. The computer system2500may further include a hard disk drive2528for reading from and writing to an internally disposed hard disk (not shown), a magnetic disk drive2530for reading from or writing to a removable magnetic disk2533(e.g., a floppy disk), and an optical disk drive2538for reading from or writing to a removable optical disk2543such as a CD (compact disc), DVD (digital versatile disc), or other optical media. The hard disk drive2528, magnetic disk drive2530, and optical disk drive2538are connected to the system bus2514by a hard disk drive interface2546, a magnetic disk drive interface2549, and an optical drive interface2552, respectively. The drives and their associated computer readable storage media provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the computer system2500. Although this illustrative example shows a hard disk, a removable magnetic disk2533, and a removable optical disk2543, other types of computer readable storage media which can store data that is accessible by a computer such as magnetic cassettes, flash memory cards, digital video disks, data cartridges, random access memories (“RAMs”), read only memories (“ROMs”), and the like may also be used in some applications of the present multitasking experiences with interactive PIP. In addition, as used herein, the term computer readable storage medium includes one or more instances of a media type (e.g., one or more magnetic disks, one or more CDs, etc.). For purposes of this specification and the claims, the phrase “computer-readable storage media” and variations thereof, does not include waves, signals, and/or other transitory and/or intangible communication media.

A number of program modules may be stored on the hard disk, magnetic disk2533, optical disk2543, ROM2517, or RAM2521, including an operating system2555, one or more application programs2557, other program modules2560, and program data2563. A user may enter commands and information into the computer system2500through input devices such as a keyboard2566and pointing device2568such as a mouse. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, trackball, touchpad, touch screen, touch-sensitive module or device, gesture-recognition module or device, voice recognition module or device, voice command module or device, or the like. These and other input devices are often connected to the processing unit2505through a serial port interface2571that is coupled to the system bus2514, but may be connected by other interfaces, such as a parallel port, game port, or USB. A monitor2573or other type of display device is also connected to the system bus2514via an interface, such as a video adapter2575. In addition to the monitor2573, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. The illustrative example shown inFIG. 25also includes a host adapter2578, a Small Computer System Interface (“SCSI”) bus2583, and an external storage device2576connected to the SCSI bus2583.

The computer system2500is operable in a networked environment using logical connections to one or more remote computers, such as a remote computer2588. The remote computer2588may be selected as another personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer system2500, although only a single representative remote memory/storage device2590is shown inFIG. 25. The logical connections depicted inFIG. 25include a local area network (“LAN”)2593and a wide area network (“WAN”)2595. Such networking environments are often deployed, for example, in offices, enterprise-wide computer networks, intranets, and the Internet.

When used in a LAN networking environment, the computer system2500is connected to the local area network2593through a network interface or adapter2596. When used in a WAN networking environment, the computer system2500typically includes a broadband modem2598, network gateway, or other means for establishing communications over the wide area network2595, such as the Internet. The broadband modem2598, which may be internal or external, is connected to the system bus2514via a serial port interface2571. In a networked environment, program modules related to the computer system2500, or portions thereof, may be stored in the remote memory storage device2590. It is noted that the network connections shown inFIG. 25are illustrative and other means of establishing a communications link between the computers may be used depending on the specific requirements of an application of multitasking experiences with interactive PIP. It may be desirable and/or advantageous to enable other types of computing platforms other than the multimedia console112to implement the present multitasking experiences with interactive PIP in some applications.

FIG. 26shows an illustrative architecture2600for a computing platform or device capable of executing the various components described herein for multitasking experiences with interactive PIP. Thus, the architecture2600illustrated inFIG. 26shows an architecture that may be adapted for a server computer, mobile phone, a PDA (personal digital assistant), a smartphone, a desktop computer, a netbook computer, a tablet computer, GPS (Global Positioning System) device, gaming console, and/or a laptop computer. The architecture2600may be utilized to execute any aspect of the components presented herein.

The architecture2600illustrated inFIG. 26includes a CPU2602, a system memory2604, including a RAM2606and a ROM2608, and a system bus2610that couples the memory2604to the CPU2602. A basic input/output system containing the basic routines that help to transfer information between elements within the architecture2600, such as during startup, is stored in the ROM2608. The architecture2600further includes a mass storage device2612for storing software code or other computer-executed code that is utilized to implement applications, the file system, and the operating system.

The mass storage device2612is connected to the CPU2602through a mass storage controller (not shown) connected to the bus2610. The mass storage device2612and its associated computer-readable storage media provide non-volatile storage for the architecture2600. Although the description of computer-readable storage media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available computer storage media that can be accessed by the architecture2600.

By way of example, and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable media includes, but is not limited to, RAM, ROM, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read only memory), Flash memory or other solid state memory technology, CD-ROM, DVDs, HD-DVD (High Definition DVD), BLU-RAY, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the architecture2600.

According to various embodiments, the architecture2600may operate in a networked environment using logical connections to remote computers through a network. The architecture2600may connect to the network through a network interface unit2616connected to the bus2610. It should be appreciated that the network interface unit2616also may be utilized to connect to other types of networks and remote computer systems. The architecture2600also may include an input/output controller2618for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus (not shown inFIG. 26). Similarly, the input/output controller2618may provide output to a display screen, a printer, or other type of output device (also not shown inFIG. 26).

It should be appreciated that the software components described herein may, when loaded into the CPU2602and executed, transform the CPU2602and the overall architecture2600from a general-purpose computing system into a special-purpose computing system customized to facilitate the functionality presented herein. The CPU2602may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the CPU2602may operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions may transform the CPU2602by specifying how the CPU2602transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the CPU2602.

Encoding the software modules presented herein also may transform the physical structure of the computer-readable storage media presented herein. The specific transformation of physical structure may depend on various factors, in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable storage media, whether the computer-readable storage media is characterized as primary or secondary storage, and the like. For example, if the computer-readable storage media is implemented as semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable storage media by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. The software also may transform the physical state of such components in order to store data thereupon.

In light of the above, it should be appreciated that many types of physical transformations take place in the architecture2600in order to store and execute the software components presented herein. It also should be appreciated that the architecture2600may include other types of computing devices, including hand-held computers, embedded computer systems, smartphones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the architecture2600may not include all of the components shown inFIG. 26, may include other components that are not explicitly shown inFIG. 26, or may utilize an architecture completely different from that shown inFIG. 26.

FIG. 27shows illustrative functional components of the camera system122that may be used as part of a target recognition, analysis, and tracking system2700to recognize human and non-human targets in a capture area of a physical space monitored by the camera system without the use of special sensing devices attached to the subjects, uniquely identify them, and track them in three-dimensional space. The camera system122may be configured to capture video with depth information including a depth image that may include depth values via any suitable technique including, for example, time-of-flight, structured light, stereo image, or the like. In some implementations, the camera system122may organize the calculated depth information into “Z layers,” or layers that may be perpendicular to a Z-axis extending from the depth camera along its line of sight.

As shown inFIG. 27, the camera system122includes an image camera component2705. The image camera component2705may be configured to operate as a depth camera that may capture a depth image of a scene. The depth image may include a two-dimensional (2D) pixel area of the captured scene where each pixel in the 2D pixel area may represent a depth value such as a distance in, for example, centimeters, millimeters, or the like of an object in the captured scene from the camera. In this example, the image camera component2705includes an IR light component2710, an IR camera2715, and a visible light RGB camera2720that may be configured in an array, as shown, or in an alternative geometry.

Various techniques may be utilized to capture depth video frames. For example, in time-of-flight analysis, the IR light component2710of the camera system122may emit an infrared light onto the capture area and may then detect the backscattered light from the surface of one or more targets and objects in the capture area using, for example, the IR camera2715and/or the RGB camera2720. In some embodiments, pulsed infrared light may be used such that the time between an outgoing light pulse and a corresponding incoming light pulse may be measured and used to determine a physical distance from the camera system122to a particular location on the targets or objects in the capture area. Additionally, the phase of the outgoing light wave may be compared to the phase of the incoming light wave to determine a phase shift. The phase shift may then be used to determine a physical distance from the capture device to a particular location on the targets or objects. Time-of-flight analysis may be used to indirectly determine a physical distance from the camera system122to a particular location on the targets or objects by analyzing the intensity of the reflected beam of light over time via various techniques including, for example, shuttered light pulse imaging.

In other implementations, the camera system122may use structured light to capture depth information. In such an analysis, patterned light (i.e., light displayed as a known pattern such as a grid pattern or a stripe pattern) may be projected onto the capture area via, for example, the IR light component2710. Upon striking the surface of one or more targets or objects in the capture area, the pattern may become deformed in response. Such a deformation of the pattern may be captured by, for example, the IR camera2715and/or the RGB camera2720and may then be analyzed to determine a physical distance from the capture device to a particular location on the targets or objects.

The camera system122may utilize two or more physically separated cameras that may view a capture area from different angles, to obtain visual stereo data that may be resolved to generate depth information. Other types of depth image arrangements using single or multiple cameras can also be used to create a depth image. The camera system122may further include a microphone2725. The microphone2725may include a transducer or sensor that may receive and convert sound into an electrical signal. The microphone2725may be used to reduce feedback between the camera system122and the multimedia console112in the target recognition, analysis, and tracking system2700. Additionally, the microphone2725may be used to receive audio signals that may also be provided by the user110to control applications such as game applications, non-game applications, or the like that may be executed by the multimedia console112.

The camera system122may further include a processor2730that may be in operative communication with the image camera component2705over a bus2740. The processor2730may include a standardized processor, a specialized processor, a microprocessor, or the like that may execute instructions that may include instructions for storing profiles, receiving the depth image, determining whether a suitable target may be included in the depth image, converting the suitable target into a skeletal representation or model of the target, or any other suitable instruction. The camera system122may further include a memory component2740that may store the instructions that may be executed by the processor2730, images or frames of images captured by the cameras, user profiles or any other suitable information, images, or the like. According to one example, the memory component2740may include RAM, ROM, cache, Flash memory, a hard disk, or any other suitable storage component. As shown inFIG. 27, the memory component2740may be a separate component in communication with the image capture component2705and the processor2730. Alternatively, the memory component2740may be integrated into the processor2730and/or the image capture component2705. In one embodiment, some or all of the components2705,2710,2715,2720,2725,2730,2735, and2740of the capture device122are located in a single housing.

The camera system122operatively communicates with the multimedia console112over a communication link2745. The communication link2745may be a wired connection including, for example, a USB (Universal Serial Bus) connection, a Firewire connection, an Ethernet cable connection, or the like and/or a wireless connection such as a wireless IEEE 802.11 connection. The multimedia console112can provide a clock to the camera system122that may be used to determine when to capture, for example, a scene via the communication link2745. The camera system122may provide the depth information and images captured by, for example, the IR camera2715and/or the RGB camera2720, including a skeletal model and/or facial tracking model that may be generated by the camera system122, to the multimedia console112via the communication link2745. The multimedia console112may then use the skeletal and/or facial tracking models, depth information, and captured images to, for example, create a virtual screen, adapt the user interface, and control apps/games2750.

A motion tracking engine2755uses the skeletal and/or facial tracking models and the depth information to provide a control output to one more apps/games2750running on the multimedia console112to which the camera system122is coupled. The information may also be used by a gesture recognition engine2760, depth image processing engine2765, and/or operating system2770.

The depth image processing engine2765uses the depth images to track motion of objects, such as the user and other objects. The depth image processing engine2765will typically report to the operating system2770an identification of each object detected and the location of the object for each frame. The operating system2770can use that information to update the position or movement of an avatar, for example, or other images shown on the display150, or to perform an action on the user interface.

The gesture recognition engine2760may utilize a gestures library (not shown) that can include a collection of gesture filters, each comprising information concerning a gesture that may be performed, for example, by a skeletal model (as the user moves). The gesture recognition engine2760may compare the frames captured by the camera system112in the form of the skeletal model and movements associated with it to the gesture filters in the gesture library to identify when a user (as represented by the skeletal model) has performed one or more gestures. Those gestures may be associated with various controls of an application and direct the system to open the personalized home screen as described above. Thus, the multimedia console112may employ the gestures library to interpret movements of the skeletal model and to control an operating system or an application running on the multimedia console based on the movements.

In some implementations, various aspects of the functionalities provided by the apps/games2750, motion tracking engine2755, gesture recognition engine2760, depth image processing engine2765, and/or operating system2770may be directly implemented on the camera system122itself.