Patent Description:
A previously proposed arrangement is disclosed in <CIT>. Digital content distribution systems conventionally include a content server, a content player, and a communications network connecting the content server to the content player. The content server is configured to store digital content files, which can be downloaded from the content server to the content player. Each digital content file corresponds to a specific identifying title. The digital content file typically includes sequential content data, organized according to playback chronology, and may comprise audio data, video data, or a combination thereof.

Digital content distribution systems generally include a content server, a content player, and a communications network connecting the content server to the content player. The content server is configured to store (or provide access to) media content made available to end users. Media content may include, movies, videos, music, games, social applications, etc. The content player is configured to download and play a digital content, in response to a user request selecting the title for playback. The user is typically presented with a set of recommended titles for selection. Such recommendations are based primarily on previous operations and/or selections performed by the user.

To improve the recommendations, content players typically track metadata for user activities as the user is engaged with media content. Metadata is defined as the data providing information about one or more aspects of the data. Metadata is used to summarize basic information about data which can allow for tracking and working with specific data.

The following presents a simplified summary of one or more embodiments in order to provide a basic understanding of present technology. This summary is not an extensive overview of all contemplated embodiments of the present technology, and is intended to neither identify key or critical elements of all examples nor delineate the scope of any or all aspects of the present technology. Its sole purpose is to present some concepts of one or more examples in a simplified form as a prelude to the more detailed description that is presented later. In accordance with one or more aspects of the examples described herein, systems and methods are provided for generating post-engagement metadata.

In an aspect, an apparatus for generating post-engagement metadata is provided as recited in claim <NUM>.

In a second aspect, a method for generating post-engagement metadata by an entertainment system is provided as recited in claim <NUM>, and a corresponding non-transitory, computer-readable storage medium, having embodied thereon a program executable to perform the method is provided as recited in claim <NUM>.

These and other sample aspects of the present technology will be described in the detailed description and the appended claims that follow, and in the accompanying drawings, wherein:.

The subject disclosure provides techniques for generating post-engagement metadata, in accordance with the subject technology. Various aspects of the present technology are described with reference to the drawings. It can be evident, however, that the present technology can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing these aspects.

A streaming media service generally includes a content server, a content player, and a communications network connecting the content server to the content player. The content server is configured to store (or provide access to) media content made available to end users. Media content may include, movies, videos, music, games, social applications, etc. The content player can suggest media content to one or more users based on the interests of the users. The content player can also suggest media content to be viewed or played together with other users connected online as a social experience. Such interests may be determined by the viewing or search history of the users.

The present disclosure provides technology that allows greater and more accurate metadata regarding user behavior in engaging with an entertainment device. To generate this post-engagement metadata, the entertainment device will record and analyze user behavior after the user has stopped watching a video, listening to a track, or playing a game.

<FIG> illustrates a block diagram <NUM> of an example entertainment device <NUM>, according to one implementation of the invention. As shown, the entertainment device <NUM> includes, without limitation, an I/O interface <NUM> a processor <NUM>, a network interface <NUM>, a bus <NUM>, a memory <NUM>, and a storage <NUM>. The bus <NUM> connects the I/O interface <NUM>, the processor <NUM>, the network interface <NUM>, the memory <NUM>, and the storage <NUM>.

The I/O interface <NUM> connects I/O devices to the entertainment device <NUM>. For example, I/O devices include input devices <NUM> such as a keyboard, mouse, or remote control. A user of the entertainment device <NUM> can use one of the input devices <NUM> to enter various commands.

Example display / audio devices <NUM> include a monitor, television, tablet, or smartphone with a liquid crystal display (LCD), light-emitting diode (LED), or organic light-emitting diode (OLED) panel, etc. The entertainment device <NUM> can output media content on the display / audio devices <NUM> for the user to enjoy.

The entertainment device <NUM> connects to a content provider <NUM> over the network <NUM>. The network interface <NUM> is configured to transmit data to and from network <NUM>, such as for example, to download media content from and/or transmit sensor data to the server system <NUM>.

The processor <NUM> can include single central processing unit (CPU), multiple CPUs, a single CPU having multiple processing cores, etc. For example, the processor can <NUM> be a chip on a motherboard that can retrieve and execute programming instructions stored in the memory <NUM>.

One or more sensors devices <NUM> are connected to or integrate into the entertainment device <NUM>. The sensors <NUM> can include, for example, cameras, microphones, radars, Light Detection and Ranging (LIDAR) sensors, pressure sensors, wearable sensors, haptic sensors, or smart home appliances for detecting user location and/or activity. The sensors <NUM> provide various sensor data to the entertainment device <NUM>.

The sensor data is interpreted by the entertainment device <NUM> and/or the content provider <NUM>. Because the entertainment device <NUM> may have much lower processing power than the content provider <NUM>, computationally intensive sensor data interpretation may occur at the content provider <NUM>. In some implementations, the entertainment device <NUM> may perform rudimentary data interpretation locally, but send the sensor data for complex processing to the content provider <NUM> over the network <NUM>.

For example, cameras can provide sensor data that allows the entertainment device <NUM> to determine a variety of information about users and their environment. In some implementations, one or more of various image processing and/or machine vision algorithms known in the art can be used to interpret the sensor data from the cameras. For example, the entertainment device <NUM> can determine from the sensor data how many users are in a room, if any, and their user identity. The entertainment device <NUM> can determine from the sensor data what activities one or more users in the room are engaged in, such as for example, resting on couch, eating a meal, in conversation, etc. In another example, the entertainment device <NUM> can determine from the sensor data facial expressions and moods of the users. For example, the entertainment device <NUM> can determine the mood of a user if the sensor data shows that the user is smiling, frowning, excited, bored, etc..

Audio sensors such as microphones can provide sensor data that allows the entertainment device <NUM> to determine a variety of information about users and their environment. One or more of various speech and sound processing algorithms known in the art can be used to interpret the sensor data from the microphones. For example, the entertainment device <NUM> can determine from the sensor data voice commands. The audio sensors can also allow the entertainment device <NUM> to passively receive sensor data for determining user preferences, behavior patterns, moods, schedules, etc. For example, the entertainment device <NUM> can determine the mood of a user based on voice tonality and/or positive (e.g., "I like this song") or negative (e.g., "what else is on TV?") speech. For example, the entertainment device <NUM> can determine the schedule of a user based on speech (e.g., "I have a doctor's appointment tomorrow morning at 8am" or "I need to get to bed in an hour").

Wearable and haptic sensors can provide sensor data that allows the entertainment device <NUM> to determine user health and other body status information. For example, a smart watch can provide sensor data to the entertainment device <NUM> for heart rate of a user.

The sensors <NUM> may be integrated to the entertainment device <NUM> or can be external to the entertainment device. Sensors <NUM> located external to the entertainment device <NUM> are connected the entertainment device <NUM> via a wired cable or via a wireless connection such as WiFi or BLUETOOTH®.

The memory <NUM> can include any physical device used to temporarily or permanently store data or programs, such as various forms of random-access memory (RAM). The storage <NUM> may be a disk drive storage device. Although shown as a single unit, the storage <NUM> may be a combination of fixed and/or removable storage devices, such as magnetic disc drives, solid state drives (SSD), removable memory cards, optical storage, network attached storage (NAS), or a storage area-network (SAN). The storage <NUM> can include any physical device for non-volatile data storage such as a HDD or a flash drive. The storage <NUM> can often have greater capacity than the memory <NUM> and can be more economical per unit of storage, but can also have slower data transfer rates.

Storage <NUM>, such as a hard disk drive or solid-state (SSD) storage drive, may store media content locally on the entertainment device such as files for an operating system, applications, games, videos, audio, user preferences, logs, and any other data.

The memory <NUM> stores data to be conveniently accessed by the processor <NUM> such as a suggestion module <NUM>, a media content interface <NUM>, a metadata module <NUM>, and an operating system <NUM>. The operating system <NUM> is system software that manages computer hardware and software resources and provides common services for computer programs.

The media content interface <NUM> is a software application that allows a user to access, browse, search, and play media content such as movies, videos, music, games, social applications, etc. The media content may be locally available on the storage <NUM> of the entertainment device <NUM> or be available over the network <NUM> (e.g., stored at a content provider or a peer entertainment device).

In some implementations, the media content interface <NUM> may permit a user to enter search queries for transmission to the content provider via the network <NUM>. Content found and downloaded from the content provider or a peer entertainment device may be stored in storage <NUM> as buffered media content prior to being decoded and played by the media content interface <NUM>.

The metadata module <NUM> is an application that generates metadata for users of the entertainment device <NUM>. In some implementations, the metadata module <NUM> is also or alternatively included in the entertainment device <NUM>. The metadata module <NUM> not only generates metadata when a user is engaging with the entertainment device <NUM>, but also continues to generate metadata after the user stops engaging. The generated metadata can be stored in the storage <NUM> as content metadata <NUM>.

For example, the metadata module <NUM> can generate metadata based on the user history. The metadata can include user/session data, including data relating to activities undertaken by various users of a number of entertainment devices <NUM>, such as for example, which games were played, which videos were watched, and/or which music tracks were listened to, and a time and duration for each.

The metadata module <NUM> also generates metadata (i.e., post-engagement metadata) after the user stops engaging with the entertainment device <NUM>. In some implementations, the metadata module <NUM> can monitor for when the user terminates engagement with the entertainment device <NUM> and record the last engagement activity. For example, the user may stop playing a first person shooter game on the entertainment device <NUM> after a streak of game losses. A record of the first person shooter game and the losing streak can be stored as post-engagement metadata related to the user.

In some implementations, the metadata module <NUM> determines post-engagement user actions using sensor data from at least one sensor. For example, post-engagement actions by the user can include going to bed, eating dinner, leaving the house to socialize, going to work, etc. The metadata module <NUM> then analyzes the sensor data to determine a termination cause, based on the post-engagement user actions. The metadata module <NUM> generates the post-engagement metadata which stores the termination cause.

The suggestion module <NUM> is an application that analyzes social interactions, the sensor data from the sensors <NUM>, the user history, and the post-engagement metadata from the metadata module <NUM> to determine one or more suggested activities for encouraging continued user engagement with the entertainment device <NUM>.

For example, a user may stop using the entertainment device <NUM> because the user's parents are asking the user to finish school work. The metadata module <NUM> may determine from sensor data from a microphone the cause of the user's termination of engagement with the entertainment device <NUM>. The metadata module <NUM> stores the termination cause into the post-engagement metadata for the user. The suggestion module <NUM> may then suggest, either immediately or during a future occurrence of the termination cause, an educational tutorial video to the user to encourage continued engagement with the entertainment device <NUM>.

In another example, a first user of the entertainment device <NUM> may speak to a second user that he is hungry, and then the first user and the second user stops engaging with the entertainment device <NUM>. The metadata module <NUM> may determine from sensor data from a microphone the cause of the users' termination of engagement with the entertainment device <NUM>. The metadata module <NUM> stores the termination cause into the post-engagement metadata for the user. The suggestion module <NUM> may then suggest, either immediately or during a future occurrence of the termination cause, food delivery services for use by the first user and the second user to encourage continued engagement with the entertainment device <NUM>.

In a third example, a user of the entertainment device <NUM> may be bored with playing a video game alone. The metadata module <NUM> may determine from the user's social network messages, from after playing the game, that the user would like more social interactions with friends. The metadata module <NUM> stores the termination cause into the post-engagement metadata for the user. In some implementations, the suggestion module <NUM> may suggest engaging watching/playing the suggest media content with an online remote user connected over a network. For example, the online remote user can be an individual from a list of friends on a social network of one of the users of the entertainment device <NUM>. The online remote user may be chosen based on similar interests and/or schedules.

<FIG> illustrates an example methodology <NUM> for generating post-engagement metadata by an entertainment device. At step <NUM>, the entertainment device monitors for termination of engagement with the entertainment system by a first user.

At step <NUM>, the entertainment device records a last engagement activity before the termination of engagement.

At step <NUM>, the entertainment device determines post-engagement user actions by the first user based on sensor data from at least one sensor, after the termination of engagement. In some implementations, the post-engagement user actions are based on sensor data from a microphone for positive or negative voice reactions of the first user. The post-engagement user actions can be based on sensor data from communications a social networking application.

At step <NUM>, the entertainment device determines a termination cause, based on the post-engagement user actions.

At step <NUM>, the entertainment device generates post-engagement metadata for the first user comprising the termination cause.

In some implementations, the entertainment device detects for presence of at least one other user at a first location, determines availability data of physical resources for the first user and the at least one other user, where the physical resources include at least one of a display device, a console system, a set-top box, a mobile gaming device, or a tablet computer, and stores the availability data in the post-engagement metadata.

In some implementations, the entertainment device determines a physical resource distribution strategy based on the availability data, and outputs the physical resource distribution strategy on at least one connected display device or audio device.

In some implementations, the entertainment device determines a suggested action for continued engagement with the apparatus based post-engagement metadata, and outputs the suggested action on at least one connected display device or audio device.

In some implementations, the entertainment device locates a second user that is available on a remotely connected apparatus, where the second user shares a common engagement activity with the first user, and outputs a suggestion for the first user to connect to the second user to share the common engagement activity on at least one connected display device or audio device.

In some implementations, the entertainment device determines a schedule for the first user from at least one of an online social network, a personal calendar application, an electronic mail application, chat application, or text application of the first user, stores the schedule data in the post-engagement metadata.

In some implementations, the entertainment device monitors for voice cues in the sensor data, where the termination cause is further based on the voice cues.

<FIG> illustrates a schematic diagram of an example infrastructure 300A for generating post-engagement metadata. Media sources <NUM>, distributed through network <NUM>, can be of several types, such as internet streamed media, broadcast media, on-demand media supplied by a cable company, photos posted on a social network, etc. A unique piece of downloaded media indented to be shown in its entirety to a user as a single unit is referred to herein as a media segment.

Different entertainment devices are available for receiving the media content and displaying the media to users <NUM> a-f. Example entertainment devices include set-top box <NUM> connected to a television display, mobile phone <NUM>, game console <NUM>, laptop <NUM>, Personal Computer <NUM>, television <NUM> configured with an ID card <NUM>, etc. Additionally, users <NUM> a-f interact with the different entertainment devices in many ways, such as with remote control <NUM>, voice commands, mobile-phone keys, camera <NUM> to capture user images in order to perform motion detection, game controllers <NUM> a-b, keyboard, etc..

A content provider <NUM> can be a server in communication with the entertainment devices to provide media content to the entertainment devices. For example, the media content can be a videos, games, software applications, music, photos, documents, presentations, billboards, coupons, videoconferences, etc. In one embodiment, media sources <NUM> coordinate activities with the content provider <NUM> for the placement of the media content, such as providing metadata for the placement of interactive commercials.

<FIG> illustrates a block diagram of an example infrastructure 300B for generating post-engagement metadata, according to one implementation of the invention. As shown, the infrastructure <NUM> includes a content provider <NUM> and an entertainment device <NUM>, each connected to a network <NUM>.

In an example implementation, one or more entertainment devices <NUM> are located in each residential home or small business, while the content provider <NUM> exists across one or more data centers. A data center is a facility used to house computer systems and associated components, such as powerful data processing, telecommunications, and storage systems.

The entertainment device <NUM> connects to the content provider <NUM> through the network <NUM> via one or more Internet Service Providers (ISPs). An ISP is an organization that provides services for accessing and using the Internet. Internet Service Providers may be organized in various forms, such as commercial, community-owned, non-profit, or otherwise privately owned. Internet services typically provided by ISPs include Internet access, Internet transit, domain name registration, web hosting, Usenet service, and colocation. Each ISP typically serves large groups of Internet users in a geographic area.

The entertainment device <NUM> communicates with the content provider <NUM> over the network <NUM> to obtain media content (e.g., movies, videos, music, games, social applications, etc.). The entertainment device <NUM> includes at least a processor <NUM> and a memory <NUM> for storing data. In this particular example, the entertainment device <NUM> represents a processing device running a media content interface <NUM>.

The media content interface <NUM> is configured to obtain media content from the content provider <NUM>, to be rendered on a display <NUM> (e.g., a tablet, a smartphone, a monitor, a television, and/or a head-mounted display (HMD) for virtual reality (VR) or augmented reality (AR). The display <NUM> can additionally or alternatively include an audio output device such as a multimedia sound system or integrated speakers for outputting audio content.

The entertainment device <NUM> represents a set-top device connected to both the network <NUM> and the display <NUM> (e.g., a flat-panel television). Accordingly, entertainment device <NUM> is representative of digital cable boxes, digital video recorder (DVR) systems, video game consoles, and other media devices capable of connecting to a network <NUM> and receiving and playing back media streams provided by content provider <NUM>. In such a case, the disc player could connect to the content provider <NUM> and download interface components used to select and playback media streams. Further, the display <NUM> may itself be an integrated device capable of connecting to the network <NUM> playing back media streams supplied by the content provider <NUM>. For example, some television displays include integrated components used to connect to a streaming media service, video on demand services, or video sharing websites.

One or more input / output (I/O) devices <NUM> are connected (e.g., via wired cable or wireless connection such as WiFi or BLUETOOTH®) to or integrate into the he entertainment device <NUM>.

One or more sensors <NUM> are connected to or integrate into the entertainment device <NUM>. The sensor data is interpreted by the entertainment device <NUM> and/or the content provider <NUM>.

Although only one entertainment device is shown in <FIG>, in some implementations, the content provider <NUM> may serve a large number of entertainment devices <NUM>. The content provider <NUM> provides a computing system configured to transmit media content to the entertainment device <NUM> (as well as other entertainment devices not shown). For example, content provider <NUM> may include a web-server, database, and/or application server configured to respond to requests for web pages and/or streaming media files received from the entertainment device <NUM>. The media content itself may be distributed from the content provider <NUM> or through broader content distribution networks.

For example, in some implementations, the content provider <NUM> may allow users to authenticate themselves to the content provider <NUM> (e.g., using a username and password or using voice, fingerprint, or facial recognition). Once a given user is authenticated, the user may receive suggestions for activities from the content provider <NUM>. The entertainment device <NUM> and/or content provider <NUM> may use relationships between user activities to generate a set of suggested activities and serve the set of activities to the user via the display <NUM>. The entertainment device <NUM> downloads from the content provider <NUM> and plays one of the suggested activities. In addition to generating the set of suggested activities, the content provider <NUM> may also use relationships and between user activities and activity history of the user to generate other activity suggestions and recommendations, to present on the entertainment device <NUM>. The disclosure below in reference to <FIG> describes in more detail the generation of suggested actions for users of the home entertainment device <NUM>.

The content provider <NUM> can include any computing or processing device, such as for example, one or more computer servers. The content provider <NUM> can be a single device, multiple devices in one location, or multiple devices in multiple locations. The disclosure below in reference to <FIG> describes in more detail example content provider configurations.

<FIG> illustrates example usage of an entertainment device <NUM> for generating post-engagement metadata. As shown, at least one user <NUM> is able to view a television display <NUM>. The entertainment device <NUM> is connected to the television display <NUM> and one or more sensors <NUM>. In some implementations, the entertainment device <NUM> is connected to a network <NUM> such as a wide area network (WAN) such as the Internet, or a local area network (LAN). The entertainment device <NUM> can also be used "offline" without connecting to the network <NUM>.

The sensors <NUM> allow the entertainment device <NUM> to automatically determines how many people are in the area, whether they are registered users, and their user identities. In this case, entertainment device <NUM> determines that the user <NUM> is in the area.

Content manager <NUM> determines how to retrieve content for the user. In one embodiment, the content can be obtained from an internal media library <NUM>, or from the Internet via Content loader <NUM>. Content loader <NUM> downloads multimedia content via network <NUM>, which is buffered by content buffer <NUM>. Content loader <NUM> can retrieve content for storage in the media library <NUM> for later retrieval by the user. Display manager <NUM> receives the media from the media library and outputs the media into the display buffer <NUM>, which drives display <NUM>.

It is noted that the television display <NUM> is merely one example of multiple display device options for use with the present disclosure. Other display options include a monitor, television, tablet, or smartphone with a liquid crystal display (LCD), light-emitting diode (LED), or organic light-emitting diode (OLED) panel, etc. In some implementations, an audio output device (e.g., speakers) can be used in addition to the display <NUM> or in alternative to the display <NUM> to play audio media content for the user <NUM>.

User interface manager <NUM> receives input from user <NUM> which can come from a variety of devices, such as single-handed controller <NUM>, or other devices such as a two-hand controller, a mobile phone, a remote control, a keyboard, etc., as shown in <FIG>. The user input is communicated to the display manager <NUM> for further processing and updating of the video sent to display buffer <NUM>. The user input may also be communicated to result delivery module <NUM> for transmission over the network <NUM>.

<FIG> illustrates example usage of various input devices with an entertainment device for generating post-engagement metadata. The user <NUM> may command the entertainment device <NUM> to search for or suggest media content. For example, the user <NUM> may command the entertainment device <NUM> using an input device <NUM>. In some implementations, the input device <NUM> may be a remote controller, a game controller, a joystick, a mouse, a keyboard, a touchscreen, a trackpad, or other similar control device. For example, the user <NUM> may command the entertainment device <NUM> using the input device <NUM> to play a recently released feature film. In some implementations, the entertainment device <NUM> can use the sensors <NUM> to receive voice or gesture commands from the user <NUM>.

<FIG> illustrates a block diagram <NUM> of an example content provider <NUM> for generating post-engagement metadata. As shown, the content provider <NUM> includes a processor <NUM>, a network interface <NUM>, a memory <NUM>, and a storage <NUM>. The content provider <NUM> connects to multiple entertainment devices 509A, 509B over a network <NUM>.

Storage <NUM> includes media content <NUM>, a user history <NUM>, and content metadata <NUM>. Media content <NUM> includes a library of media content available for download or streaming by an entertainment device <NUM>. Accordingly, the media content <NUM> may include a collection of audio/video/game data encoded at various bitrates and stored on the content provider <NUM>.

The user history <NUM> is representative of one or more log entries which store user/session data, including data relating to activities undertaken by various users of a number of entertainment devices <NUM>. Such activities may include, e.g., playing a video game title, listening to a music track, viewing a media content, performing a search, viewing links, etc. The log entries may include, for example, a user identification (ID), media content played by the user, a timestamp of when the play started, and a timestamp of when the play ended. The log entries may include text of a search query entered by the user, and a timestamp of when the search query was received. The user history <NUM> may include user IDs of each user subscribing to the streaming media service, and may also include usernames, password data, user preferences and settings, and other user information.

The content metadata <NUM> in the storage <NUM> may include media content rollups, whether media content are available at various times, and other information relating to media content. For example, content metadata for videos may include automatically collected video metadata from software or a device such as a video camera. The metadata stores camera-created information such as aperture, shutter speed, GPS coordinates and more in the video file. The standard format is Exchangeable Information File Format (EXIF). Manually written video metadata provides additional information about the video content. Manual metadata can include transcripts of dialogue or conversations and other text descriptions. Such text entries are readable by computers and provide efficient searching.

The memory <NUM> can include any physical device used to temporarily or permanently store data or programs, such as various forms of random-access memory (RAM). The memory <NUM> stores various data and applications for quick access by the processor <NUM>. The memory <NUM> stores a social module <NUM>, sensor data <NUM>, a metadata module <NUM>, and a suggestion module <NUM>.

The social module <NUM> is an application that determines possible social interactions for users of different entertainment device <NUM>. In some implementations, the social module <NUM> analyzes calendar schedules, historical activity, preferences, and interests for a group of users to determine possible social activities between the group of users. The group of users may be friends or acquaintances on a social network. The group of users may also be strangers and have no prior social interactions. The social module <NUM> may predict which group of users may have the best probability of compatibility and therefore have increased engagement with their entertainment device <NUM>. For example, the social module <NUM> may determine that two specific users are friends on a social network and both enjoy playing first person shooter games on weeknights, and therefore group those two specific users together.

The sensor data <NUM> is received from the entertainment device <NUM> over the network <NUM>. Sensors connected to the entertainment device <NUM> provide various sensor data to the entertainment device <NUM> and/or to the content provider <NUM>. For example, the sensor data <NUM> from a camera or microphone can allow the content provider <NUM> to determine how many people are in a room and their identities.

The sensor data <NUM> is interpreted by the entertainment device <NUM> and/or the content provider <NUM>. Because the entertainment device <NUM> may have much lower processing power than the content provider <NUM>, computationally intensive sensor data interpretation may occur at the content provider <NUM>. In some implementations, the entertainment device <NUM> may perform rudimentary data interpretation locally, but send the sensor data for complex processing to the content provider <NUM> over the network <NUM>.

The metadata module <NUM> is an application that generates metadata for users of each entertainment device 509A, 509B. In some implementations, the metadata module <NUM> is also or alternatively included in each entertainment device 509A, 509B. The metadata module <NUM> not only generates metadata when a user is engaging with the entertainment device <NUM>, but also continues to generate metadata after the user stops engaging. The generated metadata can be stored in the storage <NUM> as content metadata <NUM>.

For example, the metadata module <NUM> can generate metadata based on the user history <NUM>. The metadata can include user/session data, including data relating to activities undertaken by various users of a number of entertainment devices <NUM>, such as for example, which games were played, which videos were watched, and/or which music tracks were listened to, and a time and duration for each.

In some implementations, the metadata module <NUM> determines post-engagement user actions using sensor data from at least one sensor. For example, post-engagement actions by the user can include going to bed, eating dinner, leaving the house to socialize, going to work, etc. The metadata module <NUM> then analyzes the sensor data to determine a termination cause, based on the post-engagement user actions. The metadata module <NUM> generates the post-engagement metadata which stores the termination cause. The generation of the post-engagement metadata by the metadata module <NUM> is described in more detail below in reference to <FIG>.

The suggestion module <NUM> is an application that analyzes the social interactions from the social module <NUM>, the sensor data <NUM>, the user history <NUM>, and the post-engagement metadata from the metadata module <NUM> to determine one or more suggested activities for continued user engagement with the entertainment device <NUM>.

For example, a user may stop using the entertainment device <NUM> because the user's parents are asking the user to finish school work. The metadata module <NUM> may determine from sensor data from a microphone the cause of the user's termination of engagement with the entertainment device <NUM>. The suggestion module <NUM> may then suggest an educational tutorial video to the user to encourage continued engagement with the entertainment device <NUM>.

In some implementations, the suggestion module <NUM> suggests engaging watching/playing the suggest media content with an online remote user 509B connected over a network. For example, the online remote user can be an individual from a list of friends on a social network of one of the users of the entertainment device <NUM>. The online remote user may be chosen based on similar interests and/or schedules.

<FIG> illustrates a block diagram of an example processing device <NUM>. The processing device <NUM> can include a Basic Input/Output System (BIOS) <NUM>, a memory <NUM>, a storage <NUM>, a network interface <NUM>, a processor <NUM>, a northbridge <NUM>, a southbridge <NUM>, and an input / output (IO) interface <NUM>.

The processing device <NUM> can be any apparatus for processing data, such as for example, a server computer (e.g., one of many rack servers in a data center), a personal computer, a laptop computer, a smartphone, a tablet, an entertainment console, a wearable device, or a set top box. The processor (e.g., central processing unit (CPU)) <NUM> can be a chip on a motherboard that can retrieve and execute programming instructions stored in the memory <NUM>. The processor <NUM> can be a single CPU with a single processing core, a single CPU with multiple processing cores, or multiple CPUs. One or more buses <NUM> can transmit instructions and application data between various computer components such as the processor <NUM>, memory <NUM>, storage <NUM>, and networking interface <NUM>.

The memory <NUM> can include any physical device used to temporarily or permanently store data or programs, such as various forms of random-access memory (RAM). The storage <NUM> can include any physical device for non-volatile data storage such as a HDD or a flash drive. The storage <NUM> can often have greater capacity than the memory <NUM> and can be more economical per unit of storage, but can also have slower data transfer rates.

The BIOS <NUM> can include a Basic Input/Output System or its successors or equivalents, such as an Extensible Firmware Interface (EFI) or Unified Extensible Firmware Interface (UEFI). The BIOS <NUM> can include a BIOS chip located on a motherboard of the processing device <NUM> storing a BIOS software program. The BIOS <NUM> can store firmware executed when the computer system is first powered on along with a set of configurations specified for the BIOS <NUM>. The BIOS firmware and BIOS configurations can be stored in a non-volatile memory (e.g., NVRAM) <NUM> or a ROM such as flash memory. Flash memory is a non-volatile computer storage medium that can be electronically erased and reprogrammed.

The BIOS <NUM> can be loaded and executed as a sequence program each time the processing device <NUM> is started. The BIOS <NUM> can recognize, initialize, and test hardware present in a given computing system based on the set of configurations. The BIOS <NUM> can perform self-test, such as a Power-on-Self-Test (POST), on the processing device <NUM>. This self-test can test functionality of various hardware components such as hard disk drives, optical reading devices, cooling devices, memory modules, expansion cards and the like. The BIOS can address and allocate an area in the memory <NUM> in to store an operating system. The BIOS <NUM> can then give control of the computer system to the OS.

The BIOS <NUM> of the processing device <NUM> can include a BIOS configuration that defines how the BIOS <NUM> controls various hardware components in the processing device <NUM>. The BIOS configuration can determine the order in which the various hardware components in the processing device <NUM> are started. The BIOS <NUM> can provide an interface (e.g., BIOS setup utility) that allows a variety of different parameters to be set, which can be different from parameters in a BIOS default configuration. For example, a user (e.g., an administrator) can use the BIOS <NUM> to specify clock and bus speeds, specify what peripherals are attached to the computer system, specify monitoring of health (e.g., fan speeds and CPU temperature limits), and specify a variety of other parameters that affect overall performance and power usage of the computer system.

The network interface <NUM> connects the processing device <NUM> to a network <NUM>. The network interface <NUM> is configured to transmit data to and from the network <NUM>. The network <NUM> can include a wide area network (WAN) such as the Internet, or a local area network (LAN). The network <NUM> can include an intranet, a storage area network (SAN), a personal area network (PAN), a metropolitan area network (MAN), a wireless local area network (WLAN), a virtual private network (VPN), a cellular or other mobile communication network, a wireless local area networking (WiFi) network, a BLUETOOTH® wireless technology connection, a near field communication (NFC) connection, any combination thereof, and any other appropriate architecture or system that facilitates the communication of signals, data, and/or messages.

The I/O interface <NUM> allows the processing device <NUM> to connect with I/O devices such as a keyboard, a mouse, a remote control, a controller, a microphone, a television, a head mounted display (HMD), and/or a monitor, etc..

The northbridge <NUM> can be a chip on the motherboard that can be directly connected to the processor <NUM> or can be integrated into the processor <NUM>. In some instances, the northbridge <NUM> and the southbridge <NUM> can be combined into a single die. The northbridge <NUM> and the southbridge <NUM>, manage communications between the processor <NUM> and other parts of the motherboard. The northbridge <NUM> can manage tasks that require higher performance than the southbridge <NUM>. The northbridge <NUM> can manage communications between the processor <NUM>, the memory <NUM>, and video controllers (not shown). In some instances, the northbridge <NUM> can include a video controller.

The southbridge <NUM> can be a chip on the motherboard connected to the northbridge <NUM>, but unlike the northbridge <NUM>, is not directly connected to the processor <NUM>. The southbridge <NUM> can manage input/output functions (e.g., audio functions, BIOS, Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI) bus, PCI eXtended (PCI-X) bus, PCI Express bus, Industry Standard Architecture (ISA) bus, Serial Peripheral Interface (SPI) bus, Enhanced Serial Peripheral Interface (eSPI) bus, System Management Bus (SMBus), etc.) of the processing device <NUM>. The southbridge <NUM> can be connected to or can include within the southbridge <NUM> the I/O interface <NUM>, Direct Memory Access (DMAs) controllers, Programmable Interrupt Controllers (PICs), and a real-time clock.

One or more input devices <NUM> can connect to the processing device <NUM>. The input device <NUM> can also be integrated into the processing device <NUM>. The input device <NUM> can be at least one of a game controller, a joystick, a mouse, a keyboard, a touchscreen, a trackpad, or other similar control device. The input device <NUM> allows a user to provide input data to the processing device <NUM>.

One or more display devices <NUM> can connect to the processing device <NUM>. The display device <NUM> can also be integrated into the processing device <NUM>. The display device <NUM> can be at least one of a monitor, a light-emitting display (LED) screen, a liquid crystal display (LCD) screen, a head mounted display (HMD), a virtual reality (VR) display, a augmented reality (AR) display, or other such output device. The display device <NUM> allows the processing device <NUM> to output visual information to a user.

The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein can be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The operations of a method or algorithm described in connection with the disclosure herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a non-transitory computer-readable medium. Non-transitory computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blue ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of non-transitory computer-readable media.

Claim 1:
An apparatus for generating post-engagement metadata, the apparatus comprising:
a network interface (<NUM>) configured to be connected to a content provider (<NUM>) via a network (<NUM>);
at least one sensor (<NUM>) configured to record sensor data at a first location;
a user interface manager (<NUM>) configured to receive user input from a first user;
a memory (<NUM>); and
a processor (<NUM>) configured to execute instructions stored in memory, wherein execution of the instructions by the processor is configured to:
detect (<NUM>) that the first user has terminated engagement with the apparatus;
record (<NUM>) a last engagement activity before first user is detected as having terminated engagement with the apparatus;
determine (<NUM>) that the first user has performed post-engagement actions at the first location based on the sensor data from the at least one sensor recorded after the first user terminates engagement with the apparatus;
determine (<NUM>) a termination cause based on the last engagement activity and the post-engagement actions; and
generate (<NUM>) post-engagement metadata for the first user, the generated post-engagement metadata including the termination cause,
wherein the processor of the apparatus is configured to perform rudimentary data interpretation locally, but further configured to send the sensor data for computationally intensive sensor data interpretation to the content provider for processing over the network.