English:
[0001] Consumers of digital media content may discover new content for consumption via a variety of mechanisms. For example, some mechanisms involve a user actively seeking media via preexisting listings, such as by consulting an electronic programming guide. Other approaches may involve tracking a user's media consumption behavior, and then providing recommendations that relate to content consumed by the user.
[0002] Embodiments are disclosed that relate to providing digital content recommendations based upon data acquired via monitoring a digital content consumption use environment. For example, one embodiment provides a computing device configured to receive sensor data from a sensor system including one or more sensors configured to acquire the sensor data by monitoring a use environment. The computing device is further configured to detect a user present in the use environment via the sensor data, identify the user, recognize a current state of each of one or more use environment state features associated with the user via the sensor data, and store the current state of each of the one or more use environment state features. The computing device is further configured to detect a triggering condition for a selected use environment state feature, and, upon detecting the triggering condition, provide information related to the selected use environment state feature based upon one or more of the current state of the use selected use environment state feature and a previously-stored state of the selected use environment state feature.
[0004] FIGS. 1A and IB show an example embodiment of a use environment for monitoring user activity.
[0005] FIG. 2 schematically shows a use system for providing recommendations according to an embodiment of the present disclosure.
[0006] FIG. 3 shows a process flow depicting an embodiment of a method for monitoring one or more state features of a use environment according to an embodiment of the present disclosure.
[0007] FIG. 4 schematically shows a computing device according to an embodiment of the present disclosure.
[0008] Various mechanisms may be used to provide recommendations and/or advertisements for products, media content, and the like that may be of interest to a consumer. For example, various online services (e.g., electronic commerce services) may provide recommendations based on previous and/or current user interaction(s) with the service. However, such mechanisms utilize intentional, conscious user interactions with digital content to produce the recommendations.
[0009] Thus, embodiments are disclosed herein that relate to the use of one or more environmental sensors to monitor and understand a variety of user activity, features, etc. in a substantially unobtrusive manner, and thus to provide recommendations in response to a potentially richer set of environmental information. In other words, instead of relying upon one or more users to describe, define, etc. what is "going on" in the environment, it may be desirable to passively detect such information. As one non-limiting example, acoustic sensors may be used to detect and identify music being experienced, and to provide recommendations based on the detected music. As opposed to typical approaches that may rely upon a user to manually effect recommendation mechanism(s) (e.g., via text search, via audio capture mechanism, etc.), such a configuration may potentially provide recommendation(s) in a substantially more unobtrusive and intuitive manner.
[0010] FIG.1A illustrates an example use environment 100 for providing recommendations based upon passive sensing of user activity. Use environment 100 includes a sensor system 102 configured to acquire sensor data. Though environment 100 is illustrated as room within a private residence, it will be appreciated that such an environment is presented for the purpose of example, and is not intended to be limiting in any manner.
[0011] As illustrated, sensor system includes one or image sensors 104 (e.g., 2D image sensors and/or 3D depth sensors) and one or more audio sensors 106 (e.g., a microphone or directional microphone array). The use of such sensors may allow the ability to recognize positioning and/or shape of one or more state features of environment 100, as described in more detail below. In other embodiments, sensor system 102 may include additional and/or different sensor(s) without departing from the scope of the present disclosure.
[0012] As used herein, the term "state feature" refers to any characteristic of use environment 100 and/or of constituent item(s) (e.g., users, pets, furniture and other objects, etc.) thereof that is detectable via one or more sensors of sensor system 102. As non-limiting examples, state features may include, but are not limited to, a physical feature of any one or more users (e.g., body size and shape), an activity performed by any one or more users, and/or a device (e.g., mobile phone, tablet computing device, media player, gaming console, etc.) being operated by any one or more users. In other embodiments, state features may include characteristics of one or more media items (e.g., audio and/or visual media content), such as an identity of the media content item, identity of a constituent component of the content item (e.g., soundtrack content item(s)), a device accessing the content item, and a current access position (e.g., playback position) of the content item. It will be appreciated that these state features are presented for the purpose of example, and are not intended to be limiting in any manner.
[0013] Sensor system 102 may be communicatively coupled in some embodiments (e.g., via one or more wireless and/or wired connections) to computing device 108, which, although illustrated as a video game console, may have different configurations (e.g., a desktop computer, laptop computer, notebook computer, etc.) in other embodiments. Further, although illustrated as discrete devices, it will be appreciated that various components of sensor system 102 and computing device 108 may be incorporated into a single device (e.g., mobile computing device) in yet other embodiments. Additionally, sensor data also may be acquired by the computing device 108 from other devices having one or more sensors, such as a mobile device carried by a user. For ease of understanding, sensor system 102 and computing device 108 will be discussed herein as operating cooperatively, though it will be appreciated that the described functionalities may be provided by sensor system 102 and/or computing device 108 in various embodiments without departing from the scope of the present disclosure.
[0014] Using the sensor data acquired via sensor system 102 and/or via sensor(s) of other connected device(s), computing device 108 may monitor one or more state features of use environment 100 and provide information, such as metadata, media item and/or activity recommendations, advertisements, etc., related to such state features via an output device (e.g. display device 110) communicatively coupled to computing device 108. Sensor system 102 may be configured to monitor activity within use environment 100, regardless of whether or not such activity involves the direct or indirect utilization of computing device 108. For example, even if a particular media content item (e.g., audiovisual content, video game or other interactive content, etc.) is presented via a media presentation device other than computing device 108, the sensor system may be configured to monitor such activity.
[0015] As one non-limiting example depicted in FIG. 1A, user 112 is dancing while listening to audio 114 provided by an audio device 116 (e.g., a stereo). In this particular scenario, recognizable state features include, but are not limited to, user activity 118 (e.g., dancing), audio 114, and/or an identity of device 116 providing audio 114. User activity 118 may be detected, for example, using one or more modeling and/or tracking mechanism(s) (e.g., skeletal tracking pipeline(s)), inertial devices coupled to the user (e.g., via mobile devices, inertial sensors, etc.), etc. Detection of audio 114 may be provided, for example, via audio sensors 106 (e.g., via audio fingerprint identification), whereas device 116 may be detected via image sensors 104 (e.g., via shape recognition) and/or via audio sensors 106 (e.g., via beam forming mechanism(s)). As mentioned above, state features may include any features of environment 100 detectable via sensor system 102, and it will therefore be appreciated that such features are presented for the purpose of example, and are not intended to be limiting in any manner.
[0016] In some situations, continual identification of any one or more state features via sensor system 102 may utilize significant resources (e.g., power, processor bandwidth, network bandwidth, etc.) even when a user that has selected to be passively sensed is not in the use environment. Accordingly, sensor system 102 may be configured to passively monitor environment 100 until detecting activity, at which time the sensor system may begin actively monitoring the environment. Thus, when no activity (e.g., user presence, content item playback, etc.) is detected, sensor system 102 and/or computing device 108 may be configured to acquire sensor data at a lower "quality" (e.g., a lower bitrate), and to acquire sensor data at a higher "quality" once activity is detected. By acquiring sensor data at a lower bit rate in order to detect activity while utilizing sensor data of a higher bitrate for state feature identification, such a configuration may leverage the richer data set provided by sensor system 102 while potentially utilizing a decreased amount of resources, among other potential benefits. It will be understood that data acquisition at the higher bitrate may continue even after the triggering activity has ceased. For example, if a user walks into a room and then lies down, higher bitrate data acquisition may continue while the user is present in the environment even if the original triggering activity (e.g. user movement) has ceased.
[0017] Activity sufficient to trigger data acquisition and analysis at the higher "quality" level may be detected in any suitable manner. For example, in some embodiments, such activity may correspond to motion meeting a motion threshold (e.g., movement beyond a threshold velocity and/or displacement), sound meeting a sound threshold (e.g., sound exceeding a particular decibel level), a presence of one or more users in the use environment (e.g., via skeletal tracking, humanoid shape detection, etc.).
[0018] Upon detection of activity, computing device 108 may be configured to identify any one or more state features of environment 100. For example, audio 114 state features may be identified by acquiring an audio sample and comparing the audio sample to known audio information. In some embodiments, computing device 108 may be communicatively coupled to one or more remote services, and may be configured to provide at least some of the acquired audio sample to the remote service for identification. In other embodiments, computing device 108 may be configured to provide such identification instead of, or in addition to, the remote service(s). Similarly, other state features (e.g., user activity 118, audio device 116) may be identified by capturing one or more image samples and comparing the image sample(s) to known image information.
[0019] In some embodiments, sensor data and/or samples thereof representing any one or more state features (e.g., user body size and/or shape) may be compared to information regarding one or more previous states of the particular state feature(s). In this way, one or more state features may be tracked over time, thereby potentially providing improved recommendation performance by providing an understanding of state progression, as opposed to merely an understanding of individual state(s). It will be appreciated that, in order to provide such progression monitoring, computing device 108 may be configured to store state(s) (e.g., via device 108 and/or remote computing devices, services, etc.) of any one or more state features. Such storage may be provided, for example, only if the user has provided authorization to be monitored by the sensor system. Determination of such authorization may include, but is not limited to, recognizing previously-defined authorization information (e.g., user profile(s)) and/or recognizing authorization gesture(s) (e.g., verbal, physical, etc.) performed by the user. Authorization may be controlled, for example, via one or more opt-in and/or opt-out mechanisms, though other configurations are possible without departing from the scope of the present disclosure. In some embodiments, computing device 108 may provide substantially anonymous monitoring where previous state information is not stored or otherwise utilized, though it will be appreciated that such anonymous monitoring may not provide a suitably rich data set, as compared to other approaches.
[0020] Upon recognizing any one or more state features of environment 100, computing device 108 may be configured to determine whether or not of any of said state features meet or exceed a corresponding triggering condition. In other words, such triggering condition(s) determine whether or not information (e.g., recommendations) related to the triggering state feature(s) is to be provided.
[0021] As one non- limiting example of such triggering threshold detection, change(s) in any one or more physical feature(s) of user 1 12 meeting a triggering threshold (e.g., change in body size beyond a threshold level) may be identified via analysis of (e.g., comparison with) stored state information. As another example of a triggering threshold, audio 114 from audio device 116 may be identified after a temporal threshold (e.g., 30 seconds of audio) has been met, so as to ensure that audio 114 is actively being listened to, and is not merely being "scanned" or "sampled". In some embodiments, one or more triggering conditions may include an explicit user input (e.g., via gesture, via input device, etc.) requesting recommendation(s). Though each state feature has been described as having a single corresponding triggering threshold, it will be appreciated that any one or more state features may correspond to any one or more triggering conditions having any one or more triggering thresholds without departing from the scope of the present disclosure.
[0022] As mentioned above, upon recognizing satisfaction of any one more triggering conditions, computing device 108 may be configured to provide information 119 (e.g., recommendation(s)) related to the corresponding state feature(s). Where the state feature(s) include one or more media content items (e.g., audio 114), information 119 related to the state features(s) may include, but is not limited to, information 120 regarding a related content item, auxiliary information 122 for the content item, and a link 125 to acquire (e.g., download and/or purchase) an instance of the content item. In some embodiments, information 119 may include one or more mechanism(s) 124 to add the detected content item to a content playlist (e.g., online music service playlist, playlist of content items recently detected via the sensor system, etc.).
[0023] Information 120 regarding a related content item may include, for example, recommendations for related content item(s), mechanism(s) to acquire such related content items, mechanism(s) to add the related content items to a playlist, and/or other suitable information. Although illustrated as text-based information presented via a "stand-alone" recommendation user interface via display device 110, it will be appreciated that information 120 may include any information regarding related item(s), and may be provided via any suitable mechanism(s), without departing from the scope of the present disclosure.
[0024] Where the state feature(s) include a user activity (e.g., activity 118), computing device 108 may be configured to provide recommendations (e.g., advertisements, promotional offers, etc.) for goods and/or services related to the user activity. Furthermore, in some embodiments, related content items may be determined based at least on detected user activity 118. For example, computing device 108 may be configured to detect that user 112 is dancing, and may recommend dance music in response. Continuing with the dancing example, computing device 108 may be configured to determine the "style" of dance, and may provide recommendations based on the determined style. For example, recognizing user activity 118 as approximating a "disco" dance style may effect recommendations of disco-type music.
[0025] Where the state feature(s) include one or more physical features of user 112, such information may include, for example, information describing a change in the physical feature (e.g., notification of weight gain), a recommendation of goods or services related to the physical feature (e.g., advertisement for wellness goods or services), and a recommendation of one or more changes to user behavior (e.g., instructions for more exercise).
[0026] In some embodiments, information 119 may include auxiliary information 122 corresponding to the detected state feature(s). As illustrated, information 122 may include content identification (e.g., artist, album, etc.), related visual information (e.g., cover art, music visualization(s), etc.), and other auxiliary information (e.g., lyrics, etc.) related to the detected state feature(s). Presentation of information 122 may be desirable, for example, for identifying and/or learning about the detected state feature(s).
[0027] In some scenarios, the detection and understanding of multiple state features may enable computing device 108 to potentially locate more relevant recommendations. For example, by understanding multiple, related state features (e.g., audio 114 being consumed while user 112 dances), context may be provided for such state features, which may be used in determining recommendations. For example, detection of both audio 114 and user activity 118 may trigger the provision of information 119 (e.g., dance lessons advertisement 126) related to both audio 114 and activity 118 (e.g., dance music), whereas information 119 related to audio 114 (e.g., information 122 and/or information 120) may be triggered if user activity 118 is not detected along with audio 114. Such scenarios are presented for the purpose of example, and are not intended to be limiting in any manner.
[0028] Turning now to FIG. IB, environment 100 of FIG. 1A is illustrated in another use case scenario involving a user 150 viewing media content 152 (e.g., a movie) including audio 154 (e.g., soundtrack song(s)) via computing device 156, shown as a tablet computing device. In this scenario, media content 152, constituent audio 154 of media content 152, and operating computing device 156 are each examples of state features detectable via sensor system 102. Audio 154 may be detected, for example, via audio sensors 106, and media content 152 and computing device 156 may be detected by image sensors 104 and/or audio sensors 106, for example, by acquiring image sample(s) of computing device 156 and comparing the sample(s) to known image information. Additionally, computing device 156 may comprise sensors located thereon that may provide sensor data to computing device 108. Sensor information from computing device 156, and/or from other suitable mobile devices (e.g. a smart phone), may be acquired and analyzed by computing device 156 even when the user is not within the environment monitored by sensor system 102.
[0029] In the illustrated example, computing device 108 identifies both media content 152 (e.g., "Car Movie") and constituent audio 154 (e.g., "Disco Anthem" from Car Movie Soundtrack), as illustrated by auxiliary information 122. Information 120 regarding related media content item(s) includes recommendations based on either media content 152 (e.g., "Car Movie 2: More Cars") or audio 154 (e.g., "Dance Mix"). In other embodiments, at least some of information 120 may be based on both media content 152 and constituent audio 154.
[0030] As mentioned above, state feature(s) may include physical feature(s) of any one or more users and/or any activities performed by the users. As illustrated in the example scenario of FIG. IB, user 150 is presently seated on the floor while operating computing device 156. In such scenarios, information 119 provided by computing device 108 may include information 158 (e.g., sofa advertisement) based on these state features. In other words, as illustrated by information 158, computing device 108 may be configured to provide recommendations (e.g., advertisements, instructions, etc.) based on observed user behavior (e.g., sitting on the floor), based on environment feature(s) (e.g., lack of sofa), and/or a combination thereof.
[0031] As another non-limiting example, computing device 108 may be configured to provide information 119 including recommendation(s) for computing device(s) related to computing device 156. Such recommendations may include, for example, an advertisement for a competing product or an advertisement for an upgraded version of computing device 156. It will once again be appreciated that, even though an activity (e.g., consumption of media content 152) utilizes computing device(s) other than computing device 108, such activity may be leveraged in order to provide related information that is potentially more useful (e.g., specific) than typical approaches.
[0032] Although information 119 is illustrated as including text and/or other visual information provided via display device 110, it will be appreciated that such information may be presented in any other suitable manner. For example, in some embodiments, information 119 related to one or more state features may be provided via computing device(s) other than computing device 108. As more specific examples, information 119 may be displayed via another computing device, such as a tablet computing device, communicatively coupled to computing device 108 via a wireless or wired connection. This may enable, for example, a "two-screen experience," with supplemental information (e.g., character backgrounds, content previews, reviews, etc.) being provided via the other computing device(s).
[0033] In some embodiments, information 1 19 may not be displayed until suitable recommendation(s) have been determined and/or are requested. That is, computing device 108 may be configured to passively monitor environment 100, detect activity in the environment, and recognize state features, all with user 150 being substantially unaware of such operations. Thus, upon determination of one or more recommendations, computing device 108 may be configured to provide information 119 by, for example, enabling (e.g., powering on, waking from sleep mode, etc.) display device 110.
[0034] FIG. 2 illustrates a block diagram of an embodiment of a system 200 for providing recommendations based upon environmental sensor data. System 200 includes computing device 108 communicatively coupled to sensor system 102, which comprises one or more two-dimensional and/or three-dimensional image sensors 104 and/or one or more audio sensors 106 (e.g., microphone array). Image sensors 104 may include, for example, one or more depth cameras configure to determine a three-dimensional representation (i.e., depth map) of the environment monitored via sensor system 102. Such a configuration may desirable, for example, in order to track user pose and movement (e.g., via one or more skeletal tracking pipelines). Image sensors 104 also may include one or more two-dimensional image sensors, such as a color image sensor. Similarly, audio sensors 106 may include a microphone array, for example, in order to determine the location of any one or more audio sources (e.g., audio device 116, computing device 156 of FIGS. 1A-1B, etc.) using beam forming and/or other mechanisms. As mentioned above, one or more components of computing device 108 and sensor system 102 may be incorporated into any one or more discrete device(s) (e.g., mobile computing device) in various embodiments without departing from the scope of the present disclosure.
[0035] During passive monitoring (i.e., prior to detecting activity and/or user presence), computing device 108 may be configured to acquire sensor data 202 via sensor system 102 at a first, lower bit rate. As sensor data 202 may utilize a non-trivial amount of storage space and/or other computing resources, acquisition of the sensor data at a lower bit rate may enable computing device 108 to continually monitor the environment without imparting undue burden on computing device 108 and/or other computing devices, services, etc. Furthermore, in some embodiments, computing device 108 may be configured to provide at least some of sensor data 202 to remote recommendation service 204 via network 206 for analysis (e.g., via audio analysis module 208 and/or video analysis module 210). Utilizing a remote service, such as remote recommendation service 204, may be desirable in some circumstances, as analysis of sensor data 202 may rely upon resources (e.g., databases, memory, etc.) not available to, or not readily implemented via, computing device 108.
[0036] Upon detecting activity either locally or via remote recommendation service 204, computing device 108 may be configured to identify one or more users present in the environment monitored by the sensor system. Identification of the user(s) may be performed via any suitable mechanism or mechanisms (e.g., gestures, roaming profile, visual and/or audio information, etc.) without departing from the scope of the present disclosure. As with detection of activity, such identification may be provided, in whole in part, locally by computing device 108 and/or via a remote recommendation service 204 operating a user identification module 213. The identification of users may enable computing device 108 to analyze and/or store state information 214 for detected users that have authorized such analysis and/or storage. It will be understood that monitoring may be provided for an arbitrary number N of users utilizing any one or more device(s), represented as user 1 environment 216 and user N environment, without departing from the scope of the present disclosure. In some embodiments, recognition of the user(s) may not be performed.
[0037] As mentioned above, in some embodiments once activity has been detected, sensor data 202 may be acquired and/or analyzed at a second, higher bitrate corresponding to an "active" monitoring mode. At least some of the sensor data may be acquired from one or more companion devices 203 (e.g., mobile computing devices) including any one or more sensors 205 (e.g., inertial sensors, microphones, image sensors, etc.) in some embodiments. For example, computing device 108 may be configured to utilize sensor data 202 from sensor system 102 during passive monitoring, and may be configured to utilize both sensor system 102 and sensors 205 of companion device(s) 203 during active monitoring. This may help to preserve battery life on the companion devices. Likewise, sensors on companion device(s) 203 also may be utilized when a user is outside of an environment monitored by sensor system 102. It will be appreciated that these configurations are presented for the purpose of example, and sensor data 202 may be acquired from any one or more sensors during any one or more operating modes without departing from the scope of the present disclosure.
[0038] Similar to the lower bitrate data, the higher bitrate data may be analyzed by computing device 108 and/or may be provided to remote recommendation service 204. As the remote recommendation service may be configured to analyze both low bitrate streams and high bitrate streams, remote recommendation service 204 may include prioritization module 220 configured to prioritize incoming requests. In other words, as higher bitrate request may signify that a user is in a corresponding environment, it may be desirable to preferentially field such active requests so that recommendations may be provided promptly to the user.
[0039] For example, as illustrated, system 200 may include a plurality of computing devices 108 and/or sensor systems 102 for a plurality of users 216, each of which may interface with one or more shared remote recommendation services 204. Thus, in order to provide a suitable user experience, prioritization module 220 may be utilized in order to provide preferential analysis of higher bitrate (active) requests. In some embodiments, the higher-priority requests may be identified (e.g., via "flags" or other metadata) by the sending computing device 108. In other embodiments, the prioritization module may be configured to analyze the received sensor data 202, and may provide prioritization based on the analysis of the received data.
[0040] Upon entering an active monitoring mode, use environment state service 212 of computing device 108 may be configured to recognize state information 214 (e.g., current state) of any one or more state features 218. As illustrated, state information 214 may be stored via computing device 108 and/or via one or more locations accessible to remote recommendation service 204, though other configurations are possible without departing from the scope of the present disclosure.
[0041] Once state information 214 has been determined, identification of various state feature(s) 218 may be provided. For example, sensor data 202 provided to remote recommendation service 204 may include one or more sensor "samples" (e.g., audio and/or video samples) representing the monitored environment. This information may be compared to, for example, known identification information 222, including, but not limited to, known audio information 224 and known video (image) information 226. In other words, an audio and/or video "fingerprint" may be determined and compared (e.g., via audio analysis module 208 and/or video analysis module 210) to known fingerprints in order to identify at least some of the state features monitored by sensor system 102. In other embodiments, any other suitable analysis mechanism or mechanisms may be utilized.
[0042] Once the one or more state features have been recognized, based on a current state of the state feature(s) 218 and/or on a state progression (e.g., change in body size or shape) captured via state information 214, recommendation(s) may be determined. For example, as illustrated, remote recommendation service 204 may further include recommendation module 228 configured to provide such recommendation(s).
[0043] Further, in some embodiments, recommendations may be at least partially provided by one or more related information services 230 accessible via network 206. For example, upon identifying the one or more state feature(s), representative information (e.g., metadata describing the state features) may be provided to services 230, and services 230 may utilize the representative information and/or additional information (e.g., service- specific user interaction history) in order to provide recommendations. As mentioned above with reference to FIGS. 1A and IB, it will be appreciated that recommendation(s) may include any suitable information presentable via any suitable mechanism(s) without departing from the scope of the present disclosure.
[0044] FIG. 3 shows a flow diagram depicting an embodiment of a method 300 for providing recommendations based upon passively sensing a use environment. At 302, method 300 comprises receiving sensor data from a sensor system including one or more sensors. In some instances the sensor system may be configured to monitor a use environment via image and/or audio sensors. In other instances, the sensor system may be incorporated into a mobile device that a user carries in multiple environments. In yet other instances, data may be received from sensors on a plurality of different devices, including, but not limited to, stationary and mobile sensing devices. It will be understood that the sensor system may take any suitable form without departing from the scope of the present disclosure.
[0045] Receiving the sensor data may include, for example, acquiring 304 the sensor data at a first bitrate corresponding to a passive monitoring mode. The first bitrate may be lower than, for example, a second bitrate corresponding to an active monitoring mode. In some embodiments, the sensor data may be acquired at a variable bitrate that may fluctuate according to various considerations (e.g., availability of computing resources, network bandwidth, etc .) .
[0046] At 306, method 300 further comprises recognizing activity in the use environment based on the sensor data. As mentioned above, recognizing activity may include, but is not limited to, detecting motion meeting a motion threshold (e.g. where a person enters a room monitored by a sensor system in the room, where a person picks up a mobile device having a sensor system, etc.) and/or detecting sound meeting a sound threshold.
[0047] Further, in some embodiments, recognizing activity may include detecting, though not necessarily identifying, one or more users present in the scene (e.g., humanoid shape detection). Although the sensor data acquired at the lower bitrate may or may not provide suitable information to identify the user(s), such information may be usable to identify the presence and/or number of user(s) within the environment.
[0048] In some embodiments, information related to any one or more environment state features may be selected based upon the number of detected users. As one non-limiting example, detection of a plurality of users may indicate that a party is occurring, and such insight may be usable to provide recommendations that are more likely to be relevant to a party scenario. In such scenarios, detection of multiple users may effect recommendation(s) for dance music, whereas detection of a single user may effect recommendation(s) for other music selected based upon other user information and/or contextual information.
[0049] Recognizing the number of users present in a scene further may facilitate enforcement of licensing agreements for various content items. For example, typical end- user licenses may define a maximum number of simultaneous users that constitute an allowed performance of a particular content item. Accordingly, by detecting the number of users present in a scene, and thus the number of simultaneous users potentially experiencing the content item, playback of the media content item (e.g., via computing device 108) may only be allowed if the maximum number of simultaneous users has not been exceed. If the number of simultaneous users has been exceeded, presentation of the media content item may be prevented until additional license(s) are acquired and/or a suitable number of users leave the environment.
[0050] Regardless of the mechanism(s) by which activity is detected, detection of such activity may trigger active monitoring of the use environment, as described above. Accordingly, method 300 further comprises acquiring 310 sensor data at a second (e.g., higher) bitrate corresponding to an active monitoring mode. Although described in terms of varying bitrates, the sensor data, generally speaking, may be acquired at increased "quality" upon detecting activity and/or user presence. Such an approach may enable substantially continual monitoring of the use environment without unduly burdening any one or more devices and/or services.
[0051] It will be understood that the activity that triggers the higher quality monitoring mode may be temporary, yet the cessation of the activity may not trigger a return to a lower quality monitoring mode in some circumstances. For example, where a user enters a room (thereby triggering higher quality monitoring) and then takes a nap (thereby failing a motion or other activity threshold), the presence of the user represented by the sensor data may result in maintenance of a the higher quality mode, even where the user does not make any significant motions during the nap.
[0052] Continuing, at 312, method 300 may include identifying one or more users present in the scene and associating the sensor data with the user(s). In some embodiments, each user may be identified based on the sensor data (e.g., via user-specific gestures, facial recognition, voice pattern recognition, etc.) and/or via other mechanisms (e.g., Near Field Communication "NFC" mechanisms, user input devices, etc.). Identification of the user(s) present in the scene may enable storage, and thus tracking, of any one or more state features over a period of time, among other potential benefits. Additionally, a user may be identified for sensing via a mobile device by such acts as logging into the device, logging into an application running on the device, based upon motion patterns/voice patterns/other personal biometric information detected via sensors on the mobile device, or in any other suitable manner.
[0054] If the user has not provided authorization to be monitored by the system, method 300 may end. However, if the user has provided authorization, method 300 continues to 316, where the method further comprises recognizing a current state of each of one or more use environment state features associated with the user. The one or more state features may include, for example, user state feature(s) 318 (e.g., user body size or shape, user activity, etc.) and/or media state feature(s) 320 (e.g., presented media content item, constituent content item, present device, etc.), among others. Subsequently, at 322, method 300 may further comprise storing the current state of each of the one or more use environment state features. The current state may be stored, for example, via one or more local storage machines and/or via one or more remote storage machines (e.g., via remote recommendation service 204). The stored state information may include, for example, "raw" sensor data 202 and/or one or more metrics, or other representative information, computed therefrom. Storage of representative information may utilize decreased storage space, enable faster retrieval and comparison, and/or may otherwise provide a more desirable user experience.
[0055] At 324, method 300 may further comprise detecting a triggering condition for any one or more environment state features. In some embodiments, such a triggering condition may be detected for a single, user-designated state feature, whereas triggering condition(s) for each monitored state feature may be detected in other embodiments. In some embodiments, triggering conditions may include meeting of a trigger threshold 326, such as, for example, length of sensor data acquisition, change in state feature (e.g., increase in body weight beyond a set threshold), detection of one or more predefined state features (e.g., audio 114 and user activity 118), and/or a combination thereof. In other embodiments, a triggering condition may include receiving a user input 328 (e.g., via one or more input devices) requesting provision of the information related to the selected use environment state feature.
[0056] Upon detecting the triggering condition(s), method 300 further comprises providing, at 330, information related to the selected use environment state feature based upon the current state 332 of the use selected use environment state feature and/or one or more previously-stored states 334 of the selected use environment state feature. In other words, information may be provided based on a state progression (e.g., comparison between current state 332 and one or more previous states 334), based on a current state 332 (e.g., comparison between current state 332 and one or more trigger thresholds), and/or based on other suitable information.
[0057] It will be understood that the embodiments described above are presented for the purpose of example, and that the concepts illustrated may be applied to any suitable scenario. For example, a triggering condition for an environmental state feature may correspond to a lack of activity, such that a recommendation or other response may be triggered by a detected lack of motion of a user. As a more specific example, an alert sent to medical personnel (e.g. by calling 911), family members (e.g. by text message or other mechanism), or other persons or institutions may be triggered by a lack of motion of a person, as detected via sensor data, for a predetermined period of time sufficient to indicate a possible medical problem. In this example, user consent for triggering such a response may be provided by the person ahead of time. Likewise, a recommendation of an activity may be provided upon detecting someone spending a large amount of time lying on a sofa watching television. Additionally, recommendations may be made based upon sensing multiple people in different environments. As a more specific example, if two users who are friends are sitting passively in different environments, information regarding these states of the users may be used to generate a recommendation of an activity to be enjoyed together (e.g. a suggestion to go bowling).
[0058] In some embodiments, the methods and processes described above may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
[0059] FIG. 4 schematically shows a non-limiting embodiment of a computing system 400 that can enact one or more of the methods and processes described above. Computing system 400 is shown in simplified form. Sensor system 102, computing device 108, computing device 156, and remote recommendation service 204 are non-limiting examples of computing system 400. It will be understood that virtually any computer architecture may be used without departing from the scope of this disclosure. In different embodiments, computing system 400 may take the form of a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home-entertainment computer, network computing device, gaming device, mobile computing device, mobile communication device (e.g., smart phone), etc.
[0060] Computing system 400 includes a logic machine 402 and a storage machine 404. Computing system 400 may optionally include a display subsystem 406, input subsystem 408, communication subsystem 410, and/or other components not shown in FIG. 4.
[0061] Logic machine 402 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
[0062] The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
[0063] Storage machine 404 includes one or more physical devices configured to hold and/or store machine-readable instructions executable by the logic machine to implement the methods and processes described herein. For example, logic machine 402 may be in operative communication with storage machine 404. When such methods and processes are implemented, the state of storage machine 404 may be transformed— e.g., to hold different data.
[0064] Storage machine 404 may include removable and/or built-in devices. Storage machine 404 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machine 404 may include machine -readable volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file- addressable, and/or content-addressable devices.
[0065] It will be appreciated that storage machine 404 includes one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
[0066] Aspects of logic machine 402 and storage machine 404 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
[0067] When included, display subsystem 406 may be used to present a visual representation of data held by storage machine 404. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 406 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 406 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 402 and/or storage machine 404 in a shared enclosure, or such display devices may be peripheral display devices.
[0068] When included, input subsystem 408 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, microphone, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
[0069] When included, communication subsystem 410 may be configured to communicatively couple computing system 400 with one or more other computing devices. Communication subsystem 410 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide- area network. In some embodiments, the communication subsystem may allow computing system 400 to send and/or receive messages to and/or from other devices via a network such as the Internet.
[0070] It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
[0071] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.

中文:
[0001]数字媒体内容的消费者可以经由各种机制发现新内容以供消费。例如,一些机制涉及用户经由先前存在的列表主动寻找媒体,诸如通过咨询电子编程向导。其它方法可以涉及追踪用户的媒体消费行为,并且然后提供涉及用户所消费的内容的推荐。
[0002]所公开的实施例涉及基于经由监视数字内容消费使用环境所获得的数据来提供数字内容推荐。例如,一个实施例提供了一种计算设备,其被配置成从包括一个或多个传感器的传感器系统接收传感器数据,所述一个或多个传感器被配置成通过监视使用环境来获得传感器数据。计算设备还被配置成经由传感器数据检测存在于使用环境中的用户,标识用户,经由传感器数据识别与用户相关联的一个或多个使用环境状态特征中的每一个的当前状态,并且存储一个或多个使用环境状态特征中的每一个的当前状态。计算设备还被配置成检测用于所选使用环境状态特征的触发条件,并且在检测到触发条件时,基于使用所选使用环境状态特征的当前状态和所选使用环境状态特征的之前存储的状态中的一个或多个而提供涉及所选使用环境状态特征的信息。
[0004]图1A和1B示出了用于监视用户活动的使用环境的示例实施例。
[0005]图2示意性地示出了根据本公开的实施例的用于提供推荐的使用系统。
[0006]图3示出描绘了根据本公开的实施例的用于监视使用环境的一个或多个状态特征的方法的实施例的过程流。
[0007]图4示意性地示出了根据本公开的实施例的计算设备。
[0008]各种机制可以用于提供针对消费者可能感兴趣的产品、媒体内容等的推荐和/或广告。例如,各种在线服务(例如电子商务服务)可以基于之前和/或当前与服务的(多个)用户交互来提供推荐。然而,这样的机制利用与数字内容有意图的、有意识的用户交互来产生推荐。
[0009]因此,本文公开的实施例涉及使用一个或多个环境传感器来以基本上不引人注目的方式监视和理解各种用户活动、特征等并且因此响应于潜在地更加丰富的环境信息集合而提供推荐。换言之,取代于依赖一个或多个用户来描述、定义等环境中“正在进行”什么,可能合期望的是被动检测这样的信息。作为一个非限制性示例,声学传感器可以用于检测和标识正在体验的音乐,并且基于所检测的音乐提供推荐。与可能依赖用户手动实现(多个)推荐机制(例如经由文本搜索、经由音频捕获机制等)的典型方法相反,这样的配置可以潜在地以基本上更不引人注目且直观的方式提供(多个)推荐。
[0010]图1A图示了用于基于用户活动的被动感测而提供推荐的示例使用环境100。使用环境100包括被配置成获得传感器数据的传感器系统102。尽管将环境100图示为私人住所内的房间,但是将领会到的是,出于示例的目的而呈现这样的环境,并且其不意图以任何方式进行限制。
[0011]如所图示的,传感器系统包括一个或多个图像传感器104(例如2D图像传感器和/或3D深度传感器)和一个或多个音频传感器106(例如麦克风或定向麦克风阵列)。这样的传感器的使用可以允许识别环境100的一个或多个状态特征的定位和/或形状的能力,如以下更加详细地描述的。在其它实施例中,传感器系统102可以包括(多个)附加和/或不同的传感器而不脱离本公开的范围。
[0012]如本文所使用的,术语“状态特征”是指经由传感器系统102的一个或多个传感器可检测的使用环境100和/或其(多个)构成项(例如用户、宠物、家具和其它物体等)的任何特性。作为非限制性示例,状态特征可以包括但不限于,任何一个或多个用户的物理特征(例如身体尺寸和形状)、由任何一个或多个用户执行的活动,和/或由任何一个或多个用户操作的设备(例如移动电话、平板计算设备、媒体播放器、游戏控制台等)。在其它实施例中,状态特征可以包括一个或多个媒体项(例如音频和/或视觉媒体内容)的特性,诸如媒体内容项的身份、内容项的构成成分的身份(例如(多个)音轨内容项)、访问内容项的设备、以及内容项的当前访问位置(例如回放位置)。将领会到的是,出于示例的目的而呈现这些状态特征,并且其不意图以任何方式进行限制。
[0013]在一些实施例中,传感器系统102可以(例如经由一个或多个无线和/或有线连接)通信耦合到计算设备108,计算设备108尽管被图示为视频游戏控制台,但是在其它实施例中可以具有不同配置(例如台式计算机、膝上型计算机、笔记本计算机等)。另外,尽管被图示为分立的设备,但是将领会到的是,在再其它的实施例中,传感器系统102和计算设备108的各种组件可以合并到单个设备(例如移动计算设备)中。此外,传感器数据还可以由计算设备108从具有一个或多个传感器的其它设备获得,诸如由用户携带的移动设备。为了容易理解,传感器系统102和计算设备108将在本文中被讨论为协作地操作,尽管将领会到的是,在各种实施例中,所描述的功能性可以由传感器系统102和/或计算设备108提供而不脱离本公开的范围。
[0014]通过使用经由传感器系统102和/或经由(多个)其它连接设备的(多个)传感器所获得的传感器数据,计算设备108可以监视使用环境100的一个或多个状态特征,并且经由通信耦合到计算设备108的输出设备(例如显示设备110)提供涉及这样的状态特征的信息,诸如元数据、媒体项和/或活动推荐、广告等。传感器系统102可以被配置成监视使用环境100内的活动,而不管这样的活动涉及计算设备108的直接利用还是间接利用。例如,即使经由媒体呈现设备而不是计算设备108呈现特定媒体内容项(例如音频视觉内容、视频游戏或其它交互式内容等),传感器系统也可以被配置成监视这样的活动。
[0015]作为图1A中描绘的一个非限制性示例,用户112正在跳舞而同时聆听由音频设备116(例如立体声系统)提供的音频114。在该特定场景中,可识别的状态特征包括但不限于,用户活动118(例如跳舞)、音频114和/或提供音频114的设备116的身份。可以例如使用一个或多个建模和/或追踪机制(例如(多个)骨架追踪管线)、耦合到用户的惯性设备(例如经由移动设备、惯性传感器等)等来检测用户活动118。可以例如经由音频传感器106(例如经由音频指纹标识)来提供音频114的检测,而设备116可以经由图像传感器104(例如经由形状识别)和/或经由音频传感器106(例如经由(多个)波束形成机制)来检测。如以上提到的,状态特征可以包括经由传感器系统102可检测的环境100的任何特征,并且因而将领会到,出于示例的目的而呈现这样的特征,并且其不意图以任何方式进行限制。
[0016]在一些情形中,经由传感器系统102的任何一个或多个状态特征的连续标识可能利用明显的资源(例如电力、处理器带宽、网络带宽等),甚至在已经选择为被动感测的用户不处于使用环境中时亦是如此。因此,传感器系统102可以被配置成被动监视环境100直至检测到活动,此时传感器系统可以开始主动监视环境。因而,当没有检测到活动(例如用户存在、内容项回放等)时,传感器系统102和/或计算设备108可以被配置成以较低“质量”(例如较低比特率)获得传感器数据,并且一旦检测到活动则以较高“质量”获得传感器数据。通过以较低比特率获得传感器数据以便检测活动而同时将较高比特率的传感器数据用于状态特征标识,除其它潜在益处之外,这样的配置可以运用传感器系统102所提供的更丰富的数据集合而同时潜在地利用减少量的资源。将理解到,以较高比特率的数据获得甚至可以在触发活动已经终止之后继续。例如,如果用户走进房间并且然后躺下,则较高比特率的数据获得可以在用户存在于环境中时继续,即便原始触发活动(例如用户移动)已经终止。
[0017]可以以任何合适的方式检测足以触发较高“质量”水平的数据获得和分析的活动。例如,在一些实施例中,这样的活动可以对应于满足运动阈值的运动(例如超过阈值速度和/或位移的移动)、满足声音阈值的声音(例如超出特定分贝水平的声音)、使用环境中的一个或多个用户的存在(例如经由骨架追踪、人类形状检测等)。
[0018]当检测到活动时,计算设备108可以被配置成标识环境100的任何一个或多个状态特征。例如,可以通过获得音频样本并且将音频样本与已知音频信息比较来标识音频114状态特征。在一些实施例中,计算设备108可以通信耦合到一个或多个远程服务,并且可以被配置成向远程服务提供至少一些所获得的音频样本以供标识。在其它实施例中,计算设备108可以被配置成提供这样的标识而不是或者附加于(多个)远程服务。类似地,可以通过捕获一个或多个图像样本并且将(多个)图像样本与已知图像信息比较来标识其它状态特征(例如用户活动118、音频设备116)。
[0019]在一些实施例中,可以将表示任何一个或多个状态特征(例如用户身体尺寸和/或形状)的传感器数据和/或其样本与关于(多个)特定状态特征的一个或多个之前状态的信息相比较。以此方式,可以随时间追踪一个或多个状态特征,从而潜在地通过提供状态进展的理解来提供改进的推荐性能,这与简单的(多个)单独状态的理解相反。将领会到的是,为了提供这样的进展监视,计算设备108可以被配置成存储任何一个或多个状态特征的(多个)状态(例如经由设备108和/或远程计算设备、服务等)。这样的存储可以例如仅在用户已经提供要由传感器系统监视的授权的情况下提供。这样的授权的确定可以包括但不限于,识别之前定义的授权信息(例如(多个)用户简档)和/或识别由用户执行的(多个)授权姿势(例如口头、物理的等)。可以例如经由一个或多个选择加入和/或选择退出机制来控制授权,尽管其它配置是可能的而不脱离本公开的范围。在一些实施例中,计算设备108可以提供基本上匿名的监视,其中未存储或以其它方式利用之前的状态信息,尽管将领会到,相比于其它方法,这样的匿名监视可能不提供适当丰富的数据集合。
[0020]当识别到环境100的任何一个或多个状态特征时,计算设备108可以被配置成确定任何所述状态特征是否满足或超出对应触发条件。换言之,这样的(多个)触发条件确定是否要提供涉及(多个)触发状态特征的信息(例如推荐)。
[0021]作为这样的触发阈值检测的一个非限制性示例,可以经由所存储的状态信息的分析(例如与之比较)来标识满足触发阈值的用户112的任何一个或多个物理特征中的(多个)改变(例如超过阈值水平的身体尺寸中的改变)。作为触发阈值的另一示例,可以在已经满足时间阈值(例如30秒音频)之后标识来自音频设备116的音频114,以便确保音频114被主动地聆听,并且不仅仅是被“扫描”或“采样”。在一些实施例中,一个或多个触发条件可以包括请求(多个)推荐的显式用户输入(例如经由姿势、经由输入设备等)。尽管已经将每一个状态特征描述为具有单个对应触发阈值,但是将领会到,任何一个或多个状态特征可以对应于具有任何一个或多个触发阈值的任何一个或多个触发条件而不脱离本公开的范围。
[0022]如以上所提到的,当识别到满足任何一个或多个触发条件时,计算设备108可以被配置成提供涉及(多个)对应状态特征的信息119(例如(多个)推荐)。在(多个)状态特征包括一个或多个媒体内容项(例如音频114)的情况下,涉及(多个)状态特征的信息119可以包括但不限于,关于有关内容项的信息120、用于内容项的辅助信息122和获得(例如下载和/或购买)内容项的实例的链接125。在一些实施例中,信息119可以包括一个或多个机制124以将所检测的内容项添加到内容播放列表(例如在线音乐服务播放列表、经由传感器系统最近检测到的内容项的播放列表等)。
[0023]关于有关内容项的信息120可以包括例如,用于(多个)有关内容项的推荐、获得这样的有关内容项的(多个)机制、将有关内容项添加到播放列表的(多个)机制和/或其它合适的信息。尽管被图示为经由显示设备110而经由“独立式”推荐用户接口呈现的基于文本的信息,但是将领会到,信息120可以包括关于(多个)有关项的任何信息,并且可以经由任何(多个)合适机制提供,而不脱离本公开的范围。
[0024]在(多个)状态特征包括用户活动(例如活动118)的情况下,计算设备108可以被配置成提供用于涉及用户活动的物品和/或服务的推荐(例如广告、推销活动等)。另外,在一些实施例中,有关内容项可以至少基于所检测的用户活动118来确定。例如,计算设备108可以被配置成检测用户112正在跳舞,并且作为响应可以推荐舞蹈音乐。继续以跳舞为示例,计算设备108可以被配置成确定舞蹈的“风格”,并且可以基于所确定的风格而提供推荐。例如,将用户活动118识别为接近“迪斯科”舞蹈风格可以实现迪斯科型音乐的推荐。
[0025]在(多个)状态特征包括用户112的一个或多个物理特征的情况下,这样的信息可以包括例如描述物理特征中的改变的信息(例如体重增加的通知)、涉及物理特征的物品或服务的推荐(例如用于健康物品或服务的广告)、以及对用户行为的一个或多个改变的推荐(例如针对更多锻炼的指令)。
[0026]在一些实施例中,信息119可以包括对应于(多个)所检测的状态特征的辅助信息122。如所图示的,信息122可以包括内容标识(例如艺术家、专辑等)、有关视觉信息(例如封面艺术、(多个)音乐视觉化等)、以及涉及(多个)所检测的状态特征的其它辅助信息(例如歌词等)。信息122的呈现对于例如标识和/或了解(多个)所检测的状态特征而言是合期望的。
[0027]在一些场景中,多个状态特征的检测和理解可以使得计算设备108能够潜在地定位更多相关推荐。例如,通过理解多个有关状态特征(例如用户112跳舞时所消费的音频114),可以为这样的状态特征提供上下文,其可以用于确定推荐。例如,音频114和用户活动118二者的检测可以触发涉及音频114和活动118(例如舞蹈音乐)二者的信息119(例如舞蹈课广告126)的提供,而如果未连同音频114检测到用户活动118,则可以触发涉及音频114的信息119(例如信息122和/或信息120)。出于示例的目的而呈现这样的场景,并且其不意图以任何方式进行限制。
[0028]现在转向图1B,在涉及用户150正在经由计算设备156(被示为平板计算设备)观看包括音频154(例如(多个)音轨歌曲)的媒体内容152(例如电影)的另一使用情况场景中图示了图1A的环境100。在该场景中,媒体内容152、媒体内容152的构成音频154和操作计算设备156每一个都是经由传感器系统102可检测的状态特征的示例。可以例如经由音频传感器106检测音频154,并且可以例如由图像传感器104和/或音频传感器106检测媒体内容152和计算设备156,例如通过获得计算设备156的(多个)图像样本并且将(多个)样本与已知图像信息相比较。此外,计算设备156可以包括位于其上的传感器,其可以向计算设备108提供传感器数据。来自计算设备156和/或来自其它合适的移动设备(例如智能电话)的传感器信息可以由计算设备156获得和分析,甚至在用户不处于传感器系统102所监视的环境内时亦是如此。
[0029]在所图示的示例中,计算设备108标识媒体内容152(例如“汽车电影”)和构成音频154(例如来自汽车电影音轨的“迪斯科歌曲”)二者,如由辅助信息122图示的。关于(多个)有关媒体内容项的信息120包括基于媒体内容152的推荐(例如“汽车电影2:更多汽车”)或基于音频154的推荐(例如“舞蹈混合”)。在其它实施例中,至少一些信息120可以基于媒体内容152和构成音频154二者。
[0030]如以上提到的,(多个)状态特征可以包括任何一个或多个用户和/或用户所执行的任何活动的(多个)物理特征。如图1B的示例场景中图示的,用户150现在坐在地板上而同时操作计算设备156。在这样的场景中,由计算设备108提供的信息119可以包括基于这些状态特征的信息158(例如沙发广告)。换言之,如由信息158图示的,计算设备108可以被配置成基于所观察的用户行为(例如坐在地板上)、基于(多个)环境特征(例如没有沙发)和/或其组合而提供推荐(例如广告、指令等)。
[0031]作为另一非限制性示例,计算设备108可以被配置成提供信息119,其包括涉及计算设备156的(多个)计算设备的(多个)推荐。这样的推荐可以包括例如针对竞争产品的广告或针对计算设备156的升级版本的广告。再一次将领会到,即使活动(例如媒体内容152的消费)利用(多个)计算设备而不是计算设备108,也可以运用这样的活动以便提供潜在地比典型方法更有用(例如具体)的有关信息。
[0032]尽管将信息119图示为包括经由显示设备110提供的文本和/或其它视觉信息,但是将领会到,可以以任何其它合适的方式呈现这样的信息。例如,在一些实施例中,可以经由(多个)计算设备而不是计算设备108来提供涉及一个或多个状态特征的信息119。作为更具体的示例,信息119可以经由另一计算设备(诸如平板计算设备)而显示,所述另一计算设备经由无线或有线连接通信耦合到计算设备108。这可以例如使得能够实现具有经由(多个)其它计算设备提供的补充信息(例如人物背景、内容预览、回顾等)的“双屏体验”。
[0033]在一些实施例中,可以不显示信息119直到已经确定和/或请求(多个)合适的推荐。也就是说,计算设备108可以被配置成被动地监视环境100,检测环境中的活动,并且识别状态特征,其中在全部情况下用户150基本上没有意识到这样的操作。因此,当确定一个或多个推荐时,计算设备108可以被配置成通过例如启用(例如通电、从睡眠模式唤醒等)显示设备110来提供信息119。
[0034]图2图示了用于基于环境传感器数据而提供推荐的系统200的实施例的框图。系统200包括通信耦合到传感器系统102的计算设备108,传感器系统102包括一个或多个二维和/或三维图像传感器104和/或一个或多个音频传感器106(例如麦克风阵列)。图像传感器104可以包括例如一个或多个深度相机,其被配置成确定经由传感器系统102所监视的环境的三维表示(即深度图)。这样的配置可能是合期望的,例如以便追踪用户姿态和移动(例如经由一个或多个骨架追踪管线)。图像传感器104还可以包括一个或多个二维图像传感器,诸如彩色图像传感器。类似地,音频传感器106可以包括麦克风阵列,例如以便通过使用波束形成和/或其它机制来确定任何一个或多个音频源(例如图1A-1B的音频设备116、计算设备156等)的位置。如以上提到的,在各种实施例中,计算设备108和传感器系统102的一个或多个组件可以合并到任何一个或多个分立设备(例如移动计算设备)中而不脱离本公开的范围。
[0035]在被动监视期间(即在检测到活动和/或用户存在之前),计算设备108可以被配置成以第一较低比特率经由传感器系统102获得传感器数据202。由于传感器数据202可能利用非琐碎量的存储空间和/或其它计算资源,因此以较低比特率获取传感器数据可以使得计算设备108能够继续监视环境而不对计算设备108和/或其它计算设备、服务等施加不适当的负担。另外,在一些实施例中,计算设备108可以被配置成经由网络206向远程推荐服务204提供至少一些传感器数据202以供分析(例如经由音频分析模块208和/或视频分析模块210)。利用远程服务(诸如远程推荐服务204)在一些环境中可以是合期望的,因为传感器数据202的分析可能依赖于对计算设备108不可用或者经由计算设备108不容易实现的资源(例如数据库、存储器等)。
[0036]当在本地或者经由远程推荐服务204检测到活动时,计算设备108可以被配置成标识存在于传感器系统所监视的环境中的一个或多个用户。(多个)用户的标识可以经由一个或多个任何合适的机制(例如姿势、漫游配置文件、视觉和/或音频信息等)来执行而不脱离本公开的范围。至于活动的检测,可以在本地由计算设备108和/或经由操作用户标识模块213的远程推荐服务204整体或部分提供这样的标识。用户的标识可以使得计算设备108能够分析和/或存储用于已经授权这样的分析和/或存储的所检测的用户的状态信息214。将理解到,可以针对利用任何一个或多个设备的任意数目N用户提供监视,其被表示为用户1环境216和用户N环境,而不脱离本公开的范围。在一些实施例中,可以不执行(多个)用户的识别。
[0037]如以上提到的,在一些实施例中,一旦已经检测到活动,则可以以对应于“主动”监视模式的第二较高比特率获得和/或分析传感器数据202。在一些实施例中,至少一些传感器数据可以从包括任何一个或多个传感器205(例如惯性传感器、麦克风、图像传感器等)的一个或多个配套设备203(例如移动计算设备)来获得。例如,计算设备108可以被配置成在被动监视期间利用来自传感器系统102的传感器数据202,并且可以被配置成在主动监视期间利用传感器系统102和(多个)配套设备203的传感器205二者。这可以帮助保持配套设备上的电池寿命。同样地,还可以在用户处于传感器系统102所监视的环境之外时利用(多个)配套设备203上的传感器。将领会到,出于示例的目的而呈现这些配置,并且在任何一个或多个操作模式期间,传感器数据202可以从任何一个或多个传感器获得,而不脱离本公开的范围。
[0038]类似于较低比特率数据,较高比特率数据可以由计算设备108分析和/或可以被提供给远程推荐服务204。由于远程推荐服务可以被配置成分析低比特率流和高比特率流二者,因此远程推荐服务204可以包括被配置成优先化传入请求的优先化模块220。换言之,由于较高比特率请求可能意味着用户处于对应环境中,因此可能合期望的是优先应对这样的主动请求使得可以迅速向用户提供推荐。
[0039]例如,如所图示的,系统200可以包括用于多个用户216的多个计算设备108和/或传感器系统102,其中每一个可以与一个或多个共享远程推荐服务204对接。因此,为了提供合适的用户体验,可以利用优先化模块220以便提供较高比特率(主动)请求的优先分析。在一些实施例中,可以通过发送计算设备108来标识(例如经由“标志”或其它元数据)较高优先级请求。在其它实施例中,优先化模块可以被配置成分析所接收的传感器数据202,并且可以基于所接收的数据的分析来提供优先化。
[0040]当进入主动监视模式时,计算设备108的使用环境状态服务212可以被配置成识别任何一个或多个状态特征218的状态信息214(例如当前状态)。如所图示的,状态信息214可以经由计算设备108和/或经由远程推荐服务204可访问的一个或多个位置而存储,尽管其它配置是可能的而不脱离本公开的范围。
[0041]一旦已经确定状态信息214,则可以提供(多个)各种状态特征218的标识。例如,提供给远程推荐服务204的传感器数据202可以包括表示所监视的环境的一个或多个传感器“样本”(例如音频和/或视频样本)。可以将该信息与例如已知标识信息222相比较,已知标识信息222包括但不限于已知音频信息224和已知视频(图像)信息226。换言之,可以确定音频和/或视频“指纹”并且将其与已知指纹比较(例如经由音频分析模块208和/或视频分析模块210)以便标识传感器系统102所监视的至少一些状态特征。在其它实施例中,可以利用任何一个或多个其它合适的分析机制。
[0042]一旦已经识别一个或多个状态特征,则基于(多个)状态特征218的当前状态和/或基于经由状态信息214所捕获的状态进展(例如身体尺寸或形状中的改变),可以确定(多个)推荐。例如,如所图示的,远程推荐服务204还可以包括被配置成提供(多个)这样的推荐的推荐模块228。
[0043]另外,在一些实施例中,推荐可以至少部分地由经由网络206可访问的一个或多个有关信息服务230来提供。例如,当标识到一个或多个状态特征时,可以向服务230提供代表性信息(例如描述状态特征的元数据),并且服务230可以利用代表性信息和/或附加信息(例如服务特定用户交互历史)以便提供推荐。如以上参照图1A和1B提到的,将领会到,(多个)推荐可以包括经由任何(多个)合适机制可呈现的任何合适的信息而不脱离本公开的范围。
[0044]图3示出描绘了用于基于被动感测使用环境而提供推荐的方法300的实施例的流程图。在302处,方法300包括从包括一个或多个传感器的传感器系统接收传感器数据。在一些实例中,传感器系统可以被配置成经由图像和/或音频传感器来监视使用环境。在其它实例中,传感器系统可以合并到用户在多个环境中携带的移动设备中。在又一些实例中,数据可以从多个不同设备上的传感器接收,该多个不同设备包括但不限于,静止和移动感测设备。将理解到,传感器系统可以采用任何合适的形式而不脱离本公开的范围。
[0045]接收传感器数据可以包括例如以对应于被动监视模式的第一比特率获得304传感器数据。第一比特率可以低于例如对应于主动监视模式的第二比特率。在一些实施例中,传感器数据可以以可根据各种考虑(例如计算资源的可用性、网络带宽等)而波动的可变比特率来获得。
[0046]在306处,方法300还包括基于传感器数据识别使用环境中的活动。如以上所提到的,识别活动可以包括但不限于,检测满足运动阈值的运动(例如其中人员进入由房间中的传感器系统监视的房间,其中人员拾起具有传感器系统的移动设备等)和/或检测满足声音阈值的声音。
[0047]另外,在一些实施例中,识别活动可以包括检测(尽管不一定标识)存在于场景中的一个或多个用户(例如人类形状检测)。尽管以较低比特率获得的传感器数据可以或者可以不提供合适的信息以标识(多个)用户,但是这样的信息可以可用于标识环境内的(多个)用户的存在和/或数目。
[0048]在一些实施例中,可以基于所检测的用户数目来选择涉及任何一个或多个环境状态特征的信息。作为一个非限制性示例,多个用户的检测可以指示聚会正在出现,并且这样的洞察可以可用于提供更可能与聚会情境相关的推荐。在这样的情境中,多个用户的检测可以引起针对舞蹈音乐的(多个)推荐,而单个用户的检测可以引起针对基于其它用户信息和/或上下文信息所选择的其它音乐的(多个)推荐。
[0049]识别场景中存在的用户数目还可以促进用于各种内容项的许可协议的实施。例如,典型的最终用户许可可以定义构成特定内容项的所允许执行的同时用户的最大数目。因此,通过检测存在于场景中的用户数目,并且因此检测潜在地体验内容项的同时用户的数目,可以仅在尚未超出同时用户的最大数目的情况下允许媒体内容项的回放(例如经由计算设备108)。如果已经超出同时用户的数目,则可以阻止媒体内容项的呈现,直至获得(多个)附加许可和/或合适数目的用户离开环境。
[0050]不管通过哪种(哪些)机制检测活动,这样的活动的检测都可以触发使用环境的主动监视,如以上描述的。因此,方法300还包括以对应于主动监视模式的第二(例如较高)比特率获得310传感器数据。尽管在变化比特率方面进行描述,但是一般而言,当检测到活动和/或用户存在时,传感器数据可以以增加的“质量”获得。这样的方法可以使得能够基本上连续监视使用环境而不会不适当地增加任何一个或多个设备和/或服务的负担。
[0051]将理解到,触发较高质量监视模式的活动可以是临时的,但是在一些环境中,活动的终止可能不触发返回到较低质量监视模式。例如,在用户进入房间(从而触发较高质量监视)并且然后小憩(从而未能达到运动或其它活动阈值)的情况下,由传感器数据表示的用户的存在可以导致维持较高质量模式,甚至在用户在小憩期间未做出任何明显运动的情况下亦是如此。
[0052]接着,在312处,方法300可以包括标识存在于场景中的一个或多个用户并且将传感器数据与(多个)用户相关联。在一些实施例中,可以基于传感器数据(例如经由用户特定姿势、面部识别、话音模式识别等)和/或经由其它机制(例如近场通信“NFC”机制、用户输入设备等)标识每一个用户。除其它潜在益处之外,存在于场景中的(多个)用户的标识可以使得能够实现一段时间内任何一个或多个状态特征的存储及因此的追踪。此外,可以标识用户以用于通过诸如登录到设备中、登录到运行在设备上的应用中之类的动作,基于经由移动设备上的传感器所检测的运动模式/话音模式/其它个人生物信息,或者以任何其它合适的方式经由移动设备的感测。
[0054]如果用户尚未提供要由系统监视的授权,则方法300可以结束。然而,如果用户已经提供授权,则方法300继续到316,其中方法还包括识别与用户相关联的一个或多个使用环境状态特征中的每一个的当前状态。除其它之外,一个或多个状态特征可以包括例如(多个)用户状态特征318(例如用户身体尺寸或形状、用户活动等)和/或(多个)媒体状态特征320(例如所呈现的媒体内容项、构成内容项、存在的设备等)。随后,在322处,方法300还可以包括存储一个或多个使用环境状态特征中的每一个的当前状态。可以例如经由一个或多个本地存储机器和/或经由一个或多个远程存储机器(例如经由远程推荐服务204)存储当前状态。所存储的状态信息可以包括例如“原始”传感器数据202和/或从其计算的一个或多个度量或其它代表性信息。代表性信息的存储可以利用减小的存储空间,使得能够实现更快的检索和比较,和/或可以以其它方式提供更合期望的用户体验。
[0055]在324处,方法300还可以包括检测用于任何一个或多个环境状态特征的触发条件。在一些实施例中,可以针对单个用户指定状态特征检测这样的触发条件,而在其它实施例中可以检测用于每一个所监视的状态特征的(多个)触发条件。在一些实施例中,触发条件可以包括满足触发阈值326,诸如例如传感器数据获得的长度、状态特征中的改变(例如超过设定阈值的体重中的增加)、一个或多个预定义状态特征(例如音频114和用户活动118)的检测、和/或其组合。在其它实施例中,触发条件可以包括接收请求提供涉及所选使用环境状态特征的信息的用户输入328(例如经由一个或多个输入设备)。
[0056]当检测到(多个)触发条件时,方法300还包括在330处基于使用所选使用环境状态特征的当前状态332和/或所选使用环境状态特征的一个或多个之前存储的状态334而提供涉及所选使用环境状态特征的信息。换言之,可以基于状态进展(例如当前状态332与一个或多个之前状态334之间的比较),基于当前状态332(例如当前状态332与一个或多个触发阈值之间的比较)和/或基于其它合适信息来提供信息。
[0057]将理解到,出于示例的目的而呈现以上所描述的实施例,并且所说明的概念可以应用于任何合适的情境。例如,用于环境状态特征的触发条件可以对应于活动的缺乏,使得推荐或其它响应可以由所检测的用户运动的缺乏来触发。作为更具体的示例,发送到医疗人员(例如通过呼叫911)、家庭成员(例如通过文本消息或其它机制)或其他人员或机构的警报可以通过在足以指示可能的医疗问题的预确定时间段内如经由传感器数据所检测的人员运动的缺乏来触发。在该示例中,可以由人员提前提供用于触发这样的响应的用户同意。同样地,当检测到某人正花费大量时间躺在沙发上看电视时,可以提供活动的推荐。此外,可以基于在不同环境中感测到多个人而做出推荐。作为更具体的示例,如果作为朋友的两个用户在不同环境中无聊地坐着,则关于用户的这些状态的信息可以用于生成一起享受的活动的推荐(例如建议去打保龄球)。
[0058]在一些实施例中,以上描述的方法和过程可以依靠一个或多个计算设备的计算系统。特别地,这样的方法和过程可以实现为计算机应用程序或服务、应用编程接口(API)、库和/或其它计算机程序产品。
[0059]图4示意性地示出了可以实施以上描述的一个或多个方法和过程的计算系统400的非限制性实施例。以简化形式示出计算系统400。传感器系统102、计算设备108、计算设备156和远程推荐服务204是计算系统400的非限制性示例。将理解到,事实上可以使用任何计算机架构而不脱离本公开的范围。在不同实施例中,计算系统400可以采取大型计算机、服务器计算机、台式计算机、膝上型计算机、平板计算机、家庭娱乐计算机、网络计算设备、游戏设备、移动计算设备、移动通信设备(例如智能电话)等形式。
[0060]计算系统400包括逻辑机器402和存储机器404。计算系统400可以可选地包括显示子系统406、输入子系统408、通信子系统410和/或图4中未示出的其它组件。
[0061]逻辑机器402包括被配置成运行指令的一个或多个物理设备。例如,逻辑机器可以被配置成运行作为一个或多个应用、服务、程序、例程、库、对象、组件、数据结构或其它逻辑构造的部分的指令。这样的指令可以实现成执行任务、实现数据类型、变换一个或多个组件的状态、达成技术效果或以其它方式到达期望的结果。
[0062]逻辑机器可以包括被配置成运行软件指令的一个或多个处理器。此外或可替换地,逻辑机器可以包括一个或多个硬件或固件逻辑机器,其被配置成运行硬件或固件指令。逻辑机器的处理器可以是单核或多核的,并且其上运行的指令可以被配置用于串行、并行和/或分布式处理。逻辑机器的各个组件可选地可以分布在两个或更多分离设备当中,分离设备可以远程定位和/或配置用于协同处理。逻辑机器的各方面可以通过以云计算配置进行配置的可远程访问的联网计算设备虚拟化和运行。
[0063]存储机器404包括被配置成持有和/或存储由逻辑机器可执行以实现本文所描述的方法和过程的机器可读指令的一个或多个物理设备。例如,逻辑机器402可以与存储机器404操作通信。当实现这样的方法和过程时,存储机器404的状态可以变换——例如变换成持有不同数据。
[0064]存储机器404可以包括可移除和/或内置设备。除其它之外,存储机器404可以包括光学存储器(例如CD、DVD、HD-DVD、蓝光盘等)、半导体存储器(例如RAM、EPROM、EEPROM等)和/或磁性存储器(例如硬盘驱动器、软盘驱动器、磁带驱动器、MRAM等)。存储机器404可以包括机器可读易失性、非易失性、动态、静态、读/写、只读、随机存取、顺序存取、位置可寻址、文件可寻址和/或内容可寻址设备。
[0065]将领会到,存储机器404包括一个或多个物理设备。然而,本文所描述的指令的各方面可替换地可以在有限持续时间内通过未由物理设备持有的通信介质(例如电磁信号、光学信号等)而传播。
[0066]逻辑机器402和存储机器404的各方面可以一起集成到一个或多个硬件逻辑组件中。这样的硬件逻辑组件可以包括例如现场可编程门阵列(FPGA)、程序特定集成电路和专用集成电路(PASIC/ASIC)、程序特定标准产品和专用标准产品(PSSP/ASSP)、片上系统(SOC)和复杂可编程逻辑设备(CPLD)。
[0067]当包括时,显示子系统406可以用于呈现由存储机器404持有的数据的视觉表示。该视觉表示可以采取图形用户接口(GUI)的形式。当本文描述的方法和过程改变由存储机器持有的数据并且因此变换存储机器的状态时,显示子系统406的状态可以同样地变换成在视觉上表示底层数据中的改变。显示子系统406可以包括事实上利用任何类型技术的一个或多个显示设备。这样的显示设备可以与逻辑机器402和/或存储机器404组合在共享外壳中,或者这样的显示设备可以是外围显示设备。
[0068]当包括时,输入子系统408可以包括诸如键盘、鼠标、触摸屏、麦克风或游戏控制器之类的一个或多个用户输入设备或者与之对接。在一些实施例中,输入子系统可以包括所选自然用户输入(NUI)组件部分或者与之对接。这样的组件部分可以是集成的或者外围的,并且输入动作的转换和/或处理可以在板上或者在板外处置。示例NUI组件部分可以包括用于语音和/或话音识别的麦克风;用于机器视觉和/或姿势识别的红外、彩色、立体和/或深度相机;用于运动检测和/或意图识别的头部追踪器、眼睛追踪器、加速度计和/或陀螺仪;以及用于评估脑部活动的电场感测组件部分。
[0069]当包括时,通信子系统410可以被配置成将计算系统400与一个或多个其它计算设备通信耦合。通信子系统410可以包括与一个或多个不同通信协议兼容的有线和/或无线通信设备。作为非限制性示例,通信子系统可以被配置用于经由无线电话网络或有线或无线局域网或广域网的通信。在一些实施例中,通信子系统可以允许计算系统400经由诸如因特网之类的网络向其它设备发送消息和/或从其它设备接收消息。
[0070]将理解到,本文所描述的配置和/或方法在性质上是示例性的,并且这些具体实施例或示例不以限制性含义来考虑,因为大量变型是可能的。本文所描述的具体例程或方法可以表示任何数目的处理策略中的一个或多个。这样,所说明和/或描述的各种动作可以以所说明和/或描述的顺序执行、以其它顺序执行、并行执行或者可以省略。同样地,以上描述的过程的次序可以改变。
[0071]本公开的主题包括各种过程、系统和配置和本文所公开的其它特征、功能、动作和/或属性的所有新颖和非显而易见的组合和子组合,以及其任何和全部等同物。