Changing resource utilization associated with a media object based on an engagement score

In accordance with some implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a display. The method includes displaying, on the display, a representation of a first portion of a media object, wherein the first portion of the media object is associated with a first resource utilization value. The first resource utilization value characterizes a utilization of a respective resource by the electronic device. The method includes determining an engagement score that characterizes a level of user engagement with respect to the representation of the first portion of the media object. The method includes changing the utilization of the respective resource from the first resource utilization value to a second resource utilization value based on a function of the engagement score. The second resource utilization value is associated with a second portion of the media object.

TECHNICAL FIELD

The present disclosure relates to presenting a media object, and, in particular, changing device resource utilization associated with presenting the media object.

BACKGROUND

In various circumstances, a device presents a media object, such as displaying a video stream or an avatar associated with a copresence session. In order to display the media object, the device obtains the media object and renders the media object in order to generate corresponding display data for display. Obtaining and rendering the media object utilizes a combination of communication link, processing, and memory resources of the device.

Moreover, user engagement with respect to a media object is a function of various characteristics, such a level of user focus or a location of the media object within a field-of-view of a display. However, the device does not change resource utilization associated with the media object, based on a function of the level of user engagement. Accordingly, the device utilizes excessive resources.

SUMMARY

In accordance with some implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a display. The method includes displaying, on the display, a representation of a first portion of a media object, wherein the first portion of the media object is associated with a first resource utilization value. The first resource utilization value characterizes a utilization of a respective resource by the electronic device. The method includes determining an engagement score that characterizes a level of user engagement with respect to the representation of the first portion of the media object. The method includes changing the utilization of the respective resource from the first resource utilization value to a second resource utilization value based on a function of the engagement score. The second resource utilization value is associated with a second portion of the media object.

In accordance with some implementations, an electronic device includes one or more processors, a non-transitory memory, and a display. One or more programs are stored in the non-transitory memory and are configured to be executed by the one or more processors. The one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions which when executed by one or more processors of an electronic device, cause the device to perform or cause performance of the operations of any of the methods described herein. In accordance with some implementations, an electronic device includes means for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, an information processing apparatus, for use in an electronic device, includes means for performing or causing performance of the operations of any of the methods described herein.

SUMMARY

In various circumstances, a device presents a media object. For example, the device displays a three-dimensional (3D) representation of an object or a video stream. As another example, the device plays audio content via an integrated speaker. In order to present a media object, the device initially obtains the media object. For example, the device may obtain the media object, via a communication link, from local storage or from a separate content delivery server. In order to display a particular media object, the device also renders the particular media object in order to generate corresponding display data for display. Obtaining and rendering a media object utilizes a combination of communication link, processing, and memory resources of the device. Moreover, user engagement with respect to a media object may be a function of various characteristics. For example, a level of focus (e.g., eye gaze of a user) with respect to the media object may vary over time. As another example, in response to a positional change of the device (e.g., a user wearing a head-mountable device (HMD) turns head), the device moves the media object to a different location on the display or ceases to display the media object entirely. However, the device does not change resource utilization associated with the media object based on a function of the level of user engagement. Thus, the device does not, for example, reduce resource utilization in response to a reduction in the level of user engagement. Accordingly, the device utilizes excessive resources in connection with presenting the media object.

By contrast, various implementations disclosed herein include methods, electronic devices, and systems for changing a utilization of a respective resource, by an electronic device, based on a function of an engagement score that is associated with a media object. For example, the media object corresponds to an entire episode of a television show or an entire sequence of a moving three-dimensional (3D) representation of an insect. The engagement score characterizes a level of user engagement with respect to a displayed representation of a first portion of the media object. In some implementations, the engagement score is a function of eye tracking data that is indicative of a gaze of a user. In some implementations, the engagement score is a function of where the representation of the first portion of the media object is located on the display. For example, based on a positional change input that changes the position of the electronic device, the electronic device ceases to display the representation of the first portion of the media object. Accordingly, the electronic device may set the engagement score to a nominal value. Based on the nominal value, the electronic device reduces the resource utilization associated with a second portion of the media object because the second portion of the media object is not within the field-of-view of the display and is thus not viewable by the user. By changing the utilization of the respective resource based on the engagement score, the electronic device reduces resource utilization, as compared with other devices.

In some implementations, the respective resource may include a combination of a rendering resource associated with rendering the media object, and a communication link resource associated with obtaining the media object. For example, in response to determining a reduction of the engagement score, the electronic device instructs a content delivery server to provide the second portion of the media object at an output rate that is lower than an output rate at which the content delivery server provided the first portion of the media object. As another example, in response to determining an increase of the engagement score, the electronic device increases the utilization of a graphics processing unit (GPU) resource in order to render the second portion of the media object at a higher resolution (e.g., more information), as compared with rendering the first portion of the media object.

DESCRIPTION

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

FIG. 1is a block diagram of an example of a portable multifunction device100(sometimes also referred to herein as the “electronic device100” for the sake of brevity) in accordance with some implementations. The electronic device100includes memory102(which optionally includes one or more computer readable storage mediums), a memory controller122, one or more processing units (CPUs)120, a peripherals interface118, an input/output (I/O) subsystem106, a speaker111, a touch-sensitive display system112, an inertial measurement unit (IMU)130, image sensor(s)143(e.g., camera), contact intensity sensor(s)165, audio sensor(s)113(e.g., microphone), eye tracking sensor(s)164(e.g., included within a head-mountable device (HMD)), an extremity tracking sensor150, and other input or control device(s)116. In some implementations, the electronic device100corresponds to one of a mobile phone, tablet, laptop, wearable computing device, head-mountable device (HMD), head-mountable enclosure (e.g., the electronic device100slides into or otherwise attaches to a head-mountable enclosure), or the like. In some implementations, the head-mountable enclosure is shaped to form a receptacle for receiving the electronic device100with a display.

In some implementations, the peripherals interface118, the one or more processing units120, and the memory controller122are, optionally, implemented on a single chip, such as a chip103. In some other implementations, they are, optionally, implemented on separate chips.

The I/O subsystem106couples input/output peripherals on the electronic device100, such as the touch-sensitive display system112and the other input or control devices116, with the peripherals interface118. The I/O subsystem106optionally includes a display controller156, an image sensor controller158, an intensity sensor controller159, an audio controller157, an eye tracking controller160, one or more input controllers152for other input or control devices, an IMU controller132, an extremity tracking controller180, and a privacy subsystem170. The one or more input controllers152receive/send electrical signals from/to the other input or control devices116. The other input or control devices116optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate implementations, the one or more input controllers152are, optionally, coupled with any (or none) of the following: a keyboard, infrared port, Universal Serial Bus (USB) port, stylus, and/or a pointer device such as a mouse. The one or more buttons optionally include an up/down button for volume control of the speaker111and/or audio sensor(s)113. The one or more buttons optionally include a push button. In some implementations, the other input or control devices116includes a positional system (e.g., GPS) that obtains information concerning the location and/or orientation of the electronic device100relative to an operating environment. In some implementations, the other input or control devices116include a depth sensor and/or a time of flight sensor that obtains depth information characterizing an operating environment.

The touch-sensitive display system112provides an input interface and an output interface between the electronic device100and a user. The display controller156receives and/or sends electrical signals from/to the touch-sensitive display system112. The touch-sensitive display system112displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some implementations, some or all of the visual output corresponds to user interface objects. As used herein, the term “affordance” refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to inputs directed toward the graphical user interface object). Examples of user-interactive graphical user interface objects include, without limitation, a button, slider, icon, selectable menu item, switch, hyperlink, or other user interface control.

The touch-sensitive display system112has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on haptic and/or tactile contact. The touch-sensitive display system112and the display controller156(along with any associated modules and/or sets of instructions in the memory102) detect contact (and any movement or breaking of the contact) on the touch-sensitive display system112and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on the touch-sensitive display system112. In an example implementation, a point of contact between the touch-sensitive display system112and the user corresponds to a finger of the user or a stylus.

The touch-sensitive display system112optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other implementations. The touch-sensitive display system112and the display controller156optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system112.

The user optionally makes contact with the touch-sensitive display system112using any suitable object or appendage, such as a stylus, a finger, and so forth. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some implementations, the electronic device100translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.

The speaker111and the audio sensor(s)113provide an audio interface between a user and the electronic device100. Audio circuitry receives audio data from the peripherals interface118, converts the audio data to an electrical signal, and transmits the electrical signal to the speaker111. The speaker111converts the electrical signal to human-audible sound waves. Audio circuitry also receives electrical signals converted by the audio sensors113(e.g., a microphone) from sound waves. Audio circuitry converts the electrical signal to audio data and transmits the audio data to the peripherals interface118for processing. Audio data is, optionally, retrieved from and/or transmitted to the memory102and/or RF circuitry by the peripherals interface118. In some implementations, audio circuitry also includes a headset jack. The headset jack provides an interface between audio circuitry and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone).

The inertial measurement unit (IMU)130includes accelerometers, gyroscopes, and/or magnetometers in order measure various forces, angular rates, and/or magnetic field information with respect to the electronic device100. Accordingly, according to various implementations, the IMU130detects one or more positional change inputs of the electronic device100, such as the electronic device100being shaken, rotated, moved in a particular direction, and/or the like.

The image sensor(s)143capture still images and/or video. In some implementations, an image sensor143is located on the back of the electronic device100, opposite a touch screen on the front of the electronic device100, so that the touch screen is enabled for use as a viewfinder for still and/or video image acquisition. In some implementations, another image sensor143is located on the front of the electronic device100so that the user's image is obtained (e.g., for selfies, for videoconferencing while the user views the other video conference participants on the touch screen, etc.). In some implementations, the image sensor(s) are integrated within an HMD.

The contact intensity sensors165detect intensity of contacts on the electronic device100(e.g., a touch input on a touch-sensitive surface of the electronic device100). The contact intensity sensors165are coupled with the intensity sensor controller159in the I/O subsystem106. The contact intensity sensor(s)165optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor(s)165receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the physical environment. In some implementations, at least one contact intensity sensor165is collocated with, or proximate to, a touch-sensitive surface of the electronic device100. In some implementations, at least one contact intensity sensor165is located on the side of the electronic device100.

The eye tracking sensor(s)164detect eye gaze of a user of the electronic device100and generate eye tracking data indicative of the eye gaze of the user. In various implementations, the eye tracking data includes data indicative of a fixation point (e.g., point of regard) of the user on a display panel, such as a display panel within a head-mountable device (HMD), a head-mountable enclosure, or within a heads-up display.

The extremity tracking sensor150obtains extremity tracking data indicative of a position of an extremity of a user. For example, in some implementations, the extremity tracking sensor150corresponds to a hand tracking sensor that obtains hand tracking data indicative of a position of a hand or a finger of a user within an operating environment. In some implementations, the extremity tracking sensor150utilizes computer vision techniques to estimate the pose of the extremity based on camera images.

In various implementations, the electronic device100includes a privacy subsystem170that includes one or more privacy setting filters associated with user information, such as user information included in extremity tracking data, eye tracking data, and/or body position data associated with a user. In some implementations, the privacy subsystem170selectively prevents and/or limits the electronic device100or portions thereof from obtaining and/or transmitting the user information. To this end, the privacy subsystem170receives user preferences and/or selections from the user in response to prompting the user for the same. In some implementations, the privacy subsystem170prevents the electronic device100from obtaining and/or transmitting the user information unless and until the privacy subsystem170obtains informed consent from the user. In some implementations, the privacy subsystem170anonymizes (e.g., scrambles or obscures) certain types of user information. For example, the privacy subsystem170receives user inputs designating which types of user information the privacy subsystem170anonymizes. As another example, the privacy subsystem170anonymizes certain types of user information likely to include sensitive and/or identifying information, independent of user designation (e.g., automatically).

FIGS. 2A-2Kare an example of changing resource utilization associated with a media object based on an engagement score in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein.

As illustrated inFIG. 2A, an electronic device210is associated with an operating environment200. For example, in some implementations, the operating environment200includes a combination of computer-generated objects and physical objects, such as an augmented reality (AR) or a mixed reality (MR) environment. As another example, in some implementations, the operating environment200includes a physical side wall204and a physical back wall202, and the electronic device210displays, via a display212, a computer-generated display screen220. In some implementations, the operating environment200is a virtual reality (VR) environment in which the entirety of the operating environment200is comprised of computer-generated elements.

The electronic device210is being held by a user50. In some implementations, the electronic device210corresponds to a mobile device, such as a smartphone, laptop, tablet, etc. In some implementations, the electronic device210is similar to and adapted from the electronic device100inFIG. 1.

In some implementations, the electronic device210corresponds to a head-mountable device (HMD) that includes an integrated display (e.g., a built-in display) that displays a representation of the operating environment200. In some implementations, the electronic device210includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device210). For example, in some implementations, the electronic device210slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the representation of the operating environment200. For example, in some implementations, the electronic device210corresponds to a mobile phone that can be attached to the head-mountable enclosure. In various implementations, examples of the electronic device210include smartphones, tablets, media players, laptops, etc.

The electronic device210includes a display212. The display212is associated with a field-of-view214including a portion the operating environment200. The portion the operating environment200includes a display screen220and a video stream within the display screen220. The video stream represents a media object, including a dog221. For example, in some implementations, the electronic device210obtains the media object, and renders the media object in order to generate the video stream for display on the display212.

In some implementations, the display screen220corresponds to a virtual display screen (e.g., a computer-generated television) displayed by the display212, and the electronic device210displays the video stream within the virtual display screen. In some implementations, the display screen220corresponds to a physical display screen (e.g., a real-world television), and the electronic device210displays the video stream within the physical display screen. To that end, in some implementations, the electronic device210performs semantic segmentation with respect to the operating environment200in order to semantically identify, for example, a “television” or a “viewing area.” One of ordinary skill in the art will appreciate that, in some implementations, the electronic device210displays the video stream without displaying a virtual display screen, or displays the video stream independent of a position of a physical display screen within the operating environment200.

As illustrated inFIG. 2B, the display212displays a first portion of the video stream within the display screen220. The first portion of the video stream represents a first portion of the media object. The first portion of the video stream is characterized by a relatively high resolution value, such as a 4K video stream of the dog221. Accordingly, the first portion of the media object is associated with a relatively high, first resource utilization value. The first resource utilization value characterizes a utilization of a respective resource by the electronic device210.

For example, in some implementations, the first resource utilization value characterizes the electronic device210utilizing a relatively high level of a communication link resource in order to obtain the first portion of the media object. For example, the electronic device210utilizes the communication link resource in order to satisfy a performance metric with respect to a quality of service (QoS) metric (e.g., bandwidth, packet loss, jitter, etc.) associated with obtaining the first portion of the media object.

As another example, in some implementations, the first resource utilization value characterizes the electronic device210utilizing a relatively high level of a rendering resource associated with rendering the first portion of the video stream. For example, the electronic device210renders the first portion of the media object in order to generate a relatively high resolution (e.g., 4K) representation of the first portion of the video stream. In some implementations, the first resource utilization value characterizes the utilization of a combination of the communication link resource, the rendering resource, and other device resources associated with displaying a representation of the media object.

In contrast to previously available devices, the electronic device210maintains or changes the utilization of the respective resource based on an engagement score232. The engagement score232characterizes a level of user engagement with respect to a representation of the media stream, such as the displayed video stream representing the dog221. Accordingly, as compared with the previously available devices, in some circumstances the electronic device210reduces resource utilization without degrading the experience of the user50. To that end, in some implementations, as illustrated inFIG. 2C, the electronic device210includes and utilizes one or more eye tracking sensor(s)164and one or more depth sensor(s)231in order to determine the engagement score232. The eye tracking sensor(s)164output eye tracking data associated with the user50, and the electronic device210determines that an eye gaze of the user50is directed to the dog221. Moreover, the depth sensor(s)231output depth data characterizing the operating environment200, and the electronic device210determines a first depth value characterizing a distance between the display212and the display screen220(e.g., the center of the display screen220). One of ordinary skill in the art will appreciate that, in some implementations, the electronic device210determines the engagement score232based on other sensor data.

As illustrated inFIG. 2D, the electronic device210determines the engagement score232having a value of ‘7’ based on the eye tracking data and the depth data. With reference to the example illustrated inFIGS. 2A-2K, the engagement score232ranges from ‘0’ (e.g., not engaged at all) to ‘10’ (highest level of engagement). However, one of ordinary skill in the art will appreciate that other implementations include a different range and/or scale for the engagement score232. Referring back toFIG. 2D, the electronic device210determines a relatively high engagement score232of ‘7’ because the eye gaze of the user50is directed to the dog221.

As further illustrated inFIG. 2D, the electronic device210selects a high resource utilization based on a resource mapping table230, which may be locally stored. The resource mapping table230provides a mapping between engagement scores and resource utilization levels. Namely, because the engagement score is above ‘5,’ the electronic device210selects the high resource utilization, as illustrated inFIG. 2D. Accordingly, the electronic device210maintains the relatively high, first resource utilization value associated with obtaining and/or rendering the first portion of the media object. Thus, as illustrated inFIGS. 2D and 2E, the display212maintains display of the dog221with the relatively high resolution value. As illustrated inFIG. 2E, the dog221begins moving across the display screen220, as is indicated by movement line234. The movement line234is illustrated for purely explanatory purposes. Movement of the dog221from the left side of the display screen220to the right side of the display screen220is illustrated inFIGS. 2F-2K.

As illustrated inFIG. 2F, based on the eye tracking data, the electronic device210determines that the eye gaze of the user50has moved a first distance236away from the display screen220. The first distance236is illustrated for purely explanatory purposes. Moreover, the electronic device210determines, based on the depth data, that the distance between the display212and the display screen220remains at the first depth value. Accordingly, because the eye gaze of the user50is no longer focused on the display screen220or on the dog221, the electronic device210reduces the engagement score from ‘7’ to ‘5,’ as illustrated inFIG. 2F. Based on determining the engagement score of ‘5’ and based on the resource mapping table230, the electronic device210change the utilization of the respective resource from the high resource utilization to a medium resource utilization. Accordingly, as illustrated inFIG. 2F, the electronic device210displays, on the display212, a second portion of the video stream with a medium resolution value (e.g., 1080p). Namely, the dog221inFIG. 2Fis illustrated with long dotted lines in order to indicate a decrease in resolution as compared with the solid line dog221illustrated inFIG. 2E. The second portion of the video stream is associated with a second portion of the media object that is associated with the medium resource utilization. To that end, in some implementations, the electronic device210obtains the second portion of the media object according to a lower utilization of the communication link resource (e.g., lower bandwidth) as compared with obtaining the first portion of the media object. In some implementations, the electronic device210renders the second portion of the media object according to a lower utilization of the rendering resource (e.g., lower GPU processing rate) as compared with rendering the first portion of the media object.

As illustrated inFIG. 2G, based on the eye tracking data, the electronic device210determines that the eye gaze of the user50has moved a second distance238away from the display screen220. The second distance238is illustrated for purely explanatory purposes. On the other hand, the electronic device210determines, based on the depth data, that the distance between the display212and the display screen220remains at the first depth value. The second distance238is larger than the first distance236. Accordingly, the electronic device210further reduces the engagement score from ‘5’ to ‘3.’ Based on determining the engagement score of ‘3’ and based on the resource mapping table230, the electronic device210changes the utilization of the respective resource from the medium resource utilization to a low resource utilization. Accordingly, the electronic device210displays, on the display212, a third portion of the video stream with a low resolution value (e.g., 480p). Namely, the dog221inFIG. 2Gis illustrated with short dotted lines in order to indicate a decrease in resolution as compared with the long dotted line dog221illustrated inFIG. 2F. The third portion of the video stream is associated with a third portion of the media object that is associated with the low resource utilization. To that end, in some implementations, the electronic device210obtains the third portion of the media object according to a lower utilization of the communication link resource (e.g., lower bandwidth) as compared with obtaining the second portion of the media object. In some implementations, the electronic device210renders the third portion of the media object according to a lower utilization of the rendering resource (e.g., lower GPU processing rate) as compared with rendering the second portion of the media object.

As illustrated inFIGS. 2H and 21, the user50and thus the electronic device210move closer to the back wall202, as indicated by device movement line240. The device movement line240is illustrated for purely explanatory purposes. Accordingly, as illustrated inFIG. 2J, the display212displays a larger display screen220and video stream, as compared withFIG. 2G.

As illustrated inFIG. 2K, the electronic device210determines, based on the eye tracking data, that the eye gaze of the user50has moved above the display screen220and has remained at the second distance238from the display screen220. Moreover, the electronic device210determines, based on the depth data, that the distance between the display212and the display screen220(e.g., the center of the display screen220) has decreased from the first depth value to a second depth value. The change from the first depth value to the second depth value results from the movement240of the electronic device210towards the back wall202. Notably, as compared withFIG. 2G, although the eye gaze remains at the second distance238from the display screen220, the electronic device210is closer to the display screen220and to the video stream. Thus, in contrast to the relatively low engagement score of ‘3’ described with reference toFIG. 2G, the electronic device210determines a medium engagement score of ‘5,’ as illustrated inFIG. 2K. Accordingly, based on the resource mapping table230, the electronic device210changes the utilization of the respective resource from the low resource utilization to the medium resource utilization. Accordingly, the electronic device210displays, on the display212, a fourth portion of the video stream with a medium resolution value (e.g., 1080p). Namely, the dog221inFIG. 2Kis illustrated with the long dotted lines in order to indicate an increase in resolution, as compared with the short dotted line dog221illustrated inFIG. 2G.

FIGS. 3A-3Hare another example of changing resource utilization associated with a media object based on an engagement score in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. As illustrated inFIG. 3A, the electronic device210is associated with an operating environment300. The operating environment300includes a back wall302, a side wall304, and a painting311that is hanging on the back wall302. Moreover, the operating environment300includes an avatar310representing an individual associated with (e.g., currently using) a secondary device. For example, in some implementations, the electronic device210and the secondary device are in communication via a copresence session, and the operating environment300functions as a shared operating space (e.g., a real or virtual conference room). In some implementations, the electronic device210generates the avatar310. For example, the electronic device210obtains, from the secondary device, information characterizing the individual (e.g., height, hair color, etc.), and generates the avatar310based on the information.

As illustrated inFIG. 3B, the electronic device210displays, via the display212, the back wall302, the side wall304, the painting311, and the avatar310. The avatar310illustrated inFIG. 3Bis displayed according to a relatively high utilization of a respective resource, such as a high utilization of a rendering resource (e.g., 4K rendering). Moreover, as is illustrated for purely explanatory purposes, a center point312indicates the center point of the display212. As will be described below, in some implementations, the electronic device210determines the engagement score232based on a function of the center point312. One of ordinary skill in the art will appreciate that, in some implementations, the electronic device210determines the engagement score232based on other factors, such as with respect to an edge or a corner of the display212.

As illustrated inFIG. 3C, the electronic device210determines a first distance324between the avatar310and the center point312. Accordingly, as illustrated inFIG. 3D, the electronic device210determines an engagement score of ‘4’ based on the first distance. Based on the engagement score232of ‘4,’ the electronic device210sets the utilization of the respective resource to a medium resource utilization, as indicated within the resource mapping table230inFIG. 3D. Accordingly, as compared with the solid line avatar310illustrated inFIG. 3C, the avatar310inFIG. 3Dhas a dotted line appearance in order to indicate a reduced resolution. The avatar310with the reduced resolution is associated with the electronic device210utilizing the respective resource according to the medium resource utilization, such as GPU utilization for rendering a medium resolution (e.g., 1080p) or a reduced bandwidth level associated with obtaining (e.g., from the secondary device) a corresponding portion of data that represents the avatar310.

As illustrated inFIG. 3E, the electronic device210detects, via one or more input devices (e.g., the IMU130inFIG. 1), a first positional change input332. The first positional change input332is indicative of the electronic device210changing from a first position to a second position. For example, in some implementations, the first positional change input332corresponds to a rotational movement of the electronic device210or a translational movement (e.g., along the x-y axis) of the electronic device210.

In response to detecting the first positional change input332inFIG. 3E, the electronic device210correspondingly moves respective locations of the painting311and the avatar310on the display212inFIG. 3F. Accordingly, the avatar310is moved to a second distance334from the center point312. The second distance334is less than the first distance324. Moreover, the electronic device210changes the engagement score232from ‘4’ to ‘7’ because the avatar310inFIG. 3Fis closer to the center point312than the avatar310inFIG. 3E. Accordingly, based on increasing the engagement score232to ‘7,’ the electronic device210increases the utilization of the respective resource from the second value to a third value. Namely, the display212displays the avatar310inFIG. 3Fhaving a solid line appearance in order to indicate an increased resolution, as compared with the dotted line avatar310illustrated inFIG. 3E.

As illustrated inFIG. 3G, the electronic device210detects, via the one or more input devices, a second positional change input336. The second positional change input336is indicative of the electronic device210changing from the second position to a third position. In response to detecting the second positional change input336inFIG. 3G, the electronic device210determines that neither the painting311nor the avatar310are to be displayed on the display212and thus ceases to display the painting311and the avatar310. Accordingly, the electronic device210reduces the engagement score232from ‘7’ to ‘2’ and correspondingly reduces the resource utilization to a low resource utilization, as illustrated inFIG. 3H. In other words, because the avatar310is no longer displayed on the display212, the electronic device210need not continue to obtain corresponding displayable data associated with the avatar310or render the corresponding displayable data. In some implementations, the low resource utilization corresponds to the electronic device210playing spatial audio340associated with the avatar310, as is illustrated inFIG. 3H. The electronic device210reduces the utilization of communication link and rendering resources because the spatial audio340includes less information than video data, and because the spatial audio340need not be rendered. For example, the electronic device210plays the spatial audio340so as to appear to emanate from the left side of the operating environment300because the avatar310was moved off of the left side of the display212based on the second positional change input336. Thus, the electronic device210provides an undegraded experience to the user50, while saving resources associated with obtaining and rendering video data associated with the avatar310. Accordingly, the electronic device210, as described herein, utilizes less resources than other devices.

FIG. 4is an example of a flow diagram of a method400of changing resource utilization associated with a media object based on an engagement score in accordance with some implementations. In various implementations, the method400or portions thereof are performed by an electronic device including a display (e.g., the electronic device100inFIG. 1or the electronic device210inFIGS. 2A-2KorFIGS. 3A-3H). In various implementations, the method400or portions thereof are performed by a head-mountable device (HMD) including an integrated display. In some implementations, the method400is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method400is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method400are, optionally, combined and/or the order of some operations is, optionally, changed.

As represented by block402, the method400includes displaying, on the display, a representation of a first portion of a media object. For example, with reference toFIG. 2B, the display212displays the first portion of the video stream. As represented by block404, the first portion of the media object is associated with a first resource utilization value, and the first resource utilization value characterizes a utilization of a respective resource by the electronic device. Continuing with the previous example, the first portion of the video stream is associated with a relatively high resource utilization, and thus the electronic device210displays the first portion of the video stream at a correspondingly high resolution inFIG. 2B. In some implementations, the media object is a complete episode of a TV show, a complete movie, a complete sequence of moving 3D textures, etc., and the first portion of the media object corresponds to one or more sequential images of the media object. For example, in some implementations, the media object corresponds to the avatar310described with reference toFIGS. 3A-3H.

As represented by block406, the method400includes determining an engagement score that characterizes a level of user engagement with respect to the representation of the first portion of the media object. The engagement score characterizes the extent to which a user is focused on the representation of the first portion of the media object, such as for how long a user has been gazing at a video stream.

As represented by block408, in some implementations, the method400includes determining the engagement score based on a function of eye tracking data. To that end, an electronic device includes an eye tracker sensor that provides eye tracking data associated with the user. For example, when the representation of the first portion of the media object is in a periphery of the user, the engagement score is medium to low. As another example, when the representation of the first portion of the media object is within the gaze of the user, the engagement score is higher. As yet another example, in some implementations, the representation of the first portion of the media object is displayed at a first location on the display, and the method400includes identifying a gaze position based on a function of the eye tracking data. Continuing with the previous example, in accordance with a determination that the gaze position is less than a threshold distance from the first location, the method400includes setting the engagement score to a first value, and in accordance with a determination that the gaze position is not less than the threshold distance from the first location, the method400includes setting the engagement score to a second value that is less than first value. As one example, with reference toFIG. 2F, the electronic device210sets the engagement score to a value of ‘5’ when the eye gaze of the user (as determined based on eye tracking data) is the first distance236from the display screen220. As another example, with reference toFIG. 2G, the electronic device210sets the engagement score to a lower value of ‘3’ when the eye gaze of the user (as determined based on eye tracking data) is a second distance238from the display screen220. The second distance238is greater than the first di stance236.

As represented by block410, in some implementations, the method400includes determining the engagement score based on a function of depth data. To that end, an electronic device includes a depth sensor that provides depth data with respect to the first portion of the media object. For example, with reference toFIGS. 2G and 2K, the electronic device210determines the same, second distance238between the eye gaze of the user50and the display screen220. However, the electronic device210sets the engagement score232to ‘3’ inFIG. 2Gand to ‘5’ inFIG. 2Kbecause the depth data indicates that the electronic device210is farther from the display region220inFIG. 2Gthan inFIG. 2K.

As represented by block412, in some implementations, the method400includes determining the engagement score based on a function of a positional change input. To that end, an electronic device includes one or more input devices (e.g., an IMU, accelerometer, etc.), and the method400includes detecting, via the one or more input devices, a positional change input. The positional change input is indicative of the electronic device changing from a first position to a second position. For example, the positional change input corresponds to an HMD moving, such as initiated by a head turn of a wearing the HMD. As one example, in response to detecting the first positional change input332inFIG. 3E, the electronic device210repositions the avatar310closer to the center point312of the display212inFIG. 3Fand accordingly increases the engagement score232from ‘4’ to ‘7.’

In some implementations, in response to detecting the positional change input, in accordance with a determination that the representation of the first portion of the media object is located at a first location on the display that is less than a threshold distance from the center of the display, the method400includes setting the engagement score to a first value. Moreover, in accordance with a determination that the representation of the first portion of the media object is located at a second location on the display that is not less than the threshold distance from the center of the display, the method400includes setting the engagement score to a second value that is less than the first value. Moreover, in accordance with a determination that the representation of the first portion of the media object is not located on the display, the method400includes setting the engagement score to a nominal value. For example, in response to detecting the first positional change input332inFIG. 3E, the electronic device210repositions the avatar310closer to the center point312inFIG. 3Fand accordingly increases the engagement score232from ‘4’ to ‘7.’ On the other hand, in response to detecting the second positional change input336inFIG. 3G, the electronic device210determines that the avatar310is no longer displayed inFIG. 3Hand accordingly reduces the engagement score to a nominal value of ‘2.’

As represented by block414, the method400includes changing the utilization of the respective resource from the first resource utilization value to a second resource utilization value based on a function of the engagement score. The second resource utilization value is associated with a second portion of the media object. The second portion of the media object is different from the first portion of the media object. For example, the first portion of the media object corresponds to a first video frame of a video stream, whereas the second portion of the media object corresponds to a second video frame of the video stream. In some implementations, the method400includes registering a change to the engagement score, wherein changing the utilization of the respective resource from the first resource utilization value to the second resource utilization value is in response to determining that the change to the engagement score satisfies a change criterion. As one example, with reference toFIGS. 2F and 2G, in response to determining that the eye gaze is the second distance238from the display screen220and that the second distance238is greater than the first distance236, the electronic device210reduces the engagement score232from ‘5’ to ‘3.’ Continuing with this example, the electronic device210determines that the change from the engagement score232from ‘5’ to ‘3’ satisfies the change criterion, and thus the electronic device210correspondingly reduces the resource utilization from the medium resource utilization to the low resource utilization. As another example, when the change to the engagement score corresponds to an increase to the engagement score, the second resource utilization value is higher than the first resource utilization value, and when the change to the engagement score corresponds to a decrease to the engagement score, the second resource utilization value is lower than the first resource utilization value.

As represented by block416, in some implementations, the respective resource corresponds to a communication link resource. Accordingly, the first resource utilization value characterizes the electronic device obtaining the first portion of the media object, and the second resource utilization value characterizes the electronic device obtaining the second portion of the media object. For example, with reference toFIG. 2E, the electronic device210obtains, via a communication link resource, the first portion of the media object at a relatively high download rate in order to display the corresponding first portion of the video stream at a high resolution (e.g., 4K). As another example, with reference toFIG. 2F, the electronic device210obtains, via the communication link resource, the second portion of the media object at a medium download rate in order to display the corresponding second portion of the video stream at a medium resolution (e.g., 1080p). In some implementations, the method400includes obtaining, according to the first resource utilization value, the first portion of the media object that is provided by a content provisioning system at a first output rate, and obtaining, according to the second resource utilization value, the second portion of the media object that is provided by the content provisioning system at a second output rate that is different from the first output rate.

In some implementations, the content provisioning system is included in the electronic device, such as when the electronic device retrieves the media object from a local non-transitory memory. Accordingly, the communication link resource corresponds to a communication system (e.g., processing resources, memory resources) within the electronic device that facilitates retrieval of the media object from local storage.

In some implementations, the content provisioning system is a content delivery server that is separate from the electronic device, such as a content delivery server that is included in a content delivery network (CDN). For example, the communication link resource corresponds to a network interface, and the electronic device downloads (e.g., streams) the media object via the network interface. For example, in some implementations, the electronic device corresponds to an adaptive bit rate (ABR) enabled device. Continuing the previous example, the electronic device transmits, to the content delivery server, a request for the second portion of the media object in order to change the utilization of the respective resource from the first resource utilization value to the second resource utilization value. As one example, the request corresponds to an HTTP GET command for a specific segment representation of the media object that corresponds to the second portion of the media object. As another example, in some implementations, the specific segment representation is based on various factors including the subscription tier bandwidth allocated to the electronic device and the amount of data currently residing in a playout buffer of the electronic device.

As represented by block418, in some implementations, the respective resource corresponds to a rendering resource. Accordingly, the first resource utilization value characterizes the electronic device rendering the first portion of the media object, and the second resource utilization value characterizes the electronic device rendering the second portion of the media object. To that end, in some implementations, the method400includes rendering, according to the first resource utilization value, the first portion of the media object in order to generate the representation of the first portion of the media object, and rendering, according to the second resource utilization value, the second portion of the media object in order to generate a representation of the second portion of the media object. For example, the electronic device210renders a portion of the avatar310according to a medium utilization of the rendering resource in order to generate and ultimately display a medium resolution avatar310inFIG. 3D. As another example, the electronic device210renders a portion of the avatar310according to a high utilization of the rendering resource in order to generate and ultimately display a high resolution avatar310inFIG. 3E. Accordingly, in some implementations, the representation of the first portion of the media object is characterized by a first video resolution, and the representation of the second portion of the media object is characterized by a second video resolution that is different from the first video resolution. In some implementations, data representing the media object (e.g., a model of the media object) is locally stored in the non-transitory memory of the electronic device, and the electronic device obtains merely positional information characterizing the media object. Thus, the electronic device utilizes a relatively low utilization of a network resource, but may utilize a medium to high utilization of a rendering resource when the media object is to be displayed at a corresponding medium to high resolution.

As represented by block420, in some implementations, in response to determining that the engagement score falls below a threshold (e.g., is set to a nominal value), the method400includes ceasing to display the representation of the first portion of the media object, and playing, via a speaker in the electronic device, an audio representation of the second portion of the media object. The audio representation may correspond to spatial audio that is played based on a function of the engagement score. For example, in response to determining that the avatar310is no longer displayed inFIG. 3H, the electronic device210sets the engagement score to ‘2’ and accordingly plays the spatial audio340, without obtaining or rendering video data associated with the avatar310. Thus, by foregoing obtaining and rendering video in some circumstances, the electronic device210reduces overall resource utilization without degrading the user experience.

Various processes defined herein consider the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized in order to provide an improved privacy screen on an electronic device. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. As described herein, the user should have knowledge of and control over the use of their personal information.

Personal information will be utilized by appropriate parties only for legitimate and reasonable purposes. Those parties utilizing such information will adhere to privacy policies and practices that are at least in accordance with appropriate laws and regulations. In addition, such policies are to be well-established, user-accessible, and recognized as in compliance with or above governmental/industry standards. Moreover, these parties will not distribute, sell, or otherwise share such information outside of any reasonable and legitimate purposes.

Users may, however, limit the degree to which such parties may access or otherwise obtain personal information. For instance, settings or other preferences may be adjusted such that users can decide whether their personal information can be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, various aspects of these features can be implemented without the need to use such information. As an example, if user preferences, account names, and/or location history are gathered, this information can be obscured or otherwise generalized such that the information does not identify the respective user.