Patent Description:
The present description relates generally to electronic devices, including, for example, protected access to rendering information for electronic devices.

Electronic devices often include applications that generate content to be displayed using a display of the electronic device. Document <CIT> is a relevant prior-art in this field.

However, for purpose of explanation, several implementations of the subject technology are set forth in the following figures.

The invention is defined in the independent claims <NUM>, <NUM> and <NUM>.

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

Implementations of the subject technology described herein may provide protected access, by a first process such as an application, to rendering information that can be used by the first process for rendering display frames. For example, the rendering information may be, or may be based on, user information for which privacy protection is desired. In one or more implementations, the rendering information may be, for example, a resolution map generated by a system process of a device using gaze information for a user. The protected access provided by the subject technology may prevent the first process from exporting the rendering information, or information derived from the rendering information, from a protected processing environment. In this way, the first process can be allowed operate on the rendering information in the protected processing environment, such as to generate output for display, without allowing the first process to store, export, or otherwise extricate user-related information from the protected processing environment. Although user-related rendering information, used for rendering output for display, is described herein in connection with various examples, aspects of the subject technology may also be applied, as described herein, to provide protected access to other user-related data for generating device outputs other than display output.

<FIG> illustrates an example system architecture <NUM> including various electronic devices that may implement the subject system in accordance with one or more implementations. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the claims as set forth herein Additional components, different components, or fewer components may be provided.

The system architecture <NUM> includes an electronic device <NUM>, an electronic device <NUM>, an electronic device <NUM>, and a server <NUM>. For explanatory purposes, the system architecture <NUM> is illustrated in <FIG> as including the electronic device <NUM>, the electronic device <NUM>, the electronic device <NUM>, and the server <NUM>; however, the system architecture <NUM> may include any number of electronic devices and any number of servers or a data center including multiple servers.

The electronic device <NUM> may be smartphone, a tablet device, or a wearable device such as a head mountable portable system, that includes a display system capable of presenting a visualization of an extended reality environment or other display environment to a user (e.g., user <NUM>). The electronic device <NUM> may be powered with a battery and/or any other power supply. In an example, the display system of the electronic device <NUM> provides a stereoscopic presentation of the extended reality environment, enabling a three-dimensional visual display of a rendering of a particular scene, to the user. In one or more implementations, instead of, or in addition to, utilizing the electronic device <NUM> to access an extended reality environment, the user may use a handheld electronic device <NUM>, such as a tablet, watch, mobile device, and the like.

The electronic device <NUM> may include one or more cameras such as camera(s) <NUM> (e.g., visible light cameras, infrared cameras, eye tracking cameras, etc.) Further, the electronic device <NUM> may include various sensors such as sensor(s) <NUM> including, but not limited to, cameras, image sensors, touch sensors, microphones, inertial measurement units (IMU), heart rate sensors, temperature sensors, Lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communications sensors, etc.) Moreover, the electronic device <NUM> may include hardware elements that can receive user input such as hardware buttons or switches. User input detected by such sensors and/or hardware elements correspond to various input modalities for initiating recording within a given extended reality environment. For example, such input modalities may include, but not limited to, facial tracking, eye tracking (e.g., gaze direction or gaze location tracking), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breath, temperature, electroencephalogram, olfactory), recognizing speech or audio (e.g., particular hotwords), and activating buttons or switches, etc. The electronic device <NUM> may also detect and/or classify physical objects in the physical environment of the electronic device <NUM>.

The electronic device <NUM> may be communicatively coupled to a base device such as the electronic device <NUM> and/or the electronic device <NUM>. Such a base device may, in general, include more computing resources and/or available power in comparison with the electronic device <NUM>. In an example, the electronic device <NUM> may operate in various modes. For instance, the electronic device <NUM> can operate in a standalone mode independent of any base device. When the electronic device <NUM> operates in the standalone mode, the number of input modalities may be constrained by power limitations of the electronic device <NUM> such as available battery power of the device. In response to power limitations, the electronic device <NUM> may deactivate certain sensors within the device itself to preserve battery power.

The electronic device <NUM> may also operate in a wireless tethered mode (e.g., connected via a wireless connection with a base device), working in conjunction with a given base device. The electronic device <NUM> may also work in a connected mode where the electronic device <NUM> is physically connected to a base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device is charging the electronic device <NUM> while physically connected).

When the electronic device <NUM> operates in the wireless tethered mode or the connected mode, a least a portion of processing user inputs and/or rendering the extended reality environment may be offloaded to the base device thereby reducing processing burdens on the electronic device <NUM>. For instance, in an implementation, the electronic device <NUM> works in conjunction with the electronic device <NUM> or the electronic device <NUM> to generate an extended reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the extended reality environment in a real-time manner. In an example, the electronic device <NUM> provides a rendering of a scene corresponding to the extended reality environment that can be perceived by the user and interacted with in a real-time manner. Additionally, as part of presenting the rendered scene, the electronic device <NUM> may provide sound, and/or haptic or tactile feedback to the user. The content of a given rendered scene may be dependent on available processing capability, network availability and capacity, available battery power, and current system workload.

The electronic device <NUM> may also detect events that have occurred within the scene of the extended reality environment. Examples of such events include detecting a presence of a particular person, entity, or object in the scene. Detected physical objects may be classified by electronic device <NUM>, electronic device <NUM>, and/or electronic device <NUM> and the location, position, size, dimensions, shape, and/or other characteristics of the physical objects can be used to coordinate the rendering of virtual content, such as a UI of an application, for display within the XR environment.

The network <NUM> may communicatively (directly or indirectly) couple, for example, the electronic device <NUM>, the electronic device <NUM> and/or the electronic device <NUM> with the server <NUM> and/or one or more electronic devices of one or more other users. In one or more implementations, the network <NUM> may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet.

The electronic device <NUM> may include a touchscreen and may be, for example, a smartphone that includes a touchscreen, a portable computing device such as a laptop computer that includes a touchscreen, a peripheral device that includes a touchscreen (e.g., a digital camera, headphones), a tablet device that includes a touchscreen, a wearable device that includes a touchscreen such as a watch, a band, and the like, any other appropriate device that includes, for example, a touchscreen, or any electronic device with a touchpad. In one or more implementations, the electronic device <NUM> may not include a touchscreen but may support touchscreen-like gestures, such as in an extended reality environment. In one or more implementations, the electronic device <NUM> may include a touchpad. In <FIG>, by way of example, the electronic device <NUM> is depicted as a mobile smartphone device with a touchscreen. In one or more implementations, the electronic device <NUM>, the handheld electronic device <NUM>, and/or the electronic device <NUM> may be, and/or may include all or part of, the electronic system discussed below with respect to <FIG>. In one or more implementations, the electronic device <NUM> may be another device such as an Internet Protocol (IP) camera, a tablet, or a peripheral device such as an electronic stylus, etc..

The electronic device <NUM> may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In <FIG>, by way of example, the electronic device <NUM> is depicted as a desktop computer. The electronic device <NUM> may be, and/or may include all or part of, the electronic system discussed below with respect to <FIG>.

The server <NUM> may form all or part of a network of computers or a group of servers <NUM>, such as in a cloud computing or data center implementation. For example, the server <NUM> stores data and software, and includes specific hardware (e.g., processors, graphics processors and other specialized or custom processors) for rendering and generating content such as graphics, images, video, audio and multi-media files for extended reality environments. In an implementation, the server <NUM> may function as a cloud storage server that stores any of the aforementioned extended reality content generated by the above-discussed devices and/or the server <NUM>.

<FIG> illustrates an example display environment in which the display <NUM> displays a user interface (UI) <NUM> of an application running on the electronic device <NUM>. In the example of <FIG>, a rendered display frame fills the viewable area of the display <NUM> and includes the user interface <NUM>. In the example of <FIG>, UI <NUM> includes a UI element <NUM> and UI elements <NUM>. UI elements <NUM> and <NUM> may correspond to, as illustrative examples, a sub-window of a UI window, static elements such as images, and/or dynamic elements such as video streams and/or virtual characters or other game or interactive elements.

In the example of <FIG>, a user (e.g., user <NUM> of <FIG>) of the electronic device <NUM> is gazing at a gaze location <NUM> on display <NUM>. For example, images captured by camera(s) <NUM> and/or sensor data captured by sensor(s) <NUM>, and corresponding to the user <NUM>, may be used (e.g., by a system process at the electronic device <NUM>) to determine the gaze location <NUM> (e.g., in terms of the coordinates of a pixel or group of pixels of the display <NUM>). For example, the images may include optical and/or infrared images of one or both eyes of the user. In this example use case of <FIG>, the gaze location <NUM> is within the boundary of the UI element <NUM>.

In the example of <FIG>, the display frame that is displayed on the display <NUM> is a foveated display frame in which a first portion <NUM> of the display frame that is within a predefined distance from the gaze location <NUM> (e.g., within a boundary <NUM>) is displayed with a first resolution, and a second portion <NUM> of the display frame that is outside the predefined distance (e.g., outside the boundary <NUM>) is displayed with a second, lower resolution. In this way, user information, such as the gaze location <NUM>, can be used to save power and/or processing resources of the device, by allowing pixels in the second portion <NUM> to be rendered at a lower resolution, when the user is not gazing on that portion of the display. For explanatory purposes foveation is described herein with reference to the resolution of the content; however, the foveation may also be applicable to bit rate, compression, or any other encoding aspect/feature of the content.

In the example of <FIG>, the boundary <NUM> is indicated by a dashed line. However, this is merely for ease of understanding and it is appreciated that the boundary <NUM> between the first portion <NUM> (e.g., the high resolution portion) and the second portion (e.g., the low resolution portion) of the display frame may be constructed so as to be imperceptible by the user. For example, the boundary <NUM> may have a radial or other width and may have a resolution that decreases across the width from the resolution of the first portion <NUM> (e.g., a resolution at an outer edge of the first portion <NUM>) to the resolution of the second portion <NUM> (e.g., a resolution at an inner edge of the second portion <NUM>), to create a smooth transition that is imperceptible by the user. Moreover, the boundary <NUM> of <FIG> is depicted as a round boundary, but may be implemented with other forms and/or shapes (e.g., the shape of the UI element <NUM>) in various implementations. Further, the resolution of the first portion <NUM> and/or the resolution of the second portion <NUM> may also be varied as a function of distance from the gaze location <NUM> and/or as a function of the displayed content. Further, in the example of <FIG>, the UI <NUM> occupies the entire viewable area of the display <NUM>. However, it is appreciated that, in one or more use cases, the UI <NUM> may be displayed at a first location on the display while other display content (e.g., system content and/or display content from one or more other applications) is concurrently displayed at other locations on the display.

As the gaze location <NUM> moves on the display <NUM>, the electronic device <NUM> may track and update the locations and/or shapes of the first portion <NUM> and the second portion <NUM> to continue to be substantially centered on the gaze location <NUM>. As discussed herein, it may be desirable to allow the application to which the UI <NUM> corresponds to generate a user-specific output (e.g., a foveated display frame of the type shown in <FIG>), without providing the application access to user-specific information such as the gaze location <NUM>. For example, the user-specific information may include the gaze location <NUM> itself, image data and/or sensor data on which the gaze location or other user characteristics and/or behavior are based, and/or information derived from the gaze location, the image data, and/or the sensor data.

Aspects of the subject technology can provide this ability for applications to output application content based on user information, without providing the user information to the application and without providing the application with the ability to export, store, and/or otherwise output the user information or information based on the user information (e.g., other than directly to the display <NUM> or directly to another output component). For example, a system process at the electronic device <NUM> may generate rendering information, based on user information such as the gaze location <NUM>, that can be provided to the application (e.g., in a protected manner as described in further detail hereinafter). The application can then use the rendering information to render application content which is therefore effectively rendered based on the user information.

For example, <FIG> illustrates one example of rendering information that can be used by an application to render the UI <NUM> of <FIG>. In the example of <FIG>, the rendering information is a resolution map <NUM> that has been generated based on the gaze location <NUM> of <FIG>. In this example, the resolution map <NUM> is a foveation map that has a first portion <NUM> spatially corresponding to the first portion <NUM> of the display frame of <FIG> and a second portion <NUM> spatially corresponding to the second portion <NUM> of the display frame of <FIG>, and separated by a boundary <NUM>, spatially corresponding to the boundary <NUM> of <FIG>. In this example, the resolution map <NUM> does not include any content for display, and instead provides a pixel resolution at each location in the map. An application may render application content (e.g., the UI <NUM> of <FIG>) with a resolution as indicated based on the resolution map <NUM>. For example, the resolution map <NUM> may be implemented as a variable rasterization rate (VRR) map, based upon which an application can render a display frame with higher pixel density in some areas (e.g., in areas of the display frame spatially corresponding to the first portion <NUM> of the resolution map <NUM>), and relatively lower pixel density in other areas (e.g., in areas of the display frame spatially corresponding to the second portion <NUM> of the resolution map <NUM>). The resolution map <NUM> is one example of resolution information that can be provided to a process such as an application for use in rendering, and other rendering information that is based on a user position or other user characteristic(s) and that can be used for rendering of application content is also contemplated.

As discussed herein, in one or more implementations, user-related information such as rendering information (e.g., resolution map <NUM>) may be provided to an application in a protected manner (e.g., in a manner that prevents the application from storing or exporting the user-related information, the rendering information, and/or information derived from the user-related information and/or the rendering information).

For example, <FIG> illustrates an example architecture that may be implemented by the electronic device <NUM> in accordance with one or more implementations of the subject technology. For explanatory purposes, portions of the architecture of <FIG> are described as being implemented by the electronic device <NUM> of <FIG>; however, appropriate portions of the architecture may be implemented by any other electronic device. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.

Various portions of the architecture of <FIG> can be implemented in software or hardware, including by one or more processors and a memory device containing instructions, which when executed by the processor cause the processor to perform the operations described herein. For example, in <FIG>, the trapezoidal boxes may indicate that the sensor(s) <NUM>, the camera(s) <NUM>, the primary processor <NUM>, the processor <NUM>, the memory <NUM>, and the display <NUM> may be hardware components, and the rectangular boxes may indicate that an the system process(es) <NUM> and the application <NUM> may be implemented in software that is executed by the primary processor <NUM> and/or the processor <NUM>. In one or more implementations, the primary processor <NUM> may be a central processing unit (CPU) and the processor <NUM> may be a separate co-processor such as graphics processing unit (GPU), a neural processor, a secure processor, or any other co-processor that allows the application <NUM> and/or the system process(es) <NUM> to write an instruction sequence to the memory of the processor <NUM>, for execution by the processor <NUM> to generate a user-specific output (e.g., a rendered frame such a foveated frame, other display content that depends on the user's current position, motion, and/or perspective, spatial audio output, etc., and/or non-display output such as audio output or tactile output that depends on the user's current position, motion, and/or perspective, spatial audio output, etc.).

The example of <FIG> illustrates the electronic device <NUM> operating to generate rendering information based on user information, for use by a process such as an application. As shown, sensor data from sensor(s) <NUM> and/or camera data from camera(s) <NUM> may be provided to system process(es) <NUM> running on (e.g., being executed by) the primary processor <NUM>. In one or more use cases, some or all of the sensor data from sensor(s) <NUM> and/or camera data from camera(s) <NUM> may represent user-specific data such as images and/or other measurements of user physical characteristics, movements, or the like. For example, the camera data may include one or more optical and/or infrared images of the user's eyes.

In the example of <FIG>, the system process(es) <NUM> obtain user information. For example, the user information may be the sensor data and/or the camera data. In other examples, the user information may be derived from the sensor data and/or the camera data by the system process(es) <NUM>. For example, the system process(es) <NUM> may determine a gaze location (e.g., the gaze location <NUM> of <FIG>) from the sensor data and/or the camera data.

As shown in <FIG>, the system process(es) <NUM> may generate rendering information based on the user information. As an example, the rendering information may include a resolution map, such as the resolution map <NUM> of <FIG>, based on the user information (e.g., based on the gaze location <NUM>). The system process(es) <NUM> may then provide the rendering information to the processor <NUM> for storage in memory <NUM> of the processor <NUM>. In the example of <FIG>, the memory <NUM> is depicted as physical memory of the processor <NUM> (e.g., on-chip memory on a separate chip from the primary processor <NUM> and/or dedicated off-chip memory such as dynamic random access memory (DRAM)). However, in other implementations, the memory <NUM> may be a protected portion of memory that is shared with the primary processor <NUM> (e.g., on-chip memory of a system-on-chip (SoC) that includes the primary processor <NUM>, the processor <NUM>, and the memory <NUM>), the portion allocated (e.g., by a kernel process at the electronic device) as or as part of a processing environment for the processor <NUM>. The system process(es) <NUM> may then provide a location of the rendering information in the memory <NUM> of the processor <NUM>, to an application such as application <NUM> (e.g., an application having a UI such as the UI <NUM> of <FIG>). As examples, the system process(es) <NUM> may receive the location from the processor <NUM> and provide the received location to the application <NUM>, or the system process(es) <NUM> can define the location and provide the location to the application <NUM>.

In one or more implementations, the system process(es) <NUM> may generate one or more buffers <NUM> (e.g., a pool <NUM> of buffers <NUM>) in the memory <NUM> of the processor <NUM>. For example, the pool <NUM> may include three buffers <NUM>, four buffers <NUM>, five buffers <NUM>, or other number of buffers. The number of buffers may be based on application processing settings. For example, some applications may generate application content for some display frames that depend on the application content of other (e.g., prior or upcoming) display frames, and may utilize more than one buffer. In one or more implementations, the primary processor <NUM> may generate (e.g., allocate) the pool <NUM> of buffers <NUM> and provide pointers to the buffers <NUM> to the application <NUM>. In these implementations, the primary processor <NUM> may then, on a per-frame basis, generate the rendering information (e.g., the resolution map <NUM>), select a buffer <NUM> from the pool <NUM> of buffers <NUM>, store the rendering information in the selected buffer <NUM>, and provide an index of the selected buffer to the application (as the location of the memory in which the rendering information is stored).

<FIG> illustrates the example architecture of <FIG> for the electronic device <NUM>, in an operating state in which the application <NUM> uses the rendering information, provided by the primary processor (e.g., by the system process(es) <NUM>) to the processor <NUM>, to generate display content. In the example of <FIG>, the rendering information <NUM> provided by the system process(es) <NUM> has been stored in the memory <NUM> of the processor <NUM> (e.g., in a buffer <NUM>).

As shown, the application <NUM> may provide application instructions (e.g., an instruction sequence) to the memory <NUM> of the processor <NUM>, for execution within an processing environment of the processor <NUM> (e.g., a processing environment encompassing or associated with the memory <NUM>). As shown, the application instructions may include the location of the rendering information <NUM> (e.g., the location of the memory, such as the buffer <NUM>, in which the rendering information <NUM> is stored). For example, the application instructions, when executed by the processor <NUM> (within the processing environment of the processor <NUM>) may access the rendering information <NUM> (from within the processing environment of the processor <NUM>) using the location received from the system process(es) <NUM>, and generate display content using the rendering information <NUM>. As one illustrative example, the application instructions may include instructions which, when executed by the processor <NUM> (within the processing environment of the processor <NUM>) cause the processor <NUM> to generate the foveated display frame of <FIG> using the resolution map <NUM> of <FIG>), and provide the foveated display frame as display content for display by the display <NUM>.

Because the application <NUM> can only access the rendering information <NUM> using the application instructions that are executed by the processor <NUM> within the processing environment of the processor <NUM>, the rendering information <NUM>, any user information on which the rendering information <NUM> is based, and any information derived from the rendering information <NUM> using the application instructions, can be prevented from being later accessed, stored, or exported by the application <NUM> and/or generally extracted from the processing environment of the processor <NUM>.

For example, <FIG> illustrates an example of how the rendering information <NUM>, and information derived from the rendering information <NUM> within the processing environment of the processor <NUM>, may be prevented from being extricated from the processing environment of the processor <NUM>. For example, <FIG> illustrates an operating state in which the application <NUM> provides an application export request, requesting export of the rendering information <NUM> (e.g., and/or information derived from the rendering information <NUM>). As shown, extrication of the rendering information <NUM> (e.g., and/or information derived from the rendering information <NUM>) responsive to the application export request is prevented. In the example of <FIG>, the application export request is illustrated as originating from the application <NUM> running on the primary processor <NUM>. However, it is appreciated that, even if the application export request is generated from within the processing environment of the processor <NUM> (e.g., by the execution of the application instructions being executed by the processor <NUM> to generate the display content), the extrication of the rendering information <NUM> (e.g., and/or information derived from the rendering information <NUM>) responsive to the application export request is prevented. In the example of <FIG>, the application export request is illustrated as being provided from the application <NUM> to the processor <NUM> and blocked at the processor <NUM>. However, it is also appreciated that the application <NUM> may attempt to provide the application export request to the processor <NUM> by providing the request (e.g., via a method call) to one or more of the system process(es) <NUM> (e.g., a kernel process) running on the primary processor <NUM>, which can intercept the application export request and prevent the application export request from being executed before the request leaves the primary processor <NUM>.

<FIG> also illustrates an optional use case in which even the system process(es) <NUM> that generated the rendering information <NUM> are prevented from accessing and/or extricating the rendering information <NUM> (e.g., and/or information derived from the rendering information <NUM> within the processing environment of the processor <NUM>) from the processing environment of the processor <NUM>. In this example, the system process(es) <NUM> provide an export request (e.g., to a kernel process of the system process(es) <NUM> and/or to the processor <NUM>), and the extrication of the rendering information <NUM> (e.g., and/or information derived from the rendering information <NUM> within the processing environment of the processor <NUM>) responsive to the export request is prevented. In this way, one or more of the system process(es) <NUM> may allow the processor <NUM> to read data from the buffer <NUM> and write data to the buffer and prevent the primary processor <NUM> from reading data from the buffer <NUM>.

In one or more implementations, the memory <NUM> of the processor <NUM> may be a portion of a shared memory of the electronic device <NUM> (e.g., portions of which are allocated for the primary processor <NUM> and other portions of which can be allocated to the processor <NUM>), and the prevention of the extrication of the resolution information from the memory <NUM> (e.g., from the portion of the shared memory corresponding to the processing environment of the processor <NUM>) can be enforced by a kernel process (e.g., corresponding to one or more of the system process(es) <NUM>) of the electronic device <NUM>. In this way, the kernel process may configure the processing environment to prevent extrication of the rendering information, and information derived from the rendering information within the processing environment, from the processing environment. In one or more implementations, the shared memory may be on-chip memory of a system-on-chip that includes the primary processor <NUM> and/or the processor <NUM>, and/or may include off-chip memory such as DRAM. In one or more other implementations, the memory <NUM> of the processor <NUM> may be dedicated memory of the processor <NUM> that is physically inaccessible from outside the processor <NUM>.

<FIG> illustrates a flow diagram of an example process for providing protected access to rendering information according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a system process (e.g., one or more of system process(es) <NUM>) of an electronic device (e.g., electronic device <NUM>) may obtain user information corresponding to a user (e.g., user <NUM>) of the electronic device. In one or more implementations, the user information may include sensor data and/or camera data, and/or information derived from sensor data and/or camera data. For example, in one or more implementations, the user information may include gaze information, such as a gaze location (e.g., gaze location <NUM>) at which a user's gaze is directed on a display (e.g., display <NUM>) of the electronic device. In one or more implementations, the user information may include other user-related information such as head position information, user motion information, user gesture information, or the like.

At block <NUM>, the system process may generate rendering information (e.g., rendering information <NUM>) based on the user information. In one or more implementations, the rendering information may include a resolution map (e.g., resolution map <NUM>) based on the gaze information. For example, the resolution map may include a foveation map (e.g., a VRR map). The rendering information may also, or alternatively, include any other information that is user-specific and that can be used, by a process such as an application, to render a display frame including application content or the generate other user-specific device output.

At block <NUM>, the system process may store the rendering information in a memory (e.g., memory <NUM>) of a processor (e.g., a co-processor such as processor <NUM>) of the electronic device other than a primary processor (e.g., primary processor <NUM>) of the electronic device. In one or more implementations, the processor is configured to allow an application (e.g., application <NUM>) to write an instruction sequence (e.g., application instructions) to the memory of the processor for execution by the processor to generate a rendered frame. For example, the processor may be a graphics processing unit (GPU), in one or more implementations.

At block <NUM>, the system process may provide, to an application (e.g., application <NUM>) running on the electronic device, a location of the rendering information in the memory of the processor. For example, the location may be a memory index of a buffer (e.g., a buffer <NUM>) generated, by the system process, in the memory of the processor. In one or more implementations, the system process allows the processor to read data from the buffer and write data to the buffer and prevents the primary processor from reading data from the buffer. In one or more implementations, the system process may generate a pool of buffers (e.g., a pool <NUM> of buffers <NUM>) including the buffer in the memory of the processor. In one or more implementations, the system process may also select the buffer, from the pool of buffers, before storing the rendering information in the buffer.

At block <NUM>, the processor may provide for display, a rendered frame generated by the application within a processing environment of the processor based on the rendering information. For example, the rendered frame may include a foveated rendered frame (e.g., as described above in connection with <FIG>) including application content modified by the application based on the foveation map. In one or more implementations, the processing environment is configured to prevent extrication of the rendering information, and information derived from the rendering information within the processing environment, from the processing environment. The electronic device may also display the rendered frame by a display (e.g., display <NUM>) of the electronic device.

As described above, aspects of the subject technology may include the collection and transfer of data from an application to other users' computing devices. The present disclosure contemplates that in some instances, this collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include images, sensor data, gaze information, head position and/or characteristic information, motion information, environment information, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used in generating outputs, such as rendered display frames, for a specific user. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used, in accordance with the user's preferences to provide insights into their general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.

Despite the foregoing, the present disclosure also contemplates implementations in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of generating outputs, such as rendered display frames, for a specific user, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services or anytime thereafter. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.

<FIG> illustrates an example computing device with which aspects of the subject technology may be implemented in accordance with one or more implementations. The computing device <NUM> can be, and/or can be a part of, any computing device or server for generating the features and processes described above, including but not limited to a laptop computer, a smartphone, a tablet device, a wearable device such as a goggles or glasses, and the like. The computing device <NUM> may include various types of computer readable media and interfaces for various other types of computer readable media. The computing device <NUM> includes a permanent storage device <NUM>, a system memory <NUM> (and/or buffer), an input device interface <NUM>, an output device interface <NUM>, a bus <NUM>, a ROM <NUM>, one or more processing unit(s) <NUM>, one or more network interface(s) <NUM>, and/or subsets and variations thereof.

The bus <NUM> collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computing device <NUM>.

The ROM <NUM> stores static data and instructions that are needed by the one or more processing unit(s) <NUM> and other modules of the computing device <NUM>. The permanent storage device <NUM> may be a non-volatile memory unit that stores instructions and data even when the computing device <NUM> is off.

The input device interface <NUM> enables a user to communicate information and select commands to the computing device <NUM>. The output device interface <NUM> may enable, for example, the display of images generated by computing device <NUM>.

Finally, as shown in <FIG>, the bus <NUM> also couples the computing device <NUM> to one or more networks and/or to one or more network nodes through the one or more network interface(s) <NUM>. In this manner, the computing device <NUM> can be a part of a network of computers (such as a LAN, a wide area network ("WAN"), or an Intranet, or a network of networks, such as the Internet. Any or all components of the computing device <NUM> can be used in conjunction with the subject disclosure.

It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components (e.g., computer program products) and systems can generally be integrated together in a single software product or packaged into multiple software products.

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term "include", "have", or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term "comprise" as "comprise" is interpreted when employed as a transitional word in a claim.

§ <NUM>(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, the element is recited using the phrase "step for".

Claim 1:
A method, comprising:
obtaining, with a system process running on a primary processor of an electronic device, user information corresponding to a user of the electronic device;
generating, by the system process, rendering information based on the user information;
storing, by the system process, the rendering information in a memory of a secondary processor of the electronic device, the secondary processor of the electronic device being a processor other than the primary processor of the electronic device;
providing, from the system process to an application running on the primary processor of the electronic device, a location of the rendering information in the memory of the secondary processor; and
providing, by the secondary processor and for display, a rendered frame generated by executing application instructions received from the application within a processing environment of the secondary processor, the application instructions including the location of the rendering information, wherein the rendered frame is generated based on the rendering information stored at the location, further wherein the application is prevented from accessing the rendering information outside the processing environment of the secondary processor.