Systems and Methods for Performing Global Dimming in Artificial Reality Systems

In particular embodiments, a computing system may receive receiving a first image frame of a sequence of image frames to be shown on a display having a plurality of backlight zones. The computing system may compute backlight unit statistics of the first image frame. The backlight unit statistics may represent grayscale levels for the plurality of backlight zones. The computing system may compute a global dimming gain for adjusting color values and backlight unit intensity of a second image frame of the sequence of image frames based on the backlight unit statistics of the first image frame. The computing system may adjust, using the global dimming gain, the color values and the backlight unit intensity of the second image frame.

TECHNICAL FIELD

This disclosure generally relates to global dimming. In particular, the disclosure relates to a technique for performing global dimming in artificial reality systems, such as virtual reality (VR) headsets.

BACKGROUND

Global dimming (GD) is a technique used in liquid crystal displays (LCD) to reduce power consumption of a backlight unit. When frame content to be displayed is not fully white, GD scales up the RGB values (e.g., LC transmittance) and scales down the backlight intensity to achieve the same final visual image at lower power cost. More specifically, GD typically scales up RGB values and dim down the BLU intensity at the same time, so it may preserve visual quality while saving power consumption.

In the conventional approaches for local and/or global dimming, red (R), green (G), and blue (B) (collectively herein referred to as RGB) information is provided by a graphics source, and a dedicated hardware (e.g., dedicated ASIC) computes a dimming or backlight matrix. The backlight matrix is then fed back to the graphics source so that the source may adjust the RGB for local and/or global dimming. An image may then be displayed based on the adjusted RGB and backlight intensity represented by the backlight matrix. The feedback loop (i.e., sending the backlight matrix back to the graphics source), however, introduces latency. Also, the RGB color adjustment consumes too much power. Performing such local and global dimming in artificial reality space is challenging.

Performing local and global dimming in artificial reality space is challenging as artificial reality systems (e.g., VR headsets) do not have a dedicated hardware for computing backlight unit statistics or backlight matrix, and therefore the computation needs to be performed by the standard system's central processing unit (CPU). Additionally, low/no latency is an important factor when displaying content through these systems. Therefore, an improved solution or technique is needed, especially for VR headset displays, that can perform local dimming as well as global dimming with low latency and power requirements.

SUMMARY OF PARTICULAR EMBODIMENTS

Embodiments described herein relate to performing global dimming for artificial reality systems. Particular embodiments describe an efficient method, algorithm, and/or technique to perform global dimming in addition to local dimming. The global dimming technique discussed herein is an extension to the local dimming (LD) technique discussed in U.S. patent application Ser. No. 17/713,965, filed 5 Apr. 2022, which is hereby incorporated by reference in its entirety.

At a high level, to perform global dimming, image statistics (e.g., backlight unit (BLU) stats) of a previous frame are used to adjust RGB frame and BLU intensity for a current frame. By way of an example and without limitation, if there are three frames T1, T2, and T3, where T1is the first frame, T2is the second frame, and T3is the third frame in a sequence of frames to be displayed, then data associated with BLU stats of the T1frame is used to adjust the RGB values and backlight intensity for displaying the T2frame, and data associated with BLU stats of the T2frame is used to adjust the RGB values and backlight intensity for displaying the T3frame. Here, the first frame T1may be displayed at its original RGB value. However, the backlight intensity may still be adjusted, particularly in the low-level grayscale regions for local dimming.

In particular embodiments, a global dimming and local dimming algorithm (also interchangeably herein referred to as GD/LD algorithm) discussed herein may calculate a global dimming gain based on image statistics (e.g., BLU stats) associated with an image frame. The calculated global dimming gain may be used to adjust RGB values and BLU intensity of a subsequent or next frame. The global dimming gain to scale up RGB values is inversely proportional to the BLU intensity. That is, the global dimming gain increases with decreasing BLU intensity. By way of an example, if the BLU stats for the current frame results in a BLU intensity of 50%, then the global dimming gain for scaling up RGB values of next frame will be 2.0. The RGB values of the next frame will be doubled (e.g., multiplied by 2), but the BLU intensity for the backlight zones is reduced by 50%. When scaling, some of the RGB values may be clipped at a certain value (e.g., 255). The amount of clipping may depend upon whether the aim is power saving or better visual quality, and the global dimming gain may be calculated accordingly.

In particular embodiments, the GD/LD algorithm may calculate a new BLU intensity for a next or subsequent frame based on max RGB or gray values associated with BLU stats of the current frame. The GD/LD algorithm may compute the BLU stats for each backlight zone to represent a grayscale or brightness level of a portion of the image within that backlight zone. In some embodiments, computing the BLU stats may include estimating a max RGB value for each backlight zone. An accumulated histogram is generated using the maximum gray values computed from the BLU stats. For a given percentile in the accumulated histogram, a threshold gray level value (e.g., 240) is taken. Once the threshold grayscale value is determined, it may be converted into a new BLU intensity for the next or subsequent frame.

In particular embodiments, the threshold may be converted into the new BLU intensity using a lookup table (LUT). For instance, the LUT may contain different BLU intensities corresponding to different threshold grayscale values. The GD/LD algorithm may use an appropriate LUT depending on whether its aim is power saving or better visual quality. For instance, a first LUT (e.g., up-concave LUT) is directed towards better visual quality as it results in a higher BLU intensity for a given threshold, which means a low global dimming gain, and therefore low chances of clipping (e.g., values being clipped at highest RGB value 255) and occurrence of artifacts due to the clipping. Whereas a second LUT (e.g., linear LUT) may be directed towards power savings as it results in a lower BLU intensity for a given threshold, which means a higher global dimming gain to scale up the RGB.

Once a BLU intensity is calculated based on BLU stats of a current frame, the GD/LD algorithm may calculate a global dimming gain for scaling RGB values and adjusting backlight intensity of subsequent frame (e.g., frame n+1). The computed global dimming gain may be passed to the next or subsequent frame for processing. In this way, global dimming may be performed for a series of frames and iterations, where, at each iteration, RGB values and backlight intensity are globally adjusted for a current frame based on previous frame's data (e.g., global dimming gain computed based on BLU stats of previous frame). Only the first frame in the series of frames is displayed according to its original RGB value. In addition to the global dimming, local dimming may also be performed at each iteration.

In some embodiments, a dramatic change in a global dimming gain (equivalently BLU intensity) from a previous global dimming gain may cause flickering. For example, if frames change dramatically from dark to bright, then it may cause flickering. To prevent it from happening, a slow adaptation technique may be implemented using a queue or other techniques. Slow adaptation technique may be implemented based on the global dimming gains computed from previous frames and stored in a queue. The actual global dimming gain for the current frame under processing may be determined by taking an average value of all previous global dimming gains in the queue. With the slow adaptation technique discussed herein, flickering may be minimized. Also, depending on the size of the queue, flickering or amount of power save may be determined.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Local dimming (LD) is a technology for liquid crystal displays (LCD) to increase contrast ratio and improve visual quality. It controls and modulates the LC transmittance with backlight brightness to properly present content with high contrast. The LD technology ensures that the lights (e.g., backlight LEDs) with which the content is displayed are dimmed or completely turned off at specific moments so that a dark scene looks correct and black colors appear true black instead of gray. U.S. patent application Ser. No. 17/713,965, filed 5 Apr. 2022, which is hereby incorporated by reference in its entirety, describes an improved local dimming pipeline (e.g., local dimming pipeline300shown inFIG.3), technique, or algorithm for performing local dimming, especially for artificial-reality systems, such as AR/VR headsets. For instance, the LD algorithm (e.g., LD algorithm321) computes backlight unit statistics (BLU stats) or backlight matrix without needing a specialized or dedicated hardware (e.g., a dedicated ASIC) as was required in a conventional or traditional pipeline for local dimming. Also, the LD algorithm eliminates the feedback loop (e.g., sending the computed backlight matrix back to a graphics rendering source for RGB adjustments) of the conventional pipeline. Due to the elimination of the feedback loop, there is considerably low or no latency when displaying content, especially through AR/VR headsets. Furthermore, since the LD algorithm eliminates RGB adjustments and need for additional hardware (e.g., dedicated ASIC) for the backlight computation, overall computational and/or power requirements are considerably reduced.

While the improved local dimming pipeline, technique, or algorithm (discussed in reference toFIG.3) achieves improved visual contrast with low power and latency requirements for low light regions or low gray levels (e.g., black regions in a display), however, it loses the capability to modulate backlight with RGB to achieve power savings for mid-gray contents (e.g., gray regions or mid-color regions in the display). Stated differently, because the improved LD algorithm eliminates the feedback loop from a LD engine to a graphics engine for RGB adjustments, it loses the capability to adjust RGB to achieve power savings for mid-gray contents. For example, if a zone is a uniform 50% gray, because RGB values are not adjusted in the local dimming pipeline, 100% backlight may need to be used to achieve target visual performance, as shown inFIG.1A. This may not be power efficient especially for low-powered or resource-constrained devices, such as AR/VR devices. Power savings could be achieved by scaling the RGB to full white with 50% backlight on, as shown for example inFIG.1A. This is what global dimming does. That is, global dimming is a way to do both the RGB as well as backlight adjustments for power savings for mid-gray contents.

Global dimming (GD) (also interchangeably herein referred to as content adaptive backlight control) is a technique used in LCD to reduce the power consumption of the backlight unit. When frame content to be displayed is not fully white, GD scales up the RGB values (e.g., LC transmittance) and scales down the backlight intensity to achieve the same final visual image at lower power cost, as shown for example inFIGS.1A and1B. More specifically, GD typically scales up RGB values and dim down the BLU intensity at the same time, so it may preserve visual quality while saving power consumption.FIG.1Aillustrates an example of global dimming. An example comparison is shown to illustrate the difference between a scenario100when no global dimming is used or applied and a scenario120when global dimming is applied. As depicted, in the scenario100without global dimming, an image102is a uniform gray image with RGB values of 127 (e.g., R127, G127, B127). In order to output this image102, backlight unit (BLU) intensity of 100% (indicated by reference numeral104) is used to generate a display106. Whereas in the scenario120with global dimming, RGB values of the input image102are scaled up to full white (e.g., R255, G255, B255), as indicated by reference numeral122, and BLU intensity is reduced down to 50%, as indicated by reference numeral124, to generate the same display106.

FIG.1Billustrates another example of global dimming. Similar toFIG.1A, an example comparison is shown to illustrate the difference between a scenario150when no global dimming is applied and a scenario160when global dimming is applied. As depicted, in the scenario150without global dimming, BLU intensity of 100% (indicated by reference numeral152) for an image154(without RGB adjustment) is used in order to generate a display156. Whereas in the scenario160with global dimming, BLU intensity is reduced to 70% (indicated by reference numeral162) and RGB values of the image154are scaled up (indicated by reference numeral164) to generate the same display156.

To further describe the concept of global dimming,FIG.2illustrates example charts210and220respectively showing RGB adjustment and BLU intensity adjustment for an image200. The most important point in global dimming is how to balance between scaling up RGB values and dimming down the BLU intensity. For example, the image200mostly consists of mid or low gray values. This is represented by red histogram212in the chart210. To generate a display based on the RGB values represented by the red histogram212, full BLU intensity of 100% may be used, as indicated by reference numeral222. To achieve power savings, global dimming technique discussed herein (with respect toFIG.4), may aggressively scale up RGB values, as shown by blue histogram214. In return, it may reduce the BLU intensity considerably, as indicated by reference numeral224.

Particular embodiments discussed herein describes an efficient method, algorithm, and technique to perform global dimming in addition to local dimming. The technique for global dimming discussed herein is particularly suitable for artificial reality devices, such as AR/VR devices. Furthermore, the technique discussed herein is an extension to the local dimming pipeline, such as local dimming pipeline300shown inFIG.3. Before describing how the method or technique for global dimming is applied to the local dimming pipeline, it is important to first understand the local dimming pipeline and various components that are associated with this pipeline.

FIG.3illustrates an example local dimming pipeline300. The pipeline300illustrates various operations performed by computing components of a display system (e.g., display system1000) at various stages for local dimming. In particular embodiments, the pipeline300illustrates operations performed by a graphical processing unit (GPU)302and a central processing unit (CPU)320components of the display system, such as the artificial reality system1000shown inFIG.10.

A graphics rendering engine303of the GPU302of the display system may receive a RGB frame304from a particular source, such as a front-facing camera of the VR headset. Example frames that may be displayed through the VR headset are shown inFIG.7. In some embodiments, the graphics rendering engine discussed herein may be an asynchronous time warp (ATW) compositor that receives images (e.g., RGB frames) that are to be displayed to a user. The ATW compositor may make certain adjustments (e.g., move things around, distortion correction, aberration correction, etc.) to the images before displaying. The ATW compositor may ensure that the images to be displayed are at right places and compensate for any dropped frames.

The graphics rendering engine303(e.g., ATW compositor) may store the RGB frame304in a front buffer306. In particular embodiments, the front buffer306may be a dedicated memory or storage allocated to the GPU302for storing graphical content (e.g., images, videos, audio, frames, etc.). In order for the GPU302to share some portion of the graphical content with the CPU320for it to process, the GPU302may store the portion in a shared or mapped buffer308. For instance, the GPU302may transfer the RGB frame304from the front buffer306to the mapped buffer308so that the CPU320may be able to process the RGB frame304to compute the backlight unit statistics (BLU stats) discussed herein. In some embodiments, the mapped buffer308may store a reduced resolution or downscaled version of the RGB frame304. Stated differently, the mapped buffer308discussed herein may be a small resolution buffer that stores smaller/reduced resolution version of a RGB image for computing the BLU stats. Using the mapped buffer308is advantageous as it is computationally very efficient and cheap to compute the BLU stats by the CPU320. For instance, the CPU320is required to perform computation for fewer pixels of a reduced RGB frame.

In particular embodiments, a local dimming (LD) algorithm321running on the CPU320may receive the RGB frame304from the shared or mapped buffer308and perform a series of steps322,324,326, and328in sequence to compute a backlight matrix330. In some embodiments, the first step322may be optional as the RGB frame304received from the mapped buffer308may already be downscaled. If it is not downscaled or further downscaling is needed, then in the first step322, the LD algorithm321may generate a mipmap of the RGB frame304. In some embodiments, generating the mipmap may include downscaling the resolution of the RGB frame304to a reduced resolution for efficient processing. For instance, the input resolution, especially in VR, is significantly high and since there is no dedicated/specialized hardware (e.g., dedicated ASIC) for computing the backlighting like the traditional displays, the input resolution needs to be downscaled to a smaller resolution so that the RGB frame304may be processed with the CPU320. As an example and not by way of limitation, the LD algorithm321may use 10× downscaling or 8× (e.g., 3 level) mip downsize to downscale the RGB frame304. In some embodiments, the value of downscaling may depend upon RGB resolution and number of backlight zones.

In the second step324, the LD algorithm321may compute a zone statistic (also interchangeably referred to herein as BLU stats) for each backlight zone to represent a grayscale or brightness level of a portion of the image within that backlight zone. In other words, the LD algorithm321may compute how bright a particular backlight zone is based on the RGB values of pixels within that zone. Each backlight zone may encompass a subset of pixels of the RGB frame304. As an example and not by way of limitation, a backlight zone may encompass 100 pixels out of 1000 pixels that make up the RGB frame304. In particular embodiments, the LD algorithm321may compute the BLU stats for a backlight zone by first averaging values of each of the RGB channels or color components across the pixels within the backlight zone and then taking a max value of the averaged RGB. Continuing the same 100 pixels example within the backlight zone, the LD algorithm321may compute an average R value by taking an average of the red value across the 100 pixels, an average G value by taking an average of the green value across the 100 pixels, and an average B value by taking an average of the blue value across the 100 pixels. In this example, let's assume the average R value comes out to 10, the average G value comes out to 15, and the average B value comes out to 75, then the LD algorithm takes the max value i.e., 75 as the statistic or BLU stat for the particular backlight zone. Similarly, the LD algorithm321may estimate the BLU stats for other backlight zones. In particular embodiments, the LD algorithm321estimates these BLU stats for the backlight zones so that a particular color may be displayed with its intended brightness and only those backlight zones whose BLU stats are equal to or lower than a certain threshold may be dimmed, as discussed in further detail below in step326. For example, if there is a bright blue in a scene (e.g., represented by the RGB frame304), then that bright blue may be displayed as is without any adjustment.

In the third step326, the LD algorithm321may perform level mapping, which including mapping the zone statistics computed in step324, for each backlight zone, to a brightness and/or dimming value using a particular technique. The brightness and/or dimming value for a backlight zone may correspond to a BLU intensity with which to power a particular LED in that zone. For example, if the brightness values are given between 0-100 and a brightness value for a zone is given 50, then BLU intensity of 50% is used to lighten up that zone. In particular embodiments, the LD algorithm321may use a custom-built lookup table (LUT) to map the statistics or BLU stats to a brightness value. For instance, the lookup table may include predetermined brightness and/or dimming values corresponding to different BLU stats (e.g., max values) and the LD algorithm321may use this data to map each of the statistic or BLU stat associated with a backlight zone to a corresponding brightness/dimming value indicated in the lookup table. In some embodiments, the LD algorithm321may also perform the level mapping326using a machine learning (ML) technique. For instance, a trained ML model may output brightness and/or dimming values corresponding to specific statistics/BLU stats. The ML model may be trained based on ground-truth statistics and their corresponding ground-truth brightness and/or dimming values.

In some embodiments, during the level mapping operation especially in the case of local dimming, the LD algorithm321may use a gamma curve for low brightness zones (e.g., zones with low/dark gray levels) and saturate at certain threshold (e.g., max value less than or equal to 20). Stated differently, the LD algorithm321may find backlight zones whose max value is less than the certain threshold and only dim those zones to a particular brightness value using the lookup table. For instance, the backlight zones corresponding to dark or low gray levels in the RGB frame304may be dimmed by their corresponding brightness/dimming values in the lookup table, whereas medium and high gray levels may be left untouched or unadjusted. Since the LD algorithm321does not adjust the backlight of medium and/or high grayscales, the RGB does not need to be adjusted in the case of the local dimming pipeline300. However, as stated earlier, since the RGB are not adjusted, the local dimming pipeline300may not be able to achieve power savings for mid-gray contents. For example, if a zone is a uniform 50% gray (e.g., RGB values of 127), since RGB is not adjusted, 100% BLU intensity may be used to lit the zone and this may not be power efficient for low-powered or resource-constrained devices, such as AR/VR devices. To achieve power savings for mid-gray content, global dimming (e.g., adjusting both the RGB as well as backlight intensity) is needed. A modified pipeline400for global dimming or global dimming pipeline400is shown and discussed in reference to at leastFIG.4.

In particular embodiment, the level mapping step326may output an initial backlight matrix, which may include an array of brightness and/or dimming values corresponding to a plurality of backlight zones. Simply displaying based on the backlight matrix produced after the level mapping326may be prone to some artifacts. These artifacts may include, for example and without limitation, backlight flickering artifacts due to head motion or moving objects in VR, over-dimming artifacts, and halo effects (e.g., light falling into darker areas surrounding an object, such as area around a bright moon that should actually be dark).

In the fourth step328, the LD algorithm321may perform post filtering on the initial backlight matrix produced after the level mapping step326in order to correct one or more artifacts discussed above. In particular embodiments, the post filtering328may include the LD algorithm321spatially as well as temporally filtering or smoothening the brightness values in the initial backlight matrix to minimize the one or more artifacts. As an example and not by way of limitation, the LD algorithm321may perform the spatial filtering by using a spatial dilation and/or gaussian blur to minimize halo and over-dimming artifacts in the initial backlight matrix. The LD algorithm321may perform the temporal filtering by using recursive temporal averaging to remove backlight flickering artifacts occurring due to head motion or moving objects, especially in VR.

Responsive to performing the last step i.e., post filtering328(e.g., spatial-temporal filtering), the LD algorithm321generates a final backlight matrix330. The backlight matrix330may indicate by what amount (e.g., percentage) each backlight zone should be lit or dimmed by a BLU driver332. Based on the backlight matrix330, the BLU driver332may output a backlight intensity336(e.g., brightness levels) for one or more LEDs of a display component. In some embodiments, outputting the backlight intensity336may include the BLU driver332turning on/off a subset of LEDs in the display component. In the case of local dimming, the backlight intensity336may include the BLU driver332dimming a brightness level of the subset of LEDs corresponding to low/dark gray level regions of the RGB frame304. One or more backlight unit ICs (BLU-IC)340may be configured to control the backlight based on the backlight intensity336output by the BLU driver332. In particular embodiments, a BLU-IC340may receive the backlight intensity information from the BLU driver332via a communication interface/bus, such as a serial peripheral interface (SPI).

In some embodiments, in addition to computing the backlight matrix330, a display processing unit (DPU)334may receive the RGB frame304from the front buffer306and may perform one or more post-processing steps on the RGB frame304to remove one or more artifacts or further improve image quality. It should be noted that the post-processing steps performed by the DPU334are not related to the local dimming discussed herein and these steps may only be performed to correct some artifacts resulting from certain components (e.g., optics, lens) of the display system. As an example and without limitation, an image may be distorted due to lens distortions or chromatic aberrations resulting from light passing through optics (e.g., lens) of the VR headset and therefore, the DPU334may perform one or more post-processing steps on the RGB frame304to correct these lens distortions or effects of chromatic aberrations.

The DPU334may output a display RGB338. Using the backlight intensity336output by the BLU driver332and the display RGB338output by the DPU334, a display350may be generated. For generating the display350, a set of integrated circuits (ICs) may be configured to generate the display350(e.g., output an image) based on the RGB frame output by the DPU334and the backlight intensity output by the BLU driver332. For instance, one or more display driver ICs (DDIC)342may be configured to control display panel(s) and produce a rich and vibrant display based on the display RGB338output by the DPU334. In particular embodiments, a DDIC342may receive the RGB information338from the DPU334via a communication layer or interface, such as C-PHY. One or more backlight unit ICs (BLU-IC)340may be configured to control the backlight based on the backlight intensity336output by the BLU driver332. In particular embodiments, a BLU-IC340may receive the backlight intensity information from the BLU driver332via a communication interface/bus, such as a serial peripheral interface (SPI). In some embodiments, the resulting display350or image (i.e., RGB+backlight) may be presented on a screen of the display system. As an example and not by way of limitation, the resulting image may be presented on a VR headset's display.

The local dimming pipeline300discussed above may be extended to perform global dimming to achieve additional power savings, especially for mid grayscale or mid gray level content. In particular embodiments, one or more components may be added and/or adjusted in the existing local dimming pipeline300to perform global dimming. One such modified pipeline capable of performing global dimming in addition to local dimming is shown and discussed with respect toFIG.4. At a high level, to perform global dimming, data associated with BLU stats of a previous frame is used to adjust RGB frame and BLU intensity for current frame. By way of an example and without limitation, if there are three frames T1, T2, and T3, where T1is the first frame, T2is the second frame, and T3is the third frame in a sequence of frames to be displayed, then data associated with BLU stats of the T1frame is used to adjust the RGB values and backlight intensity for displaying the T2frame, and data associated with BLU stats of the T2frame is used to adjust the RGB values and backlight intensity for displaying the T3frame. Here, the first frame T1may be displayed at its original RGB value. However, the backlight intensity may still be adjusted, particularly in the low-level grayscale regions for local dimming, as discussed above for example in reference toFIG.3. Detailed description with respect to how global dimming is performed is now discussed in reference toFIG.4.

FIG.4illustrates a modified pipeline400for global dimming or a global dimming pipeline400. Although, the global dimming pipeline400here is described with respect to two image frames402a(frame n−1) and402b(frame n), it should be understood that the pipeline400is not by any way limited to these frames and any number of image frames are possible and within the scope of the present disclosure. Also, it should be noted that some of the reference numerals have been kept same throughout this disclosure for consistency, ease of understanding, and to refer to the same entities, components, and/or elements that have already been discussed and therefore, may not be repeated in its entirety again. However, the entities, components, and/or elements represented by these same reference numerals are not by any way limited to the scope or functionality discussed earlier and may incorporate additional scope or functionality.

As depicted, the graphics rendering engine303(e.g., ATW compositor) may receive a first RGB frame402a(e.g., frame n−1, where n=1). The frame402amay come from a particular source, such as a front-facing camera of a VR headset. As discussed elsewhere herein, the graphics rendering engine303may be an ATW compositor that receives images (e.g., RGB frames402a,402b, . . . ,402n) that are to be displayed to a user. The ATW compositor may make certain adjustments (e.g., move things around, distortion correction, aberration correction, etc.) to the images before displaying. The ATW compositor may ensure that the images to be displayed are at right places and compensate for any dropped frames.

Since there is no prior data available for the first RGB frame402a, the first RGB frame402amay be stored as is and without any adjustments in the front buffer306. The DPU334may receive the RGB frame402afrom the front buffer306and may perform one or more post-processing steps discussed earlier on the RGB frame402ato remove one or more artifacts or further improve image quality. The DPU334may output a display RGB422a, which may be used along with backlight intensity420ato generate a display430a, as discussed in further detail below.

In order for the GPU302to share some portion of the graphical content with the CPU320for it to process, the GPU302may store the portion in a shared or mapped buffer308. For instance, the GPU302may transfer the RGB frame402afrom the front buffer306to a reduced resolution mapped buffer308so that the CPU320may be able to efficiently process the RGB frame402ato compute BLU stats. Prior to storing the RGB frame402ain the mapped buffer308, the RGB frame402amay be downscaled (e.g., Blit) for efficient processing by the CPU320. For instance, the input resolution of the RGB frame402amay be downscaled (e.g., 10× downscaling or 8×) to a smaller resolution so that the RGB frame402amay be processed by the CPU320in a way that it reduces computational costs. In some embodiments, the value of downscaling may depend upon RGB resolution and number of backlight zones. Also, to reduce the computational costs, the RGB frame402amay be segmented into separated image block (e.g.,504blocks) for processing by the CPU320.

In particular embodiments, a global dimming (GD) and local dimming (LD) algorithm410(also interchangeably herein referred to as GD/LD algorithm410) running on the CPU320may receive the reduced resolution RGB frame402a′ from the mapped buffer308. The GD/LD algorithm410may perform a series of steps412a,414a,416a, and418ato compute a backlight intensity for backlight zones (e.g., LEDs) and a global dimming gain or scale factor for next frame402b(e.g., frame n). In particular embodiments, the GD/LD algorithm410may be configured to compute brightness level and maximum gray level of each image block for local dimming and global dimming in parallel for a better computation efficiency. Also, this computation may be carried out at multiple image blocks concurrently. It should be noted that although a downscaling step (e.g., downscale step322ofFIG.3) is not shown in the global dimming pipeline400, the downscaling step may also be included and performed prior to backlight stats estimation step412a. For instance, if the RGB frame402a′ is not sufficiently downscaled or further downscaling is needed, then the GD/LD algorithm410may perform downscaling similar to the downscaling discussed in the downscale step322prior to the backlight stats estimation412a.

In the backlight stats estimation step412a, the GD/LD algorithm410may compute BLU stats for each backlight zone to represent a grayscale or brightness level of a portion of the image within that backlight zone. As discussed with respect to step324inFIG.3, backlight stats estimation412amay include estimating a max RGB value for each backlight zone, such as max RGB values shown in matrix325ofFIG.3. In some embodiments, the BLU stats may be calculated based on original RGB values associated with a RGB frame. More specifically, the BLU stats are calculated based on original RGB values of the frame that was received by the graphics rendering engine303and not based on adjusted RGB values (e.g., scaled up RGB values due to global dimming). Since there are no RGB adjustments made in the first frame402a, the BLU stats are estimated directly based on corresponding RGB values associated with the reduced resolution RGB frame402a′. However, if a RGB frame is adjusted (e.g., RGB values of frame are scaled up) as discussed in subsequent frames (e.g., frame n, n+1, etc.), then the RGB frame needs to be re-adjusted or re-scaled back to its original RGB values prior to estimating the BLU stats. An adjusted RGB frame (e.g., RGB values scaled up) may be re-adjusted back to its original RGB values based on global dimming gain or scale factor. This is further discussed in detail below with respect to backlight stats estimation step412bperformed for next frame402b(e.g., frame n).

Responsive to estimating the BLU stats, the GD/LD algorithm410may perform global dimming gain calculation414aand level mapping416a. In some embodiments, steps414aand416amay be performed in parallel. In other embodiments, steps414aand416amay be performed in a sequential manner (i.e., one after the other). In the level mapping step416a, the GD/LD algorithm410may map the zone statistics (e.g., BLU stats) computed in step412a, for each backlight zone, to a brightness and/or dimming value using a particular technique. The brightness and/or dimming value for a backlight zone may correspond to a BLU intensity with which to power a particular LED in that zone. For example, if the grayscale values are given between 0-100 and a grayscale value for a zone is given 50, then BLU intensity of 60% is used to lighten up that zone. In particular embodiments, the BLU intensity with which the current frame402ais displayed may depend upon global dimming gain and corresponding BLU intensity calculated based on previous frame's BLU stats. Since there is no previous frame before the current frame402a, the GD/LD algorithm410may calculate the BLU intensity for the current frame402ausing a custom-built lookup table (LUT). For instance, the GD/LD algorithm410may use the LUT to map the statistics or BLU stats to a brightness value or backlight intensity. For instance, the lookup table may include predetermined brightness and/or dimming values corresponding to different BLU stats (e.g., max values) and the GD/LD algorithm410may use this data to map each of the statistic or BLU stat associated with a backlight zone to a corresponding brightness/dimming value indicated in the lookup table.

In some embodiments, to perform local dimming in addition to global dimming discussed herein, the GD/LD algorithm410may use a gamma curve for low brightness zones (e.g., zones with low/dark gray levels) and saturate at certain threshold (e.g., max value less than or equal to 20). Stated differently, the GD/LD algorithm410may find backlight zones whose max grayscale value (e.g., max RGB value) in the BLU stats is less than the certain threshold and only dim those zones to a particular brightness value using the lookup table. For instance, the backlight zones corresponding to dark or low gray levels may be dimmed by their corresponding brightness/dimming values in the lookup table, whereas medium and high gray levels may be left untouched.

In particular embodiment, the level mapping step416amay output an initial backlight matrix, which may include an array of brightness and/or dimming values corresponding to a plurality of backlight zones. The initial backlight matrix may be representative of a BLU intensity with which to display the current frame402a. Simply displaying based on the initial backlight matrix produced after the level mapping416amay be prone to some artifacts (e.g., backlight flickering artifacts, over-dimming artifacts, halo effects, etc.).

In the post filtering step418a, the GD/LD algorithm410may perform post filtering on the initial backlight matrix produced after the level mapping step416ain order to correct one or more artifacts, as discussed elsewhere herein. In particular embodiments, the post filtering418amay include the GD/LD algorithm410spatially as well as temporally filtering or smoothening the brightness values in the initial backlight matrix to minimize the one or more artifacts. As an example and not by way of limitation, the GD/LD algorithm410may perform the spatial filtering by using a spatial dilation and/or gaussian blur to minimize halo and over-dimming artifacts in the initial backlight matrix. The GD/LD algorithm410may perform the temporal filtering by using recursive temporal averaging to remove backlight flickering artifacts occurring due to head motion or moving objects, especially in VR.

Responsive to performing the post filtering418a(e.g., spatial-temporal filtering), the GD/LD algorithm410generates a final backlight matrix. Based on the final backlight matrix, a BLU driver (e.g., BLU driver332) may output a backlight intensity420a(e.g., brightness levels) that is used along with the display RGB422ato generate the display430a. The resulting display430a(i.e., RGB+backlight) may be presented on a screen of the display system, such as display system1000.

In the global dimming gain calculation414a, the GD/LD algorithm410may calculate a new BLU intensity for a subsequent/next frame (e.g., frame402bor frame n) and a corresponding global dimming scale or gain factor to scale RGB values of the subsequent frame based on the BLU intensity. In some embodiments, computations for global dimming discussed herein may be done in a time frame between two consecutive frames. For example, the computations for global dimming (e.g., scaling up RGB values) may be done between frame n−1 and frame n or frame n and frame n+1. In particular embodiments, the GD/LD algorithm410may calculate the global dimming scale or gain factor (GainGD) for the subsequent frame based on following equation:

As depicted in the above equation, the global dimming gain or scale factor to scale up RGB values is inversely proportional to the new BLU intensity. That is, the global dimming gain value increases with decreasing BLU intensity. By way of an example, if the image statistics (e.g., BLU stats) for the current frame402aresults in a new BLU intensity of 50%, then the global dimming gain for scaling up RGB values of next frame402bwill be 2.0. The RGB values of the next frame402bwill be multiplied by 2 but the BLU intensity for the backlight zones is reduced by 50%. It should be noted that the new backlight intensity calculated for the next frame may be different from backlight intensity420athat is used for displaying the current frame (e.g., frame402a). For example, the backlight intensity computed for the next frame (e.g., frame402b) may be 50%, whereas the backlight intensity used for the current frame (e.g., frame402a) may be 70%.

In particular embodiments, the GD/LD algorithm410may calculate the new backlight intensity for the next or subsequent frame (e.g., frame402bor frame n) based on max RGB or gray values computed in the BLU stats estimation step412afor the current frame (e.g., frame402aor frame n−1). An accumulated histogram is generated using the maximum gray values computed from the BLU stats estimated step412a. One such histogram212is shown for example inFIG.2. For a given percentile in the accumulated histogram, a threshold gray level value (e.g., 240) is taken. Stated differently, the threshold could be chosen using the accumulated histogram by setting a target percentile. For example, if the target percentile is set to 90%, then gray level corresponding to the 90thpercentile of the accumulated histogram is taken as the threshold. In particular embodiments, the target percentile for choosing the threshold may be determined in a heuristic way or what global dimming really aims at. For instance, the target percentile may be determined based on whether the goal is power saving or better visual quality. A particular lookup table (LUT) may be used for achieving the specific goal. For example, if the goal is better visual quality, then an up-concave LUT that is more conservative than a linear LUT may be used to determine the target percentile for choosing the threshold. In some embodiments, a max value of all RGB values in the BLU stats may be used to choose the threshold. This may avoid clipping (e.g., RGB values saturating at a certain value such as 255), but this also may limit the power saving amount.

Once the threshold grayscale value is determined, it may be converted into a new BLU intensity for the next or subsequent frame. In some embodiments, the threshold may be converted into the new BLU intensity using a LUT. For instance, the LUT may contain different BLU intensities corresponding to different threshold grayscale values. One such example correspondence between the different BLU intensities and grayscale values is shown inFIG.5. By way of an example and without limitation, the LUT may contain a BLU intensity of 30% for threshold grayscale value of 75, a BLU intensity of 50% for threshold grayscale value of 127, a BLU intensity of 75% for threshold gray value of 200, a BLU intensity of 100% for threshold grayscale value of 255, etc. The GD/LD algorithm410may use the LUT for determining a BLU intensity corresponding to the threshold it has calculated earlier. In some embodiments, different LUTs may contain varying or different BLU intensities for a same threshold grayscale value and the GD/LD algorithm410may use an appropriate LUT depending on whether its aim is power saving or better visual quality. For instance, a first LUT (e.g., up-concave LUT) is directed towards better visual quality as it results in a higher BLU intensity for a given threshold, which means a low global dimming gain, and therefore low chances of clipping (e.g., values being clipped at highest RGB value 255) and occurrence of artifacts due to the clipping. Whereas a second LUT (e.g., linear LUT) may be directed towards power savings as it results in a lower BLU intensity for a given threshold, which means a higher global dimming gain to scale up the RGB.

FIG.5illustrates an example graph500depicting an example correspondence between BLU intensities and gray levels. The example correspondence depicted here may be according to a particular lookup table. The BLU intensity varies from 0-100, where 0 is the least/no BLU intensity and 100 is the highest BLU intensity. Here, the BLU intensities are depicted in the Y-axis of the graph500. The gray levels (0-255) have been normalized to range between 0-100, where 0 may represent a black image, 50 may represent a gray image, and 100 may represent a fully white image. The normalized gray levels are depicted in the X-axis of the graph500. As depicted, the BLU intensity increases linearly with increasing normalized gray scale level. This may be the case with one of the LUTs. However, it should be noted that the BLU intensity may be different or behave differently for the same gray level according to a different LUT, as discussed above. As depicted, for a normalized gray level of 35, the BLU intensity that may be used for a frame is 55%, as indicated by reference numeral502. Also, for the BLU intensity 55%, clipping level is set to 10%, which basically means that 10% of RGB values in an image frame may be clipped at a certain value (e.g., max RGB value of 255) when the RGB values are scaled up according to the global dimming gain that is calculated based on the 55% BLU intensity using equation (1).

Generally, the amount of clipping depends on the target percentile that is chosen for the threshold grayscale. Also, the amount of clipping may generally increase with decreasing BLU intensity. More specifically, if the BLU intensity is low, then using equation (1), the global dimming gain by which the RGB values are scaled up will be high and therefore, there are high chances of more RGB values being clipped. As an example, if the BLU intensity is 50%, then using equation (1) above, the global dimming gain will be 2, which is used to scale up the RGB values of a frame by a factor of 2 (i.e., multiplying RGB values by 2). If the original RGB values are already in higher range (e.g., 100-200), then a lot of values when scaled up will be clipped at a certain value (e.g., 255). Although, the reduced backlight intensity and corresponding large amount of clipping may achieve power savings, but the large amount of clipping may also lead to visual artifacts. Whereas, if the backlight intensity is high (e.g., 90%), then the RGB values are only scaled up by a small amount (e.g., scaled up by 1.11 based on 90% BLU intensity). This may prevent clipping and achieve better visual quality, but on the downside may not be very power efficient as 90% backlight intensity is used to generate a display. Therefore, the new BLU intensity and corresponding global dimming gain may be computed based on whether the aim is power saving or better visual quality, and target percentile and LUT for the computation may be chosen accordingly.

FIG.6illustrates example images602,604,606, and608depicting global dimming with different clipping levels. Specifically, image602is an original image i.e., image with original RGB values and without any readjustments. The image602includes an area603with a group of high gray levels. These high gray levels may be clipped or converged to a certain value (e.g., 255) depending on a certain threshold. Image604is a readjusted image with 5% clipping and power saving of 15%. Image606is a readjusted image with 15% clipping and power saving of 24%. Image608is a readjusted image with 30% clipping and power saving of 36%. As can be observed from these images604,606, and608, power savings increases with the amount of clipping. However, it should be noted that while increased power savings may be achieved with increased clipping, visual quality may suffer due to the increased clipping. As such, the amount of clipping may depend on whether one wants more power save or better visual quality.

Referring back to global dimming gain calculation step414a, once a BLU intensity for the next frame is calculated as discussed above, a global dimming gain440amay be accordingly calculated using the equation (1) above. The global dimming gain440amay then be passed to the next or subsequent frame402b. As depicted inFIG.4, the global dimming gain440amay be passed to the GPU302for RGB adjustment as well as to the CPU320for backlight intensity adjustment. Using the global dimming gain440a, global dimming may be performed for the next frame i.e., frame n or RGB frame402b. The global dimming is now discussed below with respect to the frame402b.

As depicted, the graphics rendering engine303(e.g., ATW compositor) may receive a second RGB frame402b. The second RGB frame402bmay come after the first RGB frame402a. The graphics rendering engine303may also receive the global dimming gain440a, which has been calculated based on BLU stats of the previous frame402a. Using the global dimming gain440a, the graphics rendering engine303may adjust the RGB values of the second frame402bto generate an adjusted RGB frame404b. In particular embodiments, adjusting the RGB values may include scaling up the RGB values of the second frame402b. By way of an example and without limitation, if the second RGB frame402bis a uniform gray image (e.g., RGB average value of 127) and the global dimming gain is 2, then all the RGB values of the frame402bare globally scaled up by multiplying the RGB values by 2 (e.g., RGB values scaled up to 254). Stated differently, if the global dimming scalar or gain is 2, then all the RGB values are doubled in the linear RGB domain. It should be noted that the global dimming gain for the first frame (e.g.,402aor frame n−1) is 1, meaning that RGB values of the first frame do not change. Also, it should be noted that ATW compositor and scaling up RGB are computationally expensive. As such, they are incorporated into a single process.

As discussed earlier, some of the RGB values may be clipped at a certain value when scaled up. The amount of clipping may depend on whether one wants more power save or better visual quality. Different LUTs may be utilized depending on how much clipping (e.g., power save) is desired. For instance, a up-concave LUT is more conservative and may prevent clipping than a linear LUT because for a given normalized gray level, the up-concave LUT may result in a higher BLU intensity (e.g., meaning a low global dimming gain).

The graphics rendering engine303may store adjusted GB frame404b(e.g., frame with scaled up RGB values) in the front buffer306. The DPU334may receive the adjusted RGB frame404bfrom the front buffer306and may perform one or more post-processing steps, as discussed elsewhere herein. The DPU334may output an adjusted display RGB422b, which may be used along with adjusted backlight intensity420bto generate a display430b.

To compute the adjusted backlight intensity420band a new backlight intensity for subsequent frame (e.g., frame n+1), the GPU302may transfer the adjusted RGB frame404afrom the front buffer306to the mapped buffer308so that the CPU320may be able to efficiently process the adjusted RGB frame404ato compute BLU stats for current frame. As discussed elsewhere herein, prior to storing the adjusted RGB frame404bin the mapped buffer308, the adjusted RGB frame404bmay be downscaled for efficient processing by the CPU320.

In particular embodiments, the GD/LD algorithm410running on the CPU320may receive a reduced resolution version of the adjusted RGB frame404bfrom the mapped buffer308. The CPU320may also receive the global dimming gain440a(computed based on previous frame402a), as indicated by reference numeral450. In some embodiments, the CPU320and/or the GD/LD algorithm410may use the global dimming gain440afor temporal filtering to minimize certain artifacts (e.g., flickering artifacts). For example, the current frame (e.g., frame n) may be considerably different from previous frame (e.g., frame n−1) and this would cause flickering. Also, since the current frame is downscaled from its original resolution to a smaller resolution, some information may be missing that would again cause some flickering. Therefore, to reduce these flickering artifacts, the GD/LD algorithm410may perform temporal filtering based on the global dimming gain440acalculated based on previous frame stats or data.

In the backlight stats estimation step412b, the GD/LD algorithm410may compute BLU stats for each backlight zone to represent a grayscale or brightness level of a portion of the image within that backlight zone. As discussed elsewhere herein, the backlight stats estimation412bmay include estimating a max RGB value for each backlight zone. The BLU stats may be calculated based on original RGB values associated with the RGB frame402b. More specifically, the BLU stats are calculated based on original RGB values of the frame402bthat was received by the graphics rendering engine303and not based on adjusted RGB values associated with the adjusted RGB frame404breceived from the mapped buffer308. To obtain the original RGB values, the GD/LD algorithm410upon receiving the adjusted RGB frame404bmay re-scale the RGB values associated with the adjusted RGB frame404bby dividing the RGB values of the adjusted RGB frame404bby the global dimming gain440a. As an example, if the global dimming gain is 2, then the RGB values are divided by 2 to obtain the original RGB values for the adjusted RGB frame404b.

Responsive to estimating the BLU stats, the GD/LD algorithm410may perform global dimming gain calculation414band level mapping416b, as discussed elsewhere herein. In particular embodiments, the global dimming gain440amay be incorporated into the level mapping416b. From the global dimming gain440a, the BLU intensity may be determined using equation (1) discussed above. The actual BLU intensity for the current frame (e.g., frame n) may be calculated as the inverse of the global dimming gain440a. This BLU intensity may be incorporated into the level mapping416bto ensure that brightness and/or dimming values that are calculated in the level mapping416bare adjusted according to this BLU intensity. For example, if the BLU intensity associated with the global dimming gain440ais 50%, then brightness values or backlight intensity that is computed in the level mapping416bfor the current frame (e.g.,402bor frame n) is globally reduced by 50% (e.g., all brightness values are reduced by 50%).

The level mapping step416bmay output an initial backlight matrix, which may include an array of brightness and/or dimming values corresponding to a plurality of backlight zones. The initial backlight matrix may be representative of a BLU intensity with which to display the current frame402b. Simply displaying based on the initial backlight matrix produced after the level mapping416bmay be prone to some artifacts (e.g., backlight flickering artifacts, over-dimming artifacts, halo effects, etc.). In the post filtering step418b, the GD/LD algorithm410may perform post filtering (e.g., spatial-temporal filtering) on the initial backlight matrix produced after the level mapping step416ain order to correct one or more artifacts, as discussed elsewhere herein. Responsive to performing the post filtering418b, the GD/LD algorithm410generates a final backlight matrix. Based on the final backlight matrix, a BLU driver (e.g., BLU driver332) may output an adjusted backlight intensity420b(e.g., backlight intensity globally adjusted based on global dimming gain received from previous frame) that is used along with the adjusted display RGB422bto generate the display430b.

In the global dimming gain calculation414a, the GD/LD algorithm410may calculate a new BLU intensity for a subsequent/next frame (e.g., frame n+1) and a corresponding global dimming scale or gain440bto scale RGB values of the subsequent frame based on the new BLU intensity. The GD/LD algorithm may compute the new BLU intensity based on selecting a threshold for a target percentile and then converting the threshold to the BLU intensity using a specific LUT. More specifically, an accumulated histogram is generated using the maximum gray values computed from the backlight stats estimation step412b. For a given percentile, threshold may be taken. This threshold may be converted into a new BLU intensity using a LUT, as shown for example inFIG.5. From this new BLU intensity, the global dimming gain440bfor scaling RGB values and adjusting backlight intensity of subsequent frame (e.g., frame n+1) may be calculated. The global dimming gain440bmay be calculated using equation (1) discussed above. The computed global dimming gain440bis then passed to the next or subsequent frame for processing the next frame. In this way, global dimming may be performed for a series of frames and iterations, where, at each iteration, RGB values and backlight intensity are globally adjusted for a current frame based on previous frame's data (e.g., global dimming gain computed based on BLU stats of previous frame). Only the first frame in the series of frames is displayed according to its original RGB value. In addition to the global dimming, local dimming (discussed with respect toFIG.3) may also be performed at each iteration.

In some embodiments, a dramatic change in a global dimming gain (equivalently BLU intensity) from a previous global dimming gain may cause flickering. For example, if frames change dramatically from dark to bright, then it may cause flickering. One such example scenario is depicted inFIG.7.FIG.7illustrates three example consecutive frames T0, T1, and T2that may be displayed via a display system, such as display system1000. Global dimming discussed herein may be performed on these frames. As shown inFIG.7, the frames T0, T1, and T2change dramatically from dark (as shown in frame T0) to bright (as shown in frame T2). This may cause flickering. To prevent it from happening, a slow adaptation technique may be implemented using a queue or other techniques. Slow adaptation technique may be implemented based on the global dimming gains computed from previous frames and stored in a queue, as shown for example inFIG.8. So, the actual global dimming gain for the current frame under processing may be determined by taking an average value of all previous global dimming gains in the queue.

FIG.8illustrates an example slow adaptation technique to minimize artifacts associated with a plurality of frames. Specifically,FIG.8illustrates an example queue800to store global dimming gains associated with the plurality of frames T0, T1, T2, T3, and TN. Using the queue800, slow adaptation technique may be implemented to minimize artifacts, such as flickering artifacts. As illustrated, there is no global dimming gain stored in the queue800for the first frame since there is no previous frame data. Based on the first frame T0data (e.g., BLU stats), the GD/LD algorithm410may calculate a global dimming gain of 2.00 to adjust the RGB values and backlight intensity of second frame T1. The computed global dimming gain is stored in the queue800. For the next frame T2, the GD/LD algorithm410may calculate a global dimming gain of 1.7 based on the previous frame T1data. Instead of using the global dimming gain of 1.7, an average of current global dimming gain of 1.7 and previous global dimming gain(s) (e.g., 2) may instead be taken to determine a global dimming gain of 1.85 to use for the current frame i.e., T2. For the next frame T3, the GD/LD algorithm410may calculate a global dimming gain of 0.5 based on the previous frame T2data. Since there is significant difference between the current global dimming gain and previous gain, an average of current global dimming gain of 0.5 and previous global dimming gains (e.g., 2.0 and 1.7) may be taken to determine a global dimming gain of 1.40 to use for the current frame i.e., T3. Similarly, an average global dimming gain of 1.1 may be computed for frame TN. With the slow adaptation technique discussed herein, flickering may be minimized. Also, depending on the size of the queue (e.g., queue800), flickering or amount of power save may be determined.

In some embodiments, the size of a queue that is used for slow adaptation technique may vary. For instance, VR devices (e.g., display system1000) may support multiple frames rates, for example, 60 Hz, 72 Hz, 90 Hz, 120 Hz, etc. As the frame rate increases, the severity of flickering artifacts may also increase. Stated differently, higher the frame rate, higher are the changes of severe flickering artifacts. Therefore, depending on the frame rate of the device (e.g., display system1000), the queue size (or the speed of slow adaptation) may be varied. For instance, if the frame rate increases, the queue size may also increase to minimize the flickering. But, at low frame rates, the queue size may decrease to achieve more power savings.

FIG.9illustrates an example method900for global dimming, in accordance with particular embodiments. The method may begin at step910, where a computing system may receive a first image frame of a sequence of image frames to be shown on a display having a plurality of backlight zones. The sequence of image frames may be consecutive image frames, as shown, for example, frames T0, T1, and T2inFIG.7. Each of the sequence of image frames may include a plurality of color components, such as RGB color components or channels. The display may be associated with an artificial reality system, such as the artificial reality system1000shown inFIG.10. The artificial reality system may be a virtual reality headset.

At step920, the computing system may compute backlight unit statistics (i.e., BLU stats) of the first image frame. The backlight unit statistics may represent grayscale levels (e.g., RGB values) for the plurality of backlight zones. In particular embodiments, computing the backlight unit statistics may include computing, for each color component of a plurality of color components (e.g., RGB color components) in the first image frame, an average grayscale value of the color component across a plurality of pixels within each backlight zone of the plurality of backlight zones and then determining, for each color backlight zone, a maximum grayscale value among average values of the plurality of color components, as discussed for example in the backlight stats estimation step412ainFIG.4.

At step930, the computing system may compute a global dimming gain for adjusting color values and backlight unit intensity of a second image frame of the sequence of image frames based on the backlight unit statistics of the first image frame. In particular embodiments, computing the global dimming gain may include generating an accumulated histogram based on maximum grayscale values determined for the plurality of backlight zones during computation of the backlight unit statistics; determining, for a target percentile in the accumulated histogram, a threshold grayscale value; converting the threshold grayscale value to a particular backlight unit intensity using a lookup table; and computing the global dimming gain using the particular backlight unit intensity, as discussed for example in the global dimming gain calculation step414ainFIG.4. In particular embodiments, the global dimming gain is inversely proportional to the particular backlight unit intensity, as shown in the equation (1) discussed herein.

At step940, the computing system may adjust, using the global dimming gain, the color values and the backlight unit intensity of the second image frame. In particular embodiments, adjusting, using the global dimming gain, the color values and the backlight unit intensity of the second image frame may include scaling up RGB color values of the second image frame and reducing the backlight unit intensity of the plurality of backlight zones for displaying the second image frame, as discussed for example with respect to RGB frame402binFIG.4.

Particular embodiments may repeat one or more steps of the method ofFIG.9, where appropriate. Although this disclosure describes and illustrates particular steps of the method ofFIG.9as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.9occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for global dimming, including the particular steps of the method ofFIG.9, this disclosure contemplates any suitable method for global dimming, including any suitable steps, which may include a subset of the steps of the method ofFIG.9, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG.9, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG.9.

FIG.10illustrates an example of a display system, in particular, an artificial reality system1000worn by a user1002. The display system or the artificial reality system1000may be used to implement some of the embodiments/examples disclosed herein. The artificial reality system1000may be configured to operate as a virtual reality display, an augmented reality display, and/or a mixed reality display. In particular embodiments, the artificial reality system1000may comprise a head-mounted device (“HMD”)1004, a controller1006, and a computing system1008. The HMD1004may be worn over the user's eyes and provide visual content (e.g., VR or AR content) to the user1002through internal displays (not shown). The HMD1004may have two separate internal displays, one for each eye of the user1002. As illustrated inFIG.10, the HMD1004may completely cover the user's field of view. By being the exclusive provider of visual information to the user1002, the HMD1004achieves the goal of providing an immersive artificial-reality experience.

The HMD1004may have external-facing cameras, such as the two forward-facing cameras1005A and1005B shown inFIG.10. While only two forward-facing cameras1005A-B are shown, the HMD1004may have any number of cameras facing any direction (e.g., an upward-facing camera to capture the ceiling or room lighting, a downward-facing camera to capture a portion of the user's face and/or body, a backward-facing camera to capture a portion of what's behind the user, and/or an internal camera for capturing the user's eye gaze for eye-tracking purposes). The external-facing cameras are configured to capture the physical environment around the user and may do so continuously to generate a sequence of frames (e.g., as a video).

The 3D representation may be generated based on depth measurements of physical objects observed by the cameras1005A-B. Depth may be measured in a variety of ways. In particular embodiments, depth may be computed based on stereo images. For example, the two forward-facing cameras1005A-B may share an overlapping field of view and be configured to capture images simultaneously. As a result, the same physical object may be captured by both cameras1005A-B at the same time. For example, a particular feature of an object may appear at one pixel pAin the image captured by camera1005A, and the same feature may appear at another pixel pBin the image captured by camera1005B. As long as the depth measurement system knows that the two pixels correspond to the same feature, it could use triangulation techniques to compute the depth of the observed feature. For example, based on the camera1005A's position within a 3D space and the pixel location of pArelative to the camera1005A's field of view, a line could be projected from the camera1005A and through the pixel pA. A similar line could be projected from the other camera1005B and through the pixel pB. Since both pixels are supposed to correspond to the same physical feature, the two lines should intersect. The two intersecting lines and an imaginary line drawn between the two cameras1005A and1005B form a triangle, which could be used to compute the distance of the observed feature from either camera1005A or1005B or a point in space where the observed feature is located.

In particular embodiments, the pose (e.g., position and orientation) of the HMD1004within the environment may be needed. For example, in order to render the appropriate display for the user1002while he is moving about in a virtual environment, the system1000would need to determine his position and orientation at any moment. Based on the pose of the HMD, the system1000may further determine the viewpoint of either of the cameras1005A and1005B or either of the user's eyes. In particular embodiments, the HMD1004may be equipped with inertial-measurement units (“IMU”). The data generated by the IMU, along with the stereo imagery captured by the external-facing cameras1005A-B, allow the system1000to compute the pose of the HMD1004using, for example, SLAM (simultaneous localization and mapping) or other suitable techniques.

In particular embodiments, the artificial reality system1000may further have one or more controllers1006that enable the user1002to provide inputs. The controller1006may communicate with the HMD1004or a separate computing unit1008via a wireless or wired connection. The controller1006may have any number of buttons or other mechanical input mechanisms. In addition, the controller1006may have an IMU so that the position of the controller1006may be tracked. The controller1006may further be tracked based on predetermined patterns on the controller. For example, the controller1006may have several infrared LEDs or other known observable features that collectively form a predetermined pattern. Using a sensor or camera, the system1000may be able to capture an image of the predetermined pattern on the controller. Based on the observed orientation of those patterns, the system may compute the controller's position and orientation relative to the sensor or camera.

The artificial reality system1000may further include a computer unit1008. The computer unit may be a stand-alone unit that is physically separate from the HMD1004or it may be integrated with the HMD1004. In embodiments where the computer1008is a separate unit, it may be communicatively coupled to the HMD1004via a wireless or wired link. The computer1008may be a high-performance device, such as a desktop or laptop, or a resource-limited device, such as a mobile phone. A high-performance device may have a dedicated GPU and a high-capacity or constant power source. A resource-limited device, on the other hand, may not have a GPU and may have limited battery capacity. As such, the algorithms that could be practically used by an artificial reality system1000depends on the capabilities of its computer unit1008.

FIG.11illustrates an example network environment1100associated with an augmented reality (AR)/virtual reality (VR) system or a social-networking system. Network environment1100includes a client system1130, a VR (or AR) or social-networking system1160, and a third-party system1170connected to each other by a network1110. AlthoughFIG.11illustrates a particular arrangement of client system1130, VR or social-networking system1160, third-party system1170, and network1110, this disclosure contemplates any suitable arrangement of client system1130, AR/VR or social-networking system1160, third-party system1170, and network1110. As an example and not by way of limitation, two or more of client system1130, AR/VR or social-networking system1160, and third-party system1170may be connected to each other directly, bypassing network1110. As another example, two or more of client system1130, AR/VR or social-networking system1160, and third-party system1170may be physically or logically co-located with each other in whole or in part. Moreover, althoughFIG.11illustrates a particular number of client systems1130, AR/VR or social-networking systems1160, third-party systems1170, and networks1110, this disclosure contemplates any suitable number of client systems1130, AR/VR or social-networking systems1160, third-party systems1170, and networks1110. As an example and not by way of limitation, network environment1100may include multiple client system1130, AR/VR or social-networking systems1160, third-party systems1170, and networks1110.

This disclosure contemplates any suitable network1110. As an example and not by way of limitation, one or more portions of network1110may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network1110may include one or more networks1110.

Links1150may connect client system1130, AR/VR or social-networking system system1160, and third-party system1170to communication network1110or to each other. This disclosure contemplates any suitable links1150. In particular embodiments, one or more links1150include one or more wireline (such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless (such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In particular embodiments, one or more links1150each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology-based network, a satellite communications technology-based network, another link1150, or a combination of two or more such links1150. Links1150need not necessarily be the same throughout network environment1100. One or more first links1150may differ in one or more respects from one or more second links1150.

In particular embodiments, client system1130may be an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of carrying out the appropriate functionalities implemented or supported by client system1130. As an example and not by way of limitation, a client system1130may include a computer system such as a desktop computer, notebook or laptop computer, netbook, a tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), handheld electronic device, cellular telephone, smartphone, augmented/virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client systems1130. A client system1130may enable a network user at client system1130to access network1110. A client system1130may enable its user to communicate with other users at other client systems1130.

In particular embodiments, client system1130may include a client application1132operable to provide various computing functionalities, services, and/or resources, and to send data to and receive data from the other entities of the network1110, such as the AR/VR or social-networking system1160and/or the third-party system1170. For example, the client application1132may be a social-networking application, an artificial-intelligence related application, a virtual reality application, an augmented reality application, an artificial reality or a mixed reality application, a camera application, a messaging application for messaging with users of a messaging network/system, a gaming application, an internet searching application, etc.

In particular embodiments, the client application1132may be storable in a memory and executable by a processor of the client system1130to render user interfaces, receive user input, send data to and receive data from one or more of the AR/VR or social-networking system1160and the third-party system1170. The client application1132may generate and present user interfaces to a user via a display of the client system1130.

In particular embodiments, social-networking or AR/VR system1160may provide users with the ability to take actions on various types of items or objects, supported by social-networking or AR/VR system1160. As an example and not by way of limitation, the items and objects may include groups or social networks to which users of social-networking or AR/VR system1160may belong, events or calendar entries in which a user might be interested, computer-based applications that a user may use, transactions that allow users to buy or sell items via the service, interactions with advertisements that a user may perform, or other suitable items or objects. A user may interact with anything that is capable of being represented in social-networking or AR/VR system1160or by an external system of third-party system1170, which is separate from social-networking or AR/VR system1160and coupled to social-networking or AR/VR system1160via a network1110.

In particular embodiments, social-networking or AR/VR system1160may be capable of linking a variety of entities. As an example and not by way of limitation, social-networking or AR/VR system1160may enable users to interact with each other as well as receive content from third-party systems1170or other entities, or to allow users to interact with these entities through an application programming interfaces (API) or other communication channels.

In particular embodiments, a third-party system1170may include one or more types of servers, one or more data stores, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable components, e.g., that servers may communicate with. A third-party system1170may be operated by a different entity from an entity operating social-networking or AR/VR system1160. In particular embodiments, however, social-networking or AR/VR system1160and third-party systems1170may operate in conjunction with each other to provide social-networking services to users of social-networking or AR/VR system1160or third-party systems1170. In this sense, social-networking or AR/VR system1160may provide a platform, or backbone, which other systems, such as third-party systems1170, may use to provide social-networking services and functionality to users across the Internet.

In particular embodiments, social-networking or AR/VR system1160also includes user-generated content objects, which may enhance a user's interactions with social-networking or AR/VR system1160. User-generated content may include anything a user can add, upload, send, or “post” to social-networking or AR/VR system1160. As an example and not by way of limitation, a user communicates posts to social-networking or AR/VR system1160from a client system1130. Posts may include data such as status updates or other textual data, location information, photos, videos, links, music or other similar data or media. Content may also be added to social-networking or AR/VR system1160by a third-party through a “communication channel,” such as a newsfeed or stream.

FIG.12illustrates an example computer system1200. In particular embodiments, one or more computer systems1200perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems1200provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems1200performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems1200. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

In particular embodiments, computer system1200includes a processor1202, memory1204, storage1206, an input/output (I/O) interface1208, a communication interface1210, and a bus1212. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor1202includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor1202may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1204, or storage1206; decode and execute them; and then write one or more results to an internal register, an internal cache, memory1204, or storage1206. In particular embodiments, processor1202may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor1202including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor1202may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory1204or storage1206, and the instruction caches may speed up retrieval of those instructions by processor1202. Data in the data caches may be copies of data in memory1204or storage1206for instructions executing at processor1202to operate on; the results of previous instructions executed at processor1202for access by subsequent instructions executing at processor1202or for writing to memory1204or storage1206; or other suitable data. The data caches may speed up read or write operations by processor1202. The TLBs may speed up virtual-address translation for processor1202. In particular embodiments, processor1202may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor1202including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor1202may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors1202. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory1204includes main memory for storing instructions for processor1202to execute or data for processor1202to operate on. As an example and not by way of limitation, computer system1200may load instructions from storage1206or another source (such as, for example, another computer system1200) to memory1204. Processor1202may then load the instructions from memory1204to an internal register or internal cache. To execute the instructions, processor1202may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor1202may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor1202may then write one or more of those results to memory1204. In particular embodiments, processor1202executes only instructions in one or more internal registers or internal caches or in memory1204(as opposed to storage1206or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory1204(as opposed to storage1206or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor1202to memory1204. Bus1212may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor1202and memory1204and facilitate accesses to memory1204requested by processor1202. In particular embodiments, memory1204includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory1204may include one or more memories1204, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage1206includes mass storage for data or instructions. As an example and not by way of limitation, storage1206may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage1206may include removable or non-removable (or fixed) media, where appropriate. Storage1206may be internal or external to computer system1200, where appropriate. In particular embodiments, storage1206is non-volatile, solid-state memory. In particular embodiments, storage1206includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage1206taking any suitable physical form. Storage1206may include one or more storage control units facilitating communication between processor1202and storage1206, where appropriate. Where appropriate, storage1206may include one or more storages1206. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface1208includes hardware, software, or both, providing one or more interfaces for communication between computer system1200and one or more I/O devices. Computer system1200may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system1200. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces1208for them. Where appropriate, I/O interface1208may include one or more device or software drivers enabling processor1202to drive one or more of these I/O devices. I/O interface1208may include one or more I/O interfaces1208, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.