Automatic white balancing of display device to match user-preferred modes

An electronic device, method and computer program product enable an enhanced user experience by adjusting a white balance of a display device according to user preferences in combination with automatically adjusting luminance and chromaticity in response to ambient light conditions. The electronic device includes a color-sensitive ambient light sensor (ALS) and the display device. A controller of the electronic device measures luminance and chromaticity of ambient light using the ALS. The controller adjusts display brightness of the display device in relation to intensity of the luminance of the ambient light. The controller automatically determines a display target correlated color temperature (CCT) in relation to the chromaticity of the ambient light. The controller determines a display target white balance CCT adjusted in relation to user selection(s) related to white balance. The controller sets the display device to digitally map colors about a neutral color defined by the display target white balance.

CLAIM TO FOREIGN PRIORITY

This application claims priority to Chinese Patent Application No. 202210085149.9, filed Jan. 25, 2022, the content of which is fully incorporated herein.

BACKGROUND

1. Technical Field

The present disclosure relates generally to electronic devices with a color display device, and more particularly to electronic devices having an ambient light sensor that is used in adjusting the color display device for ambient light conditions.

2. Description of the Related Art

Electronic devices such as mobile phones, laptops, and tablets have display devices that present color images and videos. A number of factors contribute to accurately reproducing a particular color in the images and videos. Imperfect manufacturing process can lead to less than ideal display devices. Complex underlying physical mechanisms of the display device can degrade during the display's lifetime, leading to less than ideal display performance. Ambient lighting conditions vary in color and luminance, which causes rendering of images and video presented at the display device to appear inaccurate and to not have a subjectively preferred appearance for the person viewing the display device. White balance is the process of adjusting the presented colors so that objects which appear white in person are rendered white. White balance may also be intentionally shifted from a neutral point to create a color cast to the presented image or video (i.e., an unwanted color shift in the whole image caused by reflected light from a nearby object. The display white balance affects display image quality and video quality. Evidence shows that display white balance has a positive correlation to visual experience. Electronic devices to varying degrees may automate certain automatic white balance algorithms to adjust the white balance for changing conditions. However, users have preferred color tones, which are not the same as the ambient light correlated color temperature (CCT) that is used in known automated white balance adjustment algorithms.

DETAILED DESCRIPTION

According to a first aspect of the present disclosure, an electronic device, a method, and a computer program product improve the perceived display color and luminance, taking into consideration the ambient light and user preference. In particular, the present disclosure enables an enhanced user experience by adjusting a white balance of a display device according to user preferences in combination with automatically adjusting luminance and chromaticity in response to ambient light conditions. Examples of automatic adjustments are referred to herein as solutions 1-4. The electronic device includes a color-sensitive ambient light sensor (ALS) and the display device. A controller of the electronic device measures luminance and chromaticity of ambient light using the ALS. The controller adjusts display brightness of the display device in relation to intensity of the luminance of the ambient light. The controller automatically determines a display target white balance correlated color temperature (CCT) in relation to the chromaticity of the ambient light. The controller adjusts the display target white balance CCT in relation to user preferences (“user selection(s) related to white balance”). The controller sets the display device to digitally map colors about a neutral color defined by the display target white balance based on the display target white balance CCT.

In one or more embodiments, the electronic device provides a real-time, automatic display white balance (WB) system that is able to produce a real-time WB experience with ambient lightings on still images and videos. In addition, display luminance and chromaticity can be set differently due to user preference, display content, and viewing conditions. The present disclosure provides various automatic display WB solutions which takes into consideration user preference, and which includes measuring the ambient CCT with a color-sensitive ALS.

In the following detailed description of exemplary embodiments of the disclosure, specific exemplary embodiments in which the various aspects of the disclosure may be practiced are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, architectural, programmatic, mechanical, electrical, and other changes may be made without departing from the spirit or scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and equivalents thereof. Within the descriptions of the different views of the figures, similar elements are provided similar names and reference numerals as those of the previous figure(s). The specific numerals assigned to the elements are provided solely to aid in the description and are not meant to imply any limitations (structural or functional or otherwise) on the described embodiment. It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements.

As further described below, implementation of the functional features of the disclosure described herein is provided within processing devices and/or structures and can involve use of a combination of hardware, firmware, as well as several software-level constructs (e.g., program code and/or program instructions and/or pseudo-code) that execute to provide a specific utility for the device or a specific functional logic. The presented figures illustrate both hardware components and software and/or logic components.

Those of ordinary skill in the art will appreciate that the hardware components and basic configurations depicted in the figures may vary. The illustrative components are not intended to be exhaustive, but rather are representative to highlight essential components that are utilized to implement aspects of the described embodiments. For example, other devices/components may be used in addition to or in place of the hardware and/or firmware depicted. The depicted example is not meant to imply architectural or other limitations with respect to the presently described embodiments and/or the general invention. The description of the illustrative embodiments can be read in conjunction with the accompanying figures. Embodiments incorporating teachings of the present disclosure are shown and described with respect to the figures presented herein.

FIG.1is a functional block diagram of an electronic device in an operating environment within which the features of the present disclosure are advantageously implemented. In particular, electronic device100is an example of an electronic device that enables an enhanced user experience by automatically adjusting a white balance of display device102, in part to respond to user preferences. Electronic device100can be one of a host of different types of devices, including but not limited to, a mobile cellular phone, satellite phone, or smart-phone, a laptop, a net-book, an ultra-book, a networked smart watch or networked sports/exercise watch, and/or a tablet computing device or similar device that can include wireless communication functionality.

As more completed presented within communication device200ofFIG.2, described hereafter, electronic device100can also be a device supporting wireless communication. In these implementations, electronic device100can be utilized as, and also be referred to as, a system, device, subscriber unit, subscriber station, mobile station (MS), mobile, mobile device, remote station, remote terminal, user terminal, terminal, user agent, user device, a Session Initiation Protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), computer workstation, a handheld device having wireless connection capability, a computing device, or other processing devices connected to a wireless modem. Most importantly, it is appreciated that the features described herein can be implemented with a display device of various other types of electronic devices that are not necessarily a communication device. The specific presentation or description herein of a mobile communication device in addition to a data processing system as different examples of electronic device100are for example only, and not intended to be limiting on the disclosure.

Electronic device100is managed by controller101and includes ambient light sensor (ALS)103, which is color-sensitive. In one or more embodiments, ALS103is a red-green-blue (RGB) sensor that detects three colors. In one or more embodiments, ALS103is a hyper-spectral that detects “n” colors, where n is more than 3 colors, such as 10, 100 or more colors.

Person104viewing images or videos presented at display device102experiences the color image or video differently because of the ambient light conditions. Ambient light affects the eyes of the person104. When the luminance is low, the cones in the retina of the eyes that are sensitive to color are less effective. Ambient light also reflects back from display device102, changing the colors perceived by person104. According to one aspect of the disclosure, controller101executes automatic display white balance (ADWB) application105, which is stored in device memory106, to measure luminance and chromaticity of ambient light using ALS103and to adjust display brightness of display device102in relation to intensity of the luminance of the ambient light. Controller101automatically determines a display target white balance CCT in relation to the chromaticity of the ambient light. Controller101determines a display target white balance CCT that is adjusted in relation to user white balance selection data107. User white balance selection data107includes at least one user selection related to white balance. User white balance selection data107is related to white balance and is provided by a user of electronic device100and stored in device memory106of electronic device100. Controller101sets display device102to digitally map colors about a neutral color defined by the display target white balance to enhance experience of person104.

Referring now to the specific component makeup and the associated functionality of the presented components. In one or more embodiments, electronic device100includes device memory106data storage subsystem109, input/output (I/O) subsystem110, and network interface190, each of which is managed by controller101. Device memory106includes program code for applications, such as ADWB application105, image recognition application112, and other application(s)113. Device memory106further includes operating system (OS)114, firmware interface115, such as basic input/output system (BIOS) or Uniform Extensible Firmware Interface (UEFI), and firmware116.

Controller101includes processor subsystem117, which executes program code to provide operating functionality of electronic device100. The software and/or firmware modules have varying functionality when their corresponding program code is executed by processor subsystem117or secondary processing devices within electronic device100. Processor subsystem117of controller101can execute program code of ADWB application105and other application(s)113to configure electronic device100to perform specific functions. Device memory106can include data118, and in particular user white balance selection data107used by the ADWB application105. ADWB application105refers to user white balance selection data107when performing the automatic white balancing. Image recognition application112can refer to image library119to identify a particular object when performing one embodiment of color adjustment. Automatic color adjustments may be defined for particular types of objects that are identified by the image recognition application112, described herein as “solution 3”.

Data storage subsystem109of electronic device100includes data storage device(s)152. Controller101is communicatively connected, via system interlink153, to data storage device(s)152. Data storage subsystem109provides applications, program code, and stored data on nonvolatile storage that is accessible by controller101. For example, data storage subsystem109can provide a selection of applications and computer data such as ADWB application105and other application(s)113. These applications can be loaded into device memory106for execution by controller101. In one or more embodiments, data storage device(s)152can include hard disk drives (HDDs), optical disk drives, and/or solid-state drives (SSDs), etc. Data storage subsystem109of electronic device100can include removable storage device(s) (RSD(s))156, which is received in RSD interface157. Controller101is communicatively connected to RSD156, via system interlink153and RSD interface157. In one or more embodiments, RSD156is a non-transitory computer program product or computer readable storage device. Controller101can access RSD156or data storage device(s)152to provision electronic device100with program code, such as code for ADWB application105and other application(s)113.

I/O subsystem110includes user interface components such as vibration output device158, light output device159, image capturing device(s)160, microphone161, display device102that presents user interface163, touch/haptic controls164, and audio output device(s)166. Display device102is communicatively coupled to controller101and presents a user interface and/or operates as a user interface device.

Controller101manages, and in some instances directly controls, the various functions and/or operations of electronic device100. These functions and/or operations include, but are not limited to including, application data processing, communication with second communication devices, navigation tasks, image processing, and signal processing. In one or more alternate embodiments, electronic device100may use hardware component equivalents for application data processing and signal processing. For example, electronic device100may use special purpose hardware, dedicated processors, general purpose computers, microprocessor-based computers, micro-controllers, optical computers, analog computers, dedicated processors and/or dedicated hard-wired logic.

Controller101includes processor subsystem117, which includes one or more central processing units (CPUs), depicted as data processor179. Processor subsystem117can include one or more digital signal processors180that are integrated with data processor179. Processor subsystem117can include other processors that are communicatively coupled internally or externally to data processor179. Data processor179is communicatively coupled, via system interlink153, to device memory106, data storage subsystem109, and network interface190. Network interface190enables electronic device100to connect (via wireless or wired connection) to external network192and directly/indirectly to other devices196. Network192provides connection to and can include one or more network servers194and can provide connection to other devices196. Electronic device100is thus able to connect with servers194and other devices196to share and/or download application data that can be utilized to implement features of the disclosure. System interlink153represents internal components that facilitate internal communication by way of one or more shared or dedicated internal communication links, such as internal serial or parallel buses. As utilized herein, the term “communicatively coupled” means that information signals are transmissible through various interconnections, including wired and/or wireless links, between the components. The interconnections between the components can be direct interconnections that include conductive transmission media or may be indirect interconnections that include one or more intermediate electrical components. Although certain direct interconnections (system interlink153are illustrated inFIG.1, it is to be understood that more, fewer, or different interconnections may be present in other embodiments.

FIG.2is a functional block diagram of communication device200in an operating environment within which the features of the present disclosure are advantageously implemented. Communication device200is an implementation of electronic device100(FIG.1) that further includes communication subsystem202for communicating using a cellular connection with network node(s)204of external communication system206and for communicating using a wireless connection with access node(s)208of local communication system210. Communication subsystem202includes antenna subsystem212. Communication subsystem202includes radio frequency (RF) front end213and communication module214. RF front end213includes transceiver(s)216, which includes transmitter(s)218and receiver(s)220. RF front end213further includes modem(s)222. Communication module214of communication subsystem202includes baseband processor224that communicates with controller101and RF front end213. Baseband processor224operates in a baseband frequency range to encode data for transmission and decode received data, according to a communication protocol. Modem(s)222modulate baseband encoded data from communication module214onto a carrier signal to provide a transmit signal that is amplified by transmitter(s)218. Modem(s)222demodulates each signal received from external communication subsystem202using by antenna subsystem212. The received signal is amplified and filtered by receiver(s)220, which demodulate received encoded data from a received carrier signal.

In one or more embodiments, controller101, via communication subsystem202, performs multiple types of cellular OTA or wireless communication with local communication system210. Communication subsystem202can communicate via Bluetooth connection with one or more personal access network (PAN) devices, such as smart watch226and wireless headset228. Communication via Bluetooth connection includes both transmission and reception via a Bluetooth transceiver device. In one or more embodiments, communication subsystem202communicates with one or more locally networked devices via a wireless local area network (WLAN) link provided by access node(s)208. In one or more embodiments, access node(s)208supports communication using one or more IEEE 802.11 WLAN protocols. Access node(s)208is connected to a wide area network such as the Internet. In one or more embodiments, communication subsystem202communicates with GPS satellites230to obtain geospatial location information.

In one or more embodiments, communication device200includes network interface controller (NIC or “network interface”)242with a network connection (NC)243. Network interface242can be synonymous with and perform similar functions as network interface190(FIG.1) in some implementations. Network cable244connects NC243to wired area network245. NIC242can be referred to as a “network interface” that can support one or more network communication protocols. Wired area network245can be a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), or a wide area network (WAN). For example, NC243can be an Ethernet connection. Network device246is communicatively coupled to wired area network245.

FIG.3depicts display device102of electronic device100that presents user interface163by which user selections for color preferences can be received. User interface163presents image302, which includes visual representations of the effects of user selections for color preferences. As an example, the user can select one of three color presets radio buttons304associated with “Natural”, “Boosted”, and “Saturated” effects. As another example, color temperature slider 306 may be positioned along temperature line307extending between warm colors and cool colors. In an additional example, automatic ambient light compensation radio buttons308may be selected for linear adjusted “true tone”, non-linear adjusted, adjusted based on image content, and demographic presets.

In the described embodiments, the linear relationship is referred to as “solution 1”, which may be preferred as a true tone option for an electronic reader, presenting an easy-to-read paper-like illumination for normal illumination levels such as indoors. The non-linear relationship is referred to as “solution 2” and may be utilized for an electronic reader for outdoor lighting that vary from low to high illumination levels or other conditions that pursue an aesthetically pleasing color. In an example, a color cast of the image may be automatically adjusted to match an automatically determined lighting temperature of the content of the image or video. Landscape scenes that appear to be taken under a cloudy sky may have the color temperature adjusted to the color associated with cloudiness. Landscape scenes that appear to be taken under a bright sun may have the color temperature adjusted to the color associated with daylight. Indoor scenes with low illumination device presented such as a candle may have the color temperature adjusted to the color associated with candlelight. Alternatively, or as an additional adjustment provided as “solution 3”, particular colors and/or the white balance may be accentuated or diminished for predefined cultural or individual associations of content of presented images or video. In an example for solution 3, images recognized as pepperoni pizza may have particular shades of red boosted. In another example, an uncloudy portion of the sky in a landscape scene may be boosted to have a vibrant hue of blue. More than one color of particular objects may be boosted or reduced. Solution 3 may be preferred for colorful pictures, video, and games. In a further example, the demographic preset is referred to as “solution 4”. Demographic category entry fields310allow entry of age, individual characterization, gender, and personality. Individual characterization may include job categories, hobby categories, ethnic affiliations, etc., that have a correlation as a group to particular preferences in white balancing. To further assist in selecting a unique color temperature, multiple color temperature image312includes more than one version of unique color temperature samples to enable comparison of unique color temperature preset options for selecting one of color temperature radio buttons314: “tungsten”, “fluorescent”, “flash”, “cloudy”, “shade” and “daylight”. It is appreciated that different lists of color temperature presets can be provided in other embodiments, including more options and excluding some of the presented options. The listed preset options are presented solely as one example of the solution 4 embodiment. Solution 4 may be preferred for browsing text-like Internet webpages, using instant messaging application, performing online shopping, booking tickets, GPS navigating, playing music, participating in a video meeting, and making a voice call, etc.

According to one aspect, Controller101(FIG.1), executing ADWB application105, performs automatic white balancing by first measuring luminance and chromaticity using ALS103to obtain an ambient CCT estimation:

L=Σλ⁢Lamb(λ)⁢ALS⁡(λ)⁢Δ⁡(λ)X=Σλ⁢Lamb(λ)⁢ALS⁡(λ)⁢x_(λ)⁢Δ⁡(λ)Y=Σλ⁢Lamb(λ)⁢ALS⁡(λ)⁢y_(λ)⁢Δ⁡(λ)Z=Σλ⁢Lamb(λ)⁢ALS⁡(λ)⁢z_(λ)⁢Δ⁡(λ)x=XX+Y+Zy=YX+Y+Zz=ZX+Y+Z
Commission Internationale de l'Eclairage (CIE) color matching functions,x,y,z, that are factors in the equations above are the chromatic response of the standard observer.

Second, controller101performs automatic white balancing by adjusting display brightness based on the ambient light intensity. In an example, controller101maps luminance to display panel backlight intensity nits, where a nit is a unit of luminance, with 1 nit=1 candela per square meter.

Third, controller101performs automatic white balancing by adjusting display chromaticity based on the formula to get the target CCT:
n=(x−0.332)/(0.1858−y)
CCTambient=449×n3+3525×n2+6823.3×n+5520.33

Fourth, controller101applies delta_x, delta_y values to display device102. In order to conserve computing capacity and to reduce power consumption, controller101may determine that the change in white point from a current setting is less than a threshold amount and not continue with making a change to display device102.

FIG.4depicts graphical diagram400of two plots402,404respectively of linear and nonlinear relationships between illuminant CCT (x-axis) and display target white balance CCT (y-axis). From the illuminant CCT (x-axis value), controller101(FIGS.1-2) may use solution 1 to determine a display white point on plot402in a linear relationship that matches the illuminant CCT (x-axis value). Plot402has a slope of 1 and intersects the origin. Alternatively, from the illuminant CCT (x-axis value), controller101(FIGS.1-2) may use solution 2 to determine the display white point on plot404in a nonlinear relationship that is predefined and stored in memory. The nonlinear relationship is subjectively and empirically predetermined by an original equipment manufacturer (OEM) as having an optimal display color temperature across a range of ambient lighting conditions. The electronic device is provisioned with this nonlinear relationship such as in a lookup table or formula.

FIGS.5A-5C(collectivelyFIG.5) present a flow diagram of method500performed by electronic device100for responding to ambient light conditions using user preferences to adjust a white balance of a display device. The descriptions of method500is provided with general reference to the specific components illustrated within the precedingFIGS.1-4, and specific components referenced in method500may be identical or similar to components of the same name used in describing precedingFIGS.1-4. In one or more embodiments, controller101configures electronic device100(FIG.1) or communication device (FIG.2) to provide functionality of method500. With reference toFIG.5A, method500includes receiving an individual customized CCT that is user inputted to settings of the electronic device indicating a user preference for white balance (block502). Method500includes receiving selection(s) by the user of particular images or videos previously selected by the user that have a different white balance than other presented images or videos indicating a user preference for white balance (block504). Method500includes monitoring applications being used by a user on the electronic device for an applicable automatic solution 1, 2, 3, or 4 that are respectively adjusted by user preferences (block506). Method500includes monitoring a communication subsystem and sensors to determine whether the electronic device is indoors or outdoors as applicable to determine context of use of electronic reader for one of solutions 1 and 2 (block508). Method500includes receiving or being provisioned with library of object images with predefined color adjustment(s) for use in solution 3 (block510). Method500includes receiving or being provisioned with pre-defined demographic color adjustment data for use in solution 4 (block512). Method500includes receiving user input demographic data applicable to the user for use in solution 4 (block514). Then method500proceeds to block516(FIG.5B).

With reference toFIG.5B, method500includes measuring luminance and chromaticity of ambient light using a color-sensitive ambient light sensor (ALS) (block516). Method500includes adjusting display brightness of a display device in relation to intensity of the luminance of the ambient light (block518). Method500includes automatically determining a display target white balance correlated color temperature (CCT) in relation to the chromaticity of the ambient light (block520). Method500includes identifying automatic selection related to white balance as one of solutions 1, 2, 3, or 4 (block522). In an example, the electronic device determines a current mode of usage and automatically selects a corresponding solution. Using an electronic reader in an indoor setting indicates solution 1. Using the electronic reader in an outdoor setting indicates solution 2. Playing a game or presenting visual media content that is bright and colorful may indicate solution 3. Browsing internet content or using communication applications may indicate solution 4. Method500includes determining whether the automatic selection is solution 1 (decision block524). In response to determining that the automatic selection is solution 1, method500includes automatically determine the display target white balance CCT in relation to the chromaticity of the ambient light using a linear relation (block526). Then, method500proceeds to block542(FIG.5C). In response to determining that the automatic selection is not solution 1, method500includes determining whether the automatic selection is solution 2 (decision block528). In response to determining that the automatic selection is solution 2, method500includes automatically determining the display target white balance CCT in relation to the chromaticity of the ambient light using a predefined nonlinear relation (block530). Then method500proceeds to block542(FIG.5C).

With reference toFIG.5C, in response to determining that the automatic selection is not solution 2, method500includes determining whether the automatic selection is solution 3 (decision block532). In response to determining that the automatic selection is not solution 3, method500proceeds to block546. In response to determining that the automatic selection is solution 3, method500includes identifying at least one object in an image or video presented by the display device at a predefined white balance refresh rate (block534). Method500includes associating the at least one object with at least one predefined white balance adjustment stored in memory (block536). Method500includes identifying user selection(s) related to white balance based on a particular image or video previously selected by the user that has a different white balance than other presented images or videos (block538). Method500includes determining a display target white balance CCT, which is adjusted in relation to at least one user selection related to white balance and that shifts a neutral color setting (block540). In an example, one or more defined colors in an image or video are adjusted in luminance and/or chromaticity within a portion of an image that is recognized as the associated object. In another example, a white point is adjusted to impart a color cast to an entire image or video in response to identifying at least one object. Method500includes setting the display device to digitally map colors about a neutral color defined by the display target white balance CCT (block542). Method500includes presenting, at the display device, images or video with the display target white balance CCT (block544). Then, method500returns to block516(FIG.5B).

In response to determining that the automatic selection is not solution 3 in decision block532, solution 4 is identified as the applicable solution, and method500includes using demographic data to identify a selected display target white balance CCT as one predefined model of display target white balance CCT that is associated with the demographic data (block546). In an example, the electronic device is provisioned with the predefined white balance adjustments by the OEM. The electronic device determines which of the predefined white balance adjustments are applicable to the user of the electronic device. Method500includes identifying user selection(s) related to white balance previously received as an individual customized CCT that is user inputted to settings of the electronic device with user's demographic data (block548). Method500includes determining a display target white balance CCT adjusted in relation to the user preferences to shift a neutral color setting (block550). Then method returns to block542.

As will be appreciated by one skilled in the art, embodiments of the present innovation may be embodied as a system, device, and/or method. Accordingly, embodiments of the present innovation may take the form of an entirely hardware embodiment or an embodiment combining software and hardware embodiments that may all generally be referred to herein as a “circuit,” “module” or “system.”

While the innovation has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted for elements thereof without departing from the scope of the innovation. In addition, many modifications may be made to adapt a particular system, device, or component thereof to the teachings of the innovation without departing from the essential scope thereof. Therefore, it is intended that the innovation not be limited to the particular embodiments disclosed for carrying out this innovation, but that the innovation will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another.