Patent ID: 12260496

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known after an understanding of the disclosure of this application may be omitted for increased clarity and conciseness.

The features described herein may be embodied in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

The terminology used herein is for describing various examples only and is not to be used to limit the disclosure. The articles “a” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items. As non-limiting examples, terms “comprise” or “comprises,” “include” or “includes,” and “have” or “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.

Throughout the specification, when a component or element is described as being “connected to,” “coupled to,” or “joined to” another component or element, it may be directly “connected to,” “coupled to,” or “joined to” the other component or element, or there may reasonably be one or more other components or elements intervening therebetween. When a component or element is described as being “directly connected to,” “directly coupled to,” or “directly joined to” another component or element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.

Although terms such as “first,” “second,” and “third”, or A, B, (a), (b), and the like may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms. Each of these terminologies is not used to define an essence, order, or sequence of corresponding members, components, regions, layers, or sections, for example, but used merely to distinguish the corresponding members, components, regions, layers, or sections from other members, components, regions, layers, or sections. Thus, a first member, component, region, layer, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, or section without departing from the teachings of the examples.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term “may” herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Light estimation or modeling is a technique for estimating the illumination of a scene. Examples described herein may include using estimated light information to render a virtual object in a corresponding image or space. For example, the estimated light information may be applied to a virtual object in an augmented reality (AR) or computer graphics (CG) environment. The more accurately the light information is estimated, the more realistically the virtual object may be rendered. A machine learning-based model may be used for light estimation. The model may be trained using dues such as ambient light, shadings, specular highlights, and reflections.

FIG.1illustrates an example of a configuration of an image processing model, according to one or more embodiments. Referring toFIG.1, a light estimation model110may estimate light information111corresponding to a background image, and a rendering model120may render a virtual object image121based on the light information111.

The background image101may be an image of a scene of a captured view that is viewed from a capturing point (the view and capture point may, in some implementations, be predetermined). Light information102may include information about all light affecting the scene of the background image101. The light information102may represent the light of the scene in various forms. For example, the light information102may represent light in a form of an environment map, or may represent light using predefined attributes (e.g., direction, color, brightness, width, etc.).

The background image101may include image data of a real background behind which a virtual object is displayed in a superimposed manner, for example on a semi-transparent surface between the real background and a user viewing the virtual object, or on an opaque display which the user may in front of the real background. The light information102may be applied to a virtual object rendered in an augmented reality (AR) or computer graphics (CG) environment. For example, when an example AR system is overlaying a virtual object on the background image101, the light information102may be applied to the rendering of the virtual object such that the virtual object may be displayed on the background image101with improved harmony between the two. The more accurately the light information102represents the light in the background image101, the more realistic or scene-accurate the virtual object may be.

The light estimation model110may be a machine learning model. For example, the light estimation model110may include a deep neural network (DNN) based on deep learning. The DNN may include a plurality of layers, and at least a portion thereof may be configured as various networks such as a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). The DNN may have a general ability to, based on deep learning, map input data and output data that may have a nonlinear relationship. At least a portion of the plurality of layers of the light estimation model110may correspond to one or more CNNs, and at least another portion thereof may correspond to one or more FCNs.

In general, during a process of generating an image using a camera, image signal processing (ISP) may be performed, typically through an image processing pipeline. For example, raw image data corresponding to visual information may be generated through an image sensor, and the ISP may be applied to the raw image data to generate the image. In this example, an ISP element may be controlled according to a variable adjust value. For example, the ISP element may include, for example, one or more of an auto white balance element/stage, an auto exposure element/stage, a gamma correction element/stage, a dynamic range compression element/stage, and/or a wide dynamic range element/stage, and these ISP elements may be freely adjusted (or activated/deactivated) depending on a variable adjust value. An ISP setting allowing free control (varying) of an ISP element depending on the variable adjust value may be referred to as a variable ISP setting.

In response to the background image101being generated through the variable ISP setting, an accuracy gap between the light information111and actual light of a real background corresponding to the background image101may occur. For example, in response to the real background having low illuminance, due to an auto exposure control, the background image101thereof may have high brightness (as caused by the auto exposure control), and the light estimation model110may correspondingly generate the light information111to have high illuminance, whereas the actual light condition of the real background is low illuminance. As another example, in response to the real background being illuminated by a yellow light, auto white balance may adjust a color of the background image101such that the background image101has more white light (is whiter), and the light information111may be correspondingly generated to the white light. In the above examples, the high brightness and the white light of the respective light information do not accurately represent the actual light information. In the case of an AR function being provided through an AR device using a translucent display such as AR glasses, it may be desirable for a virtual object presented on a translucent display to be rendered correspondingly to the actual light of the real background. For example, when a virtual object corresponding to high illuminance light information (e.g., is bright) is presented on a translucent display in a low-illuminance real environment, or when a virtual object corresponding to low illuminance light information (e.g., is dim) is presented on a translucent display in a high-illuminance real environment, the virtual object may be presented unnaturally relative to a real background that a user sees through the translucent display.

The light estimation model110may be trained based on actual light information. The actual light information may be estimated through fixed ISP setting. Here, “setting” may be one or more settings of one or more ISP elements. Unlike a variable ISP setting, the fixed ISP setting may use a limited fixed adjust value. For example, the fixed ISP setting may control an ISP element with a predetermined adjust value, or may not adjust an adjustable ISP element (e.g., allowing it to operate in a default state). The latter case may correspond to a state in which an ISP function is turned off, or a state in which an adjust value is “0” (e.g., the ISP element has no, or negligible, effect on the raw image). The light estimation model110may estimate the light information111as corresponding to absolute light of the actual light (having been trained using the fixed ISP setting.

FIG.2illustrates an example of components used to train a light estimation model, according to one or more embodiments. Referring toFIG.2, actual light212may be may be incident upon a reference object211positioned on a real background210, and a light estimation model may be trained to estimate the light212.

The light estimation model may be trained based on a background image220and a reference image230. The background image220may be generated based on raw image data acquired by capturing the real background210with a camera according to a variable ISP setting213. Because the variable ISP setting213is applied, the background image220may reflect information that somewhat differs from the actual light212when the background image220is captured. The reference image230may be generated based on the fixed ISP setting214and raw image data acquired by capturing the reference object211. The reference image230may include a reference object portion231corresponding to the reference object211. According to the fixed ISP setting214, the reference image230may reflect information close to the actual light212when the reference image230is captured. For example, when the actual light212provides low illuminance, the background image220may correspond to high illuminance (e.g., may be brighter) due to an exposure being controlled according to the variable ISP setting213, in contrast to the reference image230, which may correspond to low illuminance (e.g., may be dimmer) due to the fixed ISP setting214.

The light estimation model may estimate the light information221based on the background image220. A rendering model may generate a virtual object image240by rendering a virtual object241corresponding to the reference object211based on the light information221from the background image220. The rendering model may perform rendering based on object information and plane information in addition to the light information221. For example, the rendering model may perform neural rendering. The object information and the plane information may be information known in advance about the reference object211. The object information may include one or more of a pose, a shape, or a material (e.g., type of material) of an object (e.g., corresponding to the reference object2110, or any combination thereof. The material information may represent a texture, a color, or the like. The plane information may be information about a reference plane object/structure that supports the reference object, and may include at least one of a pose (e.g., a normal direction), a shape, and a material of the reference plane, or any combination thereof. The virtual object image240may include the virtual object241corresponding to the reference object211. The rendering model may render the virtual object241based on the light information221, the object information, and the plane information.

A difference250between the reference object portion231in the reference image230and the virtual object241in the virtual object image240may correspond to an error between the actual light212and the light information221. The light estimation model may be trained according to the difference250. The light estimation model may be trained to reduce a loss of a loss function corresponding to the difference250. Accordingly, the light estimation model may have an ability to learn to estimate the actual light212from the background image220according to the variable ISP setting213.

The rendering model may render shading of the virtual object241and/or a shadow of the virtual object241, and the difference250may include the difference250between pixel data representing shading and/or a shadow of the reference object portion231and pixel data representing the shading and/or the shadow of the virtual object241. The rendering model may perform shading rendering and/or shadow rendering at the same time or at different times, and generate a rendering by fusing the result of the shading rendering with the result of the shadow rendering.

Light estimation model input data (“input data” hereafter) corresponding to the background image220may be used as input data for the light estimation model. For example, when the variable ISP setting213may not be readily determined, such as when a parameter value for the variable ISP setting213is not provided, the input data of the light estimation model may be configured without the variable ISP setting213. Alternatively, input data corresponding to the background image220and the variable ISP setting213may be used as input for the light estimation model. For example, when the variable ISP setting213may be determined, such as when the parameter value for the variable ISP setting213is provided, the input data may be configured according to the variable ISP setting213.

FIG.3illustrates an example of an image capturing environment, according to one or more embodiments. Referring toFIG.3, a camera310may generate a reference image by capturing a reference object320, and generate a background image by capturing a real background in which the reference object is positioned. To obtain light information, instead of collecting direct light, which tends to have a significantly high brightness value, collecting indirect light from the reference object320may be advantageous. The reference object320may include multiple sub-objects, each of which may have a tone or a material different from the others, and planes supporting the objects (which may or may not differ from each other). For example, the sub-objects may have a variety of tones, such as a white tone, and a gray tone. In addition, the sub-objects may be formed of a variety of materials, such as an even-textured material, a shiny (or specular) metallic material, and a bumpy material. The sub-objects may have a variety of shapes, for example, a sphere shape, a star shape, and a cup shape. A reference object or sub-object may be implemented as a combination of a sphere and a supporting plane. The combination of the sphere and the supporting plane may effectively provide clues that may be used to estimate actual light. For example, the sphere may provide a clue as to a shading, and the supporting plane may provide a clue as to a shadow. However, the reference object is not limited to being the combination of the sphere and the supporting plane.

If multiple sub-objects are used, having a respective variety of tones thereof may help to reduce the possibility of saturation in a high dynamic range (HDR) environment and may allow light information to be collected from the widest possible range of light. In particular, with the fixed ISP setting being used, it is highly likely that brightness saturation will occur because an image may be captured without adjusting exposure or sensitivity of a sensor. When sub-objects with various tones, for example, are captured and capturing information indicating which sub-object has a most suitable tone for a capturing environment that is used, appropriate light information may be collected (e.g., from the tone-appropriate object) regardless of the capturing environment. The sub-objects may be represented by respective portions of image data in different dynamic ranges in raw image data acquired by capturing the reference object320, and a portion of image data having a widest dynamic range (among the portions of image data of the respective sub-object) may be selectively used for determining a difference between the reference image and the virtual object image. For example, capturing information of a gray sphere may be used because a white sphere in a high illuminance environment may cause saturation. Alternatively, capturing information of the white sphere may be used in a low illuminance environment. In some embodiments, the portion of image data corresponding to the most suitable sub-object for a given light/capturing environment may be determined automatically using techniques such as detecting the sub-objects, segmenting the reference image to isolate the portions of image data of the respective sub-objects, and analyzing the content to determine measures of image quality of the portions, e.g., measures of hue, saturation, contrast, range, evenness of distribution, and so forth.

The camera310may be a plurality of cameras or a single camera. In one example, the camera310may include a first camera that generates a reference image using the fixed ISP setting, and a second camera that generates a background image using the variable ISP setting (if both are captured at the same time they may both represent the same capturing environment). In another embodiment, the same camera may generate the reference image using the fixed ISP setting right before or after capturing the background image using the variable ISP setting. The reference object320may be connected to the camera310through a support330, which may facilitate accuracy and consistency.

FIG.4illustrates an example of a configuration of a camera, according to one or more embodiments. Referring toFIG.4, a camera400may generate an image402corresponding to visual information401. The camera400may include a lens group410, an image sensor420, and an ISP block430. The ISP block430may generate the image402by performing ISP based on raw image data421from the image sensor420. The ISP block430may include a software module and/or a hardware module for performing the ISP.

The camera400may correspond to a sub-camera of a plurality of cameras or a single camera. In an example of using two cameras, a first camera may generate the reference image using the fixed ISP setting and a second camera may generate the background image using the variable ISP setting. That is, when the camera400is the first camera and a limited ISP setting is used, the ISP block430may use the fixed ISP setting. When an ISP function is excluded, the ISP block430may be omitted (e.g., not present, deactivated, etc.). When the camera400is the second camera, the ISP block430may generate the image402using the variable ISP setting.

FIG.5illustrates an example of a configuration of a plurality of cameras, according to one or more embodiments. Referring toFIG.5, a camera500may include a first camera510that generates an image511using a fixed (e.g., default or unchangeable) ISP setting, and a second camera520that generates an image521using a variable ISP setting (usually, sufficiently different than the fixed ISP setting to provide an effective image difference).FIG.6illustrates an example of a configuration of a single camera, according to one or more embodiments. Referring toFIG.6, a third camera610may generate an image611and an image612alternately using a fixed ISP setting and a variable ISP setting.

FIG.7illustrates an example of a configuration of a virtual object image, according to one or more embodiments. Referring toFIG.7, a virtual object image710may include a virtual object711, shading714of the virtual object711, and/or shadows712and713of the virtual object711. Each of these elements of the virtual object image710may be rendered based on light information, object information, and plane information. The shading714may be derived through a shading rendering scheme (e.g., one or more shaders in a three-dimensional graphics sub-system of an AR system), and the shadows712and713may be derived through a shadow rendering scheme. The virtual object image710may be formed through fusing or merging the shading714and the shadows712and713. The virtual object image710may be compared with the reference object in the reference image, and a light estimation model may be updated according to a comparison result (e.g., a difference). Here, pixel data of a region (e.g., a reference object region) in the reference image and pixel data of a corresponding region in the virtual object image710may be compared. For example, pixel data representing shading and a shadow of the reference object and pixel data representing the shading714and the shadows712and713of the virtual object711may be compared pixel by pixel. In some embodiments, one or both images may be transformed (e.g., resized, clipped, or the like) to align their respective objects.

FIGS.8A-8Billustrate an example of training a light estimation model, according to one or more embodiments. Referring toFIG.8A, a background image801and a reference image802may be obtained. The background image801may be generated based on raw image data acquired by capturing a real background behind which the reference object is positioned and based on a variable ISP setting. The reference image802may be generated based on raw image data acquired by capturing the reference object and based on a fixed ISP setting. The variable ISP setting may control an ISP element with a variable adjust value, and the fixed ISP setting may control the ISP element with a fixed adjust value. A light estimation model810may estimate light information811corresponding to the background image801. The light estimation model810may be, for example, a neural network model including an encoder and a decoder. A rendering model820may render a virtual object image821based on the light information811, object information803, and plane information804. The light estimation model810may be trained based on a loss830corresponding to a difference between the reference image802and the virtual object image821.

Referring toFIG.8B, ISP information840may be additionally provided to the light estimation model810. For example, when a variable ISP setting value may not be readily determined, such as when a camera does not provide a parameter value for the variable ISP setting, input data of the light estimation model may be configured without the variable ISP setting value as illustrated inFIG.8A. Alternatively, when the variable ISP setting value may be determined, such as when the camera provides the parameter value for the variable ISP setting, the input data may be configured to include the variable ISP setting value. Here, the ISP information840may include the variable ISP setting value. The light estimation model810may generate the light information811based on input data that includes the ISP information840. The ISP information840, when available, may improve estimation accuracy of the light estimation model810.

FIG.9illustrates an example of rendering a virtual object image, according to one or more embodiments. Referring toFIG.9, a rendering model920may perform rendering based on further information such as simultaneous localization and mapping (SLAM) information901. A plurality of background images and a plurality of reference images (respectively paired) may be generated according to a change in a position and/or direction of a camera and a reference object, and the change in the position and/or direction may be recorded in the SLAM information901. For example, a first reference image and a first background image may be generated at a camera first position in a first direction, and a second reference image and a second background image may be generated at a camera second position in a second direction.

The SLAM information901may include stored positions-directions of the respective background-reference image pairs, and integrated light information corresponding to the light information904according to the respective positions and the directions may be estimated based on the SLAM information901. For example, the integrated light information corresponding to the light information904may be estimated through modification910using the SLAM information901, the object information902, and the plane information903.

The rendering model920may render a virtual object image905for each position-direction based on the integrated light information. For example, the rendering model may render a first virtual object image corresponding to the first position and the first direction and a second virtual object image corresponding to the second position and the second direction. The light estimation model may be trained based on a difference between the first reference image and the first virtual object image and a difference between the second reference image and the second virtual object image. More than two virtual-real image pairs of images may be used.

FIG.10illustrates an example of generating an AR image using a light estimation model, according to one or more embodiments. The generation ofFIG.10may be for actual use after training has been performed. Referring toFIG.10, a light estimation model1010may estimate light information1011corresponding to a background image1001. The background image1001may be generated based on a variable ISP setting. For example, the background image1001may be generated through a camera having a specification that is the same as a specification of a camera used to obtain a sample background image during training (the cameras need not be the same physical cameras; training can be performed in advance with another camera/device). The light estimation model1010may be in a state in which training is completed, and may therefore estimate the light information1011to be close to actual light. A rendering model1020may render a virtual object image1021based on the light information1011, object information1002, and plane information1003. The virtual object image1021may be represented such that it is in harmony with actual light captured by the background image1001. For example, the virtual object image1021may be represented in a high illuminance state (e.g., high overall brightness/intensity) in response to actual light corresponding to a high illuminance environment, and the virtual object image1021may be represented in a low illuminance state (e.g., low brightness/intensity) in response to the actual light corresponding to a low illuminance environment.

An ISP simulator1030may adjust at least a part of an ISP element of the virtual object image1021, as necessary, to enhance the consistency of the final AR image1050containing the virtual object image1021with the display environment. The ISP simulator1030may operate in different ways according to a characteristic or type of a display device on which an AR image1050is displayed. For example, in response to the AR image1050being displayed on an opaque type of display (e.g., a display of a smartphone, a table personal computer (PC), a vehicle display, etc.), the ISP simulator1030may control a variable ISP setting of a simulated ISP element that is used to modify/generate the virtual object image1021. The simulated ISP element may be any of the types of camera ISP elements described above. The variable ISP setting may be used to adjust an ISP setting used to modify/generate the background image1001, and the ISP setting may be provided via a camera that generates the background image1001(or, from a setting outside the camera that is used to set the camera's ISP setting). Conversely, in response to the AR image1050being displayed on a translucent display (e.g., a display of AR glasses, etc.), the virtual object image1021may be used to generate the AR image1050without controlling the simulated ISP element based on the variable ISP setting (or with a setting thereof that corresponds to a translucent display).

The AR image1050may be generated by combining an adjustment result of the ISP simulator1030and the background image1001. In case no adjustment by the ISP simulator1300is performed, the AR image1050may be generated by synthesizing (e.g., overlaying the virtual object image1021on the background image1001) the virtual object image1021and the background image1001. Any given implementation may or may not include an ISP simulator.

FIGS.11A and11Billustrate examples of methods of estimating light, according to one or more embodiments. Referring toFIG.11A, in operation1110, a rendering device may obtain a reference image generated based on raw image data acquired by capturing a reference object and a first ISP setting corresponding thereto. In operation1120, the rendering device may obtain a background image generated based on raw image data acquired by capturing a real background in which the reference object is positioned and a second ISP setting corresponding thereto. The first ISP setting may a fixed adjust value that controls an ISP element, and the second ISP setting may a variable adjust value that controls the ISP element. The ISP element may be one or more of auto a white balance element, an auto exposure element, a gamma correction element, a dynamic range compression element, and/or a wide dynamic range element.

The reference object may include sub-objects, each of which has a tone or a material different from the other, and planes supporting the sub-objects. The sub-objects may be represented by respective image portions having different dynamic ranges in raw image data acquired by capturing the reference object, and an image portion having a widest dynamic range (among the image portions of the respective sub-objects) may be selectively used in determining a difference between the reference image and the virtual object image.

The reference object may be connected to a camera that generates the reference image and the background image through a support. The camera may include a first camera that generates the reference image using the first ISP setting, and a second camera that generates the background image using the second ISP setting. Alternatively, the reference image and the background image may be generated through a camera that alternately uses the first ISP setting and the second ISP setting.

In operation1130, the rendering device may estimate light information corresponding to the background image using a light estimation model. Input data corresponding to the background image and the second ISP setting may be input to the light estimation model. In operation1140, the rendering device may render a virtual object image corresponding to the light information and the reference object. In operation1150, the rendering device may train the light estimation model based on a difference between the reference image and the virtual object image. Operation1140may include rendering a shadow of the virtual object image, and operation1150may include training the light estimation model based on a difference between a shadow of the reference object in the reference image and a shadow rendered in the virtual object image.

Multiple reference images and background images may be used. A first reference image and a first background image may be obtained at a first position in a first direction, and a second reference image and a second background image may be obtained at a second position in a second direction. Operation1130may include estimating integrated light information corresponding to an integrated background image in which the first background image and the second background image are combined according to SLAM information based on the first position, the second position, the first direction, and the second direction. Operation1140may include rendering a first virtual object image corresponding to the first position and the first direction and a second virtual object image corresponding to the second position and the second direction, and operation1150may include training the light estimation model based on at least some of a difference between the first reference image and the first virtual object image and a difference between the second reference image and the second virtual object image.

Referring toFIG.11B, in operation1160, the rendering device may obtain the background image generated based on raw image data acquired by capturing the real background on which the virtual object is to be displayed and the second ISP setting. In operation1170, the rendering device may estimate the light information corresponding to the background image and the first ISP setting using the light estimation model. The light estimation model may be in a state in which training is complete. That is, the example method ofFIG.11Amay involve training a light estimation model, and the example method ofFIG.11Bmay involve using the trained model to render virtual object images using the trained light estimation model.

The light estimation model may be pre-trained through operations of obtaining a sample reference image generated based on raw image data acquired by capturing the reference object and the first ISP setting, obtaining a sample background image generated based on raw image data acquired by capturing a sample real background in which the reference object is positioned and the second ISP setting, estimating sample light information corresponding to the sample background image using the light estimation model, rendering a sample virtual object image corresponding to the sample light information and the reference object, and training the light estimation model based on a difference between the sample reference image and the sample virtual object image. These training operations may correspond to operations (operations1110to1150) illustrated inFIG.11A. To distinguish training data ofFIG.11Afrom inference data ofFIG.11B, the term “sample” may be used to identify the training data as such. For example, a reference image and a background image used for training may be respectively referred to as a sample reference image and a sample background image.

In operation1180, the rendering device may render the virtual object image corresponding to the light information and the virtual object. In operation1190, the rendering device may generate an AR image (e.g., a frame of a video sequence generated by repetition of the steps ofFIG.11B) according to the virtual object and the background image. When the AR image is being to be displayed on an opaque display, a simulated ISP element with the second ISP setting may be used to generate the AR image. When the AR image is to be displayed on a translucent display, a simulated ISP element may be omitted (or may a simulated ISP element with a corresponding setting may be used) for the AR image.

In addition, the description provided with reference toFIGS.1to10,12, and13may apply to the method of estimating light.

FIG.12illustrates an example of a configuration of a rendering device, according to one or more embodiments. Referring toFIG.12, a rendering device1200may include a processor1210and a memory1220. The memory1220is connected to the processor1210and stores instructions executable by the processor1210, data to be operated by the processor1210, or data processed by the processor1219. The instructions may be readily configured based on the description herein (e.g., by compiling source code corresponding to methods described herein). The memory1220may include non-transitory computer-readable media, for example, high-speed random access memory and/or non-volatile computer-readable storage media, such as, for example, at least one disk storage device, flash memory device, or other non-volatile solid state memory device.

The processor1210may execute the instructions to perform the operations ofFIGS.1to11and13. The processor1210may obtain a reference image generated based on raw image data acquired by capturing a reference object and a first ISP setting, obtain a background image generated based on raw image data acquired by capturing a real background in which the reference object is positioned and a second ISP setting, estimate light information corresponding to the background image using a light estimation model, render a virtual object image corresponding to the light information and the reference object, and train the light estimation model based on a difference between the reference image and the virtual object image. The processor1210may obtain a background image generated based on raw image data acquired by capturing the real background on which a virtual object is to be displayed and the second ISP setting, estimate light information corresponding to the background image and the first ISP setting using the light estimation model, render a virtual object image corresponding to the light information and the virtual object, and generate an AR image according to the virtual object image and the background image. In addition, the description provided with reference toFIGS.1to11and13may apply to the rendering device1200.

FIG.13illustrates an example of a configuration of an electronic device, according to one or more embodiments. The electronic device may be, or may include, the rendering device1200. Referring toFIG.13, an electronic device1300may include a processor1310, a memory1320, a camera1330, a storage device1340, an input device1350, an output device1360, and a network interface1370that may communicate with each other through a communication bus1380. For example, the electronic device1300may be implemented as, or a portion of, for example, a mobile device such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, and the like, a wearable device such as a smart watch, a smart band, smart glasses, and the like, a home appliance such as a television (TV), a smart TV, a refrigerator, and the like, a security device such as a door lock, a vehicle such as an autonomous vehicle, a smart vehicle, and an AR device such as AR glasses. Again, the electronic device1300may be, or may include, structurally and/or functionally, the rendering device1200ofFIG.12.

The processor1310may execute functions and instructions for execution in the electronic device1300. For example, the processor1310may process instructions stored in the memory1320or the storage device1340. The processor1310may perform the operations described throughFIGS.1to12. The memory1320may include computer-readable storage media or a computer-readable storage device. The memory1320may store instructions to be executed by the processor1310and may store related information while software and/or an application is being executed by the electronic device1300. The processor1310may be a variety of types of processor(s), for example graphics processing unit(s), central processing Links), or the like.

The camera1330may capture a photo and/or a video. The camera1330may generate a reference image based on raw image data acquired by capturing a reference object and a fixed ISP setting, and generate a background image based on raw image data acquired by capturing a real background in which the reference object is positioned and a variable ISP setting. The storage device1340may include a computer-readable storage medium or computer-readable storage device. The storage device1340may store a larger quantity of information than the memory1320for a long time. For example, the storage device1340may include a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or other types of non-volatile memory known in the art.

The input device1350may receive an input from the user through traditional input manners, such as a keyboard and a mouse, and through newer input manners such as touch, voice, and an image. For example, the input device1350may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects the input from the user and transmits the detected input to the electronic device1300. The output device1360may provide an output of the electronic device1300to the user through a visual, auditory, or haptic channel. The output device1360may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. The network interface1370may communicate with an external device through a wired or wireless network.

The computing apparatuses, the vehicles, the electronic devices, the processors, the memories, the image sensors, the vehicle/operation function hardware, the displays, the information output system and hardware, the storage devices, and other apparatuses, devices, units, modules, and components described herein with respect toFIGS.1-13are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtracters, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SSD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated inFIGS.1-13that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above implementing instructions or software to perform the operations described in this application that are performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation, Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.

Therefore, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.