Patent Description:
With a view to providing an enhanced user experience, development is being actively made on an electronic device capable of providing an augmented reality (AR) service to display computer-generated information in association with an object in the real-world. Such an electronic device may be a wearable device that may be worn by a user. For example, the electronic device may be AR glasses.

<CIT> discloses, according to its abstract, a method and system for enhancing visual perception of augmented reality presentation. The location and line-of-sight (LOS) of a user wearing a see-through head-mounted display (HMD) is detected. A future background environment to be viewed by the user through the HMD is predicted based on at least the detected location and detected LOS. At least one color-attribute of the future background environment at a background location corresponding to a user LOS is predicted. The predicted color-attribute is compared with at least one color-attribute of an intended foreground supplementary image to be projected on the HMD overlaid onto the future background environment at the background location. When an incompatibility condition is detected, at least one visual parameter of the supplementary image is adjusted to minimize the incompatibility condition, and the supplementary image is projected on the HMD at the background location with the adjusted visual parameter.

<CIT> discloses, according to its abstract, a display device configured to present a virtual field of view comprising a virtual object superimposed on a real-world scene. A controller may be provided and configured to modify display of the virtual object by the display device based on a first color of a real-world object in the real-world scene. The virtual object in the virtual field of view may be superimposed over the real-world object. A lumen output of the display device for displaying the virtual object may be decreased.

The scope of the present invention is determined according to the independent claims. Various embodiments of the present invention are outlined in the dependent claims.

According to an aspect of the present disclosure, a wearable device is described. The wearable device comprises at least one camera; a display; a memory configured to store instructions; and a processor. The processor is, when the instructions are executed, configured to: receive an input for displaying multimedia content in a display area of the display; based on receiving the input, identify whether brightness of an environment around the wearable device is greater than or equal to reference brightness; based on identifying that the brightness is greater than or equal to the reference brightness, identify whether the multimedia content includes at least one area having specified color; based on identifying that the multimedia content includes the at least one area, obtain a first image for a portion of the environment corresponding to a position in which the multimedia content is to be displayed, via the at least one camera; obtain a second image in which color of the first image is converted; and display, via the display, the multimedia content, as superimposed on the second image displayed in the position.

According to another aspect of the present disclosure, a method for operating a wearable device comprising at least one camera and a display is described. The method comprises: receiving an input for displaying multimedia content in a display area of the display; based on receiving the input, identifying whether brightness of an environment around the wearable device is greater than or equal to reference brightness; based on identifying that the brightness is greater than or equal to the reference brightness, identifying whether the multimedia content includes at least one area having specified color; based on identifying that the multimedia content includes the at least one area, obtaining a first image for a portion of the environment corresponding to a position in which the multimedia content is to be displayed, via the at least one camera; obtaining a second image in which color of the first image is converted; and displaying, via the display, the multimedia content, as superimposed on the second image displayed in the position.

Also disclosed is a non-transitory computer readable storage medium storing therein at least one program. The at least one program comprises instructions to cause, when executed by at least one processor of a wearable device including at least one camera and a display, the wearable device to receive an input for displaying multimedia content in a display area of the display. The at least one program may comprise instructions to cause, when executed by the at least one processor of the wearable device, the wearable device to, based on the reception, identify whether brightness of an environment around the wearable device is greater than or equal to reference brightness. The at least one program may comprise instructions to cause, when executed by the at least one processor of the wearable device, the wearable device to, based on identifying that the brightness is greater than or equal to the reference brightness, identify whether the multimedia content includes at least one area having specified color or not. The at least one program may comprise instructions to cause, when executed by the at least one processor of the wearable device, the wearable device to, based on identifying that the multimedia content includes the at least one area, obtain a first image for a portion of the environment corresponding to a position in which the multimedia content is to be displayed, via the at least one camera. The at least one program may comprise instructions to cause, when executed by the at least one processor of the wearable device, the wearable device to obtain a second image in which color of the first image is converted. The at least one program may comprise instructions to cause, when executed by the at least one processor of the wearable device, the wearable device to display, via the display, the multimedia content, as superimposed on the second image displayed in the position.

<FIG> shows an exemplary environment including a wearable device.

Referring to <FIG>, the wearable device <NUM> may be included in an environment <NUM>.

The wearable device <NUM> may be used to provide an augmented reality (AR) service. In order to provide the augmented reality service, the wearable device <NUM> may include at least one transparent display. Since the at least one transparent display is configured to transmit external light directed to a first surface of the at least one transparent display through a second surface of the at least one transparent display, the at least one transparent display may display a virtual object together with an external object (for example, a physical object) within the real-world. Throughout the present disclosure, the virtual object may be referred to as a visual object in terms of being viewable by a user. In an embodiment, in order to provide the augmented reality service, the wearable device <NUM> may include a camera used to recognize the external object, another camera used to track the eyes of the user wearing the wearable device <NUM>, or a combination thereof. In an embodiment, in order to provide the augmented reality service, the wearable device <NUM> may include a communication circuit. The communication circuit may be used to obtain information on the external object from an external electronic device (e.g., a server or a smartphone), or may be used to obtain information for displaying the virtual object from an external electronic device.

In an embodiment, the wearable device <NUM> within the environment <NUM> may receive a user input to control a screen (e.g., multimedia content) displayed on the display of the wearable device <NUM>. Since the screen is displayed along with an external object viewed within a display area <NUM> of the display, the user input may be defined as another input distinguished from a touch input to the display. For example, the user input may be a gesture input caused by a part of the user's body wearing the wearable device <NUM> or an eye gaze input caused by a gaze of the user wearing the wearable device <NUM>. However, the present disclosure is not limited thereto.

<FIG> is a simplified block diagram of a wearable device. The functional components indicated by the block diagram may be incorporated into the wearable device <NUM> illustrated in <FIG>.

Referring to <FIG>, the wearable device <NUM> includes a processor <NUM>, a memory <NUM>, a camera <NUM>, and a display <NUM>. The wearable device <NUM> may include a communication circuit <NUM>.

The processor <NUM> may control the overall operations of the wearable device <NUM>. For example, the processor <NUM> may write data to the memory <NUM> and read out data recorded in the memory <NUM>. For example, the processor <NUM> may obtain an image via the camera <NUM>. For example, the processor <NUM> may transmit a signal to or receive a signal from another electronic device via the communication circuit <NUM>. For example, the processor <NUM> may display information through the display <NUM>. According to various embodiments, the processor <NUM> may include multiple processors (for example, the wearable device <NUM> may comprise at least one processor). For example, the processor <NUM> may include an application processor (AP) to control an upper layer such as e.g., an application program, a communication processor (CP) to control communication, and a display controller (e.g., display driving integrated circuitry) to control a screen displayed on the display <NUM> and the like.

The processor <NUM> may be configured to implement the procedures and/or methods proposed in the present disclosure.

The memory <NUM> may store instructions, commands, control command codes, control data, or user data for controlling the wearable device <NUM>. For example, the memory <NUM> may store a software application, an operating system (OS), middleware, and/or a device driver.

The memory <NUM> may include one or more of volatile memory or non-volatile memory. The volatile memory may include, for example, a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FeRAM), and the like. The non-volatile memory may include a read only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programable ROM (EEPROM), a flash memory and the like.

The memory <NUM> may further include a non-volatile storage media such as e.g., a hard disk drive (HDD), a solid-state disk (SSD), an embedded multimedia card (eMMC), a universal flash storage (UFS), and so on.

The memory <NUM> may be operably or operatively coupled with the processor <NUM>. The memory <NUM> may store one or more programs. For example, the one or more programs may include, when executed by the processor <NUM> of the wearable device <NUM>, instructions that cause the wearable device <NUM> to execute at least a portion of the operations of the wearable device <NUM> exemplified through the following descriptions.

For example, the one or more programs may be obtained from an external electronic device (e.g., a server or a smartphone). For example, the one or more programs stored in a non-volatile memory of the external electronic device may be provided from the external electronic device to the wearable device <NUM>, in response to an input to the wearable device <NUM>. For example, the one or more programs stored in the non-volatile memory of the external electronic device may be provided from the external electronic device to the wearable device <NUM>, in response to an input to the external electronic device. However, the present disclosure is not limited thereto.

The camera <NUM> may be used to obtain an image of the environment viewed within a display area of the display <NUM> (e.g., the display area <NUM> shown in <FIG>). The camera <NUM> may be disposed to face the environment to obtain an image of the environment viewed within the display area of the display <NUM>. The field of view (FOV) of the camera <NUM> disposed to face the environment may be configured to cover an area wider than the display area of the display <NUM>, or to cover an area corresponding to the display area of the display <NUM>, in order to obtain an image of the environment viewed within the display area of the display <NUM>. The field of view of the camera <NUM> disposed to face the environment may be configured to cover an area wider than the area covered by the field of view of the user wearing the wearable device <NUM>, or to cover an area corresponding to the area covered by the user's field of view, in order to obtain an image of the environment viewed within the display area of the display <NUM>. The camera <NUM> disposed to face the environment may include a plurality of cameras to obtain an image of the environment viewed within the display area of the display <NUM>. In an embodiment, the plurality of cameras may be configured with a pair of cameras to obtain a stereoscopic image. A direction in which a first camera included in the pair of cameras faces may be the same as a direction in which a second camera included in the pair of cameras faces, in order to obtain a stereoscopic image. In order to obtain such a stereoscopic image, the field of view of the first camera and the field of view of the second camera may have disparity. In an embodiment, the camera <NUM> disposed to face the environment may include a component to obtain depth information. However, the disclosure is not limited thereto.

The camera <NUM> may be further used to track the eyes of the user wearing the wearable device <NUM>. For example, the camera <NUM> may be disposed to face the user's eyes so that the field of view of the camera <NUM> covers an area including the user's eyes wearing the wearable device <NUM>. The camera used to track the eyes of the user may be different to the camera disposed to face the environment; for example, the camera <NUM> may include a plurality of cameras, with at least one camera used to track the eyes of the user and at least one camera used to obtain an image of the environment.

The camera <NUM> may be operably or operatively coupled with the processor <NUM>.

The communication circuit <NUM> may have a variety of communication functions (e.g., cellular communication, Bluetooth, NFC, Wi-Fi, etc.) for communication between the wearable device <NUM> and at least one external device (e.g., a smartphone, a server, etc.). In other words, the communication circuit <NUM> may establish communication between the wearable device <NUM> and the at least one external device.

The communication circuitry <NUM> may be operably or operatively coupled with processor <NUM>.

The display <NUM> may include at least one transparent display so that a user wearing the wearable device <NUM> can view the real-world. For example, the display <NUM> may be configured to cause external light directed to a first surface to go through a second surface different from the first surface, and configured to display information on the second surface. For example, the second surface may be opposite to the first surface. The display <NUM> may display a graphical user interface (GUI) so that the user can interact with the wearable device <NUM>. In certain embodiments, the display <NUM> may be partitioned into different areas or regions. In certain embodiments, the display <NUM> may comprise a plurality of displays.

The display <NUM> may be operably or operatively coupled with processor <NUM>.

In an embodiment, the processor <NUM> may display multimedia content on the display area of the display <NUM> along with an external object in the real world, viewed within the display area of the display <NUM> (e.g., the display area <NUM> shown in <FIG>). In an embodiment, the multimedia content may be displayed on the display area of the display <NUM>, based on data received from another electronic device through the communication circuit <NUM>. In an embodiment, the multimedia content may be displayed on the display area of the display <NUM>, independently (or irrespectively) of the other electronic device, based on data processed by the processor <NUM>. In an embodiment, the multimedia content may include information related to the external object viewed within the display area of the display <NUM>. When the multimedia content includes information related to the external object, the multimedia content may be displayed as associated with the external object. For example, when the multimedia content includes information related to the external object, the multimedia content may be displayed in proximity to the external object or as at least partially overlapped with the external object. In an embodiment, the multimedia content may include information independent from the external object viewed within the display area of the display <NUM>.

In an embodiment, the processor <NUM> may obtain recognition information about an external object in the real world viewed within the display area of the display <NUM>. The processor <NUM> may transmit information about an image including a visual object corresponding to the external object obtained through the camera <NUM>, to another electronic device (e.g., a smartphone, a server, etc.) through the communication circuit <NUM>, and obtain the recognition information on the external object from the other electronic device through the communication circuit <NUM>. The processor <NUM> may obtain the recognition information on the external object by recognizing the image including the visual object corresponding to the external object, in a stand-alone state. For example, the processor <NUM> may obtain the recognition information on the external object by recognizing the image including the visual object corresponding to the external object without use of the other electronic device. However, the present disclosure is not limited thereto.

<FIG> is a perspective view illustrating an exemplary wearable device. For example, the exemplary wearable device may be the wearable device <NUM> illustrated in <FIG>.

Referring to <FIG>, a frame <NUM> of the wearable device <NUM> may have a physical structure worn on a part of the user's body. For example, the frame <NUM> may be configured so that, when the wearable device <NUM> is worn by the user, a first display <NUM>-<NUM> (or a first display region <NUM>-<NUM>) in the display <NUM> is positioned in front of the user's right eye and a second display <NUM>-<NUM> (or a second display region <NUM>-<NUM>) in the display <NUM> is positioned in front of the user's left right eye. In certain embodiments, the first display <NUM>-<NUM> and the second display <NUM>-<NUM> are connected to provide a continuous display in front of the user's eyes (for example, with a display portion in front of the bridge of the user's nose to join the first display <NUM>-<NUM> and the second display <NUM>-<NUM>).

In an embodiment, the display <NUM> including the first display <NUM>-<NUM> and the second display <NUM>-<NUM> may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), a micro-LED, or the like. In an embodiment, when the display <NUM> is configured of LCD, DMD, or LCoS, the wearable device <NUM> may include a light source (not shown in <FIG>) emitting light toward the display area of the display <NUM>. In an embodiment, when the display <NUM> is formed of a OLED or a micro-LED, the wearable device <NUM> may not include the light source. However, the disclosure is not limited thereto.

In an embodiment, the wearable device <NUM> may further include a first transparent member <NUM>-<NUM> and a second transparent member <NUM>-<NUM>. For example, each of the first transparent member <NUM>-<NUM> and the second transparent member <NUM>-<NUM> may be formed of a glass plate, a plastic plate, or a polymer. For example, each of the first transparent member <NUM>-<NUM> and the second transparent member <NUM>-<NUM> may be transparent or translucent.

In an embodiment, the wearable device <NUM> may include a waveguide <NUM>. For example, the waveguide <NUM> may be used to transmit a light source generated by the display <NUM> to the eyes of a user wearing the wearable device <NUM>. For example, the waveguide <NUM> may be formed of glass, plastic, or polymer. For example, the waveguide <NUM> may include a nano-pattern configured with a polygonal or curved lattice structure in the waveguide <NUM> or on a surface of the waveguide <NUM>. For example, light incident to one end of the waveguide <NUM> may be transferred to the user through the nano-pattern. In an embodiment, the waveguide <NUM> may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE), etc.) or a reflective element (e.g., a reflective mirror). For example, the at least one diffractive element or the reflective element may be used to guide light to the user's eyes. In an embodiment, the at least one diffractive element may include an input optical member and/or an output optical member. In an embodiment, the input optical member may mean an input grating area used as an input terminal of light, and the output optical member may mean an output grating area used as an output terminal of light. In an embodiment, the reflective element may include a total internal reflection optical element or a total internal reflection waveguide for total internal reflection (TIR).

In an embodiment, the camera <NUM> in the wearable device <NUM> may include at least one first camera <NUM>-<NUM>, at least one second camera <NUM>-<NUM>, and/or at least one third camera <NUM>-<NUM>.

In an embodiment, the at least one first camera <NUM>-<NUM> may be used for motion recognition or spatial recognition of three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the at least one first camera <NUM>-<NUM> may be used for head tracking or hand detection. For example, the at least one first camera <NUM>-<NUM> may be configured with a global shutter (GS) camera. For example, the at least one first camera <NUM>-<NUM> may be configured with a stereo camera. For example, the at least one first camera <NUM>-<NUM> may be used for gesture recognition.

In an embodiment, the at least one second camera <NUM>-<NUM> may be used to detect and track a pupil. For example, the at least one second camera <NUM>-<NUM> may be configured with a GS camera. For example, the at least one second camera <NUM>-<NUM> may be used to identify a user input defined by a user's gaze.

In an embodiment, the at least one third camera <NUM>-<NUM> may be referred to as a high resolution (HR) or photo video (PV) camera, and provide an auto-focusing (AF) function or an optical image stabilization (OIS) function. In an embodiment, the at least one third camera <NUM>-<NUM> may be configured with a GS camera or a remote shutter (RS) camera.

In an embodiment, the wearable device <NUM> may further include an LED unit <NUM>. For example, the LED unit <NUM> may be used to assist in tracking the pupil through at least one second camera <NUM>-<NUM>. For example, the LED unit <NUM> may be configured with an infrared LED (IR LED). For example, the LED unit <NUM> may be used to compensate for brightness when the illuminance around the wearable device <NUM> is low.

In an embodiment, the wearable device <NUM> may further include a first PCB <NUM>-<NUM> and a second PCB <NUM>-<NUM>. For example, each of the first PCB <NUM>-<NUM> and the second PCB <NUM>-<NUM> may be used to transmit an electrical signal to components of the wearable device <NUM>, such as the camera <NUM> or the display <NUM>. In an embodiment, the wearable device <NUM> may further include an interposer disposed between the first PCB <NUM>-<NUM> and the second PCB <NUM>-<NUM>. However, the present disclosure is not limited thereto.

Such a wearable device may include a display (e.g., a transparent display) configured to transmit external light directed to a first surface through a second surface to provide an augmented reality service.

Meanwhile, multimedia content displayed via the display may include an area having a certain color (this may also be referred to as a predetermined, or predefined, or specific, or specified, or set etc. color). When displaying an area having the certain color via the display, the wearable device may express the area without any light emission of at least one light emitting element (or at least one light emitting device) for the area (where, for example, the at least one light emitting element may be included in the wearable device (for example, in the display thereof)). Since the area is expressed without any light emission of the at least one light emitting element, an external object may be visible through the area. Since the external object viewed (that is, being visible) through the area may deteriorate the quality of the multimedia content, a method for enhancing the displaying of the area may be required.

<FIG> is a flowchart illustrating a method of displaying multimedia content through a display of a wearable device. This method may be executed, for example, by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

<FIG> shows exemplary multimedia contents.

<FIG> shows exemplary inputs for displaying multimedia content.

<FIG> and <FIG> show exemplary characteristics of at least one area having a specified color included in multimedia content displayed via a display of a wearable device.

<FIG> shows example messages inquiring whether to display a second image for compensating at least one area.

<FIG> shows an example of a method of generating a second image.

<FIG> shows an example of a method of displaying a second image.

<FIG> shows an example of a method of displaying multimedia content with superposition on a second image.

<FIG> shows an example of a method of rendering a second image and multimedia content.

<FIG> shows an example of a method of adjusting opacity of multimedia content.

<FIG> shows an example of a method of displaying a reflective layer with superimposing on multimedia content superimposed on a second image.

Referring to <FIG>, in operation <NUM>, the processor <NUM> receives an input for displaying multimedia content.

The multimedia content may be configured with visual information. For example, the multimedia content may include at least one of an image including at least one visual object or at least one text. For example, referring to <FIG>, the multimedia content may be an image <NUM>. For example, the image <NUM> may include a static image or a dynamic image (e.g., video), or the image <NUM> may be included in a video corresponding to the multimedia content. In an embodiment, the image <NUM> may be obtained via communication with an external electronic device. In an embodiment, the image <NUM> may be obtained by processing of the processor <NUM> without communication with an external electronic device. However, the present disclosure is not limited thereto.

In an embodiment, the image <NUM> may be related to an external object within the environment viewed through the display <NUM>. For example, the processor <NUM> may obtain an image of the external object through the camera <NUM>, and obtain the image <NUM> related to the external object based on recognition of the image. For example, the image <NUM> may include description information or attribute information about the external object. In the meantime, the image recognition may be executed in the wearable device <NUM>, in an electronic device distinct from the wearable device <NUM>, or based on interworking in between the wearable device <NUM> and the electronic device. However, the present disclosure is not limited thereto. In an embodiment, the image <NUM> may be independent of the environment viewed through the display <NUM>.

The multimedia content may be an emoji graphic object <NUM>. For example, the emoji graphical object <NUM> may represent a user of the wearable device <NUM>. For example, the graphic emoji object <NUM> may have a shape set to suit the user's intention, according to manipulation. For example, the graphic emoji object <NUM> may have a shape set based on recognizing an image of the user of the wearable device <NUM>. For example, the graphical emoji object <NUM> may be obtained based on featuring points extracted from a visual object in the image corresponding to the user (or the user's face). For example, the graphic emoji object <NUM> may indicate a service provider presented through the wearable device <NUM>. However, the present disclosure is not limited thereto. In an embodiment, the emoji graphical object <NUM> may be configured with a two-dimensional (2D) visual object or a three-dimensional (3D) visual object. In an embodiment, the emoji graphical object <NUM> may be displayed as associated with an external object <NUM> (e.g., an air conditioner) within the environment viewed within the display area of the display <NUM>. For example, the emoji graphical object <NUM> may take a gesture indicating the external object <NUM>. For example, the emoji graphical object <NUM> may be positioned adjacent to the external object <NUM>. However, the present disclosure is not limited thereto. In an embodiment, the emoji graphical object <NUM> may be associated with visual information <NUM> derived from the emoji graphical object <NUM>. For example, the emoji graphic object <NUM> and the visual information <NUM> may be adjacent to each other or connected to each other. However, the present disclosure is not limited thereto. For example, the visual information <NUM> may include information about an external object (e.g., the external object <NUM>) that is identified by a user input and is viewed within the display of the display <NUM>, or include information on various functions executed according to control of the external object or under the control of the emoji graphic object <NUM>. However, the present disclosure is not limited thereto.

Referring again to <FIG>, the input may be defined in various ways. For example, referring to <FIG>, the input may be an input <NUM> for executing a software application used to display multimedia content. For example, in operation <NUM>, the processor <NUM> may receive the input <NUM> for an executable object <NUM> for the software application used to display the multimedia content. For example, the software application may be used to play the multimedia content via a video streaming service. For example, since executing the software application may mean that the multimedia content is to be displayed, the input received in operation <NUM> may include the input <NUM>.

For example, the input may be an input <NUM> for selecting one multimedia content from among a plurality of multimedia contents, displayed on a graphical user interface (GUI) of a software application. For example, in operation <NUM>, the processor <NUM> may receive the input <NUM> to select one executable object from among executable objects <NUM> for playing each of the plurality of multimedia contents displayed within the GUI <NUM>. For example, since selecting one executable object from among the plurality of executable objects <NUM> means displaying the multimedia content, the input received in operation <NUM> may include the input <NUM>. In an example, the GUI <NUM> may be displayed via the display <NUM> of the wearable device (for example, a head-up display (HUD) arrangement), and an executable object provided within the GUI may be selected through an input (for example, a gesture input) to select a multimedia content. In another example, the input may be an input received from an external electronic device, where a multimedia content has been selected at the external electronic device (for example, via a GUI provided by the external electronic device such as a smartphone) and the selection is communicated to the wearable device and received as the input for displaying the multimedia content.

For example, the input may be an input <NUM> for selecting an external object viewed within the display area of the display <NUM>. For example, in operation <NUM>, the processor <NUM> may receive the input <NUM> for selecting an external object <NUM>. For example, since selecting the external object <NUM> means that multimedia content related to the external object <NUM> is to be displayed, the input received in operation <NUM> may include the input <NUM>.

In certain embodiments, the input may be a gesture input (for example, for selecting an object viewed or displayed within the display area of the display <NUM>), a touch input (for example, for selecting a multimedia content displayed on a GUI of a software application which is output on a touchscreen), or a voice input (for example, where voice recognition is performed to identify an object or multimedia content indicated in a voice input).

Although not shown in <FIG>, for example, the input may be either identifying that a designated condition in the wearable device <NUM> in relation to the display of the multimedia content is satisfied or receiving a designated signal in relation to displaying of the multimedia content from another electronic device distinct from the wearable device <NUM>. However, the present disclosure is not limited thereto.

Referring back to <FIG>, in operation <NUM>, the processor <NUM> identifies whether brightness of the environment around the wearable device <NUM> is equal to or greater than reference brightness, based on receiving the input. In an embodiment, the environment may mean a scene viewed in the display area of the display <NUM>. In an embodiment, the environment may mean an area in which the multimedia content is to be displayed. However, the present disclosure is not limited thereto.

The brightness of the environment around the wearable device <NUM> may be identified using various methods.

In an embodiment, the processor <NUM> may obtain, through the camera <NUM>, an image for the environment viewed within the display area of the display <NUM>, based on receiving the input, and identify the brightness of the environment based on data indicating the brightness of the obtained image. For example, when the image is encoded based on a YUV attribute, a YUV format, or a YUV model, the data may be of luma data. However, the present disclosure is not limited thereto.

In an embodiment, the processor <NUM> may obtain sensing data via an illuminance sensor of the wearable device <NUM> based on receiving the input, and identify the brightness of the environment based on the obtained sensing data. In an embodiment, the processor <NUM> may identify the brightness of the environment based on the sensing data and the data indicating the brightness of the image.

Meanwhile, in an embodiment, the reference brightness may be set as a value to identify whether an external object is viewed through at least one area within the multimedia content having a specified color, which will be described later referring to operation <NUM>. In an embodiment, the reference brightness may be set as a value to identify whether external light having an intensity greater than or equal to a specified intensity through the at least one area is received by the eyes of the user of the wearable device <NUM>, which will be described later referring to operation <NUM>. However, the present disclosure is not limited thereto.

In an embodiment, the processor <NUM> may execute operation <NUM> on condition that the brightness is equal to or greater than the reference brightness, or execute operation <NUM> on condition that the brightness is less than the reference brightness. In an embodiment, operation <NUM> or operation <NUM> may be performed, in a more general sense, based on a brightness of the environment (that is, without specification of a reference brightness).

In other embodiments, the processor <NUM> may alternatively or additionally (to operation <NUM>) identify, or detect, or determine, whether a brightness of the multimedia content (for example, a brightness of any portion of a current image of a multimedia content) is less than or equal to another reference brightness, based on receiving the input. If so, the outcome is the same as if the outcome of operation <NUM> is positive; if not, the outcome is the same as if the outcome of operation <NUM> is negative. The another reference brightness may be set as a value to identify whether one or more areas within the multimedia content have the specified color. For example, the another reference brightness may be set as a value according to an identified brightness of the environment; for instance, the another reference brightness may be set to a higher value in a brighter environment than in a less-bright environment.

In operation <NUM>, the processor <NUM> identifies (or determines, detects etc.) whether the multimedia content includes the at least one area having the specified color, based on the identifying that the brightness is equal to or greater than the reference brightness.

The specified color may be a color expressed by the display <NUM> without light emission under the control of the processor <NUM>. For example, the specified color may be black. However, the present disclosure is not limited thereto. For example, while the multimedia content is displayed on the display <NUM>, at least one first light emitting element disposed for the at least one area having the specified color among the plurality of light emitting elements may be deactivated, as opposed to at least one second light emitting element disposed for the remaining area of the multimedia content having a color distinct from the specified color among the plurality of light emitting elements. For example, referring to <FIG>, multimedia content <NUM>, which is the multimedia content, may include at least one area <NUM> having the specified color (or substantially the same as the specified color) and a remaining area <NUM> having a color or colors distinct from the specified color. When the multimedia content <NUM> is displayed via the display <NUM>, the states of the plurality of light emitting elements in the display <NUM> may be expressed as in the state <NUM>. For example, in the state <NUM>, the at least one first light emitting element for at least one area <NUM> among the plurality of light emitting elements may not emit light, while the at least one second light emitting element for the remaining area <NUM> among the plurality of light emitting elements may emit light. Deactivating of the at least one first light emitting element or emitting no light of the at least one first light emitting element may reduce the quality of the multimedia content <NUM> displayed via the display <NUM>. For example, the multimedia content <NUM> may be displayed together with the environment <NUM>, which is an environment viewed within the display area of the display <NUM>. For example, the environment <NUM>, which is an environment including at least one external object viewed within the display area of the display <NUM>, may be an environment around the wearable device <NUM>. For example, the multimedia content <NUM> may be displayed via the display <NUM> in a state that the environment <NUM> is viewed in the display area of the display <NUM>. For example, the multimedia content <NUM> may be displayed as superimposed on at least a portion of the environment <NUM> viewed within the display area of display <NUM>. The multimedia content <NUM> superimposed on the at least a portion of the environment <NUM> viewed within the display area of the display <NUM> may be provided via the display <NUM>, as in the display state <NUM>. For example, within the display state <NUM>, at least one area <NUM> in the environment <NUM> may be viewed through the at least one area <NUM>. Since the at least one area <NUM> in the environment <NUM> viewed through the at least one area <NUM> is disparate or heterogeneous with respect to the multimedia content <NUM>, the quality of the multimedia content <NUM> displayed via the display <NUM> may deteriorate by/in the at least one area <NUM>.

For example, referring to <FIG>, as the display <NUM> is configured to transmit external light directed to a first surface of the display <NUM> through a second surface of the display <NUM> (that is, the external light may pass through the second surface to the first surface) and the at least one first light emitting element for the at least one area <NUM> having the specified color is deactivated (or the at least one first light emitting element is otherwise not emitting light) while displaying the multimedia content <NUM>, an intensity of light reaching the user's eyes through the at least one area <NUM> may be greater than or equal to a reference intensity, as in state <NUM>. For example, the reference intensity may be an intensity of light that can be viewed by the user of the wearable device <NUM>. Meanwhile, although not shown in <FIG>, in the state <NUM>, the intensity of light reaching the user's eyes through the remaining area <NUM> may be less than the reference intensity.

Referring back to <FIG>, the processor <NUM> may execute operation <NUM> on condition that the multimedia content includes the at least one area, or execute operation <NUM> on condition that the multimedia content does not include the at least one area.

In operation <NUM>, the processor <NUM> obtains a first image for at least a portion of the environment viewed within the display area of the display <NUM>, via the camera <NUM>, based on the identification that the multimedia content includes the at least one area. In an embodiment, the processor <NUM> may obtain the first image of a portion of the environment corresponding to a position in the display area where the multimedia content is to be displayed, based on the identification. In an embodiment, processor <NUM> may obtain the first image for at least one portion of the environment corresponding to the at least one area of the multimedia content; for example, with the multimedia content displayed, via the display <NUM>, as superimposed over the environment, or a part thereof, the at least one portion of the environment may correspond to a portion(s) of the environment over which the at least one area of the multimedia content is superimposed.

In an embodiment, operation <NUM> may be executed based on a user input. For example, referring to <FIG>, the processor <NUM> may display a message <NUM> via the display <NUM>, based on the identification that the multimedia content includes the at least one area. For example, the message <NUM> may be displayed to identify whether to execute operations <NUM> to <NUM>. For example, the message <NUM> may include a text to enquire as to whether to generate a second image (e.g., compensation image) in operation <NUM>. For example, the message <NUM> may include an executable object <NUM> to indicate executing operations <NUM> to <NUM>, and an executable object <NUM> to indicate refraining from executing operations <NUM> to <NUM> (herein, to refrain from performing or executing an operation may be interpreted to mean to not perform or execute the operation, to refrain from executing an operation means that the operation is not executed). For example, the processor <NUM> may execute operation <NUM> based on receiving a user input <NUM> for the executable object <NUM>. In an embodiment, the message <NUM> may be displayed based on the identification in operation <NUM> that the brightness is greater than or equal to the reference brightness. When the message <NUM> is displayed based on the identification that the brightness is equal to or greater than the reference brightness, the processor <NUM> may execute operation <NUM> or refrain from executing operation <NUM>, according to a user input for the message <NUM>.

For example, the processor <NUM> may display a message <NUM> via the display <NUM> based on the identification that the multimedia content includes the at least one area. For example, the message <NUM> may be displayed to identify whether to execute operations <NUM> to <NUM>. For example, the message <NUM> may include a text to enquire as to whether to generate a second image (e.g., compensation image) in operation <NUM>. For example, the message <NUM> may include an executable object <NUM> to indicate executing operations <NUM> to <NUM> and an executable object <NUM> to indicate refraining from executing operations <NUM> to <NUM>. For example, the processor <NUM> may execute operation <NUM> based on receiving a user input <NUM> for the executable object <NUM>.

The present disclosure is not limited to operation <NUM> being executed based on a user input. For example, operation <NUM> may be performed automatically, for instance in response to identifying that the multimedia content includes at least one area having the specific color.

Referring back to <FIG>, the first image may include a scene corresponding to the environment <NUM> illustrated in <FIG>. For example, referring to <FIG>, the image <NUM>, which is the first image, may include visual objects respectively corresponding to external objects within the environment <NUM> illustrated in <FIG>.

Referring back to <FIG>, in operation <NUM>, the processor <NUM> obtains a second image in which color/colors of the first image is/are converted (or inverted - in the following, converting a color/colors may include inverting a color/colors), based on obtaining the first image. For example, in response to obtaining the first image, the processor <NUM> may obtain the second image in which the color of the first image is converted by processing the first image. For example, referring to <FIG>, the image <NUM>, which is the second image, may be an image in which the color of the image <NUM> is converted or inverted. For example, an area <NUM> in the image <NUM> corresponding to an area <NUM> in the image <NUM> configured with black may be configured with white; that is, the color of the area <NUM> in the image <NUM> is inverted in comparison to the color of the area <NUM> in the image <NUM>.

Referring back to <FIG>, in operation <NUM>, the processor <NUM> displays the multimedia content as superimposed on the second image, based on obtaining the second image. In an embodiment, the second image may be displayed at a position identified based on the position of the field of view of the camera <NUM> when obtaining the first image. For example, the second image may be displayed at the position in which at least one external object located in the field of view of the camera <NUM> when obtaining the first image is fully overlapped with at least one visual object within the second image corresponding to the at least one external object. For example, the second image displayed at the position may be aligned with the portion of the environment corresponding to a scene at the time of obtaining the first image. For example, the second image may be displayed superimposed over the portion of the environment corresponding to the scene at the time of obtaining the first image. For example, the processor <NUM> may provide a background layer having a reference color via the display <NUM>, by displaying the second image aligned with the portion of the environment where, in an example, the colors of the second image are inverted compared to the colors of the first image. For example, referring to <FIG>, the processor <NUM> may display the image <NUM> such that the image <NUM> is aligned with the environment <NUM>. Since the image <NUM> is aligned with the environment <NUM> and the environment <NUM> is viewed through the display <NUM>, the background layer <NUM> having the reference color may be provided via the display <NUM>. In an embodiment, peripheries <NUM> of the background layer <NUM> may have a blur effect. For example, before displaying the image <NUM> via the display <NUM> following obtaining the image <NUM>, the processor <NUM> may apply the blur effect to the peripheries <NUM> of the image <NUM>. Such a blur effect applied to the peripheries of the image <NUM> may be provided at the peripheries <NUM> of the background layer <NUM>. For example, the blur effect may be provided at the peripheries <NUM> for harmony in between the multimedia content superimposed on the background layer <NUM> and an external object viewed through an area around the multimedia content. For example, the blur effect may be provided at the peripheries <NUM> to minimize or prevent the background layer <NUM> from being viewed when the multimedia content is superimposed on the background layer <NUM>. However, the present disclosure is not limited thereto.

Referring back to <FIG>, the multimedia content may be superimposed on the second image arranged to provide a background layer such as the background layer <NUM>. For example, since the multimedia content superimposed on the second image is displayed on the background layer formed based on the second image, the intensity of an external light reaching the user's eyes through the at least one area in the multimedia content may be less than the reference intensity. For example, since the multimedia content superimposed on the second image is displayed on the background layer formed based on the second image, at least one area <NUM> of the environment <NUM> may not be visible through the at least one area within the multimedia content. For example, referring to <FIG>, the processor <NUM> may display the multimedia content <NUM> as superimposed on the background layer <NUM> provided by the display of the second image <NUM>, based on controlling the display <NUM> as in a state <NUM>. The multimedia content <NUM> displayed as superimposed on the background layer <NUM> may be provided via the display <NUM>, as in a display state <NUM>. For example, in the display state <NUM>, at least one area <NUM> in the environment <NUM> may not be visible through the at least one area <NUM>, as opposed to at least one area <NUM> in the environment <NUM> in the display state <NUM> shown in <FIG>. For example, referring to <FIG>, when displaying the multimedia content <NUM> with superimposing on the background layer <NUM> using the display <NUM> controlled as in the state <NUM>, the background layer <NUM> may cause at least a portion of the external light to be reflected, absorbed, or blocked, so the intensity of light reaching the user's eyes through the at least one area <NUM> may be less than the reference intensity as in state <NUM>. In other words, the processor <NUM> can enhance the quality of the multimedia content displayed via the display <NUM>, by displaying the multimedia content as superimposed on the second image.

In an embodiment, a color of the background layer <NUM> may be changed according to the color of the environment <NUM> or the color of the at least one area <NUM> of the environment <NUM>. For example, the processor <NUM> may identify the color of the environment <NUM> or a color of the at least one area <NUM> of the environment <NUM> at a designated time interval, and change, based on the identified color, the color of the background layer <NUM>. However, the present disclosure is not limited thereto.

Referring back to <FIG>, displaying the multimedia content as superimposed on the second image may be executed based on rendering the second image and the multimedia content on a virtual plane(s) located in a virtual 3D space. For example, referring to <FIG>, the processor <NUM> may convert the first image, which is a 2D (two-dimensional) image, to a 3D image (or stereoscopic image) <NUM>, based on obtaining the first image. For example, when the camera <NUM> includes a plurality of cameras defined through the description of <FIG> or <FIG> to obtain a 3D image, the processor <NUM> may respectively obtain first images via the multiplicity of cameras and obtain a 3D image <NUM> based on the first images. For example, the processor <NUM> may obtain depth information when obtaining the first images, and obtain the 3D image <NUM> converted from the first image based on the depth information. However, the present disclosure is not limited thereto. Meanwhile, the processor <NUM> may identify a position of a first virtual plane <NUM> based on the position of the 3D image <NUM> in a virtual 3D space, and render the second image <NUM> on the first virtual plane <NUM>. Meanwhile, the processor <NUM> may render the multimedia content <NUM> on a second virtual plane <NUM> (for example, a position of the second virtual plane <NUM> may be based on the position of the first virtual plane <NUM>). The processor <NUM> may display the multimedia content <NUM> superimposed on the second image <NUM>, by projecting the second image <NUM> rendered on the first virtual plane <NUM> and the multimedia content <NUM> rendered on the second virtual plane <NUM> onto a third virtual plane <NUM> on/in the virtual 3D space corresponding to the display area of the display <NUM> (for example, the position of the third virtual plane <NUM> may be based on the position of the first virtual plane <NUM> and/or the position of the second virtual plane <NUM>). As a result of this, for example, only the third virtual plane <NUM> may be viewable by a user of the wearable device, allowing for viewing of the multimedia content <NUM> superimposed over the second image <NUM> which, in turn, is superimposed over (or displayed on) a portion of the environment corresponding to the position in which the multimedia content <NUM> is to be displayed. However, the present disclosure is not limited thereto.

For example, displaying the multimedia content as superimposed on the second image may be executed by building up the second image <NUM> and the multimedia content <NUM> on different virtual layers, or virtual planes, in a virtual 3D space. In certain examples, the position of the different virtual layers relative to one another is based on a size of the multimedia content <NUM> as displayed via the display <NUM> (e.g., a size of the displayed multimedia content <NUM> in the FOV of a user wearing the wearable device), and/or a position of the displayed multimedia content <NUM> on the display <NUM> (e.g., the position of the displayed multimedia content <NUM> in the FOV of a user wearing the wearable device).

Meanwhile, referring back to <FIG>, the processor <NUM> may adjust opacity of the multimedia content before displaying the multimedia content superimposed on the second image. For example, referring to <FIG>, the processor <NUM> may obtain the multimedia content <NUM> by adjusting the opacity of the multimedia content <NUM>. For example, the opacity of the multimedia content <NUM> may be greater than the opacity of the multimedia content <NUM> (i.e., increased opacity). After obtaining the multimedia content <NUM> with adjusted opacity, the processor <NUM> may display the multimedia content <NUM> superimposed on the second image.

Meanwhile, referring back to <FIG>, the processor <NUM> may further display a reflective layer superimposed on the multimedia content superimposed on the second image. For example, the reflective layer may include blue or purple color with the properties that are robust to light reflection. For example, referring to <FIG>, the processor <NUM> may further display a reflective layer <NUM>, as superimposed on the multimedia content <NUM>, which is superimposed on the background layer <NUM> and displayed using the display <NUM> controlled as in the state <NUM>. For example, the multimedia content <NUM> between the background layer <NUM> and the reflective layer <NUM> may be provided as in a display state <NUM>. For example, the reflective layer <NUM> may be partially transparent to allow for the multimedia content <NUM> to be viewable through the reflective layer <NUM>.

Meanwhile, in operation <NUM>, the processor <NUM> may display the multimedia content based on identifying that the brightness is less than the reference brightness. For example, the processor <NUM> may refrain from executing the operation <NUM> and display the multimedia content. For example, the processor <NUM> may display the multimedia content without displaying the second image.

Meanwhile, in operation <NUM>, the processor <NUM> may display the multimedia content based on identifying that the multimedia content does not include the at least one area. For example, the processor <NUM> may refrain from executing the operations <NUM> to <NUM> and display the multimedia content. For example, the processor <NUM> may display the multimedia content without displaying the second image.

Although the foregoing description in relation to <FIG> includes description of an example in which the processor <NUM> in the wearable device <NUM> executes the operations <NUM> to <NUM>, this is only for convenience of description. At least some of those operations <NUM> to <NUM> in <FIG> may be executed by an external electronic device (e.g., smart phone, tablet, personal computer (PC), server, etc.) connected to the wearable device <NUM>. However, the present disclosure is not limited thereto.

As described above, the wearable device <NUM> can prevent, alleviate or minimize at least a portion of the external environment from being viewed through the displayed multimedia content, by displaying the multimedia content as superimposed on the second image.

<FIG> is a flowchart illustrating a method of identifying whether to display multimedia content with superimposing on a second image based on the size of at least one area. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

Referring to <FIG>, in operation <NUM>, the processor <NUM> identifies that the multimedia content includes the at least one area having the specified color. For example, the processor <NUM> may identify that the multimedia content includes the at least one area according to the identification in operation <NUM>.

In operation <NUM>, the processor <NUM> may identify whether a ratio of the size of the at least one area to the size of the multimedia content is equal to or greater than a reference value, based on the identification. For example, when the size of the at least one area is relatively small, the decrease in quality of the multimedia content owing to the at least one area may be relatively small, so the processor <NUM> may identify whether the ratio is equal to or greater than the reference value. For example, the processor <NUM> may execute operation <NUM> to reduce resource consumption of the wearable device <NUM> by the execution of operation <NUM> and operation <NUM>.

The processor <NUM> may execute operation <NUM> on condition that the ratio is equal to or greater than the reference value, or execute operation <NUM> on condition that the ratio is less than the reference value.

In operation <NUM>, the processor <NUM> may display the multimedia content as superimposed on the second image, based on identifying that the ratio is equal to or greater than the reference value. For example, the processor <NUM> may obtain the second image by executing operations <NUM> and <NUM> based on identifying that the ratio is greater than or equal to the reference value, and display the multimedia content as superimposed on the second image.

In operation <NUM>, the processor <NUM> may display the multimedia content based on identifying that the ratio is less than the reference value. For example, the processor <NUM> may display the multimedia content without displaying the second image.

As described above, through the execution of operation <NUM>, the wearable device <NUM> can adaptively execute obtaining the first image and the second image and displaying the multimedia content as superimposed on the second image. The wearable device <NUM> can optimize the efficiency of using the resource in the wearable device <NUM>, with such adaptive execution.

<FIG> is a flowchart illustrating a method of identifying whether to display multimedia content with superimposing on a second image based on a position of at least one area. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

In operation <NUM>, the processor <NUM> may identify whether the position of the at least one area is within a center area in the display area of the display <NUM>, based on the identification. For example, the center area may be an attention area of a user wearing the wearable device <NUM>. For example, the center area may be an area within the display area of the display <NUM> that the user frequently views. For example, when the at least one area is positioned within a corner area in the display area distinct from the center area, the decrease in quality of the multimedia content owing to the at least one area is relatively small, so the processor <NUM> may identify whether the position of the at least one area is within the center area. For example, the processor <NUM> may execute operation <NUM> to reduce resource consumption of the wearable device <NUM> by the execution of operations <NUM> and <NUM>.

The processor <NUM> may execute operation <NUM> on condition that the position of the at least one area is within the center area, or execute operation <NUM> on condition that the position of the at least one area is outside the center area.

In operation <NUM>, the processor <NUM> may display the multimedia content as superimposed on the second image, based on identifying that the position of the at least one area is within the center area. For example, the processor <NUM> may obtain the second image by executing the operations <NUM> and <NUM>, based on identifying that the position of the at least one area is within the center area, and display the multimedia content as superimposed on the second image.

In operation <NUM>, the processor <NUM> may display the multimedia content, based on identifying that the position of the at least one area is out of the center area. For example, the processor <NUM> may display the multimedia content without displaying the second image.

As described above, the wearable device <NUM> can adaptively execute, via the execution of operation <NUM>, obtaining the first image and the second image and displaying the multimedia content as superimposed on the second image. The wearable device <NUM> can optimize the efficiency of using resource of the wearable device <NUM> through such an adaptive execution.

<FIG> is a flowchart illustrating a method of changing color of a second image. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

<FIG> shows exemplary second images with changed color.

Referring to <FIG>, in operation <NUM>, the processor <NUM> obtains the second image. For example, operation <NUM> may correspond to operation <NUM> of <FIG>.

In operation <NUM>, the processor <NUM> may change color of the second image based on a color temperature of the first image. For example, since the second image is an image for compensating for a portion of the external environment (e.g., at least one area <NUM>) to be viewed through the at least one area (e.g., at least one area <NUM>), the processor <NUM> may estimate the color temperature of the portion of the external environment by identifying the color temperature of the first image in response to obtaining the first image. The processor <NUM> may change the color of the second image based on the estimated color temperature. For example, when the color temperature corresponds to the color temperature of blue light, the processor <NUM> may change the color of the second image by blending red with the second image. For example, when the color temperature corresponds to the color temperature of red light, the processor <NUM> may change the color of the second image by blending blue with the second image. However, the present disclosure is not limited thereto.

In operation <NUM>, the processor <NUM> may display the multimedia content as superimposed on the second image having the changed color. For example, the second image having the changed color, displayed through the display <NUM> may form the background layer. For example, the color of the background layer may be changed from the reference color by the second image having the changed color. For example, referring to <FIG>, the color of the background layer <NUM> may be changed based on the second image having the changed color. For example, when the color temperature is within a first range, the background layer <NUM> having the reference color may be changed to the background layer <NUM> having a first color distinguished from the reference color. For example, when the color temperature is within a second range distinct from the first range, the background layer <NUM> having the reference color may be changed to a background layer <NUM> having a second color distinguished from the first color and the reference color. For example, when the color temperature is within a third range distinct from the first range and the second range, the background layer <NUM> having the reference color may be changed to a background layer <NUM> having a third color distinguished from the first color, the second color, and the reference color. However, the present disclosure is not limited thereto.

As described above, the wearable device <NUM> can enhance the quality of the multimedia content displayed through the display <NUM>, by adaptively changing the color of the second image according to the color temperature of the environment in which the wearable device <NUM> is located.

<FIG> is a flowchart illustrating a method of displaying the multimedia content as superimposed on a portion of the second image. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

<FIG> shows exemplary multimedia content displayed as superimposed on a portion of the second image.

Referring to <FIG>, in operation <NUM>, in response to obtaining the second image in operation <NUM>, the processor <NUM> may obtain a portion of the second image corresponding to the at least one area. For example, referring to <FIG>, the processor <NUM> may identify at least one area <NUM> within the multimedia content <NUM>, and identify at least one area <NUM> within the second image <NUM> corresponding to the at least one area <NUM>. The processor <NUM> may obtain the at least one area <NUM> as a portion of the second image <NUM>.

Referring back to <FIG>, in operation <NUM>, the processor <NUM> may display the multimedia content as superimposed on the obtained portion of the second image. For example, referring to <FIG>, the processor <NUM> may provide a background layer <NUM> by displaying at least one area <NUM>, which is the obtained portion of the second image <NUM>. For example, the color of at least one area <NUM> corresponding to the at least one area <NUM> in the background layer <NUM> may be the reference color. The processor <NUM> may, based on controlling the display <NUM> as in the state <NUM>, display the multimedia content <NUM> as superimposed on the background layer <NUM> provided by the at least one area <NUM>. The multimedia content <NUM> displayed as superimposed on the background layer <NUM> may be provided via the display <NUM>, as in a display state <NUM>. For example, in the display state <NUM>, the at least one area <NUM> in the environment <NUM> may not be visible through the at least one area <NUM>, as in the display state <NUM>.

In an embodiment, operations <NUM> and <NUM> may be executed on condition that the multimedia content is a static image. For example, the processor <NUM> may execute operations <NUM> to <NUM> on condition that the multimedia content is a static image, or execute operations <NUM> and <NUM> on condition that the multimedia content is not a static image. However, the present disclosure is not limited thereto.

As described above, the wearable device <NUM> can adaptively obtain the portion of the second image, thereby reducing resource consumption by displaying the second image.

<FIG> is a flowchart illustrating a method of processing a second image and multimedia content superimposed on the second image according to movement of an external object. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

<FIG> illustrates an example of a method of processing a second image and multimedia content superimposed on the second image according to movement of an external object.

Referring to <FIG>, in operation <NUM>, the processor <NUM> displays the multimedia content superimposed on the second image. For example, operation <NUM> may correspond to operation <NUM> of <FIG>.

In operation <NUM>, the processor <NUM> may identify whether there is an external object moving within a portion of the environment including the wearable device <NUM>, which is hidden by displaying the multimedia content superimposed on the second image. For example, since the user wearing the wearable device <NUM> is not able to identify the movement of the external object owing to displaying of the multimedia content, the user may not recognize that he or she is in an unexpected situation. In order to prevent such unrecognized situation, the processor <NUM> may obtain images via the camera <NUM>, while displaying the multimedia content superimposed on the second image, and based on the obtained images, may identify whether there exists such a moved or moving external object. The processor <NUM> may execute operation <NUM> on condition that the (moved/moving) external object exists, or keep executing operation <NUM> while displaying the multimedia content superimposed on the second image on condition that the external object does not exist.

In operation <NUM>, the processor <NUM> may cease displaying the second image based on identifying that the external object exists. For example, ceasing displaying the second image may cause formation or provision of the background layer to be terminated. For example, referring to <FIG>, the processor <NUM> may identify that there exists the external object, while displaying the multimedia content <NUM> as superimposed on the background layer <NUM> provided by displaying of the second image <NUM>, based on controlling the display <NUM>, as in the state <NUM>. The processor <NUM> may cease displaying the second image <NUM> based on the identification. The background layer <NUM> may be removed as in a state <NUM> by cessation of displaying of second image <NUM>, and the at least one area <NUM> (not shown in <FIG>) in the environment <NUM> may be exposed through the at least one area <NUM>.

Referring back to <FIG>, in operation <NUM>, the processor <NUM> may reduce the opacity of the multimedia content based on identifying that there exists the external object. For example, the processor <NUM> may reduce the opacity of the multimedia content so that the user wearing the wearable device <NUM> can more easily recognize the external object. For example, referring to <FIG>, the processor <NUM> may display the state <NUM> by reducing the opacity of the multimedia content <NUM> in a state that the second image is removed (or in a state that the background layer <NUM> is removed). In the display state <NUM>, the user may recognize the external object <NUM> in moving.

Although <FIG> illustrates an example of executing operation <NUM> after executing operation <NUM>, the operations <NUM> and <NUM> may be executed simultaneously, or the operation <NUM> may be executed after operation <NUM> is executed, or only one of operation <NUM> and operation <NUM> may be executed.

In operation <NUM>, the processor <NUM> may identify whether the movement of the external object is ceased, while ceasing to display the second image and(/or) displaying the multimedia content having the reduced opacity. For example, the processor <NUM> may identify whether the movement of the external object is ceased or whether the external object moves out of the field of view of the camera <NUM>, based on the images obtained through the camera <NUM>. The processor <NUM> may execute operation <NUM> on condition that the movement of the external object is ceased, or maintain executing operation <NUM> on condition that the movement of the external object is maintained.

In operation <NUM>, the processor <NUM> may resume displaying the second image and restore the opacity of the multimedia content, based on identifying that the movement of the external object is ceased. For example, the processor <NUM> may resume displaying the second image and restore the opacity of the multimedia content in order to enhance the quality of displaying of the multimedia content.

As described above, the wearable device <NUM> may execute operation <NUM> so that the user wearing the wearable device <NUM> can recognize an external object moving around the wearable device <NUM> while displaying the multimedia content superimposed on the second image. For example, the wearable device <NUM> may execute operation <NUM> so that the user can view the multimedia content in a safe environment.

<FIG> is a flowchart illustrating a method of identifying color of a second visual object displayed under at least one first visual object associated with multimedia content. This method may be executed by the wearable device <NUM> shown in <FIG>, the wearable device <NUM> shown in <FIG> or <FIG>, or the processor <NUM> shown in <FIG>. In certain embodiments, one or more of the operations shown in <FIG> may be omitted and/or one or more operations shown in <FIG> may be performed in a different order to that shown.

<FIG> illustrates exemplary second visual objects displayed under at least one first visual object associated with multimedia content.

Referring to <FIG>, in operation <NUM>, the processor <NUM> may identify, while displaying the multimedia content superimposed on the second image, color of at least one first visual object to be displayed in association with the multimedia content. For example, the at least one first visual object may be visual information related to the multimedia content, but the present disclosure is not limited thereto. For example, the processor <NUM> may identify the color of the at least one first visual object to be displayed in association with the multimedia content, based on identifying an event related to the multimedia content while displaying the multimedia content.

In operation <NUM>, the processor <NUM> may identify the color of a second visual display to be displayed under the at least one first visual object, based on the color of the at least one first visual object or the color of the multimedia content. For example, the second visual object may be a visual object displayed under the at least one first visual object to enhance the quality of displaying of the at least one first visual object. For example, the second visual object may be a background of the at least one first visual object. However, the present disclosure is not limited thereto.

In an embodiment, the processor <NUM> may identify the color of the second visual object based on the color of the multimedia content amongst the at least one first visual object and the color of the multimedia content, on condition that the at least one first visual object has only at least one specified color (this "specified color" may be unrelated to the "specified color" described in combination with at least one area within a multimedia content previously), and identify the color of the second visual object based on the color of the at least one first visual object amongst the at least one first visual object and the color of the multimedia content, on condition that the at least one first visual object has a different color distinguished from the at least one specified color. For example, the at least one specified color may be a color in which visibility of the at least one first visual object is ensured, independently (or irrespectively) of which color the color of the second visual object is identified as. For example, the at least one specified color may be black and white. However, the present disclosure is not limited thereto. For example, when the visibility of the at least one first visual object having only the at least one specified color is ensured, the processor <NUM> may identify the color of the second visual object based on the color of the multimedia content, for harmonizing with the multimedia content. For example, when the visibility of the at least one first visual object having the different color is not ensured, the processor <NUM> may identify the color of the second visual object as a complementary color to the color of the at least one first visual object, in order to enhance the visibility of the at least one first visual object. For example, referring to <FIG>, in response to identifying an event for displaying at least one first visual object <NUM> having only the at least one specified color, the processor <NUM> may identify the color of the second visual object <NUM> to be displayed under the at least one first visual object <NUM>, as a color <NUM> identified based on the color of the multimedia content <NUM>. For example, in response to identifying an event for displaying at least one first visual object <NUM> having the different color, the processor <NUM> may identify the color of the second visual object <NUM> to be displayed under the at least one first visual object <NUM>, as a color <NUM> identified based on the color of the at least one first visual object <NUM>.

Referring again to <FIG>, in operation <NUM>, the processor <NUM> may display the at least one first visual object associated with the multimedia content, as superimposed on the second visual object having the color identified in operation <NUM>.

As described above, the wearable device <NUM> may, based on detecting an event for displaying the at least one first visual object related to the multimedia content while displaying the multimedia content as superimposed on the second image, identify the color of the second visual object to be displayed under the at least one first visual object based on the color of the at least one first visual object, or identify the color of the second visual object to be displayed under the at least one first visual object based on the color of the multimedia content, thereby enhancing the visibility of the at least one first visual object or displaying the second visual object in harmony with the multimedia content.

An electronic device, a method, and a non-transitory computer-readable storage medium according to an embodiment can enhance the quality of the multimedia content by displaying multimedia content superimposed on a second image in which a first image obtained through the camera is converted in color.

As described above, a wearable device may comprise at least one camera, a display, a memory configured to store instructions, and a processor. The processor may be configured to execute the instructions to obtain a user request for displaying multimedia content in a display area of the display. The processor may be configured to execute the instructions to, based on the user request, identify whether brightness of an environment around the wearable device is greater than or equal to reference brightness. The processor may be configured to execute the instructions to, based on the brightness greater than or equal to the reference brightness, identify whether the multimedia content includes at least one area having specified color. The processor may be configured to execute the instructions to, based on the multimedia content including the at least one area, generate a first image for a portion of the environment corresponding to a position in which the multimedia content is to be displayed, via the at least one camera. The processor may be configured to execute the instructions to generate a second image in which color of the first image is converted. The processor may be configured to execute the instructions to display, via the display, the multimedia content, as superimposed on the second image displayed in the position.

According to an embodiment, the color(s) of the first image is(are) inverted in the second image.

According to an embodiment, the processor may be configured to execute the instructions to, based on the brightness less than the reference brightness, refrain from identifying whether the multimedia content includes the at least one area.

According to an embodiment, the processor may be configured to execute the instructions to, based on the multimedia content not including the at least one area, refrain from generating the first image and the second image.

According to an embodiment, the processor may be configured to execute the instructions to, in response to the user request, identify the brightness of the environment, based on data indicating brightness of an image obtained via the at least one camera. According to an embodiment, the processor may be configured to execute the instructions to identify whether the brightness of the environment, identified based on the data, is greater than or equal to the reference brightness.

According to an embodiment, the processor may further comprise an illuminance sensor. According to an embodiment, the processor may be configured to execute the instructions to, in response to the user request, identify the brightness of the environment, further (or alternatively) based on data obtained via the illuminance sensor. According to an embodiment, the processor may be configured to execute the instructions to identify whether the brightness of the environment identified further (or alternatively) based on the data obtained via the illuminance sensor is greater than or equal to the reference brightness.

According to an embodiment, the processor may be configured to execute the instructions to identify a first virtual plane defined on a virtual three-dimensional (3D) space, based on the first image. According to an embodiment, the processor may be configured to execute the instructions to render the second image on the first virtual plane. According to an embodiment, the processor may be configured to execute the instructions to render the multimedia content on a second virtual plane defined on the virtual three-dimensional space and distinguished from the first virtual plane. According to an embodiment, the processor may be configured to execute the instructions to display the multimedia content as superimposed on the second image, by projecting the rendered second image and the rendered multimedia content onto a third virtual plane defined on the virtual 3D space, the third virtual plane corresponding to the display area of the display.

According to an embodiment, the processor may be further configured to execute the instructions to, based on the brightness less than the reference brightness and/or the multimedia content not including the at least one area, display the multimedia content by emitting, from among first light emitting elements for the at least one area and second light emitting elements for at least another area of the multimedia content having color distinct from the specified color, light from the second light emitting elements.

According to an embodiment, the processor may be further configured to execute the instructions to adjust opacity of the multimedia content. According to an embodiment, the processor may be further configured to execute the instructions to display, via the display, the multimedia content with the adjusted opacity, as superimposed on the second image displayed in the position.

According to an embodiment, the processor may be further configured to execute the instructions to change color of the second image, based on color temperature of the first image. According to an embodiment, the processor may be further configured to execute the instructions to display the multimedia content, as superimposed on the second image with the changed color.

According to an embodiment, the processor may be further configured to execute the instructions to, after the second image is generated, extract a portion of the second image corresponding to the at least one area. According to an embodiment, the processor may be further configured to execute the instructions to display the multimedia content, as superimposed on the extracted portion of the second image displayed in at least one position in the display area corresponding to the at least one area. According to an embodiment, the processor may be further configured to execute the instructions to display the at least one area of the multimedia content, as superimposed on the extracted portion of the second image displayed in at least one position in the display area corresponding to the at least one area.

According to an embodiment, the processor may be configured to execute the instructions to, based on the user request, identify whether the multimedia content is a static image. According to an embodiment, the processor may be further configured to execute the instructions to, based on the multimedia content that is a static image, display the multimedia content, as superimposed on the extracted portion of the second image. According to an embodiment, the processor may be further configured to execute the instructions to, based on the multimedia content that is not a static image, display the multimedia content, as superimposed on the second image.

According to an embodiment, the processor may be further configured to execute the instructions to, based on at least one image obtained via the at least one camera while the multimedia content superimposed on the second image is displayed, identify whether there exists an external object moving in a portion of the environment hidden by displaying the multimedia content superimposed on the second image. According to an embodiment, the processor may be further configured to execute the instructions to, based on the identification of the external object, cease displaying the second image. According to an embodiment, displaying of the multimedia content may be maintained while displaying of the second image is ceased.

According to an embodiment, the processor may be further configured to execute the instructions to, based on the identification of the external object, decrease opacity of the multimedia content displayed via the display. For example, the external object moving within the portion of the environment may be viewed through the display area, according to the decrease of the opacity of the multimedia content.

According to an embodiment, the processor may be further configured to execute the instructions to, based on at least one image obtained via the at least one camera while ceasing to display the second image and displaying the multimedia content, identify whether the movement of the external object is terminated. According to an embodiment, the processor may be further configured to execute the instructions to, based on termination of the movement of the external object, display the multimedia content superimposed on the second image by resuming displaying the second image in the position.

According to an embodiment, the user request may comprise an input for executing a software application used to play the multimedia content.

According to an embodiment, the processor may be further configured to execute the instructions to, while displaying the multimedia content superimposed on the second image, identify color of at least one first visual object to be displayed in association with the multimedia content. According to an embodiment, the processor may be further configured to execute the instructions to identify, based on color of the at least one first visual object or color of the multimedia content, color of a second visual object to be displayed under the at least one first visual object. According to an embodiment, the processor may be further configured to execute the instructions to display the at least one first visual object associated with the multimedia content, as superimposed on the second visual object with the identified color.

According to an embodiment, the processor may be configured to execute the instructions to, on condition that the at least one first visual object has only at least one specified color, identify the color of the second visual object, based on the color of the multimedia content from among the color of the at least one first visual object and the color of the multimedia content. According to an embodiment, the processor may be configured to execute the instructions to, on condition that the at least one first visual object has another color distinct from the at least one specified color, identify the color of the second visual object, based on the color of the at least one first visual object from among the color of the at least one first visual object and the color of the multimedia content.

According to an embodiment, the processor may be configured to execute the instructions to, based on the multimedia content including the at least one area, identify a ratio of size of the at least one area to size of the multimedia content. According to an embodiment, the processor may be configured to execute the instructions to, based on the ratio greater than or equal to a reference ratio, display the multimedia content superimposed on the second image. According to an embodiment, the processor may be configured to execute the instructions to, based on the ratio less than the reference ratio, refrain from generating the first image and the second image and display the multimedia content without displaying of the second image.

According to an embodiment, intensity of light passing through the at least one area may be greater than or equal to reference intensity, while displaying the multimedia content that is not superimposed on the second image, and may be less than the reference intensity, while displaying the multimedia content superimposed on the second image.

According to an embodiment, the size of the second image may be greater than or equal to the size of the multimedia content.

The electronic device according to various embodiments disclosed herein may be one of various types of electronic devices.

As used in connection with various embodiments of the disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be interchangeably used with other terms, for example, "logic," "logic block," "part," or "circuitry". For example, according to an embodiment of the disclosure, the module may be implemented in a form of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software including one or more instructions that are stored in a storage medium that is readable by a machine. For example, a processor of the machine may invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor.

According to an embodiment of the disclosure, a method according to various embodiments of the disclosure may be included and provided in a computer program product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., PlayStore™), or between two user devices (e.g., smart phones) directly.

Claim 1:
A wearable device (<NUM>) comprising:
at least one camera (<NUM>);
a display (<NUM>);
a memory (<NUM>) configured to store instructions; and
a processor (<NUM>),
wherein the instructions, when executed by the processor, cause the wearable device to:
receive (<NUM>) an input for displaying a multimedia content (<NUM>) in a display area (<NUM>) of the display;
based on receiving the input, identify (<NUM>) whether a brightness of an environment (<NUM>) around the wearable device is greater than or equal to a reference brightness;
based on identifying that the brightness is greater than or equal to the reference brightness, identify (<NUM>) whether the multimedia content includes at least one area (<NUM>) having a specified color;
based on identifying that the multimedia content includes the at least one area, obtain (<NUM>) a first image for a portion of the environment corresponding to a position in which the multimedia content is to be displayed, via the at least one camera;
obtain (<NUM>) a second image in which a color of the first image is converted; and
display (<NUM>), via the display, the multimedia content, as superimposed on the second image displayed in the position.