Patent Description:
<CIT> discloses an automatic variable virtual focus for augmented reality displays. <CIT>discloses a see-through computer display systems. <CIT> discloses a virtual image generator. <CIT> discloses a virtual/augmented reality system having dynamic region resolution. <CIT>discloses variable fixation viewing distance scanned light displays. <CIT> discloses materials and methods for simulating focal shifts in viewers using large depth of focus displays. In order to implement augmented reality glasses, a waveguide and a display may be used. The waveguide may be divided into a reflective type and a diffractive type, and a liquid crystal display (LCD), a liquid crystal on silicon (LCoS), a digital light processing (DLP), and a laser scanning projector (LSP) type may be used for the display.

In a method for outputting a virtual image, vergence-accommodation conflict may occur in any type of the display. Accordingly, sense of reality may deteriorate and this may cause eyestrain or motion sickness such as dizziness.

Meanwhile, since the LSP type uses linear light, a depth (sense of distance or depth) may not be expressed in a virtual image, which caused a problem that all distances of the virtual image are expressed with the same depth.

The disclosure is made to solve the above-mentioned problems and an object of the disclosure is to provide a wearable device which displays AR information corresponding to an object by adding a focus adjusting element to a laser projector and a control method thereof.

The invention is set out by the appended set of claims. In accordance with an aspect of the disclosure, there is provided a wearable device according to claim <NUM>.

The processor may be configured to identify whether a location of the object corresponds to the user's gaze by comparing the user's gaze with the location information of the object, and control the driving state of the focus adjusting element based on the identified result.

The processor may be configured to, based on the location of the object corresponding to the user's gaze, provide AR information related to the object on the display by controlling a focal length of the focus adjusting element to be a first focal length, and based on the location of the object not corresponding to the user's gaze, provide the AR information related to the object on the display by controlling the focal length of the focus adjusting element to be a second focal length.

The first focal length may be a unique focal length of the focus adjusting element, and the second focal length may be different from the unique focal length.

The processor may be configured to identify location information of each of a plurality of objects included in the obtained image, based on first AR information related to a first object corresponding to the user's gaze among the plurality of objects being provided, control a focal length of the focus adjusting element to be a first focal length, and based on second AR information related to a second object not corresponding to the user's gaze among the plurality of objects being provided, control the focal length of the focus adjusting element to be a second focal length.

The processor may be configured to, based on the second AR information being provided, control the focal length of the focus adjusting element to be the second focal length based on depth information of the second object.

The processor may be configured to obtain a difference between a depth value included in the depth information of the second object and a depth value corresponding to the identified user's gaze, and control the focal length of the focus adjusting element to be the second focal length based on the obtained difference of depth values.

The processor may be configured to, based on a plurality of objects being included in the obtained image, control the scanning mirror by determining at least one of a scanning start location, a scanning direction, and a scanning path based on location information of each of the plurality of objects.

The processor may be configured to determine at least one of the scanning start location, the scanning direction, and the scanning path so that the focal length of the focus adjusting element is changed in phases based on the location information of each of the plurality of objects.

The processor may be configured to determine at least one of the scanning start location, the scanning direction, and the scanning path to minimize a rotation operation of the scanning mirror based on the location information of each of the plurality of objects.

In accordance with another aspect of the disclosure, there is provided a method for controlling a wearable device including a laser projector including a light emitting element configured to emit laser light, a focus adjusting element configured to adjust a focal point of the emitted laser light, and a scanning mirror configured to control a scanning direction of the light with the adjusted focal point according to claim <NUM>.

The controlling the driving state of the focus adjusting element may include identifying whether a location of the object corresponds to the user's gaze by comparing the user's gaze with the location information of the object, and controlling the driving state of the focus adjusting element based on the identified result.

The providing the AR information on the display may include, based on the location of the object corresponding to the user's gaze, providing AR information related to the object on the display by controlling a focal length of the focus adjusting element to be a first focal length, and based on the location of the object not corresponding to the user's gaze, providing the AR information related to the object on the display by controlling the focal length of the focus adjusting element to be a second focal length.

The method for controlling the wearable device may further include identifying location information of each of a plurality of objects included in the obtained image, and the controlling the driving state of the focus adjusting element may include, based on first AR information related to a first object corresponding to the user's gaze among the plurality of objects being provided, controlling a focal length of the focus adjusting element to be a first focal length, and based on second AR information related to a second object not corresponding to the user's gaze among the plurality of objects being provided, controlling the focal length of the focus adjusting element to be a second focal length.

The controlling the driving state of the focus adjusting element may include, based on the second AR information being provided, controlling the focal length of the focus adjusting element to be the second focal length based on depth information of the second object.

The controlling the driving state of the focus adjusting element may include obtaining a difference between a depth value included in the depth information of the second object and a depth value corresponding to the identified user's gaze, and controlling the focal length of the focus adjusting element to be the second focal length based on the obtained difference of depth values.

The controlling the driving state of the focus adjusting element may include, based on a plurality of objects being included in the obtained image, controlling the scanning mirror by determining at least one of a scanning start location, a scanning direction, and a scanning path based on location information of each of the plurality of objects.

The controlling the scanning mirror may include determining at least one of the scanning start location, the scanning direction, and the scanning path so that the focal length of the focus adjusting element is changed in phases based on the location information of each of the plurality of objects.

The controlling the scanning mirror may include determining at least one of the scanning start location, the scanning direction, and the scanning path to minimize a rotation operation of the scanning mirror based on the location information of each of the plurality of objects.

The disclosure will be described in detail after briefly explaining the way of describing the specification and the drawings.

The terms used in the specification and claims have been selected as general terms as possible in consideration of functions in the embodiments of the disclosure. But, these terms may vary in accordance with the intention of those skilled in the art, the precedent, technical interpretation, the emergence of new technologies and the like. In addition, there are also terms arbitrarily selected by the applicant. Such terms may be interpreted as meanings defined in this specification and may be interpreted based on general content of the specification and common technical knowledge of the technical field, if there are no specific term definitions.

The same reference numerals or symbols in the accompanying drawings in this specification denote parts or components executing substantially the same function. For convenience of description and understanding, the description will be made using the same reference numerals or symbols in different embodiments. That is, although the components with the same reference numerals are illustrated in the plurality of drawings, the plurality of drawings are not illustrating one embodiment.

In addition, terms including ordinals such as "first" or "second" may be used for distinguishing components in the specification and claims. Such ordinals are used for distinguishing the same or similar components and the terms should not be limitedly interpreted due to the use of ordinals. The terms are used only to distinguish one element from another. For example, in regard to components with such ordinals, usage order or arrangement order should not be limitedly interpreted with the numbers thereof. The ordinals may be interchanged, if necessary.

Unless otherwise defined specifically, a singular expression may encompass a plural expression. It is to be understood that the terms such as "comprise" or "consist of" are used herein to designate a presence of characteristic, number, step, operation, element, part, or a combination thereof, and not to preclude a presence or a possibility of adding one or more of other characteristics, numbers, steps, operations, elements, parts or a combination thereof.

The embodiments of the disclosure may be variously changed and include various embodiments, and specific embodiments will be shown in the drawings and described in detail in the description.

A term such as "module", a "unit", or a "part" in the disclosure is for designating a component executing at least one function or operation, and such a component may be implemented as hardware, software, or a combination of hardware and software. Further, except for when each of a plurality of "modules", "units", "parts" and the like needs to be realized in an individual specific hardware, the components may be integrated in at least one module or chip and be implemented in at least one processor.

In addition, in the embodiments of the disclosure, connection of a certain part to another part may include indirect connection via still another medium, in addition to direct connection. When it is described that a certain part includes another certain part, it implies that a still another part may be further included, rather than excluding it, unless otherwise noted.

<FIG> and <FIG> are diagrams illustrating sharpness of an object according to a user's gaze.

<FIG> is a diagram illustrating a case where a user fixes user's eye on a vehicle <NUM> on a left side with respect to the front. When the user fixes user's eye on the vehicle <NUM> which is an object located on the left side, the user may blurredly recognize objects other than the portion fixed with the user's eye. For example, a building <NUM> which is a portion other than the left portion fixed with user's eye may be blurredly recognized by the user. For convenience of description, in this specification, an object sharply recognized by the user may be displayed with a solid line and an object blurredly recognized may be displayed with a dotted line. For example, the vehicle <NUM> fixed with user's eye is displayed with a solid line, and the building <NUM> located on a portion which is not fixed with user's eye is displayed with a dotted line.

<FIG> is a diagram illustrating a case where the user fixes user's eye on the building <NUM> on a right side with respect to the front. When the user fixes user's eye on the building <NUM> which is an object located on the right side, the user may blurredly recognize an object other than the portion fixed with user's eye. For example, the vehicle <NUM> may be blurredly recognized by the user.

<FIG> and <FIG> are diagrams illustrating AR information according to a gaze of a user wearing a wearable device.

The wearable device may recognize an object in a range recognizable by a user and display AR information related to the object on a display included in the wearable device. For example, the wearable device may display AR information <NUM> of the vehicle <NUM> and the AR information <NUM> of the vehicle <NUM> may include a name, a manufacturing company, a distance from the wearable device of the recognized object.

<FIG> illustrates an embodiment in which the user fixes user's eye on the vehicle <NUM> on the left after wearing the wearable device. The user may recognize objects of the vehicle <NUM> and the building <NUM> in front. The wearable device may display AR information regarding the recognized objects of the vehicle <NUM> and the building <NUM>. Referring to <FIG>, the wearable device may display AR information including at least one of a type (vehicle), a manufacturing company (Samsung), and a distance from the wearable device (<NUM>) with respect to the vehicle <NUM> object. In addition, the wearable device may display AR information including at least one of a type (building), a building name (abc), and a distance from the wearable device (<NUM>) with respect to the building <NUM> object. The AR information provided herein may be information of a type set by the user through a menu and the like, information provided from a currently operated application, information according to a type of an object, and the like but is not limited thereto.

Meanwhile, the wearable device may display various pieces of information other than the AR information of the object. For example, the wearable device may display information related to a navigation function. Referring to <FIG> and <FIG>, the wearable device may display AR information (go <NUM>) related to the navigation function in the front portion of the user and the information related to the navigation function may refer to direction and distance indexes for reaching a destination. <FIG> and <FIG> illustrate only "Go <NUM>", but the information related to the navigation function may include information displaying various directions and distances such as "left <NUM>", "right <NUM>", "turn back <NUM>", and the like.

<FIG> illustrates an embodiment in which the wearable device displays the AR information, when the user fixes user's eye on the vehicle <NUM> on the left side after wearing the wearable device.

<FIG> illustrates an embodiment in which the wearable device displays the AR information, when the user fixes user's eye on the building <NUM> on the right side after wearing the wearable device.

The user may sharply recognize the object fixed with user's eye. However, the user may blurredly recognize the object not fixed with user's eye. The wearable device may display the AR information without considering on which object the user's gaze is fixed. Accordingly, in the embodiments corresponding to <FIG> and <FIG>, the wearable device may sharply display the AR information corresponding to the vehicle <NUM> and the building <NUM>. In other words, the wearable device may display the AR information with the same sharpness, regardless of the position of the user's gaze.

<FIG> is a diagram illustrating an embodiment of displaying the AR information, when a focus adjusting element is not provided.

The wearable device may include a laser scanning projector (LSP) or laser projector <NUM>, as will be described later with reference to <FIG>. The laser projector <NUM> may include a light emitting element <NUM>, a focus adjusting element <NUM>, and a scanning mirror <NUM>. The light emitting element <NUM> herein may be an element emitting red, green, and blue laser light and may be referred to as an RGB laser.

Referring to <FIG>, the wearable device may display the AR information on the display using the laser projector <NUM> without the focus adjusting element <NUM>.

A reference numeral <NUM> may denote an image seen through a transparent display. An image <NUM> may be an AR information image displayed on the transparent display by the wearable device. The image <NUM> may include AR information <NUM> and <NUM> corresponding to each object or additional information <NUM>. The laser projector <NUM> may use the light emitting element <NUM> to display the image <NUM> on the display. The wearable device may display the image <NUM> on the transparent display using the laser projector <NUM>, and an image finally displayed by the wearable device for the user may be an image <NUM>. The image <NUM> may be an image obtained by combining the image seen through the transparent display with the image displayed with the AR information.

Meanwhile, distances, from the wearable device, of the objects of the vehicle <NUM> and the building <NUM> recognized by the user may be different from each other. However, wearable device may display the AR information with the same sharpness regardless of the position of the user's gaze, and accordingly, the user may be confused with the sense of distance or depth. For example, the objects recognized by the user may be located at different distances from the wearable device. However, if the wearable device displays the AR information corresponding to each object with the same sharpness regardless of the position of the user's gaze, the user may be confused with the sense of distance or depth regarding each object and the AR information corresponding to each object.

In order to solve such a problem, the wearable device may perform a preprocessing operation with respect to an image regarding the AR information corresponding to each object.

<FIG> is a diagram illustrating an embodiment in which a wearable device <NUM> preprocesses the AR information, when the laser projector <NUM> does not include the focus adjusting element.

Referring to <FIG>, the wearable device may identify the position of the user's gaze and perform the preprocessing operation with respect to some AR information of the AR information image, by considering the position of the user's gaze.

For example, the wearable device may distinguish an object at a position of the user's gaze from an object which is not. The wearable device may sharply display the AR information corresponding to the object at the position of the user's gaze and may blurredly display the AR information corresponding to the object not at the position of the user's gaze.

In other words, the wearable device may perform the preprocessing operation with respect to the AR information in advance, and the wearable device may obtain, in advance, an image <NUM> in which the sharpness (sharp or blurred state) of the AR information corresponding to each object is determined, by considering the position of the user's gaze. The preprocessing operation may refer to an operation of suitably changing the sharpness of the AR information in advance, and the wearable device may output an image, in which the sharpness of some AR information is changed, via the laser projector <NUM>. Herein, the laser projector <NUM> may not include the focus adjusting element.

A reference numeral <NUM> may denote an image seen through the transparent display. A reference numeral <NUM> may denote an AR information image displayed on the transparent display by the wearable device. The image <NUM> may include the AR information <NUM> and <NUM> corresponding to each object and the additional information <NUM>. The laser projector <NUM> may use the light emitting element <NUM> to display the image <NUM> on the display. The wearable device may display the image <NUM> on the transparent display using the laser projector <NUM>, and an image finally displayed by the wearable device for the user may be an image <NUM>. The image <NUM> may be an image obtained by combining the image seen through the transparent display with the image displayed with the AR information.

<FIG> is a block diagram illustrating the wearable device according to an embodiment of the disclosure.

Referring to <FIG>, the wearable device <NUM> includes a camera <NUM>, a sensor <NUM>, a display <NUM>, a processor <NUM>, the laser scanning projector (LSP) or laser projector <NUM>.

The camera <NUM> is used as a device for recognizing objects in front of the user by the wearable device <NUM> and the camera <NUM> may capture an image to identify things and environment in front. The wearable device <NUM> may detect characteristics of objects located in front of the user using the camera <NUM>. In addition, the wearable device <NUM> may detect how the objects identified by the camera <NUM> are distributed in the space (depth map) and what each object is (object recognition). In addition, the wearable device <NUM> may determine which information is to be displayed on the corresponding location by using measured information.

The sensor <NUM> is used as a device for identifying the user's gaze. For example, the sensor <NUM> may detect a movement of user's pupils, in order to identify where the user is looking at. Herein, the sensor <NUM> and the camera <NUM> are separately displayed, but the wearable device <NUM> may include a camera other than the camera <NUM> for identifying the object in front and the other camera may be attached to the sensor <NUM>.

For the display <NUM>, a transparent display may be used so that the user is able to look ahead. The user may look ahead through the display <NUM>, and AR information corresponding to a specific object may be displayed on a part of the display <NUM>.

The processor <NUM> may perform general control operations of the wearable device. Specifically, the processor identifies an object using image information captured by the camera <NUM> and may obtain AR information corresponding to the object. In addition, the processor controls the laser projector <NUM> to display the AR information on the display <NUM>.

The processor may be implemented as a digital signal processor (DSP) or a microprocessor processing digital signals. However, there is no limitation thereto, and the processor may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), or a communication processor (CP), and an ARM processor or may be defined as the corresponding term. In addition, the processor <NUM> may be implemented as System on Chip (SoC) or large scale integration (LSI) including the processing algorithm or may be implemented in form of a field programmable gate array (FPGA).

The processor <NUM> identifies the user's gaze based on sensing data obtained by the sensor <NUM> and the processor <NUM> identifies information regarding at least one object included in the image obtained by the camera <NUM>. The information regarding the object may include information of an object such as a type, a name, a shape, a color, and the like of the object and includes location information regarding a location where the object exists.

The processor <NUM> may obtain augmented reality (AR) information related to the object based on the identified user's gaze and information of the object. In order to obtain the information of the object, the user may analyze the image captured by the camera <NUM>. An object may be identified in the image captured by the camera and AR information corresponding to the corresponding object may be obtained. As a method for obtaining the AR information, the processor <NUM> may use an internal storage device. The information of the object may be obtained by using a database stored in the internal storage device in advance.

Meanwhile, in actual implementation, the processor <NUM> may use an external server to obtain the AR information corresponding to the object. The processor <NUM> may transmit the information of the object to the external server and receive the corresponding AR information from the external server. Herein, the processor <NUM> may further additionally include a communicator (not illustrated) for transmitting and receiving data.

Meanwhile, the processor <NUM> may control the laser projector <NUM> to provide the obtained AR information on the display <NUM>.

The laser projector <NUM> includes the light emitting element <NUM> which emits laser light, the focus adjusting element <NUM> which adjusts a focus of the emitted laser light, and the scanning mirror <NUM> which controls a scanning direction of the light with the adjusted focus.

Referring to <FIG>, the laser projector <NUM> may be a laser scanning projector (LSP) and includes the light emitting element <NUM>, the focus adjusting element <NUM>, and the scanning mirror <NUM>.

The laser projector <NUM> may be a device for outputting a laser light source and may use the LSP method.

The light emitting element <NUM> may output mainly light sources with three colors of red, green, and blue and may output light with various colors by additionally combining a white light source with the three light sources.

The focus adjusting element <NUM> may be an element through which the light output from the light emitting element <NUM> is transmitted and may refract light. When the focus adjusting element <NUM> is controlled, a degree of refraction of transmitted light may change. When the degree of refraction of light changes, a focus of a virtual image may change, and accordingly, the user may sharply or blurredly recognize an object.

The focus adjusting element <NUM> may be formed of a liquid lens, a liquid crystal lens, or the like. The liquid lens may refer to a lens which adjusts a refractive index by changing a thickness or a shape of the lens physically, and the liquid crystal lens may refer to a lens which adjusts a refractive index by changing a chemical configuration or arrangement of molecules.

The processor <NUM> may control a driving state of the focus adjusting element <NUM> based on the identified user's gaze and location information of the object.

Herein, the processor <NUM> identifies whether the location of the object corresponds to the user's gaze by comparing the user's gaze with the location information of the object. The processor <NUM> may control the driving state of the focus adjusting element <NUM> based on the identified result.

Specifically, when the location of the object corresponds to the user's gaze, the processor <NUM> may perform control a focal length of the focus adjusting element <NUM> to be a first focal length and may provide the AR information related to the object on the display <NUM>. Herein, the focal length may be a distance between a focal point, at which light rays meet after transmitting the focus adjusting element <NUM>, and a reference point. The reference point may refer to a vertex of the focus adjusting element <NUM>. The focal length may vary depending on a refractive index of the focus adjusting element <NUM>. For example, when the refractive index of the focus adjusting element <NUM> changes, the focal length may also change. The first focal length may correspond to a pre-stored distance in a state where the focus adjusting element <NUM> has a specific refractive index.

When the processor <NUM> changes the refractive index by adjusting the focus adjusting element <NUM>, the focal length may be changed. When the focal length is changed, the sharpness of the object may be different. The sharpness being different may imply a change in sharpness of an image formed on the retina of the user. Specifically, while the light output from the laser projector <NUM> is the same, the sharpness of the image recognized by the user may vary depending on a change in refractive index of the focus adjusting element <NUM>.

Meanwhile, when the location of the object does not correspond to the user's gaze, the processor <NUM> may perform control the focal length of the focus adjusting element <NUM> to be a second focal length and may provide the AR information related to the object on the display <NUM>.

Herein, the first focal length may refer to a unique focal length of the focus adjusting element <NUM> and the second focal length may be different from the unique focal length.

Meanwhile, the processor <NUM> may identify location information of each of a plurality of objects included in the obtained image. When providing first AR information related to a first object corresponding to the user's gaze among the plurality of objects, the processor <NUM> may perform control the focal length of the focus adjusting element <NUM> to be the first focal length.

When providing second AR information related to a second object not corresponding to the user's gaze among the plurality of objects, the processor <NUM> may perform control the focal length of the focus adjusting element <NUM> to be the second focal length.

In addition, when providing the second AR information, the processor <NUM> may perform control the focal length of the focus adjusting element <NUM> to be the second focal length based on depth information of the second object.

Further, the processor <NUM> may obtain a difference between a depth value included in the depth information of the second object and a depth value corresponding to the identified user's gaze, and perform control the focal length of the focus adjusting element <NUM> to be the second focal length based on the obtained difference in depth value. The depth value herein may refer to an index representing the sense of distance of the object. The depth value may have a distance of the object from the wearable device <NUM> as a reference.

In addition, when the obtained image includes a plurality of objects, the processor <NUM> may control the scanning mirror <NUM> by determining at least one of a scanning start location, a scanning direction, and a scanning path based on location information of each of the plurality of objects. The scanning mirror <NUM> may reflect light (light source) output from the light emitting element <NUM> to project the light at a specific location. Accordingly, the processor <NUM> may reflect light (light source) output from the light emitting element <NUM> to project the light at a specific location by adjusting the start location, the direction, the path, and the like of the scanning mirror <NUM>.

In addition, the processor <NUM> may determine at least one of the scanning start location, the scanning direction, and the scanning path so that the focal length of the focus adjusting element <NUM> is changed in phases based on the location information of each of the plurality of objects.

Further, the processor <NUM> may determine at least one of the scanning start location, the scanning direction, and the scanning path so that a rotation operation of the scanning mirror <NUM> is minimized based on the location information of each of the plurality of objects. This will be described later in detail with reference to <FIG>.

When using the laser projector <NUM>, even if the focus adjusting element <NUM> is not provided, the AR information may be blurred in advance and then displayed on the display. In such a case, similar effect may be exhibited. However, a pixel which substantially passes through a crystalline lens to be formed on the retina corresponds to an extremely small laser point, thereby not corresponding to an optically blurred image. Accordingly, the user may feel awkward due to a difference between sharpness of the image actually formed on the retina and sharpness information of an image sent to the brain.

However, since the wearable device <NUM> according to an embodiment of the disclosure outputs an image via the focus adjusting element <NUM>, a size of light corresponding to one pixel distributed (or focused or formed) to the retina may be optically the same as in a situation where the object looks blurred in reality. Therefore, the user may smoothly recognize the AR information. In addition, the AR information other than the region where the user gazes is expressed blurred, thereby being effective in protection of the retina of the user.

As a result, in a case of the wearable device not including the focus adjusting element <NUM>, the user may feel awkward with respect to the depth (sense of distance or depth) of the virtual image or the AR information. However, when using the wearable device <NUM> according to an embodiment of the disclosure, it is possible to experience natural augment reality environment by reflecting the depth (sense of distance or depth) of the AR information.

<FIG> is a diagram illustrating an embodiment of displaying the AR information, when the focus adjusting element is provided.

The embodiment in which the wearable device does not include the focus adjusting element <NUM> has been described above with reference to <FIG>, but referring to <FIG>, the laser projector <NUM> of the wearable device may additionally include the focus adjusting element <NUM>.

A reference numeral <NUM> may denote an image seen through via a transparent display. A reference numeral <NUM> may denote an AR information image displayed on the transparent display by the wearable device. The image <NUM> may include the AR information <NUM> and <NUM> corresponding to each object or the additional information <NUM>. The laser projector <NUM> may use the light emitting element <NUM> to display the image <NUM> on the display. The wearable device may display the image <NUM> on the transparent display using the laser projector <NUM>, and an image finally displayed by the wearable device for the user may be an image <NUM>. The image <NUM> may be an image obtained by combining the image seen through the transparent display with the image displayed with the AR information.

Herein, the AR information included in the image <NUM> may be information corresponding to an image not subjected to the preprocessing operation.

When the laser projector <NUM> emits the image <NUM> not subjected to the preprocessing operation through the light emitting element <NUM>, the wearable device may perform control the laser light emitted through the light emitting element <NUM> to transmit through the focus adjusting element <NUM>.

Herein, the wearable device may identify the location of each object and position of the user's gaze. In addition, the wearable device may control the focus adjusting element <NUM> by considering the location of the object and the position of the user's gaze.

Further, the wearable device distinguishes the object at the position of the user's gaze and the object which is not. The wearable device controls the focus adjusting element <NUM> to sharply display the AR information corresponding to the object at the position of the user's gaze, and controls the focus adjusting element <NUM> to blurredly display the AR information corresponding to the object which is not at the position of the user's gaze.

There are various methods for a method for controlling the focus adjusting element <NUM> by the wearable device. The wearable device may adjust sharpness of the image by changing the refractive index of the focus adjusting element <NUM>. In the focus adjusting element <NUM> according to an embodiment of the disclosure, when a thickness of the lens changes, the refractive index may change. However, in the focus adjusting element <NUM> according to an embodiment of the disclosure, although the thickness of the lens is not changed, the refractive index may be changed by a change in material and internal component of the lens, and the sharpness of the image may be adjusted according to the change in refractive index.

As a result, the wearable device may transmit the image <NUM> not subjected to the preprocessing operation through the focus adjusting element <NUM> and make the user recognize each of AR information corresponding to each object with focal points different from each other.

The user may see an image displayed with the AR information obtained by considering both the location of the object and the position of the user's gaze through the transparent display. The final image <NUM> shown through the transparent display may be displayed with suitable AR information corresponding to each object and may be specifically an image obtained by reflecting sense of distance and depth. Accordingly, the user may be less confused with the sense of distance and depth.

Meanwhile, in the description with reference to <FIG>, it is described that the focus adjusting element <NUM> is disposed between the light emitting element <NUM> and the scanning mirror <NUM>, but the focus adjusting element <NUM> may exist at various positions according to another embodiment.

<FIG> and <FIG> are diagrams illustrating a situation where one pixel of a virtual image is formed on the retina.

<FIG> is a diagram illustrating an embodiment in which the wearable device <NUM> preprocesses the AR information, when the laser projector <NUM> does not include the focus adjusting element.

A reference numeral <NUM> may denote an image seen through the transparent display. An image <NUM> may be an AR information image displayed on the transparent display by the wearable device. The image <NUM> may include the AR information <NUM> and <NUM> corresponding to each object or the additional information <NUM>. The laser projector <NUM> may use the light emitting element <NUM> to display the image <NUM> on the display. The wearable device may display the image <NUM> on the transparent display using the laser projector <NUM>, and an image finally displayed by the wearable device for the user may be an image <NUM>. The image <NUM> may be an image obtained by combining the image seen through the transparent display with the image displayed with the AR information.

Herein, the AR information included in the image <NUM> may be information corresponding to an image subjected to the preprocessing operation. The preprocessing operation has been described above in detail with reference to <FIG>.

The wearable device <NUM> may preprocess the AR information and output an image obtained by changing sharpness of some AR information via the laser projector <NUM>. In this case, sizes of light rays corresponding to the image output from the laser projector <NUM> which are distributed (or focused or formed) to the retina of the user may be the same. Specifically, when the laser projector <NUM> does not include the focus adjusting element, the size of light corresponding to one pixel of the virtual image <NUM> including the AR information which is focused (or formed or distributed) on the retina may be the same. This may be a result caused by the properties of the laser. When the user recognizes an image output from the laser, the user may recognize the output image with the same sharpness regardless of distance information.

Meanwhile, even if the AR information is preprocessed according to the distance information of the objects, the user may recognize the AR information awkwardly. The image output via the laser may be unnatural to express the sense of distance and the user may experience eyestrain or dizziness.

In order to solve such a problem, the wearable device <NUM> may additionally include the focus adjusting element <NUM> in the laser projector <NUM>.

<FIG> is a diagram illustrating an embodiment in which the wearable device <NUM> displays the AR information, when the focus adjusting element <NUM> is additionally included.

A reference numeral <NUM> may denote an image seen through a transparent display. An image <NUM> may be an AR information image displayed on the transparent display by the wearable device. The image <NUM> may include the AR information <NUM> and <NUM> corresponding to each object or the additional information <NUM>. The laser projector <NUM> may use the light emitting element <NUM> to display the image <NUM> on the display. The wearable device may display the image <NUM> on the transparent display using the laser projector <NUM>, and an image finally displayed by the wearable device for the user may be an image <NUM>. The image <NUM> may be an image obtained by combining the image seen through the transparent display with the image displayed with the AR information.

Meanwhile, the portions <NUM> and <NUM> corresponding to some AR information in the image <NUM> may not be subjected to the preprocessing operation, unlike in <FIG>.

In the wearable device <NUM>, the focus adjusting element <NUM> may be disposed between the light emitting element <NUM> and the scanning mirror <NUM> and an image may be output.

In this case, the sizes of light rays corresponding to the image output by the laser projector <NUM> which are distributed to the retina of the user may vary according to the focus adjusting element <NUM>. The wearable device <NUM> may adjust the focus adjusting element <NUM> after determining whether to display the AR information of the object sharply, and in this case, a size of an image formed on the retina of the user may vary depending on the refractive index of the focus adjusting element.

For example, the AR information of the object corresponding to the position fixed with the user's gaze may be output via the laser projector <NUM> and a graphic pixel of the AR information as a virtual image may be formed on the retina of the user as a small image. Herein, a pixel of the AR information <NUM> separated from the user's gaze by approximately <NUM> may be formed on the retina of the user as a slightly larger image. In addition, a pixel of the AR information <NUM> separated from the user's gaze by approximately <NUM> may be formed on the retina of the user as a very larger image.

When comparing the sizes, the sizes of the images may be large in the order of the AR information <NUM>, <NUM>, and <NUM>. As the size of the image formed on the retina is larger, the user may recognize the corresponding part blurredly. As a result, the user may recognize the AR information <NUM> most sharply, recognize the AR information <NUM> blurredly to a moderate degree, and recognize the AR information <NUM> most blurredly.

The wearable device <NUM> may adjust the focus adjusting element <NUM> based on the location information of the object or the distance information from the wearable device <NUM>, thereby changing the size of the image formed on the retina. Since the user recognizes the sizes of the images differently, the user may recognize the AR information displayed on the transparent display naturally. The user who uses the wearable device <NUM> according to an embodiment of <FIG> may experience less eyestrain and dizziness.

<FIG> and <FIG> are diagrams illustrating a movement path of a laser reflected on a scanning mirror.

The laser projector <NUM> includes the scanning mirror <NUM> and the wearable device <NUM> may project light output from the light emitting element <NUM> through the scanning mirror <NUM> to various positions. The scanning mirror <NUM> may play a role of reflecting light and the wearable device <NUM> may project light to a desired position by adjusting an angle of the scanning mirror <NUM>.

The wearable device <NUM> may control the scanning mirror <NUM> to project light through a desired movement path.

<FIG> is a diagram illustrating an embodiment of one movement path through which the wearable device <NUM> projects light by controlling the scanning mirror <NUM>. Images <NUM>, <NUM>, and <NUM> herein may correspond to images <NUM>, <NUM>, and <NUM> of <FIG>, respectively.

The wearable device <NUM> may control the scanning mirror <NUM> of the laser projector <NUM> to display the AR information and move the scanning mirror <NUM> to any position of the display. The image <NUM> may be an image obtained by combining the image <NUM> including the AR information with the image <NUM> seen through the transparent display. The wearable device <NUM> may output the AR information <NUM>, <NUM>, and <NUM> using the laser projector <NUM> to combine the image <NUM>. In this case, the laser projector <NUM> may control the scanning mirror <NUM> to project the light to all of positions.

<FIG> illustrates that the light is projected to all of positions, but this is merely for illustrating the position where the light is able to be projected by the scanning mirror <NUM>, and the light may actually project only to a region displayed with the AR information <NUM>, <NUM>, and <NUM> corresponding to the image <NUM>. For example, the light is emitted by the light emitting element <NUM> in the region displayed with the AR information <NUM>, <NUM>, and <NUM> corresponding to the image <NUM>, but the light may not be emitted by the light emitting element <NUM> in a region which is not.

Meanwhile, the wearable device <NUM> according to another embodiment may emit light corresponding to visible light to a region displayed with the AR information <NUM>, <NUM>, and <NUM> corresponding to the image <NUM> and may emit light corresponding to invisible light that is not recognized by human eyes to a region not displayed with the AR information <NUM>, <NUM>, and <NUM> corresponding to the image <NUM>.

Meanwhile, <FIG> is a diagram illustrating another embodiment of the movement path through which the wearable device <NUM> projects light by controlling the scanning mirror <NUM>. Images <NUM>, <NUM>, and <NUM> herein may correspond to images <NUM>, <NUM>, and <NUM> of <FIG>, respectively.

Referring to <FIG>, the wearable device <NUM> may not move the scanning mirror <NUM> with respect to all possible positions. The wearable device <NUM> may move the scanning mirror <NUM> only for a region displayed with the AR information <NUM>, <NUM>, and <NUM> in the image <NUM>. The movement path illustrated in <FIG> may be a minimum movement path for passing through only the region displayed with the AR information <NUM>, <NUM>, and <NUM> in the image <NUM>.

The movement paths of the scanning mirror <NUM> illustrated in <FIG> and <FIG> are merely a description for some of various movement paths and various methods not illustrated may be applied in actual implementation.

Meanwhile, in <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>, the image seen through the transparent display and the image displayed with the AR information are separately described, but in actual implementation, the image may be implemented by a method for combining a captured image with AR information. Specifically, the wearable device <NUM> may capture a scene in front of the user, store it as one image, and newly generate an image related to AR information. The wearable device <NUM> may generate one image by combining the image obtained by capturing the scene in front of the user with the image related to the AR information. The wearable device <NUM> may display one combined image on the display.

<FIG> are diagrams illustrating an AR information displaying method according to each embodiment.

Referring to <FIG>, a plurality of objects <NUM> to <NUM> may be located in front of the user and the user may fix a gaze <NUM> on any one of the plurality of objects. In <FIG>, it is assumed that objects of a sign <NUM>, a bear <NUM>, a vehicle <NUM>, and a building <NUM> are located in front of the user. It is assumed that the user's gaze is fixed on the sign <NUM>. The assumption of <FIG> may be the same as in <FIG>.

<FIG> is a diagram illustrating an embodiment in which the wearable device <NUM> displays the AR information, when the laser projector <NUM> does not include the focus adjusting element <NUM>.

Each AR information corresponding to each object may include a type (classification), a name, and distance information from the wearable device <NUM>.

AR information <NUM> corresponding to the sign <NUM> object may include Sign, stop, and <NUM>. In other words, regarding the AR information <NUM>, the type may be stored as sign, the name may be stored as stop, and the distance information from the wearable device <NUM> to the object may be stored as <NUM>. When the wearable device <NUM> recognizes the sign <NUM> object, the wearable device <NUM> may obtain the AR information <NUM> corresponding to the sign <NUM> object, and in this case, the AR information <NUM> may be Sign, stop, and <NUM>.

AR information <NUM> corresponding to the bear <NUM> object may include animal, bear, and <NUM>. In other words, regarding the AR information <NUM>, the type may be stored as animal, the name may be stored as bear, and the distance information from the wearable device <NUM> to the object may be stored as <NUM>. When the wearable device <NUM> recognizes the bear <NUM> object, the wearable device <NUM> may obtain the AR information <NUM> corresponding to the bear <NUM> object, and in this case, the AR information <NUM> may be animal, bear, and <NUM>.

AR information <NUM> corresponding to the vehicle <NUM> object may include Vehicle, Samsung, and <NUM>. In other words, regarding the AR information <NUM>, the type may be stored as vehicle, the name may be stored as Samsung, and the distance information from the wearable device <NUM> to the object may be stored as <NUM>. When the wearable device <NUM> recognizes the vehicle <NUM> object, the wearable device <NUM> may obtain the AR information <NUM> corresponding to the vehicle <NUM> object, and in this case, the AR information <NUM> may be vehicle, Samsung, and <NUM>.

AR information <NUM> corresponding to the building <NUM> object may include building, abc, and <NUM>. In other words, regarding the AR information <NUM>, the type may be stored as building, the name may be stored as abc, and the distance information from the wearable device <NUM> to the object may be stored as <NUM>. When the wearable device <NUM> recognizes the building <NUM> object, the wearable device <NUM> may obtain the AR information <NUM> corresponding to the building <NUM> object, and in this case, the AR information <NUM> may be building, abc, and <NUM>.

Referring to <FIG>, when the focus adjusting element <NUM> is not provided, the wearable device <NUM> may display all of the AR information <NUM>, <NUM>, <NUM>, and <NUM> corresponding to the objects, respectively, sharply regardless of the stored distance information.

However, if the wearable device <NUM> displays all of the AR information <NUM>, <NUM>, <NUM>, and <NUM> sharply, the user may experience eyestrain and confusion due to a difference in sense of distance between the object not fixed with the user's gaze and the corresponding AR information. Specifically, if the user's gaze is fixed on the sign at a distance of <NUM> from the wearable device <NUM>, the user may see the building <NUM> at a distance of <NUM> from the wearable device <NUM> blurredly. If the AR information <NUM> corresponding to the building <NUM> looks sharply while the building <NUM> looks blurredly, the user may experience eyestrain and may be confused with the sense of distance.

Therefore, the wearable device <NUM> may solve the above problem by using the focus adjusting element <NUM>.

<FIG> and <FIG> are diagrams illustrating embodiments in which the wearable device <NUM> displays the AR information using the focus adjusting element <NUM>.

Referring to <FIG> and <FIG>, the wearable device <NUM> may change a size of an image formed on the retina of the user and sharpness of the image using the focus adjusting element <NUM>. The user may see the AR information sharply or blurredly according to a change of the focus adjusting element <NUM>.

<FIG> is a diagram illustrating an embodiment of displaying the AR information with two levels of sharpness using the distance information from the wearable device <NUM> to the object.

Referring to <FIG>, the user's gaze may be fixed on the sign object and the wearable device <NUM> may control the focus adjusting element <NUM> to display the AR information <NUM> corresponding to the sign sharply.

The wearable device <NUM> may control the focus adjusting element <NUM> to display the AR information <NUM>, <NUM>, and <NUM> corresponding to the object not fixed with the user's gaze blurredly.

Herein, the wearable device <NUM> may control the focus adjusting element <NUM> to have the same blur level.

Referring to <FIG>, the wearable device <NUM> may display the AR information corresponding to the object according to the distance information of each object. Herein, the wearable device <NUM> may display sharpness differently for each AR information. It is assumed that the user's gaze is fixed on the sign. In addition, as assumed above with reference to <FIG>, the distance to the sign may be <NUM>, the distance to the bear may be <NUM>, the distance to the vehicle may be <NUM>, and the distance to the building may be <NUM>.

In this case, the wearable device <NUM> may identify the distance information of each object and identify a focal length difference. The focal length difference may refer to a difference between a distance from the wearable device <NUM> to a position of the user's gaze and a distance from the wearable device <NUM> to the object. The wearable device <NUM> may control the focus adjusting element <NUM> using the focal length difference.

Since the user's gaze is fixed on the sign and the sign is separated from the wearable device <NUM> by <NUM>, a standard of the focal length difference may be <NUM>. A difference in distance between the sign and the focal point is <NUM>, and accordingly, the focal length difference of the sign may be <NUM>. A difference in distance between the bear and the focal point is <NUM>, and accordingly, the focal length difference of the bear may be <NUM>. In addition, a difference in distance between the vehicle and the focal point is -<NUM>, and accordingly, the focal length difference of the vehicle may be -<NUM>. Further, a difference in distance between the building and the focal point is <NUM>, and accordingly, the focal length difference of the building may be <NUM>.

The wearable device <NUM> may control the focus adjusting element <NUM> to display the AR information with different sharpness using the focal length difference. Referring to <FIG>, when the focal length difference is <NUM>, the wearable device <NUM> may sharply display the AR information corresponding to corresponding object. When the focal length difference is not <NUM>, the wearable device <NUM> may display the AR information corresponding to the corresponding object blurredly at the same level (blur level <NUM> in <FIG>).

Meanwhile, the wearable device <NUM> may display different sharpness for each object using the focal length difference and may display the AR information of the objects, as illustrated in <FIG>. Specifically, when the refractive index of the focus adjusting element <NUM> included in the wearable device <NUM> is changed, the focal length of the output light may change. When the focal length changes according to the changed refractive index of the focus adjusting element <NUM>, the user may recognize the sharpness of the AR information corresponding to the object differently.

Referring to <FIG>, the wearable device <NUM> may sharply display the AR information <NUM> corresponding to the sign object having the focal length difference of <NUM>. The wearable device <NUM> may display the AR information <NUM>, <NUM>, and <NUM> corresponding to the objects having the focal length difference, which is not <NUM>, at blur level <NUM>.

Meanwhile, <FIG> is a diagram illustrating an embodiment of displaying the AR information at a plurality of levels of sharpness using the distance information from the wearable device <NUM> to the object. Herein, the distance information may refer to a distance from the wearable device <NUM> to the object.

As described above with reference to <FIG>, the wearable device <NUM> may obtain the focal distance difference of each object. In <FIG>, the sharpness is divided into two by dividing the object with the focal length difference of <NUM> and the object with the focal length difference, which is not <NUM>, but referring to <FIG>, the sharpness may be specifically divided by dividing the focal length difference in a certain unit.

For example, as illustrated in <FIG>, if the focal length difference (absolute value) is <NUM> to <NUM>, the sharpness may be identified as clear, if the focal length difference is <NUM> to <NUM>, the sharpness may be identified as blur level <NUM>, if the focal length difference is <NUM> to <NUM>, the sharpness may be identified as blur level <NUM>, and if the focal length difference is <NUM> to <NUM>, the sharpness may be identified as blur level <NUM>. Herein, a higher number of level may correspond to the sharpness with blur. When the focal length difference (absolute value) is greater than <NUM>, the sharpness may be identified as none and may be set to not display the AR information at all. Herein, the wearable device <NUM> may control the focus adjusting element <NUM> or control the light emitting element <NUM> to not display the AR information at all.

Herein, the reason for using the focal length difference as an absolute value is because a difference in distance between the focal point (gaze) and the object is actually meaningful, and a difference between a negative value of the distance difference and a positive value of the distance difference may not be generally distinguishable by the user. However, in actual implementation, the sharpness may be set different in order to distinguish the negative value of the distance difference from the positive value of the distance difference.

Meanwhile, as the control method according to another embodiment of the disclosure, if the focal length difference is <NUM> or greater, the wearable device <NUM> may control to not generate the.

The wearable device <NUM> may display the AR information based on the sharpness specified in <FIG> and this is illustrated in <FIG>.

Referring to <FIG>, the wearable device <NUM> may display the AR information <NUM> corresponding to the sign sharply. The wearable device <NUM> may display the AR information <NUM> and <NUM> corresponding to the bear and the vehicle having the sharpness of the blur level <NUM> blurredly at the level <NUM>. In addition, the wearable device <NUM> may display the AR information <NUM> corresponding to the building having the sharpness of a blur level <NUM> blurredly at the level <NUM>.

In the description with reference to <FIG> and <FIG>, the sharpness is specified with respect to the focal length difference based on certain ranges, but in actual implementation, the wearable device <NUM> may divide each sharpness according to the value of the focal length difference. For example, when the focal length difference is <NUM> to <NUM>, the wearable device <NUM> may identify the sharpness as <NUM> to <NUM>.

<FIG> are diagrams illustrating an amount of change of the focus adjusting element according to each embodiment.

<FIG> are diagrams illustrating an amount of change of the focus adjusting element according to a movement path of the scanning mirror <NUM>. The movement path of the scanning mirror <NUM> may refer to a path through which light output by the light emitting element <NUM> and reflected by the scanning mirror <NUM> is projected.

Referring to <FIG>, it is assumed that <NUM> second is taken to move from one square to another square horizontally or vertically, for convenience of calculation. In actual implementation, it may took not the same time during the movement from one square to another square and delay may occur in a specific section.

Referring to <FIG>, the wearable device <NUM> may control the scanning mirror <NUM> so that the flow path of projected light starts from a lower left end to the right and moves through all sections. In this case, the time taken for the scanning mirror to move to project light to all of the sections may be approximately <NUM> seconds.

Although the scanning mirror <NUM> moves to project light to all of the sections, the wearable device <NUM> may output light through the light emitting element <NUM>, only when displaying the AR information. Accordingly, the wearable device <NUM> may control the laser projector <NUM> not to actually project light in a section where the AR information is not displayed.

In this case, the focal length difference of the section not displayed with the AR information may be set to maintain the original focal point. This is for minimizing the change of the focus adjusting element <NUM> since the section not displayed with the artificial intelligence is not related to the change of the focus adjusting element <NUM>.

The amount of change of the focus adjusting element in the flow path of the scanning mirror <NUM> according to <FIG> may be as in <FIG>. The amount of change of the focus adjusting element may be identified by the amount of change of the focal length difference.

Referring to <FIG>, the focal length difference may be repeatedly changed between <NUM> seconds to <NUM> seconds. The time unit has been described above as a unit of one second, for convenience of calculation, but the time unit in practice may be implemented as <NUM>^-<NUM> or <NUM>^-<NUM>. Accordingly, when the focal length difference is repeatedly changed during a short period of time, the focus adjusting element <NUM> also has to be changed during a short period of time. In such a case, the wearable device <NUM> may essentially use high-speed focus adjusting element <NUM>.

Referring to <FIG>, a start position of the projection of light is a lower left end, and the movement path may reciprocate vertically to project light to all of sections. As described above with reference to <FIG>, it may take <NUM> seconds until the scanning mirror <NUM> finishes all of the movement.

When the amount of change of the focus adjusting element <NUM> is identified according to <FIG>, it may be the same as illustrated in <FIG>.

Referring to <FIG>, it may be confirmed that the focal length difference is changed during a short period of time in the vicinity of <NUM> seconds, and the wearable device <NUM> may have to use the high-speed focus adjusting element <NUM>.

<FIG> is a diagram illustrating an embodiment in which the wearable device <NUM> controls the movement path of the scanning mirror <NUM> to minimize the amount of change of the focus adjusting element <NUM>. The wearable device <NUM> may control the scanning mirror <NUM> to not project light to the entire path but to project light only to a necessary portion, unlike in <FIG> and <FIG>.

In order to minimize the amount of change of the focus adjusting element <NUM>, the wearable device <NUM> may move the scanning mirror <NUM> for each region with the same focal length difference. In addition, the wearable device <NUM> may move the scanning mirror <NUM> from a portion with a small focal length difference to a portion with a great focal length difference or from a portion with a great focal length difference to a portion with a small focal length difference. In this case, a total movement time may be <NUM> seconds.

When controlling the scanning mirror <NUM> along the movement path of <FIG>, the amount of change of the focus adjusting element <NUM> may be the same as in <FIG>. Referring to <FIG>, the focal length difference may be changed at a specific time. However, the focal adjusting element <NUM> may not be repeatedly changed, since the focal length difference changed once is maintained. For example, in the embodiment of <FIG>, the wearable device <NUM> may use the normal focus adjusting element without using the high-speed focus adjusting element. In this case, the cost of components of the wearable device <NUM> may be reduced.

<FIG> is a diagram illustrating an embodiment in which the wearable device <NUM> controls the scanning mirror <NUM> to minimize the movement path of the scanning mirror <NUM>. In order to minimize the movement path of the scanning mirror <NUM>, the wearable device <NUM> may consider only the region displayed with the AR information simply, regardless of the change of the focus adjusting element <NUM>. In this case, the total time required may be <NUM> seconds.

When controlling the scanning mirror <NUM> in the movement path according to <FIG>, the amount of change of the focus adjusting element <NUM> may be the same as in <FIG>. Referring to <FIG>, the focal length difference may be changed at a specific time. However, the focal adjusting element <NUM> may not be repeatedly changed, since the focal length difference changed once is maintained. In this case, since the focus adjusting element <NUM> is not repeatedly changed during a short period of time, the wearable device <NUM> may use the normal focus adjusting element <NUM>, and this may lead to cost reduction.

In a case of <FIG>, the movement path of the scanning mirror <NUM> was not controlled in the order from a small focal length difference to a great focal length difference, and the scanning mirror <NUM> was controlled only to minimize the movement path of the scanning mirror <NUM>. Accordingly, a degree of change of the focus adjusting element <NUM> may be greater than in the embodiment of <FIG>.

The wearable device <NUM> may control the movement path of the scanning mirror <NUM> to minimize the amount of change of the focus adjusting element <NUM> and may control the movement path of the scanning mirror <NUM> to minimize the movement path of the scanning mirror <NUM>.

Meanwhile, in actual implementation, the wearable device <NUM> may control the scanning mirror <NUM> to select an optimal method by identifying both cases described above. The wearable device <NUM> may apply a weight to a main item to select an optimal method automatically. In addition, in actual implementation, the wearable device <NUM> may provide a UI for guiding the user to select one of various methods.

Meanwhile, the wearable device <NUM> may have to select any one of the two methods. In this case, since a driving speed of the focus adjusting element <NUM> may be lower than a driving speed of the scanning mirror <NUM>, the usage of the method for minimizing the amount of change of the focus adjusting element <NUM> may be effective than the method for minimizing the movement path of the scanning mirror <NUM>.

<FIG> is a flowchart illustrating a method for controlling a wearable device according to an embodiment.

Meanwhile, according to an embodiment of the disclosure, in a method for controlling a wearable device including a laser projector including a light emitting element configured to emit laser light, a focus adjusting element configured to adjust a focal point of the emitted laser light, and a scanning mirror configured to control a scanning direction of the light with the adjusted focal point, the user's gaze may be identified based on sensing data obtained by a sensor (S1905).

In addition, location information of at least one object included in an image obtained by a camera may be identified (S1910).

In addition, a driving state of the focus adjusting element may be controlled based on the identified user's gaze and location information of the object (S1915).

In addition, augmented reality (AR) information related to the object may be provided on a display based on the identified user's gaze and location information of the object (S1920).

Herein, the step S1915 may include identifying whether the location of the object corresponds to the user's gaze by comparing the user's gaze with the location information of the object, and controlling a driving state of the focus adjusting element based on the identified result.

In addition, the step S1920 may include, based on the location of the object corresponding to the location of the user's gaze, providing AR information related to the object on the display by controlling a focal length of the focus adjusting element to be a first focal length, and based on the location of the object not corresponding to the location of the user's gaze, providing the AR information related to the object on the display by controlling the focal length of the focus adjusting element to be a second focal length.

Herein, the first focal length may be a unique focal length of the focus adjusting element and the second focal length may be different from the unique focal length.

In addition, the method for controlling the wearable device may further include identifying location information of each of a plurality of objects included in the obtained image, and the controlling the driving state of the focus adjusting element may include, based on first AR information related to a first object corresponding to the location of the user's gaze among the plurality of objects being provided, controlling the focal length of the focus adjusting element to be the first focal length, and based on second AR information related to a second objet not corresponding to the location of the user's gaze among the plurality of objects being provided, controlling the focal length of the focus adjusting element to be the second focal length.

The step S1915 may include, based on the second AR information being provided, controlling the focal length of the focus adjusting element to be the second focal length based on depth information of the second object.

The step S1915 may include obtaining a difference between a depth value included in the depth information of the second object and a depth value corresponding to the identified user's gaze, and controlling the focal length of the focus adjusting element to be the second focal length based on the obtained difference of depth values.

The control method may further include, based on the plurality of objects being included in the obtained image, controlling the scanning mirror by determining at least one of a scanning start location, a scanning direction, and a scanning path based on the location information of each of the plurality of objects.

The controlling the scanning mirror may include determining at least one of the scanning start location, the scanning direction, and the scanning path to change the focal length of the focus adjusting element in phases based on the location information of each of the plurality of objects.

The method for controlling the wearable device as in <FIG> may be executed on the wearable device having the configuration of <FIG> and may also be executed on a wearable device having other configurations.

Meanwhile, the methods according to the embodiments of the disclosure described above may be implemented in a form of an application installable on the wearable device of the related art.

In addition, the methods according to the embodiments of the disclosure described above may be implemented simply by the software upgrade or hardware upgrade in the wearable device of the related art.

Further, the embodiments of the disclosure described above may be performed through an embedded server provided in the electronic device or an external server of the electronic device.

Meanwhile, the method for controlling the wearable device according to the embodiment may be implemented as a program and provided in the wearable device. In particular, the program including the method for controlling the wearable device may be stored and provided in a non-transitory computer-readable medium.

In addition, the embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. According to the implementation in terms of hardware, the embodiments of the disclosure may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and electronic units for executing other functions. In some cases, the embodiments described in this specification may be implemented as a processor <NUM> itself. According to the implementation in terms of software, the embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described in this specification.

Computer instructions for executing processing operations on the wearable device according to the embodiments of the disclosure descried above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by the processor, the computer instructions may enable a specific machine to execute the processing operations on the wearable device according to the embodiments described above.

Claim 1:
A wearable device (<NUM>) comprising:
a camera (<NUM>);
a sensor (<NUM>);
a display (<NUM>);
a laser projector (<NUM>); and
a processor (<NUM>) configured to,
identify a user's gaze (<NUM>) based on sensing data obtained by the sensor (<NUM>),
identify location information of at least one object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) included in an image obtained by the camera (<NUM>),
distinguish an object (<NUM>, <NUM>) being at the position of the user's gaze (<NUM>), and an object (<NUM>, <NUM>, <NUM>, <NUM>) being not at the position of the user's gaze (<NUM>),
identify a focal length difference between the distinguished objects (<NUM>, <NUM> and <NUM>, <NUM>) to the wearable device (<NUM>), and
control the laser projector (<NUM>) to provide augmented reality, AR, information (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) related to the object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) through the display (<NUM>) based on the identified user's gaze (<NUM>) and location information of the object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>),
wherein the laser projector (<NUM>) comprises:
a light emitting element (<NUM>) configured to emit laser light;
a focus adjusting element (<NUM>) configured to adjust a focal point of the emitted laser light; and
a scanning mirror (<NUM>) configured to control a scanning direction of the light with the adjusted focal point, and
wherein the processor (<NUM>) is configured to:
control a driving state of the focal adjusting element (<NUM>) based on the identified user's gaze (<NUM>), location information of the object (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) and the focal length difference,
control the focus adjusting element (<NUM>) to sharply display the AR information (<NUM>, <NUM>) corresponding to the object (<NUM>) at the position of the user's gaze (<NUM>),
control the focus adjusting element (<NUM>) to blurredly display the AR information (<NUM>, <NUM>, <NUM>, <NUM>) corresponding to an object (<NUM>, <NUM>, <NUM>, <NUM>) which is not at the position of the user's gaze (<NUM>) with different sharpness based on the focal length difference, characterized in that the AR information (<NUM>, <NUM>, <NUM>, <NUM>) is not displayed, when an absolute value of the focal length difference exceeds a predetermined value, and
a movement path of the scanning mirror (<NUM>) is controlled to project light only to sections necessary for displaying the AR information (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>).