Patent Description:
Augmented reality is a type of virtual reality which combines the real world seen by a user and a virtual world, and shows the combination as a combined image. Augmented reality, which is a concept of complementing the real word with a virtual world, uses a virtual environment generated by computer graphics, but is based on the real environment. Here, computer graphics perform the role of additionally providing images or information to augment the real environment. That is, computer graphics overlap a three-dimensional virtual image with an actual image that a user is viewing, and thereby make reduce a distinction between the real environment and the virtual screen.

In this regard, a technology of rendering a virtual object in an empty space of an actual image by using an augmented reality technology is being generalized. In a case in which an object exists in an actual image, a complex operation and an apparatus with a high technical specification are required for rendering a virtual object in the place wherein the object is located. Also, a problem exists, which is that the size of a space wherein an object is located cannot be precisely identified according to the photographing angle, the photographing location, etc. even though a complex operation was performed. <NPL>, and <CIT> describe examples of the related art.

The example embodiments of the disclosure provide an electronic apparatus which improves convenience and accuracy of manipulation by providing a guide to a user, reduces operation complexity, and thereby provides an augmented reality image with a low specification, and a control method thereof.

Hereinafter, various embodiments of the disclosure will be described with reference to the accompanying drawings. However, it should be noted that the various embodiments are not for limiting the technology described in the disclosure to a specific embodiment, but they should be interpreted to include various modifications and/or alternatives of the embodiments of the disclosure. Also, with respect to the detailed description of the drawings, similar components may be designated by similar reference numerals.

In the disclosure, expressions such as "have," "may have," "include," and "may include" should be construed as denoting that there are such characteristics (e.g., elements such as numerical values, functions, operations, and components), and the expressions are not intended to exclude the existence of additional characteristics.

Also, in the disclosure, the expressions "A or B," "at least one of A and/or B," or "one or more of A and/or B," and the like, may include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" may referto all of the following cases: (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B.

In addition, the expressions "first," "second," and the like, used in the disclosure may be used to describe various elements regardless of any order and/or degree of importance. Also, such expressions are used to distinguish one element from another element, and are not intended to limit the elements.

Further, the description in the disclosure that one element (e.g., a first element) is "(operatively or communicatively) coupled with/to" or "connected to" another element (e.g., a second element) should be interpreted to include both the case where the one element is directly coupled to the another element, and the case where the one element is coupled to the another element through still another element (e.g., a third element). In contrast, the description that one element (e.g., a first element) is "directly coupled" or "directly connected" to another element (e.g., a second element) can be interpreted to mean that still another element (e.g., a third element) does not exist between the one element and the another element.

Also, the expression "configured to" used in the disclosure may be interchangeably used with other expressions such as "suitable for," "having the capacity to," "designed to," "adapted to," "made to," and "capable of," and the like. The term "configured to" may not necessarily mean that a device is "specifically designed to" in terms of hardware. Instead, under some circumstances, the expression "a device configured to" may mean that the device "is capable of" performing an operation together with another device or component. For example, the phrase "a sub-processor configured to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

In the disclosure, the term "user" may refer to a person who uses an electronic apparatus or an apparatus using an electronic apparatus (e.g., an artificial intelligence electronic apparatus).

Also, the term "subject" used in the disclosure may mean an object which is the subject of photographing. That is, the term may mean an object which is in a state of being photographed by a camera and not displayed on a display. Also, the term "object" may mean an object included in a photographed image. That is, the term may mean a state wherein a subject photographed by a camera is displayed on a display. Also, the term "virtual object" may mean an object that does not actually exist, but is generated as if it exists in a photographed image.

Hereinafter, the disclosure will be described in detail with reference to the drawings.

<FIG> is a diagram for schematically illustrating an electronic apparatus according to an embodiment.

The electronic apparatus <NUM> may photograph a space around a user through a camera. In this case, the electronic apparatus <NUM> provides an image corresponding to the space around the user to the user.

The electronic apparatus <NUM> may replace an object <NUM> included in the photographed image with a virtual object <NUM> and generate an augmented reality image. Here, the object corresponds to an object that actually exists in a space around the user, and the virtual object <NUM> corresponds to a virtual object that does not exist in a space around the user. For example, in case in which an air conditioner <NUM> is included in an image photographed by the electronic apparatus <NUM>, the electronic apparatus <NUM> may replace the air conditioner <NUM> included in the photographed image with a virtual air purifier <NUM> that does not exist in a space around the user, generate an augmented reality image, and provide the generated augmented reality image to the user.

In <FIG>, a smartphone is illustrated as the electronic apparatus <NUM>, but the electronic apparatus <NUM> is not necessarily limited thereto. In the disclosure, the electronic apparatus <NUM> may be implemented as an apparatus equipped with a display function such as a television (TV), a tablet personal computer (PC), a portable media player (PMP), a personal digital assistance (PDA), a laptop, a smart watch, a head mounted display (HMD), and a near eye display (NED). Also, the electronic apparatus <NUM> may include various forms of displays <NUM> to provide a display function.

The example embodiments of the disclosure may also be implemented through an electronic apparatus that is not equipped with a display function. For example, various types of electronic apparatuses providing an image to an external apparatus such as a Blu-ray player, a digital versatile disc (DVD) player, a streaming content output apparatus, and a set-top box may implement the various embodiments of the disclosure. As another example, various forms of home appliances such as a speaker, a refrigerator, a washing machine, an air conditioner, an air purifier, and various kinds of Internet of Things apparatuses may implement the various embodiments of the disclosure. Hereinafter, for convenience of explanation, explanation will be made based on the assumption that the electronic apparatus <NUM> is a user terminal apparatus equipped with a camera <NUM> and a display <NUM>.

<FIG> is a flow chart for illustrating a control method of an electronic apparatus according to an embodiment.

First, in the control method of an electronic apparatus according to an embodiment of the disclosure, an image is photographed at operation S210. Here, the electronic apparatus <NUM> may photograph an area opposing the electronic apparatus <NUM> via a camera arranged on the rear surface of the electronic apparatus <NUM>.

Then, a first guideline and the photographed image are displayed together at operation S220. Specifically, the electronic apparatus <NUM> may provide a guide such that the corner of a photographed object is placed on the first guideline by using the first guideline, and thereby induce a userto photograph a subject based on the first guideline when photographing a subject. Alternatively, the electronic apparatus <NUM> may provide a guide such that an object is photographed while the corner of the photographed object is in parallel to the first guideline. As a user may easily photograph a subject in an accurate angle and an accurate direction by using the first guideline, and the electronic apparatus <NUM> photographs an object while one surface of the object is in line with the first guideline, a process of an operation applying an augmented reality (AR) technology may be simplified. A method of simplifying a process of an operation will be described in detail later with reference to <FIG>.

Here, the first guideline is a line displayed on the display <NUM>, and is a user interface (Ul) for specifying the location of an object included in an image. Specifically, the first guideline is a straight line for guiding such that the location of an object included in an image can be the front surface of the object. Also, on the <NUM>/<NUM> point of the first guideline, a line orthogonal to the first guideline is placed, and the center of the object may be made to coincide with the line orthogonal to the first guideline. That is, the first guideline may be a line for guiding such that the camera is toward the front surface of an object and a vertical center axis of the object can be placed in the center of the display <NUM>. However, this is merely an embodiment, and a location wherein the first guideline is located and whether there is a line orthogonal to the first guideline may vary.

Then, the electronic device <NUM> determines whether an object was recognized in the photographed image based on the first guideline at operation S230. In this regard, the electronic apparatus <NUM> may be set such that an object is recognized in the photographed image. Here, an object corresponds to a subject that exists in reality, and if a subject is photographed by a camera, an object may be included in the photographed image. Also, an object is replaced with a virtual object in an augmented reality image, and may be various kinds of home appliances or external electronic apparatuses. For example, an object may be home appliances or external electronic apparatuses such as a TV, a monitor, a washing machine, a refrigerator, and an air conditioner. In case the electronic apparatus <NUM> photographs an image in real time and displays the image on the display <NUM>, the electronic apparatus <NUM> may recognize an object existing on the display <NUM> based on the first guideline. That is, the electronic apparatus <NUM> may recognize the size of an object or an area wherein an object exists based on the first guideline.

In addition to the first guideline, a second guideline guiding a photographing angle of a camera may be generated. The second guideline indicates a UI for guiding such that a camera photographs an object at an angle parallel to the object. For example, in case a photographing angle of a camera is not perpendicular to the ground surface, i.e., in case a camera does not photograph an object at an angle parallel to the object, the second guideline may be displayed, and in case a camera is at an angle parallel to an object, the second guideline may not be displayed.

Also, according to a photographing angle of a camera, the photographing angle of the camera may be guided by changing the length of the second guideline. For example, the length of the second guideline may be longer in a case wherein a camera faces an object from the upside than in a case wherein a camera faces an object from the downside.

If a subject is photographed while being aligned with the first guideline, and the electronic apparatus <NUM> recognizes an object in the photographed image based on the first guideline at operation S230-Y, the electronic apparatus <NUM> renders a virtual object replacing the object at operation S240.

Here, a virtual object may be an object of the same kind as the recognized object, but is not necessarily limited thereto. That is, a virtual object may be various kinds of home appliances or external electronic apparatuses. Also, a virtual object may be a still object, or a moving object.

In addition, a virtual object may be an object selected by a user. Specifically, if an object is recognized, a list including a plurality of virtual objects that can replace the recognized object may be displayed, and if a user input selecting one of the plurality of virtual objects is received, a virtual object corresponding to the user input may be rendered.

The control method of the disclosure may render a virtual object in consideration of the size of a recognized object and the surrounding blank space of the recognized object. Specifically, the length of a blank space existing in an adjacent area to the recognized object is identified, and it may be identified whether a virtual object can be located in the area wherein the object is located based on the length information of the object and the identified length of the blank space. Then, if it is identified that a virtual object can be located in the area wherein the object is located, a virtual object may be rendered.

Then, the electronic apparatus <NUM> generates an augmented reality image wherein a rendered virtual object is located in an area wherein an object is located in the photographed image at operation S250.

Specifically, based on the location information and the angle information of a camera photographing an image, the location of a virtual object in an augmented reality image may be identified, and an augmented reality image wherein a virtual object is arranged in the identified location may be generated.

To be more specific, based on the location information and the angle information of a camera, a movement value and a rotation value of a recognized object may be identified, and based on the movement value and the rotation value of the object, the size of a virtual object may be calculated and the location of the virtual object may be identified.

It may be identified whether a rendered virtual object can cover an object area inside an image. Here, the feature that a rendered virtual object can cover an object area means that a rendered virtual object is overlapped with an object in an image, and a portion of the object is not shown.

In a case in which a partial area of an object not covered by a virtual object exists, the partial object area not covered by the virtual object may be replaced with the background area around the partial object area and an augmented reality image may be generated.

Then, the electronic apparatus <NUM> may display the generated augmented reality image at operation S260.

After the generated augmented reality image is displayed, if an event occurs, a virtual object indicating a movement during a driving operation of a virtual object may be rendered. Here, an event may mean an operation of touching or dragging the rendered virtual object. For example, in case a user touches a virtual object (e.g., a refrigerator) in the generated augmented reality image for a predetermined time period (e.g., two seconds), a virtual object indicating a movement during a driving operation of the virtual object (e.g., opening of the door of the refrigerator) may be rendered.

A user input moving a virtual object included in an augmented reality image may be received. In this case, the virtual object may be rendered while the location of the virtual object in the photographed image is changed based on the user input. For example, in case a virtual object corresponding to a dryer was rendered beside a washing machine, but a user input moving the virtual object onto the washing machine was received, the virtual object corresponding to the dryer may be rendered to be located on the washing machine.

If the location information of the camera is changed while an augmented reality image including a virtual object is displayed, the virtual object may be rendered while being changed based on the changed location of the camera. For example, in case the location of the camera moves to the left and right sides of an object while an augmented reality image including a virtual object is displayed, the virtual object displayed on the image may be rendered while being changed to a virtual object corresponding to a left side surface or a right side surface of the virtual object.

Then, an augmented reality image including the rendered virtual object may be displayed.

<FIG> is a block diagram for illustrating a configuration of an electronic apparatus according to an embodiment of the disclosure.

Referring to <FIG>, the electronic apparatus <NUM> includes a camera <NUM>, a display <NUM>, and a processor <NUM>. However, examples of the electronic apparatus <NUM> not in accordance with the present invention do not include a camera <NUM>, a display <NUM>, and a processor <NUM>. In various embodiments not in accordance with the present invention, some components among the display <NUM> and the processor <NUM> are omitted. In some embodiments the camera may be omitted.

The camera <NUM> is an apparatus that may photograph a still image or a moving image, and may include at least one image sensor (e.g., a front surface sensor or a rear surface sensor), a lens, an image signal processor (ISP), and a flash component (e.g., a light emitting diode (LED), a xenon lamp, etc.).

The camera <NUM> may be located on the front surface or the rear surface of the electronic apparatus <NUM>. In the disclosure, explanation will be made based on the assumption that an image is photographed with a camera placed on the rear surface of the electronic apparatus <NUM>, but the disclosure is not necessarily limited thereto, and an image may be photographed with a camera placed on the front surface of the electronic apparatus <NUM>.

The camera <NUM> according to an embodiment may photograph a random subject according to control of the processor <NUM>, and transmit the photographed data to the processor <NUM>. The photographed data may be stored in the memory <NUM> according to control of the processor <NUM>. Here, the photographed data may referred to as a picture, an image, a still image, and a moving image, but hereinafter, the data will be generally referred to as an image for the convenience of explanation. Here, a subject may mean an object which is the subject of photographing. An image according to the various embodiments of the disclosure may mean an image received from an external apparatus or an external server, or an image stored in the memory <NUM>, etc., other than a live view image photographed through the camera <NUM>.

According to an embodiment, the processor <NUM> may display an augmented reality screen through the display <NUM> based on an image photographed through the camera <NUM>.

The display <NUM> may provide various content screens that can be provided through the electronic apparatus <NUM>. Here, a content screen may include various content such as an image, a moving image, texts, music, an application execution screen, a graphic user interface (GUI) screen, etc..

The display <NUM> may be implemented as various forms of displays such as a liquid crystal display (LCD), an organic light emitting diodes (OLED) display, a plasma display panel (PDP), a wall, and a micro LED. In the display <NUM>, a driving circuit that may be implemented in forms such as an amorphous silicon (A-Si) thin-film transistor TFT, a low temperature poly silicon (LTPS) TFT, and an organic TFT (OTFT), a backlight unit, etc. may also be included together. The display <NUM> may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc..

Also, the display <NUM> according to an embodiment may include a display panel configured to output an image, and a bezel housing a display panel. In particular, a bezel according to an embodiment may include a touch sensor (not shown) for detecting a user interaction.

In particular, the display <NUM> may display an image photographed in real time through the camera <NUM> according to control of the processor <NUM> (e.g., a live view image), an augmented reality (AR) image to which a virtual object is added to the image, or an image acquired by the processor <NUM> based on the image, etc., as will be described below. Here, a virtual object may be referred to as an AR object, a virtual object image, etc., but hereinafter, it will be generally referred to as a virtual object for the convenience of explanation.

The processor <NUM> may be electronically connected with the memory <NUM>, and control the overall operations and functions of the electronic apparatus <NUM>. For example, the processor <NUM> may drive an operating system or an application program, control hardware or software components connected to the processor <NUM>, and perform various kinds of data processing and operations. Also, the processor <NUM> may load an instruction or data received from at least one of other components on a volatile memory and process it, and store various kinds of data in a non-volatile memory.

The processor <NUM> may be implemented as a dedicated processor (e.g., an embedded processor) for performing the corresponding operation or a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory device.

In the disclosure, the processor <NUM> may be implemented as a digital signal processor (DSP) processing digital signals, a microprocessor, and a time controller (TCON). However, the disclosure is not limited thereto, and the processor <NUM> may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU) or a communication processor (CP), and an ARM processor, or may be defined by the terms. Also, the processor <NUM> may be implemented as a system on chip (SoC) having a processing algorithm stored therein or large scale integration (LSI), or in the form of a field programmable gate array (FPGA).

The processor <NUM> may recognize an object in an image photographed through a camera. Here, an object may indicate specific home appliances such as a TV, a monitor, a washing machine, a refrigerator, and an air conditioner, and such information on a recognizable object may be stored in the memory <NUM> in advance. An object may mean an object located on the display <NUM> in a specific state. Specifically, the processor <NUM> may recognize a photographed object such that one surface of the object coincides or is in parallel with the first guideline based on the first guideline displayed on the display <NUM>.

The processor <NUM> may generate a user interface (Ul) for guiding such that a photographed object is located on a line displayed on the display <NUM> or one surface of an object is in parallel with a line displayed on the display <NUM>, and control the display <NUM> to display this. Detailed explanation in this regard will be made with reference to <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Referring to <FIG>, <FIG>, and <FIG>, the processor <NUM> may guide the location of an object in an image by using the first guideline <NUM>.

As described above with reference to <FIG>, the first guideline <NUM> is a line displayed on the display, and it may be a UI for guiding the location of an object <NUM> included in an image. That is, the first guideline <NUM> indicates a straight line for guiding so that the entire front surface of the object <NUM> can be displayed in an image. For example, the first guideline may be located on the <NUM>/<NUM> point of the display <NUM> from the lower end of the display <NUM>. That is, as the first guideline is located on the <NUM>/<NUM> point from the lower end of the display <NUM>, an object included in an image may be guided to be located in the center of the display <NUM>. A location wherein the first guideline is displayed is not necessarily limited thereto, and the location may vary.

Also, according to an embodiment, for performing recognition of an object or extracting an object from a background, the electronic apparatus <NUM> may perform object recognition and extraction by using a relatively more simple mathematical formula and a relatively more simple algorithm based on the first guideline <NUM>, without using software technologies with high complexity such as deep learning and computer vision. Also, by using the first guideline <NUM>, the electronic apparatus <NUM> may perform object recognition and object extraction without using depth data of a space and a camera with a high performance generating depth data.

The processor <NUM> may generate the first guideline <NUM> for guiding such that one surface of an object <NUM> included in a photographed image coincides or is in parallel with a line displayed on the screen of the display <NUM>, and control the display <NUM> to display the generated first guideline <NUM> on the screen of the display <NUM>. For example, the processor <NUM> may provide a guide such that a corner corresponding to the front surface of the object <NUM> among a plurality of corners included in the bottom surface of the photographed object <NUM> is placed on the first guideline <NUM>. Specifically, the processor <NUM> may control the display <NUM> to display a straight line <NUM>, which is a straight line extended from a corner corresponding to the front surface of the photographed object <NUM>. Then, the processor <NUM> may provide a guide such that the object is photographed while the straight line <NUM> and the first guide line <NUM> coincide or are in parallel.

As illustrated in <FIG>, in case the straight line <NUM> and the first guideline <NUM> are not parallel, the processor <NUM> may provide a guide such that the straight line <NUM> is parallel with the first guideline <NUM>, i.e., the camera <NUM> is moved so that the camera <NUM> may face the front surface of a subject <NUM> corresponding to an object. Specifically, the processor <NUM> may control the display <NUM> to display a message <NUM>-<NUM> such as "Face the front surface of the actual product," provide a guide such that the user adjusts the location of the camera <NUM>, and provide a guide such that a corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM> is in parallel with the first guideline <NUM> in accordance thereto. The electronic apparatus <NUM> may guide such that the user can photograph the subject <NUM> by an accurate method by using the first guideline <NUM> and the message <NUM>-<NUM>. Then, in case the straight line <NUM> extended from the corner corresponding to the front surface of the object <NUM> is in parallel with the first guideline, the processor <NUM> may provide feedback informing the user that the location of the object <NUM> is located in a location parallel with the first guideline <NUM>. Here, the feedback may be provided in various forms such as indication of a message informing the location of the object <NUM>, vibration of the electronic apparatus <NUM>, flickering of the screen, the color of the object <NUM>, and change of the thickness of the shade. <FIG> illustrates a drawing wherein, in a case in which the straight line <NUM> extended from the corner corresponding to the front surface of the object <NUM> is parallel with the first guideline, feedback is provided to the user by using vibration.

As illustrated in <FIG>, the straight line <NUM> extended from the corner corresponding to the front surface of the object <NUM> may be omitted on the display <NUM>. Also, in a case in which the bottom surface of the object <NUM> does not coincide with the first guideline <NUM>, the processor <NUM> may provide a guide such that the bottom surface of the object <NUM> coincides with the first guideline <NUM>, i.e., the camera <NUM> is moved so that the camera <NUM> can face the front surface of a subject <NUM> corresponding to the object. Specifically, the processor <NUM> may control the display <NUM> to display a message <NUM>-<NUM> such as "Make the bottom surface of the actual product coincide with the guideline," provide a guide such that the user adjusts the location of the camera <NUM>, and provide a guide such that a corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM> coincides with the first guideline <NUM> in accordance thereto. The electronic apparatus <NUM> may provide a guide such that the user can photograph the subject <NUM> by an accurate method by using the first guideline <NUM> and the message <NUM>-<NUM>. Then, in case the corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM> coincides with the first guideline <NUM>, the processor <NUM> may provide feedback informing the user that the location of the object <NUM> is located on the first guideline <NUM>. Here, the feedback may be provided in various forms such as an indication of a message informing the location of the object <NUM>, vibration of the electronic apparatus, flickering of the screen, and change of the color, the shade, and the thickness of the object <NUM>. <FIG> illustrates a drawing wherein, in a case in which the corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM> coincides with the first guideline, a feedback is provided to the user by using vibration.

The processor <NUM> may guide such that a straight line <NUM> orthogonal to the first guideline is placed on the <NUM>/<NUM> point of the first guideline <NUM>, and the center axis of the object included in the image coincides with the straight line <NUM>.

Also, in case the center axis of the object <NUM> coincides with or is adjacent to the straight line <NUM>, the processor <NUM> may provide feedback informing that the center axis of the object <NUM> coincides with or is adjacent to the straight line <NUM> to the user. The feedback in this case may also be provided in various forms such as indication of a message informing the location of the object <NUM>, vibration of the electronic apparatus, flickering of the screen, and change of the color, the shade, and the thickness of the object <NUM>.

As described above, the processor <NUM> may provide the first guideline, and provide a guide such that the camera <NUM> is located on the front surface of the subject <NUM> corresponding to the object <NUM> replaced with the virtual object <NUM>.

Explanation with reference to <FIG> was made based on an embodiment wherein the first guideline <NUM> and the straight line <NUM> are parallel, but the explanation may also be applied to a case wherein the first guideline <NUM> and the straight line <NUM> coincide. Likewise, explanation with reference to <FIG> was made based on an embodiment wherein the first guideline <NUM> and the bottom surface of the object <NUM> coincide, but the explanation may be applied to a case wherein the first guideline <NUM> and the bottom surface of the object <NUM> are parallel.

As illustrated in <FIG>, the processor <NUM> may guide such that the user draws a corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM>. Specifically, the processor <NUM> may provide a UI guiding such that the user draws a corner included in the bottom surface of the object <NUM>. For example, the processor <NUM> may control the display <NUM> to display a message <NUM> such as "Draw the bottom surface line of the actual product," and provide a guide such that the user draws the bottom surface line <NUM> of the object <NUM>. Then, the processor <NUM> may receive input of the bottom surface line <NUM> of the object <NUM> from the user, and identify the input bottom surface line <NUM>. Then, the processor <NUM> may control the display <NUM> to display a virtual object based on the identified bottom surface line <NUM>. Specifically, the processor <NUM> may recognize the object <NUM> based on the bottom surface line <NUM>. That is, the input bottom surface line <NUM> may be a basis for recognizing the object <NUM> as an alternative to the first guideline <NUM>. As the bottom surface line <NUM> may perform the role of the first guideline <NUM> described above with reference to <FIG> and <FIG>, in this case, the processor <NUM> may include the first guideline <NUM> and the straight line <NUM>, or the first guideline <NUM> and the straight line <NUM> may be omitted. <FIG> illustrates a drawing including the first guideline <NUM> and the straight line <NUM>.

Then, the processor <NUM> may recognize the object <NUM> based on the bottom surface line <NUM>, and the processor <NUM> may calculate information related to the rotation angle and information related to the location of the recognized object <NUM>, and render a virtual object replacing the object <NUM>.

The user may easily follow the guide with input of the bottom surface line <NUM>, without having to control the angle of the camera <NUM> such that the corner corresponding to the front surface among the corners included in the bottom surface of the object <NUM> is in parallel or coincides with the first guideline <NUM>. For the convenience of explanation, <FIG> illustrates a drawing wherein the front surface of the object <NUM> is photographed parallel to the first guideline <NUM>, and the user inputs the bottom surface line <NUM> into the front surface of the photographed object <NUM>. However, it is obvious that, in case the front surface of the object <NUM> is not photographed to be in parallel or coincide with the first guideline <NUM>, the photographed object <NUM> can be recognized with input of the bottom surface line <NUM>.

The processor <NUM> may guide the location of the camera, and the angle of the camera.

<FIG> and <FIG> are diagrams for illustrating a second guideline guiding an angle of a camera according to an embodiment.

The processor <NUM> may acquire the pose information of a camera including the rotation information of the camera by using a simultaneous localization and mapping (SLAM) algorithm, and identify the photographing angle of the camera <NUM> based on the acquired pose information of the camera. Detailed explanation in this regard will be made with reference to <FIG>.

Also, the processor <NUM> may generate a second guideline for guiding the photographing angle of the camera <NUM> based on the identified photographing angle of the camera <NUM>, and change the length of the second guideline according to the photographing angle of the camera <NUM>.

In addition, the processor <NUM> may provide a guide such that the angle of the camera <NUM> located on the front surface of a subject <NUM> is adjusted by using the generated second guideline. Specifically, the processor <NUM> may change the length of the second guideline according to the photographing angle of the camera <NUM>, and thereby guide such that the camera <NUM> photographs the subject <NUM> at an angle parallel to the subject <NUM>.

<FIG> is a diagram illustrating the object <NUM> and the second guideline <NUM> displayed on the display <NUM> in case the camera <NUM> photographed the subject <NUM> at an angle facing the ground surface, i.e., at an angle facing the lower end of the subject <NUM>. In <FIG> and <FIG>, the line <NUM> may correspond to the first guideline <NUM> described above with reference to <FIG>, <FIG>, and <FIG>. Here, the line <NUM> may include the straight line <NUM> in <FIG>, and the straight line <NUM> may be omitted depending on cases. <FIG> and <FIG> illustrate drawings wherein the straight line <NUM> in <FIG> is omitted.

As illustrated in <FIG>, in case the camera <NUM> photographed the subject <NUM> at an angle facing the ground surface, the processor <NUM> may control the display <NUM> to display the second guideline <NUM> based on the angle of the camera <NUM>.

The processor <NUM> may change the length of the second guideline <NUM> such that the length of the second guideline <NUM> becomes longer as the photographing angle of the camera <NUM> becomes larger, i.e., the more the camera <NUM> faces the ground surface. In contrast, the processor <NUM> may change the length of the second guideline <NUM> such that the length of the second guideline <NUM> becomes shorter as the photographing angle of the camera <NUM> becomes smaller, i.e., the more the camera is located in parallel with the subject <NUM>.

<FIG> is a diagram illustrating the object <NUM> and the second guideline <NUM> displayed on the display <NUM> in case the camera <NUM> photographed the subject <NUM> at an angle facing the ceiling, i.e., at an angle facing the upper end of the subject <NUM>.

The processor <NUM> may change the length of the second guideline <NUM> such that the length of the second guideline <NUM> becomes longer as the photographing angle of the camera <NUM> becomes bigger, i.e., the more the camera <NUM> faces the ceiling. In contrast, the processor <NUM> may change the length of the second guideline <NUM> such that the length of the second guideline <NUM> becomes shorter as the photographing angle of the camera <NUM> becomes smaller, i.e., the more the camera is located in parallel with the subject <NUM>.

The processor <NUM> may respectively vary the angles that the second guideline <NUM> constitutes with the line <NUM>, for distinguishing a case wherein the camera <NUM> faces the ground surface and a case wherein the camera <NUM> faces the ceiling. For example, in case the camera <NUM> faces the ground surface, the processor <NUM> may set the angle that the second guideline <NUM> constitutes with the line <NUM> as a first angle (e.g., <NUM> degrees), and in case the camera <NUM> faces the ceiling, the processor <NUM> may set the angle that the second guideline <NUM> constitutes with the line <NUM> as a second angle (e.g., <NUM> degrees).

In case the camera <NUM> is located at an angle parallel to the subject <NUM>, the processor <NUM> may provide feedback informing that the angle of the camera <NUM> is in parallel with the subject <NUM>. The feedback in this case may be provided in various forms such as indication of a message informing the angle of the camera <NUM>, vibration of the electronic apparatus, flickering of the screen, and change of the color, the shade, and the thickness of the object <NUM> displayed on the screen.

The processor <NUM> may use various Uls in addition to the second guideline and the line <NUM> for guiding the photographing angle of the camera. For example, the processor <NUM> may display the angle of the camera, or display a message indicating the angle of the camera in one area of a photographed image. Alternatively, the processor <NUM> may display a message <NUM> which is "Align the camera in parallel with the actual product" until the angle of the camera <NUM> is located at an angle parallel to the subject <NUM>.

Returning to <FIG>, in case an object included in a photographed image is located on the first guideline displayed on the display <NUM>, or an image is photographed with the angle of the camera <NUM> in parallel with the subject <NUM>, the processor <NUM> may recognize an object included in the photographed image.

Then, the processor <NUM> may render a virtual object <NUM> replacing the recognized object. Here, the virtual object <NUM> may correspond to various kinds of external electronic apparatuses or home appliances. The virtual object <NUM> may be an object corresponding to the recognized object, but is not necessarily limited thereto. That is, in case a recognized object is an air conditioner, the virtual object <NUM> may be an air conditioner of a different kind from the recognized object, but it may also be an electronic apparatus such as a refrigerator and an air purifier which are not air conditioners.

Based on an object being recognized, the processor <NUM> may provide various virtual objects that may replace the recognized object, and render a virtual object selected according to a user input selecting one among the virtual objects.

In this regard, <FIG> and <FIG> are diagrams for illustrating an electronic apparatus providing various virtual objects according to an embodiment.

As illustrated in <FIG>, based on an object <NUM> being recognized in a photographed image, the processor <NUM> may control the display <NUM> to display a plurality of virtual objects <NUM>, <NUM>, and <NUM> that may replace the recognized object <NUM>. Here, the plurality of virtual objects may include the images of the virtual objects that can replace the recognized object, and information related to the virtual objects such as the serial numbers, the colors, and the release years of the virtual objects.

The processor <NUM> may control the display <NUM> to display various virtual objects according to a user interaction touching or dragging the plurality of virtual objects <NUM>, <NUM>, <NUM>. For example, while the virtual object <NUM> is displayed in the center, and the virtual object <NUM> and the virtual object <NUM> are displayed on the left and right sides of the virtual object <NUM>, based on a user input passing the virtual object <NUM> to the right being received, the processor <NUM> may cause the virtual object <NUM> to be displayed in the center of the screen. Also, on the left side of the virtual object <NUM>, a new virtual object (not shown) may be displayed.

The processor <NUM> may display a list <NUM> related to the plurality of virtual objects in addition to the plurality of virtual objects <NUM>, <NUM>, and <NUM>. Also, based on a user input selecting one of a plurality of list information (e.g., a TV, a washing machine, a refrigerator, etc.) included in the list <NUM> being received, the processor <NUM> may render a plurality of virtual objects corresponding to the user input. For example, based on a user input selecting a refrigerator among the plurality of list information included in the list <NUM> being received, the processor <NUM> may render a plurality of virtual objects corresponding to the refrigerator.

Based on a user input selecting one of the plurality of virtual objects <NUM>, <NUM>, and <NUM> being received, the processor <NUM> may render a virtual object corresponding to the user input. For example, based in the user selecting the virtual object <NUM> among the plurality of virtual objects <NUM>, <NUM>, and <NUM>, as illustrated in <FIG>, the processor <NUM> may render the selected virtual object <NUM>, and control the display <NUM> to display a virtual object <NUM> corresponding to the selected virtual object <NUM> in the location wherein the object <NUM> is located in the photographed image. That is, the processor <NUM> may control the display <NUM> to display an augmented reality image replacing the object <NUM> with the virtual object <NUM>.

Then, the processor <NUM> may inform the user that the object <NUM> was replaced with the virtual object <NUM> in the augmented reality image. Specifically, the processor <NUM> may control the display <NUM> to display a message <NUM> such as "Replaced with an AR product," and thereby inform that the object <NUM> was replaced with the virtual object <NUM> in the augmented reality image.

The processor <NUM> may render the virtual object in consideration of the surrounding space of the recognized object.

In this regard, <FIG>, <FIG>, <FIG> and <FIG> are diagrams for illustrating an electronic apparatus rendering a virtual object in consideration of the surrounding space of a recognized object according to an embodiment of the disclosure.

Referring to <FIG>, a drawing wherein a recognized object <NUM> and a blank space <NUM> existing in the surroundings are displayed on the display <NUM> is illustrated.

The processor <NUM> may identify a blank space <NUM> existing in the surroundings of the recognized object <NUM>. Here, the blank space <NUM> means a space wherein no object or matter exists among the surrounding spaces of the recognized object.

Referring to <FIG>, the processor <NUM> may recognize an object <NUM> in a photographed image and recognize a blank space <NUM> existing in the surroundings of the object <NUM>, and identify the length of the blank space <NUM>. Specifically, the processor <NUM> may identify the length of the blank space <NUM> identified to exist in an adjacent area of the object <NUM> in the photographed image by using a simultaneous localization and mapping (SLAM) algorithm. Also, the processor <NUM> may identify the length information of the recognized object <NUM> included in the photographed image by using the SLAM algorithm. Then, the processor <NUM> may identify whether the virtual object <NUM> can be located in the area wherein the recognized object <NUM> is located based on the identified length information of the object <NUM> and length information of the blank space <NUM>.

The processor <NUM> may acquire the length information of the actual electronic apparatus corresponding to the virtual object <NUM>. Specifically, the processor <NUM> may acquire the length information of the actual electronic apparatus corresponding to the virtual object <NUM> from an external apparatus such as a server (not shown). Alternatively, the processor <NUM> may store, in advance, the length information of the actual electronic apparatus corresponding to the virtual object <NUM> together with the virtual object <NUM>.

As a result of determining the length information of the object <NUM>, the length information of the blank space <NUM>, and the length information of the virtual object <NUM>, if it is identified that the virtual object <NUM> can be located in the area wherein the object <NUM> is located, then the processor <NUM> may render the first virtual object <NUM> in the area wherein the object <NUM> is located as illustrated in <FIG>.

Alternatively, as a result of determining the length information of the object <NUM>, the length information of the blank space <NUM>, and the length information of the virtual object <NUM>, if it is identified that the virtual object <NUM> cannot be located in the area wherein the object <NUM> is located, the processor <NUM> may inform that an actual product corresponding to the virtual object <NUM> cannot be arranged.

Specifically, the processor <NUM> may compare the length information of the actual electronic apparatus corresponding to the second virtual object <NUM> with the length information of the object <NUM> and the length information of the blank space <NUM>. If the length information of the actual electronic apparatus corresponding to the second virtual object <NUM> is bigger than the numerical value summing up the length information of the object <NUM> and the length information of the blank space <NUM>, the processor <NUM> may identify that a product corresponding to the second virtual object <NUM> cannot be arranged in the area wherein the object <NUM> is located.

Referring to <FIG>, the processor <NUM> may render the second virtual object <NUM> instead of the object <NUM> in the area wherein the object <NUM> is located, and control the display <NUM> to display an area <NUM> wherein the actual space and the second virtual object <NUM> overlap.

Then, the processor <NUM> may inform that a product corresponding to the second virtual object <NUM> cannot be arranged in the area wherein the object <NUM> is located. Specifically, the processor <NUM> may display a message <NUM> such as "As the height of the selected virtual electronic product is higher than the height of the actual space, arrangement is impossible. Please consider this at the time of purchase.

Also, the processor <NUM> may identify the location of the rendered virtual object <NUM> based on the identified location of the object.

In this regard, <FIG> and <FIG> are diagrams for illustrating an electronic apparatus determining the location of an object according to an embodiment of the disclosure. With reference to <FIG> and <FIG>, a method of reducing operation complexity will be explained based on the assumption of a situation wherein a user is guided to photograph a subject <NUM> by using the first guideline, and the subject <NUM> photographed on a display screen is indicated as an object <NUM>.

Referring to <FIG>, the processor <NUM> may identify the location of an object <NUM>-<NUM> in an augmented reality image based on the location information and the angle information of the camera <NUM>, and identify the location of a virtual object based on the identified location of the object <NUM>-<NUM>. Here, the camera <NUM> may be an RGB camera that can acquire RGB data or a 3D camera that can acquire depth data. The explanation below will be made based on the assumption of a case of using a 2D camera or an RGB camera that can acquire RGB data.

The processor <NUM> may acquire the pose information of the camera <NUM> by using various SLAM techniques such as a feature-based simultaneous localization and mapping (SLAM) technique, a direct SLAM technique, an extended Kalman filter (EKF) SLAM technique, a fast SLAM technique, and a large-scale direct monocular (LSD) SLAM technique. Here, the pose information of the camera may include the translation information (Tcx, Tcy, Tcz) and the rotation information (Rcx, Rcy, Rcz) of the camera. Alternatively, the pose information of the camera may be, for example, the X (horizontality), Y (verticality), and Z (depth) corresponding to the location of the camera and/or a <NUM> degree of freedom (<NUM> DoF) camera pose including a pitch, a yaw, and a roll corresponding to the orientation of the camera.

The processor <NUM> may acquire information regarding a plane <NUM>-<NUM> on which the subject <NUM> is placed. Also, the processor <NUM> may estimate a plane space in a photographed image. Specifically, the processor <NUM> may calculate a plane equation by using a 3D point cloud analysis algorithm. For example, the processor <NUM> may estimate a plane space in a photographed image by using a random sample consensus (RANSAC) technique which is one of 3D SLAM techniques. Here, the RANSAC technique is a method of extracting sample data randomly, and obtaining a model parameter satisfying the extracted sample data.

For example, according to spatial coherency, a plurality of points constituting a plane space may exist while being adjacent to one another. The processor <NUM> may estimate a plane space in an image section by using random points adjacent to one another. Also, the processor <NUM> may identify whether the random points adjacent to one another exist on the same plane, and acquire information on the plane space based on the identification result.

For example, the processor <NUM> may identify whether a specific space in an image section is a plane space by using first to third points adjacent to one another (or, distanced within a threshold value). If the specific space is a plane space according to the identification result, the processor <NUM> may acquire coefficients A, B, C, and D of a plane equation based on the first to third points. Then, the processor <NUM> may acquire the plane equation as information on the plane space based on the following formula <NUM>.

Here, A, B, and C are normal vectors indicating the direction of a plane, and D may be a distance between a plane including the first to third points and the camera <NUM>. However, this is merely an example, and information on a plane space may be in various forms. For example, the processor <NUM> may analyze an image section based on a machine-learned model acquiring feature information of an image, and acquire information on a plane space according to the analysis result.

The processor <NUM> may easily calculate the rotation value of the object <NUM>-<NUM> from an image photographed based on the first guideline <NUM>.

Specifically, the processor <NUM> may calculate the rotation values of the object <NUM>-<NUM> with respect to the x axis and the z axis based on a plane equation calculated based on the above formula <NUM>. For example, the processor <NUM> may acquire a first plane equation (A1x+B1y+C1z+D1=<NUM>) for a plane <NUM>-<NUM> on which a subject <NUM>-<NUM> is placed. Then, as the subject <NUM>-<NUM> is placed on the plane <NUM>-<NUM>, the processor <NUM> may identify that the rotation value Rpx of the object <NUM>-<NUM> for the x axis is A1 which is the coefficient of x in the first plane equation, and identify that the rotation value Rpz of the object <NUM>-<NUM> for the z axis is C1 which is the coefficient of z in the first plane equation.

Also, the processor <NUM> may identify the rotation value Rpy of the object <NUM>-<NUM> for the y axis based on the angle information Rcx, Rcy, Rcz of the camera <NUM>. Specifically, as the user photographs the subject <NUM>-<NUM> on the front surface, the processor <NUM> may identify that the rotation value Rpy of the object <NUM>-<NUM> for the y axis is - Rcy.

The electronic apparatus <NUM> according to the disclosure may easily calculate the rotation value of the object <NUM> by using an image photographed to be aligned with the first guideline <NUM>.

The processor <NUM> may identify the coordinate u, v of the crossing point <NUM>-<NUM> of the first guideline <NUM> and the straight line <NUM> orthogonal to the first guideline in the image. Here, the identified coordinate u, v of the crossing point <NUM>-<NUM> is a coordinate on the first guideline <NUM>, and may be fixed on a specific location of the display screen. Accordingly, the processor <NUM> may identify the identified coordinate u, v of the crossing point <NUM>-<NUM>.

The processor <NUM> may acquire information on a straight line passing through a virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> corresponding to the location and angle information of the camera by using the location and angle information of the camera acquired through the identified coordinate u, v of the crossing point <NUM>-<NUM>, the intrinsic parameter K of the camera, and the SLAM algorithm. Here, the virtual location <NUM>-<NUM> may mean a coordinate on an actual space corresponding to the location information of the camera. Specifically, the virtual location <NUM>-<NUM> may be a point existing on a straight line connecting one point on the subject <NUM>-<NUM> and one point on the object <NUM>-<NUM> corresponding to the point, and the virtual location <NUM>-<NUM> may mean a location calculated based on the focal length, the focal direction, the photographing angle, the angle information Rcx, Rcy, Rcz, the distance to the subject, etc. of the camera <NUM>.

Specifically, the processor <NUM> may acquire an equation regarding a straight line passing through the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> as information regarding a straight line by using the following formula <NUM>.

Then, based on the equation regarding the plane <NUM>-<NUM> and the equation regarding the straight line passing through the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM>, the processor <NUM> may calculate the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM> of the straight line passing through the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> and the plane <NUM>-<NUM>. Here, the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM> may indicate the center point of the corner corresponding to the front surface among the corners included in the bottom surface of the recognized object <NUM>-<NUM> in the photographed image. That is, the processor <NUM> may easily perform the calculation in the above formula <NUM> by using the first guideline <NUM> or the coordinate u, v on the first guideline. Then, the processor <NUM> may alleviate the complexity of the operation obtaining an equation regarding the straight line by making the bottom surface of the subject <NUM> and the first guideline coincide.

Then, the processor <NUM> may identify that the subject <NUM>-<NUM> is located on the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM>. Then, the processor <NUM> may generate an augmented reality image wherein a virtual object <NUM> replacing the object <NUM>-<NUM> is arranged in the identified location. Specifically, the processor <NUM> may render the virtual object <NUM> such that the center point of the corner corresponding to the front surface among the plurality of corners included in the bottom surface of the virtual object <NUM> is located on the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM>.

The processor <NUM> may calculate the size of the virtual object based on the virtual location <NUM>-<NUM> corresponding to the location information Tcx, Tcy, Tcz of the camera and the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM>. Specifically, the virtual location <NUM>-<NUM> corresponding to the location information Tcx, Tcy, Tcz of the camera may mean the location of the virtual camera, and thus the processor <NUM> may identify the size of the virtual object to be inversely proportional to the distance between the virtual location <NUM>-<NUM> corresponding to the location information Tcx, Tcy, Tcz of the camera and the coordinate Tpx, Tpy, Tpz of the crossing point <NUM>-<NUM>. For example, if the distance between the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> increases as much as a predetermined distance, the size of the virtual object may be decreased as much as a predetermined size proportion. In a case in which the distance between the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> is <NUM>, the virtual object may be implemented as the actual size of the product corresponding to the virtual object. Then, the processor <NUM> may render the virtual object in the identified size.

<FIG> illustrates the subject <NUM>-<NUM> placed on a floor, but as illustrated in <FIG>, the processor <NUM> may render the virtual object <NUM> replacing the photographed object <NUM>-<NUM> with respect to the subject <NUM>-<NUM> attached to the wall surface based on the same method.

Specifically, the processor <NUM> may identify the location of the object <NUM>-<NUM> based on the location information and the angle information of the camera <NUM>, and identify the location of the virtual object based on the identified location of the object <NUM>-<NUM>. Also, the processor <NUM> may acquire pose information of the camera <NUM> by using various SLAM techniques. That is, the processor <NUM> may use the same technique or method for identifying the location of the object <NUM>-<NUM>.

The processor <NUM> may calculate the rotation value of the object <NUM>-<NUM> with respect to the X axis and the Y axis based on the plane equation calculated based on the formula <NUM>. For example, the processor <NUM> may acquire a second plane equation (A2x+B2y+C2z+D2=<NUM>) for the plane <NUM>-<NUM> on which the subject <NUM>-<NUM> is located. Also, as the subject <NUM>-<NUM> is located on the plane <NUM>-<NUM>, the processor <NUM> may identify that the rotation value Rpx of the object <NUM>-<NUM> for the X axis is A2 which is the coefficient of x in the second plane equation, and identify that the rotation value Rpy of the object <NUM>-<NUM> for the y axis is B2 which is the coefficient of y in the second plane equation.

Also, the processor <NUM> may identify the rotation value Rpz of the object <NUM>-<NUM> for the z axis based on the angle information Rcx, Rcy, Rcz of the camera <NUM>. Specifically, as the user photographs the subject <NUM>-<NUM> from the front surface, the processor <NUM> may identify that the rotation value Rpz of the object <NUM>-<NUM> for the z axis is - Rcz.

The crossing point <NUM>-<NUM>, the virtual location <NUM>-<NUM>, and the crossing point <NUM>-<NUM> illustrated in <FIG> may respectively correspond to the crossing point <NUM>-<NUM>, the virtual location <NUM>-<NUM>, and the crossing point <NUM>-<NUM> illustrated in <FIG>. The processor <NUM> may identify the coordinate u, v of the crossing point <NUM>-<NUM> by the same method or technique as described with reference to <FIG>, and acquire information regarding the straight line passing through the virtual location <NUM>-<NUM> and the crossing point <NUM>-<NUM> corresponding to the location and angle information of the camera by using the location and angle information of the camera acquired through the intrinsic parameter K of the camera and the SLAM algorithm.

The processor <NUM> may generate an augmented reality image wherein a rendered virtual object is located in an area wherein an object is located in a photographed image, and control the display <NUM> to display the generated augmented reality image.

As described above, according to an embodiment, the electronic apparatus <NUM> may identify the location of an object in a photographed image by using only an RGB camera or a 2D camera that can acquire RGB data, without using a camera with a high performance that can acquire depth data. Also, the electronic apparatus <NUM> may calculate the location, the size, and the rotation value of an object based on the simple formula <NUM> and formula <NUM> without using a complex algorithm in object extraction and object recognition processes. Accordingly, the real time quality of an augmented reality image can be improved.

In a case in which a virtual object covers a recognized object in an image, additional image processing might not be necessary. However, in case a virtual object cannot cover a recognized object in an image, additional image processing may be necessary.

<FIG> and <FIG> are diagrams for illustrating an electronic apparatus which performs image processing in case a rendered virtual object cannot cover a recognized object according to an embodiment.

The processor <NUM> may identify the location and the size of a virtual object as described above with reference to <FIG>, and identify whether a virtual object rendered in an image can cover an object area in the image. Here, the feature that a rendered virtual object can cover an object area means that a virtual object is totally overlapped with a recognized object, and a portion of the object is not shown in an image.

In case a partial object area not covered by a virtual object exists, the processor <NUM> may generate an augmented reality image by replacing the partial object area with the background area around the partial object area. Here, the background area around the partial object area means a background area adjacent to the partial area of the object not covered.

Specifically, the processor <NUM> may remove the partial object area through a vision recognition technology, and replace the removed portion based on the background area adjacent to the removed partial object area.

For example, as illustrated in <FIG>, it will be assumed that an object <NUM> was recognized in an image photographed by the camera <NUM>. Here, the object <NUM> is an air conditioner, and a background such as a window may exist on the rear side of the object <NUM>. Then, it will be assumed that the processor <NUM> rendered an air purifier as a virtual object <NUM> replacing the recognized object <NUM>. In case the air purifier which is the virtual object <NUM> does not cover the air conditioner which is the object <NUM> in the photographed image, i.e., in case a partial area of the air conditioner exists in an augmented reality image, the processor <NUM> may remove the partial area of the air conditioner not covered in the augmented reality image, and replace the removed area based on the window, the wall of the glass window, etc. which are the background area adjacent to the removed partial object area. <FIG> illustrates an augmented reality image wherein the processor <NUM> replaced a removed partial object area <NUM> based on the window, the wall of the glass window, etc. which are the background area adjacent to the removed partial object area <NUM>.

<FIG> and <FIG> are diagrams for illustrating an electronic apparatus which changes a virtual object based on the movement of a camera after the virtual object was rendered, and renders the virtual object according to an embodiment of the disclosure.

If the location information or the angle information of the camera <NUM> is changed while an augmented reality image is displayed, the processor <NUM> may change a virtual object based on the changed location information or angle information of the camera and render the virtual object.

For this, the processor <NUM> may acquire information on the view direction (or the view angle) of the camera <NUM> in the augmented reality image from the camera <NUM>.

Information on the view direction (or the view angle) of the camera <NUM> according to an embodiment of the disclosure may mean at least one of the direction that the camera <NUM> faces which corresponds to a real-time image received from the camera <NUM>, the angle of the camera <NUM>, the location of the camera <NUM>, or the posture of the camera <NUM>. For example, information on the view direction of the camera <NUM> may mean that the camera performed photographing while moving from left to right, or photographed a subject in a specific angle or posture.

According to an embodiment of the disclosure, the electronic apparatus <NUM> may acquire information on the view direction and view angle of the camera by using at least one of an acceleration sensor or a gyro sensor. For example, the processor <NUM> may acquire a specific angle (e.g., <NUM> degrees among <NUM> degrees in all directions) corresponding to the view direction that the camera <NUM> faces (or, the direction that the electronic apparatus <NUM> faces) among all directions (e.g., <NUM> degrees) as information on the view direction based on a sensing result using at least one of an acceleration sensor or a gyro sensor.

This is merely an example, and the disclosure is not limited thereto. For example, the electronic apparatus <NUM> can obviously acquire information on the view direction of the camera <NUM> based on a GPS sensor, a tilt/gravity sensor, a location-based services (LBS) method, location information or direction information received from an external apparatus, a digital compass, etc..

The processor <NUM> may change a rendered virtual object to a virtual object corresponding to the acquired view direction information. The memory <NUM> may store a plurality of virtual objects related to a virtual object, and the processor <NUM> may change the virtual object by using the plurality of virtual objects stored in the memory <NUM>. Here, the plurality of virtual objects related to the virtual object may include images wherein an electronic apparatus corresponding to the virtual object was photographed in various angles and directions. Then, in case the processor <NUM> identified that the view direction information of the camera <NUM> was changed, the processor <NUM> may render the virtual object rendered in an augmented reality image as a virtual object corresponding to the view direction information among the plurality of virtual objects related to the virtual object.

For example, as illustrated in <FIG>, it will be assumed that, while the front surface of an air conditioner which is a virtual object <NUM> was rendered, the camera <NUM> moved by approximately <NUM> degrees in a counterclockwise direction based on the subject <NUM>. In this case, the processor <NUM> may acquire the view direction information of the camera <NUM>, and change the virtual object to a virtual object corresponding to the acquired view direction information (e.g., a virtual object including an image which moved by approximately <NUM> degrees in a counterclockwise direction from the front surface) and render the virtual object. <FIG> illustrates an augmented reality image wherein a virtual object was changed based on the changed view direction information acquired and was rendered.

<FIG> and <FIG> are diagrams for illustrating an electronic apparatus rendering a virtual object according to an embodiment of the disclosure.

When an event occurs, the processor <NUM> may render a virtual object indicating a movement during a driving operation of a virtual object. Specifically, when an event such as receiving a user input touching a virtual object while a virtual object is rendered in an augmented reality image occurs, the processor <NUM> may render a virtual object indicating a movement during a driving operation of a virtual object. Here, a movement during a driving operation of a virtual object is an object indicating a driving state of an electronic apparatus corresponding to a virtual object, and may include an object such as the door opening operation of a refrigerator (or a washing machine, a dryer), an image displaying operation of a TV, and a menu displaying operation of a display included in an air conditioner (or a refrigerator).

For example, as illustrated in <FIG>, while a refrigerator is illustrated in an augmented reality image as a virtual object <NUM>, if a user input touching the refrigerator displayed on the screen is received, the processor <NUM> may render a virtual object <NUM> wherein the door of the refrigerator is opened, as illustrated in <FIG>.

For this, the memory <NUM> may store a virtual object indicating a movement during an operation of a virtual object, and when an event occurs, the processor <NUM> may render a virtual object indicating a movement during an operation of a rendered virtual object.

In the above, it was described that an event receives a user input touching a virtual object, but the disclosure is not necessarily limited thereto. For example, an event may vary depending on embodiments, such as receiving a voice input of a user, receiving a user input selecting a UI (e.g., a UI indicating viewing of an operation state) displayed in an augmented reality image, and receiving a user input shaking the electronic apparatus <NUM>. Also, it may be possible that a predetermined time period passes after a virtual object was rendered and a virtual object indicating a movement during a driving operation of a virtual object is rendered, without a user input.

<FIG> and <FIG> are diagrams for illustrating an electronic apparatus changing the location of a rendered virtual object.

When the processor <NUM> receives a user input moving a virtual object included in an augmented reality image, the processor <NUM> may change the location of the virtual object based on the user input and render the virtual object.

For example, it will be assumed that the camera <NUM> photographs areas wherein a washing machine <NUM> and a dryer exist, and the processor <NUM> renders a virtual object <NUM> replacing the photographed dryer, and as illustrated in <FIG>, the processor <NUM> displays an augmented reality image including the washing machine <NUM> and the virtual object <NUM> corresponding to a subject existing in reality on the display <NUM>.

Here, if a user input moving the virtual object <NUM> is received, the processor <NUM> may change the location of the virtual object <NUM> based on the user input and render the virtual object <NUM>. For example, if a user input selecting a dryer which is the virtual object <NUM> and dragging it to the upper end of the washing machine <NUM> is received, the processor <NUM> may change the location of the dryer displayed in the augmented reality image to the upper end of the washing machine <NUM>.

<FIG> is a diagram for illustrating a detailed configuration of an electronic apparatus according to an embodiment of the disclosure.

As illustrated in <FIG>, the electronic apparatus <NUM> may include a camera <NUM>, a display <NUM>, a processor <NUM>, a communication interface <NUM>, a memory <NUM>, an input interface <NUM>, and a speaker <NUM>. Among the components illustrated in <FIG>, regarding the components overlapping with the components illustrated in <FIG>, detailed explanation will be omitted.

The communication interface <NUM> may be a component for the electronic apparatus <NUM> to perform communication with an external electronic apparatus (not shown) such as a server. The electronic apparatus <NUM> may receive various data such as information on a recognized object and information on a virtual object from an external electronic apparatus (not shown) through the communication interface <NUM>.

Also, the communication interface <NUM> may include various communication modules such as a wired communication module (not shown), a near field wireless communication module (not shown), and a wireless communication module (not shown).

Here, a wired communication module is a module for performing communication with an external apparatus (not shown) according to a wired communication method such as a wired Ethernet. A near field wireless communication module is a module for performing communication with an external apparatus (not shown) located in a close distance according to a near field wireless communication method such as Bluetooth (BT), Bluetooth Low Energy (BLE), and ZigBee methods. In addition, a wireless communication module is a module that is connected to an external network according to a wireless communication protocol such as Wireless Fidelity (Wi-Fi) and Institute of Electrical and Electronics Engineers (IEEE) and performs communication with an external apparatus (not shown) and a voice recognition server (not shown). Other than the above, a wireless communication module may further include a mobile communication module that is connected with a mobile communication network and performs communication according to various mobile communication standards such as 3rd Generation (<NUM>), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), and fifth generation (<NUM>) Networks.

The memory <NUM> may be a component for storing various kinds of programs and data for the operation of the electronic apparatus <NUM>. The memory <NUM> may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SDD), etc. Also, the memory <NUM> may be accessed by the processor <NUM>, and reading/recording/correction/deletion/update, etc. of data by the processor <NUM> may be performed. In the disclosure, the term memory may include the memory <NUM>, a read-only memory (ROM) (not shown) and a random access memory (RAM) (not shown) inside the processor <NUM>, or a memory card (not shown) mounted on the electronic apparatus <NUM> (e.g., a micro SD card, a memory stick).

The memory <NUM> may store information on various virtual objects of the electronic apparatus <NUM>. Also, according to an embodiment, the electronic apparatus <NUM> may store information on an electronic apparatus which the camera <NUM> photographed and can be recognized as an object.

The input interface <NUM> may receive a user input for controlling the electronic apparatus <NUM>. In particular, the input interface <NUM> may receive input of voice information of a user for a reproduced content, or receive input of a user input selecting a GUI displayed on a display. In the input interface <NUM>, as illustrated in <FIG>, a microphone <NUM> for receiving input of a user voice, a touch panel <NUM> for receiving input of a user touch using a user's hand or a stylus pen, etc., a button <NUM> for receiving input of a user manipulation, etc. may be included. However, the example of the input interface <NUM> illustrated in <FIG> is merely an example, and the input interface <NUM> may be implemented as another input apparatus (e.g., a keyboard, a mouse, a motion inputter, etc.).

The speaker <NUM> may be a component outputting various kinds of notification sounds or voice messages as well as various types of audio data for which various processing operations such as decoding or amplification, noise filtering, etc. were performed by an audio processor. In particular, the speaker <NUM> may output a response for voice information of a user as a voice message in the form of a natural language. A component for outputting audio may be implemented as a speaker, but this is merely an example, and such a component may be implemented as an output terminal that can output audio data.

The electronic apparatus <NUM> might not necessarily include all of the aforementioned components, and some components may be omitted, when not departing from the scope of the appended claims.

Computer instructions for performing the processing operations of the electronic apparatus <NUM> according to the aforementioned various embodiments of the disclosure may be stored in a non-transitory computer-program product Computer instructions stored in such a non-transitory computer-program product may implement the processing operations of the electronic apparatus <NUM> according to the aforementioned various embodiments of the disclosure performed by the aforementioned specific machine when they are executed by a processor of the specific machine.

A non-transitory computer- program product refers to a medium that stores data semi-permanently, and is readable by machines. Specifically, the aforementioned various applications or programs may be provided while being stored in a non-transitory computer- program product such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM and the like.

Claim 1:
An electronic apparatus (<NUM>) comprising:
a display (<NUM>);
a memory (<NUM>) configured to store instructions; and
a processor (<NUM>) configured to execute the instructions to:
control the display (<NUM>) to display an image captured via a camera (<NUM>),
control the display (<NUM>) to display a first guideline (<NUM>),
recognize an object (<NUM>) in the image based on the first guideline (<NUM>),
render a virtual object (<NUM>) to replace the object (<NUM>),
generate an augmented reality image including the virtual object (<NUM>) that is located in an area where the object (<NUM>) is located in the image, and
control the display (<NUM>) to display the augmented reality image;
characterized in that the processor is further configured to display the first guideline (<NUM>) as a straight line, and recognizing the object (<NUM>) in the image based on the first guideline comprises to align the object in the image with the first guideline, so that one surface of the object (<NUM>) is coincident or parallel with the straight line of the first guideline (<NUM>).