Mobile terminal and method for controlling the same

Disclosed are a mobile terminal having a hologram output unit and a method for controlling the same. The mobile terminal, comprises: a hologram output unit configured to output a hologram object to an output space outside the mobile terminal; an image sensor configured to sense a portion of a human body located within the output space of the hologram object; a feedback output unit configured to output a feedback signal; and a controller configured to: determine a relative position between the hologram object output by the hologram output unit and the portion of the human body sensed by the image sensor; detect, based on the relative position between the hologram object and the portion of the human body, that the portion of the human body approaches the hologram object, and control, based on the detection that the portion of the human body approaches the hologram object, the feedback output unit to transmit the feedback signal towards the portion of the human body that is detected to approach the hologram object.

CROSS-REFERENCE TO RELATED APPLICATION

Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of earlier filing date and right of priority to Korean Application No. 10-2015-0126506, filed on Sep. 7, 2015, the contents of which is incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a mobile terminal, and more particularly, to a mobile terminal having a hologram output unit and a method for controlling the same.

2. Background of the Invention

Efforts are ongoing to support and increase the functionality of mobile terminals. Such efforts include software and hardware improvements, as well as changes and improvements in the structural components.

As one of such structural changes and improvements, a mobile terminal having a hologram output unit, and an interaction between a hologram object output through the hologram output unit and a user may be considered.

SUMMARY OF THE INVENTION

Therefore, an aspect of the detailed description is to provide a mobile terminal capable of providing a user interaction with respect to a hologram object.

Another aspect of the detailed description is to provide a mobile terminal capable of providing a realistic user interaction according to a characteristic of an object corresponding to a hologram object.

Another aspect of the detailed description is to provide a method for more effectively sensing a user's approach, according to a characteristic of an object corresponding to a hologram object.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is provided a mobile terminal, including: a hologram output unit configured to output a hologram object to a preset space; an image sensor configured to sense a human body located at the output space of the hologram object; an output unit configured to output a feedback signal; and a controller configured to detect approach of the human body to the hologram object, based on a relative position between the hologram object and the human body, and configured to control the output unit such that the feedback signal is transmitted to the human body which has approached the hologram object, based on a result of the detection.

In an embodiment, the output unit may include at least one of a laser output module and a sound wave output module. And the feedback signal may include at least one of laser output from the laser output module, and a sound wave output from the sound wave output module.

In an embodiment, the feedback signal may be one of a plurality of different feedback signals. And the plurality of feedback signals may be different from each other in at least one of an intensity, an amplitude, a frequency and a waveform of the laser or the sound wave.

In an embodiment, when the human body's approach to the hologram object is detected in a deactivated state of the output unit, the controller may convert the deactivated state of the output unit into an activated state such that the feedback signal is output.

In an embodiment, the output space of the hologram object may include a first region where the hologram object is positioned, and a second region adjacent to the first region. When the human body is positioned on one of the first region and the second region, the controller may control the output unit such that the feedback signal is output.

In an embodiment, the controller may control the output unit to output a different feedback signal, according to whether the human body is positioned on the first region or the second region.

In an embodiment, the controller may determines a relative position between the hologram object and the human body, by comparing coordinates information of the first region where the hologram object is positioned, and the second region adjacent to the first region in the preset space, with coordinates information of the human body. And the controller may determine one of the first and second regions where the human body is positioned, based on a result of the determination.

In an embodiment, the coordinates information of the first region where the hologram object is positioned, and the second region adjacent to the first region in the preset space, and the coordinates information of the human body may be acquired through a specific type of modeling with respect to each of the hologram object and the human body.

In an embodiment, the controller may execute the specific type of modeling, with respect to an image object corresponding to the human body, from an image acquired by the image sensor.

In an embodiment, the specific type of modeling with respect to the hologram object may be executed by a method determined based on a characteristic of an object corresponding to the hologram object, among a plurality of modeling methods.

In an embodiment, the controller may control the output unit such that a different feedback signal is transmitted according to a characteristic of an object corresponding to the hologram object. The characteristic of the object may include at least one of a type, a weight, a size, an intensity, a shape and a surface of the object.

In an embodiment, the controller may output a different feedback signal according to the characteristic of the object, by controlling at least one of an intensity, an amplitude, a frequency and a waveform of the feedback signal.

In an embodiment, when the human body is positioned at a peripheral region of the hologram object, the controller may output notification information indicating that the feedback signal is outputtable.

In an embodiment, the notification information may include video information. And the controller may control the hologram output unit to output the notification information, to a region among the peripheral region of the hologram object, the region corresponding to a position of the human body and where the feedback signal is transmittable.

In an embodiment, the controller may output a first feedback signal when the human body approaches the hologram object at a first speed. And the controller may output a second feedback signal different from the first feedback signal when the human body approaches the hologram object at a second speed different from the first speed.

In an embodiment, the image sensor may be configured to sense a gesture of the human body approaching the hologram object. And the controller may control the hologram output unit such that at least one of an output position and a shape of the hologram object is changed, in response to the sensed gesture of the human body.

In an embodiment, the controller may move the hologram object in the preset output space, in response to a gesture of the human body. When the hologram object is positioned at a boundary region of the preset space, the controller may change a shape of the hologram object in response to a gesture of the human body.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is also provided a method for controlling a mobile terminal, including: outputting a hologram object to a preset space; sensing a human body located at the preset output space of the hologram object; detecting approach of the human body to the hologram object, based on a relative position between the hologram object and the human body; and controlling a haptic module such that a feedback signal is transmitted to the human body approaching the hologram object, based on a result of the detection.

The output space of the hologram object may include a first region where the hologram object is positioned, and a second region adjacent to the first region. In the controlling a haptic module, a different feedback signal may be output according to whether the human body is positioned on the first region or the second region.

The feedback signal may include at least one of laser and a sound wave. In the controlling a haptic module, one of a plurality of feedback signals different from each other in at least one of an intensity, an amplitude, a frequency and a waveform may be output, the one determined based on a characteristic of an object corresponding to the hologram object.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is also provided a mobile terminal, comprising: a hologram output unit configured to output a hologram object to an output space outside the mobile terminal; an image sensor configured to sense a portion of a human body located within the output space of the hologram object; a feedback output unit configured to output a feedback signal; and a controller configured to: determine a relative position between the hologram object output by the hologram output unit and the portion of the human body sensed by the image sensor; detect, based on the relative position between the hologram object and the portion of the human body, that the portion of the human body approaches the hologram object, and control, based on the detection that the portion of the human body approaches the hologram object, the feedback output unit to transmit the feedback signal towards the portion of the human body that is detected to approach the hologram object.

In an embodiment, the feedback output unit comprises at least one of a laser output module or a sound wave output module, and wherein the feedback signal comprises at least one of a laser output from the laser output module or a sound wave output from the sound wave output module.

In an embodiment, the feedback signal is one of a plurality of different feedback signals, and wherein the plurality of feedback signals are different from each other in at least one of an intensity, an amplitude, a frequency, or a waveform of the laser or the sound wave.

In an embodiment, the controller is configured to: determine that the detection of the portion of the human body approaching the hologram object occurs in a deactivated state of the feedback output unit; and convert the deactivated state of the feedback output unit into an activated state to output the feedback signal from the feedback output unit.

In an embodiment, wherein the output space of the hologram object comprises a first region where the hologram object is positioned and a second region adjacent to the first region, and wherein the controller is configured to: determine whether the portion of the human body is positioned in the first region or the second region; and control the feedback output unit to output the feedback signal according to the determination of whether the portion of the human body is positioned in the first region or the second region.

In an embodiment, the controller is configured to control the feedback output unit to output a first feedback signal or a second feedback signal according to whether the portion of the human body is positioned in the first region or the second region, respectively.

In an embodiment, the controller is configured to: compare coordinate information of the portion of the human body with coordinate information of the first region where the hologram object is positioned and coordinate information of the second region adjacent to the first region; determine a relative position between the hologram object and the portion of the human body based on the comparison of coordinate information of the portion of the human body with coordinate information of the first region where the hologram object is positioned and coordinate information of the second region adjacent to the first region; and determine that the portion of the human body is located in one of the first region or the second region based on the determination of the relative position between the hologram object and the portion of the human body.

In an embodiment, wherein the controller is configured to: determine a model of the hologram object and a model of the portion of the human body; and determine, based on the model of the hologram object and the model of the portion of the human body, the coordinate information of the first region where the hologram object is positioned, the coordinate information of the second region adjacent to the first region, and the coordinate information of the portion of the human body.

In an embodiment, the controller is configured to determine the model of the portion of the human body by: determining an image of the portion of the human body acquired by the image sensor; determining an image object corresponding to the portion of the human body from the image acquired by the image sensor; and determining the model of the portion of the human body based on the image object corresponding to the portion of the human body that was determined from the image acquired by the image sensor.

In an embodiment, the controller is configured to determine the model of the hologram object by: determining a characteristic of an object corresponding to the hologram object; selecting, from among a plurality of modeling methods, a modeling method based on the determined characteristic of the object corresponding to the hologram object; and determining the model of the hologram object based on executing the modeling method selected from among the plurality of modeling methods.

In an embodiment, the controller is configured to control the feedback output unit by: determining a characteristic of an object corresponding to the hologram object; and controlling the feedback output unit to transmit the feedback signal according to the determined characteristic of the object corresponding to the hologram object, wherein determining the characteristic of the object corresponding to the hologram object comprises determining at least one of a type, a weight, a size, an intensity, a shape, or a surface of the object corresponding to the hologram object.

In an embodiment, the controller is configured to control the feedback output unit to transmit the feedback signal according to the determined characteristic of the object corresponding to the hologram object by: controlling at least one of an intensity, an amplitude, a frequency, or a waveform of the feedback signal.

In an embodiment, the controller is configured to: determine that the portion of the human body is positioned in a peripheral region that is within a predetermined distance of the hologram object; and output, based on the determination that the portion of the human body is positioned in the peripheral region of the hologram object, notification information indicating that the feedback signal is capable of being output.

In an embodiment, the notification information comprises video information, and wherein the controller is configured to output the notification information indicating that the feedback signal is capable of being output by: determining a region, within the peripheral region of the hologram object, that corresponds to a position of the portion of the human body and where the feedback signal is capable of being transmitted; and controlling the hologram output unit to output the notification information to the determined region within the peripheral region of the hologram object that corresponds to the position of the portion of the human body and where the feedback signal is capable of being transmitted.

In an embodiment, the controller is configured to control the feedback output unit to transmit the feedback signal towards the portion of the human body by: determining a speed at which the portion of the human body approaches the hologram object; outputting a first feedback signal based on a determination that the portion of the human body approaches the hologram object at a first speed, and outputting a second feedback signal different from the first feedback signal based on a determination that the portion the human body approaches the hologram object at a second speed different from the first speed.

In an embodiment, the image sensor is configured to sense a gesture of the portion of the human body that approaches the hologram object, and wherein the controller is configured to control the hologram output unit to change at least one of an output position or a shape of the hologram object, in response to the sensed gesture of the portion of the human body.

In an embodiment, the controller is configured to control the hologram output unit to change the output position of the hologram object within the output space in response to a gesture of the human body, and wherein the controller is further configured to: determine whether the hologram object is positioned at a boundary region of the output space; and control the hologram output unit, based on a determination that the hologram object is positioned at the boundary region of the output space, to change the shape of the hologram object in response to a gesture of the human body.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is also provided a method for controlling a mobile terminal, including: outputting a hologram object to an output space outside the mobile terminal; sensing a portion of a human body located within the output space of the hologram object; determining a relative position between the hologram object and the portion of the human body; detecting, based on the relative position between the hologram object and the portion of the human body, that the portion of the human body approaches the hologram object; and controlling, based on the detection that the portion of the human body approaches the hologram object, a haptic module such that a feedback signal is transmitted towards the portion of the human body that approaches the hologram object.

In an embodiment, the output space of the hologram object comprises a first region where the hologram object is positioned and a second region adjacent to the first region, and wherein the controller is configured to: determine whether the portion of the human body is positioned in the first region or the second region; and control the haptic module to output the feedback signal according to determination of whether the human body is positioned in the first region or the second region.

In an embodiment, the feedback signal comprises at least one of a laser or a sound wave, and wherein the controller is configured to: determine a characteristic of an object corresponding to the hologram object; and control the haptic module to output at least one of an intensity, an amplitude, a frequency, or a waveform of the feedback signal based on the determined characteristic of the object corresponding to the hologram object.

DETAILED DESCRIPTION OF THE INVENTION

In the mobile terminal according to the present invention, the camera121may include a three-dimensional (3D) camera or a depth camera. In the present invention, a user's body positioned in an arbitrary space (preset space) where a holographic image (hologram object) is output, is sensed by an image sensor provided at a 3D camera (depth camera). The image sensor may be configured to sense a user's body or sense a user's gesture in the arbitrary space. In the mobile terminal according to the present invention, a 3D image corresponding to an object (user's body) approaching a holographic image, may be acquired by such a 3D camera.

The sensing unit140is typically implemented using one or more sensors configured to sense internal information of the mobile terminal, the surrounding environment of the mobile terminal, user information, and the like. For example, inFIG. 1A, the sensing unit140is shown having a proximity sensor141and an illumination sensor142.

The output unit150, configured to generate an output in a visible, audible or tactile manner, may include at least one of a display unit151, an audio output unit152, a haptic module153, an optical output unit154, and a hologram output unit (a holography module)155. The display unit151may have a layered structure with a touch sensor, or may be integrally formed with a touch sensor, thereby implementing a touch screen. The touch screen may serve as the user input unit123for providing an input interface between the mobile terminal100and a user, or may provide an output interface between the mobile terminal100and a user.

The memory170may store therein information on hologram interference patterns so as to support projection of a holographic image by the hologram output unit155. The controller180may output information which can be visually output from the mobile terminal, through the hologram output unit155in the form of a holographic image, based on the information stored in the memory170.

The hologram output unit (holography module155) is configured to output a holographic image to a preset space. A structure of the holography module155, and a method for projecting a holographic image using the structure will be explained in more detail with reference toFIGS. 2A to 5B.

The hologram output unit155may be provided at the terminal body. The hologram output unit155may be configured to output a holographic image155′ (refer toFIG. 2A) to a front surface of the terminal body, e.g., a space above the display unit151. In the present invention, the hologram output unit155is provided at one region of the display unit151. In the present invention, the hologram output unit155may be disposed on any region of the terminal body, and the position of the hologram output unit155may be variously changed according to a design.

The hologram output unit155may be provided on a rear surface of the terminal body, and may be configured to output the holographic image155′ (refer toFIG. 2B) to a space above the rear surface of the terminal body. In the present invention, the hologram output unit155may be provided on at least one of a front surface, a side surface and a rear surface of the terminal body.

Hereinafter, a principle to output a holographic image through the hologram output unit155, and a method thereof will be explained in more detail with reference to the attached drawings.

FIGS. 2A and 2Bare conceptual views illustrating that a holographic image is implemented through a hologram output unit.FIGS. 3A and 3Bare conceptual views illustrating a hologram principle.FIGS. 4A to 4Care conceptual views illustrating a transmission-type holography method. AndFIGS. 5A to 5Care conceptual views illustrating a reflection-type holography method.

As aforementioned, the mobile terminal of the present invention is provided with the hologram output unit155, and is configured to output a holographic image through the hologram output unit155. In order to support output of a holographic image through the hologram output unit155, the memory170may store therein information about hologram interference patterns. The controller180may output information which can be visually output from the mobile terminal, through the hologram output unit155as a holographic image, by using the information stored in the memory170.

The hologram output unit155may be provided on at least one of a front surface, a rear surface and a side surface of the terminal body, and may be configured to output a holographic image155′ to a space above the rear surface of the terminal body.

FIGS. 2A and 2Bare conceptual views illustrating that a holographic image155′ is implemented through the hologram output unit155shown inFIGS. 1B and 1C.

Referring toFIG. 2A, the hologram output unit155may be disposed on a front surface of the mobile terminal100. The hologram output unit155may be operated independently from the display unit151. As shown, the hologram output unit155may be configured to output the holographic image155′ even when no visual information has been displayed to the display unit151. The hologram output unit155may be mounted to a bezel portion formed to enclose the display unit151.

As shown, the output unit (or the haptic module153, or a feedback output unit) configured to transmit a feedback signal may be disposed near the hologram output unit155, and may be configured to transmit a feedback signal to a user, in response to a gesture of the user with respect to a hologram object output through the hologram output unit155.

An optical source configured to output light so as to output a hologram object, and the haptic module153configured to output a feedback signal may be arranged independently from each other.

Although not shown, the optical source for outputting a hologram object, and the haptic module153for outputting a feedback signal may have the same optical source. In this case, the controller180may control an intensity of laser corresponding to the optical source for outputting a hologram object, thereby implementing a feedback signal with the intensity of the laser.

As shown inFIG. 2B, the hologram output unit155may be disposed on a rear surface of the mobile terminal100, and may be configured to output the holographic image155′ in a preset space above the rear surface of the mobile terminal.

The mounting position of the hologram output unit155, and the output space of the holographic image155′ are not limited to the aforementioned ones. The hologram output unit155may be formed to be rotatable or pop-up, and may be detachably installed at the terminal body as an additional device. The holographic image155′ may be output to a space unrelated to an installation direction of the hologram output unit155, in a tilting manner, or through an additional reflection structure, etc.

The holographic image155′, which can be implemented through the hologram output unit155, may include a two-dimensional (2D) planar image and a three-dimensional (3D) stereoscopic image.

A planar display method, a monoscopic method for providing the same image to two eyes, is configured to arrange a polyhedron generated by one or more points, lines and planes or a combination thereof, in a virtual stereoscopic space, and to display an image obtained as the polyhedron is viewed from a specific view point.

Next, a stereoscopic display method, a stereoscopic method for providing different images to two eyes, is a method that allows a user to feel a cubic effect when he or she views an object with naked eyes. That is, a human's two eyes view different planar images when viewing the same object, due to a distance therebetween. The different planar images are transmitted to the brain through the retinas, and the brain synthesizes the images with each other. As a result, the human feels a depth and a reality of a stereoscopic image. A human may feel a cubic effect by a distance between two eyes, i.e., a binocular disparity, even if there is a difference by person. A method for displaying an image using such a binocular disparity is a stereoscopic display method.

The holographic image155′ formed by the hologram output unit155to be explained later, may include both the 2D image and the 3D image. Hereinafter, for the sake of explanation, an image representation by a planar display method may include an image representation by a stereoscopic display method.

Hereinafter, a method for displaying the holographic image155′, and a structure to implement the holographic image155′ according to the present invention will be explained in more detail.

An image output to the display unit151merely records a distribution of bright and dark surfaces of an object, whereas the holographic image155′ may be understood as an image which simultaneously accumulates and reproduces all of information which light as wave contains, namely, an amplitude and a phase.

FIGS. 3A and 3Bare conceptual views illustrating a hologram principle.

Referring toFIG. 3A, coherent light output from a laser optical source201is divided into two beams through a splitter202.

One of the two beams is used to illuminate a subject207, and light diffusely-reflected from the surface of the subject207reaches a holographic photosensitive material205. Hereinafter, such a beam is called an object wave. For conversion of a path of the beam, mirrors203,204may be used.

The other of the two beams reaches a mirror208along a converted path, and scatters through a lens206, thereby directly reaching a front surface of the holographic photosensitive material205. Hereinafter, such a beam is called a reference wave.

As the object wave and the reference wave interfere with each other on the holographic photosensitive material205, about 500□1,500 interference patterns per 1 mm, which are very delicate and complicated, are generated. A holography storage medium where such interference patterns are recorded, is called a hologram.

Then, as shown inFIG. 3B, when the beams such as the reference waves, namely, reconstruction waves are projected to the holographic photosensitive material205, the interference fringes may serve as diffraction grating, such that the beams can be diffracted at different positions from an incident direction of the reference waves. The diffracted beams may be converged so as to be formed the same as the beams initially reflected from the object, thereby projecting a holographic image209. That is, the initial object waves may be reconstructed through the hologram so as to realize the holographic image209.

Here, when viewed from an inside of a reconstructed wave front, the original object is viewed but it seems like the object is located inside. And, if a viewing point (view point) is moved, a position where the object is viewed is also changed. It feels like to view a 3D image. Also, since the wave front of the original object is reconstructed, it may cause interference against a wave front which comes from a very slightly deformed object.

A display method of the holographic image209may be divided into a transmission-type display method and a reflection-type display method according to a reconstruction method.FIGS. 4A to 4Care conceptual views illustrating a transmission-type holography method, andFIGS. 5A to 5Care conceptual views illustrating a reflection-type holography method.

The transmission-type holography method is a method of illuminating from a rear side of a hologram, and viewing a transmitted image in front of the hologram. In the transmission-type holography method, an object wave and a reference wave are exposed to a holographic photosensitive material in the same direction upon producing a holographic image, and a generated holographic image has a vivid and bright color.

Referring toFIG. 4A, light emitted from a laser optical source301passes through a spatial filter302, and then scatters to a smooth spherical wave. The spherical wave is divided into two beams by a beam splitter305. One of the two split spherical waves is irradiated onto an object308, thereby forming an object wave. The other of the spherical wave is irradiated onto a holographic photosensitive material307, thereby forming a reference wave. The object wave irradiated onto the object308is also irradiated onto the holographic photosensitive material307. For conversion of a path of the beams, mirrors304,306,309may be used.

As the object wave and the reference wave irradiated onto the holographic photosensitive material307interfere with each other, interference patterns are generated. The generated interference patterns are recorded in the holographic photosensitive material307.

That is, as shown inFIG. 4B, both the object wave and the reference wave are projected onto the same surface of the holographic photosensitive material307, thereby generating interference patterns.

Then, as shown inFIG. 4C, if a reconstruction wave which is the same as the reference wave is projected onto the holographic photosensitive material307, the object wave is projected in an opposite direction to the surface on which the object wave and the reference wave have been previously incident. As a result, a holographic image is generated.

A reflection-type holography method, a method for observing an image by illuminating a front side of a hologram and then viewing a reflected image in front of the hologram, is manufactured to make an object wave and a reference wave incident onto a holographic photosensitive material in opposite directions. A holographic image generated by the reflection-type holography method has an excellent cubic effect.

Referring toFIG. 5A, like inFIG. 4A, light emitted from a laser optical source401passes through a spatial filter402, and then scatters to a smooth spherical wave. The spherical wave is divided into two beams through a beam splitter405. One of the two beams is irradiated onto an object, thereby forming an object wave. The other of the two beams is irradiated onto a holographic photosensitive material407(photosensitive film), thereby forming a reference wave. For conversion of a path of the beams, mirrors404,406,409may be used. Unlike inFIG. 4A, the reference wave and the object wave are irradiated onto the holographic photosensitive material407from opposite positions to each other.

That is, as shown inFIG. 5B, the reference wave is projected onto a left surface of the holographic photosensitive material407, and the object wave is projected onto a right upper surface of the holographic photosensitive material407. Then, as shown inFIG. 5C, if a reconstruction wave which is the same as the reference wave is projected onto the holographic photosensitive material407, the object wave is transmitted in an opposite direction. As a result, a holographic image is generated.

The holographic image may be displayed according to a preset holography pattern, according to an embodiment of the present invention. The holography pattern means that a holographic image projected through the hologram output unit155is changed into a preset pattern to thus be provided to a user.

The holography pattern may be variously set according to a method which is to be explained later.

Firstly, the holography pattern may be set as a distance between the hologram output unit and a holographic image is changed according to lapse of time. With such a structure, since a holographic image projected through the hologram output unit155is moveable up and down, a predetermined holography pattern can be set.

Secondly, the holography pattern may be set as a shape of a holographic image projected from the hologram output unit155is changed according to lapse of time. For instance, the controller180may control a holographic image projected from the hologram output unit155to have a circular shape. Then, the controller180may control the circular shape of the holographic image to be changed into a quadrangular shape, as time lapses.

A holographic image projected from the hologram output unit155may be moved right and left, or may be rotated. That is, the holography pattern may be set by moving a projected holographic image right and left, rotating the holographic image, or rotating the holographic image with moving the holographic image right and left, in a state where a distance between the hologram output unit155and the holographic image is constantly maintained.

Alternatively, the holography pattern may be set as a color or a size of a projected holographic image is changed according to lapse of time. Further, the holography pattern may be set by controlling a projection brightness or a reconstruction frequency (the number of times of reconstruction), or by performing illumination (lighting), vibration feedback, sound insertion, image insertion, repetitive projection, etc.

In the above descriptions, the holography pattern is set by an individual factor. However, the holography pattern may be set by a plurality of factors. For instance, the holography pattern may be set as a holographic image is rotated while the holographic image is moved right and left, in a state where a distance between the hologram output unit155and the holographic image is changed according to lapse of time.

In the above descriptions, a holography pattern is set with respect to an entire region of a holographic image. However, this is merely exemplary. That is, a hologram pattern may be applicable to a partial region of a holographic image.

So far, the transmission-type holography display method and the reflection-type holography display method were explained. However, a holographic image may be output in various manners.

A holography method may be categorized into three according to a generation and reproduction method of a hologram. The holography method may include i) an analogue holography method such as the aforementioned transmission-type holography display method and the reflection-type holography display method, ii) a digital holography method for generating a hologram by 3D digital data captured by an imaging sensor such as a CCD camera or a CMOS sensor, and for storing, processing and editing the generated hologram, and iii) a pseudo holography method for implementing a hologram effect by capturing a stereoscopic image of ultra-multi view points even if it is not a complete hologram, and for imitating a hologram image effect by projecting an image through a semi-transmissive screen.

The pseudo holography method includes a free-format method, a peppers ghost method, and a Leia display system.

The 3D holography method includes a project vermeer, a voixe box, a 3D midair plasma display method, etc.

In the mobile terminal according to the present invention, a holographic image may be implemented by one of the aforementioned methods, or by a method rather than the aforementioned methods.

The holographic image is a three-dimensional (3D) image output to an arbitrary space. In order to touch the holographic image, a user should locate his or her hand to the arbitrary space where the holographic image has been output. However, the user cannot substantially touch the holographic image, since the holographic image is not an entity, but an image output to a 3D-space. Thus, the present invention provides a mobile terminal capable of providing a user's experience to feel that the user substantially touches a holographic image or an object corresponding to the holographic image, and a control method thereof.

The present invention is to provide an interaction between a holographic image and a user, and a more realistic user interaction based on a characteristic of an object corresponding to a hologram object.

Hereinafter, a method for providing an interaction between a holographic image and a user will be explained in more detail with reference to the attached drawings.

FIG. 6is a flowchart illustrating a method for controlling a mobile terminal, for interaction between a hologram object and a user, according to the present invention, andFIG. 7is a conceptual view illustrating an interaction between a hologram object and a user, in a mobile terminal according to the present invention.

For interaction between a holographic image output from the mobile terminal according to the present invention and a user, a hologram object is output to a preset space (S610). The hologram object may mean a holographic image, or part of the holographic image. The part of the holographic image may be a specific image included in the holographic image. The specific image may be an image corresponding to any object. Hereinafter, a holographic image output through the hologram output unit155will be called a ‘hologram object’.

The hologram object is output through the hologram output unit155, and may be output to a preset space (preset 3D space). The preset space may correspond to a projection range of the hologram output unit155.

As shown inFIG. 7, a hologram object710may be output to a preset space. In the output state of the hologram object710, a step (S620) of sensing a human body1000(e.g., a user's hand, refer toFIG. 7) disposed in the preset space is executed (refer toFIG. 6). In this specification, the human body is located at the preset space. However, the present invention is not limited to this. That is, in the present invention, an object positioned at the preset space may be also sensed.

In the present invention, not only a human body positioned at the preset space, but also a human body positioned at an output region of the hologram object710and a peripheral region may be sensed.

In the mobile terminal of the present invention, a position of the sensing unit140may be preset such that a preset space corresponding to a projection range where the output unit155projects a hologram object, is sensed.

The sensing unit (or sensing means) for sensing a human body positioned at the preset space may be implemented in various manners. For instance, the sensing unit may be implemented as a 3D camera.

More specifically, in the mobile terminal according to the present invention, the camera121may include a 3D camera or a depth camera. In the present invention, an image sensor provided at a 3D camera (depth camera) may be configured to sense a human body positioned at an arbitrary space (or a preset space) to which a holographic image (or a hologram object) is output. The image sensor may be configured to sense a user's body, and to sense a user's gesture applied to an arbitrary space. In the mobile terminal according to the present invention, an image having a cubic effect (or a 3D image), which corresponds to an object (or a human body) positioned near a hologram object, or an object (or a user body) approaching a hologram object, may be acquired by the 3D camera.

The image sensor is not always activated, but may be activated only when a hologram object has been output. In a case where a hologram object has been output, if it is sensed by the proximity sensor, the illumination sensor, etc. that an object (or a user's body) is positioned near the hologram object, the controller may activate the image sensor such that the body's approach to the hologram object is sensed.

In the present invention, a body's approach to a hologram object is sensed based on sensing information acquired by the image sensor (S630). More specifically, the controller180detects a body's approach to a hologram object, based on a relative position between the hologram object and the body.

The body's approach to the hologram object may be understood as a user's gesture with respect to the hologram object. The controller180may determine user's various gestures with respect to the hologram object, based on sensing information acquired by the image sensor, e.g., according to whether a user approaches a hologram object (i), a user touches a hologram object (ii), a user scrubs a hologram object (iii), a user passes through a hologram object (iv), a user pushes a hologram object at a constant speed (v), etc.

For instance, the controller180may compare coordinates information of an output position of the hologram object, with coordinates information of the human body, based on an image acquired from the image sensor. The controller180determines a relative position between the hologram object and the human body, based on the coordinates information. If the coordinates information of the human body is the same or similar to the coordinates information of the hologram object as a result of the comparison, it may be determined that the human body has approached (i), contacted (ii), or passed through the hologram object (iii).

The coordinates information of the hologram object and the coordinates information of the human body may be acquired through modeling with respect to the hologram object and the human body, respectively. A method for modeling the hologram object and the human body will be explained in more detail with reference toFIGS. 8A to 8C.

Once the human body's approach to the hologram object is detected, the controller180controls the output unit based on a result of the detection, such that a feedback signal is transmitted to the human body which has approached the hologram object (S640).

That is, in the present invention, a feedback signal is transmitted to an object which has approached a hologram object, such that a user feels as if he or she substantially touches the object corresponding to the hologram object.

The feedback signal may be implemented in various manners. For instance, the output unit may include at least one of a laser output module and a sound wave output module. The feedback signal may include at least one of laser output through the laser output module, and a sound wave output through the sound wave output module. The laser or the sound wave has a predetermined intensity (e.g., strength or amplitude), a period (e.g., frequency), a waveform (e. g., pulse wave, sine wave, etc.). The laser or sound wave output through the output unit is transmitted to a user's body, so the user may have an experience to substantially touch a hologram object. The feedback signal may be also called a ‘haptic signal’. The output unit may be implemented as the haptic module153.

The controller180may acquire coordinates information of a target point where a feedback signal is to be transmitted, based on coordinates information obtained after modeling a human body which has approached a hologram object. That is, the controller180may utilize coordinates information of the modeled human body, such that a feed back signal is transmitted to a position where the human body is currently positioned. In this case, the controller180may output a feedback signal not to all regions where a hologram object has been output, but only to a region where the human body is positioned.

In the present invention, the hologram output unit155may be independently implemented from an output unit for outputting a feedback signal. That is, the hologram output unit155may be independently operated regardless of an activated or inactivated state of the output unit for outputting a feedback signal. In this case, the hologram output unit155and the output unit for outputting a feedback signal, may have separate configurations. That is, the hologram output unit155may be implemented as an optical source such as an LED or a laser of a visible ray range. And the output unit for outputting a feedback signal may be implemented as a laser output module, a sound wave output module, an ultrasonic wave output module, a wind output module, a mist output module or the like separately provided from the hologram output unit155.

If the human body's approach to the hologram object is detected in an inactivated state of the output unit, the controller180may convert the inactivated state of the output unit into an activated state, such that a feedback signal is output. That is, the output unit, which is in an inactivated state in ordinary times, is converted into an activated state when a feedback signal is to be output. The inactivated state means an ‘off’ state, and the activated state means an ‘on’ state.

In the present invention, the output unit is controlled to output a feedback signal only when a user's approach to a hologram object is sensed. This can reduce power consumption more than in a case where the output unit is always in an ‘on’ state.

Further, in the present invention, when a human body has approached a hologram object, a feedback signal is output only to the human body, not to an entire region of the hologram object, based on coordinates information of the human body. This can reduce power consumption more than in a case where a feedback signal is output to the entire region of the hologram object

In the mobile terminal according to the present invention, when there is a function associated with a user's contact part, the controller may execute the function. For instance, if a hologram object of a user's contact part is an icon, the controller may execute a function related to the icon. In this case, the hologram output unit155may output screen information corresponding to an executed function, in the form of a hologram.

As aforementioned, in the mobile terminal and the control method according to the present invention, when a human body's approach to a hologram object is sensed, a feedback signal is output to the human body. This can allow a user to have a sense of substantially touching a hologram object. Hereinafter, the present invention will be explained in more detail.

Firstly, a method for detecting a human body's approach to a hologram object by modeling the hologram object and the human body, will be explained in more detail with reference to the attached drawings.

FIGS. 8A to 8Eare conceptual views illustrating a method for modeling (rendering) a hologram object and a user's body, in the mobile terminal according to the present invention.

As shown inFIG. 8A, the mobile terminal may model a hologram object810, and a human body sensed by the image sensor.

Referring toFIG. 8A, the controller180may model the hologram object810, a peripheral region of the hologram object810, and a human body1000. As a result of the modeling, there may exist a first modeling object810acorresponding to the hologram object810, a second modeling object810bcorresponding to the peripheral region of the hologram object810, and a third modeling object1000acorresponding to the human body.

The peripheral region may be a region within a predetermined distance, based on an output region of the hologram object810. A range of the peripheral region may be set under control of the controller180, or according to a user's selection.

The controller180executes modeling with respect to the hologram object810, the peripheral region, and the sensed human body, and applies a preset physical engine to modeled objects to execute a collision detection. Through the collision detection, the controller180may detect whether a collision has occurred among the human body, the hologram object810and the peripheral region.

For instance, once first to third modeling objects810a,810b,1000aare modeled, the controller180may acquire coordinates information of the modeling objects. With such a structure, the controller180may recognize an output position of the hologram object810in the preset space, and may recognize a peripheral region of the hologram object810.

Modeling the human body may mean capturing a human body using a 3D camera, and modeling an image object corresponding to the human body included in a captured image.

The controller180may acquire coordinates information of the hologram object810and the peripheral region of the hologram object810through the modeling, and acquire coordinates information of the human body. Then, the controller180may compare the coordinates information with each other, thereby determining a relative position between the peripheral region of the hologram object810and the human body.

The controller180may compare coordinates information of an output position of the hologram object, coordinates information of the peripheral region, and coordinates information of the human body, with one another. Then, the controller180may determine whether the coordinates information of the human body is the same as the coordinates information of the hologram object810or the peripheral region. Based on a result of the comparison, the controller180may determine that the human body has approached the hologram object (i), has contacted the hologram object (ii), or has passed through the hologram object (iii).

In a preset space where the hologram object810is output, coordinates information of a first region where the hologram object810is positioned, coordinates information of a second region (or the peripheral region) adjacent to the first region, and coordinates information of a region where the human body is positioned, may be acquired through a specific type of modeling with respect to the hologram object and the human body.

FIG. 8Bis a conceptual view illustrating a plurality of modeling types. A method for modeling a hologram object may include a vertex-based modeling type as shown inFIG. 8B(a), a line (edge)-based modeling type as shown inFIG. 8B(b), a sphere-based modeling type as shown inFIG. 8B(c), and a mesh-based modeling type as shown inFIG. 8B(d).

As shown inFIG. 8B(a), according to the vertex-based modeling type, when a user is to select a specific region of a hologram object, the controller180may activate vertexes of part corresponding to the specific region, among the modeled hologram object, thereby recognizing the user's approach to the hologram object.

As shown inFIG. 8B(b), according to the line (edge)-based modeling type, when a user is to select a specific region of a hologram object, the controller180may activate lines of part corresponding to the specific region, among the modeled hologram object, thereby recognizing the user's approach to the hologram object.

As shown inFIG. 8B(c), according to the sphere-based modeling type, when a user is to select a specific region of a hologram object, the controller180may activate spheres of part corresponding to the specific region, among the modeled hologram object, thereby recognizing the user's approach to the hologram object.

As shown inFIG. 8B(d), according to the mesh-based modeling type, when a user is to select a specific region of a hologram object, the controller180may activate meshes of part corresponding to the specific region, among the modeled hologram object, thereby recognizing the user's approach to the hologram object.

The above vertex-based modeling type, the line (edge)-based modeling type, the sphere-based modeling type, and the mesh-based modeling type have been well known to those skilled in the art, and thus detailed explanations thereof will be omitted.

A modeling type of the hologram object and a modeling type of the peripheral region of the hologram object may be the same or different from each other.

As shown inFIG. 8C, a user's body may be modeled by one of the aforementioned plurality of modeling types. For instance, the controller180may model a human body approaching a hologram object, based on an image acquired from a 3D camera (or a depth camera). More specifically, the controller180may apply a dot modeling type shown inFIG. 8C(a), a sphere modeling type show inFIG. 8C(b), or a mesh modeling type shown inFIG. 8C(c). As a result of such a modeling, a shape of the human body may be implemented as shown.

The controller180may model a human body (more specifically, a user's hand) using a position recognition sensor such as a ring type, a thimble type or a glove type, rather than the 3D camera.

As shown, the controller180detects a human body's approach to a hologram object, based on a modeling result with respect to the hologram object and the human body. The controller180determines whether a 3D point is included in a human body's 3D point (coordinates information) acquired as a result of the modeling, and in a model of a hologram object (e.g., mesh model, or coordinates information of a model corresponding to a hologram object). Alternatively, the controller180determines whether there is a sphere including a human body's 3D point, among spheres which constitute a hologram object, thereby sensing the human body's approach to the hologram object.

Still alternatively, the controller180determines whether there is a vertex or a line corresponding to a human body's 3D point, among vertices or lines which constitute a hologram object, thereby sensing the human body's approach to the hologram object.

The coordinates information of the hologram object, the peripheral region, and the human body, which has been acquired based on the aforementioned modeling, may be variable according to each modeling type. For instance, coordinates values by a sphere-based modeling type may be different coordinates values by a line-based modeling type.

In the above descriptions, it is determined whether a human body's 3D point is included in a model corresponding to a hologram object. But this may be understood as a determination whether a human body's 3D point (coordinates information) corresponds to coordinates information of a model corresponding to a hologram object.

In the mobile terminal according to the present invention, a hologram object may be modeled by a modeling type based on a characteristic of an object, among a plurality of modeling types. The controller180may model a hologram object, based on a characteristic of an object corresponding to the hologram object. The characteristic of the object corresponding to the hologram object may include at least one of a type, a weight, a size, an intensity, a shape and a surface of the object. For instance, as shown inFIG. 8D, if an object corresponding to a hologram object has a monotonous appearance (e.g., a book), the controller180may model the hologram object by a comparatively simple modeling type rather than a complicated modeling type. Alternatively, as shown inFIG. 8E, if an object corresponding to a hologram object has a complicated appearance (e.g., a cactus), the controller180may model the hologram object by a complicated modeling type, in order to determine a region or part to which a user has approached.

The controller may execute a modeling for visually outputting a hologram object, and a modeling for detecting a human body's approach to a hologram object, in a different manner. For instance, as shown inFIG. 8D, since a book has the same texture or shape entirely, a user has a similar tactile sense regardless of a touched region. Thus, the controller180models an object having such a characteristic, by a comparatively simple modeling type. As another example, as shown inFIG. 8E, since a cactus has a different texture or shape entirely, a user has a different tactile sense according to a touched region. Thus, the controller180models an object having such a characteristic, by a detailed modeling type.

The controller may model part of an object corresponding to a hologram object, the part to be contacted (e.g., button, protrusion (handle), etc.). Then, the controller may detect whether a user has contacted or approached the part to be contacted.

Alternatively, the controller may execute a different modeling type with respect to a single hologram object, according to a region. And the controller may model part to be contacted (e.g., a button or a protrusion), in a more detailed manner.

As aforementioned, in the mobile terminal according to the present invention, a human body's approach to a hologram object may be determined through modeling of the human body and the hologram object, and a feedback signal may be output to the human body.

Hereinafter, examples to output a feedback signal will be explained in more detail with reference to the attached drawings.

FIGS. 9A to 9Gare conceptual views illustrating a position relation between a hologram object and a user's body, in a mobile terminal according to the present invention.FIGS. 10A to 100are conceptual views illustrating a feedback signal according to the present invention.FIGS. 11A to 11Dare conceptual views illustrating a characteristic of an object corresponding to a hologram object.

In the mobile terminal according to the present invention, a feedback signal may be transmitted to a human body, not only when a user's body is positioned within an output region of a hologram object, but also when a user's body is positioned at a peripheral region of a hologram object.

For instance, as shown inFIG. 9A, an output space of a hologram object may include a first region910where the hologram object is positioned, and a second region920(peripheral region) adjacent to the first region910. The controller180may control the output unit to output a feedback signal when the human body1000is positioned on one of the first region910and the second region920.

More specifically, when the human body is positioned either in the first region910, or at a boundary of the first region910, the controller180may determine that the human body is positioned at the first region910.

When the human body is positioned between the first region910and the second region920and at a boundary of the second region920, the controller180may determine that the human body is positioned at the second region920.

As aforementioned, the controller180compares coordinates information of the first region910where the hologram object is positioned in a preset space, coordinates information of the second region920adjacent to the first region910, and coordinates information of the human body1000, with one another. Then, the controller180determines a relative position between the hologram object and the human body1000, and determines whether the human body1000is positioned on the first region910or the second region920based on a result of the determination.

FIG. 9Aillustrates that the human body1000is positioned on the second region920, andFIGS. 9B to 9Dillustrate that the human body1000is positioned on the first region910. Especially,FIG. 9Dillustrates that the human body1000has passed through a hologram object. In the present invention, a feedback control may be output not only when the human body1000is positioned at the first region910, but also when the human body1000is positioned at the second region920. The controller180may control the output unit to output a different feedback signal according to whether the human body1000is positioned at the first region910or at the second region920.

A feedback signal according to the present invention may be defined by at least one of an intensity (e.g., strength or amplitude), a period (e.g., frequency), and a waveform (e.g., pulse wave, sine wave, etc.). That is, a characteristic of a feedback signal is determined by combination of an intensity, a period and a waveform of a signal. The controller180generates a different feedback signal by the output unit, by changing at least one of an intensity, a period and a waveform of a feedback signal. That is, the output unit may generate feedback signals different from each other in at least one of an intensity, a period and a waveform, under control of the controller.

Thus, when a feedback signal is formed as a laser or a sound wave, the controller180may generate a different feedback signal by controlling at least one of an intensity, an amplitude, a frequency and a waveform of the laser or the sound wave.

As shown inFIG. 10a, a signal strength may mean one of a signal intensity and a signal size. As shown inFIG. 10B, a signal period means a signal frequency. A signal shown inFIG. 10B(b) has a lower frequency than a signal shown in FIG.10B(a). For instance, a feedback signal is implemented as a laser, the number of times that a laser signal having a frequency shown inFIG. 10B(a) reaches a human body is larger than the number of times that a laser signal having a frequency shown inFIG. 10B(b) reaches a human body.

As shown inFIG. 10C, a signal waveform may be variable. For instance, the signal waveform may be a pulse wave shown inFIG. 10C(a), a sine wave shown inFIG. 10C(b), an impulse wave shown inFIG. 10C(c), etc. As shown inFIG. 10C(d), the signal waveform may be a combination of a plurality of waveforms. Further, in case of the same pulse wave, a different type of signal may be implemented according to a pulse width.

The controller180may output a signal having a higher frequency when the human body1000is on the first region910, than when the human body1000is on the second region920.

Also, the controller180may output a different feedback signal when the human body1000comes closer to a hologram object. For instance, when the human body1000comes closer to a hologram object, the controller180may output a signal of a higher frequency, or a signal of a higher intensity. Alternatively, when the human body1000comes closer to a hologram object, the controller180may output a feedback signal with changing a waveform such that the feedback signal has a different waveform.

The controller180may output a feedback signal having at least one of a different intensity, frequency and waveform, according to a distance between a hologram object and the human body1000.

Alternatively, when the human body1000comes closer to a hologram object, the controller180may output a feedback signal having a linearly-increasing intensity, or a feedback signal having a nonlinearly-increasing intensity (e.g., nonlinearity such as logarithm).

On the contrary, when the human body becomes distant from a hologram object, the controller180may output a different feedback signal. In this case, an intensity of the feedback signal may be gradually decreased, or a frequency of the feedback signal may be gradually decreased.

In the present invention, a different feedback signal is output according to a distance between a hologram object and a human body. Thus, a user may feel that he or she is moving far away from or close to a hologram object.

The controller180may continuously output a signal having a predetermined intensity when the human body1000has reached a threshold range, e.g., when the human body1000has approached a hologram object or has reached within a preset distance.

The controller180may output a different feedback signal according to a speed of the human body1000which is approaching a hologram object, i. g., a collision speed of the human body1000against a hologram object. Under an assumption that there are a first speed and a second speed larger than the first speed, if the human body1000collides with the hologram object with the first speed, the controller180may output a signal having a first size. On the contrary, if the human body1000collides with the hologram object with the second speed, the controller180may output a signal having a second size larger than the first size.

Alternatively, when the human body1000collides with the hologram object with the first speed, the controller180may output a sine wave having a predetermined size. When the human body1000collides with the hologram object with the second speed, the controller180may output an impulse wave.

When the human body approaches the hologram object with a first speed, the controller180may output a first feedback signal. On the other hand, when the human body approaches the hologram object with a second speed different from the first speed, the controller180may output a second feedback signal different from the first feedback signal.

As shown inFIG. 9B, when the human body1000has contacted a hologram object, the controller180may output a different type of feedback signal according to a characteristic of an object corresponding to the hologram object. The characteristic of the object may include at least one of a type, a weight, a size, an intensity, a shape and a surface of the object.

The controller180may output a feedback signal the most suitable for a characteristic of an object, by controlling at least one of an intensity, a frequency and a waveform of a feedback signal, according to the characteristic of the object.

For instance, the controller180may control a feedback signal such that a user feels a texture of an object corresponding to a hologram object. For instance, if an object corresponding to a hologram object has a rough surface (e.g., tree) (refer toFIG. 9D), the controller180may output an impulse wave in response to a user's gesture to contact a hologram object (refer toFIG. 9B), or a user's gesture to scrub (touch) a hologram object (refer toFIG. 9C). As another example, if an object corresponding to a hologram object has a soft surface (e.g., cotton candy) (refer toFIG. 9E), the controller180may output a sine wave in response to a user's gesture to contact a hologram object (refer toFIG. 9B), or a user's gesture to scrub (touch) a hologram object (refer toFIG. 9C).

As still another example, if an object corresponding to a hologram object has a surface such as an irregular gravelly field, the controller180may output a feedback signal by combining at least two of a sine wave, a pulse wave and an impulse wave, in response to a user's gesture to contact a hologram object (refer toFIG. 9B), or a user's gesture to scrub (touch) a hologram object (refer toFIG. 9C).

The controller180outputs a different feedback signal in response to a user's gesture with respect to a hologram object. In this case, a characteristic of a feedback signal may be determined according to a characteristic of an object corresponding to the hologram object.

For instance, if a user's gesture is to push or pull a hologram object as shown inFIG. 9F(a) or9F(b), the controller180may output a proper feedback signal according to a characteristic of an object corresponding to the hologram object.

For instance, based on the same gesture to push or pull a hologram object, the controller may output a different feedback signal, according to a weight of an object corresponding to the hologram object, or a surface texture.

For instance, when a weight of an object corresponding to a hologram object is heavy, a feedback signal of a large intensity may be output. With such a configuration, a user may feel that he or she actually pushes or pulls the object.

The controller180of the mobile terminal according to the present invention may output a feedback signal in response to a human body's gesture with respect to a hologram object. Further, the controller180may control the hologram output unit such that at least one of an output position and a shape of the hologram object is changed.

That is, as shown inFIGS. 9F(a) and9F(b), the controller180may move the hologram object in a preset output space, in response to a gesture of the human body. Also, if an object corresponding to a hologram object is not pushable, the controller180may inform a user that the hologram object is not pushable, by changing a shape of the hologram object. For instance, the hologram object may have a distorted shape.

As shown inFIG. 9F(c), even if an object corresponding to the hologram object is moveable, if the hologram object is positioned at a boundary region of a preset space (e.g., outputtable range), the controller180may change a shape of the hologram object, without moving the hologram object in response to a gesture of the human body. Alternatively, even if the hologram object is not positioned at a boundary region of a preset space, if there is an external object (e.g., wall) near the hologram object, the controller180may not move the hologram object even if a user's gesture to push the hologram object is sensed. The external object may be sensed by a 3D camera, a proximity sensor, etc.

If an object corresponding to the hologram object is not transformable (e.g., tree), even if the hologram object is not pushable any longer, the controller180may not change a shape of the hologram object. In this case, the controller180may continuously output a signal having at least one of a predetermined intensity, frequency and waveform.

If the hologram object is moved or is transformed by a user's gesture, the controller may output a feedback signal related to the movement or the transformation of the hologram object, to a user. In this case, the feedback signal may be changed according to a characteristic of an object. For instance, the controller may output signals having different intensities when objects corresponding to a hologram object are a balloon and a tree.

If the hologram object is moved, the hologram object may become distant from a user's body. Accordingly, the controller180may additionally provide a feedback signal. That is, the controller180provides a feedback signal based on a distance between a hologram object and a user. Explanations about that have been aforementioned with reference toFIGS. 9A and 9B, and thus detailed explanations thereof will be omitted.

If an object corresponding to the hologram object is an object through which a user is to pass, the controller180may provide a visual effect that a user's body seems to have passed through the hologram object, as shown inFIG. 9F(d). In this case, the controller180may output a proper feedback signal such that the user feels as if he or she has entered the object corresponding to the hologram object. For instance, the object may be cheese, and the feedback signal may be a laser or a sound wave having a sine waveform.

In case of moving a hologram object, the controller180may move an entire part of the hologram object, or may move part of the hologram object based on a reference axis, a reference point, a reference line or a reference surface. For instance, as shown inFIG. 9G(a), if an object corresponding to a hologram object has a reference axis (a reference line or a reference surface) (e.g., a rotation point, a rotation surface, a joint point or a joint surface), the controller180may move only a region920of the hologram object based on the reference axis, as shown inFIG. 9B(b). The object may be a rotatable door, a user's body having joints, etc.

As aforementioned, in the mobile terminal according to the present invention, a different feedback signal is provided based on a characteristic of an object corresponding to a hologram object, and based on a relative position between the hologram object and a user. As a result, a user may be provided with a feedback in a more realistic manner.

In the present invention, information on a region where a feedback signal can be output may be output by using at least one of visible, tactile and audible methods. This will be explained in more detail with reference to the attached drawings.

FIG. 11is a conceptual view illustrating a method for informing a region where a feedback signal can be output, in the mobile terminal according to the present invention.

In the mobile terminal according to the present invention, when a user approaches a hologram object, guide information indicating an output region of a feedback signal may be output. In this case, the guide information may be implemented as visible information, tactile information or audible information.

The controller180may output a specific graphic object to the peripheral region of the hologram object (or the second region, refer to920ofFIG. 9A). That is, the controller180may inform a user that a feedback signal is outputtable when the user enters a corresponding region, by outputting such a graphic object. The controller180may output guide information or notification information indicating that a feedback signal can be output, when a user's body is positioned at the peripheral region of the hologram object.

Further, the controller180may control the hologram output unit155to output the notification information, to a region where the feedback signal can be transferred, the region corresponding to a position of the human body, among the peripheral region of the hologram object. In this case, the controller180may output the notification information to parts1320,1330corresponding to a region where the human body is positioned, not an entire part of the peripheral region. In this case, the notification information may be output even when a user is positioned near a region where a feedback signal can be output. With such a configuration, a user may feel that he or she is approaching the hologram object in a more realistic manner.

FIGS. 12A and 12Bare conceptual views illustrating another embodiment to output a feedback signal.

In the above embodiment, a feedback signal is transmitted through the output unit of the mobile terminal. However, in the mobile terminal according to the present invention, a feedback signal may be transmitted to a user through an external device which communicates with the wireless communication unit110. For instance, the controller180may transmit a feedback signal to a user through data communication with a ring-type external device shown inFIG. 12A, or a watch-type external device shown inFIG. 12B. The ring-type or watch-type external device may include a haptic module for outputting a feedback signal.

The ring-type or watch-type external terminal may transmit a feedback signal to a body of a user who is approaching or contacting a hologram object. In this case, information on coordinates where the feedback signal is transmitted, may be received from the mobile terminal. Since the ring-type or watch-type external terminal is always arranged close to a body of a user who is approaching or contacting a hologram object, the user may be provided with a feedback in a more realistic manner.

The ring-type or watch-type external terminal may be further provided with a pulse sensor or a photo-plethysmography (PPG) sensor. The PPG sensor means a sensor for sensing bio information indicating a blood flow rate based on an amount and a change of incident light, and indicating pulsation components generated by heartbeats. The PPG sensor is configured to measure a blood flow rate by irradiating light of a specific wavelength to a human body by using a light emitting unit provided thereat, and by receiving the irradiated light by using a light receiving unit provided thereat as the irradiated light is reflected after passing through the human body.

When a feedback signal is transmitted to a user, the ring-type or watch-type external terminal may sense a response of the user to the feedback signal, by sensing a blood flow rate of the user through a PPG sensor.

In a case where a feedback signal is transmitted to a user, if a blood flow rate of the user sensed by a PPG sensor is different from that before the feedback signal is transmitted to the user (i.e., if a blood flow rate has a different pattern or is drastically changed), the ring-type or watch-type external terminal may determine that the user has received the feedback signal.

That is, if a user's blood flow rate (or pulsation component) sensed by a PPG sensor has a drastic change, the ring-type or watch-type external terminal may determine that the user was under stress. Further, the ring-type or watch-type external terminal may determine that the stress has resulted from a tactile feedback signal applied to the user. As a result, the ring-type or watch-type external terminal may determine that the user has received a feedback signal.

Sensing a user's bio information with respect to a feedback signal, by the ring-type or watch-type external terminal through the PPG sensor may be also executed when the feedback signal is transmitted from a mobile terminal rather than the ring-type or watch-type external terminal.

In this case, a time point of sensing by the PPG sensor of the ring-type or watch-type external terminal may be the same or similar as or to a time point when the feedback signal is transmitted from the mobile terminal. Information about the time point may be transmitted from the mobile terminal to the ring-type or watch-type external terminal. The PPG sensor of the ring-type or watch-type external terminal may be controlled by the controller180of the mobile terminal.

The sensing information through the PPG sensor may be transmitted to the mobile terminal from the ring-type or watch-type external terminal. Then, the controller180of the mobile terminal may control an intensity of a feedback signal based on the received sensing information. That is, if a blood flow rate of a user is drastically changed as a sensing result, it may be determined that the user is under a lot of stress due to a feedback signal. In this case, the controller180may output the feedback signal with a lower intensity than in the conventional case.

As aforementioned, the mobile terminal according to the present invention may have the following advantages.

Firstly, a feedback signal is transmitted with a separate configuration from the mobile terminal. That is, a user may be provided with a feedback in a more realistic manner according to various modifications. With such a configuration, a feedback signal may be transmitted to a body of a user approaching a hologram object, by using the output unit, especially, the haptic module. Thus, the user may feel that he or she is actually touching a hologram object output to an arbitrary space.

Secondly, a different feedback signal may be transmitted to a body of a user, according to a characteristic of an object corresponding to a hologram object. For instance, the user may be provided with different feedback signals when objects corresponding to a hologram object are cotton candy and wood.

Thirdly, a characteristic of an object corresponding to a hologram object may be represented as an amplitude, an intensity, a frequency, a waveform, etc. of a feedback signal are variously changed. Thus, a user may feel that he or she is actually touching the object corresponding to the hologram object.

Fourthly, since modeling of a hologram object is differently executed according to a type of an object corresponding to a hologram object, a processing speed may be enhanced.