Mobile terminal and controlling method thereof

A mobile terminal and controlling method thereof are disclosed by which a user can be informed of an event associated with augmented reality based on an occurrence of the event using various haptic effects that differ from each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

Pursuant to 35 U.S.C. §119(a), this application claims the benefit of earlier filing date and right of priority to Korean Patent Application Nos. 10-2010-0082135, filed on Aug. 24, 2010, and 10-2010-0087455 filed on Sep. 7, 2010, the contents of which are hereby incorporated by reference herein in their entirety.

FIELD OF THE INVENTION

The present invention relates to a mobile terminal, and more particularly, to a mobile terminal and controlling method thereof. Although the present invention is suitable for a wide scope of applications, it is particularly suitable for enhancing convenience in using the mobile terminal.

DISCUSSION OF THE RELATED ART

Mobile terminals are presently configured to perform various functions. Such functions include data and voice communications, capturing still images and video via a camera, recording audio, playing music files and outputting music via a speaker system, and displaying images and video on a display. Some mobile terminals include additional functionality that supports game playing, while other mobile terminals are also configured as multimedia players. More recently, mobile terminals have been configured to receive broadcast and multicast signals that permit viewing of content, such as videos and television programs.

Generally, terminals can be classified into mobile terminals and stationary terminals according to a presence or non-presence of mobility. Mobile terminals can also be classified into handheld terminals and vehicle mount terminals according to whether they are configured to be carried by hand.

There are ongoing efforts to support and increase the functionality of mobile terminals. Such efforts include software and hardware improvements, as well as changes and improvements in the structural components that form the mobile terminal.

Recent developments in mobile terminal technology include the use of augmented reality (hereinafter, “AR”) technology for various purposes. AR is included in the field of virtual reality. AR is a computer graphic scheme that enables a virtual object or information to appear as if it includes a real object existing in an original environment by synthesizing a real environment with the virtual object or information.

For example, AR allows a camera of a mobile terminal that is focused on an object, such as a building, to have an object information item, such as a phone number or a homepage, appear above or on top of the object. A user is able to use the object information item either directly or indirectly.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to a mobile terminal and controlling method thereof that substantially obviate one or more problems due to limitations and disadvantages of the related art. An object of the present invention is to provide a mobile terminal and a controlling method thereof by which a user can be informed of an occurrence of an event associated with augmented reality by using various haptic effects.

To achieve these objects and other advantages and in accordance with the purpose of the invention, as embodied and broadly described herein, a mobile terminal according to the present invention includes a mobile terminal including a camera configured for capturing an image for an augmented reality (AR) function of the mobile terminal, a position location module configured for obtaining a current position of the mobile terminal, a haptic module configured for generating a plurality of haptic effects, a display unit configured for displaying the captured image as a preview image, and a controller configured for controlling the display unit to display at least one object included in the preview image and at least one object information item related to the at least one object based on the current position of the mobile terminal, and controlling the haptic module to generate at least one haptic effect indicating an occurrence of an event, wherein the event is associated with the AR function.

In another aspect of the present invention, a method of controlling a mobile terminal includes capturing an image for an augmented reality (AR) function via a camera of the mobile terminal, displaying the captured image as a preview image on a display unit of the mobile terminal, obtaining a current position of the mobile terminal via a position location module of the mobile terminal, displaying at least one object included in the preview image and at least one object information item related to at least one object based on the current position of the mobile terminal, detecting an occurrence of an event associated with the AR function, and generating at least one haptic effect, via a haptic module, indicating the occurrence of the event.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, the suffixes “module” and “unit” are used for facilitation of the detailed description of the present invention and do not have meanings or functions different from each other. Therefore, significant meanings or roles are not given to the suffixes themselves and it is understood that the ‘module’, ‘unit’ and ‘part’ can be used together or interchangeably.

The present invention can be applicable to a various types of terminals. Examples of such terminals include mobile as well as stationary terminals, such as mobile phones, user equipment, smart phones, DTV, computers, digital broadcast terminals, personal digital assistants, portable multimedia players (PMP) and navigation systems.

However, by way of non-limiting example only, further description will be provided with regard to a mobile terminal100. It should be noted that such teachings may apply equally to other types of terminals.

FIG. 1is a block diagram of a mobile terminal100in accordance with an embodiment of the present invention. Referring toFIG. 1, the mobile terminal100includes a wireless communication unit110, an audio/video (A/V) input unit120, a user input unit130, a sensing unit140, an output unit150, a memory160, an interface unit170, a controller180, and a power supply unit190. AlthoughFIG. 1shows the mobile terminal100having various components, it is understood that implementing all of the illustrated components is not a requirement. More or fewer components may alternatively be implemented.

The wireless communication unit110includes one or more components that permit wireless communication between the mobile terminal100and a wireless communication system or a network within which the mobile terminal100is located. For example, the wireless communication unit110includes a broadcast receiving module111, a mobile communication module112, a wireless Internet module113, a short-range communication module114and a position-location module115. For non-mobile terminals, the wireless communication unit110may be replaced with a wired communication unit. The wireless communication unit110and a wired communication unit (not shown) may be commonly referred to as a communication unit.

The broadcast receiving module111receives a broadcast signal and/or broadcast associated information from an external broadcast managing entity via a broadcast channel. The broadcast channel may include a satellite channel and/or a terrestrial channel.

The broadcast managing entity may refer to a system that transmits a broadcast signal and/or broadcast associated information. The broadcast managing entity may be a server that generates and transmits broadcast signals and/or broadcast associated information or a server for receiving previously generated broadcast signals and/or broadcast-related information and transmitting the broadcast signals and/or the broadcast associated information to the mobile terminal100. The broadcast signals may include not only TV broadcast signals, radio broadcast signals, and data broadcast signals, but also signals in the form of a TV broadcast signal combined with a radio broadcast signal.

The mobile terminal100may include at least two broadcast receiving modules111for providing simultaneous reception of at least two broadcast channels. The at least two broadcast receiving modules may also be used to facilitate switching of broadcast channels.

The broadcast associated information may be information about a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast associated information may even be provided over a mobile communication network. In the latter case, the broadcast associated information may be received via the mobile communication module112. Examples of broadcast associated information include an electronic program guide (EPG) of digital multimedia broadcasting (DMB) and an electronic service guide (ESG) of digital video broadcast-handheld (DVB-H).

The broadcast receiving module111may receive broadcast signals transmitted from various types of broadcast systems. As a non-limiting example, the broadcast systems include digital multimedia broadcasting-terrestrial (DMB-T), digital multimedia broadcasting-satellite (DMB-S), digital video broadcast-handheld (DVB-H), a data broadcasting system known as media forward link only (MediaFLO®) and integrated services digital broadcast-terrestrial (ISDB-T). The broadcast receiving module111may also receive multicast signals. The broadcast signals and/or the broadcast associated information received by the broadcast receiving module111may be stored in a suitable storage device, such as in the memory160.

The mobile communication module112transmits/receives wireless signals to/from at least one of a base station, an external terminal or a server over a mobile communication network. The wireless signals may represent, for example, voice call signals, video telephony call signals or data in various forms according to the transmission/reception of text and/or multimedia messages.

The mobile communication module112may also be used to transmit current position information of the mobile terminal100obtained via the position-location module115, which will be described later, to an external server (not shown) and to receive or download data relevant to the area in which the mobile terminal100is located from the external server. The downloaded data may include detailed map data of the area and an image representing the area where the mobile terminal100is located.

An object information item database relevant to objects located within a specific radius of the area can be included in the map data. As an example, the objects can include buildings. The object information items can include information about facilities located within the buildings, such as hospitals, restaurants, toilets, police stations, or community centers.

A real image of each area, position information of objects included within the real image and detailed information on the objects are included in the map data. The real image can include an image having the same view as a preview image captured via the camera121or an image corresponding to a sky view.

The wireless Internet module113supports Internet access for the mobile terminal100. The wireless Internet module113may be internally or externally coupled to the mobile terminal100. Suitable technologies for wireless Internet include, but are not limited to, WLAN (Wireless LAN), Wi-Fi®, Wibro® (Wireless broadband), Wimax® (World Interoperability for Microwave Access), and HSDPA (High Speed Downlink Packet Access). The wireless Internet module113may be replaced with a wired Internet module (not shown) in non-mobile terminals. The wireless Internet module113and the wired Internet module may be commonly referred to as an Internet module. Moreover, as mentioned in the foregoing description, the wireless internet module113can receive or download the data relevant to the area in which the mobile terminal100is located from the external server.

The short-range communication module114facilitates relatively short-range communications. Suitable technologies for short-range communication include, but are not limited to, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), as well as networking technologies such as Bluetooth® and ZigBee®. Moreover, as mentioned in the foregoing description, the short-range communication module114is able to receive or download the data relevant to the area in which the mobile terminal100is located from the external server of another terminal located in the vicinity of the mobile terminal100.

The position-location module115identifies or otherwise obtains a location of the mobile terminal100. The position-location module115may obtain position information by using a global navigation satellite system (GNSS). The GNSS is a term used to describe radio navigation satellite systems configured to send reference signals capable of determining their positions on the surface of the earth or near the surface of the earth while revolving around the earth. The GNSS includes: a global position system (GPS) operated by the U.S.A.; Galileo, operated by Europe; a global orbiting navigational satellite system (GLONASS) operated by Russia; COMPASS, operated by China; and a quasi-zenith satellite system (QZSS) operated by Japan.

As a typical example of the GNSS, the position-location module115is a GPS module. The position-location module115may calculate information related to distances between one point or object and at least three satellites and information related to the time when the distance information was measured and apply trigonometry to the obtained distance information to obtain three-dimensional position information on the point or object according to the latitude, longitude, and altitude at a predetermined time. Furthermore, a method of calculating position and time information using three satellites and correcting the calculated position and time information using another satellite may also used. The position-location module115continues to calculate a current position in real time and to calculate velocity information based on the position information.

With continued reference toFIG. 1, the audio/video (A/V) input unit120may be configured to provide audio or video signal input to the mobile terminal100. The A/V input unit120may include a camera121and a microphone122. The camera121processes image frames of still pictures or video obtained by an image sensor in a photographing mode or a video telephony mode. The processed image frames may be displayed on a display unit151.

The image frames processed by the camera121may be stored in the memory160or transmitted to an external device through the wireless communication unit110. Optionally, the mobile terminal100can include two or more cameras121, if appropriate.

The microphone122receives an external audio signal while the mobile terminal100is in a particular mode, such as a phone call mode, a recording mode and/or a voice recognition mode. The received audio signal is processed and converted into digital data. In the call mode, the processed digital data is transformed into a format transmittable to a mobile communication base station via the mobile communication module112and then output. Furthermore, the mobile terminal100, and in particular the A/V input unit120, may include a noise removing algorithm to remove noise generated during the course of receiving the external audio signal.

The user input unit130generates input data in response to user manipulation of an associated input device or devices. Examples of such devices include a keypad, a dome switch, a touchpad (e.g., static pressure/capacitance), a jog wheel and a jog switch. A specific example of the user input unit130is a touch screen in which a touchpad is combined with a display, as will be described below.

The sensing unit140provides status measurements of various aspects of the mobile terminal100. For example, the sensing unit140may detect an open/closed status of the mobile terminal100, relative positioning of components (e.g., a display and a keypad) of the mobile terminal100, a change of position of the mobile terminal100or a component of the mobile terminal100, a presence or absence of user contact with the mobile terminal100, an orientation of the mobile terminal100and/or acceleration/deceleration of the mobile terminal100.

As an example, the mobile terminal100may be configured as a slide-type mobile terminal in which the sensing unit140may sense whether a sliding portion of the mobile terminal100is open or closed. The sensing unit140may also sense a presence or absence of power provided by the power supply unit190or a presence or absence of a coupling or other connection between the interface unit170and an external device. The sensing unit140may include a proximity sensor141.

The output unit150generates output relevant to the senses of sight, hearing and touch. The output unit150may include a display unit151, an audio output module152, an alarm153, a haptic module154, and a projector module155.

The display unit151displays information processed by the mobile terminal100. For example, when the mobile terminal100is in a call mode, the display unit151may display a user interface (UI) or a graphic user interface (GUI) associated with the call. If the mobile terminal100is in a video communication mode or a photograph mode, the display unit151may display a photographed and/or received picture, a UI or a GUI.

The display unit151may include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, or a 3-dimensional display. The mobile terminal100may include one or more of such displays.

The display unit151may have a transparent or light-transmittive type configuration, hereinafter referred to as a transparent display. A transparent OLED (TOLED) is an example of a transparent display. A rear configuration of the display unit151may also have the light-transmittive type configuration. In this configuration, a user is able to see an object located behind the terminal body via the area occupied by the display unit151of the terminal body.

At least two display units151may be provided. For example, a plurality of display units151may be provided on a single face of the mobile terminal100spaced apart from each other or built in one body. Alternatively, each of a plurality of display units151may be provided on different faces of the mobile terminal100.

If the display unit151and a sensor for detecting a touch action (hereafter referred to as a ‘touch sensor’) are constructed in a mutual-layered structure (hereafter referred to as a ‘touch screen’), the display unit151may be used as an input device and an output device. For example, the touch sensor may include a touch film, a touch sheet or a touchpad.

The touch sensor can be configured to convert a pressure applied to a specific portion of the display unit151or a variation of electrostatic capacity generated from a specific portion of the display unit151to an electric input signal. The touch sensor may detect a pressure of a touch as well as a touched position or magnitude of the touch.

If a touch input is made to the touch sensor, a signal(s) corresponding to the touch input is transferred to a touch controller (not shown). The touch controller processes the signal(s) and then transfers corresponding data to the controller180. The controller180may determine, therefore, which portion of the display unit151is touched.

With continued reference toFIG. 1, a proximity sensor141can be provided within the mobile terminal100enclosed by the touch screen or around the touch screen. The proximity sensor141may detect a presence or non-presence of an object approaching a specific detecting surface or an object existing around the proximity sensor141using an electromagnetic field strength or infrared ray without mechanical contact. Accordingly, the proximity sensor141may have greater durability and greater utility than a contact type sensor.

The proximity sensor141can include a transmittive photoelectric sensor, a direct reflective photoelectric sensor, a mirror reflective photoelectric sensor, a radio frequency oscillation proximity sensor, an electrostatic capacity proximity sensor, a magnetic proximity sensor or an infrared proximity sensor. If the touch screen is an electrostatic type touch screen, the proximity sensor141may detect proximity of a pointer using a variation of an electric field according to the proximity of the pointer and the touch screen (touch sensor) may be classified as the proximity sensor141.

An action in which a pointer approaches the touch screen without contacting the touch screen, yet is recognized as being located on the touch screen, is referred to as a ‘proximity touch’. An action in which the pointer actually touches the touch screen is referred to as a ‘contact touch’. The position on the touch screen proximity-touched by the pointer refers to the position of the pointer that vertically opposes the touch screen when the pointer performs the proximity touch.

The proximity sensor141may detect a proximity touch and/or a proximity touch pattern (e.g., proximity touch distance, proximity touch duration, proximity touch position, proximity touch shift state). Information corresponding to the detected proximity touch action and/or the detected proximity touch pattern may be displayed on the touch screen.

The audio output module152may output audio data that is received from the wireless communication unit110in, for example, a call-receiving mode, a call-placing mode, a recording mode, a voice recognition mode or a broadcast receiving mode. The audio output module152may output audio data stored in the memory160. The audio output module152may output an audio signal relevant to a function (e.g., a call signal receiving sound, a message receiving sound, etc.) performed by the mobile terminal100. The audio output module152may include, for example, a receiver, a speaker or a buzzer.

The alarm153outputs a signal for announcing an occurrence of a particular event associated with the mobile terminal100. Typical events include a call signal reception, a message reception, a key signal input and a touch input. The alarm153outputs a signal for announcing the event occurrence via vibration as well as a video signal or an audio signal. The video signal is output via the display unit151and the audio signal is output via the audio output module152. Hence, at least the display unit151or the audio output module152can be regarded as part of the alarm153.

The haptic module154may generate various haptic effects that can be sensed by a user. Vibration is a representative tactile effect generated by the haptic module154. Strength and pattern of the vibration generated from the haptic module154may be controllable. For example, vibrations differing from each other can be output by being synthesized together or can be output in sequence.

The haptic module154may generate various haptic effects in addition to vibration. For example, the haptic module154may generate an effect caused by a pin array vertically moving against skin being touched, an air injection force via an injection hole, an air suction force via a suction hole, an effect of skimming on a skin surface, an effect of contact with an electrode, an effect of electrostatic power and/or an effect of a hot/cold sense using an endothermic or exothermic device.

The haptic module154can be configured to provide the haptic effect via direct contact. The haptic module154can also be configured to enable a user to experience the haptic effect via muscular sense of a finger or an arm. Two or more haptic modules154can be provided according to a configuration of the mobile terminal100.

The projector module155is configured to perform an image projection function. More particularly, the projector module155is configured to display an image that is either identical to or at least partially different from the image displayed on the display unit151on an external screen or wall based on a control signal from the controller180.

The projector module155may include a light source (not shown) configured to generate light (e.g., a laser) for projecting an image external to the mobile terminal100, an image producing means (not shown) for producing the image for external output using the light generated from the light source, and a lens (not shown) for enlarging the image to be output externally at a predetermined focus distance. The projector module155may further include a device (not shown) for adjusting a direction of the projected image by mechanically moving the lens or the entire module.

The projector module155may be classified as, for example, a cathode ray tube (CRT) module, a liquid crystal display (LCD) module, or a digital light processing (DLP) module based on a type of a display means. In particular, the DLP module is operated by a mechanism enabling the light generated from the light source to reflect on a digital micro-mirror device (DMD) chip and advantageously allows the projector module151to be a smaller size.

The projector module155may be provided in a lengthwise direction of a lateral, front or rear side of the mobile terminal100. It is understood that the projector module155may be positioned at any portion of the mobile terminal100as necessary.

The memory160is generally used to store various types of data for supporting the processing, control, and storage requirements of the mobile terminal100. Examples of such data include program instructions for applications operating on the mobile terminal100, contact data, phonebook data, messages, audio, still pictures, and moving pictures. A recent use history or a cumulative use frequency of each data (e.g., use frequency for each phonebook, each message or each multimedia) may be stored in the memory160. Moreover, data for various patterns of vibration and/or sound to be output when a touch input is received at the touchscreen may be stored in the memory160.

An object information item database including information about objects, such as buildings, shops, or road signs, can be stored in the memory160. The object information items can be searched using at least one of pattern recognition information related to an object within a preview image photographed via the camera121and position information of the mobile terminal100obtained via the position-location module115. The object information can include at least one of relevant text information (e.g., a name of a building or shop), relevant link information (e.g., link information for the building or shop), relevant image information (e.g., an image logo for the building or shop), and relevant audio information (e.g., a logged song related to the building or shop) of the pattern-recognized object.

The object information item database stored in the memory160can be downloaded or updated from a database of the external server via the wireless communication unit110. The object information item database stored in the memory160can include an entire portion of the database downloaded from the external server or only a portion of the database downloaded in part from the external server, depending on the storage capacity of the memory160. The portion of the database of the external server is provided for an object located within a predetermined distance from a current position of the mobile terminal100or an object located in a predetermined area (e.g., administrative district) related to a current position of the mobile terminal100.

Information related to a plurality of contact entries, including a plurality of video images related to the plurality of contact entries, may be stored in the memory160. The information related to the contact entries may include at least a name, video image, phone number, email address, or fax number for each corresponding contact entry.

The memory160may include, for example, a flash memory, a hard disk, a multimedia card micro type memory, a card type memory (e.g., SD memory, XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory(PROM), a magnetic memory, a magnetic disk or an optical disk. The mobile terminal100may operate in association with a web storage that performs a storage function of the memory160via the Internet.

The interface unit170couples the mobile terminal100with external devices. The interface unit170receives data from an external device. The interface unit170is supplied with power and may be configured to deliver the power to elements within the mobile terminal100. The interface unit170may be configured to enable data to be transferred from the mobile terminal100to an external device. The interface unit170may be configured to include a wired/wireless headset port, an external charger port, a wire/wireless data port, a memory card port, a port for coupling to a device having an identity module, an audio input/output (I/O) port, a video input/output (I/O) port or an earphone port.

The identity module is a chip or card that stores various types of information for authenticating a use authority of the mobile terminal100and can include a user identity module (UIM), a subscriber identity module (SIM) and/or a universal subscriber identity module (USIM). A device provided with the above identity module (hereafter referred to as an ‘identity device’) may be manufactured in the form of a smart card. The identity device is connectable to the mobile terminal100via a corresponding port.

The interface unit170may be configured as a passage for supplying power to the mobile terminal100from a cradle that is connected to the mobile terminal100. The interface unit170may facilitate delivery of various command signals, which are input via the cradle by a user, to the mobile terminal100. Various command signals input via the cradle or the power may work as a signal for recognizing that the mobile terminal100is correctly loaded in the cradle.

The controller180typically controls the overall operations of the mobile terminal100. For example, the controller180performs control and processing associated with voice calls, data communications and video conferences. The controller180may include a multimedia module181that provides multimedia playback. The multimedia module181may be configured as part of the controller180or may be configured as a separate component. The controller180may also perform pattern recognizing processing for recognizing a handwriting input performed on the touch screen as a character and/or recognizing a picture drawing input performed on the touch screen as characters or images.

Meanwhile, the controller180can be configured to implement the augmented reality (hereinafter, “AR”) technology. The AR is a form of virtual reality that provides one image generated from combining the real world, as seen via the eyes of a user, together with a virtual world having side information. The AR compensates the real world with a virtual world and plays a role in providing information necessary for the real world graphically. The controller180displays the object information item in the form of the AR.

The power supply unit190provides power required by the various components of the mobile terminal100. The power may be internal power, external power, or combinations thereof

Embodiments of the present invention described in the following description may be implemented within a recording medium that can be read by a computer or a computer-like device using software, hardware or combinations thereof. For hardware implementations, arrangements and embodiments may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors or electrical units for performing other functions. Such embodiments may also be implemented by the controller180.

For a software implementation, arrangements and embodiments described herein may be implemented with separate software modules, such as procedures and functions, each of which may perform one or more of the functions and operations described herein. Software codes may be implemented with a software application written in any suitable programming language and may be stored in the memory160and executed by the controller180.

FIG. 2Ais a front perspective view showing an example of a front of the mobile terminal100. The mobile terminal100ofFIG. 2Ais depicted as a bar-type terminal body. However, the mobile terminal100may be implemented in a variety of different configurations. Examples of such configurations include a folder-type, a slide-type, a rotational-type, a swing-type and combinations thereof. For clarity, further disclosure will primarily relate to a bar-type mobile terminal100. However such teachings apply equally to other types of mobile terminals.

The mobile terminal100includes a case (casing, housing, cover, etc.) that forms an exterior of the mobile terminal. The case may be divided into a front case101and a rear case102. Various electric/electronic parts are provided in a space between the front case101and the rear case102. A middle case may be further provided between the front case101and the rear case102. The cases can be formed by injection molding of synthetic resin or may be formed of a metallic material, such as stainless steel (STS) or titanium (Ti).

The display unit151, audio output module152, camera121, first and second user manipulating units130aand130b,microphone122and/or the interface unit170can be provided on the terminal body, and more particularly on the front case101.

The display unit151occupies most of a main face of the front case101. The audio output module152and the camera121may be provided at an area adjacent to one end portion of the display unit151, while the first manipulating unit130aand the microphone122may be provided at an area adjacent to the other, opposite end portion of the display unit151. The second manipulating unit130band the interface unit170can be provided on lateral sides of the front and rear cases101and102.

The user input unit130(FIG. 1) may receive a command for controlling an operation of the mobile terminal100. The user input unit130may include a plurality of manipulating units, such as the first and second manipulating units130aand130b.The first and second user manipulating units130aand130bcan be named a manipulating portion and may adopt any mechanism of a tactile manner that enables a user to perform a manipulation action by experiencing a tactile feeling.

Content input via the first manipulating unit130aor the second manipulating unit130bcan be set to be different. For example, commands such as start, end and scroll can be input via the first manipulating unit130a.Commands for adjusting volume of sound output from the audio output module152and for switching the display unit151to a touch recognizing mode can be input via the second manipulating unit130b.

FIG. 2Bis a rear perspective diagram of the mobile terminal100shown inFIG. 2A. Other embodiments, configurations and arrangements may also be provided.

As shown inFIG. 2B, an additional camera121′ can be provided on a rear side of the terminal body, and more particularly, on the rear case102. The camera121′ on the rear case102has a photographing direction that is substantially opposite to that of the camera121shown inFIG. 2Aand may have a different resolution.

For example, the camera121may have a smaller number of pixels, and thereby have a relatively lower resolution, to capture and transmit an image of the user's face for a video call. On the other hand, the camera121′ may have a greater number of pixels, and thereby have a relatively greater resolution, for capturing an image of a general subject for photography without transmitting the captured image. Each of the cameras121and121′ can be configured to be rotated and/or popped up from the terminal body.

A flash123and a mirror124may be disposed adjacent to the camera121′. The flash123projects light toward a subject when photographing the subject using the camera121′. When a user attempts to take a picture of himself/herself (self-photography) using the camera121′, the mirror124enables the user to view his/her face reflected by the mirror124.

An additional audio output module152′ can be disposed at the rear side of the terminal body. The additional audio output module152′ facilitates a stereo function in conjunction with the audio output module152illustrated inFIG. 2Aand may be used for implementation of a speakerphone mode when communicating via the mobile terminal100.

A broadcast signal receiving antenna116can be provided at a lateral side of the terminal body in addition to an antenna for communication. The antenna116incorporated into the broadcast receiving module111shown inFIG. 1can be retractable within the terminal body.

The power supply unit190for supplying a power to the mobile terminal100may be provided in the terminal body. The power supply unit190can be configured to be built within the terminal body or to be detachably connected to the terminal body.

FIG. 2Balso shows a touchpad135for detecting a touch input disposed at the rear case102. The touchpad135can be a light transmittive type, similar to the display unit151. If the display unit151is configured to output visual information from both its front and rear faces, it recognizes the visual information via the touchpad135as well. The information output from the front and rear faces can be entirely controlled by the touchpad135. Alternatively, a display is provided at the touchpad135so that a touch screen can be disposed at the rear case102.

The touchpad135is activated by interconnecting with the display unit151of the front case101. The touchpad135can be disposed at the rear of, and parallel to, the display unit151. The touchpad135can be of a size equal to or smaller than that of the display unit151.

In the following description, embodiments related to a controlling method, which can be implemented in the mobile terminal100, are explained with reference to the accompanying drawings.

FIG. 3is a flowchart illustrating a process for indicating an occurrence of an event associated with the AR using various haptic effects according to the present invention. Referring toFIG. 3, the controller180(FIG. 1) activates the camera121(FIG. 1) (S110) and controls the display unit151(FIG. 1) to display a preview image of a current area captured via the camera121(S120) when a menu function for the AR is selected by a user. As described previously, the display unit151may be a touch screen.

Subsequently, the controller180(FIG. 1) obtains a current position of the mobile terminal100via the position location module115(FIG. 1) (S130). More particularly, the position location module115receives position information including latitude, longitude, altitude and direction of the mobile terminal100(FIG. 1) from at least one satellite under the control of the controller180(FIG. 1).

Afterwards, the controller180(FIG. 1) searches for object information items related to at least one object included in the preview image based on the obtained current position of the mobile terminal100(FIG. 1) (S140). The object information items include at least a name, contact address, or location detail of a corresponding object.

The controller180(FIG. 1) searches map data in the memory160(FIG. 1) for the information related to the objects included in the preview image. The controller180may also access the external server via the wireless communication unit110(FIG. 1), search an object information item database included in the external server for the object information item related to the objects included in the preview image, and download the found object information item.

The controller180(FIG. 1) recognizes shapes of the objects included in the preview image and searches the map data in the memory160(FIG. 1) for the object information item corresponding to the recognized shapes. The controller180may also search the object information item database included in the external server for the object information item corresponding to the recognized shapes via the wireless communication unit110(FIG. 1) and download the found object information item via the wireless communication unit110.

As described previously, once the object information items related to the objects included in the preview image are found, the controller180(FIG. 1) controls the display unit151(FIG. 1) to display the found object information items at respective positions of the corresponding objects included in the preview image for the AR (S150). While the object information items are displayed in the preview image for the AR, the controller180monitors whether an event related to the AR occurs (S160).

As one example, the event related to the AR can include changing the object or the object information item displayed in the preview image. As another example, the event related to the AR can include a new object or new object information item being displayed in the preview image. Additionally, the event related to the AR can include selection and dragging of a specific object or specific object information item included in the preview image. Another example of the event related to the AR is changing the preview image when a photographing view point of the camera121(FIG. 1) is changed, for example, via movement of the mobile terminal100(FIG. 1) by a user. A further event related to the AR can include changing the object information item included in the preview image based on a change of the preview image caused by changing the photographing view point of the camera121. As another example, the event related to the AR can include outputting sound related to the AR or changing the corresponding output. Another example of the event related to the AR includes receiving a call from an external terminal when the AR preview image is displayed. A further example of an event related to the AR can include receiving a message from an external terminal while the AR preview image is displayed. As another example, the event related to the AR can include outputting a preset alarm while the AR preview image is displayed. The event related to the AR can include inputting a search word related to an object information item or a specific object included in the AR preview image. As another example, the event related to the AR can include inputting a user command for requesting a moving path to a specific object or object information item in the AR preview image.

As described previously, when the specific event related to the AR occurs (S170), the controller180(FIG. 1) searches the memory160(FIG. 1) for haptic effects, which were set previously according to types of events, that match the event and generates the haptic effects by controlling the haptic module154(FIG. 1) (S180). The haptic effect for different events can differ from each other by having different magnitudes and/or different patterns. Thus, a user can sense the different magnitude and/or the different pattern of the haptic effect and recognize a currently occurring event related to the AR based on the sensed haptic effect.

In the following description, a process for generating a haptic effect based on the type of event related to the AR is described in detail with reference toFIGS. 4 to 12. A process according to an embodiment of the present invention includes recognizing that at least one second object fails to be displayed on the display unit151(FIG. 1) because it is blocked by a specific first object in a preview image or at least one second object information item fails to be displayed on the display unit151(FIG. 1) because it is blocked by a specific first object information item in the preview image. The process also includes generating a haptic effect indicating a presence of the obtained at least one second object or the obtained at least one second object information item.

An embodiment of the present invention is explained in detail with reference toFIGS. 4 to 7as follows.FIG. 4is a flowchart depicting a process for generating a haptic effect indicating an event associated with AR according to an embodiment of the present invention.FIGS. 5 to 7are diagrams of screen configurations of a process for generating a haptic effect indicating an event associated with AR according to an embodiment of the present invention.

Referring toFIGS. 4 to 7, the controller180(FIG. 1) controls the display unit151(FIG. 1) to display a preview image300A including at least one object and at least one object information item according to steps (S110) to (S150) shown inFIG. 3(S210). The object information item displayed in the preview image300A is a virtual information item and is changeable based on a target photographed via the camera121(FIG. 1).

For example, when the camera121(FIG. 1) is focused on a street or object, information related to the surrounding area on the street or the object information item related to the object can be displayed on the display unit151(FIG. 1). Alternatively, when a user desires information on a book or other object, the user can photograph the book or other object via the camera121and the controller searches for information or an object information item, such as an on-line bookstore price comparison or a review of the book, which can be displayed on the display unit151. Similarly, when a user desires information related to a music disc, the user can photograph the music disk cover to search various object information items, such as music videos for songs included on the music disk or an on-line price for the music disk. When an image of a movie poster is captured via the camera121, the controller180(FIG. 1) may control the display unit to display information such as a trailer or a movie review for the movie related to the movie poster.

When the first object or the first object information item in the preview image300A is selected as the event in steps (S160) and (S170) shown inFIG. 3(S220), the controller180determines whether at least one second object or at least one second object information item exists but is not displayed because it is blocked from view by the selected first object or the selected first object information item (S230).

When it is determined that the at least one second object or at least one second object information item exists but is blocked from view by the selected first object or the selected first object information item, such that the at least one second object or at least one second object information item fails to be displayed in the preview image300A, the controller180controls the haptic module154(FIG. 1) to generate a haptic effect that indicates the existence of the blocked at least one second object or the hidden at least one second object information item (S240). When there are at least two blocked second objects or at least two blocked second object information items, the controller180(FIG. 1) controls the haptic module154to generate a number of haptic effects equal to the number of hidden objects or hidden object information items.

When the event in steps (S160) and (S170) shown inFIG. 3is the first object or the first object information item being selected and dragged in a specific direction, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a haptic effect at the position corresponding to the dragged distance differing in magnitude based on the existence of the blocked at least one object or object information item. For example, if there are four objects or object information items blocked by the first object or first object information item, the controller180controls the haptic module154to generate four vibrations that differ in magnitude based on the drag input.

The type, magnitude, pattern or other features of the haptic effect generated to correspond to the blocked object or the blocked object information item can be set by a user. A separate menu can be provided on the display unit151(FIG. 1) to set the haptic effect type, magnitude, pattern, or other haptic effect features.

FIG. 5Ashows a preview image300A for AR in which at least one object or at least one object information item is included. Referring toFIG. 5B, the preview image300A includes an increased number of objects or object information items, such that the objects and the object information items overlap each other or block each other from view. Accordingly, a user has difficulty selecting desired information or obtaining information indicating whether other information exists.

Referring toFIGS. 6A and 6B, when an object411in preview image300A is touched and selected (FIG. 6A) and an object or object information item is blocked from view in the preview image300A by the selected object411, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a haptic effect, such as vibration (FIG. 6B). In other words, when an object or object information item is hidden behind or blocked from view by the selected object411, the hidden object or object information item is not displayed in the preview image300A. Accordingly, the vibration or other corresponding haptic effect is output by the haptic module154(FIG. 1) to indicate that another object or object information item exists behind the selected object411. Similarly, when an object information item displayed in the preview image300A is touched and selected (not shown) and an object or object information item is blocked from view in the preview image300A by the selected object information item, the controller180(FIG. 1) causes the haptic module154to generate a corresponding haptic effect.

FIG. 7Adepicts a drag input400being performed on a preview image300A with reference to an object or object information item. Referring toFIGS. 7A and 7B, when a dragging of a selected object or object information item progresses on the preview image300A in the direction indicated1→2→3→4→ and objects or object information items are hidden within the progressing drag, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a haptic effect to indicate that the objects or object information items are hidden within the progressing drag (FIG. 7B). If one of the positions1,2,3, and4is selected from the preview image300A, the controller180controls the display unit151(FIG. 1) to display the object or object information item hidden at the selected position on the preview image300A.

FIGS. 8-9Bdepict a process for generating a haptic effect indicating a distance between a terminal and a selected specific object in a preview image.FIG. 8is a flowchart illustrating a process for generating a haptic effect based on an event associated with AR.FIG. 9is a diagram of a screen configuration of a process for generating a haptic effect indicating an event associated with AR.

Referring toFIGS. 8-9B, the controller180(FIG. 1) displays a preview image300A including at least one object and at least one object information item according to steps (S110) to (S150) shown inFIG. 3(S310). When a first object in the preview image300A is selected as event in steps (S160) and (S170) shown inFIG. 3(S320), the controller180obtains a distance between the mobile terminal100(FIG. 1) and the selected first object and controls the haptic module154(FIG. 1) to generate a haptic effect indicating the obtained distance (S330).

For example, the controller180(FIG. 1) may control the haptic module154(FIG. 1) to generate a haptic effect that varies based on the obtained distance between the mobile terminal100(FIG. 1) and the selected first object. The magnitude of the haptic effect may change when the distance between the mobile terminal100and the selected first object is reduced.

The controller180(FIG. 1) may control the haptic module154(FIG. 1) to generate a vibration having a greater magnitude when the distance between the selected first object and the user's current position in the preview image300A is reduced. Conversely, the controller180may control the haptic module154to generate a vibration having a smaller magnitude when the distance between the selected first object and the user's current position in the preview image300A is increased.

Since distances to objects displayed in the preview image300A for AR may not be easily determined, the varying haptic effects help a user to determine the approximate distance when an object is selected. According to the above process, a user can be notified of the approximate distance to a user-selected object in the AR preview image300A based on the haptic effect generated from the AR preview image300A.

FIG. 9Ashows a first object411and a second object412situated at different distances in a selected preview image300A. Referring toFIG. 9B, when the first object411is selected from the preview image300A, a relatively stronger vibration signal “1” is output if the first object411is situated closer to the user's current position than the second object412. When the second object412, which corresponds to a building existing at a position farther away than the first object411, is selected, a relatively weaker vibration signal “2” is output. Therefore, by generating haptic effects of differing magnitudes or patterns based on the distance between a selected object in the preview image300A and the user's current position, the corresponding tactility notifies a user of the distance to the selected object.

FIGS. 10A-13Cillustrate a process for generating a haptic effect indicating an AR event related to a change in objects or object information items included in a displayed preview image300A.FIGS. 10A to 13Care diagrams of screen configurations of a process for generating a haptic effect indicating an event associated with the AR.

Referring toFIG. 10A, a first object information item411, a second object information item412and a third object information item413are found and displayed in an AR preview image300A. When a first event occurs in which a new fourth object information item421is found and displayed in the preview image300A (FIG. 10B), the controller180(FIG. 1) searches the memory160(FIG. 1) for haptic effects set previously by type of event for the haptic effect corresponding to the occurring first event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 10C).

Referring toFIGS. 11A and 11B, when there is a plurality of the new object information items, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a number of different haptic effects in order equal to the number of the new object information items. In particular, referring toFIG. 11A, when a plurality of new object information items421to423is found and displayed in the preview image300A, the controller180(FIG. 1) determines the number of the new object information items421to423. Referring toFIG. 11B, the controller180controls the haptic module154(FIG. 1) to generate a number of different haptic effects in order equal to the number of the new object information items.FIG. 11Bdepicts the different haptic effects that correspond to the new object information items421to423.

Referring toFIGS. 12A and 12B, when there is a plurality of the new object information items, the controller180(FIG. 1) determines distances between each of the new object information items and controls the haptic module154(FIG. 1) to generate the haptic effects, in order, based on the determined distances. In particular, referring toFIG. 12B, when new fourth to sixth object information items521to523are found and displayed in the preview image300A, the controller180(FIG. 1) determines the respective distances between the fourth to sixth object information items521to523. Referring toFIG. 12C, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate the haptic effects corresponding to the respective determined distances.

FIG. 12Bshows that the determined distances are proportional to the generation cycles of the respective haptic effects. In particular, the controller180(FIG. 1) controls generation of the haptic effects in the following manner. The generation cycle of the haptic effect corresponding to each of the fourth and fifth object information items521and522displayed proximate to each other is set to be shorter than the haptic effect corresponding to the sixth object information item523, which is displayed as not being proximate to the fourth or fifth object information items521or522.

Referring toFIG. 13A, a first object310and a second object320exist in the AR preview image300A. A first object information item411on the first object310and a second object information item412on the second object320are found and displayed in the AR preview image300A. When a second event occurs, in which at least one new object330is recognized and displayed in the preview image300A (FIG. 13B), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the second event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 13C).

Meanwhile, if there is a plurality of the new objects, as described previously in relation toFIGS. 11A and 11B, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a number of different haptic effects in order equal to the number of the new objects (not shown). Moreover, when there is a plurality of the new object information items, as described previously in relation toFIGS. 12A and 12B, the controller180(FIG. 1) determines distances between each of the new object information items and controls the haptic module154(FIG. 1) to generate the haptic effects in order based on the obtained proximities (not shown).

FIGS. 14A and 14Bdepict a process for generating a haptic effect indicating an AR event related to selecting a specific object information item from a preview image.

As previously described inFIG. 13A, a first object310and a second object320exist in an AR preview image300A. A first object information item411is found and displayed on the first object310and a second object information item412is found and displayed on the second object320. If a third event occurs, in which a specific object information item411is selected from the preview image300A (FIG. 14A), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the third event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 14B).

When the haptic effect corresponding to the third event is a vibration, the controller180(FIG. 1) controls the display unit151(FIG. 1) to display a shape of the selected object information item411as being shaken by the vibration (not shown) while the haptic effect corresponding to the third event is generated. The controller180controls the haptic module154(FIG. 1) to generate the haptic effect indicating that the specific object has been selected upon selection of the specific object from the preview image300A.

FIGS. 15A-15Ddepict a process for generating a haptic effect indicating an AR event related to a change of a preview image due to a changed photographing view point of the camera121.

As previously described inFIG. 13A, a first object310and a second object320exist in an AR preview image300A. A first object information item411is found and displayed on the first object310and a second object information item412is found and displayed on the second object320. If a fourth event occurs, in which the first preview image300A is changed into a second preview image300B due to a change in a photographing view point of the camera121(FIG. 1) (FIG. 15A), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the fourth event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 15B).

If a fifth event occurs, in which a new object340and a new object information item413on the new object340are found and displayed in the second preview image300B (FIG. 15C), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the fifth event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 15D).

FIGS. 16A-16Bdepict a process for generating a haptic effect when a sound is output indicating an occurrence of an AR event. Referring toFIG. 16A, a first object310and a second object320exist in an AR preview image300A. A first object information item411is found and displayed on the first object310and a second object information item412is found and displayed on the second object320. A first sound510arelated to the AR including the first and second object information items411and412is output to indicate that two object information items are found.

If a sixth event occurs, in which the first sound510arelated to the AR is output via the audio output module152(FIG. 1), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event and controls the haptic module154(FIG. 1) to generate the haptic effect corresponding to the sixth event. More particularly, the controller180controls the haptic module154to generate the haptic effect corresponding to the sixth event while the first sound510ais being output.

Referring toFIG. 16B, if a seventh event occurs, in which the first sound510ais changed into a second sound510bwhile the haptic effect corresponding to the sixth event is generated, the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event and controls the haptic module154(FIG. 1) to generate the haptic effect corresponding to the seventh event.

FIGS. 17-19depict a process for generating a haptic effect when an external event, such as a message reception or an alarm output, occurs while a preview image for the AR is displayed.

As previously disclosed inFIG. 13A, a first object310and a second object320exist in an AR preview image300A. A first object information item411is found and displayed on the first object310and a second object information item412is found and displayed on the second object320.

If an eighth event occurs, in which a call520(FIG. 17A) is received from another terminal via the wireless communication unit110(FIG. 1), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the eighth event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 17B).

If a ninth event occurs, in which a message530(FIG. 18A) is received from another terminal via the wireless communication unit110(FIG. 1), the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the ninth event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 18C).

If a tenth event occurs, in which an alarm540(FIG. 19B) previously registered with the memory160(FIG. 1) is output, the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) by type of event for the haptic effect corresponding to the tenth event and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 19C).

FIGS. 20-23depict a process for generating a haptic effect indicating an AR related to input of a command to search for a specific object and/or a specific object information item.FIGS. 20-23are diagrams of screen configurations of a process for generating a haptic effect indicating an event associated with the AR.

Referring toFIG. 20A, a search window550for receiving input to search for a user-specific object and/or a user-specific object information item is displayed in the first preview image300A for the AR. A specific search word550ais input into the search window550.

If an eleventh event occurs, in which a command to search for a specific object and/or a specific object information item is input, as indicated by the specific search word550a, the controller180(FIG. 1) searches for an object310and/or object information item411corresponding to the search command using map data (FIG. 20B). If the object310and/or object information item411corresponding to the search command is found to exist in the first preview image300A, the controller180(FIG. 1) searches the haptic effects set previously in the memory160(FIG. 1) for the haptic effect indicating that the object310and/or object information item411corresponding to the search command exists in the first preview image300A and controls the haptic module154(FIG. 1) to generate the corresponding haptic effect (FIG. 20C). Moreover, while the corresponding haptic effect is being generated, the controller180(FIG. 1) may control the display unit151to display a shape of the found object information item411as being shaken by the vibration (not shown) while the haptic effect corresponding to the eleventh event is generated.

Meanwhile, referring toFIG. 21A, as a distance between the object310or object information item411found inFIG. 20Band the mobile terminal100(FIG. 1) decreases, the controller180(FIG. 1) changes the magnitude of the haptic effect. More particularly, as the distance between the found object310or object information item411and the mobile terminal100decreases (FIG. 21A), the controller180controls the haptic module154(FIG. 1) to increase the magnitude of the haptic effect (FIG. 21B) to notify a user that the user is getting closer to the object310or object information item411.

Referring toFIGS. 22A-22D, when an object and/or object information item for which a user inputs a search exists but is not situated in a current preview image300A, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate a haptic effect indicating a direction in which the corresponding object and/or object information item is situated.

FIG. 22Adepicts a plurality of haptic modules154a,154b,154cand154dincluded in a body of the mobile terminal100(FIG. 1). As depicted inFIG. 22A, a first haptic module154ais positioned at an upper/left portion of the body of the mobile terminal100and a second haptic module154bis positioned at an upper/right portion of the body of the mobile terminal100. The first and second haptic modules154aand154bare configured to generate haptic effects10R (FIG. 22B) having left-to-right directionality and10L (not shown) having right-to-left directionality under the control of the controller180(FIG. 1).

The controller180(FIG. 1) controls the first and second haptic modules154aand154bto generate the haptic effect having the left-to-right or right-to-left directionality by adjusting the magnitude of the haptic effect generated from each of the first and second haptic modules154aand154b.For example, the controller180controls the first and second haptic modules154aand154bto generate the haptic effect having the left-to-right directionality by controlling the first haptic module154ato generate a haptic effect having a relatively weak magnitude and by controlling the second haptic module154bto generate a haptic effect having a relatively strong magnitude.

Referring again toFIG. 22A, a third haptic module154cis positioned at a lower/left portion of the body of the mobile terminal100(FIG. 1) and a fourth haptic module154dis positioned at a lower/right portion of the body of the mobile terminal100. Accordingly, the first and third haptic modules154aand154care configured to generate, for example, a haptic effect having a top-to-bottom directionality along a left side of the body of the mobile terminal100and the second and fourth haptic modules154band154dare configured to generate, for example, a haptic effect having a top-to-bottom directionality along a right side of the body of the mobile terminal100.

If a twelfth event occurs, in which a command560ato search for an object information item440related to a specific object350is input, the controller180(FIG. 1) checks whether the object350and/or the object information item440exists in a current preview image300A. If the object350and/or the object information item440is found to exist in the current preview image300A, the controller180performs the process shown inFIGS. 20BandFIG. 20C. If, however, the object350and/or the object information item440is found to exist but is not situated in the current preview image300A, the controller180determines a position of the object350and/or the object information440and controls a plurality of haptic modules of the first to fourth haptic modules154a,154b,154cand154dto generate a haptic effect, such as the haptic effect10R, indicating the direction in which the object350and/or the object information item440is situated in relation to the preview image300A.

Referring toFIG. 22B, as an example, an actual position of the object350and/or the object information item440is depicted to the right side of the mobile terminal100. Accordingly, the controller180(FIG. 1) controls the haptic module154(FIG. 1) to generate the haptic effect10R indicating a right direction to instruct a user to lead a photographing view point of the camera121(FIG. 1) to the right. As the photographing view point of the camera121changes to the right (FIG. 22C), when a second preview image300B that includes the object350is displayed, the controller searches for the object information item440on the object350and displays the found object information item440(FIG. 22D).

Moreover, referring toFIGS. 23AtoFIG. 23C, when the actual position of the object350and/or the object information item440the user wants to search for inFIG. 22does not exist in the first preview image300A, the controller180(FIG. 1) may control the display unit151to display an information item570for leading the user to direct the view point of the camera121(FIG. 1) to the actual position of the object350and/or the object information item440.

FIGS. 24A-24Bdepict a process for generating a haptic effect indicating an AR event related to input of a command for requesting a moving path guide to a specific object and/or object information item. More particularly,FIGS. 24A-24Bare diagrams of a screen configuration of a process for generating a haptic effect indicating an event associated with the AR.

Referring toFIG. 24A, while a preview image300A for the AR is displayed, a search window580for searching for a moving path to an object and/or object information item desired by a user is displayed. A specific search word580ais input into the search window580.

If a thirteenth event occurs, in which a command for requesting a moving path to a specific object and/or object information item is input, as indicated by the specific search word580a,the controller180(FIG. 1) searches for the moving path to the object and/or object information item from a current position of the mobile terminal100using map data stored in the memory160(FIG. 1) and controls the display unit151to display the corresponding moving path (FIG. 24B). The controller180controls the haptic module154(FIG. 1) to generate a haptic effect to inform the user of the moving path to a final destination, such as the object, from the current position of the mobile terminal100.

Referring toFIG. 24B, the controller180(FIG. 1) may control the first to fourth haptic modules154a,154b,154cand154dto generate a haptic effect indicating the direction of a first intermediate point590ain relation to the current position of the mobile terminal100. When the mobile terminal100arrives at the first intermediate point590a,the controller180controls the first to fourth haptic modules154a,154b,154cand154dto generate a haptic effect indicating a direction to a second intermediate point590bfrom the first intermediate point590a. When the mobile terminal100arrives at the second intermediate point590b,the controller180controls the first to fourth haptic modules154a,154b,154cand154dto generate a haptic effect indicating a direction to a final destination from the second intermediate point590b.

If the mobile terminal100deviates from the moving path including the first and second intermediate points590aand590b,the controller180(FIG. 1) controls at least one of the haptic modules of the first to fourth haptic modules154a,154b,154cand154dto generate a haptic effect indicating that the mobile terminal100deviated from the moving path. Accordingly, a user can be notified of the deviation from the current path.

The present invention notifies a user of various events associated with the AR using various haptic effects. It will be apparent to those skilled in the art that the present invention can be specified into other form without departing from the spirit or scope of the inventions.

In addition, the above-described methods can be implemented in a program recorded medium as computer-readable codes. The computer-readable media includes all types of recording devices in which data readable by a computer system are stored. The computer-readable media includes ROM, RAM, CD-ROM, magnetic tapes, floppy discs, and optical data, as an example, and also includes carrier-wave type implementations, such as transmission via the Internet. The computer can include the control unit180(FIG. 1) of the terminal.