Display apparatus and method for controlling display apparatus thereof

A display apparatus and a controlling method are provided, in which the controlling method of the display apparatus receives movement information of a mobile terminal from the mobile terminal, obtains mobile terminal images by photographing the mobile terminal, transforms the received movement information into a screen coordinate system arrayed based on screen of the display apparatus, compensates the movement information transformed into the screen coordinate system based on the mobile terminal images, and controls the display apparatus by using the compensated movement information.

BACKGROUND

Devices and methods consistent with exemplary embodiments relate to displaying, and more specifically, to a display apparatus configured to control various functions of the display apparatus by using information regarding movements of external mobile terminals, and a controlling method thereof.

2. Description of the Related Art

Recently, in order to control display apparatuses more intuitively, new methods to control functions of display apparatuses by using user motions are being developed. For example, a mobile terminal may sense movement information according to user motions, and transmit the sensed movement information to external display apparatuses. Further, display apparatuses control various functions of the display apparatuses (e.g., moving pointing) by using the transmitted movement information. For another example, display apparatuses control various functions of the display apparatuses based on the photographed movement information after photographing movements of a mobile terminal.

However, as explained above, in a related method for transmitting movement information using a mobile terminal, relative motions of the mobile terminal may be sensed only, and there may be a problem in which it is difficult to obtain further information from the mobile terminal to ensure better accuracy with respect to the movement of the mobile terminal. Further, with respect to a method for photographing a mobile terminal by display apparatuses, there may be a problem in which it is difficult to obtain further accurate sensing of three-dimensional motions of a mobile terminal.

SUMMARY

Exemplary embodiments of the present inventive concept overcome the above disadvantages and other disadvantages not described above. Also, the present inventive concept is not required to overcome the disadvantages described above, and an exemplary embodiment of the present inventive concept may not overcome any of the problems described above.

According to an exemplary embodiment, a technical objective is to provide a display apparatus which compensates the received movement information from a mobile terminal by using the photographed mobile terminal images, so that the display apparatus is controlled more correctly and intuitively by using the mobile terminal, and a controlling method thereof.

According to an exemplary embodiment, a controlling method of a display apparatus is provided, which may include receiving movement information of a mobile terminal from the mobile terminal and obtaining mobile terminal images by photographing the mobile terminal, transforming the received movement information into a screen coordinate system arrayed based on a screen of the display apparatus, compensating the movement information transformed into the screen coordinate system based on the mobile terminal images, and controlling the display apparatus by using the compensated movement information.

The received movement information from the mobile terminal may be a value measured based on a mobile terminal coordinate system arrayed based on the mobile terminal, and the transforming may include transforming the received movement information measured based on the mobile terminal coordinate system into a global coordinate system arrayed based on geomagnetic direction; and transforming the movement information transformed into the global coordinate system into the screen coordinate system.

The transforming may additionally include, when a camera coordinate system arrayed based on a camera photographing the mobile terminal is different from the screen coordinate system, transforming the movement information transformed into the screen coordinate system into the camera coordinate system.

When inputs are received from a user while the mobile terminal is arrayed toward a screen direction of the display apparatus, transforming parameters of the screen coordinate system may be calculated based on a shape of the mobile terminal.

The compensating may include compensating the movement information by comparing a size of the mobile terminal obtained from the mobile terminal images with a previously obtained actual size of the mobile terminal.

The compensating may include comparing the movement information of the mobile terminal obtained from the mobile terminal images with the movement information transformed into the screen coordinate system, and when the movement information of the mobile terminal obtained from the mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system, synchronizing the movement information of the mobile terminal obtained from the mobile terminal images with the movement information transformed into the screen coordinate system.

The controlling may include recognizing three-dimensional movements of the mobile terminal by using the compensated movement information, and controlling the display apparatus based on the recognized three-dimensional movements.

The controlling may include when there is a plurality of mobile terminals operating, sensing movement patterns respectively corresponding to the plurality of the mobile terminals by using the compensated movement information, respectively, with respect to the plurality of the mobile terminals; and recognizing the plurality of the mobile terminals by using the movement patterns respectively corresponding to the plurality of the mobile terminals.

The controlling may include controlling the display apparatus based on the compensated movement information only when the mobile terminal is photographed.

The controlling may include calculating pointing coordinate values based on the compensated movement information, and displaying a pointer on the pointing coordinate values.

In one exemplary embodiment, a display apparatus is provided, which may include a communicator configured to receive movement information of a mobile terminal from the mobile terminal, an image acquirer configured to obtain mobile terminal images by photographing the mobile terminal, and a controller configured to transform the received movement information into a screen coordinate system arrayed based on a screen of the display apparatus, compensate the movement information transformed into the screen coordinate system based on the mobile terminal images, and control the display apparatus by using the compensated movement information.

The received movement information from the mobile terminal may be a value measured based on a mobile terminal coordinate system arrayed based on the mobile terminal, and the controller may transform the movement information measured based on the mobile terminal coordinate system into a global coordinate system arrayed based on geomagnetic direction, and transform the movement information transformed into the global coordinate system into the screen coordinate system.

When a camera coordinate system arrayed based on a camera photographing the mobile terminal is not in agreement with the screen coordinate system, the controller may transform the movement information transformed into the screen coordinate system into the camera coordinate system.

When an input is received from a user while the mobile terminal is arrayed toward a screen direction of the display apparatus, transforming parameters of the screen coordinate system may be calculated based on a shape of the mobile terminal.

The controller may compensate the movement information by comparing a size of the mobile terminal obtained from the mobile terminal images with a previously obtained actual size of the mobile terminal.

The controller may compare the movement information of the mobile terminal obtained from the mobile terminal images with the movement information transformed into the screen coordinate system, and synchronize the movement information of the mobile terminal obtained from the mobile terminal images with the movement information transformed into the screen coordinate system, when the movement information of the mobile terminal obtained from the mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system.

The controller may recognize three-dimensional movements of the mobile terminal by using the compensated movement information, and control the display apparatus based on the recognized three-dimensional movements.

When there is a plurality of mobile terminals operating, the controller may senses movement patterns respectively corresponding to the plurality of the mobile terminals by using the compensated movement information, respectively, related to the plurality of the mobile terminals, and recognizes the plurality of the mobile terminals by using the movement patterns respectively corresponding to the plurality of the mobile terminals.

The controller may control the display apparatus based on the compensated movement information only when the mobile terminal is photographed.

The display apparatus may additionally include a display, and the controller may control the display to calculate pointing coordinate values based on the compensated movement information, and display a pointer based on the pointing coordinate values.

According to another exemplary embodiment, there is provided a controlling method of a display apparatus which may include: receiving movement information of a mobile terminal and obtaining mobile terminal images by photographing the mobile terminal; transforming the received movement information into a first system; compensating the movement information transformed into the first system based on the mobile terminal images; and controlling the display apparatus by using the compensated movement information.

According to yet another exemplary embodiment, there is provided a display apparatus, which may include: a communicator configured to receive movement information of a mobile terminal; an image acquirer configured to obtain mobile terminal images by photographing the mobile terminal; and a controller configured to transform the received movement information into a first system, compensate the movement information transformed into the first system based on the mobile terminal images, and control the display apparatus by using the compensated movement information.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Certain exemplary embodiments of the present inventive concept will now be described in greater detail with reference to the accompanying drawings.

Terms such as “first” and “second” may be used to explain various units; however, the units cannot be limited with the terms. Terms are used only for the purpose of distinguishing one unit from the other.

Terms of this specification are used for the intention of explaining specific exemplary embodiments, and do not limit the scope of the invention. Singular expression includes plural expression except that both expressions indicate different meanings. Terms such as “include” or “comprise” in this specification indicate the presenting of features, numbers, processes, operations, units, parts or combinations of the foregoing described in the specification, and do not exclude the presenting of one or more than one other different features, numbers, processes, operations, units, parts or combinations of the foregoing or adding possibilities.

According to an exemplary embodiment, “module” or “unit” performs at least one function or operation; it may be implemented to be hardware, software, or combinations of hardware and software. Further, a plurality of “modules” or a plurality of “units” may be implemented to be at least one processor (not illustrated) by being united into at least one module except for “modules” or “units” which should be implemented to be specific hardware.

Referring to the attached drawings, the invention will be described in detail below.

FIG. 1illustrates a display system10according to an exemplary embodiment. Referring toFIG. 1, the display system10includes a display apparatus100and a mobile terminal200controlling the display apparatus. Meanwhile, the display apparatus100according to an exemplary embodiment may be smart TV; however, this is merely one of the exemplary embodiments. The display apparatus100may be various display apparatuses such as a digital TV, desktop PC, and notebook PC. Further, the mobile terminal200may be a smart phone; however, this is merely one of the exemplary embodiments. The mobile terminal200may be various mobile terminals such as pointing device, a remote controller, and a tablet PC.

First, the mobile terminal200senses movement information according to user motions. At this process, the mobile terminal200may sense movement information by using at least one of an acceleration sensor, a geomagnetic sensor and a gyro sensor. Further, the mobile terminal200transmits the sensed movement information to the display apparatus100.

The display apparatus100receives movement information of the mobile terminal200from the mobile terminal200. At this process, the display apparatus100photographs the mobile terminal200and obtains mobile terminal images while receiving movement information from the mobile terminal200.

Further, the display apparatus100transforms the received movement information into a screen coordinate system arrayed based on the screen of the display apparatus. Specifically, the display apparatus100may transform the movement information at a mobile terminal coordinate system which is arrayed based on the mobile terminal200into a global coordinate system which is arrayed based on a geomagnetic direction. Further, the display apparatus100may transform the movement information transformed into the global coordinate system into the screen coordinate system. Further, when the screen coordinate system is not in agreement with the camera coordinate system arrayed based on a camera, the display apparatus100may transform the movement information which has been compensated, or adjusted, at the screen coordinate system, into the camera coordinate system. Meanwhile, transforming parameters of the above screen coordinate system may be calculated based on a shape of the photographed mobile terminal200when initial input is received from a user while the mobile terminal200is arrayed toward a screen direction of the display apparatus.

Further, the display apparatus100may compensate the movement information transformed into the screen coordinate system based on mobile terminal images. Herein, the display apparatus100may compensate size and timing of the mobile terminal200based on mobile terminal images. Specifically, the display apparatus100may compensate the movement information by comparing size of the mobile terminal200obtained from mobile terminal images with previously obtained actual size of the mobile terminal200. Further, when the movement information of the mobile terminal obtained from mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system by comparing the movement information of the mobile terminal200obtained from images of the mobile terminal200with the movement information transformed into the screen coordinate system, the display apparatus100may synchronize the movement information of the mobile terminal obtained from mobile terminal images with the movement information transformed into the screen coordinate system.

Further, the display apparatus100may control various functions of the display apparatus100based on the compensated movement information. Specifically, the display apparatus100may recognize three-dimensional movements of the mobile terminal200by using the compensated movement information, and control the display apparatus based on the recognized three-dimensional movements. Further, when a plurality of mobile terminals operate, the display apparatus100may sense movement patterns respectively corresponding to a plurality of the mobile terminals by using the compensated movement information of each of the plurality of mobile terminals, respectively, and recognize a plurality of the mobile terminals by using the corresponding patterns respectively of the plurality of the mobile terminals. Further, the display apparatus100may display a pointer on the pointing coordinate values by calculating pointing coordinate values based on the compensated movement information. Other various exemplary embodiments will be specifically described by referring to drawings in a later part of the specification.

FIG. 2is a brief block diagram of the display apparatus100according to an exemplary embodiment. Referring toFIG. 2, the display apparatus100includes a communicator110, an image acquirer120, and a controller130.

The communicator110performs communication with the external mobile terminal200. At this process, the communicator110may receive movement information of the mobile terminal200according to user motions from the mobile terminal200. Meanwhile, the communicator110according to an exemplary embodiment may perform communication with the mobile terminal200by using Bluetooth method; however, this is merely one of embodiments, the communicator110may perform communication with the mobile terminal200by using other communication methods such as infrared communication method, WiFi communication method, and Zigbee communication method.

The image acquirer120photographs images by using the camera, and obtains images by processing the photographed images. At this process, the image acquirer120may obtain mobile terminal images including the mobile terminal200.

The controller130controls general operation of the display apparatus100. Specifically, the controller130transforms the received movement information from the external mobile terminal200into the screen coordinate system arrayed based on the screen of the display apparatus100, compensates the movement information transformed into the screen coordinate system, and controls the display apparatus100by using the compensated movement information.

Specifically, the controller130controls the communicator110to receive movement information from the mobile terminal200. At this process, the movement information received from the mobile terminal200is value measured based on the mobile terminal coordinate system which is arrayed based on the mobile terminal200.

Further, the controller130may transform the movement information measured based on the mobile terminal coordinate system into the global coordinate system which is arrayed based on geomagnetic direction, and transform the movement information transformed into the global coordinate system into the screen coordinate system again. At this process, the controller130may transform the movement information into a different coordinate system through the rotating transformation matrix. Specifically, when initial inputting is inputted from a user while the mobile terminal200is arrayed toward screen direction of the display apparatus100, the controller130may calculate transforming parameters based on the shape of the mobile terminal200, and store the calculated transforming parameters. Further, the controller130may transform the movement information at the mobile terminal coordinate system into the movement information at the screen coordinate system by using the stored transforming parameters.

Meanwhile, when the camera coordinate system arrayed based on the camera photographing the mobile terminal200is different from the screen coordinate system, the controller130may transform the movement information transformed into the screen coordinate system into the camera coordinate system again.

Further, the controller130may compensate the movement information by using the actual size of the mobile terminal and moving time points. Specifically, the controller130may compensate the movement information by comparing the size of the mobile terminal200obtained from mobile terminal images with the previously obtained actual size of the mobile terminal200. Further, the controller130may compare the movement information of the mobile terminal200obtained from mobile terminal images with the movement information transformed into the screen coordinate system. When the movement information of the mobile terminal200obtained from mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system, the controller130may synchronize the movement information of the mobile terminal obtained from mobile terminal images with the movement information transformed into the screen coordinate system.

Further, the controller130may control various functions of the display apparatus100by using the compensated movement information.

Specifically, the controller130may recognize three-dimensional movements of the mobile terminal200by using the compensated movement information, and control the display apparatus based on the recognized three-dimensional movements. Further, when a plurality of mobile terminals operate, the controller130may sense movement patterns respectively corresponding to a plurality of the mobile terminals by using the compensated movement information of each of the plurality of mobile terminals, respectively, and recognize a plurality of the mobile terminals by using the movement patterns respectively corresponding to a plurality of the mobile terminals. Further, the controller130may control the display apparatus100based on the compensated movement information only when the mobile terminal200is photographed. Further, the controller130may calculate pointing coordinate values based on the compensated movement information, and display a pointer on the calculated pointing coordinate values.

With the display apparatus100described above, a user may control the display apparatus more correctly and intuitively by using the mobile terminal.

The following will explain various exemplary embodiments of the invention by referring toFIGS. 3 to 13.FIG. 3is a block diagram of the display apparatus100in detail according to an exemplary embodiment. Referring toFIG. 3, the display apparatus100includes the communicator110, an image receiver140, an image processor150, a display160, a storage170, the image acquirer120, an audio output180, an input190, and a controller130.

The communicator110is a unit which performs communication with various types of external devices according to various types of communication methods. The communicator110may include various communication chips such as an infrared communication chip, WiFi chip, Bluetooth chip, NFC chip, and wireless communication chip. Herein, a WiFi chip, Bluetooth chip, and NFC chip perform communication respectively according to a WiFi method, Bluetooth method, and NFC method. NFC chip indicates a chip which operates according to NFC method using 13.56 MHz bandwidth among various RF-ID frequency bandwidths such as 135 kHz, 13.56 MHz, 433 MHz, 860˜960 MHz, and 2.45 GHz. When WiFi chip or Bluetooth chip is used, various pieces of connecting information such as SSID and session key may be first trans-received, communication connects by using the above information, and various pieces of information may be trans-received. The wireless communication chip indicates a chip which performs communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE).

Specifically, the communicator110may receive movement information of the mobile terminal200from the mobile terminal200according to user motions. Herein, the receive movement information from the mobile terminal200may be value measured based on the mobile terminal coordinate system which is arrayed based on the mobile terminal200.

The image receiver140receives image contents from various sources. Specifically, the image receiver140may receive broadcasting contents from external broadcasting stations, and video contents from external devices (e.g., DVD player and set-top box).

The image processor150is unit which performs the processing regarding the received video data from the image receiver140. The image processor150may perform various image processing such as decoding, scaling, noise filtering, frame rate converting, and resolution converting regarding video data.

The display160displays at least one frame among the video frames obtained by processing the received video data from the image receiver140with the image processor150and various screens generated with a graphic processor133. Specifically, the display160may display a pointer based on the movement information of the mobile terminal200.

The storage170stores various modules to drive the display apparatus100. For example, the storage170may store software including a base module, sensing module, communication module, presentation module, web browser module, and service module. Herein, the base module is a basic module which processes signals delivered from each of hardware included in the display apparatus100and delivers the processed signals to upper layer modules. The sensing module is a module which collects information from various sensors, analyzes and manages the collected information. It may include a face recognizing module, voice recognizing module, motion recognizing module and Near Field Communication (NFC) recognizing module. The presentation module is a module which constitutes display screens. It may include a multimedia module to reproduce and output multimedia contents and UI rendering module to perform the processing of a UI and graphics. The communication module is a module which performs communication externally. The web browser module indicates a module which accesses a web server by performing the web browsing. The service module is a module which includes various applications to provide various services.

Further, the storage170may store transforming parameters to transform the movement information measured based on the mobile terminal coordinate system into the screen coordinate system.

The image acquirer120photographs images and obtains images by processing the photographed images. Specifically, the image acquirer120may photograph areas including the mobile terminal200by using the camera, process the photographed images, and obtain mobile terminal images.

The audio output180is a unit which outputs various alarm sounds or voice messages as well as various audio data processed in an audio processor (not illustrated).

The input190receives various user commands to control the display apparatus100. Herein, the input190may be implemented to be various inputting devices such as a remote controller, touch screen, mouse, voice input, and motion input.

The controller130includes RAM131, ROM132, the graphic processor133, main CPU134, first to n interfaces135-1˜135-n, and a bus136, as illustrated inFIG. 3. Herein, RAM131, ROM132, the graphic processor133, the main CPU134, and the first to n interfaces135-1˜135-nmay be connected with each other through the bus136.

ROM132stores a set of commands for booting the system. When a turn-on command is inputted and electrical power is supplied, the main CPU134copies the stored O/S in the storage170to RAM131according to the stored commands in ROM132, and boots the system by implementing O/S. When booting completes, the main CPU134copies various application programs stored in the storage170to RAM131, and performs various operations by implementing the copied application programs in RAM131.

The graphic processor133generates screens including various objects such as a pointer, an icon, an image, and text by using a calculator (not illustrated) and a renderer (not illustrated). The calculator (not illustrated) calculates feature values such as coordinate value, shape, size, and color in which each object will be marked according to layouts of screens by using the received controlling commands from the input. The renderer generates various layouts of screens including objects based on the calculated feature values in the calculator. The screens generated in the renderer are displayed within the display area of the display160.

The main CPU134accesses the storage170and performs the booting by using the O/S stored in the storage170. Further, the main CPU134performs various operations by using a plurality of programs, contents and data which are stored in the storage170.

The first to n interfaces135-1˜135-nconnect to the above described units. One of the interfaces may be a network interface connected with external devices through a network.

Meanwhile, before explaining the operation of the controller130, the coordinate systems described in this specification will be specifically described by referring toFIG. 3. The mobile terminal coordinate system410is a coordinate system arrayed based on the mobile terminal, the global coordinate system420is a coordinate system arrayed based on geomagnetic direction, the screen coordinate system430is coordinate system arrayed based on the screen of the display apparatus100, and the camera coordinate system440is coordinate system arrayed based on the camera included in the display apparatus100. Virtual coordinate system450is a coordinate system which is separated by a certain distance from a focal point of the camera and where the image is captured, the display screen coordinate system460is a coordinate system whose size is adjusted to express mobile terminal images through the display apparatus100, and normalized screen coordinate system470is a coordinate system whose size is adjusted through mobile terminal images.

Specifically, the controller130may control the communicator110to receive movement information from the mobile terminal200. Herein, the received movement information from the mobile terminal200may be arrayed values (xd, yd, zd) at the mobile terminal coordinate system.

Further, the controller130may transform the movement information at the mobile terminal coordinate system into the screen coordinate system. Specifically, the controller130may transform the movement information at the mobile terminal coordinate system into the global coordinate system. For example, the mobile terminal coordinate system and the global coordinate system may be different to each other as illustrated inFIG. 5A.

Herein, the controller130may express the movement information with Euler angles, and transform the movement information at the mobile terminal coordinate system into values (xE,yE,zE) at the global coordinate system by using a following rotating transformation matrix formula.

Further, the controller130transforms the movement information transformed into the global coordinate system into values (xS, yS, zS) at the screen coordinate system illustrated inFIG. 5B. Herein, the controller130may transform the movement information transformed into the global coordinate system into the movement information based on the screen coordinate system by using previously stored transforming parameters at the screen coordinate system. Transforming parameters at the screen coordinate system may be calculated based on shape of the photographed mobile terminal200when initial input is received from a user while the mobile terminal200is arrayed toward a screen direction of the display apparatus100.

Meanwhile, although the above exemplary embodiment assumes and describes that the screen coordinate system and the camera coordinate system are in agreement with each other, this is merely one of the exemplary embodiments. When the screen coordinate system is not in agreement with the camera coordinate system, or when the camera coordinate system becomes different from the screen coordinate system because the camera rotates as illustrated inFIG. 5C, the controller130may finally transform the movement information at the screen coordinate system into the camera coordinate system.

Meanwhile, the above exemplary embodiment describes that the movement information at the mobile terminal coordinate system is transformed into the global coordinate system, and the global coordinate system is transformed into the screen coordinate system. However, this is merely one of the exemplary embodiments. The movement information at the mobile terminal coordinate system may be directly transformed into the screen coordinate system (or the camera coordinate system).

Further, the controller130may compensate the movement information transformed to the screen coordinate system based on mobile terminal images.

Specifically, the controller130may compensate the movement information by comparing size of the mobile terminal extracted from the mobile terminal images obtained by the image acquirer120with the actual size of the mobile terminal which is previously obtained. Specifically, the controller130may calculate normalizing parameters by comparing size of the mobile terminal obtained through mobile terminal images with the actual size of the mobile terminal which is previously obtained. Further, the controller130may normalize the movement information received from the mobile terminal200by using normalizing parameters.

Further, the controller130may compare the movement information of the mobile terminal200obtained from mobile terminal images with the movement information transformed into the screen coordinate system. When the movement information of the mobile terminal200obtained from mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system, the controller130may synchronize the movement information of the mobile terminal obtained from mobile terminal images with the movement information transformed into the screen coordinate system. Specifically, the mobile terminal200may generate information regarding time stamps with the movement information, and transmit to the display apparatus100. Further, the controller130may generate the information regarding time stamps simultaneously while obtaining mobile terminal images. The controller130may compensate timing by the sensed time shifting when certain time shifting is sensed by comparing the movement patterns transformed into the screen coordinate system with the movement patterns of mobile terminal images.

As described above, the controller130can correctly recognize the movement information of the mobile terminal200by transforming the coordinate systems and compensating the movement information through mobile terminal images. Specifically, the controller130may correctly measure movements of x, y and z axes regarding the mobile terminal200through the above described method.

Further, the controller130may control various functions of the display apparatus100by using the compensated movement information, as described above.

According to an exemplary embodiment, the controller130may determine three-dimensional movements of the mobile terminal200by using the movement information of the mobile terminal200, and transmit controlling commands to the mobile terminal200by using the determined three-dimensional movements. Specifically, when movement directed toward a y-axis direction is sensed, the controller130may transmit controlling commands to the mobile terminal200in order to perform a function corresponding to the movement directed toward the y-axis direction. For example, as illustrated inFIG. 6, when movement of the mobile terminal200approaching toward the y-axis direction of the display apparatus100(i.e., movement to throw the mobile terminal200toward the display apparatus100) is sensed, the controller130may request a file which is being reproduced by the mobile terminal200, and the mobile terminal200may transmit a currently reproducing file to the display apparatus100.

Further, when a plurality of mobile terminals operate, the controller130may sense movement patterns corresponding to the plurality of the mobile terminals, respectively, by using the compensated movement information with respect to each of the plurality of mobile terminals, respectively. Further, the controller130may recognize a plurality of the mobile terminals by using the movement patterns respectively corresponding to a plurality of the mobile terminals. Specifically, the controller130may distinguish a plurality of the mobile terminals by using the compensated movement information respectively regarding a plurality of the mobile terminals. For example, as illustrated inFIG. 7, the controller130may distinguish the mobile terminal having movement patterns directed toward the x-axis, the mobile terminal having movement patterns directed toward the y-axis, and the mobile terminal having movement patterns directed toward the z-axis. Further, the controller130may extract user faces respectively near the plurality of the mobile terminals by using the camera included in the image acquirer120, and distinguish users utilizing the mobile terminals by comparing the extracted user faces with user information stored in the storage170. Thus, when the extracted face near the mobile terminal which has patterns of moving toward the x axis, is that of user1, the extracted face near the mobile terminal which has patterns of moving toward y axis, is user2, and the extracted face near the mobile terminal which has patterns of moving toward the z axis, is user3, the controller130may recognize each user by comparing the extracted faces with user information stored in the storage170respectively. Therefore, the controller130may control functions of the display apparatus100by independently using the mobile terminals of the extracted users. Further, a plurality of users can simultaneously play games through the display apparatus100.

Further, the controller130may control the display apparatus100based on the compensated movement information only when the mobile terminal is photographed. Specifically, as illustrated inFIG. 8, only when the mobile terminal200is photographed by the camera provided from the display apparatus100, the controller130may operate the display apparatus100by using movements of the mobile terminal200. When the mobile terminal200is not photographed by the camera provided from the display apparatus100, the controller130may not perform functions corresponding to movements of the mobile terminal200even if movements of the mobile terminal are received. Thus, the controller130may perform various functions of the display apparatus100(e.g., connecting between devices, connecting applications, screen AR mode and motion controlling mode) only when movement information of the mobile terminal200is received and mobile terminal images are obtained.

Further, the controller130may calculate pointing coordinate values based on the compensated movement information and control the display160to display a pointer on the pointing coordinate values. Specifically, as illustrated inFIG. 9, the controller130may sense a current position directed by the mobile terminal200based on mobile terminal images and movement information of the mobile terminal200, and establish the sensed position to be the starting point. Further, the controller130may control the display160to calculate pointing coordinate values from the starting point based on the movement information of the mobile terminal200and display a pointer. Herein, the controller130may provide various applications using a pointer. For example, the controller130may control a web browser by using a touch pad included in the mobile terminal200, a pointing movement, text box pointing, and a character inputting pad included in the mobile terminal200. For another example, the controller130may perform various operations such as selecting objects or obtaining object information on a TV screen with a pointer. Herein, when a plurality of pointers respectively corresponding to a plurality of mobile terminals are used, the controller130may control the display160to differently display a plurality of the pointers. For example, the controller130may control the display160to display a plurality of the pointers by adjusting color, brightness and clarity of the pointers to be different to each other.

Meanwhile, the above exemplary embodiment describes that three-dimensional movements are determined by using movement information and compensated by using mobile terminal images. However, this is merely one of the embodiments. Three-dimensional movements may be obtained with other methods. Specifically, as illustrated inFIG. 10, the controller130may determine two-dimensional movements of the mobile terminal200, i.e., movements directed toward x and z axes, from mobile terminal images, and movements directed toward the y axis based on the movement information of the mobile terminal200.

Further, the controller130may control various functions of the display apparatus100based on a user's attention to the device, i.e., user eyesight. Specifically, as illustrated inFIG. 11, the controller130may extract user eyesight by using the camera included in the image acquirer120. When user eyesight directs toward the mobile terminal200, the controller130does not control the display apparatus100by using movement information of the mobile terminal200. Further, a user may use various applications when using the mobile terminal200. However, when user eyesight is directed toward the display apparatus100, the controller130may display a pointer on the display160, and move a pointer by using movements of the mobile terminal200. Herein, the controller130may stop the operation of applications in the mobile terminal200by transmitting a request to stop operation, to the mobile terminal200.

Further, the controller130may sense whether a user comes in or out of a screen area by using movement information of the mobile terminal200and mobile terminal images. Specifically, as illustrated inFIG. 12, the controller130may sense whether a user comes within a screen area by using mobile terminal images. Further, the controller130may track a user moving line within the screen area by using movement information of the mobile terminal200in order to reduce loading of vision processing, and track a user moving line by using mobile terminal images only when being requested. Thereby, the controller130may perform various application functions (e.g., function to turn on the light in an area where a user moves) by considering and tracking a user's movement. Further, the controller130analyzes and learns action radii or moving lines of a user having the mobile terminal200when modeling an interior dimension based on 2D cameras. Thus, correctness of 3D interior dimension modeling can be enhanced.

Further, the controller130may distinguish the mobile terminal200by using movement patterns of the mobile terminal200, and authorize a user with user information of the mobile terminal200. For example, as illustrated inFIG. 13, when a first mobile terminal having patterns of moving toward an x-axis direction is sensed, the controller130may perform the user authorization by using information regarding USER1 stored in the first mobile terminal. When a second mobile terminal having patterns of moving toward the z-axis direction is sensed, the controller130may perform the user authorization by using information regarding USER2 stored in the second mobile terminal. Thereby, a user may perform the user authorization such as logging-in by using movement patterns of the mobile terminal200.

The following will explain a controlling method of the display apparatus100by referring toFIG. 14.

At S1410, the display apparatus100receives movement information of the mobile terminal200from the mobile terminal200, and obtains mobile terminal images by photographing the mobile terminal200. Receiving movement information of the mobile terminal200and photographing the mobile terminal200can be simultaneously performed; however, this is merely one of the exemplary embodiments. They can be sequentially performed.

At S1420, the display apparatus100transforms the received movement information to the screen coordinate system. Specifically, the display apparatus100may transform the movement information at the mobile terminal coordinate system which is arrayed based on the mobile terminal200to the global coordinate system which is arrayed based on geomagnetic direction. Further, the display apparatus100may transform the movement information transformed into the global coordinate system to the screen coordinate system by using transforming parameters of the screen coordinate system.

At S1430, the display apparatus100compensates the movement information transformed into the screen coordinate system based on mobile terminal images. Specifically, the display apparatus100may compensate the movement information by comparing size of the mobile terminal200obtained from mobile terminal images with the actual size of the mobile terminal200which is previously obtained. Further, when the movement information of the mobile terminal obtained from mobile terminal images is not in agreement with the movement information transformed into the screen coordinate system by comparing the movement information of the mobile terminal200obtained from images of the mobile terminal200with the movement information transformed into the screen coordinate system, the display apparatus100may synchronize the movement information of the mobile terminal obtained from mobile terminal images with the movement information transformed into the screen coordinate system.

At S1440, the display apparatus100controls functions of the display apparatus100by using the compensated movement information. Specifically, the display apparatus100may control various functions illustrated inFIG. 13by using the compensated movement information.

According to the controlling method of the display apparatus described above, a user can control the display apparatus more correctly and intuitively by using the mobile terminal.

Meanwhile, the above exemplary embodiments describe that the display apparatus100receives movement information from the mobile terminal200, transforms the received movement information to the screen coordinate system which is arrayed based on the screen of the display apparatus, and compensates the movement information transformed into the screen coordinate system based on the photographed mobile terminal images. However, this is merely one of the exemplary embodiments. The technical art of the invention may also include an exemplary embodiment in which the mobile terminal200directly compensates the movement information.

The above will be explained by referring toFIG. 15. First, the mobile terminal includes the sensor210, the communicator220, and the controller230, as illustrated inFIG. 15.

The sensor210senses movement information of the mobile terminal by using a plurality of sensors. Specifically, the sensor210may sense movement information by using at least one of the acceleration sensor, the geomagnetic sensor and the gyro sensor.

The communicator220performs communication with the external display apparatus100. Specifically, the communicator220may receive mobile terminal images from the display apparatus100, and transmit the compensated movement information to the display apparatus100.

The controller230controls general operation of the mobile terminal200. Specifically, the controller230calculates coordinate values based on the sensed movement information through the sensor210, and transform the calculated coordinate values into the screen coordinate system which is arrayed based on the screen of the display apparatus100. Herein, the controller230may transform coordinate values according to a consecutive order of the mobile terminal coordinate system, the global coordinate system, then the screen coordinate system, as described above.

Further, the controller230controls the communicator220to receive mobile terminal images from the display apparatus100.

The controller230may compensate the movement information based on the received mobile terminal images. Specifically, the controller230may compensate the movement information by comparing the size of the mobile terminal obtained from the mobile terminal images with the previously obtained actual size of the mobile terminal. Further, the controller230may synchronize the movement information of the mobile terminal obtained from the mobile terminal images and the movement information transformed into the screen coordinate system by comparing the movement information of the mobile terminal obtained from the mobile terminal images with the movement information transformed into the screen coordinate system.

Further, the controller230may transmit the compensated movement information to the display apparatus100.

Meanwhile, the controlling method of the display apparatus according to the above various exemplary embodiments may be implemented to be programs and provided to the display apparatus or inputting devices. Specifically, programs including the controlling method of the display apparatus may be stored and provided in non-transitory computer readable recording medium.

Non-transitory computer readable recording medium indicate medium which store data semi-permanently and can be read by devices, not medium storing data temporarily such as register, cache, or memory. Specifically, the above various applications or programs may be stored and provided in non-transitory computer readable recording medium such as CDs, DVDs, hard disks, Blu-ray disks, USBs, memory cards, or ROM.