Pointing apparatus, pointer control apparatus, pointing method, and pointer control method

Provided is a pointing apparatus, a pointer control apparatus, a pointing method, and a pointer control method to a pointing apparatus, a pointer control apparatus, a pointing method, and a pointer control method capable of recognizing image codes included in an image frame using an image sensor to determine a pointing direction, and continuously updating the gain between the displacement of the motion of the pointing apparatus and the displacement of the motion of a displayed pointer. The pointing apparatus includes an image receiving unit sensing image patterns that exist in a sensed region, among all of the image patterns arranged in a display region; an inertial sensor sensing an input motion using at least one of the acceleration and angular velocity that are generated due to the motion; and a coordinate determining unit determining moving coordinates that are moved from the central coordinates of the sensed image pattern by coordinate displacement corresponding to the sensed motion.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No. 10-2007-0090672 filed on Sep. 6, 2007 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a pointing apparatus, a pointer control apparatus, a pointing method, and a pointer control method, and more particularly, to a pointing apparatus, a pointer control apparatus, a pointing method, and a pointer control method capable of recognizing image codes included in an image frame using an image sensor to determine a pointing direction, and continuously updating the gain between the displacement of the motion of the pointing apparatus and the displacement of the motion of a displayed pointer.

2. Description of the Related Art

Pointing apparatuses, which use inertial sensors (acceleration sensors or angular velocity sensors), sense an input motion using the inertial sensor and control the motion of a displayed pointer. When the inertial sensor is used to control the motion of a pointer, the motion of the pointer is determined according to the motion of the pointing apparatus regardless of the pointing direction of the pointing apparatus, which is called a relative pointing method.

In the relative pointing method, the user does not need to point a pointing apparatus at a display apparatus. However, the relative point method cannot be used in a case when the pointing apparatus is configured so as to operate only when the user points the pointing apparatus to the display apparatus.

Meanwhile, the gain relationship is established between the displacement of the motion of the pointing apparatus and the displacement of the motion of the pointer that is actually displayed on the display apparatus. In a pointing apparatus that senses a rotary motion using an inertial sensor, when the gain is fixed, the displacement of the motion of the pointer that is displayed on the display apparatus according to the displacement of a rotary motion of the pointing apparatus is fixed regardless of the distance between the pointing apparatus and the display apparatus, which makes it difficult to achieve an absolute pointing apparatus.

As described above, when only the inertial sensor is used, there is no way to provide an absolute pointing apparatus or update the gain. Therefore, a technique for solving these problems has been demanded.

SUMMARY OF THE INVENTION

Aspects of the present invention provide a technique for recognizing image codes included in an image frame using an image sensor to determine a pointing direction.

Aspects of the present invention also provide a technique for continuously updating the gain between the displacement of the motion of a pointing apparatus and the displacement of the motion of a displayed pointer.

According to an aspect of the present invention, there is provided an pointing apparatus including: an image receiving unit sensing image patterns that exist in a sensed region, among all of the image patterns arranged in a display region; an inertial sensor sensing an input motion using at least one of the acceleration and angular velocity that are generated due to the motion; and a coordinate determining unit determining moving coordinates that are moved from the central coordinates of the sensed image pattern by coordinate displacement corresponding to the sensed motion.

According to another aspect of the present invention, there is provided a pointer control apparatus including: a receiving unit receiving image patterns that exist in a sensed region among all of the image patterns arranged in a display region and a motion that is sensed by an inertial sensor; and a coordinate determining unit determining moving coordinates that are moved from the central coordinates of the image pattern by coordinate displacement corresponding to the sensed motion.

According to another aspect of the present invention, there is provided a pointing method including: sensing image patterns that exist in a sensed region, among all of the image patterns arranged in a display region; sensing an input motion using at least one of the acceleration and angular velocity that are generated due to the motion; and determining moving coordinates that are moved from the central coordinates of the sensed image pattern by coordinate displacement corresponding to the sensed motion.

According to another aspect of the present invention, there is provided a pointer control method including: receiving image patterns that exist in a sensed region among all of the image patterns arranged in a display region and a motion that is sensed by an inertial sensor; and determining moving coordinates that are moved from the central coordinates of the image pattern by coordinate displacement corresponding to the sensed motion.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1is a diagram illustrating a pointer control system according to an embodiment of the invention. The pointer control system includes a pointing apparatus200, a pointer control apparatus300, and a display apparatus400.

The pointer control apparatus300generates a mouse pointer100, a graphic object110, and an image pattern for allowing the pointing apparatus200to determine the range of an image recognizing region700. The image recognizing region means an image region that is sensed by an image receiving unit provided in the pointing apparatus200. The pointing apparatus200determines the range of the image recognizing region to know a pointing position on a current display region450.

The pointer control apparatus300may be an apparatus having a data processing function and an operating function, such as a personal computer provided with a central processing unit, a TV set-top box, or a CPU board provided in a TV.

The user can use the pointing apparatus200to point to a predetermined position on the display region450. The pointing apparatus200can recognize a pointing position using an image pattern displayed on the display region450.

The pointing apparatus200can analyze the displayed image pattern to determine the range of an image pattern, and extract the coordinates of a pointing position on the recognized display region450on the basis of the range of the image pattern.

Data transmission between the pointer control apparatus300and the pointing apparatus200is performed by wire or wireless communication. The pointing apparatus200transmits the coordinates of the extracted pointing position to the pointer control apparatus300. Then, the pointer control apparatus300converts the received coordinates into the actual coordinates on the display region450.

The images of the image pattern, the graphic object110and the mouse pointer100generated by the pointer control apparatus300are transmitted to the display apparatus400, and then the display apparatus400displays the received images.

The pointer control apparatus300and the display apparatus400may be integrated into one apparatus.

FIG. 2is a block diagram illustrating the structure of the pointing apparatus according to the embodiment of the invention. The pointing apparatus200includes an image receiving unit210, an inertial sensor220, a control unit230, a coordinate determining unit240, a gain calculating unit250, and a transmitting unit260.

The image receiving unit210senses an image pattern that is displayed on a recognized region, among all of the image patterns displayed on the display region450. The display apparatus400displays a continuous series of image frames to form a still picture or a moving picture. All of the image patterns may be arranged in image frames that are displayed at a predetermined time interval, among the image frames displayed on the display region450.

For example, when the display apparatus400displays thirty image frames per second, all of the image patterns may be arranged in at least one of the thirty image frames, and the image frames having the image patterns arranged therein may be displayed at a predetermined time interval. In order to sense the image pattern, the image receiving unit210may include a camera unit211and a region sensing unit212.

The camera unit211captures image patterns included in a specific image frame among the image frames displayed on the display region450. The image pattern may be formed by visible rays, infrared rays, or ultraviolet rays. When an image pattern is formed by visible rays, the camera unit211may perform the function of a general digital camera to capture the image pattern.

Meanwhile, when an image pattern is formed by infrared rays or ultraviolet rays, the camera unit211may perform the function of an infrared camera or an ultraviolet camera to capture the image pattern.

The region sensing unit212senses the range of an image pattern. For example, when a plurality of image codes forming an image pattern are arranged on the display region450, the region sensing unit212uses the arrangement of partial images to sense the range of the image pattern.

In order to sense the range of an image pattern, a plurality of different image codes forming an image pattern may be arranged at fixed positions on the display region450.

After sensing the range of the image pattern, the region sensing unit212may check the central coordinates of the image pattern.

The inertial sensor220displays the inertial force of a mass that is generated by acceleration or angular velocity as the deformation of an elastic structure connected to the mass, and displays the deformation of the structure as electric signals using an appropriate sensing method and an appropriate signal processing method. In this embodiment, the inertial sensor220senses the motion of the pointing apparatus200.

That is, the inertial sensor220uses at least one of the acceleration and the angular velocity that are generated by the actual motion input to the pointing apparatus200to sense the estimated motion of the pointing apparatus on the basis of the actual motion.

The inertial sensor220may sense a three-dimensional motion composed of a linear motion and a curvilinear motion, as well as a two-dimensional motion, such as a linear or curvilinear motion. That is, the inertial sensor220generates a series of electric signals from a two-dimensional or three-dimensional basic motion. The user may combine a plurality of basic motions to generate a desired motion.

The coordinate determining unit240determines moving coordinates that are moved from the central coordinates of an image pattern by coordinate values corresponding to the motion sensed by the inertial sensor220. The central coordinates of the image pattern sensed by the region sensing unit212may be the absolute coordinates of the image pattern on the display region450. The coordinate determining unit240determines the absolute coordinates of an image pattern that are moved from the central coordinates by coordinate values input from the inertial sensor220.

Meanwhile, in the pointing apparatus200provided with the inertial sensor220, the motion sensed by the inertial sensor220is the motion input by the user. When the same gain is applied, the coordinates on the display region450are moved in proportional to the rotation angle of the pointing apparatus200, regardless of the distance between the display apparatus400and the pointing apparatus200.

However, in order to operate the pointing apparatus200provided with the inertial sensor220like an absolute-pointing-type pointing apparatus, the coordinates need to be moved on the display region450in consideration of the rotation angle of the pointing apparatus200and the distance between the display apparatus400and the pointing apparatus200.

For example, when the distance between the display apparatus400and the pointing apparatus200is small, the user needs to make a large motion in order to form specific coordinate displacement on the display region450. However, when the distance between the display apparatus400and the pointing apparatus200is large, the user does not need to make a large motion.

Therefore, it is preferable to apply the gain depending on the distance between the display apparatus400and the pointing apparatus200to the motion sensed by the inertial sensor220, in order to determine coordinate displacement on the display region450. The gain calculating unit250compares the coordinate displacement between two image patterns that are sensed at a predetermined time interval among the image patterns with inertial coordinates displacement corresponding to the motion sensed by the inertial sensor220for a predetermined period, and calculates the gain for synchronizing the inertial coordinates displacement with the displacement of the moving coordinates on the display region450.

As described above, when the image frames each having image patterns arranged therein are displayed at a predetermined time interval, the image receiving unit210receives the image patterns at the predetermined time interval. When the user inputs a motion, the difference between two image patterns in the time direction occurs, and the coordinate determining unit240can calculate coordinate displacement due to the difference between the image patterns.

The coordinates determining unit240can calculate coordinate displacement due to the motion sensed by the inertial sensor220. The gain calculating unit250calculates gain for synchronizing two coordinate displacements. The operation of the gain calculating unit250calculating the gain will be described in detail below with reference toFIG. 9.

The coordinate determining unit240determines moving coordinates that are moved from the central coordinates of the image pattern sensed after a predetermined time by the inertial coordinate displacement to which the gain is applied.

The transmitting unit260transmits the central coordinates of the image pattern sensed by the image receiving unit210and the moving coordinates determined by the coordinate determining unit240to the pointer control apparatus300. In this case, the gain is applied to the moving coordinates.

Data communication may be performed between the transmitting unit260and the pointer control apparatus300by wire communication methods, such as Ethernet, USB, IEEE 1394, serial communication, and parallel communication, or wireless communication methods, such as infrared communication, Bluetooth, Home RF, and wireless LAN.

Meanwhile, when the pointer control apparatus300is provided with the coordinate determining unit and the gain calculating unit, only the image pattern captured by the camera unit211and the motion sensed by the inertial sensor220may be transmitted to the pointer control apparatus300through the transmitting unit260.

The control unit230controls the image receiving unit210, the inertial sensor220, the coordinate determining unit240, the gain calculating unit250, the transmitting unit260, and the pointing apparatus200.

FIG. 3is a block diagram illustrating the structure of the pointer control apparatus according to the embodiment of the invention. The pointer control apparatus300includes a receiving unit310, an image generating unit320, a coordinate determining unit330, a control unit340, a gain calculating unit350, a pointer operating unit360, and a transmitting unit370.

The image generating unit320generates image frames. The image generating unit320can generate thirty or sixty image frames per second, and image patterns may be arranged in one or more image frames. The image frames generated by the image generating unit320are transmitted to the display apparatus400through the transmitting unit370, and the display apparatus400displays the received image frames.

The receiving unit310receives the central coordinates of an image pattern and the moving coordinates from the pointing apparatus200. The moving coordinates are the coordinates of a position that is moved from the central coordinates of an image pattern in the image region sensed by the image receiving unit210of the pointing apparatus200by the coordinate displacement sensed by the inertial sensor220, and include the absolute coordinates of the image pattern on the display region450.

The received coordinates are transmitted to the pointer operating unit360, and the pointer operating unit360moves the mouse pointer100to the received coordinates.

The transmitting unit370transmits the image of the mouse pointer100whose position is moved to the display apparatus400. Then, the display apparatus400displays the mouse pointer100on the display region450.

Meanwhile, when the pointer control apparatus300is provided with the coordinate determining unit330and the gain calculating unit350, the receiving unit310can receive the motion sensed by the inertial sensor220and the image patterns that exist in a predetermined image region sensed by the image receiving unit210of the pointing apparatus200, among all of the image patterns arranged in the display region450.

The coordinate determining unit330checks the central coordinates of the image pattern, and determines the moving coordinates that are moved from the central coordinates by coordinate displacement that corresponds to the motion sensed by the inertial sensor220.

Then, the gain calculating unit350calculates gain for synchronizing inertial coordinate displacement corresponding to the motion sensed by the inertial sensor220with the moving coordinate displacement on the display region450.

The coordinate determining unit330and the gain calculating unit350of the pointer control apparatus300have the same functions as the coordinate determining unit240and the gain calculating unit250of the pointing apparatus200, and thus a detailed description thereof will be omitted.

The pointer operating unit360moves the mouse pointer100according to the results obtained from the coordinate determining unit330and the gain calculating unit350, and the transmitting unit370transmits the image of the moved mouse pointer100to the display apparatus400.

The control unit340controls the receiving unit310, the image generating unit320, the coordinate determining unit330, the gain calculating unit350, the pointer operating unit360, the transmitting unit370, and the pointer control apparatus300.

Meanwhile, the pointer control apparatus300may be provided with a display device (not shown). In this case, the display device may display the mouse pointer100moved by the pointer operating unit360.

FIG. 4is a diagram illustrating the structure of the display apparatus according to the embodiment of the invention. The display apparatus400includes a receiving unit410and a display device420.

The receiving unit410receives image frames from the pointer control apparatus300, and the display device420displays the received image frames.

The display device420is a module having an image display unit capable of displaying input image signals, such as a CRT (cathode ray tube), an LCD (liquid crystal display), an LED (light-emitting diode), an OLED (organic light-emitting diode), or a PDP (plasma display panel), and displays received image information.

InFIGS. 2 and 3, the pointer control apparatus300and the display apparatus400are separated from each other, but the invention is not limited thereto. The pointer control apparatus300and the display apparatus400may be integrated into one apparatus.

FIG. 5is a diagram illustrating image frames each having image patterns arranged therein according to the embodiment of the invention.

The image frames generated by the image generating unit320are displayed by the display device420of the display apparatus400. In this case, the display device420may display thirty or sixty different image frames510,520, and530per second.

In this embodiment, the image frame520having image patterns arranged at predetermined time intervals may be displayed. That is, the image frame520having the image patterns arranged therein may be interposed between the image frames510and530each including only general image information.

As described above, the image patterns are formed by visible rays, infrared rays, or ultraviolet rays. When the image patterns are formed by visible rays, it is preferable that the image frame520include only the image patterns. When the image patterns are formed by infrared rays or ultraviolet rays, the image patterns may be included in the image frame510or530including general image information.

When the display device displays thirty or sixty image frames per second, preferably, the camera unit211provided in the pointing apparatus200recognizes one frame in a 1/30 or 1/60 second.

FIG. 6is a diagram illustrating an image pattern composed of image codes according to an embodiment of the invention.FIG. 6shows an image frame including an image pattern600among the image frames displayed by the display device420.

As shown inFIG. 6, the image pattern600includes a plurality of different image codes, and the image codes are arranged at fixed positions on the display region450. Therefore, the region sensing unit212of the pointing apparatus200can check a position indicated by the pointing apparatus200on the basis of the image captured by the camera unit211.

FIGS. 7 and 8are diagrams illustrating image patterns710sensed by the image receiving unit210according to the embodiment of the invention.FIGS. 7 and 8show the image patterns710arranged in an image region (hereinafter, referred to a recognition region)700recognized by the camera unit211.

FIG. 7shows some image patterns700arranged in the recognition region700, among all of the image patterns600. InFIG. 7, right and left portions of the image patterns are cut. Therefore, the region sensing unit212compares the image patterns710in the recognition region700with the image patterns on the display region450that are stored beforehand to know that the pointing apparatus200points to the center of the display region450.

The pointing apparatus200may be provided with a storage unit (not shown) that stores the image patterns on the display region450.

Meanwhile, some image patterns810that are positioned at a lower left portion among the image patterns600may be arranged in the recognition region700, as show inFIG. 8. The region sensing unit212can know that the pointing apparatus200points to a lower right portion of the display region450using the image patterns810arranged at an upper left portion of the recognition region700.

The region sensing unit212compares the image patterns arranged in the recognition region700(hereinafter, referred to as a recognition image pattern) with the image patterns on the display region450that are stored beforehand (hereinafter, referred to as display image patterns) to calculate the coordinates of the image pattern on the display region450. The scales of the recognition image patterns and the display image patterns may depend on the distance between the display apparatus400and the pointing apparatus200.

That is, when the distance between the display apparatus400and the pointing apparatus200is small, the size of the recognition image pattern may be larger than that of the display image pattern. When the distance between the display apparatus400and the pointing apparatus200is large, the size of the recognition image pattern may be smaller than that of the display image pattern.

Therefore, the region sensing unit212performs image processing on the recognition region700to make the scale of the recognition image pattern equal to the scale of the display image pattern, and checks the coordinates of a position indicated by the pointing apparatus200.

Further, the recognition image pattern may be recognized as being distorted according to the pointing angle of the pointing apparatus200with respect to the display apparatus400. In this case, the region sensing unit212corrects the distorted image pattern in the forward direction, and compares the corrected image pattern with the display image pattern, thereby checking the coordinates of a position indicated by the pointing apparatus200.

FIG. 9is a diagram illustrating the relationship between the displacement of the pointing apparatus and the displacement of the pointer according to the embodiment of the invention.

The user may input a motion into the pointing apparatus200such that the mouse pointer100moves between a left end and a right end of the display region450. In this case, the mouse pointer100moves to correspond to the motion sensed by the inertial sensor220of the pointing apparatus200, and the amount930of the motion of the pointing apparatus200is regardless of the distance between the display apparatus400and the pointing apparatus200.

However, in order to achieve an absolute pointing apparatus, it is preferable that the amount930of the motion of the mouse pointer100be kept constant even when the amount910or920of the motion of the pointing apparatus200varies according to the distance between the display apparatus400and the pointing apparatus200, as shown inFIG. 9.

In order to keep the amount of the motion of the mouse pointer930constant, the gain calculating unit250continuously updates the gain, and the coordinate determining unit240applies the updated gain to the motion sensed by the inertial sensor220. The gain is calculated as follows.

In this embodiment, since the mouse pointer100is two-dimensionally moved on the display region450, only the two-dimensional motion among the motions input to the pointing apparatus200will be considered. It is assumed that the horizontal direction of the display apparatus400is an X-axis, and the vertical direction thereof is a Y-axis. It is considered only the angular velocity sensed by the inertial sensor220in this embodiment.

The image patterns are displayed at predetermined time intervals. When the angular velocity sensed by the inertial sensor220between adjacent times t1and t2is (θx, θy), an angular variation is represented by Expression 1 given below:

When the central coordinates of an image pattern are (x1, y1), the central coordinates of the next image pattern are (x2, y2), and gain is (Rx, Ry), the relationship therebetween is represented by Expression 2 given below:
Rxθx=x2−x1, and
Ryθy=y2−y1.  [Expression 2]

The gain (Rx, Ry) is rearranged as Expression 3 given below:

When the angular velocity sensed by the inertial sensor220after a predetermined time is (θx, θy), coordinate displacement (Δx, Δy) is represented by Expression 4 given below:
Δx=Rxθy, and
Δy=Ryθy.  [Expression 4]

FIG. 10is a flowchart illustrating a pointing process according to an embodiment of the invention. In order to control the motion of the mouse pointer100, first, the image receiving unit210of the pointing apparatus200senses image patterns existing in a sensed region, among all of the image patterns arranged in the display region450(S1010).

Then, the region sensing unit212of the image receiving unit210senses the range of image patterns and checks the central coordinates of the image pattern (S1020). The central coordinates include absolute coordinates on the display region450.

The inertial sensor220uses at least one of the acceleration and the angular velocity generated due to a motion input to the pointing apparatus200to sense an estimated motion for the input motion (S1030).

Then, the coordinate determining unit240determines moving coordinates that are moved from the central coordinates of the image pattern by coordinate displacement corresponding to the motion sensed by the inertial sensor220(S1040).

The central coordinates of the image pattern checked by the image receiving unit210and the moving coordinates determined by the coordinate determining unit240are transmitted to the pointing control apparatus through the transmitting unit260(S1050).

Meanwhile, the image receiving unit210receives image frames each having image patterns arranged therein at predetermined time intervals. The gain calculating unit250compares the displacement between the coordinates of two image patterns with inertial coordinate displacement for the motion sensed by inertial sensor220to calculate the gain for synchronizing the inertial coordinate displacement with the displacement of the moving coordinates on the display region450(S1060).

Then, the coordinate determining unit240determines moving coordinates that are moved from the central coordinates of an image pattern sensed after a predetermined time by inertial coordinate displacement to which gain is applied, and the determined moving coordinates are transmitted to the pointer control apparatus300through the transmitting unit260(S1070).

The updating of gain is performed whenever the image pattern is received.

FIG. 11is a flowchart illustrating a pointer control process according to an embodiment of the invention.

In order to control the motion of the mouse pointer100, first, the image generating unit320of the pointer control apparatus300generates image frames (S1110). The image frames generated by the image generating unit320may include an image frame having image patterns arranged therein. The generated image frames are transmitted to the display apparatus400through the transmitting unit370, or they are displayed by a display device (not shown) provided in the pointer control apparatus300(S1120).

The receiving unit310receives the central coordinates of the image pattern and the moving coordinates from the pointing apparatus200.

The received central coordinates and moving coordinates are transmitted to the pointer operating unit360, and then the pointer operating unit360moves the mouse pointer100by the displacement between the central coordinates and the moving coordinates.

Information on the movement of the mouse pointer100is transmitted to the image generating unit320, and the image generating unit320forms the image of the mouse pointer100at the moved position, and the image frame generated by the image generating unit320is transmitted to the display apparatus400through the transmitting unit370, or it is displayed by a display device (not shown).

Meanwhile, when the pointer control apparatus300is provided with the coordinate determining unit330and the gain calculating unit350, the receiving unit310can receive the image pattern and the motion sensed by the inertial sensor220from the pointing apparatus200(S1130).

The received image pattern and motion are transmitted to the coordinate determining unit330. Then, the coordinate determining unit330checks the central coordinates of the image pattern (S1140), and determines the moving coordinates that are moved from the central coordinates by coordinate displacement corresponding to the received motion (S1150).

The pointer operating unit360moves the mouse pointer100to the moving coordinates determined by the coordinate determining unit330(S1160), and an image frame is generated. The generated image frame is transmitted to the display apparatus400through the transmitting unit370, or it is displayed by a display device (S1170).

The gain calculating unit350compares the displacement between the coordinates of two image patterns with inertial coordinates displacement corresponding to the motion sensed by the inertial sensor220, and calculates the gain for synchronizing the inertial coordinates displacement with the displacement of the moving coordinates on the display region450(S1180).

The coordinate determining unit330determines the moving coordinates that are moved from the central coordinates of an image pattern sensed after a predetermined time by inertial coordinate displacement to which the gain is applied. The image of the mouse pointer that is moved to the moving coordinates is transmitted to the display apparatus400through the transmitting unit370, or it is displayed by a display device (S1190).

As described above, the pointing apparatus, the pointer control apparatus, the pointing method, and the pointer control method according to the above-described embodiments have the following effects.

First, it is possible to provide an absolute pointing apparatus by using an image sensor to recognize image codes included in an image frame, thereby determining a pointing direction.

Second, the gain between the displacement of the motion of the pointing apparatus and the displacement of the motion of a displayed pointer is continuously updated. Therefore, it is possible to perform accurate pointing even when the distance between the pointing apparatus and the display apparatus varies.