Three-dimensional object display control system and method thereof

Provided is a three-dimensional object display control system that enables a three-dimensional object displayed on a screen to be freely rotated by means of instinctive operation. A computer system (10) includes, e.g., a computer (12), a display (14), and a rotation information input device (20). The rotation information input device (20) includes a sphere (22), a tripod (24) that rotatably supports the sphere (22), and a camera (26) that takes an image of the sphere (22). The computer (12) selects two feature points from the image of the sphere (22) taken by the camera (26), and sets two-dimensional coordinates in the image of the feature points. While tracking the feature points, the computer (12) samples the two-dimensional coordinates and transforms them into three-dimensional coordinates on a surface of the sphere (22). Then, the computer (12) calculates a rotation matrix representing a rotation of the sphere (22), based on the three-dimensional coordinates, and multiplies the respective vertex coordinates of the three-dimensional object displayed on the display (14) by the rotation matrix, thereby calculating the respective vertex coordinates of the three-dimensional object after rotated.

TECHNICAL FIELD

A three-dimensional object display control system according to the present invention is a three-dimensional object display control system for rotating a three-dimensional object displayed on a screen, the system including a sphere, a camera, a rotation determiner and an object rotater. The sphere is rotatable, and is preferably supported by three bearings, but it may have nothing to support it. The camera takes an image of the sphere. The rotation determiner is arranged to determines a rotation of the sphere based on the image of the sphere taken by the camera. The object rotator arranged to rotates the three-dimensional object in conjunction with the rotation determined by the rotation determiner.

BACKGROUND ART

In recent years, three-dimensional graphics have been used in various fields. For example, in the scientific field, it is used, e.g., when displaying the results of numerical calculation or when viewing the structure of a protein. In the industrial field, it is used for, e.g., automobile designing, architectural designing, or CAD (Computer Aided Design). Also, for familiar examples, it can be seen in TVs, PCs, video games, etc.

Many systems employ a mouse as an input device for rotating a three-dimensional object displayed on a screen. In most applications, an operation to rotate a three-dimensional object with a mouse enables rotation of the object around the vertical axis on the screen when moving the mouse rightward or leftward and rotation of the object around the horizontal axis when moving the mouse forward or backward.

However, it is impossible to freely rotate the three-dimensional object by means of instinctive mouse operation. In other words, it is not easy to rotate the object around an axis perpendicular to the screen by means of operating the mouse to move back and forth and around. For example, in order to rotate the object around an axis perpendicular to the screen by 180 degrees, it is necessary to, first, move the mouse to the right or left to rotate it around the vertical axis by 180 degrees, and then, to move the mouse forward or backward to rotate it around the horizontal axis by 180 degrees. An operation to perform rotation around an axis perpendicular to the screen may be possible, but when it comes to rotation around an axis slightly diagonally intersecting with the screen, an operation to perform such rotation is extremely difficult.

Japanese Patent Laid-Open No. 2005-32245 describes a controller for a video game. This controller includes a joystick shaft, a spherical element mounted below the joystick shaft, an imager that sequentially inputs images of the surface of the spherical element, and a movement detector that detects movement of the element based on the results of comparison between the plural input images. This controller is basically used as an input device for two-dimensional or three-dimensional coordinates, and is not intended for rotation of a three-dimensional object displayed on a screen.

Japanese Patent Laid-Open No. 2000-330722 describes a position information input device for three-dimensionally moving a displayed image. This input device includes a rotation detecting mechanism for rotating a displayed image around an X-axis, a Y-axis and a Z-axis. The rotation detecting mechanism is configured to, upon a sphere being rolled with the tip of a finger, output the direction and amount of the rotation; however, the detection of the rotation is performed mechanically.

Japanese Patent Laid-Open No. 2000-207093 describes a rotation display device for a three-dimensional figure. This rotation display device, upon a ball body being rotated with a hand, detects such rotation to rotate a displayed image concurrently; however, the detection of the rotation is also performed mechanically.

Japanese Patent Laid-Open No. 9-218744 describes a mouse for rotating a displayed three-dimensional graphic image; however, the structure thereof is completely different from the present invention, which will be described later.

SUMMARY

An object of the present invention is to provide a three-dimensional object display control system and method, which enable a three-dimensional object displayed on a screen to be freely rotated by means of instinctive operation, and a three-dimensional object display control program used for the same.

A three-dimensional object display control system according to the present invention is a three-dimensional object display control system for rotating a three-dimensional object displayed on a screen, the system including a sphere, a camera, rotation determining means and object rotating means. The sphere is rotatable, and is preferably supported by three bearings, but it may have nothing to support it. The camera takes an image of the sphere. The rotation determining means determines a rotation of the sphere based on the image of the sphere taken by the camera. The object rotating means rotates the three-dimensional object in conjunction with the rotation determined by the rotation determining means.

According to the present invention, where the sphere is rotated by means of a user's operation, the rotation of the sphere is determined based on the image of the sphere taken by the camera, and the three-dimensional object is rotated in conjunction with the determined rotation. Accordingly, the three-dimensional object displayed on the screen can freely be rotated by means of instinctive operation.

The rotation determiner may include: a feature point setter arranged to set at least two feature points in the image of the sphere taken by the camera; feature point tracker arranged to track the feature points set by the feature point setter; and a determiner arranged to determine a rotation of the sphere based on the feature points that are being tracked by the feature point tracker.

The rotation determiner may include: a feature point setter arranged to select at least two feature points from the image of the sphere taken by the camera, and set two-dimensional coordinates in the image of the feature points; a feature point tracker arranged to track the two-dimensional coordinates of the feature points set by the feature point setter; a sampler arranged to obtain the two-dimensional coordinates of the feature points updated by the feature point tracker at predetermined time intervals; a coordinate transformer arranged to transform the two-dimensional coordinates of the feature points obtained by the sampler into three-dimensional coordinates on a surface of the sphere; and rotation matrix calculator arranged to calculate a rotation matrix representing the rotation of the sphere, based on the three-dimensional coordinates obtained as a result of the transformation by the coordinate transformer.

The three-dimensional object display control system may further comprise: a feature point resetter arranged to, where a distance in the image from any one of the feature points that are being tracked by the feature point tracker to a center of the sphere is longer than a first predetermined distance, set one new feature point instead of the one feature point. A distance in the image from the one new feature point to the center of the sphere may be shorter than the first predetermined distance.

In this case, only feature points with a large amount of displacement per unit time are used, and thus, the rotation of the sphere can be determined with high accuracy.

A distance in the image from the one new feature point to the other feature point may be longer than a second predetermined distance.

In this case, the distance between the feature points is long, and thus, the rotation of the sphere can be determined with high accuracy.

A three-dimensional object display control method according to the present invention is a three-dimensional object display control method for rotating a three-dimensional object displayed on a screen, the method comprising the steps of: rotating a sphere; taking an image of the sphere by a camera; determining a rotation of the sphere based on the image of the sphere taken by the camera; and rotating the three-dimensional object in conjunction with the determined rotation of the sphere.

According to the present invention, where the sphere is rotated by means of a user's operation, the rotation of the sphere is determined based on the image of the sphere taken by the camera, and the three-dimensional object is rotated in conjunction with the determined rotation. Accordingly, the three-dimensional object displayed on the screen can freely be rotated by means of instinctive operation.

A three-dimensional object display control program according to the present invention is a three-dimensional object display control program for rotating a three-dimensional object displayed on a screen, the program making a computer execute the steps of: obtaining an image of a sphere taken by a camera; determining a rotation of the sphere based on the image of the sphere taken by the camera; and rotating the three-dimensional object in conjunction with the determined rotation of the sphere.

According to the present invention, where the sphere is rotated by means of a user's operation, the rotation of the sphere is determined based on the image of the sphere taken by the camera, and the three-dimensional object is rotated in conjunction with the determined rotation. Accordingly, the three-dimensional object displayed on the screen can freely be rotated by means of instinctive operation.

DETAILED DESCRIPTION

Hereinafter, an embodiment of the present invention will be described in details with reference to the drawings. In the drawings, parts that are the same or correspond to each other are provided with the same reference numerals and a description thereof will not be repeated.

With reference toFIG. 1, a computer system10using a three-dimensional object display control system according to an embodiment of the present invention includes a computer12, a display14, a keyboard16, a mouse18and a rotation information input device20. The display14, the keyboard16, the mouse18and the rotation information input device20are connected to the computer12. The computer12, the display14, the keyboard16and the mouse18are general-purpose ones.

The computer12includes, e.g., a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory) and a hard disk (though the illustration thereof is omitted). In the computer12, a three-dimensional object display control program (including, e.g., a driver for the rotation information input device20and object tracking software), which will be described later, is installed in addition to an OS (Operating System) and three-dimensional graphics software. For the three-dimensional graphics software, for example, OpenGL (Open Graphics Library) (Intel Open Source Computer Vision Library) (registered trademark; the same applies to the below description) is used. OpenGL is a program interface for creating three-dimensional graphics, which was developed mainly by Silicon Graphics, Inc., U.S. It is characterized in that: high precision three-dimensional graphics can be created at high speed; and it does not depend on the OS. Also, for the object tracking software, for example, OpenCV (trademark; the same applied to the below description) is used. OpenCV is a computer vision library developed and laid open by Intel Corporation, U.S. The RGB values of each pixel can be extracted from data acquired from a camera. Using the acquired data, a certain point in an image can be tracked. Those pieces of software are stored in the hard disk, and executed upon being read by the CPU.

The rotation information input device20includes a sphere22, a tripod24that rotatably supports the sphere22at three ball bearings, and a camera26that takes an image of the sphere22from below. In order to facilitate detection of the sphere22's rotation, multiple relatively-small triangles are drawn on the surface of the sphere22. Upon a user manually operating the sphere22, the sphere22is rotated in a desired direction by a desired amount with its center fixed. The camera26is installed so that its optical axis passes through the center of the sphere22.

Next, an operation of the above-described computer system10will be described.

Upon an image of the sphere22being taken from below by the camera26, an image such as illustrated inFIG. 2is obtained. The sphere22is shown in its maximum size on the screen, and the tips of the tripod24are shown on three sides of the sphere22. The obtained image is sent to the computer12.

FIG. 3illustrates a flowchart for a three-dimensional object display control program. However, drawing of a picture of a three-dimensional object is performed by three-dimensional graphics software such as OpenGL.

With reference toFIG. 3, the computer12captures an incoming image in a predetermined cycle (S1). The computer12determines whether or not any feature points exit in the captured image (S2). At first, no feature points exist (NO in S2), and thus, the computer12, as illustrated inFIG. 2, sets two feature points r1and r2in the image of the sphere22(S3). More specifically, the computer12selects two feature points r1and r2on the image of the sphere, and sets two-dimensional coordinates (x, y) in the image of the feature points r1and r2(S3). The feature points r1and r2are set in positions that can easily be distinguished from the other part (in the present example, vertexes of triangles drawn on the surface of the sphere22).

The two-dimensional coordinates (x, y) are defined as follows. As illustrated inFIG. 2, an x-y coordinate system is introduced into the image, and the upper left corner of the image is set to the origin (0, 0). The camera26is installed so that its optical axis passes through the center of the sphere22, and thus, the coordinates (xc, yc) of the center C of the sphere22correspond to the center of the image.

Meanwhile, where feature points r1and r2exist (YES in S2), the computer12tracks the feature points r1and r2using the function of OpenCV (S4). More specifically, upon the sphere22being rotated by means of a user's operation, the feature points r1and r2are moved, and thus, the computer12updates the two-dimensional coordinates (x, y) of the feature points r1and r2(S4).

While tracking the feature points r1and r2as described above, the computer12samples (acquires) the two-dimensional coordinates (x, y) of the feature points r1and r2at predetermined time intervals (S5and S6).

Next, the computer12transforms the two-dimensional coordinates (x, y) of the feature points r1and r2into three-dimensional coordinates (X, Y, Z) on the surface of the sphere22(S7).

The three-dimensional coordinates (X, Y, Z) are defined as follows. As illustrated inFIG. 4, an X-Y-Z coordinate system is introduced into a real space (three-dimensional space). It is assumed that: the sphere22has been installed in front of the left side of the display14; and the center C of the sphere22is set to the origin (0, 0, 0). It is assumed that: the direction perpendicular to the display screen, in which a user is present, is the X-axis; the horizontal rightward direction relative to the display screen is the Y-axis; and the vertical upward direction is the Z-axis.

When the two-dimensional coordinates (x, y) of the feature points r1and r2are given, their three-dimensional coordinates (X, Y, Z) are calculated as follows. As illustrated inFIG. 5, the distance between the camera26and the center C of the sphere22is set to L; and the radius of the sphere22is set to R. Where coordinates of a point obtained as a result of projecting a feature point (X, Y, Z) on the surface of the sphere22from the camera26's position (0, 0, −L) onto the X-Y plane are (X′, Y′, 0), (X, Y, Z) is represented in expression (1) below.

Since the feature point (X, Y, Z) is on the surface of the sphere22with the radius R, t can be represented by expression (2) below.

As described above, correspondence from (X′, Y′) to (X, Y, Z) can be provided by expressions (1) and (2).

Finally, the correspondence between the coordinates (X, Y) on the X-Y plane and the coordinates (x, y) in the image is calculated. The radius Rc of the image of the sphere22projected on the X-Y plane can be represented by expression (3) below.

Using this radius Rc and the radius rc of the sphere22in the image, the coordinates (X′, Y′) of the feature point on the X-Y plane can be represented by expression (4) below.

Consequently, the correspondence from (x, y) to (X′, Y′) has been calculated, and combining it with the above-described correspondence from (X′, Y′) to (X, Y, Z), the three-dimensional coordinates (X, Y, Z) on the surface of the sphere22can be calculated from the two-dimensional coordinates (x, y) of the feature points r1and r2.

Referring back toFIG. 3, after the above-described coordinate transformation of the feature points (S7), the computer12calculates a rotation matrix representing the rotation of the sphere22, based on the three-dimensional coordinates (X, Y, Z) (S8). The details are provided below.

First, for preparation, where a unit vector l=(lx, ly, lz) is a rotation axis, a matrix R(l, θ)εSO(3) representing a transformation for rotation around the rotation axis by an angle θ can be represented by expression (5) below. Here, SO3 is a special orthogonal group, and is one having a determinant value of 1 from among three-dimensional real orthogonal matrixes.

Where, lTis a transposition of l, and I is an unit matrix, and S is represented by expression (6) below.

Expression (5) is represented by expression (7) below where it is indicated by elements.

In particular, if θ=0 (that is, the rotation angle is zero degrees), R(l, θ)=I regardless of the value of l. Since the center C of the sphere22is fixed, hereinafter, it is assumed that the origin of the coordinate system is the center C of the sphere22.

FIG. 6illustrates movement of two points as a result of rotation R of the sphere22occurring between a time t1and a time t2. The movement from r1to r1′ is represented by r1′=Rr1. The movement from r2to r2′ is represented by r2′=Rr2.

In order to obtain a rotation matrix RεSO(3) representing the rotation of the sphere22occurring between the time t1and the time t2, the movement of the two points on the surface of the sphere22is measured. This is because the rotation matrix RESO(3) can uniquely determined if the positional correspondences of the two points between before and after the rotation can be figured out.

The outline of obtainment of this rotation matrix R is as follows. With reference toFIG. 7, the sphere22is first rotated around an axis11, and then rotated around an axis12, allowing r1to be moved to r1′ and r2to be moved to r2′. Therefore, first, a rotation matrix R1(for the rotation axis11) for superposing the point r1on the point r1′ is obtained. Here, the rotation axis11is determined so that it is perpendicular to a plane including a great circle passing through the point r1and the point r1′. As a result of this rotation, r2is moved to r2′=R1r2. Next, a rotation matrix R2for rotating the sphere22around an axis12passing through the center C of the sphere22and the point r1′ to superpose the point r2′ on the point r2′ is obtained. Finally, upon combining these two rotation matrixes, a rotation matrix R=R2R1can be obtained.

Next, a method for specifically calculating the rotation matrix R will be described. The rotation axis (unit vector) of the first rotation R1can be calculated by expression (8) using an exterior product.

Also, the rotation angle φ1 around the rotation axis11can be calculated by expression (9) below.

Here, in actual calculation, it is assumed that 0≦φ1<π/2 based on the facts that: the interval between the time t1and the time t2is small; and a rotation occurring therebetween is also small. Where r1=r1′,11takes an appropriate unit vector, and it is assumed that φ1=0.

With reference toFIGS. 8(a) and8(b), the rotation axis12(unit vector) of the second rotation R2can be provided by expression (10) below.

The rotation angle φ2 will be calculated as follows. The vector12′ of the intersection point of a plane PL passing through the point r2′ and the point r2′ and perpendicular to the rotation axis12and the rotation axis is represented by expression (11) below.

The rotation angle φ2 around the rotation axis12is represented by expression (12) below.

Here, also using the fact that the rotation is small, it is assumed that 0≦φ2<π/2. Also, where r2′=r2′,12takes an appropriate unit vector, and it is assumed that φ2=0.

As described above, the rotation axis and rotation angle combinations (11, φ1) and (12, φ2) for the two rotations can be obtained. Where the rotation axes and the rotation angles are figured out, the rotation matrixes R1=R(11, φ1) and R2=R(12, φ2) for the respective steps can be calculated by expression (5), and the rotation matrix R can be obtained by expression (13) below.

Referring back toFIG. 3again, after the aforementioned rotation matrix calculation (S8), the computer12rotates the three-dimensional object displayed on the screen of the display14in conjunction with the rotation of the sphere22determined above (S9). More specifically, the computer12multiplies the respective vertex coordinates of the three-dimensional object by the rotation matrix R calculated above. Consequently, the respective vertex coordinates of the three-dimensional object after rotated.

Next, where the distance in the image from one feature point r1or r2, which is being tracked as described above, to the center of the sphere22is longer than a predetermined distance RI, the computer12resets a new feature point instead of the feature point r1or r2(S10). However, the new feature point is reset so that: the distance in the image from the new feature point to the center of the sphere22is shorter than the predetermined distance RI; and the distance in the image from the new feature point to the other feature point r2or r1is longer than a predetermined distance RP.

More specifically, with reference toFIG. 2, a limited circle LC is set in the image of the sphere22. The center of the limited circle LC corresponds to the center C of the sphere22. The radius RI of the limited circle LC is smaller than the radius R of the sphere22, and is preferably approximately half the radius R of the sphere22. When either of the feature points r1and r2that are being tracked departs from the limited circle LC, a new feature point is reset in the limited circle LC instead of the feature point. Where the sphere22is rotated at a constant speed, in the image, the speed of a feature point is high around the center of the sphere22, but becomes lower as the feature point comes closer to the outer edge of the sphere22. Accordingly, where the feature point is continuously tracked until it comes around the outer edge of the sphere22, the accuracy of detecting the rotation of the sphere22is lowered. Therefore, in the present embodiment, the rotation of the sphere22is determined from feature points within the limited circle LC.

Furthermore, when resetting the new feature point, it is preferable to reset it as far as possible from the other feature point. This is because an excessive short distance between the feature points results in lowering the accuracy of detecting the rotation of the sphere22. In the present embodiment, the feature point is reset in a position that is at least the predetermined distance RP away from the other feature point.

FIG. 9illustrates the details of feature point resetting. With reference toFIG. 9, the computer12calculates the distance DIS in the image between the feature point r1and the center C of the sphere22(S101). Next, the computer12determines whether or not the distance DIS is longer than the predetermined distance RI (S102). If the distance DIS is longer than the predetermined distance RI, that is, if the feature point r1departs from the limited circle LC (YES in S102), as in step S3above, the computer12resets a new feature point r1(S103). Next, the computer12calculates the distance DIS0in the image between the new feature point r1and the center C of the sphere22(S104), and calculates the distance DIS1between the feature point r1and the feature point r2(S105). Next, the computer12determines whether or not the distance DIS0is shorter than the predetermined distance RI and the distance DIS1is longer than the predetermined distance RP (S106). If the distance DIS0is longer than the predetermined distance RI, that is, if the new feature point r1resides outside the limited circle LC (NO in S106), or if the distance DIS1is shorter than the predetermined distance RP, that is, if the new feature point r1has a distance to the other feature point r2, which is smaller than the predetermined distance RP (NO in S106), steps S103to S105, which have been described above, are repeated. Consequently, a new feature point r1is reset in a position that is within the limited circle LC and a distance longer than the predetermined distance RP away from the other feature point R2. The same processing as in steps S101to S106, which have been described above, is also performed for the feature point r2.

Referring back toFIG. 3again, the computer12draws a picture of the three-dimensional object rotated at step S9, which has been described above, on the display14(S11).

As described above, according to an embodiment of the present invention, two feature points r1and r2are selected from an image of the sphere22taken by the camera26, two-dimensional coordinates (x, y) in the image of the feature points r1and r2are set, and the feature points are tracked. The two-dimensional coordinates (x, y) of the feature points r1and r2are sampled at predetermined time intervals, and transformed into three-dimensional coordinates (X, Y, Z) on the surface of the sphere22. A rotation matrix R representing the rotation of the sphere22is calculated based on the three-dimensional coordinates (X, Y, Z) of the feature points r1and r2before and after movement of the feature points in a predetermined time period. Accordingly, upon the sphere22being rotated by means of a user's operation, the rotation of the sphere22is determined based on the image of the sphere22taken by the camera26, and the three-dimensional object is rotated in conjunction with the determined rotation. Accordingly, the three-dimensional object displayed on the screen can freely be rotated by means of instinctive operation.

Although in the above embodiment, triangles are drawn on the surface of the sphere22, there are no specific limitations on the figure shape, size, count, etc., and it is only necessary that a pattern allowing the rotation of the sphere22to be easily detected be drawn. Furthermore, nothing may be drawn if the precision of the camera26is so high that the fine ground pattern that is invisible to the naked eye can be detected.

In general, a rotation of a sphere can be determined if movement of at least two points on the surface of the sphere is determined, and thus, in the above embodiment, two feature points are provided; however, three or more feature points may be provided. In such case, although the amount of information to be processed is increased, the calculation accuracy is enhanced.

Also, although in the above embodiment, the rotation information input device20that rotatably supports the sphere22via the tripod24to make the camera26take an image of the sphere22has been used, the sphere22and the camera26may be downsized to be embedded in the mouse18or the keyboard16.

Also, although in the above embodiment, a three-dimensional object displayed on the screen can only be rotated in its position, it may be moved three-dimensionally. For that purpose, for example, detection may be performed not only for rotation of the sphere22, but also for three-dimensional movement of the sphere22.

Although an embodiment of the present invention has been described above, the above-described embodiment is only an example for carrying out the present invention. Accordingly, the present invention is not limited to the above-described embodiment, and the present invention can be carried out by arbitrarily modifying the above-described embodiment as long as such modification does not deviate from the spirit of the present invention.