Estimation device and method

A method includes associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position and an orientation of a camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time, calculating vectors based on feature descriptors of the first feature points and feature descriptors of the second feature points for respective pairs of the first feature points and the second feature points, determining whether an association of the first feature points and the second feature points is satisfactory, and performing an estimation process that estimates a new position and a new orientation of the camera at the second time when the association is determined to be satisfactory.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2015-231989, filed on Nov. 27, 2015, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a technique of estimating a camera position and orientation.

BACKGROUND

In recent years, there has been a technique that estimates a position and orientation of a camera equipped in a personal computer (PC), a mobile terminal, a wearable terminal, or the like and, by using an estimation result, superimposes additional information onto an image displayed on a screen of the camera to support user operations.

During a user operation, since a camera position and orientation may frequently change, estimation of the camera position and orientation is likely to be temporarily unsuccessful. It is therefore desirable that a process of estimating a camera position and orientation be resumed from a state of unsuccessful estimation. In the following description, a process of resuming estimation of a camera position and orientation from a state of unsuccessful estimation may be referred to as a relocalization process.

An example of the related art for a relocalization process will be described. In the related art, keyframes each associating an image at the time of successful estimation with a camera position and orientation are generated intermittently at timings when a camera position and orientation are successfully estimated. In the related art, in attempting a relocalization process, the current camera position and orientation are estimated by detecting a keyframe that is close to a current image taken by the camera and associating a feature point of the detected keyframe with a feature point of the current image taken by the camera.

There is another related art that selects a set of feature points having a smaller error from a plurality of sets of feature points associating a keyframe with the current image and uses the selected set of feature points to perform a relocalization process.

Related arts are disclosed in U.S. Pat. No. 8,836,799 B2 and Japanese Laid-open Patent Publication No. 2012-221042, for example.

SUMMARY

According to an aspect of the invention, a method includes associating a plurality of first feature points with a plurality of second feature points, the plurality of first feature points being included in a keyframe that represents a first captured image at a first time when a position of a camera and an orientation of the camera are successfully estimated, the plurality of second feature points being included in a second captured image captured at a second time by the camera, calculating a plurality of vectors based on feature descriptors of the plurality of first feature points and feature descriptors of the plurality of second feature points for respective pairs of the plurality of first feature points and the plurality of second feature points, based on a distribution of the vectors, determining whether or not an association of the plurality of first feature points and the plurality of second feature points is satisfactory, and performing an estimation process that estimates a new position of the camera and a new orientation of the camera at the second time based on pairs of the plurality of first feature points and the plurality of second feature points when the association is determined to be satisfactory.

DESCRIPTION OF EMBODIMENTS

The related art has a problem in that a relocalization process is performed even when an association of feature points by using a keyframe is incorrect.

For example, since an incorrect association of feature points causes degeneration of estimation accuracy of a camera position and orientation, the position at which additional information is superimposed on an image taken by the camera may shift from an appropriate position. Further, in the related art that selects a set of feature points having a smaller error from a plurality of sets of feature points, an incorrect association of feature points will result in an increase in the number of attempts of a process for selecting a set of feature points having a smaller error, which may cause a problem of an increase in a processing load.

In one aspect, a technique disclosed in the embodiments has the goal of performing a relocalization process based on a satisfactory association of feature points.

The embodiments of a camera position and orientation estimation device, a camera position and orientation estimation method, and a camera position and orientation estimation program disclosed in the present application will be described below in detail with reference to the drawings. Note that the disclosure is not limited by these embodiments.

First Embodiment

FIG. 1is a functional block diagram illustrating the configuration of a camera position and orientation estimation device according to the present embodiment. As illustrated inFIG. 1, a camera position and orientation estimation device100is connected to a camera50. The camera position and orientation estimation device100has an image acquisition unit110, a feature point extraction unit120, and a camera position and orientation estimation unit130. The camera position and orientation estimation device100has a quality determination unit140, a keyframe detection unit150, an associating unit160, a vector calculation unit170, a determination unit180, a control unit190, and a relocalization process unit200. The vector calculation unit170is an example of an image association vector calculation unit.

The camera50is a single-lens red, green, and blue (RGB) camera equipped in a PC, a mobile terminal, a wearable terminal, or the like. The camera50captures an image from any point of view and outputs a captured image to the image acquisition unit110.

The image acquisition unit110is a processing unit that is connected to the camera50and acquires a captured image from the camera50. The image acquisition unit110outputs a captured image to the feature point extraction unit120.

The feature point extraction unit120is a processing unit that extracts feature points from a captured image. For example, the feature point extraction unit120extracts feature points by executing SIFT, SURF, ORB, or the like. For example, a feature point may be a point corresponding to an edge or the like of a captured image. The feature point extraction unit120outputs information of feature points to the camera position and orientation estimation unit130. The information of feature points includes two-dimensional coordinates and/or a feature descriptor for each feature point extracted from a captured image, for example. A feature descriptor is information such as a pixel value, a slope of an edge, or the like around a feature point.

The camera position and orientation estimation unit130is a processing unit that matches a feature point with a map point based on information of a feature point acquired from the feature point extraction unit120and a three-dimensional map and estimates the position and orientation of the camera50based on a matching result. As used herein, a three-dimensional map represents map points of three-dimensional coordinates that have been predefined with respect to an object.

FIG. 2is a diagram illustrating an example of the data structure of a three-dimensional map. As illustrated inFIG. 2, the three-dimensional map associates map point IDs, values X, Y, and Z, and feature descriptors. InFIG. 2, a map point ID is a number that uniquely identifies a map point. In the example illustrated inFIG. 2, the three-dimensional map includes 68 map points. Values X, Y, and Z represent three-dimensional coordinates of a map point. A feature descriptor represents a feature descriptor of a projected point on an image when a map point is projected on the image captured by a camera50. For example, a feature descriptor is information such as a pixel value, a slope of an edge, or the like around a projected point.

An example of a process in which the camera position and orientation estimation unit130matches a map point with a feature point will be described. The camera position and orientation estimation unit130uses a previously estimated position and orientation of the camera50to determine which map point of a three-dimensional map a feature point corresponds to. The camera position and orientation estimation unit130calculates coordinates at which a map point is projected on a captured image for the previous time by using Equation (1). In the following description, a map point projected on a captured image may be referred to as a projection point.

In Equation (1), the term A is a matrix with three rows by three columns and is associated with an internal parameter of the camera50. A user calibrates the camera50in advance based on a disclosure of the reference “Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334 (2000).”

In Equation (1), the term (RR) is a matrix of three rows by four columns obtained by transforming a previously estimated position and orientation of the camera50into a rotation matrix R of three rows by three columns and a translation vector t by using Rodrigues' Rotation Formula expressed by Equation (2) and combining the rotation matrix R and the translation vector t. Coordinates (u, v) are two-dimensional coordinates of a projection point when a map point is projected on a captured image. Coordinates (X, Y, Z) are three-dimensional coordinates of each map point.

In Equation (2), the term I represents a unit matrix. The term θ is defined by Equation (3). The term r is defined by Equation (4).

FIG. 3is a diagram for illustrating a process of the camera position and orientation estimation unit130. Let us assume that there are map points S1to S6inFIG. 3. Further, it is assumed that there are feature points x1to x6in a captured image20. The camera position and orientation estimation unit130projects the map points S1to S6on the captured image20based on Equation (1) to obtain projection points x1′ to x6′.

The camera position and orientation estimation unit130calculates respective distances from projection points to feature points on the current captured image located within a certain threshold range. The camera position and orientation estimation unit130determines a projection point and a feature point having the smallest distance and pairs a map point corresponding to the determined projection point and the feature point. The camera position and orientation estimation unit130repeats the above process for all the projection points and matches a map point corresponding to a projection point with a feature point on the current captured image. The threshold may be 20 pixels, for example.

For example, in the example depicted inFIG. 3, since the distance between the feature point x1and the projection point x1′ is the shortest, the feature point x1and the map point S1corresponding to the projection point x1′ are paired. In a similar manner, the feature points x2to x6are paired with the map points S2to S6, respectively.

After performing matching, the camera position and orientation estimation unit130performs a process of estimating the position and orientation of the camera50. When there are three or more pairs of feature points and map points, the camera position and orientation estimation unit130estimates a camera position and orientation by using a PnP algorithm. For example, the camera position and orientation estimation unit130may estimate a position and orientation by using a PnP algorithm described in the reference “V. Lepetit et al., EPnP: An Accurate O(n) Solution to the PnP Problem, International Journal of Computer Vision, Vol. 81, Issue 2, pp. 155-166 (2008).”

The camera position and orientation estimation unit130outputs information of the estimated camera position and orientation to the quality determination unit140. Further, the camera position and orientation estimation unit130outputs to the quality determination unit140the total number of map points included in the three-dimensional map and the number of matched pairs.

The quality determination unit140is a processing unit that determines the quality of a position and orientation of the camera50estimated by the camera position and orientation estimation unit130. For example, the quality determination unit140calculates a ratio of the number of matched pairs to the total number of map points included in the three-dimensional map. When the calculated ratio is greater than or equal to a predetermined ratio, the quality determination unit140determines that an estimate value of the position and orientation of the camera50is of high quality. On the other hand, when the calculated ratio is less than the predetermined ratio, the quality determination unit140determines that an estimate value of a position and orientation of the camera50is of low quality. For example, the predetermined ratio may be 30%.

When having determined that an estimation value of a position and orientation of the camera50is of high quality, the quality determination unit140concludes that the current position and orientation of the camera50is a position and orientation estimated by the camera position and orientation estimation unit130and stores the current position and orientation of the camera50in a predetermined storage unit. An HMD display control unit (not illustrated) looks up a camera position and orientation stored in the storage unit and utilizes the position and orientation to cause a display of the HMD to display an image where additional information is superimposed on an image captured by the camera50.

When having determined that an estimation value of a position and orientation of the camera50is of low quality, the quality determination unit140outputs a request for a relocalization process to the keyframe detection unit150.

The keyframe detection unit150is a processing unit that, when having acquired a request for a relocalization process from the quality determination unit140, compares the current captured image to a keyframe table and detects a reference keyframe. The keyframe detection unit150outputs information of a reference keyframe to the associating unit160.

FIG. 4is a diagram illustrating an example of the data structure of a keyframe table. As illustrated inFIG. 4, this keyframe table associates numbers, values of position and orientation, captured images, feature point groups, and map point ID groups. A number inFIG. 4represents a number that uniquely identifies a keyframe. A value of position and orientation represents a position and orientation of a camera. A captured image is image data captured by the camera at the time when a position and orientation of the camera is successfully estimated. A feature point group represents two-dimensional coordinates of respective feature points included in a keyframe. A map point ID group represents information that uniquely identifies respective map points associated with respective feature points.

For example, an association of feature points and map points is indicated in the order of coordinates in a feature point group and in the order in a map point ID group ofFIG. 4. For example, the coordinates of a feature point associated with a map point ID “3” are “11, 42”. In the example illustrated inFIG. 4, a keyframe table includes 25 keyframes.

A position and orientation is expressed in six dimensions (r1, r2, r3, t1, t2, t3). In these six values, values (r1, r2, r3) represent a camera orientation in the global coordinate system. Values (t1, t2, t3) represent a camera position in the global coordinate system.

An exemplary process in which the keyframe detection unit150detects a keyframe having a captured image that is the closest to the current captured image will now be described. The keyframe detection unit150reduces the current captured image and a captured image of each keyframe to a predetermined size, calculates a Sum of Squared Distance (SSD) for each pixel while blurring the reduced images by using a Gaussian filter, and determines a keyframe having the smallest SSD value as a reference keyframe. The keyframe detection unit150outputs information of the current captured image and information of the reference keyframe to the associating unit160.

The associating unit160is a processing unit that associates a plurality of feature points included in a reference keyframe with a plurality of feature points included in the current captured image. In the following description, a feature point included in a reference keyframe may be referred to as a first feature point, and a feature point included in the current captured image may be referred to as a second feature point.

The associating unit160compares a feature descriptor of each first feature point to a feature descriptor of each second feature point and associates and pairs the first feature point and the second feature point which have the highest similarity in the feature descriptor. For example, the associating unit160calculates a comparison value based on Equation (5), where the feature descriptor of a first feature point is fb and the feature descriptor of a second feature point is fc, and determines that a smaller feature descriptor indicates a greater similarity.
Comparison value=|fb−fc|(5)

The associating unit160repeats a process for determining one of the unassociated second feature points which has the highest similarity to an unassociated first feature point and pairs these first and second feature points to generate a plurality of pairs of the first feature point and the second feature point. The associating unit160outputs information of pairs of the associated first feature points and second feature points to the vector calculation unit170.

The vector calculation unit170is a processing unit that calculates an image associated vector that is based on a feature descriptor of a first feature point and a feature descriptor of a second feature point for each pair of a first feature point and a second feature point associated by the associating unit160. The vector calculation unit170outputs information of the calculated image associated vector to the determination unit180. In the following description, an image associated vector is simply referred to as a vector.

For example, the vector calculation unit170calculates a vector v of the associated first feature point and second feature point based on Equation (6), where coordinates in an image of a first feature point are (ub, vb) and coordinates in an image of a second feature point are (uc, vc). The vector calculation unit170calculates the vector v for each pair of the associated first feature point and second feature point.
Vectorv=(uc−ub,vc−vb)  (6)

The determination unit180is a processing unit that determines, based on a distribution of a plurality of vectors calculated by the vector calculation unit170, whether or not an association of first feature points and second feature points is satisfactory. The determination unit180outputs a determination result to the control unit190.

The determination unit180calculates an average “a” of the lengths of the vectors and a standard deviation σ of the lengths of the vectors, and determines whether or not the average “a” and the standard deviation σ satisfy condition 1 and condition 2. A reference value A and a reference value Σ have been set in advance by a user.

Condition 1: average “a”<reference value A

For example, when both condition 1 and condition 2 are satisfied, the determination unit180determines that an association of a first feature point and a second feature point is satisfactory. On the other hand, when the condition 1 is not satisfied or the condition 2 is not satisfied, the determination unit180determines that an association of a first feature point and a second feature point is not satisfactory.

FIG. 5is a diagram illustrating an example of a distribution of vectors. InFIG. 5, the horizontal axis is an axis corresponding to a component (the first component (uc−ub)) in the x-axis direction of vectors, and the vertical axis is an axis corresponding to a component (the second component (vc−vb)) in the y-axis direction of vectors. InFIG. 5, each circle mark represents a position corresponding to each vector where it is determined that an association of a first feature point and a second feature point is satisfactory. Each rhombus mark represents a position corresponding to each vector where it is determined that an association of a first feature point and a second feature point is not satisfactory.

As illustrated inFIG. 5, when an association of first feature points and second feature points is satisfactory, vectors are distributed near the origin (0, 0). In contrast, when an association of first feature points and second feature points is not satisfactory, vectors are not distributed near the origin (0, 0) and are widely spread.

Referring back toFIG. 1, when it is determined by the determination unit180that an association of first feature points and second feature points is not satisfactory, the control unit190does not perform a process of estimating a camera position and orientation. On the other hand, when it is determined by the determination unit180that an association of first feature points and second feature points is satisfactory, the control unit190requests a relocalization process from the relocalization process unit200.

Further, when it is determined by the determination unit180that an association of first feature points and second feature points is not satisfactory, the control unit190may cause a display device (not illustrated) to display information indicative of being unable to perform estimation of a camera position and orientation.

The relocalization process unit200is a processing unit that performs a relocalization process upon receiving a request for a relocalization process from the determination unit180. For example, the relocalization process unit200estimates a position and orientation of the camera50based on an image-to-map method. The image-to-map method is for estimating a camera position and orientation by combining processes of the associating unit160and the camera position and orientation estimation unit130. The relocalization process unit200stores estimation results of a camera position and orientation in a predetermined storage unit.

The relocalization process unit200may utilize a relationship between a feature point of a reference keyframe associated by the associating unit160and a feature point of the current captured image. As illustrated inFIG. 4, a feature point of a keyframe is associated with a map point ID. Therefore, a relationship between a feature point of the current captured image and a map point ID can be obtained from a relationship between a feature point of a reference keyframe and a feature point of the current captured image.

FIG. 6is a diagram illustrating an example of the data structure of a three-dimensional map associated with feature points on a captured image. InFIG. 6, map point IDs, values X, Y, and Z, and feature descriptors are the same as those described forFIG. 2. Values x and y indicate an x coordinate and a y coordinate of each feature point on a captured image associated with a map point.

When there are three or more pairs of a feature point and a map point, the relocalization process unit200estimates a camera position and orientation by using the PnP algorithm.

Next, steps of a process of the camera position and orientation estimation device100according to the present embodiment will be described.FIG. 7is a flowchart illustrating steps of a process of the camera position and orientation estimation device100according to the present embodiment. As illustrated inFIG. 7, the image acquisition unit110of the camera position and orientation estimation device100acquires captured images (step S101). The feature point extraction unit120of the camera position and orientation estimation device100extracts feature points from captured images (step S102).

The camera position and orientation estimation unit130of the camera position and orientation estimation device100matches map points with feature points (step S103) and estimates a camera position and orientation (step S104). The quality determination unit140of the camera position and orientation estimation device100determines the quality of an estimation value of a position and orientation (step S105).

The quality determination unit140determines whether or not an estimation value is of high quality (step S106). If the estimation value is of high quality (step S106, Yes), the quality determination unit140completes the process.

On the other hand, if the estimation value is not determined as high quality by the quality determination unit140(step S106, No), the keyframe detection unit150of the camera position and orientation estimation device100detects a reference keyframe (step S107).

The associating unit160of the camera position and orientation estimation device100associates first feature points of a reference keyframe with second feature points of the current captured image (step S108). The vector calculation unit170of the camera position and orientation estimation device100calculates vectors between respective associated points (step S109).

The determination unit180of the camera position and orientation estimation device100determines based on a distribution of a plurality of vectors whether or not an association of first feature points and second feature points is satisfactory (step S110).

If an association of first feature points and second feature points is not satisfactory, the determination unit180determines that a relocalization process is unable to be performed (step S111, No), and completes the process without performing a relocalization process. That is, if it is determined that an association of first feature points and second feature points is not satisfactory, the control unit190of the camera position and orientation estimation device100controls the process so that no relocalization process is performed by the relocalization process unit200.

On the other hand, if an association of first feature points and second feature points is satisfactory, the determination unit180determines that a relocalization process is able to be performed (step S111, Yes). The relocalization process unit200of the camera position and orientation estimation device100performs a relocalization process (step S112).

Next, advantages of the camera position and orientation estimation device100according to the present embodiment will be described. When the quality of a camera position and orientation is low and a relocalization process is thus performed, the camera position and orientation estimation device100associates first feature points of a reference keyframe and second feature points of a captured image and, based on a distribution of feature points, determines whether or not an association is satisfactory. The camera position and orientation estimation device100does not perform a relocalization process when the association is determined to be not satisfactory. This can inhibit a relocalization process under an incorrect association of feature points and allows a relocalization process based on a satisfactory relationship between feature points to be performed.

Further, being able to perform a relocalization process based on a satisfactory relationship between feature points can improve the accuracy of camera position and orientation estimation by a relocalization process. Further, being able to perform a relocalization process based on a satisfactory relationship between feature points can reduce the number of attempts for a process of selecting a pair of feature points having a smaller error and therefore reduce a processing load.

The camera position and orientation estimation device100determines that an association of first feature points and second feature points is not satisfactory when an average of the lengths of the vectors is greater than or equal to a reference average and a standard deviation of the lengths of the vectors is greater than or equal to a reference deviation. As illustrated inFIG. 5, vectors are distributed near the origin (0, 0) when an association of first feature points and second feature points is satisfactory. In contrast, there is a tendency that the vectors are widely spread out without being distributed near the origin (0, 0) when an association of first feature points and second feature points is not satisfactory. Therefore, the determination described above allows for accurate determination as to whether or not an association of first feature points and second feature points is satisfactory.

The above process of the camera position and orientation estimation device100is a mere example, and the camera position and orientation estimation device100may perform other processes. Other processes (1) and (2) of the camera position and orientation estimation device100will be described below.

Another process (1) of the camera position and orientation estimation device100will be described. The determination unit180of the camera position and orientation estimation device100calculates a median vm of a plurality of vectors v calculated by the vector calculation unit170. The determination unit180calculates a distance between a vector v and a median vm for each vector v based on Equation (7).
Distance=|v−vm|(7)

When, among all the vectors v, a percentage of vectors having a distance smaller than a threshold Dt is greater than or equal to r %, the determination unit180determines that an association of first feature points and second feature points is satisfactory. On the other hand, when the percentage of vectors having a distance smaller than a threshold Dt among all the vectors v is less than r %, the determination unit180determines that an association of first feature points and second feature points is not satisfactory.

FIG. 8andFIG. 9are diagrams for illustrating other processes of the determination unit180. The horizontal axis of each histogram illustrated inFIG. 8andFIG. 9represents the distance, the vertical axis represents the number of pairs of a first feature point and a second feature point. The number of pairs is the same as the number of vectors v. As an example, the threshold Dt is 40.

FIG. 8illustrates a histogram when a ratio of pairs having a distance smaller than a threshold Dt among all the pairs is greater than or equal to r %. Such a histogram indicates that the association of first feature points and second feature points is satisfactory. When a ratio of pairs having a distance smaller than a threshold Dt among all the pairs is greater than or equal to r %, vectors are distributed near the origin (0, 0), as illustrated for circle marks ofFIG. 5.

FIG. 9illustrates a histogram when a ratio of pairs having a distance smaller than a threshold Dt among all the pairs is less than r %. Such a histogram indicates that an association of first feature points and second feature points is not satisfactory. When a ratio of pairs having a distance smaller than a threshold Dt among all the pairs is less than r %, vectors are widely spread out without being distributed near the origin (0, 0), as illustrated by rhombus marks inFIG. 5.

As described above, the determination unit180calculates a distance between a vector v and a median vm for each vector v and determines that an association is not satisfactory when vectors whose calculated distance is greater than or equal to a threshold occupy a predetermined percentage or more of all the vectors. This can inhibit a relocalization process in the case of an incorrect association.

Another process (2) of the camera position and orientation estimation device100will be described. When having accepted a request for a relocalization process from the control unit190, the relocalization process unit200of the camera position and orientation estimation device100estimates a position and orientation of the camera50based on the image-to-map method.

In estimating a position and orientation of the camera50based on the image-to-map method, the relocalization process unit200performs a relocalization process by using one or more pairs of a first feature point and a second feature point of all the pairs associated by the associating unit160in which the one or more pairs have a distance smaller than a threshold. The distance is a value calculated by Equation (7) described above.

FIG. 10is a diagram for illustrating the above processes of the relocalization process unit200. The horizontal axis of a histogram illustrated inFIG. 10represents the distance, and the vertical axis represents the number of pairs of a first feature point and a second feature point. The number of pairs is the same as the number of vectors v. The relocalization process unit200utilizes one or more pairs having a distance smaller than a threshold Dt to estimate a position and orientation of the camera50based on the image-to-map method. That is, the relocalization process unit200performs a relocalization process without using pairs whose relationship of the distance between a vector and a median is included in an area40. This can inhibit a relocalization process from being performed in the case of an incorrect association.

Note that, although description has been provided for the case where the camera position and orientation estimation unit130and the relocalization process unit200of the present embodiment estimate a position and orientation of the camera50based on the image-to-map method as an example, the embodiments are not limited thereto. For example, the camera position and orientation estimation unit130and the relocalization process unit200may use other well-known techniques such as the image-to-image method or the like to estimate a position and orientation of a camera and/or perform a relocalization process.

Next, description will be provided for an example of a computer that executes a camera position and orientation estimation program for implementing the same functions as the camera position and orientation estimation device100illustrated in the above embodiments.FIG. 11is a diagram illustrating an example of a computer that executes a camera position and orientation estimation program.

As illustrated inFIG. 11, a computer300has a CPU301that executes various operations, an input device302that accepts data inputs from the user, and a display303. Further, the computer300has a reading device304that reads a program or the like from a storage medium, an interface device305that communicates data with other computers via a network, and a camera306. Further, the computer300has a RAM307that temporarily stores various information and a hard disk device308. Each of the devices301to308is connected to a bus309.

The hard disk device308has an associating program308a, a vector calculation program308b, a determination program308c, and a control program308d. The CPU301reads the associating program308a, the vector calculation program308b, the determination program308c, and the control program308dand expands them to the RAM307.

The associating program308afunctions as an associating process307a. The vector calculation program308bfunctions as a vector calculation process307b. The determination program308cfunctions as a determination process307c. The control program308dfunctions as a control process307d.

A process of the associating process307acorresponds to a process of the associating unit160. A process of the vector calculation process307bcorresponds to a process of the vector calculation unit170. A process of the determination process307ccorresponds to a process of the determination unit180. A process of the control process307dcorresponds to a process of the control unit190.

Note that the associating program308a, the vector calculation program308b, the determination program308c, and the control program308dmay not necessarily be stored in advance in the hard disk device308. For example, each of such programs may be stored in a “portable physical medium” such as a floppy disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, an IC card, or the like that can be inserted in the computer300. The computer300may then read and execute each of the programs308ato308d.