Patent Description:
A technique that installs a plurality of cameras (image capturing apparatuses) respectively at different positions, performs synchronous capture from multiple viewpoints, and generates a virtual viewpoint image by using a plurality of viewpoint images obtained by the capture has attracted attention. According to the technique for generating the virtual viewpoint image from the plurality of viewpoint images, for example, since a highlight scene of soccer or basketball can be seen from various angles, it enables to give a user high realistic sensation as compared with a normal image.

PTL <NUM> discloses that a plurality of cameras are arranged so as to surround a subject, the subject is captured by the cameras, and an arbitrary virtual viewpoint image is generated and displayed using captured images obtained by the cameras. In PTL <NUM>, the world coordinates X, Y and Z axes are determined with the center point of a stadium as the origin, and the plurality of cameras are installed toward the origin such that the origin is at the center of a screen. <CIT> discusses an information processing apparatus for a system generates a virtual viewpoint image based on image data obtained by performing imaging from a plurality of directions using a plurality of cameras. The information processing apparatus includes an obtaining unit configured to obtain a foreground image based on an object region including a predetermined object in a captured image for generating a virtual viewpoint image and a background image based on a region different from the object region in the captured image, wherein the obtained foreground image and the obtained background image having different frame rates, and an output unit configured to output the foreground image and the background image which are obtained by the obtaining unit and which are associated with each other.

<NPL>), the generation of a virtual viewpoint image from several camera views in a soccer stadium, via the 3D reconstruction of billboard models of the soccer players.

In the technique described in PTL <NUM>, in a case where an entire landscape is generated with the center of the stadium as a gaze point, the number of cameras that are capturing a player who is away from the gaze point is small, and moreover the relevant player is not in focus as compared with near the gaze point. As a result, for example, in case of generating an entire landscape obtained by seeing the other side stand from a side stand far from the viewpoint, the resolution of a this-side player who should be clearly visible because the distance to him/her is short is lower and thus his/her outline blurs, thereby decreasing accuracy of a 3D (three-dimensional) model. For this reason, the image quality of the this-side player is deteriorated, whereas the image quality of a player near the center farther from the this-side player is improved, so that a sense of perspective is impaired and thus realistic sensation is lowered.

The present invention has been completed in view of such a circumstance, and an object thereof is to enable, in an image processing system having a plurality of image capturing units, generation of a wide-area image from a high-quality virtual viewpoint.

According to a first aspect of the present invention a generating apparatus is provided as set out in claim <NUM>. According to a second aspect of the present invention a generating apparatus is provided as set out in claim <NUM>. According to a third aspect of the present invention a generating method is provided as set out in claim <NUM>. According to a fourth aspect of the present invention a generating method is provided as set out in claim <NUM>. According to a fifth aspect of the present invention a program is provided as set out in claim <NUM>.

According to the present invention, it is possible to generate, in an image processing system having a plurality of image capturing units, a wide-area image from a high-quality virtual viewpoint.

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

A first embodiment of the present invention will be described.

<FIG> is a diagram illustrating a constitution example of an image processing system <NUM> according to the first embodiment. The image processing system <NUM> according to the present embodiment is an image processing system that performs image capture and sound collection by installing a plurality of cameras (image capturing apparatuses) and microphones (sound collecting apparatuses) in facilities such as a stadium (field), a concert hall and the like.

The image processing system <NUM> has sensor groups <NUM>, <NUM> and <NUM> for respective gaze points. Each of the sensor groups <NUM>, <NUM> and <NUM> has a plurality of sensor systems. In the present embodiment, the sensor group <NUM> has <NUM> sensor systems 2L01 to 2L30, the sensor group <NUM> has <NUM> sensor systems 2C01 to 2C30, and the sensor group <NUM> has <NUM> sensor systems 2R01 to 2R30. Besides, each sensor system has a microphone <NUM>, a camera <NUM>, a camera platform <NUM>, and a camera adapter <NUM>. That is, each of the sensor groups <NUM>, <NUM> and <NUM> has the plurality of cameras for capturing a subject respectively from a plurality of directions.

As exemplarily illustrated in <FIG>, in each of the sensor systems 2L01 to 2L30 of the sensor group <NUM>, the camera <NUM> is installed for a gaze point <NUM> as the gaze point. Similarly, in each of the sensor systems 2C01 to 2C30 of the sensor group <NUM>, the camera <NUM> is installed for a gaze point <NUM> as the gaze point. Moreover, in each of the sensor systems 2R01 to 2R30 of the sensor group <NUM>, the camera <NUM> is installed for a gaze point <NUM> as the gaze point. The cameras <NUM> of the sensor systems 2L01 to 2L30 corresponding to the gaze point <NUM> capture the range of an area <NUM>, the cameras <NUM> of the sensor systems 2C01 to 2C30 capture the range of an area <NUM>, and the cameras <NUM> of the sensor systems 2R01 to 2R30 capture the range of an area <NUM>. Here, although an example in which the <NUM> sensor systems constitute one sensor group is shown, the number and arrangement of the sensor systems are not limited to this.

An operation of the sensor group <NUM> will be described.

The image processing system <NUM> has a control station <NUM> and a virtual camera operation UI (user interface) <NUM> in order to perform control according to a user's instruction or the like. The control station <NUM> performs management of an operation state, parameter setting/control, and the like for each function unit (block) provided in the image processing system <NUM>, via a network. An operation of transmitting the images and sounds obtained by the <NUM> sensor systems 2L01 to 2L30 from the sensor system 2L30 to a server front end <NUM> via a switching hub <NUM> will be described. Here, the sensor systems 2L01 to 2L30 are connected in a daisy chain via networks 180a, <NUM> and 180b.

Each of the sensor systems 2L01 to 2L29 inputs an image captured by the camera <NUM> to the camera adapter <NUM>, gives a camera identifier for identifying the camera to the captured image, and transmits the obtained image to the network <NUM>. The sensor system 2L30 inputs an image captured by the camera <NUM> to the camera adapter <NUM>, and gives a camera identifier for identifying the camera to the captured image. Then, the sensor system 2L30 transmits the image captured by each camera <NUM> of the sensor group <NUM> to the network 180b. The image transmitted to the network 180b is input to the server front end <NUM> via the switching hub <NUM> and a network 211a.

Incidentally, in the present embodiment, when there is no specific explanation, the <NUM> sets of the sensor systems from the sensor system 2L01 to the sensor system 2L30 are described as a sensor system <NUM> without being distinguished. Similarly, the devices in each sensor system <NUM> are respectively referred to as the microphone <NUM>, the camera <NUM>, the camera platform <NUM> and the camera adapter <NUM> without being distinguished, unless otherwise explained. In the present embodiment, the term "image" will be described as including a concept of a moving image and a concept of a still image unless otherwise specified. That is, the image processing system <NUM> according to the present embodiment can process both the still image and the moving image.

Incidentally, although the example in which the plurality of sensor systems <NUM> are cascade-connected so as to be the daisy chain is described, the present invention is not limited to this. For example, it may be a star-type network configuration in which each of the sensor systems 2L01 to 2L30 is connected to the switching hub <NUM> and data exchange is performed among the sensor systems <NUM> via the switching hub <NUM>. Besides, for example, the plurality of sensor systems <NUM> may be divided into several groups, and the sensor systems <NUM> may be daisy-chained for each of the divided groups. Of course, in a case where there is one sensor system <NUM> in the group, a star-type connection may be used.

Besides, the sensor system <NUM> is not limited to the above-described constitution. For example, the camera <NUM> and the camera adapter <NUM> may be integrally constituted in the sensor system. In this case, the microphone <NUM> may be built in the integrated camera <NUM> or may be connected to the outside of the camera <NUM>. Besides, the server front end <NUM> may have at least a part of the functions of the camera adapter <NUM>. The sensor systems 2L01 to 2L30 are not limited to have the same constitution, and may have different constitutions.

Besides, in the present embodiment, a Model Based Rendering (hereinafter abbreviated as MBR) for constituting a 3D model will be described as a method for generating a virtual viewpoint image. However, the present invention is not limited to this.

Besides, in the present embodiment, although an example in which virtual viewpoint contents provided by the image processing system <NUM> include the virtual viewpoint image and a virtual viewpoint sound will be described, the present invention is not limited to this. For example, a sound may not be included in the virtual viewpoint contents. Moreover, for example, the sound included in the virtual viewpoint contents may be the sound collected by the microphone <NUM> of the sensor system <NUM> installed at the position closest to the virtual viewpoint. Moreover, in the present embodiment, for the sake of simplicity of explanation, the description concerning the sound is partially omitted, but it is basically assumed that both the image and the sound are processed together.

That is, the sound collected by the microphone <NUM> of the sensor system 2L01 and the image captured by the camera <NUM> are subjected to an image process by the camera adapter <NUM>, and then the processed sound and image are transmitted to the camera adapter <NUM> of the sensor system 2L02 via the network <NUM>. Similarly, the sensor system 2L02 combines the collected sound, the captured image and the image and sound data obtained from the sensor system 2L01 together, and transmits the obtained data to the sensor system 2L03 via the network <NUM>. By continuing the above operation, the images and sounds obtained by the sensor systems 2L01 to 2L30 are transmitted from the sensor system 2L30 to the server front end <NUM> via the networks 180b and 211a and the switching hub <NUM>.

The same is applied to the sensor groups <NUM> and <NUM>. The sensor systems 2C01 to 2C30 are daisy-chained via networks 180c, <NUM> and 180d, and the sensor systems 2R01 to 2R30 are daisy-chained via networks 180e, <NUM> and 180f. Images and sounds respectively obtained by the sensor systems 2C01 to 2C30 are transmitted from the sensor system 2C30 to a server front end <NUM> via the networks 180d and 211b and the switching hub <NUM>. Besides, images and sounds respectively obtained by the sensor systems 2R01 to 2R30 are transmitted from the sensor system 2R30 to a server front end <NUM> via the networks 180f and 211c and the switching hub <NUM>.

Although <FIG> illustrates the constitution in which all the insides of the sensor groups <NUM>, <NUM> and <NUM> are cascade-connected so as to form the daisy chains, the present invention is not limited to this. For example, in order to form the daisy chains, the sensor system 2L30 of the sensor group <NUM> and the sensor system 2C01 of the sensor group <NUM> may be connected, and the sensor system 2C30 of the sensor group <NUM> and the sensor system 2R01 of the sensor group <NUM> may be connected.

A time server <NUM> has a function of distributing time and a synchronization signal, and distributes the time and the synchronization signal to the sensor systems 2L01 to 2L30, 2C01 to 2C30, and 2R01 to 2R30 via the switching hub <NUM>, respectively. The camera adapter <NUM> of the sensor system that has received the time and the synchronization signal performs image frame synchronization by externally synchronizing (Genlock) the camera <NUM> based on the time and the synchronization signal. That is, the time server <NUM> synchronizes the capturing timings of the plurality of cameras <NUM>.

Next, the constitution and operation of the server front end <NUM> to which the images and sounds respectively obtained by the sensor systems 2L01 to 2L30 of the sensor group <NUM> are transmitted will be described. The server front end <NUM> reconstructs a segmented transmission packet from the image and sound obtained from the sensor system 2L30, and converts a data format of frame data. Moreover, the server front end <NUM> cuts out a target such as a player or the like (hereinafter also referred to as "object") from the reconstructed frame data, and generates a 3D model of the object from the images of all the cameras using the cut-out result as a foreground image.

There are various methods for generating a 3D model. It is possible to use, e.g., a method such as Visual Hull (visual cone intersection method). For example, the generated 3D model is expressed as a point group. For example, the point group represents the number of points existing in the 3D model, and each point of the point group can be represented by an x-coordinate, a y-coordinate and a z-coordinate in a coordinate system using the gaze point as the origin. However, the present invention is not limited to this. Namely, it may be possible to divide a space into voxels with the gaze point as the origin, binarize the voxel in which the object exists to "<NUM>", binarize the voxel in which the object does not exist to "<NUM>", and encode the obtained binary data as one-dimensional data by scanning them respectively in the x-axis, y-axis and z-axis directions. The server front end <NUM> gives an identifier for identifying the 3D model to the relevant 3D model, and writes the obtained data into the database <NUM> according to a frame number together with point group data of the 3D model.

Moreover, the server front end <NUM> writes the foreground image in the database <NUM> in accordance with the camera identifier, an identifier of the gaze point, the identifier of the associated 3D model, and the frame number. Here, although the frame number is used as information representing the time, the present invention is not limited to this. Namely, a time code may be used. In the database <NUM>, the gaze point identified by the identifier of the camera, a camera position, a direction, and an angle of view are stored as camera setting information at the time of camera setting. The database <NUM> generates an object position information list in which position information of the object is described for each identifier of the 3D model in units of input frame number.

In the database <NUM>, position information of each of the gaze points <NUM> and <NUM> in a case where the gaze point <NUM> of <FIG> is used as the origin (<NUM>, <NUM>, <NUM>) of the stadium coordinates is further stored. Incidentally, it should be noted that the gaze points <NUM>, <NUM> and <NUM> have the same directions represented by the x-axis, the y-axis and the z-axis. That is, in the present embodiment, the gaze points <NUM>, <NUM> and <NUM> are on the same line, and the direction connecting them is the x-axis. Here, in the x-axis, it is assumed that the direction from the gaze point <NUM> to the gaze point <NUM> is negative and the direction from the gaze point <NUM> to the gaze point <NUM> is positive. The y-axis is orthogonal to the x-axis. In the y-axis, it is assumed that a main stand direction from the gaze point <NUM> to the front of the stadium is negative and a back stand direction is positive. In <FIG>, it is assumed that the bottom (for example, the side where the sensor system 2L08 is installed) is the main stand and the top (for example, the side where the sensor system 2L01 is installed) is the back stand. The z-axis is orthogonal to the x-axis and the y-axis. In the z-axis, it is assumed that the ground surface is the origin and the upward direction is positive.

<FIG> illustrates the x-axis, the y-axis, the z-axis, and the gaze points <NUM>, <NUM> and <NUM>. The gaze point <NUM> is separated by (-dx0) in the x direction and the gaze point <NUM> is separated by dx1 in the x direction with respect to the gaze point <NUM> that is an origin <NUM>-<NUM> of the stadium coordinates. Therefore, the gaze point <NUM> is (-dx0, <NUM>, <NUM>) with respect to the gaze point <NUM>, and the gaze point <NUM> is (dx1, <NUM>, <NUM>) with respect to the gaze point <NUM>. The position information of the objects to be described in the above object position information list is described in the world coordinates using, as the origin, the gaze point <NUM> obtained by correcting displacements of the positions of these gaze points. The axial directions and the positional relationships of the gaze points are not limited to this. Also, the position of the coordinate origin of the stadium is not limited to this. Namely, it may be possible to set any corner of the field as the origin, or place the origin on the main stand. Besides, it is not always necessary that the gaze point exists at the origin of the stadium.

Similarly, in the server front ends <NUM> and <NUM>, the data and the position information of the 3D models having the position information with each gaze point as the origin are written in the database <NUM> together with the foreground image.

<FIG> illustrates a constitution example of the data of the 3D model in the first embodiment. As shown in (A) of <FIG> as an example, the data of the 3D model is managed at each time, a time code indicating the time has been given to the head of the data, and the number of gaze points is indicated subsequently as an integer. In the example of <FIG>, since there are the three gaze points, the number of gaze points is three. Subsequently, a pointer to the first gaze point is designated. Here, it is assumed that the first gaze point is the gaze point <NUM>. As long as the same file, this pointer may be the data size to be skipped therefrom, or may be another file pointer.

As shown in (B) of <FIG> as an example, in the data designated by the pointer to the first gaze point, the position of the origin of the first gaze point is shown. The x-coordinate, the y-coordinate and the z-coordinate, i.e., the values of (-dx0, <NUM>, <NUM>), of the first gaze point from the origin of the stadium are described in order, and subsequently the number of objects included in the first gaze point is described. Thereafter, the pointer to each object is designated, and the point group of each object can be accessed by the pointer. As shown in (C) of <FIG> as an example, in the point group data of a first object, first the number of points constituting the first object is described, and the x coordinate, the y coordinate and the z coordinate of each point are described in order. Incidentally, data are similarly generated for other objects of the first gaze point. Similarly, data are generated for other gaze points.

Returning to <FIG>, a back-end server <NUM> accepts designation of the virtual viewpoint from the virtual camera operation UI <NUM>. The virtual camera operation UI <NUM> is an example of an information setting unit and an extracting unit. The back-end server <NUM> reads the corresponding foreground image, 3D model data and audio data from the database <NUM> based on the accepted virtual viewpoint, and performs a rendering process to generate the virtual viewpoint image. Here, the back-end server <NUM> generates the virtual viewpoint image for each gaze point. In the back-end server <NUM>, a virtual viewpoint image generating unit <NUM> generates the virtual viewpoint image of the area <NUM> corresponding to the gaze point <NUM>. Besides, a virtual viewpoint image generating unit <NUM> generates the virtual viewpoint image of the area <NUM> corresponding to the gaze point <NUM>, and a virtual viewpoint image generating unit <NUM> generates the virtual viewpoint image of the area <NUM> corresponding to the gaze point <NUM>. Besides, a virtual viewpoint background image generating unit <NUM> generates a background image from the virtual viewpoint. Here, the virtual viewpoint image generating units <NUM>, <NUM> and <NUM> are examples of an image generating unit.

In case of generating the virtual viewpoint image, a not-illustrated user uses the virtual camera operation UI <NUM> to set the position, direction, angle of view and the like of the virtual camera indicating the virtual viewpoint in the virtual viewpoint image. Hereinafter, information such as the position, direction, angle of view and the like of the virtual camera is also referred to as virtual camera information. The virtual camera information set by the virtual camera operation UI <NUM> is output to the back-end server <NUM>. Hereinafter, only the image will be described. For example, as illustrated fin <FIG>, it is assumed that a virtual camera <NUM> is set between the sensor system 2L19 and the sensor system 2L20. In <FIG>, the same constituent elements as those illustrated in <FIG> are denoted by the same reference numerals respectively. In <FIG>, <NUM> represents the angle of view of the virtual camera <NUM>.

The back-end server <NUM> inputs the virtual camera information to the database <NUM> in order to obtain an image necessary for generating the virtual viewpoint image seen from the virtual camera <NUM>. The database <NUM> retrieves and selects the foreground images captured by the respective sensor systems 2L01 to 2L30, 2C01 to 2C30, and 2R01 to 2R30, based on the input virtual camera information. Moreover, the database <NUM> retrieves and selects necessary data from the 3D models generated by the server front ends <NUM>, <NUM> and <NUM>, based on the input virtual camera information.

Besides, from the virtual camera information of the virtual camera <NUM>, the capturing range in the real space included in the angle of view <NUM> is determined. Incidentally, the position information of the virtual camera is represented by the position where the gaze point <NUM> is used as the origin of the world coordinates. It is determined whether or not each gaze point or each object is included in this capturing range, by comparing the position with the capturing range. In the example of <FIG>, it is assumed that an area including capturing ranges <NUM>, <NUM> and <NUM> is the capturing range. Here, since the virtual viewpoint image in the capturing range <NUM> includes the gaze point <NUM>, the virtual viewpoint image generating unit <NUM> generates the relevant virtual viewpoint image. Besides, since the virtual viewpoint image in the capturing range <NUM> includes the gaze point <NUM>, the virtual viewpoint image generating unit <NUM> generates the relevant virtual viewpoint image. Besides, since the virtual viewpoint image in the capturing range <NUM> includes the gaze point <NUM>, the virtual viewpoint image generating unit <NUM> generates the relevant virtual viewpoint image.

The image for generating the virtual viewpoint image of the angle of view <NUM> is selected by specifying based on the virtual camera information the range captured by the virtual camera <NUM>. Besides, as illustrated in <FIG>, the angle of view <NUM> includes areas <NUM>, <NUM> and <NUM> , there is an overlapping portion between the areas <NUM> and <NUM>, and there is an overlapping portion between the areas <NUM> and <NUM>. When an object is included in these portions, it is determined whether to use the 3D model data from which gazing point or its foreground image. As for a method for determination, it is assumed, in accordance with the invention, that the 3D model data of the gaze point close to the position of each object and the foreground image are used. However, other methods which do not form part of the invention may be used. For example, the 3D model data of the gaze point close to the virtual viewpoint and the foreground image may be used. That is, in case of generating the virtual viewpoint image related to the virtual camera <NUM>, for the object in the overlapping portion between the area <NUM> and the area <NUM>, the virtual viewpoint image is generated using the 3D model data of the gaze point <NUM> and the foreground image. Alternatively, in case of generating the virtual viewpoint image seen from the virtual camera <NUM>, the 3D model data of the gaze point that the number of cameras that captured the object is larger, and the foreground image may be employed. For example, a description will be given by taking an example of then object in the overlapping portion between the area <NUM> and the area <NUM>. With regard to the gaze point <NUM>, the twelve sensor systems from the sensor system 2L14 to the sensor system 2L25 are capturing the object from the positions and angles of view of the sensor systems. With regard to the gaze point <NUM>, the ten sensor systems from the sensor system 2C19 to the sensor system 2L28 are capturing the object from the positions and angles of view of the sensor systems. Accordingly, in this case, since there are more sensor systems capturing the gaze point <NUM>, the virtual viewpoint image is generated using the 3D model data of the gaze point <NUM> and the foreground image.

Generation of the virtual viewpoint image related to the area <NUM> in the virtual viewpoint image generating unit <NUM> will be described. <FIG> is a diagram illustrating a constitution example of the virtual viewpoint image generating unit <NUM>. The virtual camera information and the information of the frame number for generating the virtual viewpoint image are input from the virtual viewpoint camera operation UI <NUM> to a terminal <NUM>. A terminal <NUM> is connected to the database <NUM>, reads the object position information list from the database <NUM>, and further transmits a request for the foreground image and the 3D model data necessary for the image generation. Besides, foreground image data and the 3D model data read from the database <NUM> are input to a terminal <NUM>. A terminal <NUM> outputs image data of an object such as a player or the like, the image data representing a generated virtual viewpoint camera image.

A 3D model selecting unit <NUM> designates the identifier of the 3D model data necessary for generating the object in the area. A foreground image selecting unit <NUM> determines the foreground image data necessary for texture mapping from the identifier of the 3D model data necessary for generating the object image, the angle of view of the virtual camera, and the camera position information. An image buffer <NUM> stores the foreground image input from the database <NUM>. A model buffer <NUM> stores the 3D model data input from the database <NUM>. A rendering unit <NUM> generates the virtual viewpoint image of the object from the input 3D model data and the foreground image.

<FIG> is a flowchart for describing an example of the virtual viewpoint image generation operation in the first embodiment. In step S700, the virtual camera position, direction and angle of view are determined by the virtual camera operation UI <NUM> with a not-illustrated user. In step S701, the virtual camera operation UI <NUM> determines the capturing range based on the virtual camera information, and selects the gaze point included in the capturing range. By comparing the space indicated by the capturing range and the position of each gaze point in the world coordinates, it is determined whether or not the gaze point is included in the capturing range. In step S702, the back-end server <NUM> sets the virtual viewpoint image generating units <NUM> to <NUM> corresponding to the gaze point selected in step S701 to a processable state.

The subsequent process is the operation to be performed inside the selected virtual viewpoint image generating unit.

In step S703, the virtual camera information determined by the virtual camera operation UI <NUM> and the frame number of the frame for generating the virtual viewpoint image are input to the 3D model selecting unit <NUM>, the foreground image selecting unit <NUM> and the rendering unit <NUM> via the terminal <NUM>. In step S704, the 3D model selecting unit <NUM> requests the database <NUM> the object position information list of the input frame number via the terminal <NUM>, and receives the requested object position information list. In step S705, the 3D model selecting unit <NUM> compares the position information in the object position information list obtained in step S704 with the capturing range, determines the object included in the capturing range, and determines its 3D model identifier.

In step S706, the 3D model selecting unit <NUM> transmits the 3D model identifier determined in step S705 and the frame number to the database <NUM> via the terminal <NUM>, and requests data. In step S707, the database <NUM> reads the data of the 3D model based on the received 3D model identifier and frame number. In step S708, the 3D model data read in step S707, its 3D model identifier, and the frame number are stored in the model buffer <NUM> via the terminal <NUM>.

In step S709, the foreground image selecting unit <NUM> selects the foreground image related to the surface of the model seen from the virtual viewpoint, based on the virtual camera information input from the terminal <NUM> and the 3D model identifier of the object transmitted from the 3D model selecting unit <NUM>. Moreover, the foreground image selecting unit <NUM> selects the camera identifier that captured the selected foreground image. At this time, the camera that captured the surface that cannot be seen from the virtual viewpoint is not selected.

In step S710, the foreground image selecting unit <NUM> transmits the 3D model identifier, the selected camera identifier, and the frame number to the database <NUM> via the terminal <NUM>, and requests the data. In step S711, the database <NUM> reads out the necessary foreground image data based on the received 3D model identifier, the camera identifier, and the frame number. In step S712, the foreground image data read in step S711, its 3D model identifier, the camera identifier, and the frame number are stored in the image buffer <NUM> via the terminal <NUM>.

In step S713, the rendering unit <NUM> reads the 3D model data from the model buffer <NUM> based on the 3D model identifier and the frame number. Moreover, the rendering unit <NUM> reads the foreground image from the image buffer <NUM> based on the 3D model identifier, the camera identifier, and the frame number. Then, the rendering unit <NUM> texture-maps the foreground image on the 3D model data, and generates the image from the virtual viewpoint from the camera orientation, the angle of view, and the like of the virtual camera information input from the terminal <NUM>. Moreover, the rendering unit calculates the image position information in the final image of the generated image. The generated virtual viewpoint image and image position information are output from the terminal <NUM>.

<FIG> is a sequence diagram illustrating the flow of the generation of the virtual viewpoint image in the first embodiment. Initially, the virtual camera operation UI <NUM> determines the virtual camera position, direction and angle of view in accordance with the input by a not-illustrated user. The virtual camera operation UI <NUM> determines the capturing range based on the virtual camera information, and selects the gaze point included in the capturing range. The virtual viewpoint image generating units <NUM> to <NUM> corresponding to the gaze point selected by the virtual camera operation UI <NUM> are selected, and the virtual camera operation UI <NUM> sets the selected virtual viewpoint image generating unit to a state capable of starting a process. Then, the virtual camera operation UI <NUM> transmits the determined virtual camera information and the frame number of the frame for generating the virtual viewpoint image to the 3D model selecting unit <NUM>, the foreground image selecting unit <NUM>, and the rendering unit <NUM> (<NUM>).

The 3D model selecting unit <NUM> requests the database <NUM> the object position information list of the gaze point of the input frame number (<NUM>). The database <NUM> retrieves and reads the position information list of the corresponding frame number of the corresponding gaze point (<NUM>), and transmits the relevant list to the 3D model selecting unit <NUM> (<NUM>).

The 3D model selecting unit <NUM> compares the position information in the object position information list with the capturing range, determines the object included in the capturing range, and determines its 3D model identifier (<NUM>). After then, the 3D model selecting unit <NUM> transmits the 3D model identifier of the determined object and the frame number to the foreground image selecting unit <NUM> and the database <NUM> (<NUM>). The database <NUM> retrieves and reads the data of the 3D model based on the 3D model identifier and the frame number (<NUM>). Then, the database <NUM> transmits the read 3D model data, its 3D model identifier, and the frame number to the rendering unit <NUM> via the model buffer <NUM> (<NUM>).

Besides, the foreground image selecting unit <NUM> selects the foreground image related to the surface of the model seen from the virtual viewpoint, based on the virtual camera information transmitted from the virtual camera operation UI <NUM> and the 3D model identifier of the object transmitted from the 3D model selecting unit <NUM> (<NUM>). The foreground image selecting unit <NUM> selects the camera identifier that captured the selected foreground image. The foreground image selecting unit <NUM> transmits the 3D model identifier of the object, the selected camera identifier, and the frame number to the database <NUM> (<NUM>). The database <NUM> retrieves and reads the necessary foreground image data based on the 3D model identifier, the camera identifier, and the frame number (<NUM>). Then, the database <NUM> transmits the read foreground image data, its 3D model identifier, the camera identifier, and the frame number to the rendering unit <NUM> via the image buffer <NUM> (<NUM>).

The rendering unit <NUM> texture-maps the foreground image on the 3D model data, and generates the image from the virtual viewpoint from the camera orientation, the angle of view and the like of the virtual camera information input from the terminal <NUM>. In this way, each of the virtual viewpoint image generating units <NUM>, <NUM> and <NUM> generates the image from the virtual viewpoint for each gaze point.

Returning to <FIG>, the images respectively generated by the virtual viewpoint image generating units <NUM> to <NUM> are input to a synthesizing unit <NUM>. Besides, the virtual viewpoint background image generating unit <NUM> generates the background image from the virtual viewpoint using the background image data stored in the database <NUM>, and inputs the generated background image to the synthesizing unit <NUM>. More specifically, the virtual viewpoint background image generating unit <NUM> generates the background image from an actually captured image, CG (computer graphics) or the like based on the virtual camera position, angle of view and the like input from the virtual camera operation UI <NUM>.

The synthesizing unit <NUM> synthesizes the background image generated by the virtual viewpoint background image generating unit <NUM> and the image data generated by each of the virtual viewpoint image generating units <NUM> to <NUM> in accordance with each capturing range. Hereinafter, image synthesis by the synthesizing unit <NUM> will be described with reference to <FIG>.

<FIG> is a diagram illustrating an example of the background image generated by the virtual viewpoint background image generating unit <NUM>. In <FIG>, <NUM> indicates an entire background image, and <NUM> indicates a set angle of view. Besides, <NUM>, <NUM> and <NUM> indicates capturing ranges to be handled at the gaze point. The capturing range <NUM> indicates the capturing range constituted by the cameras pointed to the gaze point <NUM>, the capturing range <NUM> indicates the capturing range constituted by the cameras pointed to the gaze point <NUM>, and the capturing range <NUM> indicates the capturing range constituted by the cameras pointed to the gaze point <NUM>.

<FIG> illustrates an example of the image generated by each of the virtual viewpoint image generating units <NUM> to <NUM>. In the capturing range <NUM>, the image in which the 3D model data is texture-mapped to the foreground image captured by the cameras that capture the gaze point <NUM> is generated. In the capturing range <NUM>, the image in which the 3D model data is texture-mapped to the foreground image captured by the cameras that capture the gaze point <NUM> is generated. In the capturing range <NUM>, the image in which the 3D model data is texture-mapped to the foreground image captured by the cameras that capture the gaze point <NUM> is generated.

In case of the image synthesis, the synthesizing unit <NUM> performs the synthesis from the image of the distant gaze point. Thus, in a case where a distant object and a near object overlap each other, the distant object is naturally hidden by the near object. <FIG> illustrates the image that is synthesized with the background image. The synthesized image that has been synthesized as described above is transmitted to an output unit <NUM>, and output to the outside.

As described above, according to the present embodiment, it is possible to generate a wide-area image from the virtual viewpoint without deteriorating quality in relation to the generation of the virtual viewpoint image. That is, since each object can be accurately modeled and the high-quality foreground image can be used, the quality of the entire virtual viewpoint image can be improved. For example, in case of trying to generate the image as illustrated in <FIG> only from the images captured by the cameras pointed to the gaze point <NUM>, the image quality of the players in the capturing range <NUM> is not different from that of the present invention. However, since the players in the capturing ranges <NUM> and <NUM> are out of focus, blurred images are given, so that is becomes impossible to perform accurate 3D model generation and high-quality texture mapping of the foreground image. On the other hand, in the present embodiment, such a deterioration of quality is prevented, and an inaccurate image that is more blurred than the distance in spite of being close to the virtual viewpoint is not provided.

In the present embodiment, the method of generating the virtual viewpoint image by generating the 3D model has been described using the MBR. However, the method is not particularly limited to this, and Image Based Rendering (IBR) that does not construct a 3D model, or another method may be used. Hereinafter, as an example, an example using the IBR will be described.

In <FIG>, each of the server front ends <NUM>, <NUM> and <NUM> cuts out the target such as a player or the like from the reconstructed frame data, and generates only the foreground image. Moreover, each of the server front ends <NUM>, <NUM> and <NUM> writes the foreground image into the database <NUM> according to the camera identifier, the gaze point identifier, and the frame number. The back-end server <NUM> accepts designation of the virtual viewpoint from the virtual camera operation UI <NUM>. Moreover, the back-end server <NUM> reads the corresponding foreground image and audio data from the database <NUM> based on the accepted virtual viewpoint, and performs a rendering process to generate the virtual viewpoint image. Moreover, the back-end server <NUM> generates the virtual viewpoint image for each gaze point. In case of generating the virtual viewpoint image, a not-illustrated user uses the virtual camera operation UI <NUM> to set the position, the direction, the angle of view, and the like of the virtual camera indicating the virtual viewpoint in the virtual viewpoint image.

Hereinafter, the virtual viewpoint image generating unit <NUM> will be described as an example. However, the same is applied to the virtual viewpoint image generating units <NUM> and <NUM>. In order to obtain the image necessary for generating a virtual viewpoint image seen from the virtual camera <NUM>, the virtual camera information is input from the virtual camera operation UI <NUM> to the virtual viewpoint image generating unit <NUM>. Based on the virtual camera information, necessary data is retrieved and selected from the foreground images respectively captured by the sensor systems 2L01 to 2L30, 2C01 to 2C30, and 2R01 to 2R30. The database <NUM> determines the capturing range in the real space included in the angle of view <NUM> from the virtual camera information of the virtual camera <NUM>. Incidentally, the position information of the virtual camera is represented by the position where the gaze point <NUM> is used as the origin of the world coordinates.

As well as the above-described embodiment, generation of the virtual viewpoint image related to the area <NUM> in the virtual viewpoint image generating unit <NUM> will be described. <FIG> is a diagram illustrating another constitution example of the virtual viewpoint image generating unit <NUM>. In <FIG>, the constituent elements having the same functions as those of the constituent elements illustrated in <FIG> are denoted by the same reference numerals respectively, and redundant descriptions are omitted. A foreground image selecting unit <NUM> selects the camera necessary to generate an image of the object from the angle of view of the virtual camera and the camera position information, and determines necessary data of the foreground image. An image buffer <NUM> stores the foreground image input from the database <NUM>. A rendering unit <NUM> generates the virtual viewpoint image of the object from the input foreground image. The foreground image selecting unit <NUM> stores in advance the position information of the cameras capturing the respective gaze points. Incidentally, the position information of the camera may be read from the outside.

The method of the IBR is not particularly limited. For example, as described in PTL <NUM>, an image is generated based on the images from the two cameras. The foreground image selecting unit <NUM> selects the two nearest cameras from the position of the virtual viewpoint camera. In the virtual viewpoint image generating unit <NUM>, the foreground images of the cameras of the sensor system 2L19 and the sensor system 2L20 are selected for the virtual viewpoint camera <NUM> illustrated in <FIG>. Similarly, in the foreground image selecting unit <NUM> of the virtual viewpoint image generating unit <NUM>, the sensor system 2C23 and the sensor system 2C24 are selected. Moreover, in the foreground image selecting unit <NUM> of the virtual viewpoint image generating unit <NUM>, the sensor system 2R27 and the sensor system 2R28 are selected.

The foreground image selecting unit <NUM> transmits the corresponding frame number and the identifier of the camera of each sensor system to the database <NUM> via the terminal <NUM>, and requests data. The database <NUM> reads the necessary foreground image data based on the received frame number and camera identifier. The read foreground image data, its frame number, and the camera identifier are stored in the image buffer <NUM> via the terminal <NUM>. The rendering unit <NUM> reads the foreground image from the image buffer <NUM> based on the camera identifier and the frame number. From the foreground images of the two cameras, the rendering unit <NUM> generates an image from the virtual viewpoint using a technique such as morphing or the like. Moreover, the rendering unit calculates the image position information in the final image of the generated image. The generated virtual viewpoint image and image position information are output from the terminal <NUM>.

As just described, in relation to generation of the virtual viewpoint image, it is possible to generate a wide-area image from the virtual viewpoint without deteriorating quality even by the IBR that does not use a 3D model. That is, in the present embodiment, since the high-quality foreground image can be used for each virtual viewpoint camera, it is possible to improve the quality of the entire virtual viewpoint image.

Incidentally, it should be noted that the above-described image processing system <NUM> in the present embodiment is not limited to the above-explained physical constitution, and this system may be logically constituted. Besides, although the sensor groups <NUM>, <NUM> and <NUM> are connected to the switching hub <NUM>, the present invention is not limited to this. It is of course possible to perform a cascade connection for these sensor groups. Besides, although the example in which the plurality of virtual viewpoint image generating units are used has been described, the present invention is not limited to this. Namely, the present embodiment may be achieved by a time-division system or parallel processes in a plurality of threads, using one virtual viewpoint image generating unit.

Incidentally, in the above embodiment, although the difference of the position information between the gaze points has been described, the present invention is not limited to this. Of course, the camera position, the gaze point position and the like may be calculated using the world coordinates that are based on one origin. That is, in (C) of <FIG>, as the information of each point group, it may be possible to store information obtained by adding, to the coordinates of each point, not the coordinates from each gaze point origin but the coordinates of the gaze point from the stadium origin. Incidentally, in the present embodiment, since the gaze point is fixed, it is also possible to store the position information of the gaze point as a list in the database <NUM>, or store the position information of the gaze point as a fixed value in the server front ends <NUM> to <NUM> and the back-end server <NUM>.

Moreover, in the above-described embodiment, it is possible to perform the rendering with the resolution of the object included in the gaze point far from the virtual camera being lower than the resolution of the object included in the gaze point close to the virtual camera. That is, since a distant object becomes small at the time of synthesizing, it becomes possible to perform a high-speed process by suppressing the original resolution. Thus, in the IBR, by lowering the resolution of the 3D model of the object included in the distant gaze point, it becomes possible to perform high-speed model generation and rendering. Incidentally, although the above embodiment has been described using the stadium of soccer or the like, as an example, the present invention is not limited to this. For example, it may be a game such as baseball, basketball, skating or the like, or it may be a stage or a movie set.

<FIG> is a diagram illustrating a constitution example of the image processing system <NUM> according to the second embodiment. In <FIG>, the constituent elements having the same functions as those of the constituent elements illustrated in <FIG> are denoted by the same reference numerals respectively, and redundant descriptions are omitted. Server front ends <NUM>, <NUM> and <NUM> process data obtained from the sensor systems. The server front ends <NUM> to <NUM> are different from the server front ends <NUM> to <NUM> in the first embodiment in the point of obtaining the position information of each gaze point from a control station <NUM> and giving the position information from the stadium origin to each data.

<FIG> illustrates an example of gaze points of the stadium in the second embodiment. In the present embodiment, ski aerials will be described as an example. Although the description will be made assuming that there are the three gaze points also in the present embodiment, the present invention is not limited to this. It is assumed that gaze points <NUM>, <NUM> and <NUM> represent the gaze points at the start of capturing, and a gaze point <NUM> is an origin <NUM>-<NUM> of the field. The sensor systems 12L01 to 12L06 correspond to the gaze point <NUM>, and each of the sensor systems has one camera <NUM> and one camera platform <NUM>. The sensor systems 12C01 to 12C06 correspond to the gaze point <NUM>, and each of the sensor system has one camera <NUM> and one camera platform <NUM>. The sensor systems 12R01 to 12R06 correspond to the gaze point <NUM>, and each of the sensor systems has one camera <NUM> and one camera platform <NUM>. Although an example in which the six sensor systems are used for each gaze point is described in the present embodiment, the present invention is not limited to this.

<FIG> illustrates a capturing range of each gaze point. The cameras <NUM> of the sensor systems 12R01 to 12R06 corresponding to the gaze point <NUM> capture the range of an area <NUM>. The cameras <NUM> of the sensor systems 12C01 to 12C06 corresponding to the gaze point <NUM> capture the range of an area <NUM>, and the cameras <NUM> of the sensor systems 12L01 to 12L06 corresponding to the gaze point <NUM> capture the range of an area <NUM>. Here, as illustrated in <FIG>, the gaze point <NUM> is the origin of the field, and its coordinates are (<NUM>, <NUM>, <NUM>). The gaze point <NUM> is (dx1, dy1, -dz1) in the field coordinates, and the gaze point <NUM> is (-dx0, -dy0, dz0) in the field coordinates.

The sound collected by the microphone <NUM> of the sensor system 12L01 and the image captured by the camera <NUM> are subjected to an image process by the camera adapter <NUM>, and then transmitted to the camera adapter <NUM> of the sensor system 12L02 via the network <NUM>. Similarly, the sensor system 12L02 combines the collected sound, the captured image and the image and sound data obtained from the sensor system 12L01 together, and transmits the obtained data to the sensor system 12L03 via the network <NUM>. By continuing the above operation, the images and sounds obtained by the sensor systems 12L01 to 12L06 are transmitted from the sensor system 12L06 to the server front end <NUM> via the networks 180b and 211a and the switching hub <NUM>.

The control station <NUM> can move the gaze point by controlling the camera platform <NUM> at the capture or between the captures and thus moving the direction of the camera <NUM>. A case where the control station <NUM> sets a new gaze point using the camera platform <NUM> will be described. For example, in case of moving the gaze point <NUM> by (sx1, sy1, sz1), the control station <NUM> controls the camera platform <NUM> of each of the sensor systems 12L01 to 12L06, and points the camera <NUM> to the intended direction, thereby controlling the focus and angle of view. Then, the information related to the change of the position of the gaze point is input to the server front end <NUM> via a network 311a.

Similarly, in case of moving the gaze point <NUM>, the control station <NUM> controls the camera platform <NUM> of each of the sensor systems 12C01 to 12C06, and points the camera <NUM> to the intended direction, thereby controlling the focus and angle of view. Then, the information related to the change of the position of the gaze point is input to the server front end <NUM> via a network 311b. Besides, in case of moving the gaze point <NUM>, the control station <NUM> controls the camera platform <NUM> of each of the sensor systems 12R01 to 12R06, and points the camera <NUM> to the intended direction, thereby controlling the focus and angle of view. Then, the information related to the change of the position of the gaze point is input to the server front end <NUM> via a network 311c.

In the present embodiment, the server front end <NUM> reconstructs a segmented transmission packet from the image and sound obtained from the sensor system 12L06, and converts a data format of frame data. Moreover, as well as the server front end <NUM> in the first embodiment, the server front end <NUM> cuts out a target (object) such as a player or the like from the reconstructed frame data, and generates a 3D model of the object from the images of all the cameras using the cut-out result as the foreground image. Here, as well as the first embodiment, it is assumed that the generated 3D model is expressed as a point group. The server front end <NUM> gives an identifier for identifying the 3D model to the relevant 3D model, and writes the obtained data into the database <NUM> according to a frame number together with point group data of the 3D model.

<FIG> illustrates a constitution example of the data of the 3D model in the second embodiment. As shown in (A) of <FIG> as an example, the data of the 3D model is managed at each time, a time code indicating the time has been given to the head of the data, and the number of gaze points is indicated subsequently as an integer. In the example of <FIG>, since there are the three gaze points, the number of gaze points is three. Subsequently, a pointer to the first gaze point is designated. Here, it is assumed that the first gaze point is the gaze point <NUM>. As long as the same file, this pointer may be the data size to be skipped therefrom, or may be another file pointer.

As shown in (B) of <FIG> as an example, in the data designated by the pointer to the first gaze point, the position of the origin of the first gaze point is shown. The x-coordinate, the y-coordinate and the z-coordinate, i.e., the values of (-dx0, -dy0, dz0), of the first gaze point from the origin of the field are described in order, and subsequently the number of objects included in the first gaze point is described. Thereafter, the pointer to each object is designated, and the point group of each object can be accessed by the pointer. Moreover, the number of points constituting the first object is subsequently described. The data amount of the 3D data of the first object can be calculated from the data length of the data representing the coordinates, and the number of points, so that it is possible to obtain the data in a lump.

As shown in (C) of <FIG> as an example, in the point group data of the first object, the x coordinate, the y coordinate and the z coordinate of the origin of the circumscribed cube of the first object are described. Subsequently, the size in the x-axis direction (x size), the size in the y-axis direction (y size) and the size in the z-axis direction (z size) are described, and these sizes represent the size of the circumscribed cube of the first object. Subsequently, the x-coordinate, the y-coordinate and the z-coordinate of each point are described in order. Similarly, data are generated for other objects of the first gaze point. Similarly, data is generated for other gaze points.

Here, a circumscribed cube including the first object is assumed. Aspects of this circumscribed cube are illustrated in <FIG>. The position in the case where the origin of the circumscribed cube is the origin of gaze point is described as the circumscribed cube origin coordinates of the first object. In the present embodiment, as illustrated in <FIG>, it is assumed that the coordinate positions with the gaze point as the origin are x<NUM>, y<NUM> and z<NUM>. Besides, as illustrated in <FIG>, it is assumed that the size of the circumscribed cube is represented by xs0, ys0 and zs0. In the following, the x coordinate, the y coordinate and the z coordinate of each point constituting the first object are described in order as the relative positions from the origin of the circumscribed cube.

As well as the first embodiment, the back-end server <NUM> reads the 3D model data and the foreground image from the database <NUM>, and performs a rendering process to generate the virtual viewpoint image. Here, the back-end server <NUM> generates a virtual viewpoint image for each gaze point.

In case of generating the virtual viewpoint image, a not-illustrated user uses the virtual camera operation UI <NUM> to generate virtual camera information. The back-end server <NUM> inputs the virtual camera information to the database <NUM> in order to obtain an image necessary for generating the virtual viewpoint image seen from a virtual camera <NUM> illustrated in <FIG>. The database <NUM> retrieves and selects the foreground images captured by the respective sensor systems 12L01 to 12L06, 12C01 to 12C06, and 12R01 to 12R06, based on the input virtual camera information. Moreover, the database <NUM> retrieves and selects necessary data from the 3D models generated by the server front ends <NUM>, <NUM> and <NUM>, based on the input virtual camera information.

Besides, from the virtual camera information of the virtual camera <NUM>, the capturing range in the real space included in an angle of view <NUM> is determined. Incidentally, the position information of the virtual camera is represented by the position where the gaze point <NUM> is used as the origin of the world coordinates. In the present embodiment, since the gaze point moves, whether or not the gaze point is included in the angle of view is decided by the coordinates of each gaze point from the origin of the field and whether or not its area is included in the angle of view, as shown in (B) of <FIG>. Besides, whether or not each gaze point or each object is included in the capturing range is determined by a comparison between its position and the capturing range. In this case, initially, in order to represent the object with the circumscribed cube, the database <NUM> first decides whether or not the circumscribed cube is included in the field of view from the virtual viewpoint <NUM>. This can be decided by whether or not the points of the corners of the circumscribed cube are included in the angle of view.

In the example of <FIG>, since the virtual viewpoint image includes the gaze point <NUM>, a virtual viewpoint image generating unit <NUM> generates the virtual viewpoint image. Moreover, since the virtual viewpoint image includes the gaze point <NUM>, a virtual viewpoint image generating unit <NUM> generates the virtual viewpoint image. Incidentally, since the virtual viewpoint image does not include the gaze point <NUM>, a virtual viewpoint image generating unit <NUM> does not operate.

The generation of the virtual viewpoint image related to the area <NUM> in the virtual viewpoint image generating unit <NUM> will be described. Since the constitution of the virtual viewpoint image generating unit <NUM> is the same as the constitution of the virtual viewpoint image generating unit <NUM> in the first embodiment, a description thereof will be omitted. Moreover, the virtual viewpoint image generation operation in the second embodiment is the same as that in the first embodiment shown in the flowchart of <FIG>. However, in the present embodiment, since the gaze point moves, it is necessary in step S701 to read and compare the information of the gaze point position from the 3D model data in the database <NUM>. Besides, in step S705, each point of the circumscribed cube is referred to as the position information of the object.

<FIG> is a sequence diagram illustrating the flow of the generation of the virtual viewpoint image in the second embodiment. Initially, the virtual camera operation UI <NUM> determines the virtual camera position, direction and angle of view in accordance with the input by a not-illustrated user. The virtual camera operation UI <NUM> transmits the time of frame for image generation to the database <NUM> and requests the gaze point position information at that time (<NUM>). The database <NUM> transmits the position information of each gaze point to the virtual camera operation UI <NUM> (<NUM>). The virtual camera operation UI <NUM> determines the capturing range based on the virtual camera information and the gaze point position information, and selects the gaze point included in the capturing range. The virtual viewpoint image generating units <NUM> to <NUM> corresponding to the gaze point selected by the virtual camera operation UI <NUM> are selected, and the virtual camera operation UI <NUM> sets the selected virtual viewpoint image generating unit to a state capable of starting a process. Then, the virtual camera operation UI <NUM> transmits the determined virtual camera information and the frame number of the frame for generating the virtual viewpoint image to the 3D model selecting unit <NUM>, the foreground image selecting unit <NUM>, and the rendering unit <NUM> (<NUM>).

The rendering unit <NUM> texture-maps the foreground image on the 3D model data, and generates the image from the virtual viewpoint from the camera orientation, the angle of view and the like of the virtual camera information input from the terminal <NUM>. In this way, the virtual viewpoint image generating unit that generates the virtual viewpoint image among the virtual viewpoint image generating units <NUM>, <NUM> and <NUM> generates the image from the virtual viewpoint for each gaze point.

As well as the first embodiment, the synthesizing unit <NUM> synthesizes the background image generated by the virtual viewpoint background image generating unit <NUM> and the image data generated by each of the virtual viewpoint image generating units <NUM> to <NUM> in accordance with each capturing range.

As described above, according to the present embodiment, in relation to the generation of the virtual viewpoint image, it is possible to prevent a deterioration of quality, and to generate a wide-area image from the virtual viewpoint without providing an inaccurate image that is more blurred than the distance in spite of being close to the virtual viewpoint. That is, since each object can be accurately modeled and the high-quality foreground image can be used, it is possible to improve the quality of the entire virtual viewpoint image. In addition, it is possible to track a moving object, thereby always allowing to set the object as the gaze point. As a result, since the object can always be captured with the best focus, it is possible to generate an accurate object, and it is also possible to obtain a high-quality image even in case of generating the virtual viewpoint image at long range.

In the present embodiment, although the position information of the gaze point is included in the 3D model data and recorded, the present invention is not limited to this. The position information of the gaze point may separately be listed in the database <NUM> in association with the frame number. Besides, although the above embodiment has been described using the difference of the position information between the gaze points, the present invention is not limited to this. Of course, the camera position, the gaze point position, and the like may be calculated using the world coordinates based on one origin. That is, in <FIG>, it may be possible to store the information of each point group obtained by adding, to the coordinates of each point, not the coordinates from each circumscribed cube origin but the coordinates from the gaze point origin. Moreover, in <FIG>, it may be possible to store the information of each point group obtained by adding, to the coordinates of each point, not the coordinates from each circumscribed cube origin but the coordinates from the gaze point origin and the field origin.

Incidentally, in the present embodiment, although the synthesis is performed by overwriting the distant object with the near object, the present invention is not limited to this. More specifically, with respect to the position and size of each object can, it is possible to derive their anteroposterior relationships from, for example, the field origin, the gaze point coordinates, the circumscribed cube coordinates and their sizes. Thus, since generation of the distant object that is hidden by the near object can be omitted, it is possible to perform image generation at high speed and at low cost.

Incidentally, as well as the first embodiment, the method of generating the virtual viewpoint image is not limited to this.

Next, a third illustrative embodiment which does not form part of the present invention will be described.

<FIG> is a diagram illustrating a constitution example of the image processing system <NUM> according to the third embodiment. In <FIG>, the constituent elements having the same functions as those of the constituent elements illustrated in <FIG> are denoted by the same reference numerals respectively, and redundant descriptions are omitted. Sensors <NUM>, <NUM> and <NUM> sense the weather conditions of a stadium. Examples of the weather conditions include humidity, temperature, weather and the like. A back-end server <NUM> has virtual viewpoint image correcting units <NUM>, <NUM> and <NUM> in addition to the virtual viewpoint image generating units <NUM>, <NUM> and <NUM>. The stadium will be described as being the same as in the first embodiment.

The sensors <NUM>, <NUM> and <NUM> are arranged at various locations in the stadium, and measure humidity and temperature as the environmental conditions at the time of capturing. The measured weather conditions are called weather information. The weather information is recorded in the database <NUM> at each time. For example, when the humidity is high, there are many water molecules in the atmosphere, so that a long-distance image appears blurred. More specifically, in the example of <FIG>, the object in the capturing range <NUM> is hazily seen from the virtual camera <NUM> depending on the weather conditions. The haze has already been modeled as Mie scattering. Similarly, if it is raining, a distant image is hazily seen.

In the present embodiment, the generation of the virtual viewpoint image in each capturing range is the same as that in the first embodiment. Each of the virtual viewpoint image correcting units <NUM>, <NUM> and <NUM> calculates haze and light attenuation based on the weather information and the distance between the virtual camera and the gaze point, and perform a haze process to the generated virtual viewpoint image. Thus, as well as the first embodiment, in relation to the generation of the virtual viewpoint image, since it is possible to generate a wide-area image from the virtual viewpoint without deteriorating quality, it is possible to improve the quality of the virtual viewpoint image. Moreover, by performing a correcting process to the virtual viewpoint image generated based on the weather conditions at the time of capturing, atmosphere or air feeling of the stadium can be reproduced, so that it is possible to generate the virtual viewpoint image closer to reality.

Incidentally, although the example in which the haze process is performed according to the distance of the gaze point has been described, the present invention is not limited to this. Namely, it is possible to perform the haze process also according to the distance between the object and the virtual camera. For example, the position of the circumscribed cube in the second embodiment can be easily calculated by referring to the coordinates of the gaze point from the stadium origin and the coordinates of the circumscribed cube from the gaze point.

Next, a fourth embodiment of the present invention will be described.

In the second embodiment, although the position of each object and the point coordinates of the point group are represented by the gaze point coordinates from the origin of the field and the coordinates from the gaze point of the circumscribed cube, the present invention is not limited to this. <FIG> is a diagram illustrating a constitution example of the image processing system <NUM> according to the fourth embodiment. In <FIG>, the constituent elements having the same functions as those of the constituent elements illustrated in <FIG> and <FIG> are denoted by the same reference numerals respectively, and redundant descriptions are omitted.

A coordinate converting unit <NUM> converts, in relation to the gaze point <NUM>, each circumscribed cube origin and each coordinate of the point group not into the relative coordinates from the gaze point origin but into the coordinates for the stadium origin. Similarly, coordinate converting units <NUM> and <NUM> convert, respectively in relation to the gaze points <NUM> and <NUM>, the circumscribed cube origin and each coordinate of the point group not into the relative coordinates from the gaze point origin but into the coordinates for the stadium origin. The 3D model data converted in this way is stored as exemplarily illustrated in <FIG>. However, the present invention is not limited to such a format. Namely, it is possible to omit the coordinates of the first gaze point from the stadium origin. Thus, all the points are represented by the coordinates for the stadium origin. Accordingly, by calculating them in advance, it is not necessary to calculate the respective positions from the differences of the coordinates in the rendering process, so that it is possible to speed up the image generation.

Incidentally, there may be an area where the plurality of gaze point areas overlap depending on the installation states of the cameras. In that case, for example, in relation to the 3D model data of the object, there is a case where the data of the same object exists for each gaze point area. In this case, all the data can be stored. However, if generation accuracy of the 3D model is different, there is a case where the quality of the finally generated virtual viewpoint image is affected. Therefore, for example, as described in the first embodiment, the data may be selected based on information of the position of the object and the gaze point area. For example, with respect to certain 3D model data, it is possible to perform a process of leaving only the data of the gaze point with the closest coordinates, and deleting the overlapping data in other gaze points.

The present invention can be realized also by a process in which a program for realizing one of more functions of the above embodiments are supplied to a system or an apparatus via a network or a storage medium and one or more processors in the system or the apparatus read and execute the supplied program. Besides, the present invention can be realized also by a circuit (e.g., an ASIC) of realizing one or more functions of the above embodiments.

For example, each of the image processing systems described in the first to fourth embodiments has a computer function <NUM> as illustrated in <FIG>, and a CPU <NUM> thereof performs the operations in the first to fourth embodiments.

As illustrated in <FIG>, the computer function <NUM> has the CPU <NUM>, a ROM <NUM> and a RAM <NUM>. Moreover, the computer function has a controller (CONSC) <NUM> of an operation unit (CONS) <NUM>, and a display controller (DISPC) <NUM> of a display (DISP) <NUM> serving as a display unit such as a CRT, an LCD or the like. Moreover, the computer function has a controller (DCONT) <NUM> of a storage device (STD) <NUM> such as a hard disk (HD) <NUM>, a flexible disk and the like, and a network interface card (NIC) <NUM>. These functional units <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> are configured to be communicably connected to others via a system bus <NUM>.

The CPU <NUM> comprehensively controls the respective constituent elements connected to the system bus <NUM> by executing the software stored in the ROM <NUM> or the HD <NUM> or the software supplied from the STD <NUM>. That is, the CPU <NUM> performs the control for realizing the operations in the first to fourth embodiments, by reading and executing processing programs for performing the above operations from the ROM <NUM>, the HD <NUM> or the STD <NUM>. The RAM <NUM> functions as a main memory, a work area or the like for the CPU <NUM>. The CONSC <NUM> controls the instruction input from the CONS <NUM>. The DISPC <NUM> controls the display of the DISP <NUM>. The DCONT <NUM> controls the access to the HD <NUM> and the STD <NUM> that store a boot program, various applications, user files, a network management program, the processing programs in the first to fourth embodiments, and the like. The NIC <NUM> bidirectionally exchanges the data with other apparatuses and devices on a network <NUM>.

The above embodiments are merely the examples of concretization for carrying out the present invention. Accordingly, the technical scope of the present invention should not be interpreted restrictively or limitedly by these embodiments.

Claim 1:
A generating apparatus (<NUM>) that generates a virtual viewpoint image on the basis of a plurality of captured images obtained by a plurality of image capturing devices, the plurality of image capturing devices being classified into a plurality of image capturing device groups including a first image capturing device group comprising the plurality of image capturing devices pointed to a first gaze point and a second image capturing device group comprising the plurality of image capturing devices pointed to a second gaze point different from the first gaze point, the generating apparatus comprising:
a first obtaining unit (<NUM>) configured to obtain first image data based on the captured image obtained by the image capturing device belonging to the first image capturing device group;
a second obtaining unit (<NUM>) configured to obtain second image data based on the captured image obtained by the image capturing device belonging to the second image capturing device group;
a third obtaining unit (<NUM>) configured to obtain information related to a position and a direction of a virtual viewpoint; and
a generating unit (<NUM>) configured to generate the virtual viewpoint image based on the first image data obtained by the first obtaining unit, the second image data obtained by the second obtaining unit, and the information related to the position and the direction of the virtual viewpoint obtained by the third obtaining unit,
characterized in that the generating unit is configured to generate the virtual viewpoint image so that objects closer to the virtual viewpoint are displayed in front of objects farther from the virtual viewpoint based on a distance between the position of the virtual viewpoint specified by the information obtained by the third obtaining unit and a position of the first gaze point, and a distance between the specified position of the virtual viewpoint and a position of the second gaze point,
wherein image content of a particular portion of the virtual viewpoint image corresponding to one of the objects or to a scene background is generated from an image capturing device group, among the plurality of image capturing device groups, having a gaze point closest in three-dimensional spatial distance to the corresponding three-dimensional position of the one of the objects or of the scene background.