Patent Description:
In recent years, special attention has been paid to a technique in which a plurality of imaging apparatuses (cameras) are installed at different positions, images are captured thereby from different directions, and a virtual viewpoint image is generated using a plurality of captured images. The virtual viewpoint image generation technique enables a user to see an image viewed from a specified viewpoint (a virtual viewpoint). According to the technique for generating virtual viewpoint images as described above, for example, highlight scenes of a soccer game, a basketball game, or the like can be viewed from various angles, which gives a user a high sense of realism compared to normal images. <CIT> discloses a technique for generating a virtual viewpoint image corresponding to a virtual viewpoint specified based on an operation by a user.

However, the technique disclosed in <CIT> does not consider a case where a plurality of virtual viewpoints are specified by a plurality of operations. Furthermore, in a case where a plurality of virtual viewpoints are specified and a specific virtual viewpoint is selected from among them, it is necessary to generate a plurality of virtual viewpoint images and present them to the user, and thus an increase occurs in a processing load related to the generation of the virtual viewpoint images. There is also a demand for generating a selected virtual viewpoint image with appropriate quality.

<CIT> discusses a virtual viewpoint image generation apparatus including: a first generation unit configured to generate, based on a plurality of captured images obtained by a plurality of first cameras, a first virtual viewpoint image in accordance with a position and direction of a virtual viewpoint; a determination unit configured to determine, in accordance with evaluation results of the first virtual viewpoint image, whether or not to generate a second virtual viewpoint image whose image quality is higher than that of the first virtual viewpoint image based on one or a plurality of captured images obtained by one or a plurality of second cameras; and a second generation unit configured to generate the second virtual viewpoint image whose image quality is higher than that of the first virtual viewpoint image in accordance with determination by the determination unit.

A paper by <NPL>, describes a system that allows the generation and rendering of <NUM>-D models in preview quality for on-set visualization in real time and in high quality for post production applications in an offline phase. The shape reconstruction is based on the computation of the visual hull. The view-dependent projection component allows actors to be immersed into a virtual scene and to interact with virtual components.

The present disclosure provides a technique of appropriately generating a virtual viewpoint image corresponding to a selected virtual viewpoint while suppressing the processing load in selecting a specific virtual viewpoint from among a plurality of virtual viewpoints.

The present invention in its first aspect provides an information processing apparatus as specified in claims <NUM> to <NUM>.

The present invention in its second aspect provides an information processing method as specified in Claim <NUM>.

Embodiments of the present disclosure are described below with reference to drawings. Note that constituent elements described in the following embodiments are merely examples according to the present disclosure, and the scope of the present disclosure is not limited by these examples. Each of the embodiments of the present invention described below can be implemented solely or as a combination of a plurality of the embodiments or features thereof where necessary or where the combination of elements or features from individual embodiments in a single embodiment is beneficial.

A hardware configuration of an information processing apparatus according to a first embodiment is described. <FIG> is a block diagram illustrating the hardware configuration of the information processing apparatus <NUM>. The information processing apparatus <NUM> includes a CPU <NUM>, a RAM <NUM>, a ROM <NUM>, an operation unit <NUM>, an output unit <NUM>, an auxiliary storage apparatus <NUM>, an I/F <NUM>, and a bus <NUM>.

The CPU <NUM> controls an entire computer using computer programs and data stored in the RAM <NUM> or the ROM <NUM>, and executes processes performed by the information processing apparatus <NUM> according to the present embodiment. That is, the CPU <NUM> functions as each processing unit of the information processing apparatus <NUM>. Each processing unit will be described later.

The RAM <NUM> has an area for temporarily storing computer programs and data loaded from the auxiliary storage apparatus <NUM>, data obtained from outside via the I/F <NUM>, and the like. Furthermore, the RAM <NUM> has also a work area used by the CPU <NUM> in executing various processes. That is, the RAM <NUM> can be allocated, for example, as a frame memory and can provide other various areas as required.

The ROM <NUM> stores setting data of the computer, a boot program, and the like. The operation unit <NUM> includes a keyboard, a mouse, a joystick, a touch panel, etc., and is operated by a user of the computer to input various instructions to the CPU <NUM>. The output unit <NUM> displays a result of processing performed by the CPU <NUM>. The output unit <NUM> includes, for example, a liquid crystal display, a touch panel, or the like.

The auxiliary storage apparatus <NUM> is a large-capacity information storage apparatus typified by a hard disk drive. The auxiliary storage apparatus <NUM> stores an OS (operating system) and a computer program for causing the CPU <NUM> to realize the functions of the units shown in <FIG>. The auxiliary storage apparatus <NUM> may further store one or more pieces of image data to be processed. Computer programs and data stored in the auxiliary storage apparatus <NUM> are loaded into the RAM <NUM>, as required, under the control of the CPU <NUM> and processed by the CPU <NUM>.

The I/F <NUM> is an interface for connecting with a network such as a LAN, the Internet or the like, other devices such as a projector, a display apparatus, or the like. The computer is allowed to send/receive various kinds of information via this I/F <NUM>. For example, to connect the information processing apparatus <NUM> to an external apparatus via a communication cable, the communication cable is connected to the I/F <NUM>. In a case where the information processing apparatus <NUM> has a function of wirelessly communicating with an external apparatus, the I/F <NUM> includes an antenna. In the present embodiment, a plurality of imaging apparatuses are connected to the I/F <NUM> to enable it to obtain captured images and control each imaging apparatus. The bus <NUM> is a bus via which the units described above are connected.

In the present embodiment, it is assumed by way of example that the operation unit <NUM>, the output unit <NUM>, and the auxiliary storage apparatus <NUM> exist inside the information processing apparatus <NUM>, but the present disclosure is not limited to this configuration. The information processing apparatus <NUM> may be configured such that at least one of the operation unit <NUM>, the output unit <NUM>, and the auxiliary storage apparatus <NUM> is disposed as a separate apparatus outside the information processing apparatus <NUM> and is connected to the information processing apparatus <NUM>. The hardware configuration of the information processing apparatus <NUM> has been described above. Note that this hardware configuration described above with reference to <FIG> may also be applied to other apparatuses included in an information processing system <NUM>, which will be described later.

Next, an example of a configuration of the information processing system that generates a virtual viewpoint image according to the present embodiment is described below with reference to <FIG>. The information processing system <NUM> according to the present embodiment includes a plurality of imaging units <NUM>, a synchronization unit <NUM>, a three-dimensional shape estimation unit <NUM>, a storage unit <NUM>, a viewpoint generation unit <NUM>, an information processing apparatus <NUM>, a selection unit <NUM>, an image generation unit <NUM>, a display unit <NUM>, a time control unit <NUM>, and a video distribution apparatus <NUM>. Note that in the present embodiment, it is assumed by way of example that the generated virtual viewpoint image is a moving image, but this is merely an example, and the generated virtual viewpoint image is not limited to a moving image. That is, the present embodiment can be applied also to a virtual still viewpoint image. The processing units are described below unit by unit.

The imaging unit <NUM> is, for example, an imaging apparatus such as a camera. The synchronization unit <NUM> is an apparatus such as a time server, and outputs synchronization signals to the plurality of imaging units <NUM>. The plurality of imaging units <NUM> perform imaging of imaging areas with high synchronization accuracy with each other based on synchronization signals transmitted from the synchronization unit <NUM>. The plurality of imaging units <NUM> may be installed so as to surround an imaging area, for example, as shown in <FIG>. Note that the arrangement and number of imaging units <NUM> are not limited to the examples shown in <FIG>. The imaging area may be, for example, an outdoor area such as a sports stadiums, a field, a park, or the like, or an indoor area such as a gymnasium, a concert hall, a stage, a studio, or the like. Each imaging unit <NUM> outputs a captured image obtained as a result of imaging to the three-dimensional shape estimation unit <NUM>.

The three-dimensional shape estimation unit <NUM> generates a three-dimensional model representing a three-dimensional shape of a subject based on a plurality of captured images acquired from the plurality of imaging units <NUM>. An example of processing performed by the three-dimensional shape estimation unit <NUM> to generate a three-dimensional model is described below. The three-dimensional shape estimation unit <NUM> generates a foreground image and a background image using the acquired plurality of captured images.

Here, the foreground image is an image obtained by extracting an object area (a foreground area) from a captured image obtained by capturing by an imaging apparatus. The object extracted as the foreground area refers to a dynamic object (a moving object) that moves (its absolute position and shape can change) when the image thereof is captured in time series from the same direction. The object is, for example, a person such as a player or a referee in a field where a game is played, a ball in a ball game, or a singer, a performer, a moderator in a concert or entertainment, or the like. Furthermore, the object extracted as the foreground area may be, for example, an article used by a person.

The background image is an image of an area (a background area) different from at least the foreground object. More specifically, the background image is an image obtained by removing the foreground object from the captured image. The background image refers to an imaging target object that is stationary or continues to be nearly stationary when imaging is performed in time series from the same direction. More specifically, such an imaging target object is, for example, a stage of a concert, a stadium where an event such as a game is held, a structure such as a goal used in a ball game, a field, or the like. Note that the background area is an area different at least from the foreground object, and the area to be imaged as the background area may include, in addition to the object and the background, another object. The three-dimensional model data is data representing the above-described object in a three-dimensional shape.

An example of a method of generating a foreground image and a background image by the three-dimensional shape estimation unit <NUM> is described below. The three-dimensional shape estimation unit <NUM> compares a plurality of captured images obtained by capturing images at consecutive times by the plurality of imaging units <NUM>, and detects an area in which pixel values do not change. The three-dimensional shape estimation unit <NUM> determines the detected area as a background area, and generates a background image based on the detected area. Furthermore, the three-dimensional shape estimation unit <NUM> compares the generated background image and the captured image, and determines that an area in which differences in the image value are equal to or greater than a predetermined threshold value is a foreground area and generates a foreground image based on the detected foreground area.

An example of a method used by the three-dimensional shape estimation unit <NUM> to generate a three-dimensional model of an object corresponding to a foreground area is described below. Based on the foreground area and background area detected in the above-described process, the three-dimensional shape estimation unit <NUM> generates a silhouette image of the object in which each area is represented in binary values. In this process, by using captured images captured from a plurality of directions, the three-dimensional shape estimation unit <NUM> generates silhouette images of the object viewed from a plurality of directions. The three-dimensional shape estimation unit <NUM> generate a three-dimensional model, for example, by using a known technique such as a shape-from-silhouette method based on the plurality of silhouette images.

In the above example, the three-dimensional shape estimation unit <NUM> generates the foreground image, the background image, and the silhouette image, but the present disclosure is not limited to this example. For example, the imaging unit <NUM> or another apparatus connected to the imaging unit <NUM> may have a function of generating at least one of the foreground image, the background image, and the silhouette image. In this case, the imaging unit <NUM> or the apparatus outputs the generated image to the three-dimensional shape estimation unit <NUM>, and the three-dimensional shape estimation unit <NUM> generates the three-dimensional model using the acquired image. The three-dimensional shape estimation unit <NUM> outputs the generated three-dimensional model to the storage unit <NUM>.

The storage unit <NUM> is an apparatus such as a database server that stores and accumulates a group of data for use in generating a virtual viewpoint image. The three-dimensional model input from the three-dimensional shape estimation unit <NUM> is one of specific examples of the data. Other specific examples of the data are camera parameters representing a position, an orientation, optical characteristics, and the like of each imaging unit <NUM>. The storage unit <NUM> may store a background model, a background texture image, and/or the like in advance for use in drawing a background of a virtual viewpoint image. The storage unit <NUM> may acquire a captured image from the imaging unit <NUM> and store the acquired captured image.

The viewpoint generation units 5a to 5d each include an operation unit functioning as an input apparatus such as a joystick, and a display unit for displaying a virtual viewpoint image corresponding to a viewpoint being operated. Each of the viewpoint generation units 5a to 5d generates viewpoint information representing the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint based on the input from the operation unit, and inputs the resultant viewpoint information to the information processing apparatus <NUM>. In addition to the information representing the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint, the viewpoint information may include information corresponding to internal parameters of the camera, such as information regarding the focal length, the angle of view, and/or the like. In the example described above with reference to <FIG>, it is assumed that there are four viewpoint generation units 5a to 5d, but the number thereof may be three or less or five or more.

In the following description, when there is no particular need to distinguish among the viewpoint generation units 5a to 5d, an expression "viewpoint generation unit <NUM>" will be used.

The information processing apparatus <NUM> acquires viewpoint information from the viewpoint generation unit <NUM>, and outputs it to a first image generation unit 8a and a second image generation unit 8b, which will be described later. Furthermore, the information processing apparatus <NUM> receives a viewpoint information selection instruction from the selection unit <NUM>, which will be described later. The configuration of the information processing apparatus <NUM> will be described later.

The selection unit <NUM> is an input apparatus such as a mouse, a keyboard, a switch, or the like. The selection unit <NUM> transmits an instruction, to the information processing apparatus <NUM>, to select one or more pieces of viewpoint information from among the viewpoint information output by the viewpoint generation units <NUM>.

The first image generation unit 8a and the second image generation unit 8b each are an apparatus such as a rendering server which generates a virtual viewpoint image based on the viewpoint information acquired from the information processing apparatus <NUM>. In the generation of the virtual viewpoint images, the first image generation unit 8a and the second image generation unit 8b acquire data necessary for generating the virtual viewpoint images from the storage unit <NUM> based on the time information included in the viewpoint information. Here, the time information is information regarding the time related to the virtual viewpoint image corresponding to the viewpoint information. More specifically, it can be a time code or the like representing the time of capturing the image by the imaging unit <NUM>. In the present embodiment, time information is similarly added to the data such as the three-dimensional model and the foreground image stored in the storage unit <NUM>. By specifying time information, the necessary data can be acquired from the storage unit <NUM>. Note that the time information is not limited to the time information indicating the time of capturing an image by the imaging unit <NUM>, and the time information may represent, for example, a relative time from a predetermined time, a frame number of a moving image, or the like.

The first image generation unit 8a and the second image generation unit 8b perform rendering processing using the three-dimensional model and the foreground image to generate a virtual viewpoint image of an object representing the appearance viewed from the virtual viewpoint. Furthermore, the first image generation unit 8a and the second image generation unit 8b generate a background image using the background model and the background texture, and combine the background image with the virtual viewpoint image of the object to obtain a virtual viewpoint image with a background.

According to a plurality of pieces of viewpoint information generated by operating a plurality of viewpoint generation units <NUM>, the first image generation unit 8a generates a plurality of virtual viewpoint images based on predetermined quality conditions. Here, it is assumed that the conditions regarding the quality of the virtual viewpoint image in the present embodiment are conditions regarding the resolution of the generated image and the frame rate of the moving image. It is assumed that the higher the resolution and the higher the frame rate, the higher the quality of the virtual viewpoint image. The first image generation unit 8a outputs the generated virtual viewpoint image to the multiple-image display unit 9a. Note that, in the present embodiment, the first image generation unit 8a and the second image generation unit 8b are assumed to be different apparatuses.

The second image generation unit 8b acquires one or more pieces of viewpoint information selected by the selection unit <NUM> from among a plurality of pieces of viewpoint information, and generates a virtual viewpoint image. It is assumed that the virtual viewpoint image generated in the above process has higher quality than the virtual viewpoint image generated by the first image generation unit 8a.

The second image generation unit 8b outputs the generated virtual viewpoint image to the output image display unit 9b and the video distribution apparatus <NUM>. That is, the first image generation unit 8a generates the plurality of virtual viewpoint images so as to satisfy conditions for generating virtual viewpoint images having at least one of the predetermined resolution and the predetermined frame rate, and the second image generation unit 8b generates the plurality of virtual viewpoint images so as to satisfy conditions for generating virtual viewpoint images having at least one of the resolution higher than the predetermined resolution and the frame rate higher than the predetermined frame rate.

The multiple-image display unit 9a and the output image display unit 9b respectively display virtual viewpoint images obtained from the first image generation unit 8a and the second image generation unit 8b. The time control unit <NUM> controls time information such that the time information used by the first image generation unit 8a in generating virtual viewpoint images is synchronized with the time information used by the second image generation unit 8b in generating virtual viewpoint images. The video distribution apparatus <NUM> distributes the virtual viewpoint image transmitted from the second image generation unit 8b via, for example, television broadcasting or the Internet.

Next, a functional configuration of the information processing apparatus <NUM> is described below with reference to <FIG>. The information processing apparatus <NUM> includes a viewpoint information acquisition unit <NUM>, a selection acceptance unit <NUM>, and a viewpoint information output unit <NUM>. The viewpoint information acquisition unit <NUM> acquires viewpoint information transmitted from the viewpoint generation unit <NUM> and outputs the acquired viewpoint information to the viewpoint information output unit <NUM>. In a case where a plurality of viewpoint generation units <NUM> are provided, the viewpoint information acquisition unit <NUM> acquires a plurality of pieces of viewpoint information and outputs them to the viewpoint information output unit <NUM>. The selection acceptance unit <NUM> accepts a viewpoint information selection instruction from the selection unit <NUM> and outputs information indicating the accepted instruction to the viewpoint information output unit <NUM>.

Based on the acquired viewpoint information and the selection instruction information, the viewpoint information output unit <NUM> outputs viewpoint information to the first image generation unit 8a and the second image generation unit 8b. In this process, the viewpoint information output unit <NUM> outputs a plurality of pieces of viewpoint information to the first image generation unit 8a, and outputs one or more pieces of viewpoint information selected from among the plurality of pieces of viewpoint information according to the selection instruction to the second image generation unit 8b. As described above, the viewpoint information output unit <NUM> outputs the viewpoint information to the first image generation unit 8a and the second image generation unit 8b, and performs control such that the virtual viewpoint images generated by processing units described above are displayed on the multiple-image display unit 9a and the output image display unit 9b.

Next, processing performed by the information processing system <NUM> according to the present embodiment is described below with reference to <FIG> is a flowchart of processing performed by the information processing apparatus <NUM>. The processing shown in <FIG> is performed by the CPU <NUM> of the information processing apparatus <NUM> by reading and executing programs stored in the RAM <NUM> and the ROM <NUM>. In the following description, processing steps are simply denoted as S. The processing is started when viewpoint information is input from the viewpoint generation unit <NUM>. The processing from S201 to S205 is executed repeatedly for each video frame of the generated virtual viewpoint image. That is, in a case where a moving image to be generated is at <NUM> FPS (frames per second), the processing from S201 to S205 is repeated every <NUM>/<NUM> of a second.

In S201, the viewpoint information acquisition unit <NUM> acquires a plurality of pieces of viewpoint information generated by operating the operation units of the plurality of viewpoint generation units <NUM>. In S202, the viewpoint information output unit <NUM> outputs the acquired plurality of pieces of viewpoint information to the first image generation unit 8a.

In S203, the selection acceptance unit <NUM> determines whether an instruction to select viewpoint information is newly acquired from the selection unit <NUM>. For example, when a selection instruction is accepted for the first time or when a selection instruction is accepted which instructs to select viewpoint information different from the viewpoint information previously selected in S204 in the iteration loop from S201 to S205, the processing flow proceeds to S204. On the other hand, in a case where no new selection instruction is acquired, or in a case where there is no difference from the viewpoint information from that selected in the S204 previously in the iteration loop from S201 to S205, the processing flow proceeds to S205.

In S204, the viewpoint information output unit <NUM> outputs, to the second image generation unit 8b, one or more pieces of viewpoint information selected by the selection acceptance unit according to the selection instruction acquired by the selection acceptance unit <NUM> from among the plurality of pieces of viewpoint information acquired by the viewpoint information acquisition unit <NUM>. In S205, the viewpoint information acquisition unit <NUM> determines whether acquisition of viewpoint information is completed, for example, based on an end instruction given from the viewpoint generation unit <NUM>.

In a case where it is determined that the acquisition is completed, the processing is ended, but in a case where the acquisition is not completed, the processing from S201 to S205 is repeated.

Next, processing performed by the first image generation unit 8a and processing performed by the second image generation unit 8b are described with reference to <FIG>. The processing shown in <FIG> is performed by the CPUs of the first image generation unit 8a and the second image generation unit 8b by reading and executing programs stored in the RAM and the ROM. It is assumed that the first image generation unit 8a and the second image generation unit 8b independently perform the processing shown in <FIG>. In the following description, when it is not necessary to distinguish among the first image generation unit 8a and the second image generation unit 8b, a simple expression "image generation unit <NUM>" will be used. When the image generation unit <NUM> acquires viewpoint information, the processing is started.

In S211, the image generation unit <NUM> acquires viewpoint information from the information processing apparatus <NUM>. In this processing, the first image generation unit 8a acquires a plurality of pieces of viewpoint information output in S202.

On the other hand, the second image generation unit 8b acquires the selected viewpoint information output in S204.

In S212, the image generation unit <NUM> acquires control information regarding time synchronization from the time control unit <NUM>. An example of control performed by the time control unit <NUM> is described. The plurality of viewpoint generation units <NUM> are operated independently. For this reason, the timing of acquiring viewpoint information is not necessarily equal among the viewpoint information acquisition units <NUM>, and there is a possibility that there is a difference in the time information included in the plurality of pieces of viewpoint information. In view of the above, the time control unit <NUM> outputs, to the image generation units <NUM>, synchronization signals as control information for synchronizing the times represented by the time information included in the plurality of pieces of viewpoint information. The image generation units <NUM> synchronize the times of the plurality of pieces of viewpoint information based on the synchronization signals. This enables the image generation units <NUM> to acquire a plurality of pieces of viewpoint information at synchronized times. Note that the time control method performed by the time control unit <NUM> is not limited to that described above. For example, a signal specifying time information for acquiring data may be output to the image generation unit <NUM>. This makes it possible for the image generation unit <NUM> to acquire data corresponding to the time specified by the signal acquired from the time control unit <NUM> from the storage unit <NUM>.

In S213, based on the time information included in the viewpoint information, the image generation unit <NUM> acquires data used for generating a virtual viewpoint image from the storage unit <NUM>. Examples of the acquired data are a three-dimensional model of an object, a foreground image, a background model, a background texture, or the like. In S212, since the time information included in the plurality of pieces of viewpoint information is synchronized, it is sufficient to acquire data corresponding to one piece of time information even in a case where a plurality of virtual viewpoint images corresponding to a plurality of pieces of viewpoint information are generated. This makes it possible to reduce the processing load on the image generation unit <NUM> in reading the data from the storage unit <NUM>, and it is also possible to reduce the amount of data transmitted in the reading.

In S214, the image generation unit <NUM> generates a virtual viewpoint image based on the viewpoint information and the data acquired from the storage unit <NUM>. An example of generating a virtual viewpoint image is described. Here, it is assumed that the image generation unit <NUM> has already obtained, from the storage unit <NUM>, a three-dimensional model corresponding to certain time information and a foreground image for coloring the three-dimensional model. In a case where there are a plurality of pieces of viewpoint information related to virtual viewpoint images to be generated, the plurality of virtual viewpoint images are generated by coloring the three-dimensional model based on the position of the virtual viewpoint represented by each piece of viewpoint information and the line-of-sight direction from the virtual viewpoint.

In S214, the first image generation unit 8a generates a virtual viewpoint image with quality lower than the quality of the virtual viewpoint image generated by the second image generator 8b. For example, in a case where four pieces of viewpoint information are acquired, the first image generation unit 8a generates four virtual viewpoint images with a resolution of Half HD (<NUM> × <NUM> pixels). When one of the four pieces of viewpoint information is selected, the second image generation unit 8b generates virtual viewpoint image with a resolution of a Full HD (<NUM> × <NUM> pixels).

In another example, when four pieces of viewpoint information are acquired, the first image generation unit 8a generates four virtual viewpoint images in the form of moving images with a frame rate of <NUM> FPS. When one of the four pieces of viewpoint information is selected, the second image generation unit 8b generates one virtual viewpoint image in the form of moving images with a frame rate of <NUM> FPS.

In S215, the first image generation unit 8a outputs the generated virtual viewpoint images to the multiple-image display unit 9a. The second image generation unit 8b outputs the generated virtual viewpoint image to the output image display unit 9b and the video distribution apparatus <NUM>. <FIG> illustrates an example of a manner in which images are displayed on the multiple-image display unit 9a and the output image display unit 9b for a case where there are four pieces of viewpoint information. As shown in <FIG>, the multiple-image display unit 9a simultaneously displays a plurality of virtual viewpoint images A to D corresponding to the plurality of pieces of viewpoint information. The output image display unit 9b displays the virtual viewpoint image A corresponding to the viewpoint information selected from among the four pieces of viewpoint information. Here, it is assumed that the virtual viewpoint image A displayed on the output image display unit 9b has higher quality than the quality of the plurality of virtual viewpoint images A to D displayed on the multiple-image display unit 9a.

A user selects a virtual viewpoint image to be supplied to the video distribution apparatus <NUM> by operating one of a plurality of switches (switcher) of the selection unit <NUM> shown in <FIG> while viewing a plurality of virtual viewpoint images displayed on the multiple-image display unit 9a. Since the selected virtual viewpoint image is displayed on the output image display unit 9b, the user can view, for the confirmation, the virtual viewpoint image with high quality to be distributed.

Although there are four pieces of viewpoint information in the above example, the number of pieces of viewpoint information is not limited to four. For example, in a case where there are <NUM> pieces of viewpoint information, the first image generation unit 8a generates <NUM> virtual viewpoint images with a further lowered resolution of <NUM> × <NUM> pixels. This makes it possible to reduce the processing load associated with rendering on the first image generation unit 8a and the processing load associated with displaying on the multiple-image display unit 9a. In the above example, one piece of viewpoint information is selected by the selection unit <NUM>, but two or more pieces of viewpoint information may be selected.

In S216, the image generation unit <NUM> determines whether acquisition of viewpoint information is completed. In a case where it is determined that the acquisition is completed, the processing is ended, but if the acquisition is not completed, the processing from S211 to S216 is repeated.

In the present embodiment, as described above, the information processing apparatus <NUM> controls displaying by outputting the viewpoint information to the image generation unit <NUM> such that the virtual viewpoint images are displayed on the multiple-image display unit 9a and the output image display unit 9b. The user can easily confirm viewpoint information to be used for distribution from among a plurality of pieces of viewpoint information obtained via a plurality of operations while suppressing the processing load associated with displaying virtual viewpoint images.

Furthermore, the present disclosure makes it possible to adequately generate a virtual viewpoint image corresponding to the selected virtual viewpoint while suppressing the processing load when selecting a specific virtual viewpoint from among a plurality of virtual viewpoints.

In the first embodiment described above, it is assumed by way of example that a plurality of virtual viewpoint images are displayed in the form of tiles on the multiple-image display unit 9a, which is one of display apparatuses. However, the present disclosure is not limited to this. For example, a plurality of virtual viewpoint images may be displayed separately on different display apparatuses.

Furthermore, in the first embodiment, the viewpoint generation unit <NUM> includes the operation unit and the display unit, and the viewpoint generation unit <NUM> is operated by a user. However, the present disclosure is not limited to this. For example, the viewpoint generation unit <NUM> may be an apparatus that automatically generates the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint in accordance with the time. In this configuration, the operation unit used by a user to perform an inputting operation and a display unit for displaying a virtual viewpoint image are not necessary.

Furthermore, in the first embodiment, the first image generation unit 8a and the second image generation unit 8b are assumed to be different apparatuses, but the present disclosure is not limited to this configuration. For example, the first image generation unit 8a and the second image generation unit 8b may be the same apparatus. In this case, this apparatus is assumed to be capable of switching the quality of the virtual viewpoint image generated in response to the timing of generating the virtual viewpoint image, such as the timing at which the viewpoint information is selected by the selection unit <NUM>. Note that, in this configuration, a plurality of virtual viewpoint images corresponding to a plurality of pieces of viewpoint information based on a plurality of operations cannot be generated when the apparatus is functioning as the second image generation unit 8b. Instead, virtual viewpoint images which have already been generated may be displayed on the multiple-image display unit 9a.

Furthermore, in the first embodiment, the viewpoint selection unit <NUM>, the first image generation unit 8a, and the second image generation unit 8b are separate components, but the present disclosure is not limited to this configuration.

Furthermore, in the first embodiment, the time control unit <NUM> outputs control information to the first image generation unit 8a and the second image generation unit 8b, respectively, but the present disclosure is not limited to this. For example, as shown in <FIG>, the time control unit <NUM> may output control information to the information processing apparatus <NUM>. In this case, the information processing apparatus <NUM> outputs the time information to the image generation units <NUM> together with the viewpoint information. In this configuration, it is not necessary to change the connection destination of the time control unit <NUM> when a change occurs in the configuration such as the number of the image generation units <NUM>. Furthermore, in this configuration, since the time information and viewpoint information are all collected once in the information processing apparatus <NUM>, such information may be stored in an internal storage unit of the information processing apparatus <NUM> or an external storage unit. This makes it possible to store even viewpoint information that is not selected at the time of distributing an image.

In this case, it is possible to generate a virtual viewpoint image based on the store information, and edit and use it as a virtual viewpoint image viewed from another viewpoint. The time control unit <NUM> may be included in the information processing apparatus <NUM>.

The present disclosure may be realized by supplying a program for realizing one or more functions of the one or more embodiments described above to a system or an apparatus via a network or a storage medium, and reading and executing the program by one or more processors of a computer in the system or the apparatus. The present disclosure may also be realized by a circuit (for example, an ASIC circuit) for realizing one or more functions.

While the present invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments.

Claim 1:
An information processing apparatus (<NUM>) comprising:
acquisition means (<NUM>) configured to acquire a plurality of pieces of viewpoint information, each of the plurality of pieces of viewpoint information representing a position of a virtual viewpoint and a line-of-sight direction from the virtual viewpoint used for generating a virtual viewpoint image based on a plurality of captured images obtained by imaging performed by a plurality of imaging apparatuses (<NUM>);
first output means (<NUM>) arranged to output the plurality of pieces of viewpoint information acquired from the acquisition means (<NUM>) to first generation means (8a) configured to generate a plurality of virtual viewpoint images based on a condition of first quality according to the plurality of pieces of viewpoint information, wherein the plurality of virtual viewpoint images corresponds to a same time and is displayed simultaneously on a display unit (9a); and
second output means (<NUM>) arranged to output a piece of viewpoint information selected by a user from among the plurality of pieces of viewpoint information to second generation means (8b) configured to generate a virtual viewpoint image based on a condition of second quality higher than the first quality according to the pieces of viewpoint information, wherein the selected piece of viewpoint information is a piece of viewpoint information corresponding to a virtual viewpoint image selected by the user from among the plurality of virtual viewpoint images displayed on the display unit.