Control device, imaging system, and 3D model data generation method

There is provided a control device, an imaging system, and a 3D model data generation method capable of appropriately extracting a subject from a moving image. The control device includes a control unit that performs control to synchronize an imaging timing at which an imaging device performs imaging with a display timing at which a display device located in an imaging range of the imaging device displays a first moving image, and an image processing unit that performs foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing. The second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized between the second moving image and the third moving image. The present technology can be applied to, for example, an imaging system for capturing a moving image used for generating a 3D model.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2021/001159 (filed on Jan. 15, 2021) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2020-013883 (filed on Jan. 30, 2020), which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present technology relates to a control device, an imaging system, and a 3D model data generation method, and more particularly to a control device, an imaging system, and a 3D model data generation method capable of appropriately extracting a subject from a moving image.

BACKGROUND ART

There is a technology of generating a 3D model of a subject from moving images captured from multiple viewpoints and generating a virtual viewpoint image of the 3D model according to an arbitrary viewpoint position to provide an image of a free viewpoint. Such a technology is also referred to as a volumetric capture technology or the like.

For example, Patent Document 1 describes a technology of generating a 3D model using a method such as visual hull in which a three-dimensional shape of a subject is cut out on the basis of a plurality of captured images obtained by imaging from different directions.

CITATION LIST

Patent Document

Patent Document 1: International Patent Application Publication No. 2018/150933

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

In the generation of a 3D model, it is necessary to separate the subject and the background appearing in each of the captured images captured from multiple viewpoints. For example, by taking the difference between a background image obtained by imaging only the background in advance and a captured image obtained by imaging in a state where a subject such as a person is actually present, the subject and the background appearing in each captured image are separated.

Incidentally, in imaging for generating a 3D model, a situation is conceivable in which a person as a subject displays a moving image on a display device to confirm the content of acting or displays lines and lyrics on a prompter for reference.

However, in a case where, at the time of actual imaging in a state where a person is present, there is an object that was not present at the time of background imaging in the imaging space, the object itself is also treated as a subject. For example, in a case where a moving image is displayed on a display device arranged in an imaging space, the display of the moving image is also separated as a subject together with the person, and is used for generating a 3D model. Accordingly, at the time of actual imaging, it is difficult for a person as a subject to check the content of acting with reference to the display of a display device, or to check lines or lyrics with reference to the display of a prompter.

The present technology has been made in view of such a situation, and aims to appropriately extract a subject from a moving image.

Solutions to Problems

A control device according to one aspect of the present technology is a control device including a control unit that performs control to synchronize an imaging timing at which an imaging device performs imaging with a display timing at which a display device located in an imaging range of the imaging device displays a first moving image, and an image processing unit that performs foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, in which the second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized between the second moving image and the third moving image.

An imaging system according to one aspect of the present technology is an imaging system including: a plurality of imaging devices; at least one display device; a control unit that performs control to synchronize an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which the display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image; and an image processing unit that performs foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, in which the second moving image and the third moving image both include at least a part of the first moving image.

A 3D model data generation method according to one aspect of the present technology is a 3D model data generation method including: synchronizing an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which a display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image, and imaging by a plurality of imaging devices; generating difference data by performing foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; and generating 3D model data on the basis of the difference data.

In the control device according to one aspect of the present technology, control is performed to synchronize an imaging timing at which an imaging device performs imaging with a display timing at which a display device located in an imaging range of the imaging device displays a first moving image, and foreground/background difference processing is performed on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing. The second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized between the second moving image and the third moving image.

In the imaging system according to one aspect of the present technology, control is performed to synchronize an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which a display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image, and foreground/background difference processing is performed on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing. The second moving image and the third moving image both include at least a part of the first moving image.

In the 3D model data generation method according to one aspect of the present technology, imaging is performed by a plurality of imaging devices while synchronizing an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which a display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image, difference data is generated by performing foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing and 3D model data is generated on the basis of the difference data.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, a mode for carrying out the present technology will be described. The description will be given in the following order.1. Overview of imaging system2. Configuration of imaging system3. Operation of imaging system4. Processing when performing monitoring display5. Modification6. Configuration example of computer

<1. Overview of Imaging System>

FIG.1is a diagram illustrating a configuration example of an imaging system1to which the present technology is applied.

The imaging system1inFIG.1includes a control device11, N (N>1) cameras12-1to12-N, and a monitor13. The cameras12-1to12-N and the monitor13are arranged in the same space such as an imaging studio.

The control device11includes, for example, a computer, a server device, or the like, and controls an imaging timing at which the cameras12-1to12-N perform imaging and a display timing at which the monitor13displays a moving image.

Specifically, the control device11generates a synchronization signal and supplies the synchronization signal to each of the cameras12-1to12-N and the monitor13. An imaging timing at which the cameras12-1to12-N perform imaging and a display timing at which the monitor13displays a moving image are controlled on the basis of the synchronization signal supplied from the control device11. Additionally, the control device11also reproduces a moving image displayed on the monitor13.

Moreover, the control device11generates, for example, a 3D model of an object in which a person appearing as a subject is an object from a plurality of captured moving images supplied from each of the cameras12-1to12-N.

Data of the 3D model of the object includes, for example, moving image data of the object generated from the captured moving images obtained by the cameras12-1to12-N and 3D shape data representing the 3D shape of the object. The 3D shape data is represented by, for example, a depth image corresponding to a captured moving image captured by the plurality of cameras12-1to12-N, a point cloud representing a three-dimensional position of the object by a set of points, a polygon mesh represented by connections between vertices, or the like.

The cameras12-1to12-N are imaging devices that capture a subject to be extracted as an object and a background thereof under the control of the control device11. The cameras12-1to12-N supply captured moving images obtained by imaging to the control device11. The cameras12-1to12-N are arranged so as to surround the subject, for example, to image the same subject. The captured moving images supplied from the cameras12-1to12-N are multi-view moving images including a plurality of captured moving images having different viewpoints for capturing the subject.

In the following description, in a case where it is not necessary to distinguish among the cameras12-1to12-N, the cameras are simply referred to as a camera12. The same applies to other configurations provided in a plurality of units.

The monitor13is a display device that displays a reference moving image as a first moving image under the control of the control device11. The reference moving image is a moving image to be referred to by a person as a subject at the time of actual imaging. For example, at the time of actual imaging, a moving image serving as a model of an action of a person who is a subject is displayed on the monitor13as a reference moving image.

Additionally, the monitor13can also perform monitoring display in which a monitoring image is displayed as a reference moving image. The monitoring image is a captured moving image captured in real time by any one of the cameras12-1to12-N. In the monitoring display, a person as a subject can perform a predetermined action while confirming his/her movement.

Additionally, the monitor13can also display a moving image representing lines and lyrics as a reference moving image. By displaying the moving image representing lines and lyrics, a person as a subject can act while confirming the lines or sing while confirming the lyrics.

Note that the monitor13may be formed by a device such as a smartphone or a tablet terminal (portable terminal) including a display.

Communication between the control device11and the cameras12-1to12-N and communication between the control device11and the monitor13may be directly performed via a cable or the like, or may be performed via a predetermined network such as a local area network (LAN) or the Internet. Additionally, the communication may be wired communication or wireless communication.

Next, an outline of 3D model generation executed by the control device11will be described with reference toFIG.2.

FIG.2illustrates an example in which a person performing a predetermined operation is imaged as a subject Ob1using three cameras12. As illustrated on the left side ofFIG.2, the three cameras12-1to12-3arranged to surround the subject Ob1image the subject Ob1.

3D modeling is performed by the control device11using the captured moving images obtained from the plurality of cameras12arranged at different positions, and a 3D model MO1of the subject Ob1is generated as illustrated in the center ofFIG.2. The 3D model MO1is generated by, for example, a method such as visual hull in which a three-dimensional shape is cut out using captured moving images obtained by imaging the subject Ob1from different directions.

The data (3D model data) of the 3D model of the subject generated as described above is transmitted to a device on the reproduction side and reproduced. That is, in the device on the reproduction side, the 3D model is rendered on the basis of the 3D model data, whereby a 3D shape video is displayed on a viewing device. InFIG.2, a display D1and a head mounted display (HMD) D2are illustrated as the viewing device used by the viewer.

Incidentally, in the generation of the 3D model, it is necessary to separate the subject and the background appearing in each captured moving image and extract data of only the subject. As a method for extracting data of only a subject, keying using a green screen background is known. Additionally, the subject and the background can be separated by obtaining a difference between a background moving image that is a moving image in which only the background is imaged and an actual moving image that is a moving image in which the subject and the background are imaged by foreground/background difference processing.

The imaging system1inFIG.1performs background imaging that is imaging of a background moving image and actual imaging that is imaging of an actual moving image, and obtains a difference between the background moving image and the actual moving image by foreground/background difference processing, thereby separating a subject and a background. Here, actual imaging is imaging in a situation where the subject is present, whereas background imaging is imaging in a situation where no subject is present. The actual moving image is a second moving image captured by the camera12at a predetermined first timing (timing of actual imaging), and the background moving image is a third moving image captured by the camera12at a second timing (timing of background imaging) different from the first timing.

FIG.3is a plan view illustrating an example of a situation when the imaging system1performs background imaging with N=8.

In the example ofFIG.3, eight cameras12-1to12-8are annularly arranged at substantially equal intervals so as to face a central part of an imaging space such as a room. The camera12-1and the camera12-5, the camera12-2and the camera12-6, the camera12-3and the camera12-7, and the camera12-4and the camera12-8are arranged to face each other.

The monitor13is arranged between the camera12-5and the camera12-6with the front of the monitor13facing the center. Here, the monitor13is located in an imaging range of the camera12-2. As illustrated in balloon #1, a moving image in which a person is dancing is displayed on the monitor13as a reference moving image.

In this manner, background imaging is performed in a state where no subject is present. In the background moving image captured by each of the cameras12-1to12-8, another camera12present in the angle of view, the monitor13, a wall of the imaging space, a ceiling, a floor, and the like appear as the background.

For example, as illustrated in balloon #2, in the background moving image captured by the camera12-2, the camera12-6appears in the vicinity of the center, and the monitor13appears on the left side of the camera12-6. Since the reference moving image is displayed on the monitor13at the time of background imaging, the reference moving image showing a state where a person is dancing is also displayed on the monitor13appearing in the background moving image captured by the camera12-2.

FIG.4is a plan view illustrating an example of a situation when the imaging system1performs actual imaging with N=8.

As illustrated inFIG.4, the arrangement of the cameras12-1to12-8and the monitor13at the time of actual imaging is similar to the arrangement at the time of background imaging described with reference toFIG.3.

Actual imaging is performed in a situation where a person as a subject Ob11is present in the central part of the imaging space. In the actual moving image captured by each of the cameras12-1to12-8, the subject Ob11present in the angle of view and the background appear. Note that at the time of actual imaging, as illustrated in balloon #11, the same moving image as the moving image displayed at the time of background imaging is displayed on the monitor13as the reference moving image.

As illustrated in balloon #12, in the actual moving image captured by the camera12-2, the subject Ob11appears in the vicinity of the center, and the monitor13appears at a position on the left back side of subject Ob11. Since the reference moving image is displayed on the monitor13at the time of actual imaging, the reference moving image showing a state where a person is dancing is also displayed on the monitor13appearing in the actual moving image captured by the camera12-2.

As described above, actual imaging is performed similarly to background imaging except that the imaging is performed by arranging the subject in the central part of the imaging space.

In a case where the subject and the background are separated by foreground/background difference processing, if an object that was not present at the time of background imaging is in the imaging space at the time of actual imaging, the object itself is also treated as a subject.

For example, regarding the time of actual imaging and the time of background imaging, if the reference moving image of the monitor13appearing in the actual moving image captured by the camera12-2is different from the reference moving image displayed on the monitor13at the time of background imaging, the reference moving image of the monitor13appearing in the actual moving image is also extracted as a subject.

Against this background, the control device11of the imaging system1generates a synchronization signal and supplies the synchronization signal to the cameras12-1to12-8and the monitor13, thereby performing control so that imaging timings of the cameras12-1to12-8and a display timing of the monitor13are synchronized with each other.

FIG.5is a diagram illustrating an example of control of the imaging timing and the display timing based on a synchronization signal.

At the time of each of background imaging and actual imaging, for example, a synchronization signal as illustrated on the upper side ofFIG.5is supplied to each of the camera12and the monitor13. The synchronization signal illustrated inFIG.5is a pulse signal that alternately repeats 0 (low) and 1 (high) in a predetermined cycle. In the example ofFIG.5, the synchronization signal rises at times t1, t3, t5, t7, . . . , and falls at times t2, t4, t6, t8, . . . .

The camera12performs exposure (imaging) for acquiring a frame image included in a moving image (actual moving image or background moving image) at each timing of times t1, t3, t5, and t7which are the rising times of the synchronization signal.

Additionally, the monitor13sequentially displays one or more frame images included in the reference moving image at each timing of times t1, t3, t5, and t7, which are the rising times of the synchronization signal. As described above, the reference moving image is displayed during each of the actual imaging and the background imaging.

Note that the control device11decodes the frame image prior to the update timing (display timing) of the frame image displayed on the monitor13. For example, as illustrated inFIG.5, a frame image displayed at the timing of time t1is decoded before time t1. Similarly, the frame image displayed at each timing of the times t3, t5, and t7is decoded before each timing.

At each of the time of actual imaging and the time of background imaging, the control device11generates a synchronization signal and supplies the synchronization signal to the cameras12-1to12-8and the monitor13, so that at least a part of the same frame image of the reference moving image appears in frame images of the same frame number of the actual moving image and the background moving image.

Since the same frame image of the reference moving image is included in the frame images of the same frame number of the actual moving image and the background moving image, when the subject and the background are separated by the foreground/background difference processing, the region of the reference moving image displayed on the monitor13in the actual moving image can be treated as the background. As a result, it is possible to prevent the reference moving image on the monitor13appearing in the actual moving image from being extracted as a subject. Consequently, the imaging system1can perform imaging while providing a reference moving image to a person as a subject in imaging for 3D model generation.

Note that while the imaging by the camera12and the display of the reference moving image by the monitor13are performed at the rising timing of the synchronization signal in the above-described example, the imaging by the camera12and the display of the reference moving image by the monitor13may be performed at the falling timing of the synchronization signal.

FIG.6is a diagram illustrating an example of the foreground/background difference processing.

The actual moving image illustrated on the left side ofFIG.6is the actual moving image captured by the camera12-2described with reference toFIG.4. Additionally, the background moving image illustrated in the center ofFIG.6is the background moving image captured by the camera12-2described with reference toFIG.3.

By obtaining the difference between the actual moving image and the background moving image, the control device11extracts data of a region in which the subject Ob11appears as difference data as illustrated on the right side ofFIG.6.

Specifically, the control device11compares pixel values of a pixel included in the actual moving image with a pixel of the background moving image at the same pixel position as the pixel included in the actual moving image, and extracts a pixel in the actual moving image in which the difference between the pixel values is equal to or greater than a predetermined threshold as a pixel of the foreground. Such comparison of pixel values is performed for all the pixels included in the actual moving image, and the pixels of the foreground, that is, the pixels included in the subject are generated as the difference data.

The comparison of the pixel values may be performed by comparing RGB values, or may be performed by comparing HSV (hue, saturation, value) values obtained by converting RGB values. In the comparison using the HSV value, it is possible to perform control such that a change in the pixel value in which only the brightness changes and the color (hue) does not change, such as the presence or absence of a shadow, is not extracted as a change. Additionally, difference data may be generated by comparing pixel values of not only one pixel to be compared but also pixels surrounding the pixel to be compared. The threshold used to determine the pixel of the foreground can be appropriately set according to the imaging environment.

The foreground/background difference processing as described above is performed on the actual moving image and the background moving image captured by the plurality of cameras12, and difference data of the moving images is generated for each camera12.

As described above, according to the imaging system1, it is possible to appropriately extract the subject from the actual moving image in which the monitor13displaying the reference moving image appears. Even if the image displayed on the monitor13changes during imaging, the foreground/background difference processing can be performed without being affected by the change.

<2. Configuration of Imaging System>

FIG.7is a block diagram illustrating a configuration example of the imaging system1.

The imaging system1includes the control device11, N cameras12-n(n=1 to N), and the monitor13. The cameras12-nare provided with an imaging unit51. Additionally, the monitor13is provided with a display unit52.

The control device11includes an operation unit31, a control unit32, a reproduction unit33, a synchronization signal generation unit34, a storage control unit35, a storage unit36, and an image processing unit37.

The operation unit31includes a touch panel monitor, a keyboard, a mouse, a controller, a remote operation device, and the like. The operation unit31detects an operation by an image capturing person or a person as a subject and supplies information indicating the content of the detected operation to the control unit32.

For example, the operation unit31detects an operation for starting background imaging, an operation for starting actual imaging, an operation for generating 3D model data, and the like. An operation for designating a reference moving image displayed on the monitor13at the time of background imaging and at the time of actual imaging may be detected by the operation unit31.

The control unit32controls each configuration of the control device11and the cameras12-non the basis of the information supplied from the operation unit31.

For example, in a case where an operation for starting actual imaging or background imaging is detected by the operation unit31, the control unit32controls the imaging unit51of the cameras12-nand performs various settings such as setting of a frame rate and resolution of a moving image. Additionally, the control unit32performs control to cause the reproduction unit33to display a predetermined reference moving image on the monitor13. Moreover, the control unit32performs control to cause the synchronization signal generation unit34to generate a synchronization signal.

On the other hand, in a case where an operation for generating 3D model data is detected by the operation unit31, the control unit32performs control to cause the image processing unit37to perform 3D model data generation processing.

The reproduction unit33acquires and reproduces the reference moving image designated by the control unit32from reference moving images stored in the storage unit36. The reproduction unit33decodes the reference moving image prior to the display timing of the monitor13, and supplies the reference moving image obtained by the decoding to the display unit52. The decoding of the reference moving image is performed by a method corresponding to an encoding method used when the reference moving image is stored in the storage unit36. Note that the frame rate of the reference moving image is assumed to be the same as the frame rate when the imaging unit51performs actual imaging or background imaging.

The synchronization signal generation unit34generates the synchronization signal described with reference toFIG.5under the control of the control unit32, and supplies the synchronization signal to each of the imaging unit51and the display unit52.

The storage control unit35encodes the moving image supplied from the imaging unit51by a predetermined encoding method such as the MPEG2 method or the AVC method, for example, and stores the encoded moving image in the storage unit36.

The storage unit36includes an auxiliary storage device including an internal or external storage such as a semiconductor memory. The storage unit36includes a reproduction data storage unit41and an imaging data storage unit42.

The reproduction data storage unit41stores a reference moving image reproduced by the reproduction unit33.

The imaging data storage unit42stores the background moving image and the actual moving image supplied from the storage control unit35. The imaging data storage unit42also stores information regarding the cameras12-nused for capturing moving images, together with the background moving image and the actual moving image. The information regarding the cameras12-nincludes, for example, internal parameters and external parameters of the cameras12-n.

Note that the reproduction data storage unit41and the imaging data storage unit42may be provided as separate storage units in the control device11.

The image processing unit37generates 3D model data of the subject under the control of the control unit32. The image processing unit37includes a foreground/background difference processing unit43and a 3D model data generation unit44.

The foreground/background difference processing unit43acquires the actual moving image and the background moving image from the storage unit36for each camera12. The foreground/background difference processing unit43performs foreground/background difference processing using the actual moving image and the background moving image to generate difference data. The difference data generated by the foreground/background difference processing unit43is supplied to the 3D model data generation unit44.

The 3D model data generation unit44acquires information regarding each camera12from the storage unit36. The 3D model data generation unit44generates 3D model data of the subject using a method such as visual hull on the basis of the difference data for each camera12and the information regarding each camera12supplied from the foreground/background difference processing unit43. The 3D model data of the subject generated by the 3D model data generation unit44is stored in the storage unit36or output to an external device.

Note that the image processing unit37may be provided not as a part of the control device11but as an image processing device different from the control device11.

The imaging unit51of the cameras12-ncaptures the actual moving image or the background moving image at an imaging timing based on the synchronization signal supplied from the synchronization signal generation unit34. The moving image acquired by capturing the actual moving image or the background moving image is supplied to the storage control unit35of the control device11.

The display unit52of the monitor13displays the reference moving image supplied from the reproduction unit33at a display timing based on the synchronization signal supplied from the synchronization signal generation unit34.

<3. Operation of Imaging System>

Next, an operation of the imaging system1will be described with reference to flowcharts ofFIGS.8to11.

First, synchronous moving image generation processing of the control device11will be described with reference to the flowchart ofFIG.8.

Synchronous moving image generation processing inFIG.8is a series of processing for capturing a background moving image and an actual moving image. The processing is started, for example, when an operation for starting background imaging is detected by the operation unit31.

In step S1, the control device11performs background moving image capturing processing. Background moving image capturing processing is processing of performing moving image capturing processing inFIG.9described later in a state where no subject is present in the imaging space. A background moving image is generated by the background moving image capturing processing and stored in the storage unit36.

Next, when an operation for starting actual imaging is detected by the operation unit31, in step S2, the control device11performs actual moving image capturing processing. Actual moving image capturing processing is processing of performing moving image capturing processing ofFIG.9described later in a state where a subject exists in the imaging space. An actual moving image is generated and stored in the storage unit36by the actual moving image capturing processing. When the background moving image and the actual moving image are stored in the storage unit36, the synchronous moving image generation processing ends.

Next, moving image capturing processing performed as processing in step S1or step S2inFIG.8will be described with reference to the flowchart ofFIG.9.

In step S11, the control unit32prepares the camera12and the monitor13. Specifically, the control unit32controls the imaging unit51to perform various settings such as setting of a frame rate and resolution of a moving image. Additionally, the control unit32performs control to cause the reproduction unit33to display a reference moving image on the monitor13. The reproduction unit33acquires the reference moving image designated by the control unit32, decodes the reference moving image, and appropriately supplies a frame image of the reference moving image to the display unit52.

In step S12, the control unit32causes the synchronization signal generation unit34to start generating a synchronization signal. Under the control of the control unit32, the synchronization signal generation unit34generates a synchronization signal and supplies the synchronization signal to the imaging unit51and the display unit52of each camera12.

In step S13, the imaging unit51of each camera12captures the background moving image or the actual moving image on the basis of the synchronization signal supplied from the synchronization signal generation unit34. At the same time, the display unit52displays the reference moving image on the basis of the synchronization signal supplied from the synchronization signal generation unit34.

In step S14, the control unit32causes the synchronization signal generation unit34to stop generating the synchronization signal. For example, in a case where an operation for ending imaging is detected by the operation unit31, the processing of step S14is performed. The flow of processing may be configured such that the processing of step S14is performed in a case where a predetermined time has elapsed from the start of generation of the synchronization signal.

In step S15, the control unit32receives an imaging end notification supplied from the imaging unit51.

In step S16, the storage control unit35stores the moving image captured by each camera12in the storage unit36. That is, the storage control unit35encodes the background moving image or the actual moving image captured and supplied by the imaging unit51in the processing in step S13by a predetermined encoding method such as the MPEG2 method or the AVC method, for example, and stores the encoded moving image in the storage unit36.

Thus, the moving image capturing processing performed as the processing of step S1or step S2inFIG.8ends. The difference between the background moving image capturing processing of step S1and the actual moving image capturing processing of step S2inFIG.8is whether or not the processing is performed in a state where a subject exists in the imaging space.

Next, subject 3D model data generation processing by the control device11will be described with reference to the flowchart ofFIG.10. The processing is started, for example, when an operation for generating 3D model data of the subject is detected by the operation unit31.

First, in step S31, the foreground/background difference processing unit43performs foreground/background difference processing for each actual moving image captured by each camera12. Difference data of the moving image for each camera12is generated by the foreground/background difference processing. Note that details of the foreground/background difference processing will be described later with reference to the flowchart ofFIG.11.

In step S32, the 3D model data generation unit44generates 3D model data of the subject on the basis of the difference data of the moving image for each camera12.

Details of the foreground/background difference processing performed in step S31ofFIG.10will be described with reference to the flowchart ofFIG.11.

In step S41, the foreground/background difference processing unit43reads one predetermined frame image among a plurality of frame images included in the actual moving image from the storage unit36. For example, the foreground/background difference processing unit43sequentially reads a plurality of frame images included in the actual moving image frame by frame from the first frame image in the processing of step S41repeatedly performed as described later.

In step S42, the foreground/background difference processing unit43reads a frame image of the background moving image having the same frame number as the frame image of the actual moving image read in step S41from the storage unit36among the plurality of frame images included in the background moving image.

In step S43, the foreground/background difference processing unit43sets a predetermined pixel among the plurality of pixels included in the frame image of the actual moving image read from the storage unit36as a processing target pixel, and calculates a difference in pixel value between the processing target pixel and a pixel corresponding to the processing target pixel among the pixels included in the frame image of the background moving image. Then, the foreground/background difference processing unit43determines whether or not the calculated difference between pixel values is equal to or greater than a predetermined threshold.

If it is determined in step S43that the difference between pixel values is equal to or greater than the predetermined threshold, the processing proceeds to step S44, and the foreground/background difference processing unit43sets the processing target pixel as a foreground pixel.

On the other hand, if it is determined in step S43that the difference between pixel values is less than the predetermined threshold, the processing proceeds to step S45, and the foreground/background difference processing unit43sets the processing target pixel as a background pixel.

In step S46, the foreground/background difference processing unit43determines whether or not there is a next pixel to be set as a processing target pixel among the pixels of the frame image of the actual moving image read in step S41. In step S46, in a case where all the pixels of the frame image of the actual moving image read in step S41are set as processing target pixels, it is determined that there is no next pixel to be set as a processing target pixel, and in a case where not all the pixels have been set as processing target pixels yet, it is determined that there is a next pixel to be set as a processing target pixel.

If it is determined in step S46that there is a next pixel, the processing returns to step S43, a pixel that has not yet been set as a processing target pixel is set as a processing target pixel, and the subsequent processing is performed.

On the other hand, if it is determined in step S46that there is no next pixel, the processing proceeds to step S47, and the foreground/background difference processing unit43generates a frame image of only the foreground. That is, in step S47, a frame image in which only the pixels set as foreground pixels are collected in the frame image of the actual moving image read in step S41is generated. For example, a fixed pixel value such as 0 is stored in the pixel set as a background pixel.

Next, in step S48, the foreground/background difference processing unit43determines whether there is a next frame image, that is, whether or not there is a frame image for which generation of a frame image of only the foreground has not yet been performed in the actual moving image.

If it is determined in step S48that there is a next frame image, the processing returns to step S41, the frame image of the actual moving image for which generation of the frame image of only the foreground has not yet been performed is read as the next frame image, and the subsequent processing is performed.

On the other hand, if it is determined in step S48that there is no next frame image, the processing proceeds to step S49, and the foreground/background difference processing unit43generates difference data of the moving image. That is, the foreground/background difference processing unit43generates difference data of the moving image by arranging the frame images of only the foreground generated by the repeatedly performed processing in step S47in the same order as the frame images of the actual moving image.

Thus, the foreground/background difference processing inFIG.11ends. The foreground/background difference processing inFIG.11is performed on each of the actual moving images captured by the plurality of cameras12. The foreground/background difference processing may be performed by sequentially selecting the actual moving images corresponding to the cameras12one by one, or the foreground/background difference processing may be performed by simultaneously selecting two or more or all of the actual moving images. After the difference data of the moving image is generated for all the actual moving images corresponding to all the cameras12, the processing returns to step S31ofFIG.10, and the processing of step S31and subsequent step is performed.

According to the imaging system1described above, a person as a subject can perform imaging while confirming a reference moving image displayed on the monitor13at the time of actual imaging. At that time, it is possible to prevent the moving image displayed on the monitor13from being extracted as a subject.

<4. Processing when Performing Monitoring Display>

Next, a case where the imaging system1captures a moving image while performing monitoring display will be described.

In a case where the imaging system1performs monitoring display, as described above, the moving image captured by the camera12is displayed on the monitor13in real time as a reference moving image. That is, at the time of actual imaging, the actual moving image being captured is displayed as it is on the monitor13as a reference moving image.

In this case, since the actual moving image displayed as a reference moving image on the monitor13is required at the time of background imaging, the actual imaging is performed before the background imaging, and the background imaging is performed after the actual imaging. At the time of background imaging, the actual moving image displayed on the monitor13at the time of actual imaging and stored in the control device11is displayed as a reference moving image.

FIG.12is a block diagram illustrating a configuration example of the imaging system1in a case where monitoring display is performed.

InFIG.12, the same configurations as those of the imaging system1inFIG.7are denoted by the same reference numerals, and redundant description will be omitted as appropriate.

The configuration of an imaging system1illustrated inFIG.12is different from the configuration of the imaging system1described with reference toFIG.7in that the actual moving image obtained by a camera12-mis supplied to the display unit52.

The camera12-m(m is any one of 1 to N) is a camera that captures an actual moving image as a reference moving image displayed on a monitor13. The same synchronization signal as that of a camera12-n(n is 1 to N excluding m) is supplied from a synchronization signal generation unit34to the camera12-m.

Similarly to the cameras12-n, the camera12-mis provided with an imaging unit51-m. The imaging unit51-mcaptures an actual moving image or a background moving image at an imaging timing based on the synchronization signal supplied from the synchronization signal generation unit34. At the time of actual imaging, the actual moving image captured by the imaging unit51-mis supplied to a storage control unit35and a display unit52. At the time of background imaging, the background moving image captured by the imaging unit51-mis not supplied to the display unit52but is supplied to the storage control unit35.

At the time of actual imaging, the display unit52displays the actual moving image supplied from the imaging unit51-mas a reference moving image at a display timing based on the synchronization signal supplied from the synchronization signal generation unit34. On the other hand, at the time of background imaging, the display unit52displays the reference moving image supplied from a reproduction unit33at a display timing based on the synchronization signal supplied from the synchronization signal generation unit34. The reference moving image supplied from the reproduction unit33is the same as the actual moving image supplied from the imaging unit51-mat the time of actual imaging.

A control unit32controls the camera12-msimilarly to the case of the cameras12-non the basis of the information supplied from an operation unit31. In a case where an operation for starting actual imaging or background imaging is detected by the operation unit31, the control unit32controls the imaging unit51-mand the imaging unit51-n, and performs various settings such as setting of a frame rate and resolution of a moving image.

Additionally, at the time of actual imaging, the control unit32does not cause the reproduction unit33to reproduce a predetermined moving image since the actual moving image captured by the imaging unit51-mis displayed on the monitor13as a reference moving image. In other words, the control unit32causes the reproduction unit33to not reproduce a predetermined moving image at the time of actual imaging. On the other hand, at the time of background imaging, the control unit32performs control to cause the reproduction unit33to display the actual moving image displayed on the monitor13at the time of actual imaging on the monitor13again as a reference moving image.

At the time of actual imaging, under the control of the control unit32, the reproduction unit33does not reproduce the predetermined moving image. On the other hand, at the time of background imaging, the reproduction unit33acquires the actual moving image designated by the control unit32(actual moving image captured by imaging unit51-m) from the actual moving images stored in the storage unit36, and reproduces the acquired actual moving image. Similarly toFIG.7, the reproduction unit33decodes the reference moving image prior to the display timing of the monitor13and supplies the actual moving image as a reference moving image to the display unit52.

A storage control unit35encodes the moving images supplied from the imaging units51-mand51-nby a predetermined encoding method, and stores the encoded moving images in the storage unit36.

A reproduction data storage unit41stores the actual moving image captured by the imaging unit51-mas a reference moving image.

<Operation of Control Device11>

Next, the synchronous moving image generation processing of the control device11in the case of performing the monitoring display will be described with reference to a flowchart ofFIG.13.

The synchronous moving image generation processing inFIG.13is started, for example, when an operation for starting actual imaging is detected by an operation unit31.

First, in step S101, the control device11performs actual moving image capturing processing. In the processing of step S101, the moving image capturing processing ofFIG.9is performed in a state where the actual moving image captured by the camera12-mis displayed on the monitor13and the subject is present in the imaging space. The actual moving image captured by each camera12is stored in the storage unit36by the actual moving image capturing processing.

Next, when an operation for starting background imaging is detected by the operation unit31, in step S102, the control device11performs background moving image capturing processing. In the processing of step S102, the actual moving image displayed on the monitor13in the processing of step S101is displayed on the monitor13as a reference moving image, and the moving image capturing processing ofFIG.9is performed in a state where no subject is present in the imaging space. The background moving image captured by each camera12is stored in the storage unit36by the background moving image capturing processing.

3D model data of the subject is generated using the actual moving image and the background moving image generated by the synchronous moving image generation processing inFIG.13. Note that the processing of generating the 3D model data of the subject is similar to the subject 3D model data generation processing illustrated inFIG.10, and thus the description thereof will be omitted.

According to the imaging system1described above, it is possible to generate moving images necessary for generating 3D model data while a person as a subject confirms his/her own movement at the time of actual imaging. At that time, it is possible to prevent the moving image displayed on the monitor13from being extracted as a subject.

<Example in which Time Code is Displayed>

At the time of each of actual imaging and background imaging, a control unit32can perform control so that information indicating a time code is displayed on a monitor13together with a reference moving image.

In this case, it is possible to confirm whether the time code of the moving image itself captured by a camera12matches the time code appearing in the moving image captured by the camera12. As a result, it is possible to easily confirm synchronization between the imaging timing of the camera12and the display timing of the monitor13.

<Example of Arranging a Plurality of Monitors13>

There are cases where actual imaging is performed in a state where a plurality of persons exists as subjects, and where actual imaging is performed in a state where persons as subjects face various directions. In these cases, it is preferable that a plurality of monitors13is arranged in the imaging space.

FIG.14is another configuration example of the imaging system1, and is a diagram illustrating a configuration example of an imaging system1in a case where a plurality of monitors13is arranged in the imaging space.

The imaging system1inFIG.14includes a control device11, N cameras12-1to12-N, and N monitors13-1to13-N. The cameras12-1to12-N and the monitors13-1to13-N are arranged in the same imaging space.

The monitors13-1to13-N may display different reference moving images, or may display the same reference moving image.

Even in a case where a plurality of monitors13is provided, the imaging timings of the cameras12-1to12-N and the display timings of the monitors13-1to13-N are controlled to be synchronized on the basis of a synchronization signal generated by the control device11.

FIG.15is a plan view illustrating a situation at the time of actual imaging in the imaging system1ofFIG.14with N=8.

In the example ofFIG.15, eight cameras12-1to12-8and eight monitors13-1to13-8are alternately arranged in an annular shape.

The monitors13-1to13-8are arranged between the camera12-1and the camera12-2, between the camera12-2and the camera12-3, between the camera12-3and the camera12-4, between the camera12-4and the camera12-5, between the camera12-5and the camera12-6, between the camera12-6and the camera12-7, between the camera12-7and the camera12-8, and between the camera12-8and the camera12-1with the front facing the center.

Note that in the examples ofFIGS.14and15, an example in which the same number of monitors13as the number of cameras12is provided in the imaging system1has been described. However, the number of monitors13may be different from the number of cameras12in the imaging system1.

As described above, even in a case where a plurality of monitors13is provided, the control device11can separate only a person assumed as a subject, for example, from an actual moving image as the foreground by performing the foreground/background difference processing using the actual moving image and the background moving image. At this time, the imaging timings of the cameras12-1to12-N and the display timings of the monitors13-1to13-N are controlled to be synchronized on the basis of the synchronization signal, so that it is possible to prevent the reference moving images on the monitors13-1to13-N appearing in the actual moving image from being extracted as a subject.

<Example in which Synchronization Signal Generation Device is Provided>

While the synchronization signal generation unit34is provided in the control device11in the imaging system1ofFIG.7, the function of the synchronization signal generation unit34may be implemented by a device outside the control device11.

FIG.16is a diagram illustrating still another configuration example of the imaging system1, and is a diagram illustrating a configuration example of the imaging system1in a case where the function of the synchronization signal generation unit34is a separate device.

An imaging system1illustrated inFIG.16is configured by connecting a control device11A, cameras12-1to12-N, a monitor13, and a synchronization signal generation device101directly or via a network.

The synchronization signal generation device101is provided with the configuration of the synchronization signal generation unit34inFIG.7. That is, the synchronization signal generation device101generates a synchronization signal under the control of the control device11and supplies the synchronization signal to each of the cameras12-1to12-N and the monitor13. The control device11A is configured by removing the synchronization signal generation unit34from the control device11inFIG.7.

As described above, a part of the configuration provided in the control device11may be provided outside the control device11.

<Example of Performing Frame Rate Conversion>

In each of the above-described embodiments, the description has been given on the assumption that the frame rate when the camera12performs actual imaging or background imaging is the same as the frame rate when the reproduction unit33reproduces the reference moving image. However, in some cases, the frame rate of the reference moving image may be different from the frame rate at which the camera12performs imaging. In that case, the control unit32can designate the frame rate to the reproduction unit33, and the reproduction unit33can convert the reference moving image designated from the control unit32to the designated frame rate, reproduce the reference moving image, and supply the reference moving image to the display unit52.

Additionally, for example, in a case where the lines or lyrics to be displayed on the monitor13as a reference moving image is not a moving image but includes one or more still images, the reproduction unit33can convert the still images into a moving image having a frame rate designated by the control unit32and supply the moving image to the display unit52.

<6. Configuration Example of Computer>

The series of processing described above can be performed by hardware or software. In a case where the series of processing is performed by software, a program forming the software is installed from a program recording medium to a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

FIG.17is a block diagram illustrating a hardware configuration example of a computer that performs the series of processing described above according to a program.

A central processing unit (CPU)301, a read only memory (ROM)302, and a random access memory (RAM)303are mutually connected via a bus304.

An input/output interface305is also connected to the bus304. An input unit306, an output unit307, a storage unit308, a communication unit309, and a drive310are connected to the input/output interface305.

The input unit306includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit307includes a display, a speaker, an output terminal, and the like. The storage unit308includes a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit309includes a network interface and the like. The drive310drives a removable recording medium311such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU301loads a program stored in the storage unit308onto the RAM303through the input/output interface305and the bus304, and executes the program to perform the above-described series of processing, for example. The RAM303also appropriately stores data and the like necessary for the CPU301to perform various processing.

The program executed by the computer (CPU301) can be provided by being recorded on the removable recording medium311such as a package medium, for example.

Additionally, the program can be provided through a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit308through the input/output interface305by attaching the removable recording medium311to the drive310. Additionally, the program can be received by the communication unit309through a wired or wireless transmission medium and be installed in the storage unit308. In addition, the program can be installed in advance in the ROM302or the storage unit308.

Note that in the specification, steps described in the flowcharts may be performed chronologically according to the described order, as a matter of course, but does not necessarily have to be processed in chronological order, and may be performed in parallel or at a necessary timing such as when a call is made.

Note that in the present specification, a system means a collection of a plurality of components (devices, modules (parts), and the like), and it does not matter whether or not all the components are in the same case. Accordingly, a plurality of devices housed in separate cases and connected through a network, and one device housing a plurality of modules in one case are both systems.

Note that the effect described in the present specification is merely an illustration and is not restrictive. Hence, other effects can be obtained.

Embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, the present technology can have a cloud computing configuration in which one function is shared and processed by a plurality of devices through a network.

Additionally, each step described in the above-described flowchart can be executed by one device or be executed in a shared manner by a plurality of devices.

Moreover, in a case where a plurality of processing is included in one step, the plurality of processing included in one step can be performed by one device or be performed in a shared manner by a plurality of devices.

<Exemplary Combination of Configuration>

The present technology can also be configured as follows.

A control device includinga control unit that performs control to synchronize an imaging timing at which an imaging device performs imaging with a display timing at which a display device located in an imaging range of the imaging device displays a first moving image, andan image processing unit that performs the foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, in whichthe second moving image and the third moving image both include at least a part of the first moving image whose display timing is synchronized between the second moving image and the third moving image.

The control device according to (1) above, in which:the second moving image is an actual moving image which is a moving image including a subject and a background;the third moving image is a background moving image which is a moving image including the background; andthe image processing unit performs the foreground/background difference processing using the background moving image and the actual moving image.

The control device according to (2) above, in whichthe first moving image is a reference moving image to be referred to when the actual moving image is captured, andthe control unit further performs control to cause the display device to display the reference moving image.

The control device according to (3) above, in whichthe reference moving image is a moving image read from a predetermined storage unit and reproduced.

The control device according to (4) above further including a reproduction unit that reproduces the reference moving image stored in the predetermined storage unit, in whichthe reproduction unit decodes the reference moving image prior to the display timing of the display device.

The control device according to (3) above, in whichthe reference moving image is a moving image captured by the imaging device in capturing the actual moving image.

The control device according to any one of (3) to (6) above, in whichthe control unit causes the display device to display the same reference moving image at the first timing and the second timing.

The control device according to (7) above, in whichthe control unit causes the display device to display a frame image of the same reference moving image for the same frame number of the actual moving image and the background moving image when imaging the actual moving image and the background moving image.

The control device according to any one of (1) to (8) above, in whichthe control unit generates a synchronization signal, causes the imaging device to perform imaging at a rising or falling timing of the synchronization signal, and causes the display device to display the first moving image.

The control device according to any one of (1) to (9) above, in whichthe control unit performs control to synchronize an imaging timing at which a plurality of the imaging devices performs imaging with a display timing at which a plurality of the display devices displays the first moving image.

The control device according to (10) above, in whichthe control unit causes the plurality of the display devices to display different first moving images respectively.

The control device according to any one of (1) to (11) above, in whichthe control unit further performs control to cause the display device to display information indicating a time code together with the first moving image.

The control device according to (9) above further includinga synchronization signal generation unit that generates the synchronization signal on the basis of control of the control unit.

The control device according to any one of (1) to (13) above, in whichthe image processing unit generates difference data by performing foreground/background difference processing between the second moving image and the third moving image captured by the imaging device, and further generates 3D model data on the basis of the difference data.

An Imaging System Including:a plurality of imaging devices;at least one display device;a control unit that performs control to synchronize an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which the display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image; andan image processing unit that performs foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing, in whichthe second moving image and the third moving image both include at least a part of the first moving image.

A 3D model data generation method including:synchronizing an imaging timing at which the plurality of imaging devices performs imaging with a display timing at which a display device located in an imaging range of at least one of the plurality of imaging devices displays a first moving image, and imaging by a plurality of imaging devices;generating difference data by performing foreground/background difference processing on the basis of a second moving image captured by the imaging device at a first timing and a third moving image captured by the imaging device at a second timing different from the first timing; andgenerating 3D model data on the basis of the difference data.

REFERENCE SIGNS LIST