INFORMATION PROCESSING APPARATUS FOR VIRTUAL VIEWPOINT IMAGES, CONTROL METHOD, AND PROGRAM FOR THE SAME

An information processing apparatus includes one or more memories storing instructions and one or more processors executing the instructions to: obtain a user operation; set, based on the obtained user operation, positions and orientations of a plurality of virtual cameras in a virtual viewpoint image generated from a plurality of images captured by a plurality of image capturing apparatuses; generate a camera path from the positions and the orientations of the plurality of virtual cameras; and differentiate a procedure for generating the camera path between generating the camera path using the set positions and orientations of the plurality of virtual cameras and generating the camera path using the set positions and orientations of the virtual cameras and a preset position and orientation of a virtual camera.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an information processing apparatus for generating a camera path in a virtual viewpoint image.

Description of the Related Art

Recent known image processing systems can generate an image viewed from a virtual viewpoint that is specified by a user from multiple images obtained by capturing a subject with a plurality of cameras. The image viewed from the virtual viewpoint specified by the user is hereinafter referred to as a virtual viewpoint image. These image processing systems create a subject model by presuming the shape of an image acquisition target, such as a person, and generates a virtual viewpoint image viewed from freely changeable viewpoint. The virtual viewpoint image can be continuously generated while the positions and orientations of the virtual camera are being changed.

Such virtual viewpoint images are used for achieving more realistic visual representation specifically in competitive sports broadcasting, such as live coverage of sports events. In sports broadcasting, clip video using virtual viewpoint images is sometimes generated for replays of impactful moments during the match. One of a method for generating a camera path for generating clip video using virtual viewpoint images is a key frame method. This is a method of generating a camera path by the user registering multiple virtual camera parameters of any time, position, and orientation for key frames and interpolating the key frames using a predetermined method.

Japanese Patent Laid-Open No. 2007-25979 discloses a method for generating a camera path by interpolating between specified key frames using spline interpolation.

However, the method disclosed in Japanese Patent Laid-Open No. 2007-25979 may generate a redundant camera path in moving to a preset position.

SUMMARY OF THE INVENTION

The present invention provides an information processing apparatus including one or more memories storing instructions and one or more processors executing the instructions to: obtain a user operation; set, based on the obtained user operation, positions and orientations of a plurality of virtual cameras in a virtual viewpoint image generated from a plurality of images captured by a plurality of image capturing apparatuses; generate a camera path from the positions and the orientations of the plurality of virtual cameras; and differentiate a procedure for generating the camera path between generating the camera path using the set positions and orientations of the plurality of virtual cameras and generating the camera path using the set positions and orientations of the virtual cameras and a preset position and orientation of a virtual camera.

Further features of the present invention will become apparent from the following description of example embodiments with reference to the attached drawings.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

A first embodiment will be described in detail. In this embodiment, the image processing system has predetermined interpolation procedures for key frames in virtual viewpoint images and for interpolating from the current position of the virtual viewpoint to the preset position of the virtual camera. The image processing system generates a camera path following the predetermined interpolation procedures.

Here, the virtual viewpoint is a viewpoint specified by the user in the three-dimensional space of the virtual environment. The following description is made using a camera (virtual camera) that is virtually placed at a virtual viewpoint position for the convenience of description. In other words, the position of the virtual viewpoint and the eye gaze direction from the virtual viewpoint correspond to the position and orientation of the virtual camera. The field of view from the virtual viewpoint corresponds to the angle of view of the virtual camera. This virtual viewpoint can freely be specified by the user.

The virtual viewpoint images of this embodiment are not limited to the image corresponding to the viewpoint that is freely specified by the user and include an image corresponding to a viewpoint that is selected from a plurality of candidates by the user. In this embodiment, the specification of the virtual viewpoint is primarily described with respect to user operations; however, the specification of the virtual viewpoint may be automatically performed based on the result of image analysis or similar methods. In this embodiment, the description focuses on the case where the virtual viewpoint image is a moving image. In other words, the virtual viewpoint image is a video captured by a virtual camera.

The image processing system in this embodiment has the function of switching between a video captured by an actual image capturing apparatus, for example, a broadcasting camera, and a virtual viewpoint image corresponding to a virtual viewpoint.

FIG.2is a diagram illustrating the overall configuration of an image processing system20for generating a virtual viewpoint image. The image processing system20includes an image capturing system201, an image processing apparatus202, an information processing apparatus203.

The image capturing system201includes a plurality of digital cameras (image capturing apparatuses) at different positions so as to surround (around) the image-capturing region and captures images in time synchronization and transmits the multiple images captured from multiple viewpoints in synchronization to the image processing apparatus202. The multiple images are transmitted via a communication medium such as a local area network (LAN) cable. Examples of the image-capturing region include a photo studio for creating a virtual viewpoint image, an athletic field for sporting events, and a stage for acting performance.

The image processing apparatus202generates three-dimensional shape data on the subject based on multiple images synchronously captured from multiple cameras. The three-dimensional shape data is generated using, for example, a visual hull method. As a result of this process, a three-dimensional (3D) point group (a set of points with three-dimensional coordinates) represents the three-dimensional shape of the subject. The method for deriving the three-dimensional shape of the subject from the captured image is illustrative only. A virtual viewpoint image corresponding to the virtual camera is generated from the three-dimensional shape data and a specified background model. The background model is a computer graphics (CG) model of, for example, an athletic field in which a physical camera group of the image capturing system201is placed. This background model is created in advance and stored in the image processing apparatus202(for example, in a read-only memory (ROM)403inFIG.4, described later). An example of a method for generating the virtual viewpoint image uses model-based rendering (MBR). This processing allows generating a three-dimensional shape data image viewed from the positions and directions of the virtual cameras. The method for generating virtual viewpoint images is illustrative only. The position, the direction, and other information on the virtual camera are represented by camera parameters that the information processing apparatus203determines, described later. In this embodiment, both the three-dimensional shape data on the subject and the virtual viewpoint image are generated by the image processing apparatus202; alternatively, the generation of the data may be shared by a plurality of image processing apparatuses.

FIGS.3A and3Bare diagrams illustrating an example of the information processing apparatus203. More details of the information processing apparatus203are described below with reference to the examples shown inFIGS.4and5. The information processing apparatus203connects to one or more input devices, such as joysticks301aand301b, for controlling the virtual camera. The user sets (determines) a camera path representing the viewpoint of the virtual camera by operating the joysticks301aand301b. In this embodiment, the multiple operating members of the joystick301aare assigned commands to control parameters (x, y, and z) indicating the position of the virtual camera in the three-dimensional coordinates in the virtual space. The multiple operating members of the joystick301bare assigned commands to control parameters (pan, tilt, and roll) for pan, tilt, and roll directions representing the orientation of the virtual camera in the virtual space.

The information processing apparatus203also connects to an input device, such as a keyboard302. The individual keys of the keyboard302are assigned commands to generate a virtual viewpoint image. For example, a key303is assigned a command to register a key frame, and a key304is assigned a command to generate a camera path for a clip video from the registered key frame. A key305is assigned a command to register a preset position, and a key306is assigned a command to cancel the playback of the clip video.

An application for generating a virtual viewpoint image, stored in the information processing apparatus203, has the function of assigning any command to any operating member among the joystick301a, the joystick301b, and the keyboard302. The user can set any command to any operating member using the function.

In this embodiment, the information processing apparatus203further connects to two or more display units.

In this embodiment, the information processing apparatus203obtains a virtual viewpoint image generated by the image processing apparatus202from the image processing apparatus202and displays the virtual viewpoint image on a display unit307. The information processing apparatus203displays a graphic user interface (GUI)308that displays a camera path for generating a clip video of the virtual viewpoint image, parameters (for example, present position) of the virtual camera, and other information on a display unit309. The GUI308corresponds to the GUI of the application for generating a virtual viewpoint image.

Referring toFIG.3B, the details of the GUI308displayed on the display unit309will be described. The GUI308contains a clip list310, a key-frame list311, and a preset list312.

First, a method of operation on the GUI308in registering key frames and generating/playing back a clip video will be described. The user sets the position and orientation of the virtual camera at any time (time code) in the virtual viewpoint image and presses the key303. In response to the user operation, the information processing apparatus203adds a key frame having a key frame ID and camera parameters and time code of the virtual camera to the key-frame list311. The key frame IDs are assigned sequentially in the order added by the user. However, the user can freely change the order of key frame IDs. When the user performs a user operation, such as when the user presses the key304, with one or more key frames set on the key-frame list311, the information processing apparatus203generates a camera path from the key frames contained in the key-frame list311. In this embodiment, the information processing apparatus203generates a camera path by interpolating the position and orientation of the virtual camera using the camera parameters of the virtual camera in the order of key frame IDs. The generated camera path is added to the clip list310every time the camera path is generated. For example, when the user clicks (selects) any camera path in the clip list310, a virtual viewpoint image (clip video) to which the clicked camera path is applied is displayed on the display unit307.

Next, a method for registering a preset position and a method for applying the preset position using the GUI308will be described. First, the method for registering a preset position will be described. When the user sets a virtual camera at any position and orientation and performs a user operation (e.g. presses the key305), a preset position with a preset ID and information on the camera parameters of the virtual camera is added to the preset list312. This allows the user to record the location of the virtual camera at a predetermined position and orientation on the information processing apparatus203. Next, the method for applying the preset position will be described. When the user clicks (selects) any preset position in the preset list312, the virtual camera shifts from the current state (position and orientation) to the preset state. In this embodiment, the information processing apparatus203generates a camera path for shifting the virtual camera from the current state (position and orientation) to a preset state and updates the position and orientation of the virtual camera (moves the virtual camera) following the camera path.

The information processing apparatus203can assign a preset position to any key on the keyboard302. In this case, for example, when the user presses a key to which the preset position of the virtual camera is assigned, the information processing apparatus203moves the virtual camera to a preset position corresponding to the pressed key.

The information processing apparatus203transmits the camera path for the virtual camera selected based on the user operation to the image processing apparatus202. The image processing apparatus202generates a virtual viewpoint image based on the camera path received from the information processing apparatus203.

An example of interpolation between two key frames in camera path generation is illustrated inFIG.1.FIG.1illustrates a method for interpolating one value p of the camera parameters at a start point and an end point. When the user sets the time and the parameter at the start point to t1and p1and the time and the parameter at the end point to t2and p2, respectively, the value p is interpolated between t1and t2as in the graph ofFIG.1. The parameters of the virtual camera interpolated in this way include position coordinates (x, y, and z), orientations (pan, tilt, and roll), and a magnification ratio (Zoom).

The number of key frames registered in generating a desired camera path can be decreased by unlimiting the upper limit and the lower limit of the parameters representing the orientations, pan, tilt, and roll. For example, consider the case where the gazing point is set at any position, and a camera path with two rotations while facing in the direction of the gazing point is to be generated. Unlimiting the upper limit and the lower limit of the rotation angle allows generation of the camera path only by setting the parameter at two points, the start point and the end point, to 0° and 720°, respectively. In contrast, if the minimum limit 0° and the maximum limit 359° are set, rotation angles of 360° or more cannot be set, which requires three or more key frames. In this embodiment, the camera path is generated using key frames with unlimited upper limit and lower limit. The method for generating a camera path using key frames is hereinafter also referred to as a key frame method.

In generating a camera path for another application, using the same interpolation procedure as in generating a camera path for clip video using the key frame method may generate a camera path that is not desired by the user. One example is moving the virtual camera from a current position to a registered position and orientation (preset position). One example is a use case in a real-time live broadcast of a baseball game. In a live broadcast, it is easy to obtain a realistic video from a viewpoint from the mound on the field or from the catcher's position. Real-time live broadcasting requires frequent switching between the viewpoints, which requires the function of registering multiple preset positions and switching the position of the virtual camera to a desired preset position. Instant switching of the virtual camera from the current position to the preset position may take much time for the user who is operating the virtual camera to determine whether the switched position is the desired preset position. This needs to devise methods such as generating a camera path for moving the virtual camera from the current position to the preset position and displaying the process of the movement. However, generating the camera path by interpolating between the current position and the preset position using the same method as the key frame interpolation described above may generate a redundant camera path, such as unnecessary rotation. For this reason, the position relationship needs to be taken into account in generating a camera path from the current position to the preset position.

FIG.4is a diagram illustrating an example of the hardware configuration of the information processing apparatus203configured to generate and edit the camera path for a virtual viewpoint image according to this embodiment. The information processing apparatus203includes a central processing unit (CPU)401, a random-access memory (RAM)402, a read-only memory (ROM)403, an input/output device404, and a communication unit405. For virtual viewpoint images, users, such as viewers or camera operators, can freely operate the position and orientation of the virtual camera. The virtual viewpoint image may be either moving images or still images.

The CPU401is a processor that executes programs stored in the ROM403using the RAM402as a work memory and controls the components of the information processing apparatus203as a whole.

Thus, the functions of the processing units (e.g. information processing apparatus203) illustrated inFIG.2are performed by the CPU401executing the various programs.

The RAM402temporarily stores computer programs read from the ROM403and intermediate results of calculation.

The ROM403holds computer programs and data that require no change. The ROM403also stores data necessary for interpolating camera parameters and data necessary for determining an interpolation method, such as the key frame method described above.

The input/output device404includes one or a plurality of controllers for controlling the virtual camera and one or a plurality of display units that displays, for example, the state of the virtual camera. The one or a plurality of controllers includes general devices for the user to perform an input operation, such as a keyboard and a mouse, a joystick, a tab, and a jog dial for controlling the virtual camera. The display units include one or a plurality of display devices (hereinafter referred to as “monitor”) for displaying information necessary for the user.

FIG.5is a block diagram illustrating an example of the functional configuration of the information processing apparatus203according to the first embodiment. The information processing apparatus203generates a camera path for the virtual camera based on a user operation and transmits the camera path to a virtual-viewpoint-image generating unit.

The individual functions of the information processing apparatus203will be described in sequence.

An operating-information acquisition unit501obtains a user operation, such as on the joystick301, for each frame and converts the operation to parameters, such as the position and orientation of the virtual camera according to the amount of operation. The parameters represent the amount of change in the virtual camera parameters per frame. In response to a user operation (input), such as on the keyboard302or the GUI308, the operating-information acquisition unit501transmits an input signal responsive to the operation to a control-mode determination unit502, a key-frame holding unit504, a clip creating unit505, and a virtual-camera-parameter calculation unit511.

The control-mode determination unit502switches a plurality of control modes according to a user operation. The control modes include a manual control mode, a clip playback mode, and a preset movement mode. In the manual control mode, the information processing apparatus203moves the virtual camera in response to a user operation on the joystick301. In the clip playback mode, the information processing apparatus203plays back a virtual viewpoint image generated using a camera path generated for a clip video.

The virtual viewpoint image is generated by an external virtual-viewpoint-image generating unit (not shown). In the preset movement mode, the information processing apparatus203moves the position of the virtual camera to the preset position. In the preset movement mode, the information processing apparatus203generates a camera path for moving the position of the virtual camera to the preset position and plays back a virtual viewpoint image generated using the camera path on the display unit.

A control-mode holding unit503is a recording unit that holds (records) the current control mode. The held control mode is updated when a switching signal from the control-mode determination unit502is input. The control-mode holding unit503transmits the currently held control mode to the control-mode determination unit502.

FIG.6illustrates a state transition diagram of the three control modes. In this embodiment, the manual control mode is normally set, in which the virtual camera is operated by the user. When a camera path in the clip list310is clicked by the user, the control-mode determination unit502switches the control mode to the clip playback mode and transmits a signal to indicate the switching of the modes to the control-mode holding unit503and the virtual-camera-parameter calculation unit511. When any preset position on the preset list312is clicked by the user, the control-mode determination unit502switches the control mode to the preset movement mode and transmits a signal to indicate the switching of the modes to the control-mode holding unit503and the virtual-camera-parameter calculation unit511. When the movement to the preset position ends while the control mode is the preset movement mode, the control-mode determination unit502receives a signal to switch to the manual control mode from the virtual-camera-parameter calculation unit511and switches the control mode to the manual control mode.

In response to a user input, such as to the key303on the keyboard302, the key-frame holding unit504receives an input signal to register the key frame from the operating-information acquisition unit501. The key303is assigned a command to register the key frame. In response to the input signal, the key-frame holding unit504obtains the current parameters of the virtual camera from the virtual-camera-parameter calculation unit511and holds the parameters as a key frame. The key-frame holding unit504can hold a plurality of key frames. Each key frame holds a key frame ID, which are integers, external parameters such as the position and orientation of the virtual camera, internal parameters such as an optical center and a focal length, and time information, such as time code, as parameters. The key frame ID is assigned to each key frame when the key-frame holding unit obtains the key frame. The key frame ID is a parameter that can be changed by the user. The held key frame is output to the clip creating unit505at a user input, such as to the key304on the keyboard302, and is then cleared. The key304is assigned a command to generate a camera path for a clip video from the registered key frame.

In response to a user input, such as to the key304on the keyboard302, the clip creating unit505receives an input signal from the operating-information acquisition unit501. In response to the input signal, the clip creating unit505obtains a plurality of key frames held in the key-frame holding unit504and interpolates between the key frames to generate a camera path.

FIGS.7A and7Bare diagrams illustrating an example of a method for interpolating between key frames for the x-coordinate of the position coordinates of the virtual camera. InFIG.7A, the positions and orientations of the virtual cameras at time code t1to t4are set as virtual cameras701to704. The graph inFIG.7Aillustrates the x-coordinate of the virtual camera at time t1to t4. As illustrated inFIG.7B, the clip creating unit505generates a camera path705so as to interpolate between the virtual cameras701to704using an interpolation method to smooth the movement locus of the virtual camera. The graph inFIG.7Billustrates the movement of the camera path705in the x-coordinate from time t1to t4. One example of the interpolation method is spline interpolation. The clip creating unit505also performs interpolation for the y-coordinate and the z-coordinate of the position coordinates of the virtual camera, as well as orientation information, pan, tilt, and roll.

A clip holding unit506holds a camera path for a clip video generated by the clip creating unit505. When any camera path on the clip list310is clicked (selected) by the user, the clip holding unit506transmits the selected camera path to the virtual-camera-parameter calculation unit511in response to an input signal received from the operating-information acquisition unit501.

In response to a user input, such as to the key305on the keyboard302, a preset holding unit507obtains the camera parameters of the virtual camera from the virtual-camera-parameter calculation unit511and holds the parameters as the preset position of the virtual camera. The key305is assigned a command to register the preset position.

The preset-movement-time holding unit508holds the movement time from the virtual camera position to the preset position. The movement time is registered (input) in advance by the user using the keyboard302or the like. The preset movement time differs from the time when the clip video is generated. For example. the preset movement time or movement time is a time, set by the user, for use in generating a camera path using a preset position (e.g. is a value of time set by the user for the virtual camera to move from a current position to the preset position).

A remaining-preset-movement-time holding unit509holds a remaining time to move to the preset position. The remaining-preset-movement-time holding unit509holds the preset movement time when the preset position in the preset list312of the GUI308is clicked (selected) by the user. Thereafter, the remaining-preset-movement-time holding unit509decreases the held remaining time by one frame every time one frame of the virtual viewpoint image is played back. The remaining-preset-movement-time holding unit509transmits the remaining time to a virtual-camera-parameter calculation unit511per frame.

A virtual-camera-parameter holding unit510obtains the virtual camera parameters calculated by the virtual-camera-parameter calculation unit511and holds the parameters. The virtual-camera-parameter holding unit510transmits the virtual camera parameters held at the current time code to the virtual-camera-parameter calculation unit511. Thereafter, the virtual-camera-parameter holding unit510obtains the virtual camera parameters of the next frame calculated by the virtual-camera-parameter calculation unit511and updates the held virtual camera parameters.

When the control mode is updated, the virtual-camera-parameter calculation unit511receives a signal from the control-mode determination unit502and calculates virtual camera parameters using a method matching the control mode. Methods for calculating the virtual camera parameters for the individual control modes will be described hereinbelow.

In the manual control mode, the virtual-camera-parameter calculation unit511adds up the amount of change in the virtual camera parameters obtained from the operating-information acquisition unit501and the camera parameters obtained from the virtual-camera-parameter holding unit510to calculate the virtual camera parameters of the next frame.

In the clip playback mode, the virtual-camera-parameter calculation unit511obtains clip information selected by the user from the operating-information acquisition unit501to obtain a corresponding camera path from the clip holding unit506. The obtained camera path is a camera path generated using the key frames.

In the preset movement mode, the virtual-camera-parameter calculation unit511calculates a camera path from the current position and orientation of the virtual camera to the preset position. Specifically, if the frame rate is 60 fps, the virtual-camera-parameter calculation unit511calculates the camera path using Math. 1.

where t is the time (remaining time) taken to move the virtual camera from the current position to the preset position, obtained from the remaining-preset-movement-time holding unit509. In this embodiment, t is expressed, for example, in the form of HH:MM:SS:FF, p(t) is the virtual camera parameter at the current time obtained from the virtual-camera-parameter holding unit510at remaining time t, and p_pre is the virtual camera parameter of the preset position obtained from the preset holding unit507, and f(t) is the number of remaining frames at remaining time t. The initial value of f(t) is obtained by multiplying the initial value of remaining time t converted into seconds (e.g. the initial value is the value of the preset movement time preset by the user) by the frame rate. The value dp is the amount of change in the virtual camera parameter per frame. Accordingly, dp can take either a positive value or a negative value. At t=0, the virtual-camera-parameter calculation unit511transmits a signal to switch to the manual control mode to the control-mode determination unit502. If the frame rate is a non-integer, such as 59.94 fps, the virtual-camera-parameter calculation unit511performs the calculation of the above equations in consideration of drop frames.

In the preset movement mode, the virtual-camera-parameter calculation unit511obtains the virtual camera parameters of the preset position from the preset holding unit507and virtual camera parameters from the virtual-camera-parameter holding unit510. Then, the orientation parameters of the virtual camera are corrected so that the path between the two points is the shortest.

A method for correcting the orientation parameters of the virtual camera will be described with reference toFIG.8. For example, a camera path for moving the virtual camera from a virtual camera position801to a preset position802inFIG.8is to be generated. The values of pan at the virtual camera position801and the preset position802when the virtual camera is rotated while facing the point of gaze803are set at 0°, 270°, respectively. If the path between the virtual camera position801and the preset position802is interpolated using the same method as the key frame interpolation, the value of pan increases monotonically from 0° to 270°, and the camera path of the virtual camera takes a path804. However, in this embodiment, the preset position is used to move the virtual camera to a desired position quickly in real-time video. In other words, the camera path indicated by the path804is a roundabout camera path from the virtual camera position801to the preset position802, which is not an appropriate camera path. A desirable camera path from the virtual camera position801to the preset position802is a path805. For this reason, the virtual-camera-parameter calculation unit511corrects the values of pan, tilt, and roll, which are parameters representing the orientation of the virtual camera, with the current position and orientation of the virtual camera position801fixed. Specifically, if the absolute value of the difference in the value of pan, tilt, or roll between the current position and the preset position of the virtual camera is greater than 180°, the virtual-camera-parameter calculation unit511corrects the value.

For example, the virtual-camera-parameter calculation unit511corrects the orientation parameters of the current virtual camera position whose difference from the preset position is greater than 180° to decrease the absolute value of the difference to 180° or less. Specifically, for the parameters, if the value of the current virtual camera position is greater than the value at the preset position by 180°, the virtual-camera-parameter calculation unit511subtracts an integral multiple of 360° from the value of the current camera path position so that the difference falls to 180° or less. In contrast, for the parameters, if the value of the current position of the virtual camera is less than the value of the preset position by1800or more, the virtual-camera-parameter calculation unit511adds an integral multiple of 360° to the value of the current camera path position so that the difference falls to 180° or less. Thus, the virtual-camera-parameter calculation unit511can correct the orientation parameters of the virtual camera by adding or subtracting a multiple of 360° to or from the parameters to be corrected without changing the position and orientation of the virtual camera. In addition, by decreasing the difference to 180° or less as described above, a camera path that does not take a detour can be generated.

The virtual-camera-parameter calculation unit511transmits the corrected camera parameters to a virtual-camera-parameter providing unit512.

The virtual-camera-parameter providing unit512obtains camera parameters for each frame from the virtual-camera-parameter calculation unit511and transmits the camera parameters to the virtual-viewpoint-image generating unit.

A time-code holding unit513obtains the current time code from a time-code calculation unit514and holds the time code.

The time-code calculation unit514obtains a time code (e.g. time information) for each frame from the time-code holding unit513, updates the time codes, and transmits the time codes to the time-code holding unit513and a time-code providing unit515.

The time-code providing unit515obtains the time codes from the time-code calculation unit514and transmits the time codes to the virtual-viewpoint-image generating unit.

A display516displays a virtual viewpoint image generated by the virtual-viewpoint-image generating unit. Examples of the display516include a liquid crystal display and an organic electroluminescence (EL) display.

The information processing apparatus203communicates with the image processing apparatus202using the communication unit405. The communication unit405communicates in accordance with a wired communication standard such as Ethernet or a wireless communication standard such as Wi-Fi®.

Referring next toFIG.9, a processing procedure for generating a camera path for clip video in this embodiment will be described. This flowchart is started when the key304on the keyboard302is pressed by the user. The key304is assigned a command to generate a camera path for clip video from the registered key frames.

In step S901, the operating-information acquisition unit501transmits a signal to generate a camera path to the clip creating unit505.

In step S902, the clip creating unit505receives the signal to generate a camera path from the operating-information acquisition unit501and obtains key frames from the key-frame holding unit504. The clip creating unit505generates a camera path by interpolating between the key frames in order of the key frame IDs as discussed above.

In step S903, the clip holding unit506obtains the camera path from the clip creating unit505and adds the obtained camera path to the clip list310.

In step S904, in response to the addition of the generated camera path to the clip list310, the key-frame holding unit504clears the held key frames (the key-frame list).

Next, a processing procedure for playing back clip video in this embodiment will be described with reference toFIG.10. This flowchart is started when the user clicks any camera path from the clip list310on the GUI308. The user operation of clicking any camera path is an operation for playing back clip video.

In step S1001, the operating-information acquisition unit501transmits a signal to switch the control mode to the clip playback mode to the control-mode determination unit502.

In step S1002, the control-mode determination unit502receives the signal from the operating-information acquisition unit501and switches the control mode to the clip playback mode. The control-mode determination unit502transmits a signal indicating the switching to the clip playback mode to the control-mode holding unit503and the virtual-camera-parameter calculation unit511.

In step S1003, the virtual-camera-parameter calculation unit511obtains the clip information selected by the user from the operating-information acquisition unit501and obtains a corresponding camera path from the clip holding unit506. The virtual-camera-parameter calculation unit511transmits the obtained camera path to the virtual-camera-parameter providing unit512.

In step S1004, the virtual-camera-parameter providing unit512, which has received the camera path, transmits the camera path to the virtual-viewpoint-image generating unit. The time-code providing unit515also obtains a time code from the time-code calculation unit514and transmits the time code to the virtual-viewpoint-image generating unit. Here, the virtual-viewpoint-image generating unit generates a virtual viewpoint image. The information processing apparatus203transmits the camera path and the time code via the communication unit405.

In step S1005, the display516displays the virtual viewpoint image obtained from the virtual-viewpoint-image generating unit. The displayed virtual viewpoint image corresponds to the clip video.

Here, the information processing apparatus203obtains (receives) the virtual viewpoint image via the communication unit.

Referring next toFIG.11, a processing procedure performed when the user selects (presses) any preset position on the preset list312on the GUI308in this embodiment will be described. The processing of this flowchart is started when the user selects any preset position.

In step S1101, the operating-information acquisition unit501transmits a signal to switch the control mode to the preset movement mode to the control-mode determination unit502.

In step S1102, in response to receiving the signal to switch the control mode to the preset movement mode from the operating-information acquisition unit501, the control-mode determination unit502, obtains information on the current control mode from the control-mode holding unit503. If the current control mode is the clip playback mode, the processes from S1103onward are not performed. If the current control mode is the manual control mode or the preset movement mode, the control-mode determination unit502performs the process of step S1103.

If the control mode is the manual control mode, then in step S1103the control-mode determination unit502switches the control mode to the preset movement mode. If the control mode is the preset movement mode, the control-mode determination unit502keeps the preset movement mode. The control-mode determination unit502transmits a signal indicating the switching to the preset movement mode to the control-mode determination unit502and the virtual-camera-parameter calculation unit511.

In step S1104, the virtual-camera-parameter calculation unit511receives the signal to indicate the switching to the preset movement mode from the control-mode determination unit502. The virtual-camera-parameter calculation unit511obtains the camera parameters of the preset position selected by the user from the preset holding unit507. The virtual-camera-parameter calculation unit511also obtains the current camera parameters from the virtual-camera-parameter holding unit510. The virtual-camera-parameter calculation unit511calculates the absolute value of the difference, for each of the obtained parameters (pan, tilt, and roll) of the orientations, and determines whether the value is greater than 180°. In the flowchart, for the parameters, the value of the current virtual camera position is referred to as current orientation, and the value of the preset position is referred to as preset orientation.

If in step S1104there are parameters whose absolute values of the difference are greater than 180°, then in step S1105the virtual-camera-parameter calculation unit511compares the value of the preset position and the value of the current virtual camera position for each of the parameters. For each parameter, if the value of the current virtual camera position is greater than the value of the preset position, the process of step S1106is executed. If the value of the parameter of the current virtual camera position is less than or equal to the value of the preset position, the process of step S1107is executed.

In step S1106, the virtual-camera-parameter calculation unit511subtracts 360° from the value of the current virtual camera position, for each parameter, and substitutes the obtained value for the value of the virtual camera position held by the virtual-camera-parameter holding unit510.

In step S1107, the virtual-camera-parameter calculation unit511adds 360° to the value of the current virtual camera position, for each parameter, and substitutes the obtained value for the value of the virtual camera position held by the virtual-camera-parameter holding unit510.

The processes from step S1104to S1107are repeated until, for each of the parameters of pan, tilt, and roll, the difference between the value held by the virtual-camera-parameter holding unit510and the value of the preset position falls to 180° or less. Thus, for the parameters of pan, tilt, and roll, an integral multiple of 360° is added or subtracted to/from the value of the camera path position.

In step S1108, the remaining-preset-movement-time holding unit509obtains the preset movement time from the preset-movement-time holding unit508and sets the preset movement time for the remaining time to move to the preset position.

The above processes from step S1101to step S1108are performed before the current virtual camera is changed to the preset state.

A processing procedure for changing the virtual camera to the preset state will be described with reference to the flowchart ofFIG.12. The processing of this flowchart is executed next to the flowchart ofFIG.11. The following processing is executed for each frame.

In step S1201, the control-mode determination unit502obtains information on the current control mode from the control-mode holding unit503. If the control mode is the clip playback mode or the manual control mode, the following processes are not performed, and the processing goes to the processing for the next frame. If the control mode is the preset movement mode, the process of step S1202is executed.

In step S1202, the virtual-camera-parameter calculation unit511obtains the camera parameters of the preset position, the camera parameters of the current virtual camera position, and the remaining movement time to calculate the amount of movement per frame.

In step S1203, the virtual-camera-parameter calculation unit511obtains the camera parameters of the current virtual camera position from the virtual-camera-parameter holding unit510and transmits a new camera parameter position to which the amount of movement is added to the virtual-camera-parameter holding unit510.

In step S1204, the remaining-preset-movement-time holding unit509subtracts the time corresponding to one frame from the remaining movement time to update the held remaining present movement time.

In step S1205, the virtual-camera-parameter calculation unit511obtains the remaining preset movement time from the remaining-preset-movement-time holding unit509. If the remaining preset movement time is 0, the process of step S1206is executed. If the remaining preset movement time is not 0, the process for the next frame is executed.

If the remaining preset movement time that the virtual-camera-parameter calculation unit511obtained is 0, then in step S1206the virtual-camera-parameter calculation unit511transmits a signal to switch the control mode to the manual control mode to the control-mode determination unit502. The control-mode determination unit502, which has received the signal, switches the control mode to the manual control mode. After switching the control mode to the manual control mode, the control-mode determination unit502transmits the current control mode (the manual control mode) to the control-mode holding unit503.

The above processing procedure is executed for each frame to move the position of the virtual camera to the preset position.

This embodiment shows an example in which the interpolation procedure (a method for generating a camera path) is selected between generation of a camera path for clip video and movement to the preset position. In other words, in the movement to the preset position, adding a camera parameter correction process so that the difference in orientation parameter between two points falls to 180° or less prevents the camera path from becoming redundant, thereby generating a short path.

Second Embodiment

Next, a second embodiment of the present invention will be described in detail.

In the first embodiment, as illustrated inFIG.8, the information processing apparatus203performs the process of correcting the parameters to move from the current virtual camera position801to the preset position802so as to decrease the path in real-time video. However, some user may want to move the virtual camera along the camera path while imaging some area, not simply passing through a short path. For example, the display target, which is present in a subject region806at the point of gaze803, cannot be viewed from the virtual camera passing through the path805. In this case, the path804may be desirable for the user.

In contrast, a short path may be desirable, such as when the virtual camera is moved to the preset position not using a normal interpolation procedure in generating clip video.

For this reason, in second embodiment, the procedure of interpolation between key frames and the procedure for interpolation between the current position and the preset position can be selected by the user, and the camera path is automatically generated according to the selected interpolation method.

FIG.13illustrates an example of the GUI in the second embodiment. InFIG.13, the components having the same functions as the functions of the GUI inFIG.3are denoted by the same reference signs, and only the difference from the first embodiment will be described.

The camera paths in the clip list310and the preset positions in the preset list312are each assigned a button1301and a button1302indicating whether to correct the parameter for making the path shortest, respectively. The user can switch the ON/OFF of parameter correction by clicking the button1301or the button1302. If the parameter correction is ON, the information processing apparatus203performs the process of correcting the parameters of pan, tilt, and roll of the current virtual camera so that the difference in the values of the parameters between the present virtual camera and the current virtual camera falls to 180° or less, as in the first embodiment. The information processing apparatus203generates a camera path using the corrected parameters. If the correction of the parameters is OFF, the information processing apparatus203generates a camera path without correcting the parameters.

FIG.14is a block diagram illustrating the functional configuration of the information processing apparatus203according to the second embodiment. The information processing apparatus203generates a camera path for the virtual camera using a method selected by the user and transmits the camera path to the virtual-viewpoint-image generating unit. InFIG.14, the components having the same functions as the functions of the information processing apparatus203inFIG.5are denoted by the same reference signs, and only the difference from the first embodiment will be described.

In response to receiving information on a user input, such as to the button1301on the GUI308, the operating-information acquisition unit501transmits a signal to switch the ON/OFF of parameter correction to an interpolation-switching determination unit1401and the virtual-camera-parameter calculation unit511.

The interpolation-switching determination unit1401receives the signal to switch the ON/OFF of parameter correction from the operating-information acquisition unit501. Thereafter, the interpolation-switching determination unit1401switches the ON/OFF of parameter correction and transmits the switched setting to an interpolation-switching holding unit1402. The interpolation-switching determination unit1401obtains the current ON/OFF information from the interpolation-switching holding unit1402and transmits the information to the virtual-camera-parameter calculation unit511.

The interpolation-switching holding unit1402transmits the ON/OFF information to the interpolation-switching determination unit1401. When the ON/OFF is changed by the interpolation-switching determination unit1401, the interpolation-switching holding unit1402obtains the information from the interpolation-switching determination unit1401and holds the information.

In response to receiving the signal to switch the ON/OFF of parameter correction from the operating-information acquisition unit501, the virtual-camera-parameter calculation unit511obtains the information on the ON/Off of parameter correction from the interpolation-switching determination unit1401. If the virtual-camera-parameter calculation unit511obtains information of parameter correction ON, then the virtual-camera-parameter calculation unit511corrects the orientation parameters of the virtual camera so as to interpolate between key frames or the path from the current virtual camera position to the preset position to achieve the shortest path, as described in the first embodiment. If the virtual-camera-parameter calculation unit511obtains information of parameter correction OFF, then the virtual-camera-parameter calculation unit511interpolates between key frames or the path from the current virtual camera position to the preset position without correcting the orientation parameters.

Referring next toFIG.15, a processing procedure for generating a camera path for clip video in this embodiment will be described. InFIG.15, the processes for the same functions as the functions of the processing procedure inFIG.9are denoted by the same reference signs, and only the difference from the first embodiment will be described.

In step S1501, the interpolation-switching determination unit1401obtains ON/OFF information on current parameter correction setting from the interpolation-switching holding unit1402and transmits the information to the clip creating unit505. If the parameter correction setting is ON, the process of step S1502is executed. If the parameter correction setting is OFF, the process of step S902is executed.

In step S1502, the clip creating unit505obtains a list of key frames from the key-frame holding unit504. The difference between the value of a key frame and the value of the key frame with the next frame ID is taken in the order of frame IDs for each of pan, tilt, and roll.

If the absolute value of the difference between the target key frame and the next key frame for each of pan, tilt, and roll is greater than 180°, then in step S1503the clip creating unit505determines which key frame value is greater for each parameter. If the difference between the absolute values is less than or equal to 180°, the processing goes to a process for the next frame.

If the value of the target key frame is greater for each of pan, tilt, and roll, then in step S1504, the clip creating unit505subtracts 360° from the values of all the key frames prior to the target key frame.

If the value of the target key frame is greater for each of pan, tilt, and roll, then in S1505, the clip creating unit505adds 360° to all the key frames prior to the target key frame.

The processes from step S1502to S1505are repeated until the difference between two key frames falls below 180° for each of the parameters of pan, tilt, and roll. Thus, an integral multiple of 360° is added or subtracted to/from the value of the target key frame for each of the parameters of pan, tilt, and roll.

Referring next toFIG.16, a processing procedure when the user presses any preset position on the preset list312on the GUI308in this embodiment will be described. InFIG.16, the processes for the same functions as the functions of the processing procedure inFIG.11are denoted by the same reference signs, and only the difference from the first embodiment will be described.

In this embodiment, user input means, such as buttons on the GUI308, are provided to enable a user to select whether to correct the parameters of the virtual camera for all camera paths for clip video and the preset position so that a camera path can be generated using different interpolation procedures depending on the selection of the user. This allows automatic generation of a camera path desired by the user even when the user wants to interpolate between key frames so as to make the path shortest or when the user wants to take a non-shortest path to the preset position.

In the above embodiments, at least one of A and B may be A or B, or may include A and B.

OTHER EMBODIMENTS

This application claims the benefit of Japanese Patent Application No. 2023-056851, filed Mar. 31, 2023, which is hereby incorporated by reference herein in its entirety.