Information processing device, control method of information processing device, and storage medium

An information processing device decides a viewpoint position and generates a virtual viewpoint image based on the decided viewpoint position by using a plurality of images shot by a plurality of imaging apparatuses. The information processing device includes a determining unit configured to determine a scene related to the virtual viewpoint image to be generated, and a deciding unit configured to decide the viewpoint position related to the virtual viewpoint image in the scene determined by the determining unit, based on the scene determined by the determining unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to technology for generating a virtual viewpoint image.

Description of the Related Art

There is a technology for installing multiple cameras at different positions and synchronously shooting from multiple viewpoints, and generating a virtual viewpoint image using the multiple viewpoint images obtained by the shooting. Japanese Patent Laid-Open No. 2008-015756 discloses a technology for generating a virtual viewpoint image from images taken by multiple imaging devices.

However, there has been the concern that setting viewpoints regarding virtual viewpoint images would be troublesome. For example, a case will be considered where a user views a virtual viewpoint image of a sport event at a stadium. In this case, generating virtual viewpoint images in accordance with viewpoints set by the user enables the user to watch the match from viewpoints preferred by the user. For example, scenes of scoring points in soccer or basketball preferably are viewed from a viewpoint where a player who has made the goal or basket is in the scene, while a long shot following the path of the ball might be desirable for viewing scenes other than scenes of scoring points. However, users not used to setting viewpoints for virtual viewpoint images might take time to set viewpoints appropriate for each scene, or might not be able to set viewpoints at desired positions. Note that the object of viewing regarding virtual viewpoint images is not restricted to sports, and the same problem can occur regarding concerts and other events as well.

SUMMARY OF THE INVENTION

An information processing device decides a viewpoint position and generates a virtual viewpoint image based on the decided viewpoint position by using a plurality of images shot by a plurality of imaging apparatuses. The information processing device includes a determining unit configured to determine a scene related to the virtual viewpoint image to be generated, and a deciding unit configured to decide the viewpoint position related to the virtual viewpoint image in the scene determined by the determining unit, based on the scene determined by the determining unit.

DESCRIPTION OF THE EMBODIMENTS

An embodiment will be described with reference to the drawings.FIG. 1is a diagram illustrating an example of an image processing system according to the present embodiment. The image processing system illustrated inFIG. 1is an information processing system that generates virtual viewpoint images from multiple images obtained by shooting a shooting object such as a field where a sport event or the like is actually to be held, for example, from multiple cameras. A virtual viewpoint image is an image generated using images shot from multiple viewpoints, and is an image expressing virtual shooting results from an optional viewpoint or optional line-of-sight direction.

The image processing system includes a storage device100, an image processing device300, and a user terminal400, as illustrated inFIG. 1. The storage device100stores, for example, multiple images obtained by shooting with multiple cameras installed in a stadium, or information necessary for automatically setting viewpoints regarding virtual viewpoint images. The image processing device300is an information processing device that decides positions of viewpoints relating to virtual viewpoint images, generates virtual viewpoint images, and so forth. The image processing device300generates virtual viewpoint images expressing virtual shooting results from the position of the decided viewpoint. The image processing device300also may generate virtual viewpoint images expressing shooting results in a case of directing a virtual camera in a line-of-sight direction that has been decided. The image processing device300generates virtual viewpoint images using multiple images, obtained by shooting with multiple cameras, that are stored in the storage device100. The image processing device300transmits generated virtual viewpoint images to the user terminal400.

The user terminal400displays virtual viewpoint images output from the image processing device300on a display screen that the user terminal400has. The user terminal400also accepts instruction for moving the position of the viewpoint relating to virtual viewpoint images, changing the line-of-sight direction relating to virtual viewpoint images, and switching viewpoints, based on user input, for example, and transmits transmission signals indicating the contents thereof to the image processing device300as virtual viewpoint information. The user terminal400also requests the image processing device300for automatic setting of virtual viewpoints based on later-described composition scenarios.

Next, an example of the configuration of the storage device100, image processing device300, and user terminal400will be described.FIG. 13is a diagram illustrating an example of a hardware configuration of the storage device100, image processing device300, and user terminal400. A device7000(storage device100, image processing device300, and user terminal400) includes a controller unit700, operating unit709, and display device710.

The controller unit700has a CPU701. The CPU701activates an operating system (OS) by a boot program stored in read only memory (ROM)702. The CPU701executes an application program stored in a hard disk drive (HDD)704, on this OS. The CPU701realizes various types of processing by executing application programs. Random access memory (RAM)703is used as a work area for the CPU701. The HDD704stores applications programs and so forth. Note that the CPU701may be a single processor or multiple processors.

The CPU701is connected to the ROM702and RAM703, and an operating unit interface705, a display unit interface706, and a communication interface707, via a system bus708. The operating unit interface705is an interface to an operating unit709. The operating unit interface705sends information input by the user from the operating unit709to the CPU701. The operating unit709has a mouse and keyboard or the like, for example. The display unit interface706outputs image data to be displayed on a display device710, to the display device710. The display device710has a display such as a liquid crystal display or the like. The communication interface707is an interface for performing communication by Ethernet (registered trademark) or the like, for example, and is connected to a transmission cable. The communication interface707performs input/output of information with external devices via the transmission cable. Note that the communication interface707may be a circuit and antenna for performing wireless communication. The device7000may also perform display control, to display images on the display device710. In this case, the device7000realizes display control by outputting display data to the display device710. Note that not all of the elements illustrated inFIG. 13are indispensable to the storage device100, image processing device300, or user terminal400. For example, the display device710is not an indispensable element for the storage device100and image processing device300. Although description has been made where the controller unit700has the CPU701, this is not restrictive. For example, the controller unit700may have hardware such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA) instated of the CPU701or along with the CPU701. In this case, the hardware such as the ASIC or FPGA may perform part of all of the processing of the CPU701.

Next, an example of the function configuration of the storage device100, image processing device300, and user terminal400will be described with reference toFIG. 1. The configurations illustrated inFIG. 1are realized by the CPU701inFIG. 13reading out various types of programs recorded in the ROM702, and executing control of various parts. Note that part or all of the configurations illustrated inFIG. 1may be realized by dedicated hardware. Examples of dedicated hardware include ASIC and FPGA.

An example of the functional configuration of the storage device100will be described. the storage device100includes an image storing unit10and an event information storing unit20. The image storing unit10stores multiple images synchronously shot by multiple cameras installed surrounding the sports field or the like. The data that the image storing unit10stores may be data indicating still images, data indicating moving images, or data indicating both still images and moving images. Description will be made in the present embodiment with the understanding that that the term “image” includes still images and moving images, unless specifically stated otherwise.

The event information storing unit20stores event informs ion regarding images that the image storing unit10stores. Event information as used here includes basic information such as the title of an event that is the object of shooting, the day it was held, and the venue where it was held, person information such as performers, organizers, participants, and so forth in the event, and log information recording incidents occurring during the event in time-series. Note that event information is not restricted to these, and may include part of these or other information.FIGS. 2A through 2Cillustrate examples of event information in a case where the event is a soccer match. Event basic information may include the title of the event, the date on which the event was held, the venue where the event was held, and team information regarding the names of the teams playing the match, as illustrated inFIG. 2A. Event person information may include the team names, uniform features such as color, pattern, and so forth, as illustrated inFIG. 2B. The event person information may include player information, such as the names of players, uniform numbers, and so forth, as illustrated inFIG. 2B. If the event to be shot is a sumo match, the event person information may include the names of the sumo wrestlers who will be facing off. If the event to be shot is a figure skating competition, the event person information may include the names of skaters performing.

Event log information includes the name of incidents and times of occurrence. In the example illustrated inFIG. 2C, event log information may include information indicating the time at which an incident occurred in terms of elapsed time from a reference time, such as the amount of time elapsed from the start of the match (start of the event), such as “3 minutes into the first half”. In the example illustrated inFIG. 2C, event log information may also include supplemental information such as information for identifying the entity of the incident and individuals related to the incident, such as the name of players. Further, in the example illustrated inFIG. 2C, event log information may include information indicating the state at the time of the incident occurring, such as score information indicating the score at the time of the incident occurring. In the example illustrated inFIG. 2C, event log information may also include the time at which the incident started and scene duration time. Moreover, event log information may also include the start time and end time of the incident.

In a case where the event being shot is a ball sport such as a soccer match or the like, an incident indicated by the event log information might be a “shot (scored)”, “shot (failed to score)”, “pass”, “corner kick”, “foul”, “violation (offside, etc.)” and so forth. Also, an incident indicated by the event log information might be a “goal”, “penalty kick”, “first half over”, “halftime”, “start of second half”, and so forth. In a case where the event being shot is a concert, an incident indicated by the event log information may be the compositions to be performed. In a case where the event being shot is a sport event, an incident indicated by event log information might be the name of the sport. Note that the image processing device300or user terminal400may have part or all of the functional configuration of the storage device100described above.

Next, an example of the functional configuration of the image processing device300will be described. The image processing device300has a generating unit301, a position information obtaining unit302, a calculating unit303, an event information obtaining unit304, and a compiling unit305.

The generating unit301generates a 3D model from multiple images obtained from the image storing unit10, and generates a virtual viewpoint image by model-based rendering where texture mapping is performed according to a viewpoint relating to a virtual viewpoint image obtained from the calculating unit303. Model-based rendering uses a 3D shape (mode) obtained by a 3D shape restoration technique such as silhouette volume intersection, multi-view stereo (MVS, or the like. Model-based rendering is a technique that generates an image taken by a virtual camera, from a viewpoint relating to the virtual viewpoint image, using the generated 3D shape. Other methods may be used for generating a virtual viewpoint image as well, such as image-based rendering or the like, for example. Image-based rendering is a rendering method where a virtual viewpoint image is generated from images shot from multiple viewpoints, without performing modeling (a process of creating shapes of objects using geometric shapes). The generating unit301outputs the generated virtual viewpoint image to the user terminal400via a network.

The position information obtaining unit302analyzes the images that have been shot, and obtains position information regarding a particular subject. For example, the position information obtaining unit302obtains position information of a particular subject based on a 3D model that the generating unit301has generated. The position information obtaining unit302also obtains information indicating the orientation of a particular subject, based on a 3D model that the generating unit301has generated. In the case of a sports event, the particular subject may be a person such as a particular player or referee or the like, or a ball.

The calculating unit303calculates the position and/or line-of-sight direction relating to the viewpoint relating to the virtual viewpoint image being generated. In the present embodiment, simulating a virtual camera to shoot the virtual viewpoint image being generated, and setting the position and/or line-of-sight direction of the viewpoint relating to the virtual viewpoint image being generated, will be referred to as “setting the position and orientation” of the virtual camera. A successive transition of position and orientation of the virtual camera is referred to as a “virtual camera path”. That is to say, the image processing device300can generate multiple virtual viewpoint images based on the virtual camera path that has been set. The image processing device300can also generate multiple virtual viewpoint images generated based on the set virtual camera path as a single moving image.

In a case of having obtained virtual viewpoint information based on user operations from the user terminal400, the calculating unit303decides the position and orientation corresponding to the virtual viewpoint information as being the viewpoint and line-of-sight direction relating to the virtual viewpoint image to be output. Virtual viewpoint information includes at least position information and direction information. In a case of having obtained information indicating a composition scenario, which is conditions for deciding a virtual camera path, from the user terminal400, the calculating unit303calculates the virtual camera path based on the information indicated in the composition scenario. The calculating unit303uses subject position information obtained from the position information obtaining unit302to calculate a temporally successive virtual camera path, based on information indicating the composition scenario. Note that the calculating unit303sets the virtual camera path by setting virtual camera parameters. The virtual camera parameters include the position and/or orientation of the virtual camera. correlated with frame number or time code. The calculating unit303may also set the angle of view, focal length, zoom power, and so forth of the virtual camera, as virtual camera parameters.

The event information obtaining unit304obtains event information from the event information storing unit20. The compiling unit305compiles a subject list from the event person information obtained from the event information storing unit20and the multiple images obtained from the image storing unit10. The subject list here is a list indicating whether or not a virtual viewpoint image can be generated from viewpoints regarding each of subjects such as players, the ball, and so forth, in a scene of generating a virtual viewpoint image. The subject list will be described later in detail.

The compiling unit305also compiles a composition scenario from a scene regarding which the user has requested generation of a virtual viewpoint image obtained from the user terminal400, an incident that has occurred in the scene, and a subject list. The composition scenario gives conditions and ideas for setting a virtual camera path. The composition scenario is represented as templates and rules relating to conditions on the position and orientation for the virtual camera. A composition scenario list is a list of composition scenarios that are selectable in a specified scene. The composition scenario list will be described later in detail. Note that the compiling unit305may have functions of generating a later-described scene selection screen. Also, the storage device100or user terminal400may have part of all of the functional configuration of the image processing device300described above.

Next, an example of the functional configuration of the user terminal400will be described. The user terminal400has a display unit401, a determining unit402, a selecting unit403, and a setting unit404.

The display unit401displays virtual viewpoint images obtained from the image processing device300via a network, and various types of information. The determining unit402performs determination to divide an event that has been shot into multiple scenes.

The determining unit402creates a scene selection screen showing multiple scenes as options, and displays the created scene selection screen on the display unit401. The determining unit402divides the event that has been shot into multiple scenes based on event information. The determining unit402displays on the display unit401the scene selection screen for the user to select a scene for which to generate a virtual viewpoint image, in which the multiple scenes into which the event has been divided based on the event information are options. Note that the user terminal400may receive a scene selection screen from the image processing device300and display the received scene selection screen on the display unit401. The determining unit402also determines a scene relating to generating a virtual viewpoint image that has been identified based on user operations. The determining unit402may also transmit information indicating the scene relating to generating a virtual viewpoint image that has been identified based on user operations to the image processing device300via a network. The determining unit402identifies at least one of the time relating to the scene that has been determined, the location, and an occurring incident. The determining unit402may also identify, based on the determined scene, at least one of the shooting time of the virtual viewpoint image to be generated, the location of the virtual viewpoint image to be generated, and the incident shown in the virtual viewpoint image to be generated.

The selecting unit403creates a composition scenario selection screen from the composition scenario list compiled by the compiling unit305, and displays the compiled composition scenario selection screen on the display unit401. Note that the user terminal400may receive the composition scenario selection screen from the image processing device300and display the received composition scenario selection screen on the display unit401. The selecting unit403also selects the composition scenario relating to the generating of the virtual viewpoint image, that has been identified from the composition scenario list based on user operations. Also, in a case where a subject needs to be specified from the selected composition scenario, the selecting unit403creates a subject selection screen from the subject list obtained from the image processing device300, and displays this on the display unit401. Note that the user terminal400may receive a subject selection screen from the image processing device300and display the received subject selection screen on the display unit401.

The setting unit404sets virtual camera path conditions based on the composition scenario and subject selected by the user, and transmits the set virtual camera path conditions to the image processing device300via a network. The virtual camera path conditions here are conditions for deciding virtual camera parameters, such as the position and orientation of the virtual camera, regarding the scene for generating the virtual viewpoint image. The virtual camera path conditions are conditions stipulating what the position and orientation of the virtual camera should be to shoot a certain subject at a time where a certain incident has occurred, for example. Note that the storage device100or image processing device300may have part of all of the functional configuration of the user terminal400described above. Although the image processing system is described as being made up of three devices, this may be one, two, or four or more devices.

The operations of the image processing system having the above-described configuration will be described.FIG. 3is a sequence diagram of virtual viewpoint image generating processing by the image processing system. The image processing system starts the virtual viewpoint image generating processing by the user terminal400activating a virtual viewpoint image viewing application. The user terminal400activates the virtual viewpoint image viewing application, and starts scene selection screen presenting processing (S4001). The user terminal400requests the image processing device300for event information (S4002).

Upon receiving the request for event information from the user terminal400, the image processing device300obtains event information that the event information storing unit20stores. The image processing device300transmits the obtained event information to the user terminal400(S3001). The user terminal400generates a scene selection screen based on the obtained event information, and displays the scene selection screen.

FIG. 4Aillustrates an example of a scene selection screen displayed based on the event log information in a case where the event information that the user terminal400obtains is that illustrated inFIG. 2C. In the example illustrated inFIG. 4A, the scene selection screen presents, out of information included in event information, information indicating the time of occurrence of an incident, the name of the incident, and score information regarding incidents regarding which there is a possibility of change in score, in the scene selection screen. A scene selection screen may be used that presents part of this information. Also, the image processing device300may generate the scene selection screen illustrated inFIG. 4A, and the user terminal400may obtain the scene selection screen from the image processing device300and display it.

The user terminal400displays the scene selection screen, and accepts selection of a scene from the user (S4003). By accepting selection of a scene from the user, the user terminal400identifies the scene regarding the time, location, and situation for which the virtual viewpoint image is to be generated.

The user terminal400starts composition scenario selection screen presenting processing, to display a composition scenario selection screen (S4004). The user terminal400requests the image processing device300for a composition scenario list relating to this scene, along with the information for identifying the screen that the user has selected (S4005). The image processing device300compiles a subject list relating to the selected scene (S3002). The image processing device300compiles a composition scenario list relating to the selected scene using the compiling unit305(S3003).

Now, the subject list compiling processing in S3002and the composition scenario list compiling processing in S3003will be described in detail.FIG. 5Ais a flowchart illustrating the flow of the subject list compiling processing by the compiling unit305of the image processing device300. The flowchart inFIG. 5Astarts with reception of a request to obtain a composition scenario list. Note that the processing of the flowchart described below is realized by the CPU701of the image processing device300computing information and controlling the hardware. Note that at least part of the steps in the flowchart may be realized by dedicated hardware. Examples of dedicated hardware include ASIC and FPGA.

The compiling unit305starts subject list compiling processing by receiving a request to obtain a composition scenario list from the user terminal400. A subject list here is a list that indicates whether or not a virtual viewpoint image can be generated regarding the subjects such as players and the ball, for each frame from the start time to end corresponding to the specified scene. Depending on the state in which multiple cameras are installed, there can be cases where a virtual viewpoint image of a subject situated at a particular position cannot be generated. Accordingly, there can be periods of time where a virtual viewpoint image including a certain subject such as a player or the like cannot be generated, for example, depending on the position of the field that is the object of shooting.

The compiling unit305obtains subject position information relating to the specified scene from the position information obtaining unit302(S1501). The subject position information is information indicating the position of a particular subject. The subject position information may also be information relating to a subject that can be included in a virtual viewpoint image in the specified scene. The subject position information also may be information regarding a subject that can be included in the virtual viewpoint image from a viewpoint relating to a subject, such as the line-of-sight of the subject in the specified scene. Information regarding the position and orientation of each subject, and information for identifying each subject, may be included in the subject position information in correlation with time. The subject position information may also include the ID of each subject.

FIG. 5Bis a flowchart illustrating the flow of subject position information obtaining processing by the position information obtaining unit302. The position information obtaining unit302starts the subject position information obtaining processing by accepting a request to obtain subject position information from the compiling unit305. The position information obtaining unit302obtains a 3D model at the start time of the specified scene from the generating unit301(S2501). Subsequently, the position information obtaining unit302obtains the position of all particular subjects included in the 3D mode at the start time of the scene (S2502). An arrangement may be made where the position information obtaining unit302obtains positions of particular subjects included a predetermined range according to the position where an incident occurs in the scene selected in S4003, to reduce the amount of processing.

The position information obtaining unit302imparts IDs to individually distinguish each subject situated in a range regarding which a virtual viewpoint image can be generated (S2503). In a case where the position of the subject at the time of starting of the scene is that illustrated inFIG. 6B, there are three players wearing checked uniforms, three players wearing striped uniforms, and one ball, in a circle indicated by a dotted line which is a region that a virtual viewpoint image can be generated, so ID01through ID07are imparted. Next, the position information obtaining unit302obtains a 3D model of time corresponding to the next frame in the virtual viewpoint image to be generated from the generating unit301(S2504). The position information obtaining unit302obtains the position of all subjects included in the 3D model for the time corresponding to the next frame that has been obtained (S2505).

The position information obtaining unit302identifies, for each of the subjects situated within the region regarding which a virtual viewpoint image can be generated for the time corresponding to the next frame, the subject that has the smallest positional difference regarding the time corresponding to the previous frame. The position information obtaining unit302then imparts the same ID as the previous frame to subjects of which the positional different is a predetermined value or smaller (S2506). In a case where there is a subject of which the positional difference as to time corresponding to the previous frame is not the predetermined value or smaller in the subjects situated within the region regarding which a virtual viewpoint image can be generated for the time corresponding to the next frame, the position information obtaining unit302imparts a new ID thereto (S2507). In a case where the subject positions at the time corresponding to the next frame are as illustrated in the example ofFIG. 6C, there are three players wearing checked uniforms, three players wearing striped uniforms, and one ball, in the dotted circle which is a region that a virtual viewpoint image can be generated. The subjects are imparted the same IDs of ID01through ID06, based on the positional difference in this frame as to the time corresponding to the previous frame. In the example illustrated inFIG. 6C, the player in the checked uniform at the lower right has moved outside of the circle in the start frame, so no ID is imparted thereto. The player in the checked uniform at the upper right has moved into the circle from outside of the circle in the start frame, so a new ID08is imparted thereto. The position information obtaining unit302repeatedly performs the processing of S2504through S2507until the end time of the screen for generating the virtual viewpoint image. In a case of having ended processing up to the portion corresponding to the end time of the scene (Yes in S2508), the position information obtaining unit302ends the subject position information obtaining processing.

Returning to the description inFIG. 5A, the compiling unit305obtains event person information from the event information storing unit20(S1502). The compiling unit305analyzes multi-viewpoint video obtained from the image storing unit10, and extracts features such as the color, pattern, and uniform number of uniforms, for each of the subjects distinguished by IDs (S1503). Next, the compiling unit305matches the features such as color, pattern, and uniform number of uniforms included in the event person information, with the extracted features, and identifies who the individual subjects distinguished by IDs are (S1504). Subjects that are not people, such as the ball or the like, can be identified by storing features such as shape and size in the compiling unit305. The timing at which S1502is processed may be any timing as long as it is before S1504, and accordingly maybe before S1501or after S1503.

Next, the compiling unit305compiles a subject list (S1505).FIG. 6Aillustrates an example of a subject list. It can be seen from the example illustrated inFIG. 6Athat five players and one ball have been situated in a region regarding which a virtual viewpoint image can be generated for all frames from the start to the end of the scene, and two players were situated outside the region regarding which a virtual viewpoint image can be generated at part of the frames.

Next, the composition scenario list compiling processing of S3003will be described in detail.FIG. 7is a flowchart illustrating the flow of the composition scenario list compiling processing by the compiling unit305. Upon the subject list compiling processing ending, the compiling unit305starts the composition scenario list compiling processing. The compiling unit305sets a common composition scenario that is not dependent on the scene content (S1701). An example of a common composition scenario that is not dependent on the scene content is, in a case that the event is a soccer match, “view movement of entire scene from above” as inFIG. 8. In this composition scenario, a plane view virtual viewpoint image where the virtual camera is situated overhead is generated, regardless whether the selected scene is a shot scene, foul scene, or any other such scene. Another example of a common composition scenario that is not dependent on the scene content is “view from sideline with ball in center of view”. Yet another example of a common composition scenario that is not dependent on the scene content is “view toward the goal with the position of the ball as the viewpoint”.

Next, the compiling unit305identifies the type of selected scene (S1702). In a case where the event is a soccer match, the compiling unit305may classify the scene for generating a virtual viewpoint image into one of scoring, failing to score, offside, and fouling, for example. In this case, the compiling unit305classifies the types of scenes such as shots, corner kicks, penalty kicks, and other such scenes, into whether scored or failed to score. Scene types of offside and fouling are offside and fouling, respectively.

The compiling unit305adds a composition scenario that is dependent on the identified scene type (S1703).FIG. 8illustrates examples of composition scenarios where the scene type is scoring, in a case where the event is a soccer match. Examples of composition scenarios inFIG. 8include “follow ball at front line of * (where * is offense, defense, etc.)” and “watch goal being scored at point of * (where * is kick, heading, goal, etc.)”. In a composition scenario of “follow ball at front line of offense”, a virtual viewpoint image is generated where the virtual camera successively follows offense players in contact with the ball. This sort of composition scenario is an example of a composition scenario where the position and orientation of the virtual camera is decided from the positional relation of the ball and multiple offense-side players in each frame from the start to the end of the scene.

The “watch goal being scored at point of goal” scenario generates a virtual viewpoint image where the virtual camera is placed behind the goal on a straight line from the point where the ball was kicked to the point at which the goal line was crossed, thus capturing the way in which the ball flies into the goal. Such a composition scenario is an example of composition scenarios where the position and orientation of the virtual camera is decided from the positional relation of the ball and players, at a particular time such as a kick scene or goal scene.

Also, the “carefully watch instant of foul” composition scenario inFIG. 8generates a virtual viewpoint image where the virtual camera is rotated around the position at which the foul occurred, while stopping time at the frame in which the foul occurred, so as to observe the instant of the foul from 360 degrees. This composition scenario is an example of a composition scenario where the virtual camera is moved in a state with time stopped, based on the position of a player in a particular frame. That is to say, in this case, multiple virtual viewpoint images are generated of the subject at a particular point in time, from various viewpoints.

Note that the above-described composition scenarios are examples and that other composition scenarios may be used. Further, composition scenarios relating to the scene may be used, without classifying scenes into types. For example, in a case where the incident occurring in the scene regarding which a virtual viewpoint image is to be generated is “shot (scored)” or “shot (failed to score)”, then “watch from line-of-sight of goalie” or “watch from line-of-sight of striker” may be used as composition scenarios. Also, in a case where the incident occurring in a scene regarding which the virtual viewpoint image is to be a generated is a “pass”, “watch from line-of-sight of player receiving pass” or “watch from line-of-sight of player delivering pass” may be used as composition scenarios. Alternatively, composition scenarios relating to scenes may be decided by user input.

Next, the compiling unit305confirms the subject list (S1705), and determines whether or not to add a person-specified scenario. The compiling unit305determines in S1704whether or not there is a person situated in a region regarding which a virtual viewpoint image can be generated for all frames from the start of the scene to the end, for example. In a case where there is a person situated in a region regarding which a virtual viewpoint image can be generated for all frames, the compiling unit305determines to add a person-specified scenario, and in a case where there is no person situated in a region regarding which a virtual viewpoint image can be generated for all frames, the compiling unit305does not determine to add a person-specified scenario. Also, for example, the compiling unit305determines in S1704whether or not there is a person in the subject list from the start to end of the scene. In a case where there is a person in the subject list from the start to end of the scene, the compiling unit305determines to add a person-specified scenario, and in a case where there is no person in the subject list from the start to end of the scene, the compiling unit305does not determine to add a person-specified scenario. Although description is made here to determine to add a person-specified scenario in a case where there is a person situated in a region regarding which a virtual viewpoint image can be generated for all frames from start to end, this is not restrictive. For example, an arrangement may be made to add a person-specified scenario in a case where there is a person situated in a region regarding which a virtual viewpoint image can be generated for a predetermined number of frames or more. Also, for example, an arrangement may be made to determine to add a person-specified scenario in a case where there is a person facing the point of occurrence of the incident occurring in the scene. Also, an arrangement may be made to determine to add a person-specified scenario in a case where there is a person by whose tine-of-sight a virtual viewpoint image can be generated in the scene. Further, an arrangement may be made where the processing of S1705is omitted, and a person-specified scenario is always added.

Next, in a case of determining to add a person-specified scenario (Yes in S1705), the compiling unit305adds a person-specified composition scenario (S1706). Examples of a person-specified scenario include, in a case where the event is a soccer match, “view from viewpoint of selected player” or “continue to follow selected player from * (where * is behind, in front, etc.)”. In the “view from viewpoint of selected player” composition scenario, a virtual viewpoint image is generated that reproduces the field of view of the particular player, by matching the position and orientation of the virtual camera with the position and orientation of the face of the particular player. Such a composition scenario is an example of a composition scenario where the position and orientation of the virtual camera are decided from the position and orientation of one particular player over all the frames. A configuration may also be made where a configuration scenario is added that is person-specific regarding the person related to the scene regarding which the virtual viewpoint image is to be generated. For example, a person-specified scenario may be added regarding a player who has come into contact with the ball, in the scene regarding which the virtual viewpoint image is to be generated.

Next, the compiling unit305inspects whether or not a virtual viewpoint image can be generated for the selected scene regarding the added composition scenarios (S1707). In a case where the event is a soccer match, a scoring compositional scenario “watch goal being scored at point of heading”, for example, is not satisfied if the shot was kicked, so judgement is made that a virtual viewpoint image cannot be generated. The compiling unit305excludes composition scenarios regarding which judgment has been made that a virtual viewpoint image cannot be generated. The compiling unit305compiles a composition scenario list of composition scenarios regarding which judgment has been made that a virtual viewpoint image can be generated (S1708). The composition scenario list that the compiling unit305has created is transmitted to the user terminal400and presented to the user as a composition scenario selection screen.

FIG. 4Billustrates an example of a composition scenario selection screen in a case where the selected screen is a point scoring scene such as “3 minutes into first half, shot, (1-0)” in the example illustrated inFIG. 2Cor the like, and a person-specified scenario is added. Multiple composition scenarios that are conditions for setting the virtual camera path are presented to the user in a selectable manner inFIG. 4B.

Returning to the description ofFIG. 3, the user terminal400displays a composition scenario selection screen such as illustrated inFIG. 4Bfor example, based on the composition scenario list. The user terminal400selects a composition scenario identified from the composition scenario selection screen, based on user specification (S4006). In a case where the selected composition scenario relates to a person, the user terminal400presents a subject selection screen (S4007). The user terminal400displays a subject selection screen such as illustrated inFIG. 4C, for example, based on the subject list.FIG. 4Cis an example of a subject selection screen in a case where the viewer has selected “view from viewpoint of selected player” in the composition scenario selection screen illustrated inFIG. 4B. The user is presented with multiple people inFIG. 4C, for selection of a person as a condition for setting the virtual camera path. The subject selection screen may also include additional information such as name of person, uniform number, team, facial photograph, and so forth.

The user terminal400selects a particular person from the subject selection screen (S4009). Next, the user terminal400starts virtual viewpoint image creation processing (S4010). The setting unit404of the user terminal400sets virtual camera path conditions based on the selected scene received from the selecting unit403, and the composition scenario and person (S4011). Virtual camera path conditions are conditions stipulating the relation between the position and orientation of the virtual camera, and positional information of the subject, in order to realize the composition scenario.

FIG. 9is an example of settings for virtual camera path conditions by the setting unit404, in a case where the composition scenario list is the example illustrated inFIG. 8. For example, in a case where the event is a soccer match, the position of the virtual camera in the composition scenario “view movement of entire scene from above” is “8 meters above center of circle encompassing path of movement of ball”, and the orientation of the virtual camera is “straight down”. Note that these specific numbers are only exemplary, and that the position of the virtual camera in the composition scenario “view movement of entire scene from above” may be another position.

Now, a specific example of virtual camera path conditions will be described with reference toFIGS. 10A through 10C. In the scene regarding which a virtual viewpoint image is to be generated, a player of ID02kicks the ball with ID03at a time corresponding to the start frame of the virtual viewpoint image to be generated, and delivers a pass to a player with ID06, as illustrated inFIG. 10A. The player of ID06who has received the pass kicks a goal shot. The frame of the virtual viewpoint image corresponding to the time at which the player of ID06has kicked the shot is thus a shot frame.

The ball shot by the player of ID06crosses the goal line. The frame in the virtual viewpoint image corresponding to the time of crossing the goal line is the goal frame. The frame of the virtual viewpoint image corresponding to the time of ending the virtual viewpoint image to be generated is the end frame.

A traveling path52of the ball from the start frame to the end frame is as shown inFIG. 10B. A circle53that encompasses this traveling path52of the ball is as shown inFIG. 10B. Accordingly, the position of the virtual camera that satisfies the virtual camera path conditions for “view movement of entire scene from above” is “8 meters above center of circle encompassing path of movement of ball”. Thus, the setting unit404sets how to use which subject position information as virtual camera path conditions, so that the calculating unit303can decide the position and orientation of the virtual camera using the traveling path of the ball.

Next, an example of virtual camera conditions for the composition scenario “watch goal being scored at point of goal” will be described with reference to the scene illustrated inFIG. 10A. The virtual camera position in the composition scenario “watch goal being scored at point of goal” is “3 meters behind goal point”, the orientation of the virtual camera is “ball” before the shot and “point of shot” after the shot. The position of the virtual camera51is a position where a straight line from the ball position in the shot frame to the ball position in the goal frame has been extended by three meters, as illustrated inFIG. 10C. The orientation of the virtual camera51is set so that the ball is at the middle of the screen from the start frame to the shot frame, and fixed at the shot position from the shot frame to the end frame, as illustrated inFIG. 10C. Thus, the setting unit404uses position information of the subject so that the calculating unit303can decide the virtual camera position, and set conditions regarding which position of which frame to locate the virtual camera, and how to orient the virtual camera. Note that these specific numbers and specific positions are only exemplary, and that the virtual camera position for the composition scenario “watch goal being scored at point of goal” may be other positions.

Next, an example of virtual camera conditions for the composition scenario “carefully watch instant of foul” regarding a foul scene will be described. The position of the virtual camera in the composition scenario “carefully watch instant of foul” is the position of the assistant referee from the start to the occurrence of the foul. The orientation of the virtual camera is set to where the player that is the subject of the foul is in the image from the start until the foul occurs. In a case where a foul occurs, the position of the virtual camera is rotated around the player regarding which the foul has occurred 360 degrees in the clockwise direction, at the timing at which the foul occurred (staying at the time at which the foul occurred). That is to say, the timing at which the foul occurred can be taken from various angles in the virtual viewpoint image. When the virtual camera returns to the position of the assistant referee, the virtual camera is fixed and elapsing of time is resumed. Thus, the setting unit404sets conditions regarding at what time to stop the virtual camera, and from which subject position to rotate the camera, in which orientation, around which subject, so as to be able to decide the timing for stopping elapsing of time in the virtual viewpoint image and the rotating state of the virtual camera. Note that these specific numbers and specific positions are only exemplary, and that the virtual camera position for the composition scenario “carefully watch instant of foul” may be other positions. Also, a composition scenario may be provided where the virtual camera rotates in the same way, around a player kicking a shot, and the time of shooting.

The setting unit404also sets, as virtual camera path conditions, how to use the position information of which subject in which frame, regarding composition scenarios other than those described above which are illustrated inFIG. 9. Although an example has been described where the user terminal400sets virtual camera path conditions, a configuration may be made where the image processing device300calculates the virtual camera path, without the user terminal400setting virtual camera path conditions. In this case, the user terminal400may transmit to the image processing device300the selected scene, composition scenario, and information for identifying a person. The image processing device300may be arranged to set the virtual camera path based on the received selected scene, composition scenario, and information for identifying a person.

Returning to the description ofFIG. 3, the user terminal400transmits a virtual viewpoint image creation request, including the selected scene, composition scenario, information for identifying a person, and information indicating virtual camera conditions, to the image processing device300(S4012). Upon receiving the virtual viewpoint image creation request from the user terminal400, the image processing device300creates a virtual viewpoint image based on the received information (S3005).

Now, the virtual camera path calculation processing of the image processing device300will be described with reference toFIG. 11.FIG. 11is a flowchart illustrating the flow of virtual camera path calculation processing performed by the calculating unit303of the image processing device300.

The calculating unit303of the image processing device300starts the virtual camera path calculation processing by receiving the virtual camera path conditions included in the virtual viewpoint image creation request from the user terminal400. The calculating unit303obtains the virtual camera path conditions (S1101). The calculating unit303identifies the subject regarding setting the virtual camera path indicated by the virtual camera path conditions (S1102). For example, out of the examples illustrated inFIG. 9, in a case where the virtual camera path conditions are that the position of the virtual camera is “8 meters above center of circle encompassing path of movement of ball”, and the orientation of the virtual camera is “straight down”, the calculating unit303identifies the ball to be the subject regarding which the position will be used from start to end. Also, out of the examples illustrated inFIG. 9, a case where the virtual camera path conditions are that the position of the virtual camera is “3 meters behind goal point” will be considered. In this case, if the orientation of the virtual camera is “ball/kick point”, the calculating unit303identifies the ball from the start to a player shooting the ball, and the ball in the goal frame, to be the subject regarding which the position is used. Also, in this case, the player kicking the shot may be identified to be the subject regarding which the position is used.

The calculating unit303may identify time information used at the time of setting the virtual camera path, such as the time of a shot or the time of a goal being scored, from event log information in the event information storing unit20, or may identify the time from the positional relation between the player who has scored a goal, the goal line, and the ball. Also, time information used at the time of setting the virtual camera path, such as the time of a shot or the time of a goal being scored, may be identified based on scene determination by image processing of shot images. For example, the time of a shot may be determined based on the ball leaving the scoring player by image processing, and the time may be obtained from the time at which that image was shot. A configuration may be made where the last frame in which the positional difference between the scoring player and the ball is a predetermined value or smaller is found and identified from the frames of the shot image, for the image processing where the ball is determined to have left the player. Also for example, the time at which a goal is scored is identified by determining the first frame at which the ball is situated on the inner side of the goal line, and the time is identified from the time at which that frame was shot.

Thus, the calculating unit303identifies which subject and at which shooting time to use for the position, based on the virtual camera path conditions. lithe calculating unit303cannot identify a subject to use for the position due to, for example, the shot point being outside of the range regarding which a virtual viewpoint image can be generated (No in S1103), an error is notified to the user terminal400(S1108), and the virtual camera path calculation processing is ended. If the calculating unit303can identify a subject to use for the position (Yes in S1103), the position information of the subject at a particular time is obtained from the position information obtaining unit302(S1104). The calculating unit303decides the position and orientation of the virtual camera for each frame in the virtual viewpoint image to be generated, based on the position information of the subject that has been obtained, and the virtual camera path conditions (S1105). In a case where the composition scenario is “view movement of entire scene from above” out of the examples illustrated inFIG. 9, the virtual camera path will be one facing straight down from 8 meters above the center of the circle53illustrated in the example inFIG. 10Bfor all frames from the start of the scene to the end. In a case where the composition scenario is “watch goal being scored at point of goal” out of the examples illustrated inFIG. 9, the virtual camera path will be one where the position of the virtual camera51is as illustrated inFIG. 10C, panning from the left to the right following the ball, with the panning stopping at the position where the player kicks a shot. Thus, the calculating unit303decides virtual camera parameters for all frames to create a virtual camera path from the start to the end of the scene, so as to satisfy virtual camera path conditions, thereby automatically calculating a virtual camera path. The virtual camera parameters decided here are, for example, the position and/or line-of-sight direction for each frame in the generated virtual viewpoint image. Note that in a case where the movement of the virtual camera that has been calculated exceeds a predetermined threshold value, reproduction speed may be set by the calculating unit303as a virtual camera parameter, such as setting the reproduction speed to half speed, so that the viewer can view the virtual viewpoint image better.

Next, a line connecting the positions of the virtual camera in each of the frames identified by the calculated virtual camera path is smoothed, and the coordinates on the smoothed line are identified as positions of the virtual camera by the calculating unit303(S1106). Smoothing of the virtual camera path is to suppress the virtual camera from shaking and giving the impression of the virtual viewpoint image having been shot from a camera with a shaking hand, thereby preventing the viewer from experiencing visually induced motion sickness. Note that the processing of S1106may be omitted. Next, the calculating unit303requests the generating unit301to generate a virtual viewpoint image in accordance with the calculated virtual camera path (S1107).

Returning to the description inFIG. 3, the generating unit301of the image processing device300generates a virtual viewpoint image satisfying the virtual camera parameters of the virtual camera that the calculating unit303has calculated. That is to say, the generating unit301generates a virtual viewpoint image in accordance with the position and/or line-of-sight direction of the viewpoint relating to the virtual viewpoint image decided in accordance with the composition scenario.

The image processing device300outputs the generated virtual viewpoint image to the user terminal400. The user terminal400displays the received virtual viewpoint image on the display unit401. The user terminal400presents the received virtual viewpoint image to the viewer by reproducing at the display unit401(S4013).

As described above, according to the present embodiment, in a case of shooting an event and generating a virtual viewpoint image from the results of shooting, the event is divided into multiple scenes, and the user can select a scene for which to generate a virtual viewpoint image out of the multiple scenes. Accordingly, the user can generate a virtual viewpoint image regarding a desired time, location, or incident in an event without having to perform troublesome operations. Also, according to the present embodiment, selecting desired conditions from multiple conditions relating to a selected scene that are for deciding the position of the viewpoint for the virtual viewpoint image enables the virtual viewpoint image to be generated without having to perform troublesome operations. Accordingly, the user can view a virtual viewpoint image along a virtual camera path that is appropriate for a selected scene, according to the preferences of the user, without having to perform troublesome operations. The user also can view a virtual viewpoint image of a selected scene on a virtual camera path without having to perform troublesome operations. The viewer can also be provided with a virtual viewpoint image not restricted to a single subject position or orientation, by setting virtual camera conditions from a selected scene, composition scenario, and person, and automatically calculating a virtual camera path from the start to end of the scene. Thus, according to the present embodiment, even users who are not accustomed to performing setting of viewpoints regarding virtual viewpoint images can easily set the viewpoint for the virtual viewpoint image.

Note that in the above-described embodiment, an arrangement may be made where the number of composition scenarios relating to a selected scene is confirmed, and if the number is one, that composition scenario is automatically selected without presenting the user with a composition scenario selection screen. Also, description has been made in the above-described embodiment regarding an arrangement where the user terminal400obtains event log information, distinguishes scenes, and generates a scene selection screen, but this is not restrictive. The image processing device300may obtain event log information, and distinguish scenes and generate a scene selection screen.

An arrangement may also be made where a time or period is specified at the user terminal400based on user operations to identify a scene regarding generating of a virtual viewpoint image.FIG. 12illustrates an example of a screen of accepting user instructions, to identify a scene regarding generating of a virtual viewpoint image by specifying a time or period. The screen inFIG. 12is for specifying at least the two of a scene start time and the duration to the end of the scene. The screen may be a screen that accepts the scene start time and scene end time. The duration to the end of the scene may be restricted to within a predetermined value, so as to facilitate identification of what sort of a scene has been specified. The image processing device300or user terminal400may be arranged to identify scenes from event log information. If the specified scene is from 10:03:48 to 10:03:56 eight seconds later as illustrated inFIG. 12, it can be seen that this includes the scoring scene of the shot at 10:03:50, two seconds after the specified scene starts, in the example of the event log information illustrated inFIG. 2C.

Modification

A modification of the above-described embodiment will be described with reference to a flowchart illustrating the flow of operations of the image processing device300, illustrated inFIG. 14. Note that the configurations of the devices in the modification are the same as in the embodiment described above.

In S1901, the image processing device300presents the user with multiple scenes that are candidates for generating a virtual viewpoint image. Note that the image processing device300generates the above-described scene selection screen based on event information, and transmits the scene selection screen to the user terminal400.

In S1902, the image processing device300determines whether or not a scene for generating a virtual viewpoint image has been identified by a user having specified a scene by user operations based on information received from the user terminal400. In a case of having received information identifying a scene from the user terminal400, the image processing device300determines that a scene for generating a virtual viewpoint image has been identified.

In a case of having identified a scene for generating a virtual viewpoint image, in S1903the image processing device300identifies the location, time, and incident of the identified scene, based on the event information. For example, in a case where the identified scene is a scene of a shot being kicked, the location based on the position of the shot is determined by the image processing device300to be the location of the scene, a series of time of play over which the shot was kicked to be the identified time of the scene, and the shot to be the incident of the scene.

In a case of having identified the scene for generating a virtual viewpoint image, in S1904the image processing device300identifies people related to the location, time, and incident identified in S1903. For example, in a case where the identified scene is a scene of a shot being kicked, the player who has kicked the shot, the goalie, and a defending player closest to the offense player who has kicked the shot, are identified by the image processing device300as being people related to the scene.

In S1905, the image processing device300obtains a composition scenario related to at least one of the location, time, and incident of the identified scene, and a composition scenario relating to a subject identified in S1903. In a case where the identified scene is a scene of a shot being kicked, the image processing device300obtains, for example, “watch goal being scored at point of goal” illustrated inFIG. 9, as a composition scenario related to at least one of the location, time, and incident of the identified scene. The image processing device300also obtains “watch from viewpoint of the player who has kicked the shot, the goalie, or the defending player closest to the offense player who has kicked the shot” as a composition scenario related to the subject identified in S1903. The image processing device300creates a composition scenario selection screen for selecting a composition scenario from the obtained composition scenarios, and transmits to the user terminal400to be presented to the user.

The image processing device300determines in S1906whether or not a composition scenario relating to the virtual viewpoint image to be generated has been identified by the user having specified a composition scenario by user operations, based on information received from the user terminal400. In a case of having received information identifying a composition scenario from the user terminal400, the image processing device300determines that the composition scenario has been identified.

In a case of having identified a composition scenario related to the virtual viewpoint image to be generated, in S1907the image processing device300identifies virtual camera parameters based on the identified composition scenario. For example, in a case where “watch from viewpoint of the player who has kicked the shot” is identified as the composition scenario related to the virtual viewpoint image to be generated, the image processing device300obtains position information and orientation of the player who kicked the shot in the scene to be generated, from the position information obtaining unit302. The image processing device300sets the virtual camera parameters where the position information and orientation of the player who kicked the shot in the scene to be generated are the virtual camera path.

In S1908, the image processing device300generates a virtual viewpoint image based on the identified virtual camera parameters, and in S1909, the image processing device300outputs the generated virtual viewpoint image to the user terminal400.

Thus, according to the above modification, even users who are not accustomed to performing setting of viewpoints regarding virtual viewpoint images can easily set the viewpoint for the virtual viewpoint image. According to the above embodiment, setting of viewpoints regarding virtual viewpoint images can easily be performed.

Other Embodiments

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It will of course be understood that this invention has been described above by way of example only, and that modifications of detail can be made within the scope of this invention.

This application claims the benefit of Japanese Patent Application No. 2017-166106 filed Aug. 30, 2017, which is hereby incorporated by reference herein in its entirety.