Information processing apparatus, information processing system, information processing method, and storage medium

An information processing apparatus includes a first detection unit configured to detect a first object located on a first direction side of a refracting surface from an image obtained by an imaging apparatus located on the first direction side, a second detection unit configured to detect a second object located on a second direction side of the refracting surface from an image obtained by an imaging apparatus located on the second direction side, an obtaining unit configured to obtain position information indicating at least either one of a positional relationship between the first and second objects and positions of the first and second objects, and an identification unit configured to identify positions of the plurality of imaging apparatuses in a common coordinate system, based on a result of detection performed by the first detection unit, a result of detection performed by the second detection unit, and the position information.

BACKGROUND

Field of the Disclosure

The present disclosure relates to a technique for identifying a position of an imaging apparatus based on a captured image, and a technique for generating shape data on an object based on captured images.

Description of the Related Art

There is a technique for installing a plurality of imaging apparatuses at different positions, performing synchronous imaging, and generating a virtual viewpoint image of which the point of view can be freely changed by using a plurality of captured images obtained by the synchronous imaging. More specifically, the virtual viewpoint image is generated by generating three-dimensional shape data on an object included in the captured images based on the plurality of captured images, and performing rendering processing based on the position and direction of a virtual viewpoint.

To generate the three-dimensional shape data on the object based on the captured images obtained by the plurality of imaging apparatuses, the positions of the respective imaging apparatuses needs to be identified in a common coordinate system. Japanese Patent Application Laid-Open No. 2018-207252 discusses identifying the positions of a respective plurality of imaging apparatuses in a common coordinate system by using a plurality of images obtained by capturing images of the same marker by the imaging apparatuses.

According to a method discussed in Japanese Patent Application Laid-Open No. 2018-207252, the identification results of the positions of the plurality of imaging apparatuses can be erroneous if the plurality of imaging apparatuses is located on both sides of an interface between a plurality of regions filled with substances having different refracting indexes. For example, in a case where a virtual viewpoint image is generated with an aquatic sport such as artistic swimming as an imaging target, imaging apparatuses can be installed above and below the water surface. Since the air and water have different optical refractive indexes, the imaging apparatuses are unable to stably detect a marker located on the other side of the water surface due to occurrence of light reflection at the water surface and fluctuations of the water surface. If images captured when the marker is installed only in the water are used, the identification results of the positions of the imaging apparatuses arranged above the water surface can be erroneous. Similarly, if images captured when the marker is installed only in the air are used, the identification results of the positions of the imaging apparatuses arranged below the water surface can be erroneous.

Japanese Patent Application Laid-Open No. 2008-191072 discusses a technique for correctly identifying a three-dimensional shape of a target object even if there are imaging apparatuses that only cover part of the object in their imaging range. More specifically, Japanese Patent Application Laid-Open No. 2008-191072 discusses identifying three-dimensional shapes of individual parts of an object by using captured images obtained by imaging apparatuses that cover the respective parts of the object with their imaging range. Japanese Patent Application Laid-Open No. 2008-191072 also discusses generating shape data expressing the three-dimensional shape of the entire object by combining the identified three-dimensional shapes of the respective parts.

According to the conventional techniques, shape data expressing the three-dimensional shape of an object can be erroneous if the object exists on both sides of a refracting surface where light is refracted. For example, in a case where a virtual viewpoint image is generated when an aquatic sport such as artistic swimming is the imaging target, the body of a swimmer as a target to generate shape data can exit in part above the water surface (above water) and in part below the water surface (in the water). Since the air and water have different optical refractive indexes, an object on the other side of the water surface as seen from an imaging apparatus can appear distorted in the captured image due to occurrence of light reflection at the water surface and fluctuations of the water surface. If shape data on the in-water part of the competitor's (swimmer's) body is generated by using a captured image obtained by an imaging apparatus in the air covering the in-water part with its imaging range, shape data correctly expressing the three-dimensional shape of the competitor's body can fail to be generated.

SUMMARY

According to an aspect of the present disclosure, an information processing apparatus includes a first detection unit configured to detect a first object located on a first direction side of a refracting surface from an image obtained by an imaging apparatus located on the first direction side of the refracting surface among a plurality of imaging apparatuses configured to capture images of an imaging region including at least part of the refracting surface, light being refracted at the refracting surface in a three-dimensional space, a second detection unit configured to detect a second object located on a second direction side of the refracting surface from an image obtained by an imaging apparatus located on the second direction side of the refracting surface among the plurality of imaging apparatuses, the second direction side being an opposite side of the first direction side with respect to the refracting surface, an obtaining unit configured to obtain position information indicating at least either one of a positional relationship between the first and second objects and positions of the first and second objects, and an identification unit configured to identify positions of the plurality of imaging apparatuses in a common coordinate system, based on a result of detection performed by the first detection unit, a result of detection performed by the second detection unit, and the position information obtained by the obtaining unit.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. Configurations described in the following exemplary embodiments are merely examples, and the present disclosure is not limited to the illustrated configurations.

[Configuration of Information Processing System]

A configuration example of an information processing system10will be described with reference toFIGS. 1 and 2. The information processing system10is a system for generating a virtual viewpoint image expressing an appearance from a specified virtual viewpoint based on a plurality of images (multi-viewpoint images) captured by a plurality of imaging apparatuses and the specified virtual viewpoint. The virtual viewpoint image according to the present exemplary embodiment is also referred to as a free-viewpoint video image. However, the virtual viewpoint image is not limited to an image corresponding to a point of view freely (arbitrarily) specified by a user, and includes, for example, an image corresponding to a point of view that the user selects from a plurality of candidates. In the present exemplary embodiment, a case where the virtual viewpoint is specified by a user operation will be mainly described. However, the virtual viewpoint may be automatically specified based on a result of image analysis. In the present exemplary embodiment, a case where the virtual viewpoint image is a moving image will be mainly described. However, the virtual viewpoint image may be a still image.

The information processing system10includes a plurality of cameras serving as imaging apparatuses that capture images of an imaging region in a plurality of directions. In the present exemplary embodiment, the imaging region to an imaging target is near a swimming pool where aquatic sports such as an artistic swimming, a swimming race, and a water polo are played. The plurality of cameras is installed at different positions to surround such an imaging region, and synchronously captures images. The plurality of cameras included in the information processing system10is classified into a plurality of imaging apparatus groups corresponding to different installation locations. In the example illustrated inFIG. 1, an in-air camera system100ais an imaging apparatus group including a plurality of cameras110ainstalled above the water surface, specifically, in the air around the swimming pool. The in-air camera system100amainly captures images in the air above the water. An in-water camera system100wis an imaging apparatus group including a plurality of cameras110winstalled below the water surface, specifically, at the corners of the swimming pool in the water. The in-water camera system100wmainly captures images in the water. Both the in-air camera system100aand the in-water camera system100win the water capture images of the imaging region, which includes at least part of the water surface of the swimming pool. The water surface is an interface between an in-air region and an in-water region.

With such a camera installation configuration, images of an entire object900such as a swimmer can be captured even if the object900lies across above and in the water in the swimming pool as illustrated inFIG. 1. Thus, images of the above-water part of the entire object900lying in the air (e.g., the upper half of the swimmer's body) are captured at least by the cameras110ain the air. Images of the in-water part of the entire object900lying in the water (e.g., the lower half of the swimmer's body) are captured at least by the cameras110win the water.

The plurality of cameras does not necessarily need to be installed all around the imaging region. Depending on restrictions on installation positions, the plurality of cameras may be installed in only some of the directions of the imaging region.FIG. 1illustrates cameras110-1a,110-2a,110-1w, and110-2w.FIG. 2further illustrates cameras110-3aand110-3w. However, the number of cameras is not limited thereto. For example, the in-air camera system100aand the in-water camera system100wmay each include about 30 cameras. The imaging object is not limited to the foregoing, either. The number of cameras may be changed based on the imaging object. Cameras with different functions, such as telephoto cameras and wide-angle cameras, may be installed.

As employed herein, components each denoted by a reference numeral with the alphabetical letter “a” at the end are located in the air. Components each denoted by a reference numeral with the alphabetical letter w at the end are located in the water. The cameras included in the in-air camera system100awill be referred to as cameras110aif no distinction is intended. The cameras included in the in-water camera system100wwill be referred to as cameras110wif no distinction is intended. The cameras, whether in the air or in the water, will be referred to simply as cameras110if no distinction is intended. Similar notation is used for the reference numerals of other components in the in-air and in-water camera systems100aand100w.

As illustrated inFIGS. 1 and 2, the plurality of cameras110aincluded in the in-air camera system100ais daisy-chained via respective corresponding camera control units210a. The plurality of cameras110wincluded in the in-water camera system100wis similarly daisy-chained via respective corresponding camera control units210w. The in-air and in-water camera systems100aand100wdo not necessarily need to be separately connected. The in-air and in-water camera systems100aand100wmay be connected in series. The cameras110do not necessarily need to be daisy-chained, either. The plurality of camera control units210may be connected in network topology such as star topology.

The plurality of cameras110aincluded in the in-air camera system100ais installed at positions and in directions appropriate to capture images above the water in the swimming pool. For example, the focal lengths and focuses of the cameras110aare set so that images of a position of interest such as near the water surface can be captured with predetermined image quality. Similarly, the cameras110wincluded in the in-water camera system100ware installed at positions and in directions appropriate to capture images in the water in the swimming pool. The focal lengths and focuses of the cameras110ware set so that images of a position of interest such as near the water surface can be captured with predetermined image quality. In the present exemplary embodiment, the cameras110wincluded in the in-water camera system100whave a waterproof function and are directly installed in the water. However, this is not restrictive, and the cameras110wmay be accommodated in waterproof housings installed in the water. The swimming pool may be made of transparent acrylic glass, and the cameras10wmay be installed outside the swimming pool so that images in the water can be captured from below the water surface.

As illustrated inFIG. 2, the in-air camera system100aincludes the plurality of cameras110aand the plurality of camera control units210arespectively connected to the cameras110a. Each camera control unit210aincludes a synchronization client211aand an image transmission unit212a. The plurality of camera control units210aincluded in the in-air camera system100ais daisy-chained, and the endmost camera control unit210-1ais connected to a time server300and a data storage unit400. The in-water camera system100whas a similar configuration.

The synchronization client211in each camera control unit210communicates with the time server300and the synchronization clients211of other camera control units210to perform synchronization processing. The Precision Time Protocol (PTP) is used as a synchronization protocol. However, the synchronization protocol is not limited thereto. The plurality of cameras110are synchronized by the respective synchronization clients211outputting a GenLock signal and a timecode to the cameras110based on a result of the synchronization processing. The plurality of cameras110then synchronously captures images. The cameras110output obtained captured images and a timecode indicating the imaging time to the image transmission units212of the respective camera control units210. The image transmission units212transmit the captured images obtained from the cameras110to the data storage unit400.

The data storage unit400stores various types of information used to generate a virtual viewpoint image as well as the captured images obtained by the camera110. The information stored in the data storage unit400includes information obtained by calibration performed by a calibration unit500. The calibration unit500includes a calibration calculation unit510(hereinafter, referred to as a calculation unit510) and a calibration condition input unit520(hereinafter, referred to as an input unit520), and performs calibration based on captured images obtained from the data storage unit400. The calibration performed by the calibration unit500is information obtaining processing for obtaining camera parameters of each of the plurality of cameras110included in the information processing system10. The camera parameters to be obtained by the calibration includes at least parameters indicating the respective positions of the cameras110. However, this is not restrictive. The camera parameter to be obtained by the calibration may include parameters indicating the orientations of the cameras110, parameters indicating the focal lengths of the cameras110, and parameters indicating the states of lens distortion of the cameras110. Details of the calibration processing will be described below.

An image generation unit600obtains multi-viewpoint images and the information (camera parameters) obtained by the calibration from the data storage unit400, and generates a virtual viewpoint image based on the multi-viewpoint images, the information, and viewpoint information obtained from a viewpoint setting unit700. The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction of a virtual viewpoint. More specifically, the viewpoint information is a parameter set including parameters expressing a three-dimensional position of the virtual viewpoint, and parameters expressing directions of the virtual viewpoint in pan, tilt, and roll directions. The contents of the viewpoint information are not limited thereto. For example, the parameter set serving as the viewpoint information may include a parameter expressing the size (angle of view) of the field of view of the virtual viewpoint. The viewpoint information may include a plurality of parameter sets. For example, the viewpoint information may include a plurality of parameter sets corresponding to a respective plurality of frames constituting a virtual viewpoint moving image, and indicate the positions and directions of the virtual viewpoint at a plurality of consecutive points in time, respectively. The viewpoint setting unit700generates viewpoint information based on a user operation, and outputs the viewpoint information to the image generation unit600.

For example, a virtual viewpoint image is generated by the following method. First, foreground images and background images are obtained from a plurality of images (multi-viewpoint images) obtained by the plurality of cameras110capturing images in respective different directions. The foreground images are formed by extracting foreground regions corresponding to an object such as a person or a ball. The background images are formed by extracting background regions other than the foreground regions. A foreground model expressing a three-dimensional shape of the person or the like and texture data for coloring the foreground model are generated based on the foreground images. Texture data for coloring a background model expressing a three-dimensional shape of the background such as a swimming pool is generated based on the background images. The foreground model is generated by using information about the positions and orientations of the respective cameras110. Specifically, the camera parameters obtained by the calibration performed by the calibration unit500is used. The texture data is mapped onto the foreground model and the background model, and rendering is performed based on the virtual viewpoint indicated by the viewpoint information, whereby a virtual viewpoint image is generated. However, the method for generating a virtual viewpoint image is not limited thereto, and various methods may be used. Examples thereof include a method for generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model. The image generation unit600outputs the generated virtual viewpoint image to a display device and/or a storage device.

Next, a hardware configuration of the calibration unit500that is one of the information processing apparatuses included in the information processing system10will be described with reference toFIG. 3. Other apparatuses included in the information processing system10, such as the camera control units210and the image generation unit600, may have a similar hardware configuration to that of the calibration unit500. The calibration unit500includes a central processing unit (CPU)501, a read-only memory (ROM)502, a random access memory (RAM)503, an auxiliary storage device504, a display unit505, an operation unit506, a communication interface (I/F)507, and a bus508.

The CPU501implements functions of the calibration unit500by controlling the entire calibration unit500with use of computer programs and data stored in the ROM502and the RAM503. The calibration unit500may include one or a plurality of pieces of dedicated hardware different from the CPU501, and the dedicated hardware may execute at least part of the processing by the CPU501. Examples of the dedicated hardware include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a digital signal processor (DPS). The ROM502stores programs that do not need to be changed. The RAM503temporarily stores programs and data supplied from the auxiliary storage device504, and data supplied from outside via the communication I/F507. The auxiliary storage device504includes a hard disk drive, for example. The auxiliary storage device504stores various types of data such as captured images and the information obtained by the calibration.

The display unit505includes a liquid crystal display and a light-emitting diode (LED), for example. The display unit505displays a graphical user interface (GUI) for a user to operate the calibration unit500. Examples of the operation unit506include a keyboard, a mouse, a joystick, and a touch panel. The operation unit506receives the user's operations and inputs various instructions to the CPU501. The CPU501operates as a display control unit that controls the display unit505, and an operation control unit that controls the operation unit506. The communication I/F507is used to communicate with an external apparatus. For example, if the calibration unit500is connected to an external apparatus in a wired manner, a communication cable is connected to the communication I/F507. If the calibration unit500has a function of wirelessly communicating with an external apparatus, the communication I/F507includes an antenna. The bus508connects the components in the calibration unit500and transmits information.

In the present exemplary embodiment, the display unit505and the operation unit506are included in the calibration unit500. However, at least either one of the display unit505and the operation unit506may exist as a separate apparatus. The display unit505and the operation unit506may be omitted.

The calibration unit500performs calibration by using calibration images captured before the start of a scene, such as a competition, that is a virtual viewpoint image generation target. In the following description, the capturing of images to obtain images for performing calibration (calibration images) will be referred to as calibration imaging. The capturing of images of a scene to be a virtual viewpoint image generation, which is performed after calibration imaging, will be referred to as scene imaging. If a virtual viewpoint image is not generated in real time and generated based on recorded captured images, calibration imaging may be performed after the scene imaging.

Calibration imaging is performed in a state where a reference object is placed in the imaging region. In the present exemplary embodiment, the reference object is a marker810that can be detected from captured images obtained by the cameras110. A board800displaying the marker810is installed at a position capable of being imaged by the plurality of cameras110. The marker810may be printed on the board800. Alternatively, the board800may include a display for displaying the marker810. The shape of the board800and the reference object are not limited to such an example.

FIG. 5illustrates an example of the imaging region at a time of calibration imaging. The board800is an object that includes two markers810aand810wand exists across both regions above the water surface (in the air) and below the water surface (in the water). The marker810ais located on a part of the board800in the air. The marker810wis located on a part of the board800in the water. The cameras110-1aand110-2ain the air can capture images of the marker810a. The cameras110-1wand110-2win the water can capture images of the marker810w. The markers810are two-dimensional markers having respective different contents. For example, identification information about the markers810can be read from the captured images of the markers810. However, the contents of the markers810are not limited thereto. In the following description ofFIG. 4, the board800is assumed to be movable in the swimming pool. However, the board800may be fixed to a predetermined position on the bottom of the swimming pool.

Details of processing related to the calibration by the calibration unit500will be described with reference to the flowchart ofFIG. 4. The processing illustrated inFIG. 4is implemented by the CPU501of the calibration unit500loading a program stored in the ROM502into the RAM503and executing the program. At least part of the processing illustrated inFIG. 4may be implemented by one or a plurality of pieces of dedicated hardware different from the CPU501. The processing illustrated inFIG. 4is started at a timing when a plurality of captured images obtained by calibration imaging is stored into the data storage unit400and the calibration unit500becomes ready to communicate with the data storage unit400. However, the start timing of the processing illustrated inFIG. 4is not limited thereto.

In the description ofFIG. 4, calibration imaging is performed for a predetermined period while changing the positions of the markers810, and a plurality of frames of images is stored in the data storage unit400. Performing calibration using a plurality of frames of images in this way can improve the accuracy of calibration. However, this is not restrictive. The positions of the markers810may be fixed during the calibration period. Images captured at a time may be stored into the data storage unit400, and calibration may be performed by using the images.

In step S101, the calculation unit510selects which time (hereinafter, referred to as time T) to obtain the captured images of from the data storage unit400. In step S102, the calculation unit510obtains a plurality of images captured by the plurality of cameras110at the selected time T from the data storage unit400. The images obtained in step S102include the images of the marker810ain the air, captured by the cameras110aincluded in the in-air camera system100a. The images obtained in step S102also include the images of the marker810win the water, captured by the cameras110wincluded in the in-water camera system100w.

In step S103, the calculation unit510performs detection processing on each obtained image. If a marker810is detected from the image by the detection processing (YES in step S103), the processing proceeds to step S104. If no marker810is detected (NO in step S103), the processing proceeds to step S105. In step S103, the calculation unit510performs detection processing for detecting the marker810aabove water on each of the images captured by the cameras110ain the air. The calculation unit510performs detection processing for detecting the marker810win the water on each of the images captured by the cameras110win the water. The calculation unit510may detect a marker810from an image and then determine whether the marker810is the marker810aabove water or the marker810win the water.

In step S104, the calculation unit510records a set (T, N) of the time T and a marker number N read from the detected marker810, and coordinates (x, y) indicating the position of the marker810in the image, in association with identification information about the camera by which the image is captured. In step S105, the calculation unit510determines whether the detection processing of the markers810has been performed on all the frames of calibration images stored in the data storage unit400. If there is a frame of images on which the detection processing has not yet been performed (NO in step S105), the processing returns to step S101. In step S101, the calculation unit510selects the time T of the unprocessed frame. If all the frames of images have been processed (YES in step S105), the processing proceeds to step S106.

In step S106, the input unit520obtains position information about the markers810and calculation initial values of the positions and orientations of the cameras110. The position information about the markers810indicates a positional relationship between the marker810ain the air and the marker810win the water. For example, the input unit520obtains position information about three positions of the marker810ain the air and three positions of the marker810win the water. The position information to be obtained by the input unit520is not limited thereto. For example, if the positions of the markers810are fixed, the input unit520may obtain information indicating the positions of the respective markers810aand810wby constant coordinates as the position information. The input unit520obtains such information based on inputs corresponding to user operations. However, this is not restrictive, and the input unit520may obtain a measurement result of the positions of the markers810from an external apparatus.

In step S107, the calculation unit510performs calibration processing based on the information recorded in step S104and the information obtained in step S106with the time T and the marker number N, i.e., set (T, N), as an index identifier (ID). The index to be the reference of calibration, if an identical marker810are captured at different times, is handled as different indexes. Thus, the set (T, N) is handled as an index ID. In step S108, the calculation unit510outputs camera parameters obtained by the calibration to the data storage unit400for storage. The camera parameters stored in the data storage unit400are used to generate a virtual viewpoint image by the image generation unit600.

In the calibration processing according to the present exemplary embodiment, the detection results of the marker810ain the air are mainly used to calibrate the cameras110ain the air. The detection results of the marker810win the water are mainly used to calibrate the cameras110win the water. By such independent calibration, the camera parameters of the cameras110ain the air expressed by using coordinates in a coordinate system and the camera parameters of the cameras110win the water expressed by using coordinates in another coordinate system are obtained. If a virtual viewpoint image including an image in the water and an image above water, such as a virtual viewpoint image near the water surface, is generated based on the camera parameters obtained in such a manner, an image disturbance can occur due to a difference between the coordinate systems.

Thus, the calculation unit510obtains the camera parameters of the cameras110ain the air and the cameras110win the water expressed by using coordinates in a common coordinate system by performing processing based on the position information about the markers810obtained by the input unit520. For example, the calculation unit510obtains camera parameters indicating the positions of the cameras110ain the air in a first coordinate system based on the detection results of the marker810ain the air from the images captured by the cameras110ain the air. The calculation unit510obtains camera parameters indicating the positions of the cameras110win the water in a second coordinate system based on the detection results of the marker810win the water from the images captured by the cameras110win the water.

The calculation unit510converts the camera parameters of the cameras110win the water in the second coordinate system into values in the first coordinate system based on the identification results of the markers810aand810widentified from the position information obtained by the input unit520. In this way, the calibration unit500obtains the camera parameters of the plurality of cameras110in a common coordinate system based on the images captured by the cameras110ain the air and the images captured by the cameras110win the water. An image disturbance can be reduced by generating a virtual viewpoint image using the camera parameters obtained in this way by calibration.

The method for obtaining the camera parameters expressed by coordinate values in a single coordinate system is not limited thereto. For example, the calculation unit510may convert the camera parameters of the cameras110ain the air in the first coordinate system into values in the second coordinate system. For example, the calculation unit510may convert both the camera parameters of the cameras110ain the air in the first coordinate system and the camera parameters of the cameras110win the water in the second coordinate system into values in a third coordinate system. The control unit510may correct a difference in scale between the coordinate systems by using the refractive index of water in converting the camera parameters. However, the use of the refractive index for coordinate calculations is not indispensable. For example, using a plurality of markers as illustrated inFIG. 5, corrections may be made based on differences between known position information about the markers in the air and in the water and position information calculated from the detection results of the respective markers.

The camera parameters in the first and second coordinate systems can be obtained by using a conventional method for performing calibration based on the detection positions of markers810in a plurality of captured images obtained by a plurality of cameras110. In other words, the camera parameters of the cameras110ain the air in the first coordinate system can be obtained based on the detection results of the markers810afrom the plurality of images obtained by the in-air camera system100a. The camera parameters of the cameras110win the water in the second coordinate system can be obtained based on the detection results of the markers810wfrom the plurality of images obtained by the in-water camera system100w.

In the foregoing description with reference toFIG. 4, the calibration unit500is described to perform the calibration processing after the calibration imaging is ended and the captured images are stored in the data storage unit400. However, it is not limited thereto. If the calibration imaging is performed for a certain period, the calibration unit500may perform the calibration processing in parallel with the image capturing by the cameras110.

[Installation Example of Markers]

An installation example of the markers810in performing the calibration imaging will now be described. In the installation example illustrated inFIG. 5described above, the marker810ain the air and the marker810win the water are displayed on the same object, i.e., the board800. In other words, the water surface in the swimming pool is located between the two markers810on the board800. The positional relationship between the position where the marker810ais displayed and the position where the marker810wis displayed on the board800is determined in advance. The camera parameters with reference to the marker810aand the camera parameters with reference to the marker810wcan therefore be appropriately integrated by performing calibration using the position information about the markers810aand810wobtained by the input unit520.

In performing calibration by using the markers810illustrated inFIG. 5, the detection results of the marker810win the water from the captured images of the cameras110ain the air and/or the detection results of the marker810ain the air from the captured images of the cameras110win the water may be used. For example, the marker810ain the air expresses information from which the presence of the marker810ain the air can be identified. The marker810win the water expresses information from which the presence of the marker810win the water can be identified. More specifically, the marker810ain the air and the marker810win the water express information from which respective different IDs can be read. The calculation unit510determines whether a marker810detected from a captured image is the marker810ain the air or the marker810win the water.

If the calculation unit510determines that a marker810detected from the captured image of a camera110ain the air is the marker810win the water, the calculation unit510corrects a positional deviation of the marker810wdue to optical refraction at the water surface by correcting the detection result with a known relative refractive index. If the calculation unit510determines that a marker810detected from the captured image of a camera110win the water is the marker810ain the air, the calculation unit510corrects the detection result with the known relative refractive index. The calculation unit510then performs calibration by using the corrected detection result in addition to the detection results of the marker810ain the air from the captured images of the camera110ain the air and the detection results of the marker810win the water from the captured images of the cameras10win the water. Such a method can improve the accuracy of calibration since the calibration can be performed by using more detection results of the markers810.

The method for installing the markers810for calibration is not limited to the example ofFIG. 5.FIG. 6illustrates another installation example of the markers810. A board800-1adisplaying a marker810-1aand a board800-2adisplaying a marker810-2aare installed in the air on the poolside. Aboard800-1wdisplaying a marker810-1wand a board800-2wdisplaying a marker810-2ware installed in the water in the swimming pool. The markers810-1a,810-2a,810-1w, and810-2ware installed at positions capable of being imaged by the cameras110-1a,110-2a,110-1w, and110-2w, respectively. The plurality of markers810is fixed during calibration imaging, and the positional relationship between the markers810remains unchanged. The input unit520obtains position information indicating the positions of the respective markers810in a global coordinate system. The calculation unit510performs calibration by using identification results of the positions of the markers810identified from the position information and the images captured by the respective cameras110.

If a plurality of markers810is installed in such a manner, the markers810ain the air may be installed outside the imaging ranges of the cameras110win the water, and the markers810win the water may be installed outside the imaging ranges of the cameras110ain the air. Such installation can prevent the accuracy of calibration from dropping that is caused when the detection results of the markers810win the water detected from the captured images of the cameras110ain the air are erroneously used for calibration. Similarly, the accuracy of calibration can be prevented from dropping that is caused when the detection results of the markers810ain the air detected from the captured images of the cameras110win the water are erroneously used for calibration. In such a case, the markers810aand the markers810wmay have the same contents.

Alternatively, as described above with reference toFIG. 5, the markers810may express information from which the IDs of the respective markers810can be read. In such a case, both the markers810ain the air and the markers810win the water may be included in the captured image of the same camera110since the calculation unit510can determine whether each of the markers810detected from the captured image is a marker810ain the air or a marker810win the water. This reduces restrictions on the installation of the markers810. In this case, the detection results of the markers810win the water from the captured images of the cameras110ain the air and the detection results of the markers810ain the air from the captured images of the cameras110win the water may be left unused for calibration.

FIGS. 7A, 7B, and 7Cillustrate another installation example of the markers810. In the example ofFIGS. 7A, 7B, and 7C, a board800is located across the water surface that is the interface between in the air and in the water. The board800has a marker810aon its top surface (surface in the air) and a marker810won its bottom surface (surface in the water).FIG. 7Ais a lateral view of the swimming pool where the board800is floated on the water surface.FIG. 7Bis a diagram illustrating the board800seen in the direction of the arrow A inFIG. 7A.FIG. 7Cis a diagram illustrating the board800seen in the direction of the arrow B inFIG. 7A. Since the board800is not fixed, calibration imaging can be performed while moving the board800. The input unit520obtains position information indicating the positional relationship between the markers810aand810w. The calculation unit510performs calibration by using the positional relationship between the markers810identified from the position information and the images captured by the respective cameras110.

If the water surface ripples during calibration imaging, the accuracy of calibration can drop due to changes in the tilt of the markers810. In view of this, at least either one of the markers810aand810wmay display information from which the tilt of the markers810can be identified. For example, the board800may include a gyro sensor or a level, and a marker810of different contents may be displayed on a display included in the board800based on the tilt detected by the gyro sensor or the level. The calculation unit510may determine the tilt of the markers810by reading the contents of the marker810detected from captured images, and calculate the camera parameters by using the detection results corrected based on the tilt of the markers810. This can improve the accuracy of calibration in a case where the tilt of the markers810changes during calibration imaging. The calculation unit510may determine the normal directions to the surfaces of the markers810, and calculate the camera parameters based on the determination results. As another method, markers including perfect circles may be used to estimate the directions of the markers from the shapes of ellipses included in the captured images.

As described above, the calibration unit500according to the present exemplary embodiment detects the marker(s)810ain the air from the images obtained by the cameras110alocated above the water surface among the plurality of cameras110for capturing images of the imaging region including at least part of the water surface. The calibration unit500detects the marker(s)810win the water from the images obtained by the cameras110wlocated below the water surface. The calibration unit500identifies the positions of the plurality of cameras110in a common coordinate system based on the detection results of the markers810and the position information indicating the positions of or the positional relationship between the marker(s)810ain the air and the marker(s)810win the water.

With such a configuration, the positions of the plurality of cameras110in the common coordinate system can be identified even if the plurality of cameras110is located on both sides of the water surface that is the interface between the regions in the air and in the water with different refractive indexes. A drop in the image quality of a virtual viewpoint image can be reduced by generating the virtual viewpoint image based on the camera parameters indicating the positions of the plurality of cameras110identified in such a manner.

In the foregoing description, the imaging region is described to include a region filled with the air (region above the water) and a region filled with water that is a substance having a different refractive index from that of the air (region in the water). However, the application of the calibration method according to the present exemplary embodiment is not limited thereto. For example, the calibration method according to the present exemplary embodiment can also be applied to improve the accuracy of calibration in a case where the imaging region includes a region filled with the air and a region filled with glass, resin, or oil. In another example, the imaging region may include a region filled with water and a region filled with oil. If the substances filling the imaging region are fluids, especially the air and a liquid in particular, the effect of improving the accuracy of calibration according to the present exemplary embodiment is high since the interface is likely to fluctuate. As employed in the present exemplary embodiment, a substance filling a region refers to one that mainly constitutes the three-dimensional region (e.g., a substance occupying one half or more of the volume of the three-dimensional region).

According to the exemplary embodiment described above, the positions of the plurality of imaging apparatuses in a common coordinate system can be identified even if the plurality of imaging apparatuses is located on both sides of the interference of a plurality of regions filled with substances having different refractive indexes from each other.

[Another Configuration Example of Information Processing System]

FIG. 8illustrates another configuration example of the information processing system10, obtained by modifying the configuration example illustrated inFIG. 2. The difference from the configuration illustrated inFIG. 2is that the information processing system10includes a shape data generation unit1000. In the present configuration, the information stored in the data storage unit400includes object's shape data generated by the shape data generation unit1000. The shape data generation unit1000includes an image separation unit1010, a model generation unit1020, and a model adjustment unit1030, and generates shape data based on images obtained from the data storage unit400. The shape data according to the present exemplary embodiment is data expressing by a group of points the three-dimensional shape of an object located in the imaging region of which images are captured by the plurality of cameras110. The contents of the shape data are not limited thereto. For example, the shape data may be data expressing the three-dimensional shape by a polygon mesh. Details of processing for generating the shape data will be described below.

The configuration of the information processing system10is not limited to that illustrated inFIG. 8. As an example, the shape data generation unit1000and the image generation unit600may be configured as one unit. In such a case, the generated shape data may be simply used to generate a virtual viewpoint image without being stored in the data storage unit400. As another example, the camera control units210may include the image separation unit1010and output foreground images and background images to the data storage unit400. In such a case, the shape data generation unit1000may obtain a plurality of foreground images from the data storage unit400as a plurality of images based on imaging by the plurality of cameras110.

The shape data generation unit1000generates shape data on each frame of a moving image obtained by the cameras110capturing images of a competition to generate a virtual viewpoint image. In other words, shape data expressing the three-dimensional shape of an object at time T is generated based on a plurality of images captured by the plurality of cameras110at time T. The shape data may be generated in real time in parallel with the imaging by the cameras110, or may be generated based on images stored in the data storage unit400after the end of the competition to capture images. In the following description, processing for generating shape data expressing the three-dimensional shape of the object900lying both in the air and in the water as illustrated inFIG. 1based on the plurality of images captured by the plurality of cameras110, will be described.

Details of the processing related to the generation of the shape data by the shape data generation unit1000will be described with reference to the flowchart illustrated inFIG. 9. The processing illustrated inFIG. 9is implemented by a CPU501of the shape data generation unit1000loading a program stored in a ROM502into a RAM503and executing the program. At least part of the processing illustrated inFIG. 9may be implemented by one or a plurality of pieces of dedicated hardware different from the CPU501. The processing illustrated inFIG. 9is started at a timing when a plurality of captured images obtained by the plurality of cameras110is stored into the data storage unit400and an instruction to generation shape data is input to the shape data generation unit1000. However, the start timing of the processing illustrated inFIG. 9is not limited thereto. The processing illustrated inFIG. 9is repeated for each frame of the captured images.

In step S201, the model generation unit1020obtains information, from the data storage unit400, enabling identification of the position of the water surface that is the interface between the region in the air and the region in the water in the imaging region. The information indicates the height of the water surface in the swimming pool, and is stored in the data storage unit400in advance along with the positions of the cameras110obtained by calibration. The positions of the cameras110and the water surface in the three-dimensional space are expressed by using coordinates in the same coordinate system. However, the contents of the information enabling the identification of the position of the water surface are not limited thereto. For example, the information may indicate the region in the water and the region in the air by using the coordinates of the vertexes or sides of the regions. The model generation unit102may obtain the information enabling the identification of the position of the water surface based on a user operation.

Then, the model generation unit1020sets a global coordinate system expressing a virtual space corresponding to the imaging region to be imaged by the plurality of cameras110based on the information enabling the identification of the position of the water surface. In the present exemplary embodiment, an upward direction perpendicular to the water surface in the swimming pool is referred to as a z-axis positive direction, and the global coordinate system is set so that the water surface corresponds to a plane of z=0. In a space expressed by the global coordinate system set in this manner, a position where the z coordinate is positive corresponds to a position in the air above the water surface. A position where the z coordinate is negative corresponds to a position in the water. If the position of the water surface changes, the water surface position information stored in the data storage unit400may be updated. The setting of the global coordinate system in step S201may be performed at least once during generation of a series of pieces of shape data corresponding to an imaging period, not for each frame of the captured images.

In step S202, the image separation unit1010obtains, from the data storage unit400, a plurality of captured images obtained by the plurality of cameras110capturing images in different directions at time T corresponding to the frame to be processed. In step S203, the image separation unit1010performs foreground/background separation on the obtained captured images to obtain foreground images and background images. Examples of the technique for foreground/background separation include, but not limited to, a technique of extracting the foreground by using a parallax obtained by comparing the images of adjoining cameras110.

A foreground image refers to an image obtained by extracting the region of an object (foreground region) from a captured image. The object to be extracted as a foreground region refers to a moving object (moving body) that is moving (can change in absolute position or shape) when a time series of images is captured in the same direction. In this case, suppose that the object900that is a person performing in a competition near the water surface in the swimming pool is extracted as a foreground region. However, the object is not limited thereto. Examples of the object may include a person in the field of a game, such as a player and a judge, a ball in a ball game, and a singer, player, performer, and master of ceremonies in a concert or entertainment show.

A background image refers to an image of a region (background region) at least different from the object900serving as the foreground. More specifically, a background image is a captured image from which the image of the object900serving as the foreground is removed. A background refers to an imaging object that is stationary or maintains in a state close to being stationary when a time series of images is captured from the same direction. In the present exemplary embodiment, the background includes structures such as the swimming pool where the competition is held and the poolside. However, the imaging object serving as a background is not limited thereto. Examples thereof may include a concert stage, a stadium where a game or an event is held, a structure such as goal posts used in a ball game, and a field. The background is a region at least different from the object900serving as the foreground. Objects other than the object900and the background may be included as imaging objects. For example, the water in the swimming pool may be handled as a background, or handled differently from the foreground and the background.

In step S204, the model generation unit1020identifies the three-dimensional shapes of the above-water part and in-water part of the object900based on the foreground images obtained in step S203. More specifically, the model generation unit1020generates shape data on the above-water part of the object900located in the air based on the foreground images extracted from the captured images obtained by the cameras110ain the air. The model generation unit1020also generates shape data on the in-water part of the object900located in the water based on the foreground images extracted from the captured images obtained by the cameras110win the water. The three-dimensional shape of the entire object900is a sum of three-dimensional shapes expressed by the two pieces of shape data.

The processing for generating shape data on each of the above-water part and in-water part of the object900will be described with reference toFIGS. 10A, 10B, 10C, 10D, and 10E.FIG. 10Aillustrates a state where there is a water surface901at z=0, and the object900is located near the water surface901(i.e., standing across the water surface901). Foreground images are extracted from captured images obtained by the cameras110ain the air capturing images of the object900in the state illustrated inFIG. 10A.FIG. 10Billustrates the shape of a three-dimensional (3D) model801aexpressed by shape data generated from the foreground images.

The part of the 3D model801awhere z<0 is generated based on the images of the in-water part included in the captured images obtained by the cameras110ain the air. Since the air and water have different refractive indexes, an object in the water appears deformed in the captured images obtained by the cameras110ain the air due to a change in the refractive index at the water surface901. As illustrated inFIG. 10B, the part of the 3D model801awhere z<0 has a shape compressed in the z direction, compared to the actual shape of the object900illustrated with a broken line. The model generation unit1020obtains a 3D model802aof the above-water part of the object900illustrated inFIG. 10Cby deleting the part of the 3D model801ain the water, i.e., the part where z<0. The 3D shape of the above-water part of the object900is thereby identified.

FIG. 10Dillustrates the shape of a 3D model801wexpressed by shape data that is generated from foreground images extracted from captured images obtained by the cameras110win the water capturing images of the object900in the state illustrated inFIG. 10A. The part of the 3D model801wwhere z<0 is generated based on the images of the in-water part included in the captured images obtained by the cameras110win the water. The part where z>0 is generated based on the images of the above-water part included in the captured images obtained by the cameras110win the water. Since the air and water have different refractive indexes, the object900in the captured images obtained by the cameras110win the water becomes greater than the object900in the captured images obtained by the cameras110ain the air. In other words, the captured images obtained by the cameras110win the water look as if captured at focal lengths on the telephoto side of the focal lengths actually set in the cameras110w.

Thus, as illustrated inFIG. 10D, the 3D model801wdiffers at least in size, compared to the actual shape of the object900illustrated with the broken line. Since an object in the air is deformed in the images captured by the cameras110win the water, the part of the 3D model801wwhere z>0 can be different from the actual object900in shape. In addition, since the captured images obtained by the cameras110win the water look as if captured at focal lengths different from those actually set in the cameras110w, the position of the 3D model801wcan be different from the actual position of the object900. The model generation unit1020obtains a 3D model802wof the in-water part of the object900illustrated inFIG. 10Eby deleting the part of the 3D model801W in the air, i.e., the part where z>0. In this way, the 3D shape of the in-water part of the object900is identified.

In step S205, the model adjustment unit1030adjusts and combines the 3D models generated in step S204. Details of the processing will be described with reference toFIGS. 11A and 11B.FIG. 11Aillustrates the 3D model812of the entire object900obtained by simply combining the 3D model802aof the above-water part of the object900generated in step S204with the 3D model802wof the in-water part of the object900at the water surface901. The part of the 3D model802wcontacting the water surface901is greater than the part of the 3D model802acontacting the water surface901due to the refractive indexes. Thus, the 3D model812is out of shape at the position of the water surface901and unable to correctly express the 3D shape of the object900.

Thus, the model adjustment unit1030performs modification processing for modifying at least either one of the 3D models802aand802wbased on the difference between the 3D models802aand802wat the water surface901, whereby the difference at the water surface901is corrected. More specifically, the model adjustment unit1030modifies the size of either one of the 3D models802aand802wto that of the other.FIG. 11Billustrates a 3D model811of the entire object900obtained by modifying the size of the 3D model802wto that of the 3D model802aand combining the 3D models802aand802wat the water surface901. Since the 3D model802wis modified in size, the difference between the 3D models802aand802wat the water surface901is corrected, and the 3D model811correctly expresses the 3D shape of the object900.

If there is a difference in position between the 3D models802aand802wat the water surface901, the model adjustment unit1030may modify the position of either one of the 3D models802aand802wto that of the other. The model adjustment unit1030may modify both the 3D models802aand802wso that a difference in position and size between the 3D models802aand802wis corrected.

The position(s) and/or size(s) of the 3D models802are modified based on cross-sectional shapes of the 3D models802on a plane of z=0 corresponding to the water surface901where the refractive index changes. More specifically, the model adjustment unit1030identifies the cross section of the 3D model802aand that of the 3D model802won the plane of z=0, and modifies at least either one of the 3D models802aand802wso that the positions and shapes of the cross sections approach each other. For example, the model adjustment unit1030modifies the position of the 3D model802wso that the gravity center positions of the two cross sections coincide, and modifies the size of the 3D model802wto minimize difference between the outlines of the two cross sections.

The method for adjusting the 3D models802is not limited thereto. For example, the model adjustment unit1030may obtain information indicating the refractive indexes of the substances filling the two respective regions with the plane of z=0 as the interface or a relative refractive index between the substances based on inputs made by the user or from an external apparatus. Then, the model adjustment unit1030may modify the 3D models802based on the obtained information. In the example of the present exemplary embodiment where the plane of z=0 corresponds to the water surface901, the model adjustment unit1030can correct a difference by making the size of the 3D model802w1/1.333 times since the refractive index of water is 1.333. If the refractive indexes are known, the accuracy of correction can be improved by using such a method.

In step S206, the model generation unit1020generates shape data on the object900expressing the 3D model811adjusted and combined by the model adjustment unit1030, and outputs the shape data to the data storage unit400. The shape data stored in the data storage unit400is used for the generation of a virtual viewpoint image by the image generation unit600. Since the shape data on which the result of the adjustment made by the model adjustment unit1030is reflected correctly expresses the 3D shape of the object900, the object900is correctly reproduced in the virtual viewpoint image generated based on the shape data.

If the multi-viewpoint images capturing the imaging region include a plurality of objects each lying across the water surface (i.e., object having above-water part and in-water part), the shape data generation unit1000generates 3D models802aand802wof the above-water part and in-water part of each object, and associates the 3D models802aand802wwith the object. Then, the shape data generation unit1000performs processing for reducing a difference for each object by adjusting the associated 3D models802aand802w, and generates shape data on each object. In this way, correct shape data on each object can be generated even if there is a plurality of objects.

If an object in the imaging region is entirely located in the air, the shape data generation unit1000does not need to perform the foregoing adjustment processing on the 3D model of the object generated from the foreground images. If an object is entirely located in the water, the shape data generation unit1000may adjust the size of the entire 3D model of the object generated from the foreground images based on the refractive indexes.

In the foregoing description with reference toFIG. 9, the shape data on the above-water part of the object900and the shape data on the in-water part are initially generated separately, and at least either one of the pieces of shape data is adjusted to correct a difference at the water surface901. However, the method for correcting a difference in the shape data of the object900is not limited thereto.

For example, the plurality of cameras110may be installed and set based on the refractive indexes of the regions where the cameras110are installed. If the cameras110are installed in the air and in the water, the focal lengths of the cameras110win the water may be set to be shorter than those of the cameras110ain the air so that images can be captured with the same angle of view as that of the cameras110ain the air. The positions and orientations of the cameras110may be adjusted based on the angles of view. Such precise adjustments of the cameras110enable correction of a difference in the shape data on the object900while reducing processing related to the adjustment of shape data. On the other hand, the method described above with reference toFIG. 9can simplify the installation and setting of a large number of cameras110. In addition, a difference can be corrected even if the refractive indexes of the substances filling the respective regions are unknown.

As a method for reducing a difference, for example, the shape data generation unit1000may convert the foreground images or captured images for generating the shape data or the camera parameters obtained by calibration based on the refractive indexes. If the cameras110are installed in the air and in the water, the sizes of the captured images or foreground images based on imaging by the cameras110win the water or the camera parameters of the cameras110win the water may be converted based on a change in the focal lengths due to the refractive index of water. The generation of shape data using the foreground images or camera parameters converted in advance enables correction of a difference in the shape data on the object900while reducing the processing related to the adjustment of the shape data. In addition, the shape data generation unit1000can also directly generate the shape data on the entire object900by performing different conversion processes respectively on the above-water regions and the in-water regions in the foreground images, without generating the pieces of shape data on the above-water part and the in-water part of the object900separately. On the other hand, the method described above with reference toFIG. 9can reduce the processing related to the conversion of the images and camera parameters. In addition, a difference can be corrected even if the refractive indexes of the regions are unknown.

In the present exemplary embodiment, the cameras110for capturing images of the object900are described to be installed both in the air and in the water that are a plurality of regions filled with substances having different refractive indexes. However, the cameras110are not limited to such a layout. For example, the plurality of cameras110may be installed only in the air. The shape data generation unit1000may generate shape data on both the above-water part and the in-water part of the object900from the images obtained by the cameras110ain the air, and adjust the shape data on the in-water part based on the refractive indexes. The shape data on the entire object900can be generated by combining the adjusted shape data on the in-water part with the shape data on the above-water part. Such a method facilitates the installation of the cameras110.

However, if this method is used, the shape data on the in-water part of the object900is generated based on the images obtained by the cameras110ain the air capturing images of the in-water part lying on the other side of the water surface. The water surface not only refracts light but also reflects light and causes a change in the incident angle of light due to rippling. The cameras110acan therefore fail to capture stable images of the in-water part of the object900. In such a case, the 3D shape of the in-water part of the object900is unable to be accurately identified even by taking the refractive indexes into account, and correct shape data on the entire object900may fail to be generated.

On the other hand, the shape data generation unit1000according to the present exemplary embodiment identifies the 3D shape of the above-water part of the object900from the images obtained by the cameras110ain the air, and identifies the 3D shape of the in-water part of the object900from the images obtained by the cameras110win the water. Then, the shape data generation unit1000generates the shape data on the entire object900based on the identified 3D shapes of the above-water and in-water parts. This prevents the effect of disturbance of the images on the other side of the water surface as seen from the cameras110from appearing on the shape data. As a result, correct shape data on the entire object900can be generated.

As described above, the shape data generation unit1000according to the present exemplary embodiment obtains a plurality of images obtained by the plurality of cameras110capturing images of the object900lying both in the air and in the water in different directions. The shape data generation unit1000generates shape data expressing the 3D shape of the object900based on the plurality of obtained images. In generating the shape data, the shape data generation unit1000performs processing for correcting a difference, at the water surface901, between the 3D shape of the above-water part of the object900located in the air and the 3D shape of the in-water part of the object900located in the water.

Through such a configuration, the shape data expressing the 3D shape of the object900can be generated even if the object900exist across both regions in the air and in the water with different refractive indexes. A virtual viewpoint image in which the shape of the object900is correctly reproduced can be generated by using the shape data generated in such a manner.

In the foregoing description, the refracting surface at which light is refracted in the imaging region is described to be the interface between the region filled with air (above-water region) and the region filled with water that is a substance having a different refractive index from that of the air (in-water region). However, the application of the method for generating shape data according to the present exemplary embodiment is not limited thereto. For example, the method for generating shape data according to the present exemplary embodiment can also be applied to improve the accuracy of the shape data even if the imaging region includes an interface between a region filled with the air and a region filled with a solid substance such as glass and resin or a liquid such as oil. As another example, the imaging region may include a region filled with water and a region filled with oil. If the imaging region includes layers of two types of fluids having different refractive indexes, or a gas layer and a liquid layer in particular, the effect of improving the accuracy of generation of shape data according to the present exemplary embodiment is high since the interface between the layers is likely to fluctuate easily. Even if the object900lies across three or more regions filled with different substances, the present exemplary embodiment can be applied to identify the 3D shape of the entire object900. In the present exemplary embodiment, a substance filling a region is not limited to one that fully occupies the region, and refers to one that mainly constitutes the region (e.g., a substance occupying one half or more of the volume of the 3D region).

According to the exemplary embodiment described above, shape data expressing the 3D shape of an object can be generated even if the object exists on both sides of a refracting surface where light is refracted.

Other Embodiments

While the present disclosure has been described with reference to exemplary embodiments, the scope of the following claims are to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Applications No. 2019-083242, filed Apr. 24, 2019, and No. 2019-108888, filed Jun. 11, 2019, which are hereby incorporated by reference herein in their entirety.