Image processing apparatus, image processing method, storage medium, manufacturing method of learned model, and image processing system

An image processing apparatus includes at least one processor or circuit configured to execute a plurality of tasks including an acquisition task configured to acquire two first images made by capturing the same object at two different viewpoints, and an image processing task configured to input the two first images into a machine learning model and to estimate a second image at one or more viewpoints different from the two viewpoints.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing method that estimates another viewpoint image from two-viewpoint images using machine learning or deep learning (DL).

Description of the Related Art

N. K. Kalantari, T-C. Wang, R. M. Ramamoorthi, “Learning-Based View Synthesis for Light Field Camera,” SIGGRAPH Asia (2016) (“Kalantari et al.”) discloses a method of estimating another viewpoint image from four-viewpoint images using DL. The four-viewpoint images are images made by capturing the same object at four different viewpoints, and the other viewpoint image is an image as if it is made by capturing the same object at one or more viewpoints different from the four viewpoints.

When the other viewpoint image is estimated from two-viewpoint images with a disparity (parallax) in a lateral direction using the method disclosed in Kalantari et al., the estimation accuracy of the rectangular-shaped object in the two-viewpoint images is lowered. This is because it is so difficult to find corresponding points of the rectangular-shaped object between the two-viewpoint images that a depth from the camera to the rectangular-shaped object cannot be determined.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus that can estimate another viewpoint image from two-viewpoint images with high accuracy.

An image processing apparatus according to one aspect of the present invention includes at least one processor or circuit configured to execute a plurality of tasks including an acquisition task configured to acquire two first images made by capturing the same object at two different viewpoints, and an image processing task configured to input the two first images into a machine learning model and to estimate a second image at one or more viewpoints different from the two viewpoints. An image processing method corresponding to the above image processing apparatus also constitutes another aspect of the present invention. A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method also constitutes another aspect of the present invention.

An image processing apparatus according to another aspect of the present invention includes at least one processor or circuit configured to execute a plurality of tasks including an image generating task configured to generate two first images made by capturing the same object at two different viewpoints and a second image at one or more viewpoints different from the two viewpoints, and a learning task configured to input the two first images into a machine learning model, to compare an output image and the second image with each other, and to learn the machine learning model. An image processing method and a method of manufacturing a learned model each corresponding to the above image processing apparatus also constitutes another aspect of the present invention. A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method also constitutes another aspect of the present invention.

An image processing system according to another aspect of the present invention includes a first device and a second device communicable with the first device, the image processing system. The first device includes a transmitter configured to transmit a request that requests the second device to process two first images made by capturing the same object at two different viewpoints. The second device includes a receiver configured to receive the request transmitted from the transmitter, and at least one processor or circuit configured to execute a plurality of tasks including an image processing task configured to input the two first images into a machine learning model and to estimate a second image at one or more viewpoints different from the two viewpoints.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a duplicate description thereof will be omitted.

A description will now be given of a gist of each embodiment before a specific description is given. In each embodiment, in estimating another viewpoint image from two-viewpoint images with a disparity in a lateral direction using DL, another viewpoint image is robustly estimated of a rectangular-shaped object in the two-viewpoint images.

A neural network is used for image processing by DL. The neural network uses a filter to be convolved with an image, a bias added to it, and an activation function that performs a nonlinear transformation. The filter and bias are called weights and generated by learning from training images. The present invention uses two-viewpoint images and another viewpoint image corresponding to them as the training image. Alternatively, a parametric activation function may be used to simultaneously generate the parameters by learning.

A multidimensional array obtained in an intermediate layer in the neural network as a result of repeating processing of convolving the filter with the image, of adding the bias to the image, and of performing the nonlinear transformation for the image is called a feature map. One point of the feature map obtained by convolving a plurality of filters with the image many times includes information on a corresponding wide area (receptive field) in the image.

Referring now toFIG.10, a description will be given of an outline of each embodiment. Each embodiment first generates two feature maps (first feature amounts) of the two-viewpoint images (first images) from the two viewpoint images using a first neural network. Next, each embodiment compares the two generated feature maps with each other, and extracts information (second feature amount) on corresponding points between the two-viewpoint images. Here, the information on the corresponding points between the two-viewpoint images is information used to find a shift amount according to a depth (distance) of the same object in the two-viewpoint image. Finally, another viewpoint image (second image) is estimated by a second neural network from the information (second feature amount) on the corresponding points between the two viewpoint images.

Each embodiment characteristically finds the corresponding points by comparing the two feature maps obtained from the two-viewpoint images. This makes it easier to find the corresponding points of the rectangular-shaped object than the prior art that compares the two-viewpoint images for each pixel. This is because even for the rectangular-shaped object, a clue of the corresponding points (edges of the rectangular-shaped object and the texture of the surface of the rectangular-shaped object) can be found when it is viewed in a wide area. As a result, each embodiment can robustly estimate the other viewpoint image of a rectangular-shaped object in the two-viewpoint images having a disparity in the lateral direction.

A description will now be given of finding the corresponding points between the two-viewpoint images using template matching. A larger template might be able to find the corresponding points of the rectangular-shaped object between the two-viewpoint images. However, this is not realistic because the larger the template is, the heavier the computational load becomes. Each embodiment using the feature map can expand the receptive field only by deepening the network, and search for the corresponding points in a wider area in the two-viewpoint images. As a result, it becomes easy to find the corresponding points of the rectangular-shaped object between the two-viewpoint images.

In general, the two-viewpoint images with a disparity in the lateral (or horizontal) direction have no disparity information in the longitudinal (or vertical) direction. It is therefore difficult to estimate, based on the two-viewpoint images, an image at a viewpoint that shifts in the longitudinal direction (a direction orthogonal to or different from the predetermined direction) rather than the lateral direction (predetermined direction) determined by the two viewpoints. However, each embodiment can estimate the distance from the camera to the object with high accuracy by comparing the two feature maps obtained from the two viewpoint images and by finding the corresponding points. Estimating the disparity information in the longitudinal direction from the result, each embodiment can estimate, based on the two-viewpoint images, an image at a viewpoint that shifts in the longitudinal direction rather than the lateral direction determined by the two viewpoints.

The above image processing method is merely illustrative, and each embodiment is not limited to this example. Details of other image processing methods and the like will be described in each of the following embodiment. Each embodiment is not limited to a case where the two viewpoints of the two-viewpoint images shift in the lateral direction, but is applicable to an image in which the viewpoints shift in an arbitrary direction (predetermined direction).

First Embodiment

A description will now be given of an image processing system according to a first embodiment of the present invention. This embodiment provides learning and execution of image processing for estimating another viewpoint image from two-viewpoint images using a neural network.

FIG.2is a block diagram of the image processing system100according to this embodiment.FIG.3is an external view of the image processing system100. The image processing system100includes a learning apparatus101, an image pickup apparatus102, an image estimating apparatus103, a display apparatus104, a recording medium105, an output apparatus106b, and a network107.

The learning apparatus (image processing apparatus)101includes a memory101a, an image generating unit101b, a feature amount generating unit101c, a feature amount comparing unit101d, an image reconstructing unit101e, and a learning unit101f.

The image pickup apparatus102includes an optical system (imaging optical system)102aand an image sensor102b. The optical system102acollects light incident on the image pickup apparatus102from an object space. A captured image (two-viewpoint image) is acquired by receiving an optical image of an object formed via the optical system102a. The image sensor102bis an image-plane phase detection sensor. Here, the image-plane phase detection sensor is an image sensor in which one pixel includes a single microlens and a plurality of photoelectric conversion elements (photodiodes, etc.), and can acquire a plurality of viewpoint images at the same time because light beams that have passed through different pupil areas of the imaging optical system is recorded by the photoelectric conversion element. The present invention uses an image sensor in which one pixel includes two independent photoelectric conversion elements on the left and right sides. This structure can simultaneously provide two-viewpoint images in which the same object is viewed with the right eye and the left eye. The image pickup apparatus102provided with the image-plane phase detection sensor is mainly used to automatically focus (autofocus) on the object. The image pickup apparatus102according to this embodiment is not limited to this structure, and may be, for example, a stereo camera as long as it can acquire the two-viewpoint images. In that case, the image sensor102bis a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like. A memory for storing the acquired two-viewpoint images and a display unit for displaying the acquired two-viewpoint images, a transmitter for transmitting information to the outside, an output unit for storing the information in an external storage medium, a control unit for controlling each unit in the image pickup apparatus102, and the like are not illustrated inFIGS.2and3, respectively.

The image estimating apparatus (image processing apparatus)103has a memory103a, an acquisition unit103b, and an image processing unit (estimation unit)103c. The image estimating apparatus103performs image processing for estimating another viewpoint image from the acquired two-viewpoint images. A neural network is used for the image processing, and weight information of the neural network is read out of the memory103a. The weight has been learned by the learning apparatus101, and the image estimating apparatus103reads the weight information out of the memory101ain advance via the network107and stores the weight in the memory103a. The weight information to be stored may be a weight value itself or may be in an encoded form. The details of weight learning and image processing using the weight will be described later.

The estimated other viewpoint image is output to at least one of the display apparatus104, the recording medium105, and the output apparatus106b. The display apparatus104is, for example, a liquid crystal display or a projector. The user can perform editing work and the like while checking an image that is being processed via the display apparatus104. The recording medium105is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output apparatus106bis a printer or the like. The image estimating apparatus103serves to perform image processing such as development processing and refocusing, if necessary. Refocusing is image processing that changes an in-focus position (focus position or focus plane) to the corresponding points by adding the multi-viewpoint images after shifting them relative to each other so as to overlap the corresponding points. That is, the image processing unit103ccan generate a third image (refocused image) in which the in-focus position has been changed from the estimated other viewpoint image.

Referring now toFIGS.1and4, a description will be given of a weight learning method (a method of manufacturing a learned model) executed by the learning apparatus101in this embodiment.FIG.1illustrates a flow of learning weights of a neural network (machine learning model).FIG.4is a flowchart relating to the weight learning. Each step inFIG.4is mainly executed by the image generating unit101b, the feature amount generating unit101c, the feature amount comparing unit101d, the image reconstructing unit101e, or the learning unit101f.

First, in the step S101inFIG.4, the image generating unit101bgenerates two-viewpoint patches (two first images) and corresponding, another viewpoint patch (at least one second image). In this embodiment, the two-viewpoint patches are image (first images) made by capturing the same object at two different viewpoints, and the other viewpoint patch is an image (second image) made by capturing the same object at one or more viewpoints different from the two viewpoints. The patch refers to an image having a predetermined number of pixels (such as 120×120 pixels). In this embodiment, the one or more viewpoints different from the two viewpoints include, but are not limited to, a viewpoint that shifts in a direction orthogonal to a direction determined by the two viewpoints, and may include a viewpoint that shifts in a direction different from the direction determined by the two viewpoints.

This embodiment acquires multi-viewpoint images by a light field camera, and generates two-viewpoint patches and another viewpoint patch from the multi-viewpoint images. For example, four-viewpoint images may be acquired by the light field camera, and corresponding two-viewpoint images acquired by a camera with the image-plane phase detection sensor may be generated from the acquired four-viewpoint images. The four-viewpoint images to the two-viewpoint images may be generated by interpolating the viewpoint images based on the relationship between the viewpoints. The relationship between the viewpoints is determined by the configuration of the light field camera sensor and the image-plane phase detection sensor.

However, this embodiment is not limited to this example, and the two-viewpoint patches and the corresponding, other viewpoint patch may be generated by numerical calculation. For example, the two-viewpoint patches and the corresponding other viewpoint patch may be generated by arranging an object and a camera in a three-dimensional space of 3DCG and by rendering a corresponding image captured by the camera.

Next, in the step S102, the feature amount generating unit101cgenerates feature maps of the two-viewpoint patches (two first feature amounts) using the first neural network from the two-viewpoint patches (two first images). One first neural network is prepared for each of the two-viewpoint patches (exists for each of the two-viewpoint patches). In this embodiment, each first neural network to which the two-viewpoint patch is input has the same network architecture and weight. However, this embodiment is not limited to this example, and each first neural network into which the two-viewpoint patch is input may have a different network architecture and/or weight.

This embodiment inputs each of the two viewpoint patches to the first neural network, but the present invention is not limited to this example. For example, information on an imaging condition and a viewpoint of the estimated, other viewpoint patch may be input to the first neural network together with the two-viewpoint patches. Here, the imaging condition includes an aperture value (F-number) of the lens that has captured the two-viewpoint patches, a focal length, a distance from the camera to the in-focus object, and an ISO speed (sensor sensitivity), a shutter speed, a pixel pitch, and various filter characteristics, etc. of the sensor on the camera side.

One method of inputting information on the imaging condition and the viewpoint of the estimated, other viewpoint image into the neural network is, for example, a method of combine an image having a value corresponding to the F-number of the lens as a pixel value with the two-viewpoint patch and input the combination into the first neural network. Alternatively, a method may be used which inputs the value corresponding to the F-number of the lens into a third neural network, combines an image having the obtained value as the pixel value with the two-viewpoint patch, and inputs the combination into the first neural network. That is, the value corresponding to the F-number of the lens may not be input into the first neural network as it is, but may be input into it after the value is nonlinearly converted by the third neural network that is separately prepared. Further, instead of inputting the two-viewpoint patch directly into the first neural network, for example, an image made by interpolating and shifting the two-viewpoint patch by a subpixel in the lateral direction may be input.

Next, in the step S103, the feature amount comparing unit101dcompares two feature maps (two first feature amounts) with each other generated from the two-viewpoint patches (two first images), respectively, and extracts information on the corresponding points between the two-viewpoint patches (second feature amounts). In this embodiment, the feature amount comparing unit101dcalculates the corresponding points between the two feature maps (similarity of the two feature maps) and makes a comparison. More specifically, the feature amount comparing unit101dperforms processing based on the matrix product of the two feature maps and calculates the corresponding points between the two feature maps. However, this embodiment is not limited to the comparison method using the processing based on the matrix product of the feature maps. For example, classical feature amounts such as SIFT, SURF, and HoG, that have been used for alignment may be used. Alternatively, the feature map may be shifted by the subpixel in the lateral direction and then subtracted for each pixel for comparison. Alternatively, the feature maps may be concatenated and processed by a fourth neural network for comparison.

Next, in the step S104, the image reconstructing unit101egenerates an estimated patch (output image)202using the second neural network based on the information (second feature amount) on the corresponding points between the two-viewpoint patches. The estimated patch202is an estimation (estimated image) of the other viewpoint patch (second image)200obtained from the two-viewpoint patches (two first images)201.

This embodiment inputs information (second feature amount) on the corresponding points between the two-view patches into the second neural network, but the present invention is not limited to this example. For example, information on the imaging condition and the viewpoint of the estimated, other viewpoint image may be input into the second neural network together with the information on the corresponding points between the two-viewpoint patches. Here, the imaging condition is the same as that in the step S102. A method of inputting the information on the imaging condition and the viewpoint of the estimated, other viewpoint image into the second neural network is the same as that in the step S102. The information on the corresponding points between the two-viewpoint images may be used to warp the two-viewpoint patches and then input into the second neural network.

In this embodiment, as illustrated inFIG.1, a first half branched network represents the first neural network, and a second half network represents the second neural network. CN inFIG.1represents a convolution layer. In the convolution layer CN, the convolution of the input and the filter and an addition of the bias are calculated, and the result is nonlinearly transformed by the activation function. Each component of the filter and the initial value of the bias are arbitrary, and are determined by random numbers in this embodiment. The activation function can use, for example, ReLU (Rectifier Liner Unit), a sigmoid function, or the like. The multidimensional array output in each layer except the final layer is called a feature map. A skip connection203is a shortcut path for synthesizing feature maps output from nonsuccessive layers. The feature map may be synthesized by element-wise summation or by concatenation in the channel direction. In this embodiment, the element-wise summation is adopted.

Reference numeral204inFIG.1denotes a comparing unit for comparing two feature maps (two first feature quantities) with each other generated from the two-viewpoint patches (two first images)201, respectively, and corresponds to the function of the feature amount comparing unit101dinFIG.2. As described above, in this embodiment, the feature map comparison method is a matrix product, but the method is not limited to this example.

An element in a dotted-line frame205inFIG.1represents a residual block (Residual Block). Although not surrounded by a dotted line, in the first neural network, those modules before and after the residual block are also residual blocks. Similarly, the second neural network includes residual blocks. Thus, a neural network in which residual blocks are stacked is called a residual network, and is widely used for image processing by DL. However, this embodiment is not limited to this example, and other modules may be stacked to form a neural network. For example, an Inception Module can be used in which convolutional layers having different convolution filter sizes and a plurality of obtained feature maps are integrated to form a final feature map. A dense block having a dense skip connection may be used.

The processing load (number of convolutions) may be reduced by downsampling the feature maps in the layers close to the input and upsampling the feature maps in the layers close to the output so as to reduce the size of the feature map in the intermediate layer. Downsampling the feature map can use pooling, strided convolution, inverse pixel shuffle, or the like. Upsampling the feature map can use a deconvolution or transposed Convolution, pixel shuffle, interpolation, or the like.

Next, in the step S105inFIG.4, the learning unit101fupdates the weight of the neural network illustrated inFIG.1using an error between the other viewpoint patch (second image)200and its estimated patch (output image or estimated image)202. The weight includes the filter component and bias of each layer. The backpropagation method is used to update the weight, but this embodiment is not limited to this example. The mini batch learning finds the error between the plurality of other viewpoint patches200and the estimated patches202corresponding to them, and updates the weight. The error function (Loss function) may use, for example, the L2 norm or the L1 norm. The weight updating method (learning method) is not limited to mini batch learning, and may use batch learning or online learning. Only the second neural network may be trained using the weight of the first neural network learned previously. That is, the weight of the first neural network may be fixed without learning, and only the second neural network may be learned. Alternatively, only the first neural network may be learned. Although not illustrated inFIG.1, when the third neural network is used in the step S102, its weight is also updated at the same time. This is similarly applied when the fourth neural network is used in the step S103.

Next, in the step S106, the learning unit101fdetermines whether or not the weight learning has been terminated. The termination can be determined by determining whether the number of iterations of learning (weight updates) has reached a specified value, or whether a weight changing amount during updating is smaller than the specified value. If it is determined that the learning has not yet been terminated, the flow returns to the step S101to acquire a plurality of new two-viewpoint patches (first images)201and the other viewpoint patch (second image)200. On the other hand, when it is determined that the learning has been terminated, the learning apparatus101(learning unit101f) ends the learning and stores the weight information in the memory101a.

This embodiment uses the configuration illustrated inFIG.1as the machine learning model, but the present invention is not limited to this example. For example, the configuration illustrated inFIG.11or12may be used.FIG.11illustrates a learning flow of a machine learning model according to a modification of this embodiment. As illustrated inFIG.11, a machine learning model may be used in which two-viewpoint patches (two first images)201are input into a common neural network to generate an estimated patch (output image)202. That is, the machine learning model according to this embodiment does not have to be separated into two neural networks like the first neural network and the second neural network.FIG.12illustrates a learning flow of a machine learning model according to another modification of this embodiment. As illustrated inFIG.12, a machine learning model can be used which inputs two outputs from the first neural networks of the two-viewpoint patches (two first images)201directly into the second neural network and generates the estimated patch (output image)202. That is, the machine learning model according to this embodiment does not need to use the comparing unit204for comparing the outputs of the first neural networks.

Referring now toFIG.5, a description will be given of an estimation of the other viewpoint image executed by the image estimating apparatus103in this embodiment.FIG.5is a flowchart relating to the estimation of the other viewpoint image. Each step inFIG.5is mainly executed by the acquisition unit103band the image processing unit103cin the image estimating apparatus103.

First, in the step S201, the acquisition unit103bacquires information on the captured images and weight. The captured images are two-viewpoint images similar to learning, and transmitted from the image pickup apparatus102in this embodiment. The weight information is transmitted from the learning apparatus101and stored in the memory103a. When the information on the imaging condition and the viewpoint of the estimated, other viewpoint image in the step S102inFIG.4is input to the first neural network together with the first image, the information is also acquired. The imaging condition is transmitted from the image pickup apparatus102together with the captured images and stored in the memory103a. The information on the viewpoint of the other viewpoint image to be estimated is input by the user, for example, from the input apparatus106a, and is stored in the memory103a.

Next, in the step S202, the image processing unit103cinputs the captured images into the neural network to which the acquired weight information is applied, and estimates the other viewpoint image. Here, the other viewpoint image is an estimated image captured at one or more viewpoints different from that of the captured images. A neural network similar to the configuration illustrated inFIG.1can be used to estimate the other viewpoint image. In inputting the captured images into the neural network, it is unnecessary to cut it out to the same size as that of the two-viewpoint patch used during learning, but in order to accelerate the processing, the captured image may be decomposed into multiple patches that overlap each other and processed. In this case, the patches obtained after the processing may be fused to form another viewpoint image.

As described above, the image estimating apparatus103according to this embodiment has an acquisition unit103band an image processing unit103c. The acquisition unit103bacquires two first images made by capturing the same object at two different viewpoints. The image processing unit103cinputs the two first images into the machine learning model and estimates a second image at one or more viewpoints different from the two viewpoints. The learning apparatus101according to this embodiment has an image generating unit101band a learning unit101f. The image generating unit101bgenerates two first images made by capturing the same object at two different viewpoints and a second image at one or more viewpoints different from the two viewpoints. The learning unit101finputs two first images into the machine learning model, compares the output image and the second image with each other, and learns the machine learning model.

In this embodiment, the learning apparatus101and the image estimating apparatus103are separate members as an example, but the present invention is not limited to this example, and the learning apparatus101and the image estimating apparatus103may be integrated with each other. That is, a single device may perform learning (processing illustrated inFIG.4) and an estimation (processing illustrated inFIG.5).

The configuration according to this embodiment can provide an image processing apparatus that robustly estimates another viewpoint image of a rectangular-shaped object in the two-viewpoint images having a disparity in the lateral direction.

Second Embodiment

Next follows a description of an image processing system according to a second embodiment of the present invention. In this embodiment, similar to the first embodiment, DL image processing for estimating another viewpoint image from two-viewpoint images is learned and executed using a neural network. The image processing system according to this embodiment is different from that of the first embodiment in that the image pickup apparatus acquires captured images (two-viewpoint images) and performs image processing.

FIG.6is a block diagram of an image processing system300according to this embodiment.FIG.7is an external view of the image processing system300. The image processing system300includes a learning apparatus (image processing apparatus)301and an image pickup apparatus302connected the learning apparatus301via a network303. The learning apparatus301and the image pickup apparatus302do not have to be always connected via the network303.

The learning apparatus301has the same configuration as that of the learning apparatus101of the first embodiment. That is, the learning apparatus301includes a memory311, an image generating unit312, a feature amount generating unit313, a feature amount comparing unit314, an image reconstructing unit315, and a learning unit316. In this configuration, the learning apparatus301learns the weight for performing image processing for estimating another viewpoint image from the two-viewpoint images using the neural network.

The image pickup apparatus302captures the object space, acquires captured images (two-viewpoint images), and estimates another viewpoint image from the two-viewpoint images using the read information on the weight. Details of the image processing performed by the image pickup apparatus302will be described later. The image pickup apparatus302includes an optical system (imaging optical system)321and an image sensor322. The image estimating unit323has an acquisition unit323aand an image processing unit323b, and estimates another viewpoint image from the two-viewpoint images using the weight information stored in the memory324. Since learning of the weight for the neural network executed by the learning apparatus301is the same as that of the first embodiment, a description thereof will be omitted, and only a description of the image processing executed by the image pickup apparatus302will be described later in detail.

The weight information is learned in advance by the learning apparatus301and stored in the memory311. The image pickup apparatus302reads the weight information out of the memory311via the network303and stores it in the memory324. The estimated, other viewpoint image is stored in the recording medium325a. When the user gives an instruction regarding the display of the estimated, other viewpoint image, the saved image (estimated, other viewpoint image) is read out and displayed on the display unit325b. The captured images (two-viewpoint images) already stored in the recording medium325amay be read out, and the image estimating unit323may estimate the other viewpoint image. When the instruction is given by the user, a refocused image may be generated from the estimated, other viewpoint image. The above series of control is performed by the system controller327.

Next follows a description of the estimation of the other viewpoint image executed by the image estimating unit323in this embodiment. The procedure of image processing is the same as that ofFIG.5in the first embodiment. Each step of image processing is mainly executed by the acquisition unit323aor the image processing unit323bin the image estimating unit323.

First, in the step S201, the acquisition unit323aacquires the captured images (two-viewpoint images) and the weight information. The two-viewpoint images have already been acquired by the image pickup apparatus302and stored in the memory324. The weight information has already been transmitted from the learning apparatus301and stored in the memory324.

Next, in the step S202, the image processing unit323binputs the captured images (two-viewpoint images) into the neural network to which the acquired weight is applied, and estimates the other viewpoint image. A neural network similar to the configuration illustrated inFIG.1can be used for estimating the other viewpoint image.

Due to the above configuration, this embodiment can provide an image processing system that robustly estimates another viewpoint image of a rectangular-shaped object in the two-viewpoint images having a disparity in the lateral direction.

Third Embodiment

Next follows a description of the image processing system according to a third embodiment of the present invention. The image processing system according to this embodiment is different from the image processing systems according to the first and second embodiments in including a processing apparatus (computer) that transmits the captured images (two-viewpoint images) that are targets to be image-processed to an image estimating apparatus and receives another viewpoint image estimated from the two-viewpoint images from the image estimating apparatus.

FIG.8is a block diagram of an image processing system400according to this embodiment. The image processing system400includes a learning apparatus401, an image pickup apparatus402, an image estimating apparatus (second device)403, and a computer (processing apparatus, first device)404. The learning apparatus401and the image estimating apparatus403are, for example, servers. The computer404is, for example, a user terminal (personal computer or smartphone). The computer404is connected to the image estimating apparatus403via the network405. The image estimating apparatus403is connected to the learning apparatus401via the network406.

The computer404and the image estimating apparatus403are communicable with each other, and the image estimating apparatus403and the learning apparatus401are communicable with each other. Since the configurations of the learning apparatus401and the image pickup apparatus402are the same as those of the learning apparatus101and the image pickup apparatus102of the first embodiment, a description thereof will be omitted.

The image estimating apparatus403includes a memory403a, an acquisition unit403b, a communication unit (receiver)403c, and an image processing unit403d. The memory403a, the acquisition unit403b, and the image processing unit403dare the same as the memory103a, the acquisition unit103b, and the image processing unit103cin the image estimating apparatus103according to the first embodiment, respectively. The communication unit403cserves to receive a request transmitted from the computer404and to transmit the other viewpoint image estimated by the image estimating apparatus403to the computer404.

The computer404includes a communication unit (transmitter)404a, a display unit404b, an input unit404c, an image processing unit404d, and a recording unit404e. The communication unit404aserves to transmit a request for causing the image estimating apparatus403to perform processing for the captured images (two first images and two-viewpoint images) to the image estimating apparatus403, and to receive the output image (other viewpoint image) estimated by the image estimating apparatus403. The display unit404bserves to display various information. The information displayed on the display unit404bincludes, for example, the captured images (two-viewpoint images) transmitted to the image estimating apparatus403, another viewpoint image received from the image estimating apparatus403, and a refocused image (third image) generated from the other viewpoint image. The input unit404cinputs an instruction from the user. The image processing unit404dserves to receive the other viewpoint image estimated by the image estimating apparatus403and to perform image processing for the other viewpoint image. The image processing applicable to the estimated, other viewpoint image includes refocus processing and the like. The recording unit404erecords the captured images acquired from the image pickup apparatus402, the output image received from the image estimating apparatus403, and the like.

Referring now toFIG.9, a description will be given of the image processing according to this embodiment.FIG.9is a flowchart relating to the estimation of the other viewpoint image in this embodiment. The image processing in this embodiment is equivalent to the image processing (FIG.5) described in the first embodiment. The image processing illustrated inFIG.9is started when the user gives an instruction to start the image processing via the computer404.

A description will now be given of the operation of the computer404. First, in the step S401, the computer404transmits a processing request for the captured images (two-viewpoint images) to the image estimating apparatus403. A method of transmitting the two-viewpoint images to be processed to the image estimating apparatus403is not limited. For example, the two-viewpoint images may be uploaded to the image estimating apparatus403at the same time as the step S401or prior to the step S401. The two-viewpoint images may be images stored on a server different from the image estimating apparatus403. In the step S401, the computer404may transmit an ID for authenticating the user, an imaging condition, information on the viewpoint of the other viewpoint image to be estimated, and the like, as well as the processing request for the two-viewpoint images. Next, in the step S402, the computer404receives the other viewpoint image estimated by the image estimating apparatus403.

Next follows a description of the operation of the image estimating apparatus403. First, in the step S501, the image estimating apparatus403receives the processing request for the captured images (two-viewpoint images) transmitted from the computer404. The image estimating apparatus403determines that the processing for the two-viewpoint images has been instructed and executes the processing subsequent to the step S502.

Next, in the step S502, the image estimating apparatus403acquires weight information. The weight information is information (learned model) learned by the same method as that of the first embodiment (FIG.4). The image estimating apparatus403may acquire weight information from the learning apparatus401, or weight information that has been previously acquired from the learning apparatus401and stored in the memory403a. The subsequent step S503is the same as the step S202of the first embodiment. Next, in the step S504, the image estimating apparatus403transmits the estimated, other viewpoint image to the computer404.

As discussed, the image estimating apparatus403may be controlled with the computer404communicatively connected to the image estimating apparatus403as in this embodiment.

Other Embodiments

When estimating another viewpoint image from two-viewpoint images having a disparity in the lateral direction using a machine learning DL, each embodiment can robustly estimate the other viewpoint image of a rectangular-shaped object in the two-viewpoint images. Therefore, each embodiment can provide an image processing apparatus, an image processing method, a storage medium, a method for manufacturing a learned model, and an image processing system, each of which can estimate another viewpoint image from the two-viewpoint images with high accuracy.

This application claims the benefit of Japanese Patent Application No. 2020-103809, filed on Jun. 16, 2020 which is hereby incorporated by reference herein in its entirety.