An information processing apparatus includes an identification unit configured to identify a partial region of an input image, and a processing unit configured to perform image processing for reducing degradation of the input image on the input image by inference using a neural network. The processing unit is configured to change the image processing between the partial region and another region.

BACKGROUND

Field of the Disclosure

The present disclosure relates to an information processing technique for reducing image degradation.

Description of the Related Art

Deep neural networks (DNNs) have been applied to various information processing application programs in recent years. A DNN refers specifically to a neural network including two or more hidden layers, and its performance improves as the number of hidden layers increases. An example of information processing using a DNN is image processing for reducing image degradation. Degradation elements of an image include, for example, noise, blur, low resolution, and missing data. The processing for reducing image degradation may include noise reduction, deblurring, super-resolution, and missing data compensation.

Zhang, Kai; Zuo, Wangmeng; Zhang, Lei, “FFDNet: Toward A Fast AND Flexible Solution for CNN based Image Denoising”, Institute of Electrical and Electronics Engineers (IEEE) Transactions on Image Processing, vol. 27, issue 9, pp. 4608-4622 (hereinafter, referred to as Non-Patent Literature 1) discusses a method for training a neural network using a plurality of images having different noise levels. Guo, Shi; Yan, Zifei; Zhang, Kai; Zuo, Wangmeng; Zhang, Lei, “Toward Convolutional Blind Denoising of Real Photographs”, 2019 IEEE/Computer Vision Foundation (CVF) Conference on Computer Vision and Pattern Recognition (CVPR) (hereinafter, referred to as Non-Patent Literature 2) discusses a method for estimating noise in an actually captured image from Poisson distribution variance by information processing using a multilayer neural network, and obtaining a noise-reduced image based on the estimation result.

However, the methods discussed in the foregoing Non-Patent Literature 1 and Non-Patent Literature 2 are unable to favorably reduce degradation in each local partial region of an image to be processed separately.

SUMMARY

According to an aspect of the present disclosure, an information processing apparatus includes an identification unit configured to identify a partial region of an input image, and a processing unit configured to perform image processing for reducing degradation of the input image on the input image by inference using a neural network. The processing unit is configured to change the image processing between the partial region and another region.

DESCRIPTION OF THE EMBODIMENTS

Some exemplary embodiments will be described below with reference to the drawings. The following exemplary embodiments are not intended to limit the present disclosure, and not all combinations of the features described in the exemplary embodiments are used as the solving means of the present disclosure. The configurations of the exemplary embodiments can be modified or changed as appropriate depending on the specifications and various conditions (such as use condition and use environment) of the apparatuses to which the present disclosure is applied. Parts of the exemplary embodiments described below may be combined as appropriate. In the following description of the exemplary embodiments, like numbers will refer to likes components.

A convolutional neural network (CNN) that is used in deep learning-based information processing techniques in general used in the following exemplary embodiments will initially be described. A CNN is a technique for performing convolution of a filter generated by training or learning on image data and nonlinear calculation repeatedly. Filters are also referred to as local receptive fields (LRFs). Image data obtained by the convolution of a filter on image data, followed by nonlinear calculation, is called a feature map. Training is performed using training data (training images or data sets) including pairs of input image data and output image data. Simply put, training refers to generating filter values capable of converting input image data into corresponding output image data with high precision from the training data. Details thereof will be described below.

If the image data includes red, green, and blue (RGB) color channels or if a feature map includes image data on a plurality of images, the filter to be used for convolution also includes a plurality of channels accordingly. More specifically, the convolution filter is expressed by a four-dimensional array including vertical and horizontal sizes, the number of images, and the number of channels. Processing for performing the convolution of a filter on image data (or feature map) and nonlinear calculation is expressed in units of layers, like an nth-layer feature map and an nth-layer filter. For example, a CNN that repeats filter convolution and nonlinear calculation three times has a three-layer network structure. Such nonlinear calculation processing can be formulated by the following Eq. (1):

In Eq. (1), Wnis the nth-layer filter, bnis an nth-layer bias, f is a nonlinear operator, Xnis the nth-layer feature map, and * is the convolution operator. (1) indicates that the filter or feature map is the lth one. The filters and biases are generated by training to be described below, and are referred to collectively as “network parameters”. The nonlinear calculation uses a sigmoid function or a rectified linear unit (ReLU), for example. A ReLU is given by the following Eq. (2):

As expressed by Eq. (2), negative components of the input vector X become zero, and positive components are maintained intact.

Among known CNN-based networks are Residual Network (ResNet) in the image recognition field and its application Residual Encoder-Decoder Network (RED-Net) in the super-resolution field. Both include a multilayer CNN to perform filter convolution many times for high-precision processing. For example, the ResNet is characterized by a network structure including a path for shortcutting convolution layers, whereby a multilayer network including as many as 152 layers is constructed to achieve high-precision recognition close to human's recognition rates.

The reason why a multilayer CNN increases processing precision is, simply put, that a nonlinear relationship between the input and output can be represented by repeating nonlinear calculation many times.

Next, CNN training will be described. A CNN is trained by minimizing an objective function with respect to training data including pairs of input training image (hereinafter, also referred to as student image) data and corresponding output training image (hereinafter, also referred to as teacher image) data. The objective function is typically expressed by the following Eq. (3):

In Eq. (3), L is a loss function for measuring an error between a correct answer and its estimation. Yiis ith output training image data, and Xi is ith input training image data. F is a function collectively expressing the calculation performed in each layer of the CNN (Eq. (1)). θ is a network parameter (filter and bias). ∥Z∥2is the L2 norm, or simply put, the root sum square of the components of a vector Z. n is the total number of pieces of training data used in training. Since the total number of pieces of training data is typically large, stochastic gradient descent (SGD) selects some of the pieces of training image data at random and uses the selected pieces for training. This can reduce calculation load for training using a lot of pieces of training data. There are known various methods for minimizing (optimizing) the objective function, including the momentum method, adaptive gradient (AdaGrad), AdaDelta, and adaptive moment estimation (Adam). Adam is given by the following Eqs. (4):

In Eqs. (4), θitis the ith network parameter at the tth repetition, and g is the gradient of the loss function L for θit. m and v are moment vectors, α is the base learning rate, β1and β2are hyper parameters, and ε is a small constant. Since there is no selection guideline on the optimization method for training, basically any method can be used. However, different methods have different convergence properties and are known to make a difference in training time.

In the present exemplary embodiment, information processing (image processing) for reducing image degradation is performed using the CNN mentioned above. Examples of degradation elements of an image include degradation such as noise, blur, aberration, compression, low resolution, and missing data, and degradation such as a drop in contrast due to the weather during imaging, including fog, mist, snow, and rain. Examples of the image processing for reducing image degradation may include noise reduction, deblurring, aberration correction, missing data compensation, correction of compression-based degradation, super-resolution processing on a low-resolution image, and processing for correcting a drop in contrast due to the weather during imaging. Image degradation reduction processing according to the present exemplary embodiment is processing for generating or restoring a degradation-free (or little degraded) image from a degraded image. In the following description, such image degradation reduction processing will be referred to as image restoration processing.

In other words, image restoration according to the present exemplary embodiment covers the case of enabling a reduction in degradation included in the original image itself, as well as the case of restoring an image that is degradation-free (little degraded) itself and subsequently degraded by amplification, compression, decompression, or other image processing.

For image degradation that can be expressed by a specific parameter or parameters, image restoration processing using a neural network can provide image restoration performance surpassing that of conventional processing not using a neural network. However, the image restoration performance of a single neural network can be insufficient if there are various types of image degradation. For example, in the case of noise reduction, a neural network trained using images with a single noise level or a sufficiently narrow range of noise levels can provide a sufficient noise reduction effect if the target image of the image restoration processing has a noise level similar to in the training. On the other hand, if the target image of the image restoration processing has a noise level different from that of the images used in training, the neural network can provide an insufficient noise reduction effect. The foregoing Non-Patent Literature 1 discusses a method for training a neural network using a plurality of images of different noise levels so that the single neural network can handle a plurality of images of different noise levels. According to the method discussed in Non-Patent Literature 1, a sufficient noise reduction effect can be obtained if the target image of the image restoration processing has a noise level similar to that of one of the images used in training. However, as described above, the methods discussed in Non-Patent Literature 1 and Non-Patent Literature 2 are unable to favorably reduce degradation in local partial regions of the target image of the image restoration processing.

A first exemplary embodiment deals with a method for estimating the intensity of image quality degradation in an input image and adjusting the intensity of restoration in each local region of the input image based on the estimation result so that the degradation can be reduced region by region of the image to be processed, without changing the configuration of the neural network. The intensity of restoration refers to the amount of reduction in degradation in the degradation reduction processing, i.e., the amount of restoration in the image restoration processing. The present exemplary embodiment will be described below by using noise as an example of a degradation element of an image, using an example where noise reduction processing is performs as the image restoration processing.

Configuration Example of Information Processing System

FIG.1is a diagram illustrating an example of a system configuration to which an information processing apparatus according to the first exemplary embodiment is applied. The information processing apparatus illustrated inFIG.1includes a cloud server200and an edge device100connected via the Internet. The cloud server200is in charge of generating training data, estimating image quality degradation, and doing training for restoration. The edge device100is in charge of degradation restoration on an image to be processed. The generation of training data, the estimation of image quality degradation, and the training for restoration by the cloud server200will hereinafter be referred to as degradation restoration training. The degradation restoration by the edge device100will be referred to as degradation restoration inference.

<Hardware Configuration of Edge Device>

The edge device100according to the present exemplary embodiment obtains raw image data (Bayer arrangement) input from an imaging apparatus10as an input image to perform the image restoration processing on. The edge device100then performs degradation restoration inference on the input image to be processed by applying trained network parameters provided by the cloud server200. In other words, the edge device100is an information processing apparatus that reduces noise in raw image data by using neural networks provided by the cloud server200and running an information processing application program installed in advance. The edge device100includes a central processing unit (CPU)101, a random access memory (RAM)102, a read-only memory (ROM)103, a mass storage device104, a general-purpose interface (I/F)105, and a network I/F106. These components are connected to one another by a system bus107. The edge device100is also connected to the imaging apparatus10, an input apparatus20, an external storage device30, and a display device40via the general-purpose I/F105.

The CPU101runs programs stored in the ROM103using the RAM102as a work memory, and controls the components of the edge device100via the system bus107in a centralized manner. The mass storage device104is a hard disk drive (HDD) or a solid-state drive (SSD), for example, and stores various types of data and image data to be handled by the edge device100. The CPU101writes data to the mass storage device104and reads data stored in the mass storage device104via the system bus107. The general-purpose I/F105is a serial bus I/F such as a Universal Serial Bus (USB), Institute of Electrical and Electronics Engineers (IEEE) 1394, and High-Definition Multimedia Interface (HDMI)® I/Fs. The edge device100obtains data from the external storage device30(various storage media such as a memory card, a CompactFlash (CF) card, a Secure Digital (SD) card, and a USB memory) via the general-purpose I/F105. The edge device100also accepts user instructions from the input apparatus20, such as a mouse and a keyboard, via the general-purpose I/F105. The edge device100outputs image data processed by the CPU101to the display device40(various image display devices such as a liquid crystal display) via the general-purpose I/F105. The edge device100obtains data on a captured image (raw image) to perform the noise reduction processing on from the imaging apparatus10via the general-purpose I/F105. The network I/F106is an I/F for connecting to the Internet. The edge device100accesses the cloud server200using an installed web browser, and obtains network parameters for degradation restoration inference.

<Hardware Configuration of Cloud Server>

The cloud server200according to the present exemplary embodiment is an information processing apparatus that provides cloud services on the Internet. More specifically, the cloud server200generates training data, performs degradation restoration training, and generates a trained model storing network parameters resulting from the training and network structures. The cloud server200then provides the trained model in response to a request from the edge device100. The cloud server200includes a CPU201, a ROM202, a RAM203, a mass storage device204, and a network I/F205. These components are connected to one another by a system bus206. The CPU201controls operation of the entire cloud server200by reading control programs stored in the ROM202and performing various types of processing. The RAM203is used as a temporary storage area such as a main memory and a work area of the CPU201. The mass storage device204is a large-capacity secondary storage device such as an HDD and an SSD, and stores image data and various programs. The network I/F205is an I/F for connecting to the Internet. The network I/F205provides the trained model storing the foregoing network parameters and network structures in response to a request from the web browser on the edge device100.

While the edge device100and the cloud server200also include other components than the foregoing, a description thereof will be omitted here. In the present exemplary embodiment, the trained model obtained by the cloud server200generating the training data and performing the degradation restoration training is assumed to be downloaded to the edge device100, and the edge device100to perform degradation restoration inference on the input image data to be processed. Such a system configuration is just an example and not restrictive. For example, the functions of the cloud server200may be subdivided and the generation of the training data and the degradation restoration training may be performed by separate apparatuses. The imaging apparatus10may be configured to have both the functions of the edge device100and those of the cloud servers200, and perform all the generation of the training data, the degradation restoration training, and the degradation restoration inference.

<Functional Blocks of Entire Information Processing System>

Next, a functional configuration of the entire information processing system according to the present exemplary embodiment will be described with reference toFIG.2.

As illustrated inFIG.2, the edge device100includes a specific region extraction unit111and an inference unit112. As will be described in detail below, the inference unit112has the function of image restoration processing for reducing image degradation. The inference unit112includes an inference-specific degradation estimation unit113, an intensity adjustment unit114, and an inference-specific degradation restoration unit115. In other words, the inference unit112includes two neural networks, namely, a degradation inference network including the inference-specific degradation estimation unit113and a degradation restoration network including the inference-specific degradation restoration unit115.

The cloud server200includes a degradation addition unit211and a training unit212. As will be described in detail below, the training unit212has a degradation estimation function of estimating degradation of a student image using a teacher image and the student image, and a degradation restoration function of performing image restoration processing on the student image based on the result of the degradation estimation. The training unit212includes a training-specific degradation estimation unit213, a training-specific degradation restoration unit214, an error calculation unit215, and a model update unit216. In other words, the training unit212includes two neural networks, namely, a degradation estimation network including the training-specific degradation estimation unit213and a degradation restoration network including the training-specific degradation restoration unit214.

The configuration illustrated inFIG.2can be modified or changed as appropriate. For example, one functional unit may be divided into a plurality of functional units. Two or more functional units may be integrated into one. The configuration illustrated inFIG.2may be implemented by two or more apparatuses.

In such a case, the apparatuses are connected via a circuit or a wired or wireless network, and perform the processes according to the present exemplary embodiment by performing data communication with each other for cooperative operation.

The functional units of the edge device100will initially be described.

The specific region extraction unit111obtains input image data116, and extracts local partial regions from the input image data116(input image). In the present exemplary embodiment, the local partial regions of the input image will hereinafter be referred to as specific regions. The specific region extraction unit111then outputs a specific region map indicating the extraction result of the specific regions. In the present exemplary embodiment, raw image data where each pixel has a pixel value corresponding to the R, G, or B color is used as the input image data116. The raw image data is image data captured using a color filter of Bayer arrangement where each pixel has information about one color.

In the present exemplary embodiment, a specific region may be the region of a main object or a specific object included in the input image data116, or the region of a different object. There may be one main object to extract a specific region of, or a plurality of main objects. There may be one different object to extract a specific region of than a main object, or a plurality of such objects. Which of the regions of such main and other objects to extract as a specific region may be determined in advance or freely selected by the user, for example. Specific regions are not limited to regions inside the image like that of an object. For example, components within a specific frequency band included in the image can be extracted as specific regions. As an example, components within a specific frequency band such as a high frequency band detected using an edge detection filer, like a Sobel filter and a Laplacian filter, may be extracted as specific regions. Undetected components in a lower frequency band may be extracted as specific regions. Both the regions of specific objects or main objects (or the other regions) and components within a specific frequency band (or the other frequency bands) may be extracted as specific regions. Which regions to extract may be selectively switched as appropriate. The methods for extracting specific regions are not limited thereto, and a method for extracting a region freely specified by the user in the input image as a specific region may be used as well. The user-specified specific region may be the region of a main object or a specific object, or a different region. The user-specified specific region may be a region including components within a specific frequency band or one including components in the other frequency bands.

The inference unit112estimates degradation of the input image data116using a trained model220received from the cloud server200, and performs degradation restoration inference based on the estimation result.

In the present exemplary embodiment, the inference unit112reduces degradation (performs degradation restoration) while controlling the amount of restoration by restoration processing in the specific region(s) and the other regions separately. As employed herein, the amount of restoration refers to an amount by which the intensity of degradation restoration is adjusted in each of the specific region(s) and the other regions, i.e., the amount of reduction in degradation by the degradation reduction processing. In the present exemplary embodiment, the degradation restoration inference is performed by the inference-specific degradation estimation unit113, the intensity adjustment unit114, and the inference-specific degradation restoration unit115.

The inference-specific degradation estimation unit113obtains the input image data116, and estimates the amount of degradation indicating the degree of degradation of the input image data116using the trained model220. The amount of degradation is estimated using a neural network.FIG.3Ais a diagram illustrating a processing procedure for the inference unit112. As illustrated inFIG.3A, the inference-specific degradation estimation unit113inputs the input image data116into a first CNN301to repeat the convolution calculation and the nonlinear calculation expressed by Eqs. (1) and (2) a plurality of times, and outputs a degradation estimation result302that is the estimation result of image degradation.

FIGS.4A and4Bare diagrams for describing the structure of CNNs and a procedure for inference and training.

The processing by the first CNN301will initially be described with reference toFIGS.3A and4A.

The first CNN301includes a plurality of filters401for preforming the calculation of the foregoing Eq. (1). The inference-specific degradation estimation unit113initially inputs the input image data116into this CNN. The inference-specific degradation estimation unit113then sequentially applies the filters401to the input image data116to calculate a feature map (not illustrated). The inference-specific degradation estimation unit113outputs the result of application of the last filter401as the degradation estimation result302. The degradation estimation result302has the same channels as those of the input image data116.

The intensity adjustment unit114processes the degradation estimation result302estimated by the inference-specific degradation estimation unit113, using a specific region map303provided by the specific region extraction unit111. In the present exemplary embodiment, the processing of the degradation estimation result refers to intensity adjustment processing for adjusting the amount of estimation of degradation pixel by pixel in the specific region(s) included in the degradation estimation result302. As intensity adjustment processing304, the intensity adjustment unit114adjusts the amount of estimation of degradation by multiplying the specific region(s) in the degradation estimation result302by a coefficient α117pixel by pixel. If α>1, the amount of restoration of the input image data116increases. If α<1, the amount of restoration decreases.

Next, the inference-specific degradation restoration unit115receives the degradation estimation result302processed by the intensity adjustment unit114, and performs restoration processing on the degradation of the input image data116based on the processed degradation estimation result302. In other words, the inference-specific degradation restoration unit115performs the restoration processing on the degradation of the input image data116by controlling the amount of reduction in degradation based on the amount of estimation processed by the intensity adjustment unit114in the specific region(s) pixel by pixel. More specifically, the inference-specific degradation restoration unit115inputs the input image data116and the processed degradation estimation result302into a second CNN305. The inference-specific degradation restoration unit115then repeats the convolution calculation and the nonlinear calculation using the filters expressed by Eqs. (1) and (2) a plurality of times, and outputs the restored output image data118.

Next, the processing by the second CNN305will be described with reference toFIGS.3A and4B.

As illustrated inFIG.4B, the second CNN305includes a plurality of filters401and a connection layer402. The inference-specific degradation restoration unit115initially inputs the input image data116and the processed degradation estimation result302connected or added to each other in the channel direction into the second CNN305. The inference-specific degradation restoration unit115then applies filters401to the input data in succession to calculate a feature map. The inference-specific degradation restoration unit115then connects the feature map and the input data in the channel direction using the connection layer402. The inference-specific degradation restoration unit115further applies filters401to the connected result in succession, and outputs the output image data118having the same number of channels as that of the input image data116from the last filter401.

Next, the functional units of the cloud server200will be described.

The degradation addition unit211generates student image data by adding at least one or more types of degradation elements to teacher image data taken out of a degradation-free teacher image group. In the present exemplary embodiment, noise is described as an example of the degradation elements. The degradation addition unit211therefore generates student image data by adding noise as a degradation element to the teacher image data. In the present exemplary embodiment, the degradation addition unit211analyzes the physical properties of the imaging apparatus10, and generates student image data by adding noise corresponding to a wider range of amounts of degradation than that of possible amounts of degradation occurring in the imaging apparatus10as a degradation element to the teacher image data. The reason why a wider range of amounts of degradation than in the analysis result are added is to provide margins for improved robustness since the range of amounts of degradation can vary due to individual differences of imaging apparatuses10. More specifically, as illustrated inFIG.5, the degradation addition unit211generates student image data504by adding502noise based on an analysis result218of the physical properties of the imaging apparatus10as a degradation element to teacher image data501taken out of a teacher image group217. The degradation addition unit211then pairs the teacher image data501with the student image data504to generate training data. The degradation addition unit211generates a student image group including a plurality of pieces of student image data by adding a degradation element to each piece of teacher image data501in the teacher image group217, whereby training data505is generated. While the present exemplary embodiment deals with noise as an example, the degradation addition unit211may add any one or a combination of two or more of a plurality of types of degradation elements to the teacher image data501. As described above, examples of the degradation elements include blur, aberration, compression, low resolution, missing data, and a drop in contrast due to the weather in imaging.

The teacher image group217includes various types of image data. Examples include nature photographs including landscape photographs and animal pictures, portrait photographs such as studio portraits and sport pictures, and artificial pictures such as building and product pictures. In the present exemplary embodiment, like the input image data116, the teacher image data501is raw image data where each pixel has a pixel value corresponding to the R, G, or B color. The analysis result218of the physical properties of the imaging apparatus10includes, for example, the amount of noise occurring from the built-in image sensor of the camera (imaging apparatus)10at each sensitivity, and the amount of aberration caused by a lens. Using the analysis result218, how much degradation in image quality occurs can be estimated with respect to each imaging condition. In other words, by adding degradation estimated under an imaging condition to the teacher image data501, an image similar to one obtained in imaging can be generated.

The training unit212obtains network parameters219to be applied to the CNNs for degradation restoration training, initializes the weights of the CNNs using the network parameters219, and performs degradation restoration training using the training data505generated by the degradation addition unit211. The network parameters219include the initial values of parameters of the CNNs, and hyper parameters indicating the structures of and optimization methods for the CNNs. The degradation restoration training in the training unit212is performed by the training-specific degradation estimation unit213, the training-specific degradation restoration unit214, the error calculation unit215, and the model update unit216.

FIG.3Bis a diagram illustrating a processing procedure for the training unit212.

The training-specific degradation estimation unit213receives training data306from the degradation addition unit211and estimates the amount of degradation added307to student image data308. Specifically, the training-specific degradation estimation unit213initially inputs the student image data308into a first CNN301to repeat the convolution calculation and the nonlinear calculation using the filters expressed by Eqs. (1) and (2) a plurality of times, and outputs a degradation estimation result310.

The error calculation unit215inputs the amount of degradation added307and the degradation estimation result310to first loss processing311that is loss function calculation, and calculates an error therebetween. Here, the amount of degradation added307, the student image data308, and the degradation estimation result310all have the same number of pixels. Next, the model update unit216inputs the error calculated by the error calculation unit215into first update processing312, and updates the network parameters of the first CNN301to reduce (minimize) the error.

The training-specific degradation restoration unit214receives the student image data308and the degradation estimation result310estimated by the training-specific degradation estimation unit213, and performs restoration processing on the student image data308. Specifically, the training-specific degradation restoration unit214initially inputs the student image data308and the degradation estimation result310into a second CNN305to repeat the convolution calculation and the nonlinear calculation using the filters expressed by Eqs. (1) and (2) a plurality of times, and outputs a restoration result313.

The error calculation unit215then inputs the teacher image data309and the restoration result313into second loss processing314to calculate an error therebetween. Here, the teacher image data309and the restoration result313have the same number of pixels. The model update unit216then inputs the error calculated by the error calculation unit215into second update processing315, and updates the network parameters of the second CNN305to reduce (minimize) the error. The training-specific degradation estimation unit213and the training-specific degradation restoration unit214calculate the errors at different timing, but the network parameters are updated at the same timing. The first CNN301and the second CNN305used by the training unit212are the same neural networks as the first CNN301and the second CNN305used by the inference unit112, respectively.

<Processing Procedure for Entire Information Processing System>

Next, various types of processing performed by the information processing system according to the present exemplary embodiment will be described with reference toFIGS.6A and6B.FIGS.6A and6Bare flowcharts illustrating a processing procedure for the information processing system according to the present exemplary embodiment. The functional units illustrated inFIG.2are implemented by the CPUs101and201running information processing computer programs according to the present exemplary embodiment. All or some of the functional units illustrated inFIG.2may be implemented by hardware. A description will now be given with reference to the flowcharts ofFIGS.6A and6B.

An example of the procedure of the degradation restoration training performed by the cloud server200will initially be described with reference to the flowchart ofFIG.6A.

In step S601, the teacher image group217prepared in advance and the analysis result218of the physical properties of the imaging apparatus10, such as the properties of the image sensor, imaging sensitivity, an object distance, the focal length of the lens, an f-number, and an exposure value, are input to the cloud server200. Teacher image data is a raw image in the Bayer arrangement, and can be obtained by the imaging apparatus10capturing an image, for example. This is not restrictive. An image captured by the imaging apparatus10can be directly uploaded to the cloud server200. Images captured in advance may be stored in an HDD and subsequently uploaded to the cloud server200. The data on the teacher image group217and the analysis result218of the physical properties of the imaging apparatus10input to the cloud server200are delivered to the degradation addition unit211.

In step S602, the degradation addition unit211generates student image data by adding noise based on the analysis result218of the physical properties of the imaging apparatus10to the teacher image data in the teacher image group217input in step S601. Here, the degradation addition unit211adds the amounts of noise previously measured based on the analysis result218of the physical properties of the imaging apparatus10in preset order or random order.

In step S603, the network parameters to be applied to the CNNs for the degradation restoration training are input to the cloud server200. As described above, the network parameters here include the initial values of the parameters of the CNNs, and the hyper parameters indicating the structures of and optimization methods for the CNNs. The input network parameters are delivered to the training unit212. The training unit212initializes the weights of the first and second CNNs301and305using the received network parameters.

In step S604, the training-specific degradation estimation unit213estimates degradation of student image data generated in step S602. The training-specific degradation restoration unit214then restores the student image data based on the estimation result.

In step S605, the error calculation unit215calculates an error between the restoration result and the teacher image data based on the loss function expressed by Eq. (3).

In step S606, the model update unit216updates the network parameters to reduce (minimize) the error obtained in step S605as described above.

In step S607, the training unit212determines whether to end the training. For example, the training unit212can determine to end the training if the number of updates of the network parameters has reached a predetermined number. If the training unit212determines to not end the training (NO in step S607), the processing returns to step S604. In the processing of step S604and the subsequent steps, the cloud server200performs training using another pair of student image data and teacher image data.

Next, an example of the procedure of the degradation restoration inference performed by the edge device100will be described with reference to the flowchart ofFIG.6B.

In step S608, the trained model220trained by the cloud server200and the input image data116that is a Bayer-arrangement raw image to perform the degradation restoration processing on are input to the edge device100. For example, an image captured by the imaging apparatus10may be directly input as the raw image. An image captured in advance and stored in the mass storage device104may be read. The input image data116is delivered to the specific region extraction unit111and the inference unit112. The trained model220is delivered to the inference unit112.

In step S609, the specific region extraction unit111extracts a specific region or regions from the input image data116. The extraction result is delivered to the intensity adjustment unit114as the specific region map303.

In step S610, the inference-specific degradation estimation unit113constructs the same first CNN301as that used in the training by the training unit212, and estimates degradation of the input image data116. Here, the existing network parameters are initialized with the updated network parameters received from the cloud server200. The inference-specific degradation estimation unit113thus inputs the input image data116into the first CNN301to which the updated network parameters are applied, and performs degradation estimation to obtain a degradation estimation result302by the same method as that performed by the training unit212.

In step S611, the intensity adjustment unit114adjusts the amount of degradation restoration from the degradation estimation result310output in step S610, using the specific region map303output in step S609.

In step S612, the inference-specific degradation restoration unit115constructs the same second CNN305as that used in the training by the training unit212, and performs degradation restoration on the input image data116using the degradation estimation result adjusted in step S611. More specifically, like step S610, the inference-specific degradation restoration unit115initializes the existing network parameters with the updated network parameters received from the cloud server200, and performs degradation restoration on the input image data116by the same method as that performed by the training unit212. The image data degradation-restored by the inference-specific degradation restoration unit115is then output as the output image data118.

The entire processing procedure performed by the information processing system according to the present exemplary embodiment has been described above. Image degradation in the input image data116can thus be estimated and the intensity of restoration can be adjusted in each region of the input image based on the estimation result without changing the neural network configuration of the inference unit112.

In the present exemplary embodiment, the training data306is generated in step S602. However, the training data306may subsequently be generated. Specifically, the cloud server200may be configured to generate student image data corresponding to teacher image data in the subsequent degradation restoration training.

In the present exemplary embodiment, training is performed from scratch using the data on the teacher image group217prepared in advance. However, the processing of the present exemplary embodiment may be performed based on trained network parameters.

The present exemplary embodiment has been described in conjunction with raw images captured using a color filter in the Bayer arrangement. However, other color filter arrangements may be employed. The image data format is not limited to raw images, either. For example, demosaiced RGB images or YUV-converted images may be used.

The present exemplary embodiment has been described by using noise as an example of the degradation element. However, the degradation element is not limited thereto. As described above, degradation elements can include any one or a combination of the following: blur, aberration, compression, low resolution, missing data, and a drop in contrast due to the effect of fog, mist, snow, or rain in imaging.

In the present exemplary embodiment, the inference unit112is described to output the output image data118alone obtained by the restoration processing on the input image data116. However, the degradation estimation result302output by the inference-specific degradation estimation unit113may be output along with the output image data118.

In the present exemplary embodiment, the edge device100is described to perform the degradation restoration based on the input image data116alone, using the trained model220. However, parameters for assisting degradation restoration may also be used. For example, a lookup table including estimations about the degree of degradation to occur in image quality depending on imaging conditions such as the distance to an object, a focal length, a sensor size, and exposure may be stored in advance, and the amount of restoration may be adjusted by referring to the lookup table in degradation restoration. In other words, the inference unit112of the edge device100may adjust the intensity of degradation restoration based on the imaging conditions under which the image of the input image data116is captured.

The present exemplary embodiment has been described by using a case with there is a single piece of input image data116as an example. However, sequential image data such as frames of moving image data can also be processed. In such a case, continuous teacher image data in a time series and student image data generated by adding degradation thereto are used as the training image in the degradation restoration training. In performing degradation restoration on the sequential pieces of input image data, the same number of degradation estimation results are output. Here, the edge device100determines differences between the degradation estimation results, and sets the amount of noise reduction to be greater in regions of larger difference values to reduce ghosts and smoothen motion, since regions of large difference values can include a moving object, camerawork-based motion, or camera shake.

A second exemplary embodiment will be described. The first exemplary embodiment has dealt with an example where one type of degradation element (in the foregoing example, noise) is estimated from the input image data116and the amount of restoration is adjusted region by region based on the estimation result.

In the second exemplary embodiment, a method for estimating a plurality of degradation elements with respective priority levels from input image data, and performing restoration processing based on the estimated result and degradation estimation priority levels indicating the priority levels will be described. The description of a basic configuration of the information processing system common with that described in the first exemplary embodiment will be omitted, and differences will mainly be described below.

FIG.7is a block diagram illustrating a functional configuration of the entire information processing system according to the second exemplary embodiment.

As illustrated inFIG.7, an edge device700according to the second exemplary embodiment includes an inference-specific priority level determination unit701, a specific region extraction unit702, and an inference unit703. The inference unit703includes an inference-specific degradation estimation unit704, an intensity adjustment unit705, and an inference-specific degradation restoration unit706.

A cloud server710according to the second exemplary embodiment includes a training data generation unit711, a training-specific priority level determination unit714, and a training unit715. The training data generation unit711includes a data analysis unit712and a degradation addition unit713. The training unit715includes a training-specific degradation estimation unit716, a training-specific degradation restoration unit717, an error calculation unit718, and a model update unit719.

The functional units of the edge device700will initially be described.

The inference-specific priority level determination unit701determines the order in which a plurality of degradation elements included in input image data707and the intensities thereof are estimated, i.e., priority levels for inference.

It is suitable that the order of estimation is determined in that reverse order to that of the process of conversion from photons into pixel values. The process will now be briefly described. Photons flying from an object pass through a lens, an optical low-pass filter, and a color filter in order and reach photodiodes. The photodiodes convert the photons into electric charges, which are converted into voltages by capacitors, amplified by amplifiers, and then converted into pixel values by analog-to-digital (A/D) conversion circuits. Blur and aberration to occur before the photons reach the photodiodes can be optically analyzed. As for noise to occur after the arrival at the photodiodes and before the conversion into the pixel values, analysis of the sensor allows reproduction of how much image quality degradation occurs in the captured image. Image quality degradation such as a drop in contrast due to the weather like fog, mist, rain, and snow can also be analyzed since such degradation occurs when the photons pass through the lens. After the conversion of the photons into the pixel values, image quality is also degraded due to demosaicing processing for generating RGB values from a raw image in converting the raw image into a color image, color thinning processing from an RGB color space into a YUV color space, and bit compression. Such factors can also be analyzed if the image processing method and the compression method are known. The inference-specific priority level determination unit701determines the order in which the plurality of degradation elements included in the input image data707and the intensities thereof are estimated, i.e., the estimation priority levels based on the analysis results.

The specific region extraction unit702extracts a specific region or regions by processing similarly to that by the specific region extraction unit111according to the first exemplary embodiment.

The inference unit703estimates the plurality of degradation elements included in the input image data707based on the estimation priority levels determined by the inference-specific priority level determination unit701, and performs degradation restoration inference based on the estimation results, using a trained model723received from the cloud server710. The degradation restoration inference is performed by the inference-specific degradation estimation unit704, the intensity adjustment unit705, and the inference-specific degradation restoration unit706.

The inference-specific degradation estimation unit704obtains the input image data707and the estimation priority levels from the inference-specific priority level determination unit701, and estimates the plurality of degradation elements included in the input image data707and the amounts of degradation thereof based on the priority levels, using the trained model723. As many degradation estimation results as the number of degradation elements to be estimated are thereby obtained. The amounts of degradation are obtained in the form of a pixel-by-pixel degradation amount map.

The intensity adjustment unit705processes the degradation estimation results estimated by the inference-specific degradation estimation unit704using the specific region map generated by the specific region extraction unit702. The processing of the degradation estimation results is similar to that in the first exemplary embodiment, whereas the intensity adjustment unit705according to the second exemplary embodiment can adjust the intensity with respect to each degradation element. For example, if there are degradation estimation results of noise, blur, and aberration, the intensity adjustment unit705adjusts the amounts of degradation estimation by multiplying the degradation amount maps by a coefficient α708pixel by pixel. In the case of performing noise reduction alone on the specific region(s), the intensity adjustment unit705activates the degradation amount map about noise alone. Here, all the pixel values in the degradation amount maps about blur and aberration are set to 0 as if there were no degradation.

The inference-specific degradation restoration unit706receives the degradation estimation results processed by the intensity adjustment unit705, and outputs the result of restoration made of the degradation of the input image data707based on the priority levels as an output image data709.

Next, the functional units of the cloud server710according to the second exemplary embodiment will be described.

The data analysis unit712analyzes features of teacher image data taken out of a teacher image group720. Specifically, the data analysis unit712extracts high frequency components using a spatial filter, and calculates the proportion of the high frequency components as a feature value. The data analysis unit712also make a setting to add a plurality of types of degradation elements such as noise, blur, and aberration, and the amounts of degradation thereof to teacher image data having a higher feature value, i.e., including a higher proportion of high frequency components by priority. The feature analysis technique is not limited thereto. The data analysis unit712may make a setting to add the plurality of types of degradation elements and the amounts of degradation thereof to teacher image data including a specific object by priority.

The degradation addition unit713performs processing similar to that by the degradation addition unit211according to the first exemplary embodiment as many times as the number of degradation elements to be added.

The training-specific priority level determination unit714determines the order in which the plurality of degradation elements added to the student image data and the intensities thereof are estimated, i.e., training-specific priority levels. The order of estimation is determined by processing similar to that by the inference-specific priority level determination unit701.

The training unit715obtains network parameters722to be applied to the CNNs for the degradation restoration training. The training unit715initializes the weights of the CNNs with the network parameters722, and performs degradation restoration training using the training data generated by the degradation addition unit713. The network parameters722include the initial values of the parameters of the CNNs, and hyper parameters indicating the structures of and optimization methods for the CNNs. The degradation restoration training of the training unit715is performed by the training-specific degradation estimation unit716, the training-specific degradation restoration unit717, the error calculation unit718, and the model update unit719.

The training-specific degradation estimation unit716receives the training data from the training data generation unit711, and estimates the plurality of degradation elements included in the student image data based on the priority levels determined by the training-specific priority level determination unit714.

The training-specific degradation restoration unit717receives the student image data and the degradation estimation results estimated by the training-specific degradation estimation unit716, and performs degradation restoration processing corresponding to the plurality of degradation elements included in the student image data.

The error calculation unit718has the same function as that of the error calculation unit215according to the first exemplary embodiment. The model update unit719has the same function as that of the model update unit216according to the first exemplary embodiment.

As described above, a difference of the second exemplary embodiment from the first exemplary embodiment is that a plurality of degradation elements is estimated based on the priority levels determined by the training-specific priority level determination unit714, and degradation restoration processing is performed based on the estimation results.

<Processing Procedure for Entire Information Processing System>

Next, various types of processing performed by the information processing system according to the second exemplary embodiment will be described with reference toFIGS.8A and8B.FIGS.8A and8Bare flowcharts illustrating a processing procedure for the information processing system according to the second exemplary embodiment. The functional units illustrated inFIG.7are implemented by the CPU101or201running computer programs corresponding to the respective functional units. The processing procedure performed by the cloud server710according to the second exemplary embodiment will initially be described with reference to the flowchart ofFIG.8A.

In step S801, the teacher image group720prepared in advance and an analysis result721of the physical properties of the imaging apparatus10are input to the cloud server710. The teacher image data and its uploading are similar to in the foregoing first exemplary embodiment. The data on the teacher image group720and the analysis result721of the physical properties of the imaging apparatus10input to the cloud server710are delivered to the data analysis unit712.

In step S802, the data analysis unit712analyzes features of the teacher image data. For example, if, as a result of the analysis, a piece of teacher image data is found to include a lot of high frequency components, the degradation addition unit713adds degradation elements such as noise, blur, and aberration to the piece of teacher image data at various intensities. In such a manner, various pieces of student image data are generated.

In step S803, the training-specific priority level determination unit714determines the priority levels to estimate the degradation elements included in the student image data.

In step S804, the network parameters to be applied to the CNNs for the degradation restoration training are input to the cloud server710. Like the first exemplary embodiment, the network parameters here include the initial values of the parameters of the CNNs and the hyper parameters indicating the structures of and optimization methods for the CNNs. The input network parameters are delivered to the training unit715.

In step S805, the training-specific degradation estimation unit716estimates the plurality of degradation elements included in the student image data based on the priority levels. The training-specific degradation restoration unit717performs degradation restoration processing based on the estimation results.

In step S806, the error calculation unit718calculates an error between the restoration result and the teacher image data based on the loss function expressed by Eq. (3).

In step S807, the model update unit719updates the network parameters to reduce (minimize) the error obtained in step S806.

In step S808, the training unit715determines whether to end the training. Like the foregoing first exemplary embodiment, the training unit715can determine to end the training if the number of updates of the network parameters has reached a predetermined number. If the training unit715determines to not end the training (NO in step S808), the processing returns to step S805. In the processing of step S805and the subsequent steps, the cloud server710performs training using another pair of student image data and teacher image data.

Next, the processing procedure performed by the edge device700according to the second exemplary embodiment will be described with reference to the flowchart ofFIG.8B.

In step S809, the trained model723trained by the cloud server710and the input image data707to perform the degradation restoration processing on are input to the edge device700. Like the first exemplary embodiment, the input image data707is a raw image. The input image data707is delivered to the inference-specific priority level determination unit701and the specific region extraction unit702. The trained model723is delivered to the inference unit703.

In step S810, the inference-specific priority level determination unit701determines the priority levels to estimate a plurality of degradation elements included in the first image data707.

In step S811, the specific region extraction unit702extracts a specific region or regions from the input image data707. The extraction result is delivered to the intensity adjustment unit705as a specific region map.

In step S812, the inference-specific degradation estimation unit704estimates a plurality of degradation elements included in the input image data707based on the priority levels determined in step S810.

In step S813, the intensity adjustment unit705processes the plurality of degradation estimation results, i.e., adjusts the intensities of restoration.

In step S814, the inference-specific degradation restoration unit706performs the degradation restoration processing on the input image data707based on the degradation estimation results processed in step S813. The image data degradation-restored by the inference-specific degradation restoration unit706is output as the output image data709.

The entire processing procedure performed by the information processing system according to the second exemplary embodiment has been described above. A plurality of image quality degradation elements in the input image data707can thus be estimated and the intensities of restoration can be adjusted region by region of the input image based on the estimation results without changing the neural network configuration of the inference unit703.

In the present exemplary embodiment, the priority levels of the degradation elements to be estimated are determined by the training-specific priority level determination unit714in performing the degradation restoration training. However, the degradation addition unit713may add the degradation elements in the reverse order to that of estimation, and the training-specific priority level determination unit714may be skipped.

The foregoing first and second exemplary embodiments have dealt with an example of extracting specific regions and adjusting the intensity of restoration region by region. However, the degradation restoration inference may be performed with the regions of the input image data other than the specific regions masked in advance. This can provide a result where degradation of the specific regions alone of the input image data is restored.

An exemplary embodiment of the present disclosure can also be implemented by processing for supplying a program for implementing one or more functions of the foregoing exemplary embodiments to a system or an apparatus via a network or a storage medium, and reading and running the program by one or more processors in a computer of the system or apparatus. A circuit for implementing one or more functions (such as an application specific integrated circuit [ASIC]) can also be used for implementation.

All the foregoing exemplary embodiments are just examples of embodiment in implementing the present disclosure, and the technical scope of the present disclosure should not be interpreted as limited to the foregoing exemplary embodiments.

In other words, exemplary embodiments of the present disclosure can be implemented in various forms without departing from the technical concept or essential features of the present disclosure.

According to the exemplary embodiments of the present disclosure, degradation can be reduced in each partial region of an image to be processed.

OTHER EMBODIMENTS

This application claims the benefit of Japanese Patent Application No. 2021-156591, filed Sep. 27, 2021, which is hereby incorporated by reference herein in its entirety.