Patent Description:
Image deblurring is an important research direction of image processing, and aims to restore detail information lost in blurry images. With the research advancement of neural network models, image deblurring methods based on an image processing model have achieved better effects than conventional methods. The image processing model is a neural network model used for performing image deblurring on the blurry images to obtain clear images. How to obtain an image processing model with perfect performance through model training is particularly important for the effect of subsequent image deblurring. In existing model training methods, it is generally considered that a blurry image is composed of a plurality of different blurry regions, and convolution model assumptions are applied on the different blurry regions to restore the different blurry regions to clear images in the different regions respectively, to further train the image processing model. Specifically, it is necessary to segment the blurry image into different regions first, then continuously perform two operations of iterative convolution kernel estimation and image deconvolution on the different regions to gradually optimize a deblurring effect of each region, and finally synthesize the regions after deblurring to obtain a complete clear image. D1(XP33687039A) discloses the general principle for constraining the deblurring network structure by proposing the generic and effective selective sharing scheme. Inside the subnetwork of each scale, a nested skip connection structure for the nonlinear transformation modules is proposed to replace stacked convolution layers or residual blocks. Comprehensive experimental results show that our parameter selective sharing scheme, nested skip connection structure, and the new dataset are all significant to set a new state-of-the-art in dynamic scene deblurring. D2(XP33473740A) discloses a Scale-recurrent Network (SRN-DeblurNet) for this deblurring task. Compared with the many recent learning-based approaches, it has a simpler network structure, a smaller number of parameters and is easier to train. Results show that our method can produce better quality results than state-of-the-arts, both quantitatively and qualitatively. D3 (XP55516211A) discloses a very deep fully convolutional encoding-decoding framework for image restoration such as denoising and super-resolution. The network is composed of multiple layers of convolution and de-convolution operators. Symmetrically link convolutional and de-convolutional layers with skip-layer connections, with which the training converges much faster and attains a higher-quality local optimum. First, the skip connections allow the signal to be back-propagated to bottom layers directly, and thus tackles the problem of gradient vanishing, making training deep networks easier and achieving restoration performance gains. Second, these skip connections pass image details from convolutional layers to de-convolutional layers, which is beneficial in recovering the original image. D4(XP33249361A) discloses a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various source. Together, multi-scale loss function that mimics conventional coarse-to-fine approaches is proposed. Furthermore, a new large-scale dataset that provider pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera.

An embodiment of this disclosure provides a training method for an image processing model for processing blurry images, performed by a network device, the image processing model at least including a first network and a second network; the first network and the second network being codec networks with different scales; the sizes of the scales corresponding to the measurements of the sharpness of to-be-processed blurry images; and the method including:.

An embodiment of this disclosure provides a training apparatus for an image processing model for processing blurry images, the image processing model at least including a first network and a second network; the first network and the second network being codec networks with different scales; the sizes of the scales corresponding to the measurements of the sharpness of to-be-processed blurry images; and the apparatus including:.

An embodiment of this disclosure provides a network device, including a processor, and a memory connected to the processor, the memory storing machine-readable instructions, and the machine-readable instructions being executable by the processor to perform the training method for an image processing model and the image processing method according to the embodiments of this disclosure.

An embodiment of this disclosure further provides a computer-readable storage medium, storing a plurality of instructions, the instructions being configured to be loaded by a processor, to perform the training method for an image processing model and the image processing method according to the embodiments of this disclosure.

To describe the technical solutions in the embodiments of this disclosure more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments. Apparently, the accompanying drawings in the following description show only some embodiments of this disclosure, and a person of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.

The following clearly and completely describes technical solutions in embodiments of this disclosure with reference to the accompanying drawings in the embodiments of this disclosure.

The AI technology is a comprehensive discipline, and relates to a wide range of fields including both hardware-level technologies and software-level technologies. The basic AI technologies generally include technologies such as a sensor, a dedicated AI chip, cloud computing, distributed storage, a big data processing technology, an operating/interaction system, and electromechanical integration. AI software technologies mainly include several major directions such as a computer vision technology, an audio processing technology, a natural language processing technology, and machine learning/deep learning.

Currently, deep learning is a technology of machine learning and one of research fields. Artificial intelligence (AI) is implemented in a computer system by building an artificial neural network with a hierarchical structure.

Due to successful application of deep learning (DL) in the field of vision, researchers also introduce the DL to the field of image processing. A deep learning neural network model is trained by using a large quantity of training images to enable the model to perform image processing, for example, process blurry images.

Image blurring is a common problem in image capture. In an example, when a user is in a dynamic scene or a relatively dark environment, a movement of an object in the dynamic scene and/or a movement of a recording camera may cause images obtained by recording to be blurry to various degrees. In another example, when a user record a target object, slight shaking of a hand of the user may also cause images obtained by recording to be blurry to various degrees. When facing a blurry image obtained by recording, the user usually chooses to re-record to obtain a clear image. The blurry image herein refers to an image of which the sharpness is less than a preset threshold, and the clear image refers to an image of which the sharpness is greater than the preset threshold. The sharpness refers to a degree of clearness of detail textures and borders thereof in the image. However, due to various factors such as camera movement, object movement, and hand shaking, the user still cannot obtain a clear image after a plurality of times of re-recording. In addition, in some recording scenes of instantaneous snapshots, a user usually does not have a second chance of recording. For example, in a scene of recording a landscape outside a window on a high-speed moving car/train, or in a scene of recording a fast-moving object in a static scene, the user does not have a chance of re-recording.

In the process of processing a blurry image, because an actual recording scene of the blurry image is extremely complex, including a plurality of factors such as camera movement and object movement in a recording scene, existing model training methods cannot satisfy convolution model assumptions on all blurry regions in movement, resulting in poor image deblurring performance of the image processing model obtained through training. Moreover, model training requires the processing on the blurry image of first segmenting, then calculating respectively, and finally synthesizing, which has a low model training efficiency.

Based on this, an embodiment of this disclosure provides an image processing model for processing blurry images. The image processing model may be used for performing sharpness restoration on a blurry image to obtain a clear image.

The image processing model for processing blurry images provided in one embodiment consistent with the present disclosure may be formed by sequentially connecting at least two networks with different scales in series according to the scales in descending order or in ascending order, and the networks with different scales may perform sharpness restoration on blurry images with different sharpness. The scales of the networks are used for indicating the levels of the sharpness of images inputted to the networks, that is, the sizes of the scales correspond to the measurements of the sharpness of to-be-processed blurry images. A coarse scale represents that an original blurry image is downsampled to a lower resolution to obtain an image with higher sharpness, and a blur degree of the image is relatively low. A fine scale represents that the original blurry image is downsampled to a higher resolution to obtain an image with lower sharpness, and the blur degree of the image is relatively high. Referring to <FIG> (an example in which an image processing model includes three networks is used for description in <FIG>), the image processing model may include such three networks with different scales as a coarse-scale network <NUM>, a middle-scale network <NUM>, and a fine-scale network <NUM>. In the three networks with different scales, the scale of the coarse-scale network <NUM> is the largest, the scale of the middle-scale network <NUM> is the second largest, and the scale of the fine-scale network <NUM> is the smallest. Each network is an codec network, which may specifically include a plurality of feature extraction units <NUM> (as shown in black units in <FIG>), a plurality of feature transformation units <NUM> (as shown in gray units in <FIG>), and a plurality of feature reconstruction units <NUM> (as shown in white units in <FIG>) that have different channel quantities. The channel quantity of each unit may be set according to an empirical value or service requirements, for example, setting to <NUM> channels, <NUM> channels, <NUM> channels, and the like. In some embodiments, each of the feature extraction units <NUM>, the feature transformation units <NUM>, and the feature reconstruction units <NUM> may include one or more convolutional layers, and each convolutional layer may include two convolution kernels of <NUM>×<NUM>. The parameter quantity may be reduced by using the two convolution kernels of <NUM>×<NUM> to improve the speed of model training. <FIG> only schematically represents a structure of the image processing model, and does not limit the structure of the image processing model provided in one embodiment consistent with the present disclosure. In an example, the network quantity in the image processing model is not limited to <NUM> shown in <FIG>, but may alternatively be <NUM>, <NUM>, or the like. In another example, each convolutional layer may alternatively include three convolution kernels of <NUM>×<NUM> or one convolution kernels of <NUM>×<NUM>.

Network parameters of the feature extraction units <NUM> in any two networks are independent, and/or network parameters of the feature reconstruction units <NUM> in any two networks are independent. Network parameters of the feature transformation units <NUM> in any two networks are shared. In some embodiments, the feature transformation unit <NUM> may include at least two residual units, each residual unit may include two or more convolutional layers, and the residual units may be connected to each other by using a multi-order nested skip connection structure. A residual unit may be defined with reference to formula <NUM>: <MAT> where, xn-<NUM>, xn, and Fn respectively represent an input, an output, and a residual function of an nth residual unit; and formula <NUM> may also be referred to as a first-order residual function corresponding to a structure shown in <FIG>. In one embodiment consistent with the present disclosure, assuming that an input of an (n-<NUM>)th residual unit is also generated by another residual function, the input is substituted into formula <NUM> to obtain a second-order residual function shown in formula <NUM> corresponding to a second-order nested skip connection structure shown in a schematic structural diagram on the left side in <FIG>. In addition, it may be learned by comparing schematic structural diagrams on the left and right sides in <FIG> that, compared with directly connecting two residual units in series in the related art, the second-order nested skip connection structure provided in one embodiment consistent with the present disclosure has one more connection.

Formula <NUM> is expanded to further obtain a third-order residual function shown in formula <NUM> corresponding to a third-order nested skip connection structure shown in <FIG>.

Similarly, a multi-order residual function and a corresponding multi-order nested skip connection structure may be obtained. The multi-order nested skip connection structure may be combined into a nested module to be embedded in the feature transformation unit, thereby improving gradient propagation and reducing the complexity of network parameter optimization. Taking an example in which the feature transformation unit <NUM> includes four residual units Fi, each residual unit includes two convolutional layers, and the residual units are connected to each other by using a fourth-order nested skip connection structure, a schematic structural diagram of the feature transformation unit <NUM> may be shown with reference to <FIG> is the schematic diagram of the internal structure of the feature transformation unit <NUM> in <FIG>. <FIG> only schematically represents a structure of the feature transformation unit <NUM>, and does not limit the structure of the feature transformation unit <NUM> provided in one embodiment consistent with the present disclosure. In an example, the quantity of the residual units in the feature transformation unit <NUM> is not limited to <NUM> shown in <FIG>, but may alternatively be <NUM>, <NUM>, or the like. In another example, each residual unit is not limited to only including two convolutional layers, but may alternatively include three, five, or more convolutional layers.

For the foregoing image processing model, an embodiment of this disclosure further provides a model training solution to better train and update the image processing model, optimize the deblurring performance of the image processing model, and improve the efficiency of model training. When the model training solution is used for training and updating the image processing model, a blurry image and a clear image that are paired may be obtained; networks in the image processing model are sequentially called to perform sharpness restoration on the blurry image for training to obtain a restored image; and then network parameters of the networks in the image processing model are updated according to the restored image and the clear image. When each network performs sharpness restoration on an image, a plurality of encoding stages and a plurality of decoding stages may be included (three encoding stages and three decoding stages are used as an example for description in <FIG>). In each encoding stage, the feature extraction unit <NUM> may be first called to perform feature extraction on a received image, and then the feature transformation unit <NUM> is called to perform feature transformation on an image obtained after the feature extraction. In each decoding stage, the feature transformation unit <NUM> may be first called to perform feature transformation on a received image, and then the feature reconstruction unit <NUM> is called to perform feature reconstruction on an image obtained after the feature transformation.

Based on the foregoing description, the embodiments of this disclosure provide a training method and apparatus for an image processing model for processing blurry images, a network device, and a storage medium.

The training apparatus for an image processing model may be specifically integrated into the network device such as a terminal or a server. The terminal herein may include, but is not limited to: a smart terminal, a tablet computer, a laptop computer, a desktop computer, or the like. For example, referring to <FIG>, a network device <NUM> may obtain a sample pair for training, the sample pair including a clear image and a blurry image corresponding to the clear image; and the sharpness of the clear image being greater than a preset threshold, and the sharpness of the blurry image being less than the preset threshold; call the image processing model to perform sharpness restoration on the blurry image to obtain a restored image; and update network parameters of the first network and/or network parameters of the second network in the image processing model according to the restored image and the clear image to obtain a trained image processing model.

A training method for an image processing model for processing blurry images provided in an embodiment of this disclosure may be performed by a network device. Referring to <FIG>, the training method for an image processing model for processing blurry images may include the following steps S301 to S303:
S301. Obtain a sample pair for training, the sample pair including a clear image and a blurry image corresponding to the clear image.

When the sample pair for training is obtained, the clear image and the blurry image corresponding to the clear image may be obtained in a data-driven manner. The so-called data-driven manner refers to a manner of blurring a dynamic scene by superimposing a plurality of consecutive frames of images captured by a camera to obtain the blurry image and the clear image in the dynamic scene. The sharpness of the clear image is greater than a preset threshold, and the sharpness of the blurry image is less than the preset threshold. The preset threshold herein may be set according to an empirical value or actual service requirements (for example, the requirement on the accuracy of deblurring performance of the image processing model). The clear image and the blurry image that are paired are obtained in the data-driven manner, which may reduce the acquisition difficulty of the sample pair.

Call the image processing model to perform sharpness restoration on the blurry image to obtain a restored image.

In one embodiment consistent with the present disclosure, the image processing model at least includes a first network and a second network; and the first network and the second network are codec networks with different scales, the first network corresponds to a first scale, and the second network corresponds to a second scale. Values of the first scale and the second scale are different, and the value of the first scale may be greater than the value of the second scale, that is, the first scale may be a coarse scale, and the second scale may be a fine scale. When the image processing model is called to perform sharpness restoration on the blurry image, the first network and the second network may be sequentially called to perform sharpness restoration on the blurry image to obtain the restored image. If the image processing model further includes other networks such as a third network and a fourth network, the first network, the second network, and the other networks may be called to perform sharpness restoration on the blurry image.

The sharpness restoration herein refers to the processing of improving the sharpness of the image. <FIG> is a specific flowchart of sharpness restoration according to an embodiment of this disclosure. In some embodiments, as shown in <FIG>, the following steps S321 to S323 are specifically included:
S321. Perform feature extraction on an image.

Specifically, a plurality of convolution operations may be performed on the image to implement the feature extraction on the image, or a feature extraction algorithm may be used to perform the feature extraction on the image. The feature extraction algorithm herein may include, but is not limited to: a local binary patterns (LBP) algorithm, a histogram of oriented gradient (HOG) feature extraction algorithm, a speeded up robust features (SURF) algorithm, or the like.

Perform, by using a multi-order residual function, feature transformation on an image obtained after the feature extraction.

The multi-order residual function herein refers to a residual function of which an order is greater than or equal to <NUM>.

Perform feature reconstruction on an image obtained after the feature transformation.

Specifically, a plurality of deconvolution operations may be performed on the image obtained after the feature transformation to implement the feature reconstruction on the image obtained after the feature transformation.

Update network parameters of the first network and/or network parameters of the second network in the image processing model according to the restored image and the clear image to obtain a trained image processing model.

Through the study of blurry images (images shown on the left side in <FIG>) captured in a dynamic scene, it is found that, in the images shown on the left side in <FIG>, an image of a building part in a background region is relatively clear, while an image of a crowd part in a foreground region is relatively blurry. A blurry image region <NUM> in the foreground region and a clear image region <NUM> in the background region are randomly selected, and the two selected image regions are analyzed in an image pyramid. For an analysis result thereof, refer to the right side in <FIG>. According to the analysis result shown on the right side in <FIG>, it may be learned that, after the image region <NUM> in the background region is downsampled, edges of the image thereof are still clear after downsampling; and after the image region <NUM> in the foreground region is downsampled, edges of the image thereof become increasingly clear after downsampling. If the same feature extraction parameters are allocated to the networks with different scales in the image processing model, the image processing model cannot both extract clear image features and blurry image features. Therefore, in an embodiment of this disclosure, different feature extraction parameters are allocated to feature extraction units of the networks with different scales, which enables the networks with different scales to learn important image information at the different scales, so as to extract more image features at the different scales.

Because feature transformation functions of the feature transformation units in the networks with different scales are similar, and all aim to transform corresponding blurry image features into clear image features, in an embodiment of this disclosure, the same feature transformation parameters are allocated to feature transformation units of the networks with different scales, as shown in <FIG>. Three rows in <FIG> represent the three networks at the coarse scale to the fine scale respectively from top to bottom. FE represents a feature extraction unit, T represents a feature transformation unit, and the same background represents the same parameters. Further, because functions of the feature transformation at different scales and the same scale are similar, the same feature transformation parameters may be further allocated to the feature transformation units in the networks with the same scale, as shown in <FIG>. <FIG> and <FIG> only schematically represent encoding stages of the networks, and decoding stages thereof are not shown in <FIG> and <FIG>.

Based on the foregoing description, for the first network and the second network in the image processing model, the network parameters of the first network and the network parameters of the second network meeting a selective sharing condition may be set, and the selective sharing condition is used for indicating shared network parameters between the first network and the second network, and is used for indicating independent network parameters between the first network and the second network. Specifically, the network parameters include a feature extraction parameter and a feature transformation parameter; the selective sharing condition, in a case of being used for indicating the shared network parameters between the first network and the second network, is specifically used for indicating that the feature transformation parameter of the first network and the feature transformation parameter of the second network are the shared network parameters, that is, the feature transformation parameter of the first network and the feature transformation parameter of the second network are the same network parameter; and the selective sharing condition, in a case of being used for indicating the independent network parameters between the first network and the second network, is specifically used for indicating that the feature extraction parameter of the first network and the feature extraction parameter of the second network are the independent network parameters, that is, the feature extraction parameter of the first network and the feature extraction parameter of the second network are different network parameters. In some embodiments, the network parameters further include a feature reconstruction parameter; and the selective sharing condition, in a case of being used for indicating the independent network parameters between the first network and the second network, is further used for indicating that the feature reconstruction parameter of the first network and the feature reconstruction parameter of the second network are the independent network parameters, that is, the feature reconstruction parameter of the first network and the feature reconstruction parameter of the second network are different network parameters.

The selective sharing condition being specifically used for indicating that the feature transformation parameter of the first network and the feature transformation parameter of the second network are the shared network parameters may include the following two implementations: (<NUM>) in a case that the quantity of the feature transformation parameter is greater than <NUM>, a plurality of feature transformation parameters of the first network and a plurality of feature transformation parameters of the second network are the shared network parameters, and each of the feature transformation parameters of the first network is an independent network parameter and each of the feature transformation parameters of the second network is an independent network parameter, as shown in the image on the right side in <FIG>; and (<FIG>) in a case that the quantity of the feature transformation parameter is greater than <NUM>, a plurality of feature transformation parameters of the first network and a plurality of feature transformation parameters of the second network are the shared network parameters, and each of the feature transformation parameters of the first network is a shared network parameter and each of the feature transformation parameters of the second network is a shared network parameter, as shown in the image on the right side in <FIG>.

In one embodiment consistent with the present disclosure, the image processing model at least includes the first network with the first scale and the second network with the second scale. Because there are shared network parameters and independent network parameters between the first network and the second network, when performing sharpness restoration on the blurry image, the image processing model can learn more image features in the blurry image to obtain a more accurate restored image. The network parameters of the first network and/or the network parameters of the second network are updated according to the more accurate restored image and the clear image, which may improve the deblurring performance of the trained image processing model. In addition, because there are shared network parameters between the first network and the second network, the quantity of parameters of the image processing model may be reduced, and the efficiency of model training is improved. Moreover, by using the corresponding clear image and blurry image to perform end-to-end training and learning on the image processing model, there is no need to segment the blurry image into blurry regions in movement, and there is no need to make any assumption on the blurry image, which may further improve the deblurring performance of the trained image processing model and the efficiency of model training.

<FIG> is a schematic flowchart of another training method for an image processing model for processing blurry images according to an embodiment of this disclosure. The training method for an image processing model may be performed by a network device. Referring to <FIG>, the training method for an image processing model may include the following steps S601 to S605:
S601. Obtain a sample pair for training, the sample pair including a clear image and a blurry image corresponding to the clear image.

The network device may obtain a large quantity of sample pairs and perform a subsequent model training update operation on the image processing model by using the large quantity of sample pairs. In one embodiment, because the production of a blurry image is usually caused by camera movement during recording or object movement in a recording scene, and is essentially because a shutter speed of a camera is not fast enough. As a result, the camera movement or the object movement in the recording scene causes a sensor of the camera to acquire not only the luminance at a certain fixed location, but also an integral of all luminance at related locations within a period of time in which a shutter is enabled and then disabled, resulting in image blurring. However, studies show that the integral of all luminance at related locations in consecutive frames of images captured by the camera may be approximately the summation of adjacent consecutive images.

<FIG> is a flowchart of a method for obtaining a sample pair for training according to an embodiment of this disclosure. As shown in <FIG>, the method for obtaining a sample pair for training may specifically include the following steps S611 to S613:
Step S611. Obtain image sequence frames for training.

In some embodiments, the image sequence frames may be obtained by acquiring, by using an action camera (for example, a GoPro high-speed camera) and a high-speed mode of a network device, a large number of videos, and performing image frame analysis on the acquired videos. The videos may be high-speed videos at <NUM> frames per second, high-speed videos at <NUM> frames per second, or the like.

Randomly select one frame of image from the image sequence frames as a clear image, and determine a plurality of frames of reference images associated with the clear image.

In some embodiments, the reference image being associated with the clear image refers to: a difference between a frame sequence number of the reference image and a frame sequence number of the clear image being less than a preset difference. For example, a frame sequence number of the clear image is <NUM>, that is, the clear image is a <NUM>th frame of image in the image sequence frames. If the preset difference is <NUM>, a <NUM>rd frame of image, a <NUM>th frame of image, a <NUM>th frame of image, and a <NUM>th frame of image in the image sequence frames may all be used as the reference images.

Obtain a blurry image corresponding to the clear image according to the plurality of frames of reference images, and construct, by using the blurry image and the clear image, the sample pair for training.

In some embodiments, a specific implementation of the obtaining a blurry image corresponding to the clear image according to the plurality of frames of reference images may be: superimposing and averaging the plurality of frames of reference images to obtain the blurry image corresponding to the clear image.

Call the image processing model to perform sharpness restoration on the blurry image to obtain a restored image.

In one embodiment consistent with the present disclosure, the image processing model at least includes a first network and a second network; the first network corresponds to a first scale, and the second network corresponds to a second scale; and values of the first scale and the second scale are different. As can be learned from the foregoing, the networks with different scales may perform sharpness restoration on blurry images with different sharpness.

<FIG> is a flowchart of a method for calling an image processing model to perform sharpness restoration on a blurry image in step S602 according to an embodiment of this disclosure. As shown in <FIG>, the method includes the following steps S621 to S624:.

Downsample the blurry image according to the first scale to obtain a blurry image with first sharpness.

Call the first network to perform sharpness restoration on the blurry image with the first sharpness to obtain an intermediate image.

In some embodiments, the first network may perform sharpness restoration on the blurry image with the first sharpness by using formula <NUM>. <MAT> where, Net<NUM> is a function used by the first network to perform sharpness restoration, B<NUM> represents the blurry image with the first sharpness inputted to the first network, θ<NUM> represents a network parameter, independent of the second network, in the first network, η represents a network parameter shared between the first network and the second network, and I<NUM> represents the intermediate image outputted by the first network.

Downsample the blurry image according to the second scale to obtain a blurry image with second sharpness.

Call the second network to perform sharpness restoration according to the blurry image with the second sharpness and the intermediate image to obtain a restored image.

In some embodiments, the second network may perform sharpness restoration on the blurry image with the second sharpness and the intermediate image by using formula <NUM>. <MAT> where, Net<NUM> is a function used by the second network to perform sharpness restoration, B<NUM> represents the blurry image with the second sharpness inputted to the second network, I<NUM> represents the intermediate image outputted by the first network, θ<NUM> represents a network parameter, independent of the first network, in the second network, η represents a network parameter shared between the first network and the second network, and I<NUM> represents the restored image outputted by the second network.

When the image processing model includes at least three networks, the networks may be sequentially called to perform sharpness restoration on the blurry image according to a connection order of the image processing model. A first network in the image processing model may perform sharpness restoration by using formula <NUM>, any network other than the first network in the image processing model may perform sharpness restoration by using formula <NUM>, and an image obtained by performing sharpness restoration by a last network is a restored image. <MAT> where, Neti is a function used by an ith network to perform sharpness restoration, Bi represents a blurry image with ith sharpness inputted to the ith network, θ<NUM> represents a network parameter, independent of other networks with different scales, in the ith network, η represents a network parameter shared between the networks, Ii-<NUM> represents an intermediate image outputted by an (i-<NUM>)th network, and Ii represents an intermediate image outputted by the ith network.

Obtain an optimization function of the image processing model.

Determine a value of the optimization function according to the restored image and the clear image.

Update, by reducing the value of the optimization function, the network parameters of the first network and/or the network parameters of the second network in the image processing model, to obtain a trained image processing model.

The network device may perform the foregoing steps S601 and S602 to obtain restored images and clear images of a large quantity of sample pairs, and perform steps S603 to S605 after obtaining a large quantity of pairs of restored images and clear images. In steps S603 to S605, the optimization function of the image processing model may be shown as formula <NUM>: <MAT> where, N represents the quantity of the sample pairs, <MAT> and <MAT> respectively represent the blurry image and the clear image in a kth sample pair at scale i, S represents the total quantity of scales in the image processing model, θi represents an independent network parameter in a network corresponding to the scale i, η represents a shared network parameter,Ti represents the total quantity of pixels of the image at the scale i, and Fi represents a function for performing sharpness restoration on the blurry image <MAT>.

After the optimization function is obtained, the restored image and the clear image may be substituted into the optimization function to determine the value of the optimization function, and then the network parameters of the first network and/or the network parameters of the second network in the image processing model are continuously updated according to the principle of reducing the value of the optimization function, until the value of the optimization function is minimized, and the image processing model is in a converged state. The image processing model may further include other networks different from the first network and the second network. Then, after the value of the optimization function is determined, network parameters of the other networks in the image processing model may be further continuously updated by reducing the value of the optimization function.

In one embodiment consistent with the present disclosure, the image processing model for processing blurry images at least includes the first network with the first scale and the second network with the second scale. Because there are shared network parameters and independent network parameters between the first network and the second network, when performing sharpness restoration on the blurry image, the image processing model can learn more image features in the blurry image to obtain a more accurate restored image. The network parameters of the first network and/or the network parameters of the second network are updated according to the more accurate restored image and the clear image, which may improve the deblurring performance of the trained image processing model. In addition, because there are shared network parameters between the first network and the second network, the quantity of parameters of the image processing model may be reduced, and the efficiency of model training is improved. Moreover, by using the corresponding clear image and blurry image to perform end-to-end training and learning on the image processing model, there is no need to segment the blurry image into blurry regions in movement, and there is no need to make any assumption on the blurry image, which may further improve the deblurring performance of the trained image processing model and the efficiency of model training.

Based on the foregoing related description of the image processing model, an embodiment of this disclosure further provides an image processing method, and the image processing method may be performed by the network device in <FIG>. Referring to <FIG>, the image processing method may include the following steps S701 to S703:
S701. Obtain a to-be-processed original image.

The sharpness of the original image is less than a preset threshold. A to-be-processed original image may be obtained using the following two methods:.

In one embodiment, after the network device detects that the user uses the camera assembly of the network device to capture an image, the image captured by the camera assembly may be obtained, and the captured image is displayed in a user interface, for the user to view. If finding that the captured image is not clear and the sharpness thereof is less than the preset threshold, the user may input an image processing instruction to the network device. If the network device receives the image processing instruction, the captured image may be used as the to-be-processed original image. In another embodiment, if the user finds that some historical images in an image gallery of the network device are blurry and the sharpness thereof is less than the preset threshold, the user may also input an image processing instruction to the network device to trigger the network device to obtain the historical images as the to-be-processed original images. The foregoing image processing instruction may be an instruction generated by the user by clicking or pressing an image, or may be an instruction generated by the user by pressing a designated key on the network device, or may be an instruction generated by the user by inputting voice to the network device, or the like.

Call the image processing model to perform sharpness restoration on the original image to obtain a target image.

The sharpness of the target image is greater than the preset threshold. The sharpness restoration includes: performing feature extraction on an image, performing, by using a multi-order residual function, feature transformation on an image obtained after the feature extraction, and performing feature reconstruction on an image obtained after the feature transformation. Correspondingly, in a specific implementation process of step S702, the image processing model may be called to first perform feature extraction on the original image to obtain a first image obtained after the feature extraction; feature transformation is then performed on the first image by using a multi-order residual function to obtain a second image obtained after the feature transformation; and finally feature reconstruction is performed on the second image to obtain the target image.

The image processing model herein may be obtained by training by using the training method for an image processing model shown in <FIG> or <FIG>. The image processing model at least includes a first network and a second network; the first network corresponds to a first scale, and the second network corresponds to a second scale; and network parameters of the first network and network parameters of the second network meet a selective sharing condition, and the selective sharing condition is used for indicating shared network parameters between the first network and the second network, and is used for indicating independent network parameters between the first network and the second network. In one embodiment, the network parameters include a feature extraction parameter and a feature transformation parameter. Correspondingly, the selective sharing condition, in a case of being used for indicating the shared network parameters between the first network and the second network, is specifically used for indicating that the feature transformation parameter of the first network and the feature transformation parameter of the second network are the shared network parameters; and the selective sharing condition, in a case of being used for indicating the independent network parameters between the first network and the second network, is specifically used for indicating that the feature extraction parameter of the first network and the feature extraction parameter of the second network are the independent network parameters. In another embodiment, the network parameters further include a feature reconstruction parameter. Correspondingly, the selective sharing condition, in a case of being used for indicating the independent network parameters between the first network and the second network, is further used for indicating that the feature reconstruction parameter of the first network and the feature reconstruction parameter of the second network are the independent network parameters.

In one embodiment consistent with the present disclosure, because the image processing model is obtained by training by using the training method for an image processing model shown in <FIG> or <FIG>, the deblurring performance of the image processing model is good. Therefore, by calling the image processing model to perform sharpness restoration on the original image with low sharpness, the original image may be better deblurred to obtain a relatively clear target image, which may improve the sharpness of the target image and further improve the image quality of the target image.

Based on the foregoing description of the embodiment of the training method for an image processing model, an embodiment of this disclosure further discloses a training apparatus for an image processing model for processing blurry images. The image processing model at least includes a first network and a second network; the first network and the second network are codec networks with different scales; the sizes of the scales correspond to the measurements of the sharpness of to-be-processed blurry images; and the training apparatus for an image processing model may be a computer program (including program code) run on a network device. The training apparatus for an image processing model may perform the method shown in <FIG> or <FIG>. Referring to <FIG>, the training apparatus for an image processing model may operate the following units:.

In one embodiment, the network parameters include a feature extraction parameter and a feature transformation parameter;.

In another embodiment, the network parameters further include a feature reconstruction parameter; and
the selective sharing condition, in a case of being used for indicating the independent network parameters between the first network and the second network, is further used for indicating that the feature reconstruction parameter of the first network and the feature reconstruction parameter of the second network are the independent network parameters.

In another embodiment, the selective sharing condition being specifically used for indicating that the feature transformation parameter of the first network and the feature transformation parameter of the second network are the shared network parameters includes:.

In another embodiment, the first network corresponds to a first scale, and the second network corresponds to a second scale; and the processing unit <NUM>, when being configured to call the image processing model to perform sharpness restoration on the blurry image to obtain a restored image, is specifically configured to:.

In another embodiment, the sharpness restoration includes: performing feature extraction on an image, performing, by using a multi-order residual function, feature transformation on an image obtained after the feature extraction, and performing feature reconstruction on an image obtained after the feature transformation.

In another embodiment, the update unit <NUM>, when being configured to update network parameters of the first network and/or network parameters of the second network in the image processing model according to the restored image and the clear image, is specifically configured to:.

In another embodiment, the obtaining unit <NUM>, when being configured to obtain a sample pair for training, is specifically configured to:.

In another embodiment, the obtaining unit <NUM>, when being configured to obtain a blurry image corresponding to the clear image according to the plurality of frames of reference images, is specifically configured to:
superimpose and average the plurality of frames of reference images to obtain the blurry image corresponding to the clear image.

According to an embodiment of this disclosure, the steps in the method shown in <FIG> or <FIG> may be performed by the units of the training apparatus for an image processing model shown in <FIG>. In one example, steps S301 to S303 shown in <FIG> may be respectively performed by the obtaining unit <NUM>, the processing unit <NUM>, and the update unit <NUM> shown in <FIG>. In another example, steps S601 and S602 shown in <FIG> may be respectively performed by the obtaining unit <NUM> and the processing unit <NUM> shown in <FIG>, and steps S603 to S605 may be performed by the update unit <NUM> shown in <FIG>.

According to another embodiment of this disclosure, the units of the training apparatus for an image processing model for processing blurry images shown in <FIG> may be separately or wholly combined into one or several other units, or one (or more) of the units herein may further be divided into a plurality of units of smaller functions. In this way, same operations may be implemented, and the implementation of the technical effects of the embodiments of this disclosure is not affected. The foregoing units are divided based on logical functions. In an actual application, a function of one unit may also be implemented by a plurality of units, or functions of a plurality of units are implemented by one unit. In other embodiments of this disclosure, the training apparatus for an image processing model may also include other units. In an actual application, the functions may also be cooperatively implemented by other units and may be cooperatively implemented by a plurality of units.

According to another embodiment of this disclosure, a computer program (including program code) that can perform the steps in the corresponding method shown in <FIG> or <FIG> may be run on a general computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), to construct the training apparatus for an image processing model shown in <FIG> and implement the training method for an image processing model in the embodiments of this disclosure. The computer program may be recorded on, for example, a computer-readable recording medium, and may be loaded into the foregoing computing device by using the computer-readable recording medium and run on the computing device.

Based on the foregoing description of the embodiment of the image processing method, an embodiment of this disclosure further discloses an image processing apparatus. The image processing apparatus may be a computer program (including program code) run on a network device. The image processing apparatus may perform the method shown in <FIG>. Referring to <FIG>, the image processing apparatus may operate the following units:.

According to an embodiment of this disclosure, the steps in the method shown in <FIG> may be performed by the units of the image processing apparatus shown in <FIG>. Specifically, steps S701 to S703 shown in <FIG> may be respectively performed by the obtaining unit <NUM>, the processing unit <NUM>, and the output unit <NUM> shown in <FIG>. According to another embodiment of this disclosure, the units of the image processing apparatus shown in <FIG> may be separately or wholly combined into one or several other units, or one (or more) of the units herein may further be divided into a plurality of units of smaller functions. In this way, same operations may be implemented, and the implementation of the technical effects of the embodiments of this disclosure is not affected. The foregoing units are divided based on logical functions. In an actual application, a function of one unit may also be implemented by a plurality of units, or functions of a plurality of units are implemented by one unit. In other embodiments of this disclosure, the image processing apparatus may also include other units. In an actual application, the functions may also be cooperatively implemented by other units and may be cooperatively implemented by a plurality of units. According to another embodiment of this disclosure, a computer program (including program code) that can perform the steps in the corresponding method shown in <FIG> may be run on a general computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), to construct the image processing apparatus shown in <FIG> and implement the image processing method in the embodiments of this disclosure. The computer program may be recorded on, for example, a computer-readable recording medium, and may be loaded into the foregoing computing device by using the computer-readable recording medium and run on the computing device.

Based on the descriptions of the foregoing method embodiments and apparatus embodiments, an embodiment of this disclosure further provides a network device. Referring to <FIG>, the network device at least includes a processor <NUM>, an input device <NUM>, an output device <NUM>, and a computer storage medium <NUM>. The input device <NUM> may further include a camera assembly, and the camera assembly may be configured to acquire images. The camera assembly may be an assembly configured on the network device when the network device leaves the factory, or may be an external assembly connected to the network device. In some embodiments, the network device may be further connected to other devices to receive images transmitted by the other devices.

The computer storage medium <NUM> may be stored in a memory of the network device. The computer storage medium <NUM> is configured to store a computer program. The computer program includes program instructions. The processor <NUM> is configured to execute the program instructions stored in the computer storage medium <NUM>. The processor <NUM> (or referred to as a central processing unit (CPU)) is a computing core and a control core of the network device, is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement a corresponding method procedure or a corresponding function. In an embodiment, the processor <NUM> in the embodiments of this disclosure may be configured to perform a series of training on the image processing model for processing blurry images, including: obtaining a sample pair for training, the sample pair including a clear image and a blurry image corresponding to the clear image; and the sharpness of the clear image being greater than a preset threshold, and the sharpness of the blurry image being less than the preset threshold; calling the image processing model to perform sharpness restoration on the blurry image to obtain a restored image; and updating network parameters of the first network and/or network parameters of the second network in the image processing model according to the restored image and the clear image; the network parameters of the first network and the network parameters of the second network meeting a selective sharing condition, and the selective sharing condition being used for indicating shared network parameters between the first network and the second network, and being used for indicating independent network parameters between the first network and the second network. In another embodiment, the processor <NUM> in the embodiments of this disclosure may be further configured to perform a series of image processing on the original image, including: obtaining a to-be-processed original image, the sharpness of the original image being less than a preset threshold; and calling the image processing model to perform sharpness restoration on the original image to obtain a target image, the sharpness of the target image being greater than the preset threshold.

Claim 1:
A training method for an image processing model for processing blurry images, the image processing model at least comprising a first network and a second network; the first network and the second network being codec networks with different scales; the sizes of the scales corresponding to the measurements of the sharpness of to-be-processed blurry images; and the method comprising:
obtaining (S301) a sample pair for training, the sample pair comprising a clear image and a blurry image corresponding to the clear image; and the sharpness of the clear image being greater than a preset threshold, and the sharpness of the blurry image being less than the preset threshold;
calling (S302) the image processing model to perform sharpness restoration on the blurry image to obtain a restored image; and
updating (S303) network parameters of the first network and/or network parameters of the second network in the image processing model according to the restored image and the clear image to obtain a trained image processing model;
wherein the network parameters of the first network and the network parameters of the second network meet a selective sharing condition, and the selective sharing condition indicates the network parameters between the first network and the second network are shared or independent;
wherein the network parameters comprise a feature extraction parameter and a feature transformation parameter;
the selective sharing condition, when indicating the network parameters between the first network and the second network are shared, indicates that the feature transformation parameter of the first network and the feature transformation parameter of the second network are the shared network parameters; and
the selective sharing condition, when indicating the network parameters between the first network and the second network are independent, indicates that the feature extraction parameter of the first network and the feature extraction parameter of the second network are the independent network parameters.