Patent ID: 11928792
Assignee: XI'AN JIAOTONG UNIVERSITY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A fusion network-based method for image super-resolution and non-uniform motion deblurring, comprising:
step 1: preprocessing an original video to obtain a set of triples of image patches as training and testing data: converting a number of segments of videos captured by a high-speed motion camera into a plurality of triples of image patches {lblur, l, h} that are spatially aligned, where lblur denotes a non-uniform motion-blurred image patch at a low resolution and serves as an input of neural network training; and l and h denote a low-resolution sharp image patch and a high-resolution sharp image patch, respectively, and serve as true values of different branches in the neural network training;
step 2: building a deep neural network: using, by the deep neural network, two branch modules to respectively extract features for image super-resolution and for non-uniform motion deblurring, and performing, by the deep neural network, adaptive fusion on the features extracted by the two branch modules through a feature fusion module that is trainable; and using an upsampling reconstruction module to perform a super-resolution and non-uniform motion deblurring task:
step 3: model training: using the triples of image patches of a training set obtained in the step 1 to update parameters of the built neural network to obtain a deep neural network model for restoring a low-resolution non-uniform motion-blurred image; and
step 4: model testing: preprocessing the low-resolution non-uniform motion-blurred image to be processed, and inputting the low-resolution non-uniform motion-blurred image into the deep neural network model that has been trained, to obtain a high-resolution sharp image,
wherein in the step 2, a module of the neural network for extracting a deblurring feature adopts an encoding-decoding structure, the encoding-decoding structure module comprises a convolutional layer configured to directly process an input, a number of residual network blocks without Batch-Normalization (BN) layers, two downsampling convolutional layers, and two deconvolutional layers; a plurality of skip connections are provided between features of a same scale in the encoding-decoding structure; an output of the encoding-decoding structure module is the deblurring feature Φdeblur; and a pixel regression network composed of two convolutional layers is connected after the encoding-decoding structure to generate a low-resolution sharp image {circumflex over (L)};
wherein in the step 2, a module for extracting a super-resolution feature comprises a convolutional layer configured to directly process an input, and a number of residual network blocks without Batch-Normalization (BN) layers, and an output of the branch module is a super-resolution feature ΦSRF; and
wherein in the step 2, the upsampling reconstruction module takes a feature Φdeblur obtained after fusion as an input, adopts a number of residual network blocks without the BN layers to optimize and refine the fused feature, and achieves a recovery of a high-resolution sharp image Ĥ through two pixel shuffle layers and a number of convolutional layers.