Patent ID: 12198297

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An aspect of the present invention is to provide an image enlarging apparatus and an image enlarging method having super resolution enlarging mechanism to perform deep learning on the input image based on different image characteristics to generate final output image residual values corresponding to these image characteristics, such that the enlarged image can be enhanced accordingly to accomplish image enlarging having super resolution enlarging mechanism.

Reference is now made toFIG.1.FIG.1illustrates a block diagram of an image enlarging apparatus100having super resolution enlarging mechanism according to an embodiment of the present invention. The image enlarging apparatus100includes a storage circuit110and a processing circuit120.

In an embodiment, the storage circuit110can be such as, but not limited to a CD, a random access memory (RAM), a read only memory (ROM), a floppy disc, a hard drive or an optical disc. The storage circuit110is configured to store a plurality of computer executable commands115.

The processing circuit120is electrically coupled to the storage circuit110. In an embodiment, the processing circuit120is configured to retrieve and execute the computer executable commands115and execute the function of the image enlarging apparatus100. More specifically, the processing circuit120performs super resolution image enlarging on an input image LR having a lower resolution by using deep learning mechanism, to generate an output enlarged image HR. When a size of the input image LR is W×H and an enlargement ratio is n, a size of the output enlarged image HR is nW×nH.

The operation of the image enlarging apparatus100is further described in the following paragraphs in accompany withFIG.2andFIG.3at the same time.

FIG.2illustrates a flow chart of an image enlarging method200having super resolution enlarging mechanism according to an embodiment of the present invention. The image enlarging method200can be used in the image enlarging apparatus100as illustrated inFIG.1, or can be implemented by using other hardware components such as a database, a normal processor, a computer, a server or other unique hardware devices having specific logic circuits or equipments having specific functions, e.g., a unique hardware integrating computer codes and processor/chip.

More specifically, the image enlarging method200can be implemented by computer programs to control the components in the image enlarging apparatus100. The computer programs can be stored in a non-transitory computer readable medium, such as a read-only memory, a flash memory a floppy disc, a hard disc, an optical disc, a flash drive, a magnetic tape, a database accessible from network or other computer readable medium having the same function that is known by those skilled in the art.

FIG.3illustrates a block diagram of a neural network system300implemented based on the operation of the image enlarging apparatus100according to an embodiment of the present invention. More specifically, when the computer executable commands115are executed by the processing circuit120, the computer executable commands115are operated as the neural network system300to execute the image enlarging method200. In other words, modules in the neural network system300inFIG.3can be implemented by the software operated by the processing circuit120. However, the present invention does not exclude the embodiments that use firmware or hardware (such as, but not limited to units of a microprocessor, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof) to replace one or more than one module thereof, e.g., the unique hardware integrating computer codes and processor/chip.

In an embodiment, the neural network system300includes an enlarging module310, neural network module320and an enhancing module330. The neural network module320includes a front-end convolutional path340, a branching convolutional path350A, a branching convolutional path350B, a mixing module360and a weight generating module370.

The image enlarging method200includes steps outlined below (The steps are not recited in the sequence in which the steps are performed. That is, unless the sequence of the steps is expressly indicated, the sequence of the steps is interchangeable, and all or part of the steps may be simultaneously, partially simultaneously, or sequentially performed).

In step S210, the input image LR is received to perform image enlarging thereon to generate an enlarged image ER by the enlarging module310.

In an embodiment, the enlarging module310may use any suitable operation mechanism to perform image enlarging, such as but not limited to perform interpolation operation based on the pixels included in the input image LR. In an embodiment, when the size of the input image LR is W×H and the enlargement ratio is n, the size of the enlarged image ER is nW×nH.

In step S220, the input image LR is received by the front-end convolutional path340included by the neural network module320to perform convolutional operation thereon to generate a front-end operation output result FO.

In an embodiment, the front-end convolutional path340includes a plurality of front-end convolutional units CNN0˜CNN2connected in series. The front-end convolutional unit CNN0corresponds to a head layer and includes a single convolutional layer. Each of the front-end convolutional units CNN1and CNN2corresponds to a residual block and includes one or more than one convolutional layer. The front-end convolutional units CNN0˜CNN2performs convolutional operation on the input image LR in series to generate the front-end operation output result FO.

It is appreciated that the number of the front-end convolutional units illustrated inFIG.3is merely an example. In different embodiments, the number of the front-end convolutional units included in the front-end convolutional path340can be any number that is one or more than one.

In step S230, the front-end operation output result FO is respectively received by the branching convolutional paths included by the neural network module320to perform convolutional operation thereon to generate a plurality of groups of output image residual values.

In the present embodiment, the neural network module320includes two branching convolutional paths, which are a branching convolutional path350A and a branching convolutional path350B. Each of the branching convolutional path350A and the branching convolutional path350B includes a plurality of branching convolutional units connected in series and a pixel reconstruction unit.

The branching convolutional path350A includes branching convolutional units CNA0˜CNAMand a pixel reconstruction unit PSA.

Each of the branching convolutional units CNA2˜CNAM-1corresponds to a residual block and includes one or more than one convolutional layer. The branching convolutional unit CNAMcorresponds to a tail layer and includes a single convolutional layer. The branching convolutional units CNA0˜CNAMperform convolutional operation on the front-end operation output result FO is series to generate a branching operation output result BOA.

In an embodiment, the branching operation output result BOA includes n×n pieces of data having the size of W×H. The pixel reconstruction unit PSA further performs pixel reconstruction on the branching operation output result BOA to generate a piece of data having the size of nW×nH as a group of output image residual values RVA.

The branching convolutional path350B includes branching convolutional units CNB0˜CNB1and a pixel reconstruction unit PSB.

The branching convolutional unit CNB0corresponds to a residual block and includes one or more than one convolutional layer. The branching convolutional unit CNB1corresponds to a tail layer and includes a single convolutional layer. The branching convolutional units CNB0˜CNB1perform convolutional operation on the front-end operation output result FO in series to generate a branching operation output result BOB.

In an embodiment, the branching operation output result BOB includes n× n pieces of data having the size of W×H. The pixel reconstruction unit PSB further performs pixel reconstruction on the branching operation output result BOB to generate a piece of data having the size of nW×nH as a group of output image residual values RVB.

In an embodiment, each of the convolutional units included in the branching convolutional path350A and the branching convolutional path350B performs convolutional operation according to a plurality of groups of convolutional operation parameters. Each of the groups of the convolutional operation parameters corresponds to one of a plurality of image characteristics of the input image RL. The image characteristics can be such as, but not limited to edges, textures or a combination thereof.

For example, the branching convolutional path350A is configured to perform training specifically related to the edges of objects such that the output image residual values RVA enhance the edges of the objects, to make the edges clearer, having lesser noise and smoother. On the contrary, the branching convolutional path350B is configured to perform training specifically related to the textures of the objects such that the output image residual values RVB enhance the textures of the objects, to make the texture more obvious.

It is appreciated that the corresponding relation between the branching convolutional paths and the image characteristics described above is merely an example. In other embodiments, the branching convolutional paths may correspond to other types of image characteristics such that the enhancement of these characteristics can be obtained through the use of deep learning.

Furthermore, the number and the configuration of the branching convolutional paths illustrated inFIG.3are merely an example. In different embodiments, the number of the branching convolutional paths included in the neural network module320can be any number that is two or more than two to perform deep learning specifically related to different image characteristics. Moreover, in different embodiments, the number of the convolutional units included in each of the branching convolutional paths can be any number that is one or more than one.

In step S240, the group of the output image residual values RVA and the group of the output image residual values RVB are weighted according to a weight setting WS related to a plurality of image regions of the input image RL, and mixing is performed thereon to generate a group of final output image residual values RVF by the mixing module360included by the neural network module320.

In an embodiment, the weight setting WS is generated by a weight generating module370included by the neural network module320. More specifically, the weight generating module370is configured to receive the input image LR to determine an image regional characteristic of each of the image regions included in the input image LR corresponding to the image characteristics.

For example, the weight generating module370may include a high-pass filter, a Sobel filter used to perform edge detection, an object edge direction determining unit or a combination thereof to distinguish the object edges and the texture regions in the input image LR. In another example, the weight generating module370may also include a color determining unit, an image segmentation unit or a combination thereof to distinguish different objects, e.g., sky or grass.

Further, the weight generating module370generates a plurality of weights corresponding to the groups of the output image residual values RVA and the groups of the output image residual values RVB according to the image regional characteristic, such that the weights serve as the weight setting WS.

In an example, for the regions that belong to the edges of the objects in the input image LR, the weight generating module370may assign larger weights to the output image residual values RVA. For the regions that correspond to the textures of the objects in the input image LR, the weight generating module370may assign larger weights to the output image residual values RVB.

In another example, the weight generating module370may distinguish the colors and the objects in the input image LR and determines the corresponding image characteristics based on the distinguished objects. For example, the weight generating module370may distinguish the grass and the trees regions and other regions in the input image LR, and enhance the edges of the grass and the trees regions. Under such a condition, for the grass and the trees regions in the input image LR, the weight generating module370assigns larger weights to the output image residual values RVA. For the other regions in the input image LR, the weight generating module370assigns larger weight to the output image residual values RVB.

As a result, the mixing module360can weight the output image residual values RVA and the output image residual values RVB by using the weight setting WS generated based on the image regional characteristic of each of the image regions. The weighted results can be mixed by using operations such as, but not limited to superimposition and/or multiplication to generate the group of final output image residual values RVF. The final output image residual values RVF include a piece of data having the size of nW×nH.

In step S250, the enlarged image ER is enhanced according to the group of the final output image residual values RVF by the enhancing module330to generate an output enlarged image HR. In an embodiment, the enhancing module330is configured to perform operations such as, but not limited to superimposition and/or multiplication on the final output image residual values RVF and the corresponding pixels of the enlarged image ER to generate the output enlarged image HR. The size of the output enlarged image HR is nW×nH.

It is appreciated that the embodiments described above are merely an example. In other embodiments, it should be appreciated that many modifications and changes may be made by those of ordinary skill in the art without departing, from the spirit of the disclosure.

In summary, the image enlarging apparatus and the image enlarging method having super resolution enlarging mechanism of the present invention performs deep learning on the input image based on different image characteristics to generate final output image residual values corresponding to these image characteristics, such that the enlarged image can be enhanced accordingly to accomplish image enlarging having super resolution enlarging mechanism.

The aforementioned descriptions represent merely the preferred embodiments of the present invention, without any intention to limit the scope of the present invention thereto. Various equivalent changes, alterations, or modifications based on the claims of present invention are all consequently viewed as being embraced by the scope of the present invention.