Image processing device, image processing method, and non-transitory computer readable medium for image processing

An image processing device executes image processing by each object of an object group in which plural objects are connected to each other in a directed acyclic graph form. A processing unit performs updating processing and imparting processing, the updating processing for updating image processing which is executed by each object of the object group to partial processing which performs image processing on division image data representing a division image obtained by dividing an input image represented by input image data into plural partial regions, and the imparting processing for imparting a dependency relationship between pieces of the partial processing of the objects connected to each other. A controller performs control for causing plural computation devices to execute, in parallel, the updating processing and the imparting processing by the processing unit and the partial processing which becomes executable based on the dependency relationship.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119 from Japanese Patent Application No. 2016-60526 filed on Mar. 24, 2016

BACKGROUND

Technical Field

The present invention relates to an image processing device, an image processing method, and a non-transitory computer readable medium for image processing.

SUMMARY

According to an aspect of the present invention, there is provided an image processing device that executes image processing by each object of an object group in which plural objects are connected to each other in a directed acyclic graph form. The image processing device includes: a processing unit that performs updating processing and imparting processing, the updating processing for updating image processing which is executed by each object of the object group to partial processing which performs image processing on division image data representing a division image obtained by dividing an input image represented by input image data into plural partial regions, and the imparting processing for imparting a dependency relationship between pieces of the partial processing of the objects connected to each other; and a controller that performs control for causing plural computation devices to execute, in parallel, the updating processing and the imparting processing by the processing unit and the partial processing which becomes executable based on the dependency relationship.

According to an aspect of the present invention, it is possible to improve a processing speed of image processing, as compared with a case where each partial processing is executed after processing of updating each object of an object group in which objects for executing image processing are connected to each other in a DAG form to plural pieces of partial processing is performed for all objects.

DETAILED DESCRIPTION

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the drawings.

First, a configuration of a computer10that functions as an image processing device will be described referring toFIG. 1. The computer10may be a copier, a printer, a facsimile machine, a multifunctional machine having the functions of these devices, or a computer incorporated in an image handling device such as a scanner that performs image processing. In addition, the computer may be an independent computer such as a personal computer (PC), and may be a computer incorporated in a portable device such as a personal digital assistant (PDA) or a mobile phone.

As illustrated inFIG. 1, the computer10according to the present exemplary embodiment includes a computation unit12, a memory14, a display unit16, an operation unit18, a storage unit20, an image data supply unit22, and an image output unit24. In addition, each unit of the computation unit12, the memory14, the display unit16, the operation unit18, the storage unit20, the image data supply unit22, and the image output unit24are connected to each other via a bus26.

The computation unit12according to the present exemplary embodiment is a main processor of the computer10, and is a central processing unit (CPU) including plural processor cores13(hereinafter, referred to as “cores13”) as an example. Each of the cores13is an example of a computation device that executes image processing. In the following description, in a case of distinguishing each of the cores13, as in the core13A and the core13B, an alphabet is added to the end of the reference numeral13.

As described above, in the present exemplary embodiment, although a case where one computation unit12is provided is described, the present invention is not limited thereto. Plural computation units12may be provided. In a case where plural computation units12are provided, the plural computation units12may be the same type of CPUs, or different types of CPUs. In addition, in a case where the plural the computation units12are provided, the plural computation units12may include a graphics processing unit (GPU) or a computation device such as a field programmable gate array (FPGA).

The memory14is nonvolatile storage means for temporarily storing data by the computation unit12.

In a case where the computer10is incorporated in the image handling device, as the display unit16and the operation unit18, for example, a display panel such as a liquid crystal display (LCD), a ten key, and the like, which are provided on the image handling device, may be used. In a case where the computer10is an independent computer, as the display unit16and the operation unit18, for example, a display, a keyboard, a mouse, and the like, which are connected to the computer10, may be used. In addition, the display unit16and the operation unit18may be a touch panel display or the like formed by integrating a touch panel and a display. In addition, as the storage unit20, a nonvolatile storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory may be used.

The image data supply unit22may be any device as long as the image data supply unit supplies image data as a processing target. For example, an image reading unit that reads an image recorded on a recording material such as paper or a photographic film and outputs image data may be used. In addition, as the image data supply unit22, for example, a receiving unit that receives image data from an external device via a communication line and an image storage unit (the memory14or the storage unit20) that stores image data may be used.

The image output unit24may be any device as long as the image output unit outputs image data subjected to image processing or an image represented by image data subjected to image processing. For example, as the image output unit, an image recording unit that records an image represented by image data on a recording material such as paper or a photosensitive material may be used. In addition, as the image output unit24, a display unit (the display unit16) that displays an image represented by image data on a display or the like or a writing device that writes image data on a recording medium such as a compact disk read only memory (CD-ROM) may be used. In addition, as the image output unit24, a transmission unit that transmits image data subjected to image processing to an external device via a communication line may be used. In addition, the image output unit24may be an image storage unit (the memory14or the storage unit20) that stores image data subjected to image processing.

As illustrated inFIG. 1, the storage unit20stores various programs to be executed by the computation unit12. The storage unit20stores programs of an operating system30, as various programs, the programs including a program for resource management, a program for program execution management, a program for communication between the computer10and an external device, and the like. In addition, the storage unit20stores an image processing program group34for causing the computer10to function as an image processing device, as various programs. In addition, the storage unit20stores a various-application-program group32(hereinafter, referred to as an “application32”) for causing the image processing device to perform desired image processing, as various programs.

The image processing program group34includes a program developed for a purpose of reducing a load when developing an image processing program to be executed by the image handling device, the portable device, the PC, and the like. In addition, the image processing program group34includes a program developed so as to be commonly executed on various kinds of devices (platforms) such as the image handling device, the portable device, the PC, and the like.

The image processing device realized by the image processing program group34configures an image processing DAG50A (to be described in detail) that performs image processing instructed by the application32, according to a configuration instruction from the application32. The image processing device executes processing of the image processing DAG50A according to an execution instruction from the application32. In this manner, the image processing program group34provides an interface for the application32, the interface instructing a configuration of the image processing DAG50A that performs desired image processing or instructing an execution of image processing by the configured image processing DAG50A.

With the configuration, even in a case of newly developing a certain device that needs to perform image processing internally, for development of a program that performs the image processing, an application32, which causes the image processing program group34to perform the image processing required in the certain device, may be developed by using the interface. Therefore, a developer does not need to newly develop a program that actually performs image processing, and thus a load of the developer is reduced.

Next, the image processing program group34according to the present exemplary embodiment will be described in detail. As illustrated inFIG. 1, the image processing program group34includes a module library36, a program functioning as a processing configuration unit42, and a program functioning as a processing control unit46.

In the module library36, programs of plural types of image processing modules38that perform predetermined image processing different from each other are registered. Examples of the image processing include, for example, input processing, filter processing, color conversion processing, enlargement processing and reduction processing (denoted as “enlargement/reduction processing” inFIG. 1), skew angle detection processing, image rotation processing, image synthesis processing, output processing, and the like.

In addition, in the module library36, image processing modules38having the same image processing type and different image processing contents to be executed are also registered. InFIG. 1, the image processing modules are distinguished from each other by adding numbers at the ends, as in “module1”, “module2”, and the like. For example, as the image processing module38that performs the enlargement/reduction processing, an image processing module38that performs processing of reducing horizontal and vertical sizes of an image to 50% by thinning-out input image data for each one pixel in each direction of a horizontal direction and a vertical direction, is prepared. Further, for example, as the image processing module38that performs enlargement/reduction processing, an image processing module38that performs enlargement/reduction processing of input image data at a specified enlargement/reduction ratio, is prepared.

In addition, for example, as the image processing module38that performs color conversion processing, an image processing module38that converts an image in a red, green, and blue (RGB) color space into an image in a cyan, magenta, yellow, and key-plate (black) (CMYK) color space, and an image processing module38that converts an image in a CMYK color space into an image in an RGB color space are prepared. Further, for example, as the image processing module38that performs color conversion processing, an image processing module38that converts an image in an RGB color space into an image in an YCbCr color space and an image processing module38that converts an image in an YCbCr color space into an image in an RGB color space are prepared.

In addition, in the module library36, a buffer module40including a storage area (buffer) for storing image data is also registered.

According to an instruction from the application32, the processing configuration unit42according to the present exemplary embodiment configures the image processing DAG50A in a DAG form. As illustrated inFIG. 2A, in the image processing DAG50A, as an example, one or more image processing modules38are connected to each other via the buffer module40disposed in at least one of a pre-stage and a post-stage of each of the image processing modules38.

Each of the image processing modules38is an example of an object that executes image processing on input image data. In addition, the image processing DAG50A is an example of an object group in which plural image processing modules38are connected to each other. In addition, in an example illustrated inFIG. 2A, for the image processing module38connected to the pre-stage image processing module38via the buffer module40, in a case where image processing by the pre-stage image processing module38is completed, image processing by the image processing module38can be executed. In addition, in the example illustrated inFIG. 2A, for the image processing module38connected to the plural pre-stage image processing modules38via the buffer modules40, in a case where all image processing by the plural pre-stage image processing modules38is completed, image processing by the image processing module38can be executed.

In addition,FIG. 2Aillustrates a DAG in which only the modules are connected to each other in order of processing. On the other hand, when the DAG is executed, as illustrated inFIG. 2B, image data stored in the memory is input, and the image processing is performed according to the DAG. A processing result such as image data subjected to the processing is finally stored in the memory14.

Next, a functional configuration of the processing control unit46according to the present exemplary embodiment will be described with reference toFIG. 3. As illustrated inFIG. 3, the processing control unit46includes a processing unit60, a control unit62, a task queue64, and an output unit66.

The processing unit60according to the present exemplary embodiment divides an image represented by a portion as a processing target of the input image data into plural partial regions (hereinafter, referred to as “division images”). As illustrated inFIG. 4, as an example, the processing unit60divides an image G represented by a portion as a processing target of the input image data into plural (in the example illustrated inFIG. 4, three) division images B1to B3. In the following description, image data representing the division image is referred to as “division image data”. In addition, in image processing such as color conversion processing in which the entire input image is a processing target, a portion which is a processing target of the input image data means a portion in which the entire input image data is a processing target. In addition, in image processing such as clipping (trimming) processing in which a portion of the input image is a processing target, a portion which is a processing target of the input image data means a portion of the input image. In the following description, in order to avoid confusion, the portion which is a processing target of the input image data is simply referred to as “input image data”.

In addition, in an example illustrated inFIG. 4, although the processing unit60divides the image G vertically when viewed from the front, the invention is not limited thereto. For example, the processing unit60may divide the image G horizontally when viewed from the front, or divide the image G vertically and horizontally when viewed from the front.

The number of divisions of the input image data by the processing unit60is not particularly limited. For example, the processing unit60may divide the input image data by a predetermined number or size. In addition, for example, the processing unit60may divide the input image data into a number equal to or smaller than the number of processor cores of the computation unit that executes image processing by the image processing module38, and equal to or larger than two.

In addition, for example, the processing unit60may divide the input image data by a size equal to or smaller than a capacity of a cache memory of the computation unit that executes image processing by the image processing module38. In this case, for example, a form in which the processing unit60divides the input image data by a size is exemplified, the size being equal to or smaller than a capacity of a cache memory, which is at the farthest level from the processor of the computation unit that executes image processing by the image processing module38, a so-called last level cache (LLC), and being maximally matched to the capacity of the LLC.

As illustrated inFIG. 5, as an example, the processing unit60divides the image processing which is executed by each image processing module38of the image processing DAG50A into partial processing39corresponding to each of the division image data, and updates the image processing DAG50A to an image processing DAG50B.FIG. 5illustrates the image processing DAG50B in a case where the input image data of the image processing DAG50A illustrated inFIG. 2Ais divided into four pieces of division image data. InFIG. 5, in order to avoid confusion, the buffer module40is not illustrated.

According to the type of the image processing executed by the image processing module38, the processing unit60according to the present exemplary embodiment imparts a dependency relationship between the partial processing39of the image processing module38connected to the pre-stage and the partial processing39of the image processing module38connected to the post-stage. InFIG. 5, the dependency relationship is indicated by a broken-line arrow.

For example, as in color conversion processing, in processing of performing image processing on only pixels as processing targets, the control unit imparts a one-to-one dependency relationship to each partial processing39. On the other hand, for example, as in filter processing, in image processing in which pixels adjacent to the pixels as processing targets are also required, the control unit also imparts a dependency relationship to the pre-stage partial processing39which performs image processing on the adjacent pixels. That is, the dependency relationship is a relationship between the image processing modules38connected to each other, in which the partial processing of the image processing module38connected to the post-stage can be executed in a case where the partial processing39of the image processing module38connected to the pre-stage is completed. Therefore, each partial processing39can be executed in a case where there is no pre-stage partial processing39to which a dependency relationship is imparted, or in a case where all of pre-stage partial processing39to which a dependency relationship is imparted are completed.

Specifically, for example, the partial processing39A and the partial processing39B illustrated inFIG. 5can be executed when starting an execution of image processing. In addition, for example, the partial processing39C illustrated inFIG. 5can be executed when both of the pre-stage partial processing39A and the pre-stage partial processing39B to which a dependency relationship is imparted are completed.

The control unit62according to the present exemplary embodiment performs control for causing the plural cores13to execute, in parallel, updating processing of the image processing DAG50A and imparting processing of the dependency relationship by the processing unit60, and partial processing39that becomes executable based on the dependency relationship. Here, parallel processing means that at least a part of processing of the updating processing and the imparting processing, and the partial processing39that becomes executable based on the dependency relationship is executed in parallel (simultaneously) by the plural cores13. Specifically, the control unit62stores tasks (processing) for executing the updating processing of the image processing DAG50A and the imparting processing of the dependency relationship for each image processing module38, in the task queue64such that each of the plural cores13sequentially reads and executes the stored tasks. Further, during the updating processing and the imparting processing by the processing unit60, the control unit62sequentially stores the partial processing that becomes executable based on the dependency relationship, in the task queue64.

Under the control of the control unit62, the output unit66according to the present exemplary embodiment outputs output image data obtained by executing the image processing by the final-stage image processing module38of the image processing DAG50B. In the present exemplary embodiment, the output unit66displays an output image represented by the obtained output image data, on the display unit16. The output unit66may output (transmit) the output image data to an external device. In addition, in a case where the computer10is incorporated in a printer, the output unit66may output (form) an output image represented by the output image data on a recording material such as paper.

Next, an operation of the computer10according to the present exemplary embodiment will be described with reference toFIGS. 6 to 11.FIG. 6is a flowchart illustrating a flow of processing of a DAG configuration processing program that is executed by the computation unit12in a case where an instruction to start an execution of image processing is input by the application32. In addition, the DAG configuration processing program is installed in the storage unit20in advance. In addition, any task is not executed in the computation unit12, and the core13capable of executing a task executes the DAG configuration processing program. Thus, the program functions as the processing configuration unit42and the control unit62.

In step100ofFIG. 6, the processing configuration unit42configures an image processing DAG50A that performs image processing instructed by the application32, according to a configuration instruction from the application32. In next step102, the control unit62stores tasks for executing updating processing of the image processing DAG50A configured in step100and imparting processing of the dependency relationship for each image processing module38, in the task queue64, and then ends the DAG configuration processing.

As illustrated inFIG. 7, as an example, by the processing of step102, the tasks T for executing the updating processing of the image processing DAG50A and the imparting processing of the dependency relationship for each image processing module38are stored in the task queue64. The core13(in the example illustrated inFIG. 7, the core13C) capable of executing a task reads the task T from the task queue64, and executes the task T. Thus, the DAG updating processing illustrated inFIG. 8is executed.

FIG. 8is a flowchart illustrating a flow of processing of a DAG updating processing program, and the DAG updating processing program is also installed in the storage unit20in advance. In addition, the core13capable of executing a task executes the DAG updating processing program, and thus, the program functions as the processing unit60and the control unit62.

In step110ofFIG. 8, the processing unit60sorts all of the image processing modules38included in the image processing DAG50A, by a known sorting method such as topological sorting. In repetition processing of the following steps112to118, it is assumed that the image processing modules38are sequentially executed one by one in order from the top of all of the image processing modules38sorted in step110as processing targets (hereinafter, referred to as “processing target modules38”).

In step112, as described above, the processing unit divides the input image data into plural pieces of division image data. As described above, the processing unit60updates the image processing which is executed by the processing target module38to the partial processing39corresponding to each of the pieces of division image data. In next step114, the processing unit60determines whether or not an image processing module38is connected to a pre-stage of the processing target module38. In a case where the determination result is Yes, the processing unit60proceeds to step118, and in a case where the determination result is No, the processing unit60proceeds to step116.

In step116, the control unit62sequentially stores the partial processing39of the processing target module38that is divided in step112, in the task queue64. On the other hand, in step118, as described above, according to the type of the image processing which is executed by the processing target module38, the processing unit60imparts a dependency relationship between the partial processing39of the image processing module38connected to the pre-stage and the image processing module38connected to the post-stage.

In step120, the processing unit60determines whether or not the processing of steps112to118is executed for all of the image processing modules38of the image processing DAG50A. In a case where the determination result is No, the processing unit60returns to step112, and in a case where the determination result is Yes, the processing unit60ends the DAG updating processing.

As illustrated inFIG. 9, as an example, by the processing of step116, the partial processing39of the foremost-stage image processing module38is sequentially stored in the task queue64. The cores13(in the example illustrated inFIG. 9, the cores13A and13B) capable of executing tasks read the partial processing39from the task queue64, and execute the partial processing39. On the other hand, for the image processing modules38other than the foremost-stage image processing module, the processing of step112and step116is subsequently executed. That is, as illustrated inFIG. 9, the task T and the partial processing39are executed in parallel by the plural cores13.

On the other hand,FIG. 10is a flowchart illustrating a flow of processing of a partial-processing execution processing program that is executed by the computation unit12in a case where an instruction to start an execution of image processing is input by the application32. In addition, the partial-processing execution processing program is also installed in the storage unit20in advance. In addition, the core13capable of executing a task executes the partial-processing execution processing program, and thus the program functions as the control unit62.

In step130ofFIG. 10, the control unit62determines whether or not the partial processing39is stored in the task queue64. In a case where the determination result is No, the control unit62repeatedly executes the processing of step130, and in a case where the determination result is Yes, the control unit62proceeds to step132.

In step132, the control unit62reads one piece of partial processing39from the top of the task queue64, and executes the partial processing39which is read. In next step134, when the execution of the partial processing39by the processing of step132is completed, the control unit62updates the dependency relationship, and sequentially stores partial processing39which becomes newly executable based on the dependency relationship, in the task queue64.

In next step136, the control unit62determines whether or not all the partial processing39of all the image processing modules38of the image processing DAG50B is completed. In a case where the determination result is No, the control unit62returns to step130, and in a case where the determination result is Yes, the control unit62ends the partial-processing execution processing. In addition, when all the partial processing of all the image processing modules38is completed, the output unit66outputs an output image represented by output image data which is output by the partial processing39of the final-stage image processing module38of the image processing DAG50B, to the display unit16.

As illustrated inFIG. 11, as an example, the partial processing39stored in the task queue64is executed by the cores13(in the example illustrated inFIG. 11, the cores13A and13B). When the execution of the partial processing39is completed, the dependency relationship of the image processing DAG50A is updated, and the partial processing39which becomes newly executable based on the dependency relationship is stored in the task queue64. InFIG. 11, the partial processing39of which the execution is completed is illustrated by a shaded circle, and the partial processing39which becomes newly executable based on the dependency relationship is illustrated by a black circle.

As described above, according to the present exemplary embodiment, the task T and the partial processing39are executed in parallel. Therefore, as illustrated inFIG. 12, as an example, a processing speed of the image processing is improved as compared with a case where the task T and the partial processing39are executed in series. In an upper portion ofFIG. 12, a case where the task T and the partial processing39are executed in series is illustrated, and in a lower portion ofFIG. 12, a case where the task T and the partial processing39are executed in parallel as in the present exemplary embodiment is illustrated.

In addition, as illustrated inFIG. 13, as an example, in a case where the image processing is continuously executed on plural pieces of input image data, even when all the partial processing39on the input image data which is previously input is not completed, execution of the task T and the partial processing39on the input image data which is later input is started. Therefore, in this case, a processing speed of the image processing is further improved as compared with a case where the task T and the partial processing39are executed in series. In an upper portion ofFIG. 13, a case where the task T and the partial processing39are executed in series is illustrated, and in a lower portion ofFIG. 13, a case where the task T and the partial processing39are executed in parallel as in the present exemplary embodiment is illustrated.

Here, in a case where the image processing is continuously executed on the plural pieces of input image data, for example, a higher priority may be imparted to the partial processing39for executing the image processing on the input image data which is previously input than the partial processing39for executing the image processing on the input image data which is later input. In this case, in a case where the plural pieces of partial processing39become executable, the partial processing39may be stored in the task queue64in descending order of priority. Thereby, it is possible to prevent completion of the image processing on the image data which is later input before completion of the image processing on the image data which is previously input.

In addition, in the above-described embodiment, although a mode in which various programs are stored (installed) in the storage unit20in advance is described, the present invention is not limited thereto. The various programs may be provided by being recorded on a recording medium such as a CD-ROM, a digital versatile disk read only memory (DVD-ROM), or a universal serial bus (USB) memory. In addition, various programs may be downloaded from an external device via a network.

While various exemplary embodiments have been described above, these embodiments may be combined with each other as appropriate.

In addition, the present disclosure is not limited to each of the above-described embodiments, and can be freely modified in various forms without departing from the spirit of the present disclosure.

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2016-060526 filed on Mar. 24, 2016, the entire contents of which are incorporated herein by reference.

REFERENCE SIGNS LIST