Image processing apparatus, and non-transitory computer readable medium for reducing undesirable effect during image capturing

An image processing apparatus includes a receiving unit that receives a first image that is captured with an object irradiated with light from a light irradiating unit and a second image that is captured with the object not irradiated with light from the light irradiating unit, and a generating unit that generates a composite image by adjusting brightness of a whole area of the second image such that brightness at a specific location in the first image is approximately equal to brightness at a corresponding specific location in the second image, and by replacing in the first image a pixel value of a pixel that is affected by light irradiation by the light irradiating unit with a pixel value of a pixel in the second image that has been adjusted in brightness.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2018-177197 filed Sep. 21, 2018.

BACKGROUND

(i) Technical Field

The present disclosure relates to an image processing apparatus, and a non-transitory computer readable medium.

(ii) Related Art

Japanese Unexamined Patent Application Publication No. 2010-74693 discloses a digital camera. The digital camera includes an electronic flash device that irradiates an object with flash light, an imaging device that consecutively acquires a first image that is captured with the object not irradiated with flash light, and a second image that is captured with the object irradiated with flash light, an image correlation detection circuit that detects an image portion of each image having no correlation by comparing the first image with the second image, and a correction image generating circuit that generates a third image by extracting from the second image the image portion having no correlation, removes from the first image the image portion having no correlation, and embedding the image portion extracted from the second image into the corresponding portion of the first image from which the image portion having no correlation has been removed.

SUMMARY

Aspects of non-limiting embodiments of the present disclosure relate to an image processing apparatus and a non-transitory computer readable medium. The image processing apparatus acquires an image that is captured in a more reduced effect that is caused with a light irradiating unit irradiating an object with light in comparison with an image of the object that is captured with the light irradiation unit irradiating the object with light at one time only.

According to an aspect of the present disclosure, there is provided an image processing apparatus. The image processing apparatus includes a receiving unit that receives a first image that is captured with an object irradiated with light from a light irradiating unit and a second image that is captured with the object not irradiated with light from the light irradiating unit, and a generating unit that generates a composite image by adjusting brightness of a whole area of the second image such that brightness at a specific location in the first image is approximately equal to brightness at a corresponding specific location in the second image, and by replacing in the first image a pixel value of a pixel that is affected by light irradiation by the light irradiating unit with a pixel value of a pixel in the second image that has been adjusted in brightness.

DETAILED DESCRIPTION

An image processing apparatus20of an exemplary embodiment of the disclosure is described with reference toFIGS. 1A and 1B.FIG. 1Ais a perspective external view of the image processing apparatus20of the exemplary embodiment, andFIG. 1Bis a perspective view illustrating how an object30is photographed using the image processing apparatus20. Referring toFIGS. 1A and 1B, the image processing apparatus20of the exemplary embodiment is a tablet computer. The disclosure is not limited to the tablet computer. For example, the image processing apparatus20may be a smart phone or a digital camera as long as the configuration thereof is the one described below. The image processing apparatus20may be a terminal device, such as a notebook computer with a camera. A housing200of the image processing apparatus20includes on the rear surface thereof a camera206, and a flash generating device207. The housing200of the image processing apparatus20includes on the front surface thereof a display204that occupies most of the front surface.

As described below, when the object30is photographed, the video of the object30captured by the camera206is displayed on the display204on a real-time basis. A user may photograph the object30by operating an input interface (described below) mounted on the display204while observing the object30displayed on the display204.

In accordance with the exemplary embodiment, the object30may be a test image of a printer or a multi-function apparatus, output by an image forming apparatus. The test image is an image printed on an entire paper sheet with one or a mixture of black, cyan, magenta, and yellow colors at a given density, or may be an entire half-tone image. The test image is used to check the presence or absence of nonuniformity or banding when image data at a specific gradation value is printed out. The object that is photographed in accordance with the exemplary embodiment of the disclosure is not limited to the test image. The object30may be a different image.

If an entire image filled with the same color (solid image) is photographed as an object by the image processing apparatus20with the object not irradiated with light from the flash generating device207, a shadow may be created on the object30by light incoming from the outside as illustrated inFIG. 2. For example, nonuniformity of light reflected from the object30, such as luminance nonuniformity, may be caused by the shadow of the image processing apparatus20itself or the shadow of a hand of a photographer.

The object30may be photographed within close distance with the flash generating device207irradiating the object30with light in order to capture an image that is free from the effect of the shadow caused by light incoming from the outside. Referring toFIG. 3, concentric gradations about the center where light is projected appear on the object30. If such a problem occurs, it is difficult to check the presence or absence of nonunformity or the presence or absence of banding when the image data at a fixed gradation value is printed out. It is desirable to reduce an effect involved in image capturing.

The effect caused by the flash generating device207radiating light is desirably reduced in the image captured by the image processing apparatus20of the exemplary embodiment. The image is thus captured in a manner described below.

The configuration of the image processing apparatus20of the exemplary embodiment is described with reference toFIG. 4.FIG. 4illustrates the hardware configuration of the image processing apparatus20of the exemplary embodiment.

Referring toFIG. 4, the image processing apparatus20includes a controlling microprocessor201, a memory202, a storage device203, a display204, an input interface205, a camera206, and a flash generating device207. Each of these elements is connected to control bus208.

The controlling microprocessor201controls the operation of each element in the image processing apparatus20in accordance with a control program stored on the storage device203.

An image of the object30photographed by the camera206and a composite image generated by an image generating unit are temporarily stored on the memory202.

The storage device203includes a solid-state drive (SSD) or a hard disk (HDD), and stores a control program that controls each element in the image processing apparatus20.

The display204includes a liquid-crystal display or an organic electroluminescent (EL) display, mounted on the housing200of the image processing apparatus20, and displays information processed by a display controller described below.

The input interface205is a touch panel overlaid on the front surface of the display204, and serves as an input unit on which a user operating the image processing apparatus20inputs instructions.

The camera206is arranged on the rear surface of the housing200of the image processing apparatus20. The camera206captures the image of the object30in response to an instruction that the user inputs by operating the input interface205, and an instruction from a camera controller described below. The captured image is stored on the memory202.

The flash generating device207includes a light-emitting diode (LED) light. The flash generating device207serves as a light irradiation unit that irradiates the object30with flash light when the object30is photographed in response to the instruction from the camera controller.

The functionality of the image processing apparatus20of the exemplary embodiment is described with reference toFIG. 5.FIG. 5is a functional block diagram illustrating the image processing apparatus20ofFIG. 4. Referring toFIG. 5, the image processing apparatus20performs the functionalities of a camera controller211, an image receiving unit212, an image generating unit213, and a display controller214when the controlling microprocessor201executes the control program stored on the storage device203.

The camera controller211controls the operations of the camera206and the flash generating device207. If the object30in the video captured by the camera206satisfies a specific condition, for example, if the object30is recognized in a predetermined size, the camera controller211adjusts focus. The camera controller211captures a first image by causing the flash generating device207to irradiate the object30with flash light, and immediately in succession to the capturing of the first image, captures a second image by not causing the flash generating device207to irradiate the object30with flash light, and stores the first and second images on the memory202.

The image receiving unit212receives the first image that has been captured with the flash generating device207irradiating the object30with flash light and the second image that has been captured with the flash generating device207not irradiating the object30with flash light. In other words, the image receiving unit212retrieves the first image and the second image from the memory202.

The image generating unit213performs projective transformation such that the first and second images retrieved by the image receiving unit212are corrected to be in a specified size. More specifically, when the object30is photographed by the camera206in the image processing apparatus20, the image of the object30is captured at a slight slant angle. By performing the projective transformation on the captured images, distortion is removed from the images and the images are adjusted in size.

The image generating unit213adjusts brightness of the whole second image such that brightness at a specific location in the first image retrieved and projective-transformed by the image receiving unit212is approximately equal to brightness at a specific location in the second image retrieved and projective-transformed by the image receiving unit212. The image generating unit213generates a composite image by replacing a pixel value of a pixel that has been affected by the irradiation of flash light from the flash generating device207with a pixel value of a pixel in the brightness-adjusted second image.

More specifically, the image generating unit213generates the composite image by replacing in the first image a pixel value of a pixel higher in brightness than a pixel in the second image adjusted in brightness, as a pixel that has been affected by the flash light irradiation by the flash generating device207, with the pixel value of the pixel in the second image adjusted in brightness. Alternatively, the image generating unit213generates the composite image by selecting a pixel value of the first image retrieved by the image receiving unit212or a pixel value of the second image retrieved by the image receiving unit212, whichever is lower.

The display controller214processes an image to be displayed on the display204. The display controller214performs control to display on the display204a video that is captured by the camera206when an object is photographed by the camera206, to display on the display204an image generated by the image generating unit213, or to display on the display204a user interface (UI) that allows a user to input a variety of instructions.

The image processing apparatus20of the exemplary embodiment is described with reference toFIGS. 6, 7A and 7B, and8A through8C.FIG. 6is a flowchart illustrating an image composing process performed by the image processing apparatus20of the exemplary embodiment.FIGS. 7A and 7Billustrate images that have undergone projective transformation. Specifically,FIG. 7Aillustrates the first image that is captured with the object irradiated with flash light, andFIG. 7Billustrates the second image that is captured with the object not irradiated with the flash light.FIGS. 8A through 8Cillustrate the concept of the image composing process of the exemplary embodiment.FIG. 8Aillustrates luminance of a pixel row of the image captured with the object irradiated with the flash light from the flash generating device207and luminance of a pixel row of the image captured with the object not irradiated with the flash light from the flash generating device207.FIG. 8Billustrates the concept of corrected luminance.FIG. 8Cillustrates luminance of the pixel row of the composite image.

In step S401, the camera controller211causes the camera206to start operating in response to an instruction from a user. The display controller214displays a video captured by the camera206on the display204. The camera controller211determines whether the object30to be photographed is recognized as satisfying a predetermined condition. The condition is related to whether the object30is recognized as being in a predetermined size or larger. If the predetermined condition is not satisfied, an attempt to recognize the object30continues with the camera206inactive. If the object30that satisfies the predetermined condition is not recognized even after the time elapse of a predetermined period of time, the image composing process may end.

If the object30to be photographed is recognized as satisfying the predetermined condition, the process proceeds to step S402. The camera controller211causes the flash generating device207to irradiate the object30with flash light for image capturing.

In step S403, the camera controller211stores on the memory202, as the first image, an image that is captured with the flash generating device207irradiating the object30with the flash light.

In step S404, the camera controller211immediately captures the image of the object30with the flash generating device207not irradiating the object30with the flash light.

In step S405, the camera controller211stores on the memory202, as the second image, an image that is captured with the flash generating device207not irradiating the object30with the flash light.

In step S406, the image receiving unit212retrieves the first and second images from the memory202. The image generating unit213projective-transforms each of the first and second images, and corrects the projective-transformed first and second images to be equal to each other in size. During the projective transformation, distortion in the first and second image is removed. The intended image size during the projective transformation is pre-set by the user. Alternatively, during the projective transformation, the display controller214may request the user to specify the image size, and the user may operate the input interface205to specify the image size to which the images have to be scaled.

In step S407, the image generating unit213adjusts the brightness of the whole second image such that brightness at specific locations in the first image projective-transformed may be approximately equal to brightness at specific locations in the second image projective-transformed. The specific locations are multiple locations off the central region of each of the first image and second images projective-transformed. For example, if four imaginary lines are drawn from the center of each of the first image and second images projective-transformed to the four corners thereof, the specific locations are four points, each point at one-third length point along each imaginary line from the center of each image. The specific locations may be other multiple points in the images. The specific locations are not limited to four points, and may be a different number of points.

The brightness may be any index indicating a luminance value, a gradation value, lightness, or a degree of brightness of an image. In the following discussion, the brightness is luminance. According to the exemplary embodiment, the image generating unit213adjusts the luminance value of the whole area of the second image that has been projective-transformed such that the mean of the luminance values at the four locations in the first image projective-transformed and the mean of the luminance values at the four locations in the second image projective-transformed are approximately equal to each other, specifically within a predetermined range, or more preferably converge to the same value. Besides the mean of the luminance values, a variance of luminance or contrast may be calculated, and the second image may be adjusted such that the variance falls within a predetermined range.

The operation in step S407is described with reference toFIGS. 7A and 7B, andFIGS. 8A through 8C.FIG. 7Aillustrates the state of the first image that is obtained by capturing the image of the object30with the object30irradiated with flash light and then by projective-transforming the image.FIG. 7Billustrates the state of the second image that is obtained by capturing an image of the object30with the object30not irradiated with flash light and then by projective-transforming the image.FIG. 8Aillustrates luminance of the pixel row of the first image ofFIG. 7Aalong line VIIIA-VIIIA and luminance of the pixel row of the second image ofFIG. 7Balong line VIIIA-VIIIA.FIG. 8Billustrates luminance of pixel rows of the first and second images with brightness of the second image adjusted, andFIG. 8Cillustrates luminance of the pixel row of the composite image into which the first and second images are combined.

Referring toFIGS. 7A and 7B, andFIGS. 8A through 8C, the horizontal axis ofFIGS. 8A through 8Cillustrating the luminance of the pixel rows is an X coordinate when the image is cut in a horizontal direction, and the vertical axis represents luminance values. Referring toFIG. 8A, luminance of the image captured with the flash generating device207irradiating the object with the flash light is typically higher than luminance of the image captured with the flash generating device207not irradiating the object with the flash light. Furthermore, the plot of the luminance of the image captured with the flash generating device207irradiating the object with the flash light has a hill-like shape along the horizontal axis with the peak point thereof at the central region and the slopes on both sides. Referring toFIG. 8B, the image generating unit213adjusts the luminance of the whole second image projective-transformed such that luminance values at points P in the first and second images projective-transformed are equal to each other (in other words, luminance values are generally increased as illustrated inFIG. 8B).

In step S408ofFIG. 6, the image generating unit213replaces the pixel value of a pixel that is affected by the flash light radiated by the flash generating device207with the pixel value of a pixel in the second image that has been adjusted in step S407. More specifically, the image generating unit213replaces the pixel value of a pixel brighter in the first image than in the second image that has been adjusted in brightness, as a pixel that has been affected by the flash light irradiated by the flash generating device207, with a pixel value of a pixel in the second image that has been adjusted in brightness. Alternatively, the image generating unit213generates a composite image by comparing the projective-transformed first image with the bright-adjusted second image on a per pixel basis, and selecting a pixel value, whichever is less bright.

FIG. 8Cillustrates the composite image as the resulting state. Within a range defined by specific points P, pixels in the second image adjusted in brightness are used as pixels in the composite image ofFIG. 8C. Outside the range defined by the specific points P, the pixels in the first image projective-transformed are used as pixels in the composite image ofFIG. 8C.

In step S409ofFIG. 6, the image generating unit213stores the composite image generated in step S408on the memory202. The display controller214displays the composite image on the display204such that the user may observe the composite image. The image composing process then ends.

In the image composing process described above, in the first image projective-transformed, the pixels brighter than those in the second image adjusted in brightness are determined to be the pixels that are affected by the flash light radiated by the flash generating device207, and the pixel values of those pixels are thus replaced with the pixel values in the second image adjusted in brightness. The disclosure is not limited to the process. The pixel values of pixels in a region serving as a target to be replaced and of pixels surrounding the region are replaced with values that are calculated using pixel values in the first image projective-transformed and pixel values in the second pixel adjusted in brightness. In such a case, the pixel value of the pixel serving as a replacement may be obtained by combining, using SoftMinimum function, a pixel value in the first image projective-transformed and a pixel value in the second image adjusted in brightness.

According to the exemplary embodiment described above, the second image is captured with the object not irradiated with the flash light immediately after the first image is captured with the object irradiated with the flash light. Conversely, the first image may be captured with the object irradiated with the flash light immediately after the second image is captured with the object not irradiated with the flash light.

Modification of the exemplary embodiment of the disclosure is described with reference toFIGS. 9 through 13.FIG. 9illustrates the concept of an image processing system60A of a modification of the exemplary embodiment of the disclosure.FIG. 10is a functional block diagram illustrating a terminal device20A of the modification.FIGS. 11A and 11Billustrate the configuration of an image processing server80A of the image processing system20A of the modification.FIG. 11Aillustrates the hardware configuration of the image processing server80A, andFIG. 11Bis the functional block diagram of the image processing server80A.FIG. 12is a flowchart illustrating the process of the terminal device20A of the modification.FIG. 13is a flowchart illustrating the process of the image processing server80A of the modification.

In the exemplary embodiment described with reference toFIG. 1throughFIG. 8, the photographing of the object30and the composing process of the images are performed by the single image processing apparatus20. In the modification described with reference toFIG. 9throughFIG. 13, the photographing of the object30and the displaying of the composite image subsequent to image processing are performed on the terminal device20A, and the image processing is performed on the image processing server80A.

Referring toFIG. 9, the image processing system60A of the modification includes the terminal device20A and the image processing server80A. The terminal device20A and the image processing server80A are respectively connected to a network70A. In the modification, the terminal device20A is generally identical in hardware configuration to the image processing apparatus20except that the terminal device20A includes a communication interface (not illustrated). Elements identical to those in the image processing apparatus20are designated with the same reference symbols and the detailed discussion is omitted herein.

Referring toFIG. 10, a controlling microprocessor201in the terminal device20A of the modification performs a control program stored on the storage device203. The terminal device20A thus has functionalities of a camera controller211A, a display controller212A, and an image transceiver unit213A. The camera controller211A and the display controller212A are respectively identical in functionality to the camera controller211and the display controller214in the image processing apparatus20of the exemplary embodiment, and the detailed discussion thereof is omitted herein.

The first image is captured with the flash generating device207irradiating the object and the second image is captured with the flash generating device207not irradiating the object. The first image and the second image are stored on the memory202or the storage device203. In response to an instruction input by the user operating the input interface205, the image transceiver unit213A transmits the first image and the second image to the image processing server80A via the network70A or receives the composite image produced on the image processing server80A from the image processing server80A via the network70A.

Referring toFIG. 11A, the image processing server80A of the modification includes a controlling processor801A, a memory802A, a storage device803A, and a communication interface804A, each thereof connected to a control bus805A.

The controlling processor801A controls the operation of each element in the image processing server80A in accordance with the control program stored on the storage device803A. The first image captured with the flash light radiating and the second image with the flash light not radiating are temporarily stored on the memory802A. The storage device803A includes a hard disk (HDD) and/or a solid-state drive (SDD), and stores the control program that controls each element in the image processing server80A. The communication interface804A controls communication that the image processing server80A performs with the terminal device20A via the network70A.

Referring toFIG. 11B, the controlling processor801A in the image processing server80A executes the control program stored on the storage device803A. The image processing server80A has thus the functionalities of an image receiving unit811A, an image generating unit812A, and an image transmitting unit813A.

The image receiving unit811A receives from the terminal device20A the first image captured with the flash light radiating and the second image with the flash light not radiating, and temporarily stores the first and second images on the memory802A.

The image generating unit812A adjusts the brightness of the whole second image such that the brightness at a specific location in the first image acquired by the image receiving unit811A is approximately equal to the brightness at a corresponding specific location in the second image acquired by the image receiving unit811A. The image generating unit812A generates a composite image by replacing in the first image the pixel value of a pixel that is affected by the irradiation of the flash light with the pixel value of a pixel in the second image that is adjusted in brightness. The process described above remains unchanged from the process of the image generating unit213in the image processing apparatus20of the exemplary embodiment, and the detailed discussion thereof is omitted herein.

The image transmitting unit813A transmits to the terminal device20A the composite image generated by the image generating unit812A via the communication interface804A and the network70A.

The process of the image processing system60A of the modification is described below. The process of the terminal device20A is described first with reference toFIG. 12. In step S901ofFIG. 12, the camera controller211A causes the camera206to start operating in response to an instruction that is input by the user operating the terminal device20A. The terminal device20A photographs the object by causing the flash generating device207to irradiate the object with the flash light, and temporarily stores the captured image as the first image on the memory202.

In step S902, the image transceiver unit213A transmits the first image to the image processing server80A via the network70A.

In step S903, the camera controller211photographs the object with the flash generating device207not irradiating the object, and stores the captured image as the second image on the memory202.

In step S904, the image transceiver unit213A transmits the second image to the image processing server80A via the network70A. In the modification of the exemplary embodiment, the first image and the second image are transmitted to the image processing server80A each time each of the first and second images is captured. Alternatively, the first and second images, after being captured, may be together transmitted to the image processing server80A.

Referring toFIG. 13, the composing process of the images is performed on the image processing server80A. In step S905, the image transceiver unit213A receives the composite image from the image processing server80A, and stores the composite image on the memory202.

In step S906, the display controller212A displays on the display204the composite image received in step S905such that the user may view the composite image. The process thus ends.

The process of the image processing server80A is described with reference toFIG. 13. In step S1001ofFIG. 13, the image receiving unit811A in the image processing server80A receives the first image and the second image from the terminal device20A. The first image and the second image may be received together at the same time or separately with a time difference therebetween. The first image and the second image are temporarily stored on the memory802A.

In step S1002, the image generating unit812A projective-transforms each of the first and second images such that the two images are equal in size. The image size serving as a target of the projective transformation may be preset by the user, and the information on the image size may be received together with the first and second images. Alternatively, the terminal device20A may be requested to specify the image size, and the user may specify the image size by operating the input interface205in the terminal device20A. The information on the specified image size may be transmitted to the image processing server80A.

In step S1003, the image generating unit812A adjusts the brightness of the whole second image such that the brightness at a specific location in the first image projective-transformed is approximately equal to the brightness at a specific location in the second image projective-transformed. The operation is generally identical to the operation that the image generating unit213in the image processing apparatus20performs in step S407ofFIG. 6, and the detailed discussion thereof is omitted herein.

In step S1004, the image generating unit812A generates a composite image by replacing in the first image projective-transformed the pixel value of a pixel that is affected by the irradiation of the flash light with the pixel value of a pixel in the second image that is adjusted in brightness in step S1003. The operation is generally identical to the operation that the image generating unit213in the image processing apparatus20of the exemplary embodiment performs in step S408ofFIG. 6, and the detailed discussion thereof is omitted herein.

In step S1005, the image transmitting unit813A transmits the composite image generated in step S1004to the terminal device20A via the communication interface804A and the network70A. The process thus ends.

In the modification of the exemplary embodiment, the terminal device20A captures the first image and the second image, and successively transmits the first image and the second image to generate the composite image. The disclosure is not limited to this method. The first image and the second image may be first stored on the storage device203in the terminal device20A, and the image processing server80A may later retrieve the first and second images, and generate the composite image.