STORAGE MEDIUM STORING COMPUTER-READABLE INSTRUCTIONS CAUSING COMPUTER TO EXECUTE IMAGE PROCESS ON IMAGE DATA TO GENERATE OUTPUT DATA RELATED TO DEFECT IN APPEARANCE OF OBJECT

A computer generates processed original image data by executing a first image process on original image data. The original image data represents an object image to be printed. The computer generates processed capture image data by executing a second image process on captured image data. The captured image data represents a captured object image. The captured object image is obtained by capturing an image of a printed object. The printed object is produced by printing the object image. The computer generates output data by executing a third image process on the processed original image data and the processed captured image data. The output data is related to a defect in an appearance of the printed object. The first image process includes a first process, which is not included in the second image process. The second image process includes a second process, which is not included in the first image process.

BACKGROUND ART

Conventionally, captured image data has been used in visual inspections and other processes related to the appearance of an object. For example, a captured image of a workpiece is inputted into a trained model to inspect the appearance of the workpiece. The model is trained using supervised data. Japanese Patent Application Publication No. 2021-060962 proposes a method of generating supervised data in which a captured image of the workpiece is segmented into a plurality of regions, and each region is marked to indicate whether the region contains prescribed information.

SUMMARY

However, it is not easy to generate data related to defects in the appearance of a workpiece, and the conventional technology leaves room for improvement.

In view of the foregoing, it is an object of the present disclosure to provide a technology for generating data related to defects in the appearance of an object.

In order to attain the above and other object, according to one aspect, the present disclosure provides a non-transitory computer-readable storage medium storing a set of computer-readable instructions for a computer. The computer is configured to perform processing on image data. The set of computer-readable instructions, when executed by the computer, causes the computer to perform: generating processed original image data; generating processed captured image data; and generating output data. The generating processed original image data is performed by executing a first image process on original image data. The original image data represents an object image to be printed. The generating processed captured image data is performed by executing a second image process on captured image data. The captured image data represents a captured object image. The captured object image is obtained by capturing an image of a printed object. The printed object is produced by printing the object image. The generating output data is performed by executing a third image process on the processed original image data and the processed captured image data. The output data is related to a defect in an appearance of the printed object. The first image process includes a first process. The first process is not included in the second image process. The second image process includes a second process. The second process is not included in the first image process.

With the above configurations, the processed original image data is generated by executing the first image process including the first process, which is not included in the second image process, and the processed captured image data is generated by executing the second image process including the second process, which is not included in the first image process. Hence, by using the processed original image data and the processed captured image data, the computer can generate suitable output data related to a defect in the appearance of the printed object.

According to another aspect, the present disclosure also provides a non-transitory computer-readable storage medium storing a set of computer-readable instructions for a computer. The computer is configured to perform processing on image data. The set of computer-readable instructions, when executed by the computer, causes the computer to perform: generating processed original image data; generating processed captured image data; and generating output data. The generating processed original image data is performed by executing a first image process on original image data. The original image data represents an object image which is a design image of a target object. The generating processed captured image data is performed by executing a second image process on captured image data. The captured image data represents a captured object image. The captured object image is obtained by capturing an image of the target object. The generating output data is performed by executing a third image process on the processed original image data and the processed captured image data. The output data is related to a defect in an appearance of the target object. The third image process includes a process using a machine learning model that has been trained. The first image process includes a first process. The first process is not included in the second image process. The second image process includes a second process. The second process is not included in the first image process. The first process includes a pre-process. The pre-process includes at least one of a noise adding process and a blurring process. The noise adding process, when executed on first target image data representing a first target image, adds noise to the first target image. The blurring process, when executed on second target image data representing a second target image, blurs the second target image. The machine learning model has been trained using training image data. The training image data is generated by executing processes including a process identical to the pre-process on the original image data.

With the above configurations, the processed original image data is generated by executing the first image process including the first process, which is not included in the second image process, the processed captured image data is generated by executing the second image process including the second process, which is not included in the first image process, and the machine learning model has been trained using the training image data which is generated by executing processes including a process identical to the pre-process which is included in the first process on the original image data. Hence, by using the processed original image data and the processed captured image data and executing the third image process including a process using the machine learning model that has been trained, the computer can generate suitable output data related to a defect in the appearance of the target object.

According to still another aspect, the present disclosure also provides a non-transitory computer-readable storage medium storing a set of computer-readable instructions for a computer. The computer is configured to perform processing on image data. The set of computer-readable instructions, when executed by the computer, causes the computer to perform: generating processed original image data; generating processed captured image data; and generating output data. The generating processed original image data is performed by executing a first image process on original image data. The original image data represents a photographed image of an appearance of a target object. The generating processed captured image data is performed by executing a second image process on captured image data. The captured image data represents a captured object image. The captured object image is obtained by capturing an image of a printed object. the printed object is produced by printing an object image. The generating output data is performed by executing a third image process on the processed original image data and the processed captured image data. The output data is related to a defect in an appearance of the printed object. The first image process includes a first process. The first process is not included in the second image process. The second image process includes a second process. The second process is not included in the first image process.

The technology disclosed herein can be implemented in various aspects, such as image processing methods and data processing apparatuses, methods and devices for training machine learning models, computer programs for implementing the functions of those methods, apparatuses, or devices, recording media storing those computer programs (e.g., non-transitory storage media), machine learning models that have been trained, and the like.

DESCRIPTION

A1. Device Configuration

FIG.1is a block diagram illustrating a data processing apparatus200according to an embodiment of the present disclosure. The data processing apparatus200in the present embodiment is a personal computer. The data processing apparatus200performs various data processes for inspecting the appearance of an object (e.g., a label sheet provided on a multifunction peripheral or other product). The following description will focus on inspecting the appearance of a label sheet800disposed on a multifunction peripheral (MFP)900.

The data processing apparatus200includes a processor210, a storage device215, a display unit240, an operating unit250, and a communication interface270. The above components are interconnected via a bus. The storage device215includes a volatile storage device220, and a nonvolatile storage device230.

The processor210is a device, such as a CPU, that is configured to perform data processing. The volatile storage device220is DRAM, for example. The nonvolatile storage device230is flash memory, for example. The nonvolatile storage device230stores a generation program231, a training program232, an inspection program233, an object detection model M1, an image generation model300, a plurality of sets of training image data I1td, and document image data700d. In the present embodiment, the object detection model M1and image generation model300are program modules used for forming machine learning models. The programs231-233, models M1and300, and image data I1tdand700dwill be described later in greater detail.

The display unit240is a device configured to display images, such as a liquid crystal display or an organic light-emitting diode display. The operating unit250is a device configured to receive operations by the operator, such as a touchscreen arranged over the display unit240, buttons, levers, and the like. By operating the operating unit250, the operator can input various requests and instructions into the data processing apparatus200. The communication interface270is an interface for communicating with other devices, such as a USB interface, a wired LAN interface, or a wireless communication interface conforming to the IEEE 802.11 standard. A digital camera110is connected to the communication interface270. The digital camera110is used for photographing the label sheet800, i.e., capturing images of the label sheet800.

FIG.2is a perspective view illustrating the digital camera110, the MFP900, and a belt190. In the present embodiment, the belt190is part of a belt conveyor used to convey MFPs900.FIG.2illustrates only a section of the belt190. The belt190is arranged to form a flat upper surface191. The MFP900is placed on the belt190so that a bottom surface909of the MFP900contacts the upper surface191of the belt190. The belt conveyor is configured to move the MFP900in a conveying direction D190so that the MFP900passes in front of the digital camera110. The conveying direction D190crosses a photographing (shooting) direction D110of the digital camera110. For example, the conveying direction D190is roughly orthogonal to the photographing direction D110. The label sheet800is affixed to a first side surface901of the MFP900. The digital camera110is disposed in a position for photographing the label sheet800.

A2. Label Sheet

FIG.3Ais a schematic diagram illustrating an example of a label sheet. The label sheet800is a rectangular sheet.FIG.3Aillustrates a label sheet800that is free of defects (a label sheet800with no defects). The label sheet800can depict various objects. In this example, the label sheet800depicts a logotype810, a mark820, and explanatory text830. Details of the mark820and explanatory text830have been omitted from the drawing.

FIG.3Bis a schematic diagram illustrating an example of a design image for the label sheet800. A design image is an image showing the design of an object such as the label sheet800. In the present embodiment, the label sheet800is produced by printing an image of the label sheet800on a sheet. The image of the label sheet800is an object image800i(also called the design image800i). A document image700inFIG.3Bdepicts the design image800itogether with trim marks790. The trim marks790are also called crop marks and indicate edge portions of the document image700that should be cut off after printing. When producing the label sheet800, the document image700is printed on a sheet in accordance with the document image data700drepresenting the document image700. The label sheet800is then finished by cutting off the edge portions defined by the trim marks790. The document image data700dis also called “block copy data”. In the present embodiment, the document image data700dis color bitmap data. However, the document image data700dmay have any data format.

FIG.4Ais a block diagram illustrating an example of the image generation model300. The image data inputted into the image generation model300is input image data I11d. The image data generated by the image generation model300is generated image data I12d. Each of the image data I11dand I12din the present embodiment is grayscale bitmap data. The image data I11dand I12drepresent images I11and I12, respectively. The images I11and I12are rectangular in shape and have two parallel sides aligned in a first direction Dx, and two parallel sides aligned in a second direction Dy orthogonal to the first direction Dx. The images I11and I12are represented by color values (luminance values in the present embodiment) for each of a plurality of pixels. The pixels are arranged in a matrix configuration having rows in the first direction Dx and columns in the second direction Dy. The number of pixels in the first direction Dx and the number of pixels in the second direction Dy are both predetermined and are the same for both the image data I11dand I12d.

As will be described later, image data representing an image of the label sheet800captured by the digital camera110(seeFIG.2) is inputted into the image generation model300. The input image I11represented by the input image data I11dincludes a captured image I11aof the label sheet800(seeFIG.3A). The label sheet800may contain defects. The image generation model300is trained to produce a generated image I12containing an image I12aof the same label sheet800that is free of defects, i.e., that contains no defects. The size and position in the generated image I12of the image I12afor the label sheet800is the same as the size and position in the input image I11of the captured image I11afor the label sheet800.

The image generation model300in the present embodiment is referred to as a variation autoencoder (VAE). The image generation model300has an encoder302and a decoder307. The encoder302performs dimensionality reduction on the input image data I11dto generate latent data305indicating features of the input image I11. The decoder307performs dimensionality restoration on the latent data305to produce the generated image data I12d. The encoder302and decoder307may each have any of various configurations. For example, the encoder302may have one or more fully-connected layers that calculate the mean and standard deviation using the input image data I11d, and a layer that generates the latent data305using this mean and standard deviation. When generating the latent data305, noise can be introduced according to a method known as the reparameterization trick. The decoder307may also have one or more fully-connected layers that calculate the generated image data I12dusing the latent data305. The number of dimensions of the latent data305may be any of various numbers smaller than the number of dimensions of the image data I11dand I12d. The mean and standard deviation may be calculated for each element in the latent data305.

A4. Training Image Data Generation Process

FIG.5is a flowchart illustrating an example of steps in a process for generating training image data used to train the image generation model300(training image data generation process). In the present embodiment, the processor210(seeFIG.1) uses the document image data700d(seeFIG.3B) to generate a plurality of sets of training image data representing images that are similar to the captured image. The processor210executes the process inFIG.5according to the generation program231.

In S110ofFIG.5, the processor210acquires the document image data700d(seeFIG.3B) from the nonvolatile storage device230. In S120the processor210acquires original image data from the document image data700d. The original image data is bitmap data for the region that contains the design image800i.FIG.3Cis a schematic diagram illustrating an example of an original image710represented by original image data710d. The original image710represented by the original image data710dincludes the design image800iand its surrounding area in the document image700(seeFIG.3B). The number of pixels aligned in the first direction Dx of the original image710and the number of pixels aligned in the second direction Dy of the original image710are the same as those in the image inputted into the image generation model300(e.g., the input image I11; seeFIG.4A).

Here, the processor210extracts a predetermined region of the document image700(seeFIG.3B) in S120as the region of the original image710. As an alternative, the processor210may analyze the trim marks790to determine the region of the original image710.

In S130the processor210performs grayscale conversion on the original image data710dto generate gray original image data. In this conversion, RGB color values are converted to luminance values using prescribed relational expressions (e.g., a color conversion formula for converting values in the RGB color space to values in the YCbCr color space).FIG.3Dis a schematic diagram illustrating an example of a gray original image720represented by gray original image data720d. The gray original image720includes a gray design image720a, which is the design image800iconverted to a grayscale image. In S130the processor210stores the generated gray original image data720din the storage device215(e.g., the nonvolatile storage device230).

In S140the processor210uses the gray original image data720dto generate P sets of training image data (where P is an integer greater than or equal to two). Step S140includes steps S150and S160.

In S150the processor210executes a noise adding process P times on the gray original image data720dto generate P sets of training image data I1td. The noise adding process is performed to generate training images that are similar to the captured image. In the present embodiment, the processor210adds noise values randomly generated from a Gaussian distribution to respective color values for the plurality of pixels. This noise is also referred to as Gaussian noise. In the present embodiment, the processor210generates a noise value for each pixel and a noise value for each image.

The mean value and standard deviation of the Gaussian distribution is determined experimentally in advance using captured images. For example, the experiment uses a plurality of captured images captured by the digital camera110(seeFIG.2) in a dark environment where no light illuminates the digital camera110. These captured images are all plain black images. However, the color value for any one pixel may vary among the plurality of captured images due to noise. The standard deviation of the Gaussian distribution may be set to a value corresponding to this variation in color values. The mean value of the Gaussian distribution may be set to zero.

In S160the processor210performs a blurring process on each of the P sets of training image data I1td. The blurring process serves to generate training images that are similar to the captured image. The image of the label sheet800may be blurred in the captured image due to a variety of reasons. For example, the MFP900(seeFIG.2) is conveyed by the belt190in the present embodiment, and the digital camera110photographs the MFP900while the MFP900is stopped in front of the digital camera110. However, deformation in the belt190may cause the MFP900to vibrate after the belt190is halted. The MFP900may vibrate parallel to the conveying direction D190, for example. This vibration diminishes over time. In a case where the MFP900is vibrating when the label sheet800is photographed by the digital camera110, the image of the label sheet800may be blurry. Such blurring caused by movement of the subject is also called “motion blur.”

Motion blur can be reproduced through convolution using the point spread function. The point spread function indicates the degree of spreading caused by motion blur of a color value for a single pixel. In the present embodiment, motion of the MFP900is approximated by linear motion parallel to the conveying direction D190. A line segment defined by two parameters, angle and length in the image, can be used as a point spread function that indicates linear motion. The angle and length representing the point spread function are determined experimentally in advance using a captured image. For example, the digital camera110is used to photograph the label sheet800under the same conditions as those for photographing the label sheet800for inspection. The captured image is then analyzed to calculate a motion vector indicating movement of the label sheet800within the captured image. The angle and length of this motion vector are used as the angle and length representing the point spread function. In S160the processor210executes a convolution process on the sets of training image data I1tdusing this point spread function to add motion blur to the training image data I1td. Here, a plurality of captured images may be used for calculating the motion vector. For example, the motion vector may be calculated from a plurality of images captured within a length of time equivalent to the exposure time of the inspection photographs (captured images). Alternatively, the motion vector may be calculated using video image data taken over a period of time equivalent to the exposure time.

Through the process of S140(and specifically, the processes of S150and S160), the processor210generates P sets of training image data I1td. Each of the P sets of training images has noise and blur and is similar to the captured image of the label sheet800.

In S170the processor210stores the P sets of training image data I1tdin the storage device215(the nonvolatile storage device230in this case). Hence, P training pairs of training image data I1tdand original image data710are generated. Subsequently, the processor210ends the process ofFIG.5. The total number P of the sets of training image data I1tdis preset to a number sufficient for training the image generation model300appropriately.

A5. Training Process for Training Image Generation Model300

FIG.6is a flowchart illustrating an example of steps in a training process for training the image generation model300(seeFIG.4A).FIG.4Bis a block diagram illustrating an overview of the training process in which the image generation model300is being trained. In the present embodiment, the image generation model300is trained so that when training image data I1tdrepresenting a training image I1tis inputted into the image generation model300, the image generation model300produces generated image data I1xdrepresenting a generated image I1xthat depicts the same image as the training image I1trepresented by the inputted training image data I1td. The processor210executes the process inFIG.6according to the training program232.

In S210ofFIG.6, the processor210acquires a subset of the P sets of training image data I1tdgenerated in the process ofFIG.5. The subset is configured of a plurality of sets of training image data I1td. The processor210selects a plurality of sets of unprocessed training image data I1tdas the subset. The total number of sets of training image data I1tdin the subset is preset.

In S220the processor210inputs each set of training image data I1tdin the subset into the image generation model300to produce a set of generated image data I1xd. The processor210produces the generated image data I1xdby performing the operation (calculation) for each layer of the image generation model300using model parameters for the corresponding layers. A single set of generated image data I1xdis generated for each set of training image data I1tdincluded in the subset.

In S230the processor210calculates evaluation values using the training image data I1tdand the generated image data I1xd. An evaluation value is calculated for each set of training image data I1td. Next, the processor210calculates a loss L using the plurality of evaluation values calculated from the plurality of sets of training image data I1tdincluded in the subset. The loss L may be the sum of the plurality of evaluation values, for example.

Any method suitable for training the image generation model300may be used for calculating the evaluation values. Since the image generation model300is a VAE in the present embodiment, various values capable of maximizing the variational lower bound can be employed as an evaluation value. For example, the evaluation value may be the sum of a reconstruction error and a regularization error. The reconstruction error is a parameter indicating the difference between the training image I1tand generated image I1x, such as a cross-entropy error. The regularization error is a parameter indicating the difference between a combination of the mean and standard deviation used to calculate the latent data305and the standard normal distribution, for instance. As an example, the regularization error may be calculated using the formula “−(1+log(s2)−m2−s2)/2” (where m is the mean and s is the standard deviation). This formula specifies the error in one element of the latent data305. The regularization error may be the sum of errors for all elements of the latent data305.

In S240the processor210adjusts the plurality of model parameters in the image generation model300to reduce the loss L. An algorithm using error backpropagation and gradient descent, for example, may be employed to adjust the plurality of model parameters. Here, the optimizer known as Adam may be used.

In S250the processor210determines whether a training termination condition has been satisfied. The training termination condition may be any condition indicating that the image generation model300has been properly trained. In the present embodiment, the training termination condition is the condition that the operator has inputted a termination instruction. Specifically, the processor210randomly acquires a prescribed number of sets of training image data I1tdthat have not been used in training from the P sets of training image data I1td. The processor210then inputs each of the plurality of sets of the acquired training image data I1tdinto the image generation model300to produce generated image data I1xd, thereby acquiring a plurality of sets of generated image data I1xdrepresenting a plurality of generated images I1x. The processor210displays a plurality of pairs of the training images I1tand generated images I1xon the display unit240. By viewing the display unit240, the operator confirms whether the generated images I1xadequately represent the training images I1t. Depending on the results of this confirmation, the operator may operate the operating unit250to input an instruction to terminate or to continue the training.

Note that the condition for terminating training may be various other conditions. For example, the termination condition may be that the loss L calculated using the prescribed number of sets of training image data I1tdnot used in training is less than or equal to a prescribed loss threshold.

When the processor210determines that training has not been terminated (S250: NO), the processor210returns to S210and executes the above process on a new subset. When the processor210determines that training has been terminated (S250: YES), in S260the processor210stores the trained image generation model300in the storage device215(the nonvolatile storage device230in this case). Subsequently, the processor210ends the process inFIG.6.

The trained image generation model300(seeFIG.4A) produces a generated image I12from an input image I11containing a captured image I11aof the label sheet800. The generated image I12contains an image I12aof the same label sheet800. The layout of an image of the label sheet800within the image is also the same for both the input image I11and the generated image I12.

The image generation model300is also trained using the training image data I1tdrepresenting a trained image I1tcontaining an image I1taof a label sheet800with no defects (defect-free label sheet800) to produce generated image data I1xdrepresenting a generated image I1xcontaining an image I1xaof the same label sheet800, i.e., an image of the defect-free label sheet800. When the label sheet800whose image is contained in an image inputted into the image generation model300has a defect, the image generation model300suitably reconstructs a defect-free partial image (a partial image without defects) from the partial image showing this defect and produces generated image data representing a generated image containing an image of the defect-free label sheet800.

A6. Inspection Process

FIGS.7and8are flowcharts illustrating an example of steps in an inspection process.FIG.8is a continuation of the process illustrated inFIG.7. By performing the inspection process, the data processing apparatus200(seeFIG.1) inspects the label sheet800on the MFP900(seeFIG.2). The processor210executes the inspection process according to the inspection program233.

In S310ofFIG.7, the processor210performs a process to acquire image data. Step S310includes steps S312, S314, S316, and S318. In S312the processor210supplies a photographing instruction to the digital camera110(seeFIG.2). In response to this instruction, the digital camera110photographs an area of the MFP900that includes the label sheet800and generates captured inspection data, which is image data representing the photographed (captured) image. In the present embodiment, the captured inspection data is color bitmap data that specifies the color of each pixel in the image for the three channels red (R), green (G), and blue (B). As described above, the digital camera110performs photographing while the MFP900is halted in front of the digital camera110. The processor210acquires data indicating the operating state of the belt190from a drive device (not illustrated) of the belt conveyor, for example. The processor210then supplies a photographing instruction to the digital camera110while the belt190is in a halted state.

In S314the processor210acquires captured image data from the captured inspection data. The captured image data is color bitmap data for the area of the captured inspection data that includes an image of the label sheet800.FIGS.9and10are schematic diagrams illustrating an example of changes in an image undergoing the inspection process. The first image from the top of the left column inFIG.9is an example of an image610represented by captured image data610d(hereinafter called the “captured label image610”). The captured label image610is a rectangular image having two parallel sides aligned in the first direction Dx and two parallel sides aligned in the second direction Dy, which is orthogonal to the first direction Dx. The captured label image610includes a captured image610aof the label sheet800. In the example ofFIG.9, the label sheet800has a scratch (not illustrated inFIG.3A), and thus the captured image610aof the label sheet800contains a portion depicting this scratch (hereinafter called the “scratch image801”). The captured image610aof the label sheet800is also skewed with respect to the captured label image610. The captured label image610has noise and blur.

In S314ofFIG.7, the processor210in the present embodiment uses the trained object detection model M1(seeFIG.1) to detect the region in the image represented by the captured inspection data that contains an image of the label sheet800, i.e., captured image610a. The processor210then acquires the captured image data610drepresenting this detected region from the captured inspection data. In the present embodiment, the object detection model M1is a model called YOLOv4 (You Only Look Once) that has been pretrained to detect an image of the label sheet800. Note that the object detection model M1may be any of various other object detection models, such as a Single Shot MultiBox Detector (SSD) or Region-based Convolutional Neural Networks (R-CNN). Any suitable method may be used to train the object detection model M1.

Steps S316and S318ofFIG.7are identical to steps S110and S120ofFIG.5. The processor210acquires the document image data700d(seeFIG.3B) in S316and acquires the original image data710dfrom the document image data700din S318. The first image from the top of the right column inFIG.9is the original image710represented by the original image data710d. This original image710is identical to the original image710inFIG.3C.

In S320ofFIG.7, the processor210executes a first pre-process. Step S320includes steps S322, S324, S326, and S328. In S322the processor210executes grayscale conversion on the captured image data610dto generate grayscale captured image data. The method of grayscale conversion is identical to the method used in S130ofFIG.5. The second image from the top of the left column inFIG.9is an example of a gray captured image620. The gray captured image620is represented by gray captured image data620dgenerated from the captured image data610d. The gray captured image620includes an image620aof the label sheet800.

In S324ofFIG.7, the processor210executes grayscale conversion on the original image data710dto generate gray original image data720d. The second image from the top of the right column inFIG.9is an example of a gray original image720represented by the gray original image data720dgenerated from the original image data710d. This gray original image720is identical to the gray original image720inFIG.3D.

In S326ofFIG.7, the processor210executes a second color adjustment process to adjust the color distribution in the gray captured image620represented by the gray captured image data620dto be closer to the color distribution in the gray original image720represented by the gray original image data720d. In the present embodiment, histogram matching is performed as the color adjustment process.FIG.11Ais an explanatory diagram illustrating an example of the histogram matching process. Specifically,FIG.11Aillustrates a graph of cumulative frequencies VC in which the horizontal axis represents the luminance value V, and the vertical axis represents the cumulative frequency VC (units: %). In the present embodiment, each luminance value V is expressed as one of 256 levels from 0 to 255. The cumulative frequency VC is calculated using a histogram of the luminance values V. Here, each luminance value V constitutes one category of the histogram (referred to as a “bin”). The cumulative frequency VC of a target category is the ratio of the sum of frequencies from the smallest category to the target category (i.e., the cumulative frequency) to the total number of frequencies in the histogram (i.e., the number of pixels).

The graph inFIG.11Aincludes a first curve C1and a second curve C2. The first curve C1depicts the cumulative frequencies VC for the image data undergoing the color adjustment (hereinafter called the “target image data”). The second curve C2depicts the cumulative frequencies VC for image data having a reference color distribution. A first luminance value V1of the target image data is converted to a second luminance value V2. The second luminance value V2is determined as follows. First, the processor210references the first curve C1for the target image data to acquire the cumulative frequency VC1corresponding to the first luminance value V1. Next, the processor210references the second curve C2to identify the second luminance value V2corresponding to the acquired cumulative frequency VC1. An adjusted luminance value V is similarly determined for other luminance values V.

FIG.11Billustrates an example of a histogram for luminance values VA1in the gray captured image620(seeFIG.9). This histogram has a first peak P1formed by a plurality of pixels representing background areas, and a second peak P2formed by a plurality of pixels representing objects (characters, marks, etc.). In addition to the peaks P1and P2, this histogram has several other small peaks. As an alternative, the image of the label sheet800may be composed of three or more colors. In this case, the histogram of luminance values could have three or more large peaks.

FIG.11Cillustrates an example of a histogram for luminance values VA2in the gray original image720(seeFIG.9). This histogram has a third peak P3formed by a plurality of pixels representing background areas, and a fourth peak P4formed by a plurality of pixels representing objects.

The widths of the peaks P1and P2in the gray captured image620(FIG.11B) are greater than the widths of the corresponding peaks P3and P4in the gray original image720(FIG.11C). This difference is due to the gray captured image620having various noise and various blurring, unlike the gray original image720.

The third image from the top of the left column inFIG.9is an example of a first pre-processed captured image630. The first pre-processed captured image630is represented by first pre-processed captured image data630dthat has been generated from the gray captured image data620din S326ofFIG.7. The first pre-processed captured image630includes an image630aof a label sheet800identical to the label sheet800depicted by the image620ain the gray captured image620.

FIG.11Dillustrates an example of a histogram for luminance values VB1in the first pre-processed captured image630. This histogram has a first adjusted peak P1xthat has been converted from the first peak P1(FIG.11B), and a second adjusted peak P2xthat has been converted from the second peak P2. The luminance values VB1of the adjusted peaks P1xand P2xapproach the luminance values VA2of the corresponding peaks P3and P4in the gray original image720(FIG.11C). Thus, the histogram matching process brings the color distribution in the first pre-processed captured image630closer to the color distribution in the gray original image720with respect to the peak positions (the luminance values in this case). The widths of the adjusted peaks P1xand P2xare also smaller than the respective widths of the original peaks P1and P2. The reason for the widths of the peaks being smaller is that when the color distribution having narrow peaks P3and P4(FIG.11C) is used as the reference color distribution for histogram matching, a large number of luminance values VA1contained in a peak with a large width (e.g., peak P1) are converted to a small number of luminance values VB1within a narrow range through histogram matching. Thus, the histogram matching process brings the color distribution in the first pre-processed captured image630closer to the color distribution in the gray original image720with respect to the widths of peaks.

In S328ofFIG.7, the processor210executes a first color adjustment process (histogram matching in the present embodiment) to adjust the color distribution in the gray original image720represented by the gray original image data720dto be closer to the color distribution in the gray captured image620drepresented by the gray captured image data620d. The third image from the top in the right column ofFIG.9is an example of a first pre-processed original image730. The first pre-processed original image730is represented by first pre-processed original image data730dthat has been generated from the gray original image data720din S328. The first pre-processed original image730includes a design label image730awhose colors have been adjusted from the gray design image720ain the gray original image720.

FIG.11Eillustrates an example of a histogram for luminance values VB2in the first pre-processed original image730. This histogram includes a third adjusted peak P3xthat has been converted from the third peak P3(FIG.11C), and a fourth adjusted peak P4xthat has been converted from the fourth peak P4. The luminance values VB2of the adjusted peaks P3xand P4xapproach the luminance values VA1of corresponding peaks P1and P2in the gray captured image620(FIG.11B). Thus, the process of histogram matching adjusts the color distribution in the first pre-processed original image730to be closer to the color distribution in the gray captured image620with respect to the peak positions (the luminance values in this case). Note that the widths of the adjusted peaks P3xand P4xare slightly larger than the widths of the original peaks P3and P4. However, since the widths of the peaks P3and P4are small, the widths of the adjusted peaks P3xand P4xcan be smaller than the widths of the corresponding peaks P1and P2in the reference color distribution (FIG.11B).

FIG.11Fillustrates an example of a histogram for the absolute values of differences in luminance values for the same pixels in the gray captured image620and the gray original image720(|VA1−VA2|).FIG.11Gillustrates an example of a histogram for the absolute values of differences in luminance values for the same pixels in the first pre-processed captured image630and the first pre-processed original image730(|VB1−VB2|). As illustrated in the graphs, the difference in color-adjusted luminance values (FIG.11G) is smaller than the difference in luminance values when no color adjustments have been performed (FIG.11F). Hence, the difference in color distribution between the first pre-processed captured image630and the first pre-processed original image730is smaller than the difference in color distribution between the gray captured image620and the gray original image720. As will be described below, image data representing a color-adjusted captured image is inputted into the image generation model300in order to detect the scratch image801(seeFIG.9) in the color-adjusted captured image, i.e., the scratch in the label sheet800. As described inFIGS.5and6, sets of training image data I1tdgenerated from the gray original image data720dare inputted into the image generation model300to train the image generation model300. Since the color adjustment process adjusts the color distribution in each captured image inputted into the image generation model300for inspection to be closer to the color distribution in the training image, the image generation model300can produce a suitable generated image.

In S330ofFIG.7, the processor210executes a second pre-process. Step S330includes a pre-process (S332) performed on the first pre-processed captured image data630d, and a pre-process (S340) performed on the first pre-processed original image data730d.

Step S332includes steps S334and S336. In S334the processor210executes a blur reduction process on the first pre-processed captured image data630d. The blur reduction process reduces the motion blur described in S160ofFIG.5. The blur reduction process is also referred to as “de-blurring”. Any of various blur reduction processes may be employed. In the present embodiment, the processor210determines a point spread function using the motion vector described in S160(and specifically, the angle and length). The processor210then generates a Wiener filter using the point spread function. By applying the Wiener filter in the frequency domain, the processor210performs filtering on the first pre-processed captured image data630dto generate blur-reduced captured image data.

In S336the processor210performs a smoothing process on the blur-reduced captured image data. The smoothing process is also known as a noise reduction process. For the smoothing process in the present embodiment, the processor210performs a filtering process using a mean filter. The size of the kernel in the mean filter is set experimentally in advance in order to reduce noise appropriately.

Through S334and S336described above (i.e., S332), the processor210generates second pre-processed captured image data640dfrom the first pre-processed captured image data630d. The fourth image from the top of the left column inFIG.9is an example of a second pre-processed captured image640represented by the second pre-processed captured image data640d. The second pre-processed captured image640includes an image640aof a label sheet800identical to the label sheet800corresponding to the image630ain the first pre-processed captured image630. However, the second pre-processed captured image640has less noise and blur than the first pre-processed captured image630.

Step S340ofFIG.7includes steps S342, S344, and S346. The process of S342and S344is performed to add noise to the first pre-processed original image730represented by the first pre-processed original image data730dand overall is similar to the noise adding process described in S150ofFIG.5. Specifically, in S342the processor210randomly generates a noise value (also called “Gaussian noise”) for each pixel according to a Gaussian distribution (Gaussian noise generation process). In S344the processor210combines the first pre-processed original image data730dwith the Gaussian noise to produce noise-added original image data (combining process).

In S346the processor210performs a blurring process on the noise-added original image data. The blurring process of S346is identical to the blurring process described in S160ofFIG.5.

Through S342-S346described above (i.e., S340), the processor210generates second pre-processed original image data740dfrom the first pre-processed original image data730d. The fourth image from the top of the right column inFIG.9is an example of a second pre-processed original image740represented by the second pre-processed original image data740d. The second pre-processed original image740includes a design label image740a, which is the design label image730ain the first pre-processed original image730with added noise and blur.

In S350ofFIG.8, the processor210performs an outline matching process. Step S350includes steps S352-S362. In S352the processor210analyzes the second pre-processed captured image data640d(seeFIG.9) to extract edges (edge extraction process). In S354the processor210uses the extracted edges to calculate coordinates in the second pre-processed captured image640of four predetermined portions in the image640aof the label sheet800. In the present embodiment the predetermined portions correspond to the four corners of the label sheet800. Any method may be used for extracting the edges. For example, the processor210calculates an edge amount for each pixel in the second pre-processed captured image640using a Laplacian filter and extracts pixels that have an edge amount greater than a prescribed edge threshold as edge pixels. Any method may be used to calculate the coordinates for each of the four portions. For example, the processor210may apply a thinning process, such as Hilditch's thinning algorithm, to produce thinner edges. The processor210then applies the Hough transform using the thinned edges to detect four straight lines that outline a portion in the second pre-processed captured image640corresponding to the label sheet800(image640a). From these four lines, the processor210calculates the coordinates of the four intersecting points (i.e., the four corners).

Steps S356and S358are identical to the respective steps S352and S354, except that the second pre-processed original image data740dis used in place of the second pre-processed captured image data640d. The processor210calculates the coordinates in the second pre-processed original image740for the four corners of the design label image740a(seeFIG.9).

In S360ofFIG.8, the processor210determines the projection function for a projective transformation using the coordinates of the four portions in the second pre-processed captured image640(FIG.9) and the coordinates of the four portions in the second pre-processed original image740. This projection function is used to match the outline of the design label image740ain the second pre-processed original image740with the outline of the image640aof the label sheet800in the second pre-processed captured image640. The outline is the shape of the image's contour. In S362the processor210performs a projective transformation of the second pre-processed original image data740daccording to the projection function to generate transformed original image data750d. The fifth image from the top of the right column inFIG.9is an example of a transformed original image750represented by the transformed original image data750d, which has been generated from the second pre-processed captured image data640dand second pre-processed original image data740d. The transformed original image750includes a design label image750a, which is an image projected from the design label image740ain the second pre-processed original image740. The outline of the design label image750ain the transformed original image750is identical to the outline of the image640aof the label sheet800in the second pre-processed captured image640.

Note that any method may be used for projective transformation (S360and S362). For example, the OpenCV (Open Source Computer Vision Library) projective transformation function may be used.

In S372ofFIG.8, the processor210performs a resizing process on the second pre-processed captured image data640d(seeFIG.9) to generate resized second pre-processed captured image data645d(hereinafter called “processed captured image data645d”). This resizing process converts the size of an image represented by the second pre-processed captured image data640d(and specifically the number of pixels in the first direction Dx and the number of pixels in the second direction Dy) to a size that can be inputted into the image generation model300. Any of various resolution converting processes may be used as the resizing process. The fifth image from the top of the left column inFIG.9is an example of a processed captured image645represented by the processed captured image data645d. The processed captured image645is identical to the second pre-processed captured image640, except that the resolution is different. The processed captured image645includes an image645a, which is identical to the image640aexcept for the resolution. Note that the resizing process may be performed not only on the second pre-processed captured image data640d, but also on any of the image data610d,620d,630d, and640dthat represent an image containing a captured image of the label sheet800.

In S374the processor210inputs the processed captured image data645d(seeFIG.9) into the image generation model300(seeFIG.4A) to produce generated image data650d(generation process).FIG.10illustrates an example of a generated image650represented by the generated image data650d, which has been generated from the processed captured image data645d. The generated image650includes an image650aof the label sheet800with no defects (without the scratch (scratch image801) in this example). As described inFIGS.5and6, the image generation model300has been trained to produce generated image data representing an image of the defect-free label sheet800. Hence, even when the processed captured image data645dinputted into the image generation model300represents an image of the label sheet800with the scratch (an image containing the scratch image801), the generated image data650dproduced by the image generation model300represents an image of the label sheet800without the scratch (an image without the scratch image801).

In S378ofFIG.8, the processor210generates difference image data660dshowing the differences between the generated image data650dand the processed captured image data645dinputted into the image generation model300.FIG.10illustrates an example of an image660represented by the difference image data660d(hereinafter called the “difference image660”). Each pixel in the difference image660indicates the difference in color values (luminance values in this case) between the corresponding pixels in the two images645and650. The difference image660includes an image660arepresenting differences in pixels contained in the image of the label sheet800. Of the plurality of pixels in the image660a, pixels contained in the scratch image801shows the largest difference. The remaining pixels in the image660ashow smaller differences (e.g., values close to zero).

In S379ofFIG.8, the processor210performs a resizing process on the difference image data660dto generate resized difference image data665d. This resizing process matches the resolution of the difference image data660dto the resolution of the transformed original image data750d. The resized difference image (not illustrated) represented by the resized difference image data665dis identical to the difference image660represented by the difference image data660d(seeFIG.10), except that the resolution is different. The portion of the resized difference image corresponding to the object image800iand the design label image750ain the transformed original image750(seeFIG.9) have the same resolution and the same outline. This resizing process may be the inverse of the resolution conversion performed in the resizing process of S372, for example. Alternatively, the processor210may analyze the difference image660and the transformed original image750and perform a resizing process that gives the portion of the resized difference image corresponding to the object image800ithe same resolution and same outline as the design label image750a.

In S380ofFIG.8, the processor210generates superimposed image data670drepresenting a superimposed image of the resized difference image (identical to the difference image660) and the transformed original image750by combining the resized difference image data665dand the transformed original image data750d.FIG.10illustrates an example of a superimposed image670represented by the superimposed image data670d. The superimposed image670includes a superimposed label image670a, which is a superimposed image of the design label image750a(seeFIG.9) and the portion of the resized difference image corresponding to the object image800i(identical to the image660ain the difference image660ofFIG.10). The superimposed label image670ashows both the image corresponding to the design image800iand the scratch image801.

In S382ofFIG.8, the processor210performs a projective transformation of the superimposed image data670dto generate transformed superimposed image data680d. Here, the processor210performs the inverse of the projective transformation applied in S350.FIG.10illustrates an example of an image680represented by the transformed superimposed image data680d(hereinafter called the “transformed superimposed image680”). The transformed superimposed image680includes a superimposed label image680a. The superimposed label image680ais obtained through a projective transformation of the superimposed label image670a. The outline of the superimposed label image680ais the same as the outline of the design label image740a, (i.e., the outline of the design image800iin the original image710) in the second pre-processed original image740(seeFIG.9), which is the processing target of S350. The superimposed label image680ais not skewed relative to the transformed superimposed image680.

In S384ofFIG.8, the processor210uses the transformed superimposed image data680dto generate output data690d.FIG.10illustrates an example of an image690represented by the output data690dgenerated using the transformed superimposed image data680d(hereinafter called the “output image690”). The output image690includes a superimposed label image690a. The superimposed label image690ashows the superimposed label image680aand a frame image699depicting a frame around the scratch depicted by the scratch image801. In the present embodiment, the processor210analyzes the resized difference image to detect a defective portion having a plurality of contiguous pixels indicating a difference greater than or equal to the prescribed difference threshold and superimposes an image of a frame surrounding the detected defective portion on the transformed superimposed image680. When the label sheet800has no defects, the processor210does not detect a defective portion in the resized difference image and, hence, the output image690is substantially the same as the second pre-processed original image740(seeFIG.9).

In S386ofFIG.8, the processor210performs a results outputting process using the output data690d. In the present embodiment, the processor210displays the output image690represented by the output data690don the display unit240(seeFIG.1). When viewing the display unit240, the operator can easily discern whether the label sheet800has any defects in appearance. When the label sheet800has any defects, the operator can easily determine the locations of the defects on the label sheet800based on the displayed frames (e.g., the frame image699inFIG.10). Moreover, the operator can easily find these defects (e.g., the scratch corresponding to the scratch image801) on the actual label sheet800by comparing the displayed output image690to the actual label sheet800. Here, the output image690, like the captured image, has noise and blur. Therefore, any sense of unease that the operator may experience when visually comparing the output image690to the actual label sheet800can be reduced. After completing the process in S386, the processor210ends the inspection process ofFIGS.7and8. The operator may perform various processes based on the outputted results. For example, when the label sheet800has a defect, the operator may remove the MFP900having this label sheet800from the production line for MFPs900.

In the inspection process ofFIGS.7and8described in the above embodiment, the processor210(FIG.1) generates the transformed original image data750dfrom the original image data710dby executing a first image process S910comprising steps S324, S328, S340, and S350(FIGS.7and8). In other words, by executing the first image process S910using the original image data710d, the processor210generates the transformed original image data750d(hereinafter referred to as the “processed original image data750d”). Here, the original image data710d(seeFIG.9) is image data representing the object image800ito be printed.

The processor210generates the processed captured image data645dfrom the captured image data610dby executing a second image process S920comprising steps S322, S326, S332, and S372(FIGS.7and8). In other words, by executing the second image process S920using the captured image data610d, the processor210generates the processed captured image data645d. Here, the captured image data610d(FIG.9) represents a captured image610aof a label sheet800, and the label sheet800is produced by printing an object image800i, that is, the label sheet800has a printed object image800i. Hereinafter, the captured image610awill be also called the captured object image610a, and the printed object image800iwill be also called the printed object800.

The processor210generates the output data690dfrom the processed original image data750dand processed captured image data645dby executing a third image process S990comprising steps S374, S378, S379, S380, S382, and S384(FIG.8). As illustrated inFIG.10, the output data690dis an example of data related to defects in the appearance of a printed object800.

Here, the first image process S910(FIGS.7and8) includes a first process S810(and specifically S328, S342, S344, and S346ofFIG.7and S350ofFIG.8), which is not included in the second image process S920. Various processes suitable for generating the processed original image data750dfrom the original image data710dmay be employed as the first process. Additionally, the second image process S920includes a second process S820(and specifically S326, S334, and S336ofFIG.7and S372ofFIG.8), which is not included in the first image process S910. Various processes suitable for generating the processed captured image data645dfrom the captured image data610dmay be employed as the second process. Hence, by using suitably generated processed original image data750dand suitably generated processed captured image data645d, the processor210can generate suitable output data690drelated to defects in the appearance of a printed object800.

The first process S810included in the first image process S910(FIGS.7and8) includes a pre-process (S340) involving both a noise adding process (S342and S344) and a blurring process (S346). Therefore, the processed original image650represented by the processed original image data750dhas noise and blur, just as in the captured image represented by the captured image data. By using the processed original image data750dand processed captured image data645d, the processor210can generate suitable output data690drelated to defects. For example, the output image690may have noise and blur, just like the processed original image (transformed original image)750represented by the processed original image data750d. Therefore, the unease experienced by an operator observing the output image690and the actual label sheet800can be reduced.

Further, the object image800irepresented by the original image data710d(FIG.9) is a design image of the label sheet800. The processor210generates the processed original image data750dby executing the first image process S910(FIGS.7and8) on such original image data710d. The processor210also generates the processed captured image data645dby executing the second image process S920on the captured image data610drepresenting the captured object image610a, which is a photographed image of the label sheet800. By subsequently executing the third image process S990using the processed original image data750dand processed captured image data645d, the processor210generates the output data690d, which is data related to defects in the appearance of the label sheet800. As described above, the third image process S990includes a process (S374) using the image generation model300that has been trained. Additionally, the first image process S910includes the first process S810, which is not included in the second image process S920, while the second image process S920includes the second process S820, which is not included in the first image process S910. In the present embodiment, the first process S810includes a pre-process (S340) that involves both a noise adding process (S342and S344) and a blurring process (S346). The processor210also trains the image generation model300using sets of training image data I1td. The sets of training image data I1tdare generated when the processor210executes a generation process S930on the original image data710d. The generation process S930is an image process including steps S130-S160inFIG.5. The generation process S930includes a process (S140) identical to the pre-process ofFIG.7(S340). Thus, the same degree of noise and the same degree of blur are added to the images in the generation process S930for generating the training image data I1tdand the first image process S910for generating the processed original image data750d. This configuration improves consistency between the first image process S910and the process that uses the trained image generation model300(and specifically S374). By using image data750dand650dgenerated according to these processes S910and S374, the processor210can generate suitable output data690drelated to defects in the appearance of the label sheet800.

As described inFIGS.4A,4B,5, and6, the image generation model300is also trained to produce sets of generated image data I1xdrepresenting generated images I1xfrom sets of training image data I1tdrepresenting training images I1t. Each generated image I1xcontains an image I1xaof a defect-free label sheet800corresponding to an image I1taof the label sheet800contained in the respective training image I1t. With this configuration, the processor210can generate data representing images of defects in the label sheet800(the difference image data660din this embodiment) using image data generated by the image generation model300. The processor210uses such difference image data660dto generate suitable output data690d.

The first process S810in the first image process S910(FIG.7) also includes a first color adjustment process (S328). The first color adjustment process adjusts the color distribution in the gray original image720represented by the gray original image data720dbeing processed to be closer to the color distribution in the gray capture image620represented by the gray captured image data620dbased on the captured image data610d. The second process S820in the second image process S920includes the second color adjustment process (S326). The second color adjustment process adjusts the color distribution in the gray captured image620represented by the gray captured image data620dbeing processed to be closer to the color distribution in the gray original image720represented by the gray original image data720dbased on the original image data710d. These color adjustment processes (S326and S328) reduce the difference in color distribution between the first pre-processed captured image630(FIG.9) and the first pre-processed original image730. Hence, the difference in color distribution between images represented by image data750dand645dgenerated through the respective image processes S910and S920is reduced. The processor210can then generate suitable output data690dusing this image data750dand645d.

The second process S820in the second image process S920(FIGS.7and8) includes a pre-process (S332) that involves both a noise reduction process (S336) and a blur reduction process (S334). Therefore, when the captured label image610has an unexpectedly large amount of noise or blur, the processor210can mitigate the effects of such noise and blur on the output data690d. This can reduce the possibility that the processor210will generate output data690dthat does not show any defects when the label sheet800in fact has defects, for example. Thus, by using the processed original image data750dand the processed captured image data645d, the processor210can generate suitable output data690drelated to defects.

The first process S810in the first image process S910(FIG.8) includes an outline matching process (S350). The process of S350is performed to match the outline of a portion of the second pre-processed original image740represented by the second pre-processed original image data740dbeing processed (FIG.9) that corresponds to the object image800i(the design label image740ain this case) with the outline of the image640aof the label sheet800in the second pre-processed captured image640(identical to the outline of the captured target image610ain the captured label image610). Through this process, the processor210can reduce differences in the outlines of the image of the label sheet800between the image data750dand645dgenerated through the respective image processes S910and S920. By using this image data750dand645d, the processor210can generate suitable output data690d.

The first process S810in the first image process S910(FIGS.7and8) includes a first color adjustment process (S328), a pre-process (S340), and an outline matching process (S350). The first color adjustment process (S328) is performed to adjust the color distribution in the gray original image720represented by the gray original image data720dbased on the original image data710d(FIG.9) to be closer to the color distribution in the gray captured image620represented by the gray captured image data620dbased on the captured image data610d. The pre-process (S340) is performed on the first pre-processed original image data730dproduced from the first color adjustment process (S328). The outline matching process (S350) is performed to match the outline of a portion of the second pre-processed original image740represented by the second pre-processed original image data740dproduced from the pre-process (S340) that corresponds to the object image800i(the design label image740ain this case) with the outline of the image640aof the label sheet800in the second pre-processed captured image640(identical to the outline of the captured target image610ain the captured label image610). This configuration reduces differences between the image data750dand645dgenerated through the respective image processes S910and S920, excluding any defects in the appearance of the label sheet800(the scratch corresponding to the scratch image801, etc.), and specifically differences in color distribution, noise, blurring, and the outline of the image of the label sheet800. By using this image data750dand645d, the processor210can generate suitable output data690d.

The third image process S990ofFIG.8includes S374, S378, S379, S380, S382, and S384. In S374and S378, the processor210generates difference image data660drepresenting a difference image660. The difference image660depicts portions that differ between the processed captured image645represented by the processed captured image data645dand an image of a defect-free label sheet800(the generated image650represented by the generated image data650din this case). In S379the processor210resizes the difference image660represented by the difference image data660dto generate the resized difference image data665d. In S380the processor210combines the resized difference image data665dwith the processed original image data750dto produce superimposed image data670drepresenting a superimposed image of the resized difference image (identical to the difference image660) and the transformed original image750. In S382and S384, the processor210uses the superimposed image data670dto generate the output data690d. In the present embodiment, in S382the processor210matches the outline of the portion670acorresponding to the object image800iin the super imposed image670represented by the superimposed image data670dwith the outline of the object image800iin the original image710represented by the original image data710d(FIG.9). The outline of the portion680aof the transformed superimposed image680corresponding to the design image800iis identical to the outline of the design image800i. In S384the processor210generates output data690drepresenting the output image690that depicts a superimposed image of the transformed superimposed image680and an image of a frame (e.g., the frame image699). The transformed superimposed image680depicts a superimposed image of the resized difference image represented by the resized difference image data665d(identical to the difference image660represented by the difference image data660d) and the transformed original image750represented by the processed original image data750d. Accordingly, the processor210can generate suitable output data690dshowing defective portions of the label sheet800.

B. Modifications of the Embodiment

While the invention has been described in conjunction with various example structures outlined above and illustrated in the figures, various alternatives, modifications, variations, improvements, and/or substantial equivalents, whether known or that may be presently unforeseen, may become apparent to those having at least ordinary skill in the art. Accordingly, the example embodiments of the disclosure, as set forth above, are intended to be illustrative of the invention, and not limiting the invention. Various changes may be made without departing from the spirit and scope of the disclosure. Therefore, the disclosure is intended to embrace all known or later developed alternatives, modifications, variations, improvements, and/or substantial equivalents. Some specific examples of potential alternatives, modifications, or variations in the described invention are provided below:(1) The filter used in the noise reduction process of S336(FIG.7) may be any of various filters in place of the mean filter, such as a Gaussian filter or a median filter. The noise reduction process may also be any of various processes for smoothing color values instead of a process using a filter.(2) The noise adding process in S150ofFIG.5and S342and S344ofFIG.7may be various other processes instead of the process described in the embodiment. For example, the noise adding process may superimpose a noise image indicating the noise value for each pixel on the image being processed. The noise image may be determined experimentally in advance. The processor210may randomly select a noise image to be superimposed on the image being processed from among a plurality of prepared noise images. In either case, the noise adding process employed in S340ofFIG.7is preferably the same as the noise adding process employed in the process for generating training image data inFIG.5, but the processes may be different from each other.(3) Any method may be used to determine the motion vector used in the blur reduction process of S334(FIG.7). For example, a speed sensor may be fixed to the MFP900, and the motion vector may be determined on the basis of the speed of the MFP900when the MFP900is being photographed for inspection. The blur reduction process may be implemented using any other method in place of the above method using a point spread function and Wiener filter. For example, the blur reduction process may be performed using a procedure known as Lucy-Richardson deconvolution. This method can reduce blur based on a point spread function. Another possibility is an algorithm called “interactive back projection” or a technique employing a convolutional neural network called “Deep Generative Filter for Motion Deblurring.” In addition to motion blur, a process may be performed to reduce blur caused by the focal point of the camera deviating from the label sheet800. For example, blur may be reduced through a process known as “Image deblurring” in the Python library called “PyLops”.(4) The blurring process in S160ofFIG.5and S346ofFIG.7may be achieved through various other processes instead of the process in the above embodiment. For example, the blurring process may include smoothing with a smoothing filter.(5) The four portions used in the projective transformation of S350(FIG.8) are not limited to the four corners of the label sheet800but may be any portions of the label sheet800. Additionally, any of various methods for calculating the coordinates of the four portions may be employed instead of the process in the above embodiment. For example, the coordinates of each of the four portions may be determined through pattern matching using four reference images representing the four portions.(6) The inspection process may be various other processes instead of the process inFIGS.7and8. For example, the first image process S910may include a resizing process similar to S372performed on the processed original image data750d. Additionally, execution of the plurality of processes in the first image process S910(the grayscale conversion in S324, the first color adjustment in S328, the noise addition in S342and S344, the blurring in S346, and the projective transformation in S350) may be performed in various other orders. For example, the first color adjustment in S328may be performed on image data produced through the pre-process of S340. The first image process S910and the second image process S920form three pairs PP1-PP3of image processes having the same focus, as indicated below.(PP1) S324, S322(color space)(PP2) S328, S326(color distribution)(PP3) S340, S332(image degradation (noise and blur))

In order to improve consistency between the image processes S910and S920, the elements of these three pairs PP1-PP3are preferably performed in the same order between the image processes S910and S920. For example, if S328(first color adjustment) is executed after S340(pre-process) in the first image process S910, S326(second color adjustment) is preferably executed after S332(pre-process) in the second image process S920.

One or more of the plurality of processes in the first image process S910(the grayscale conversion in S324, the first color adjustment in S328, the noise addition in S342and S344, the blurring in S346, and the projective transformation in S350) may be omitted. For example, when positioning accuracy between the MFP900(FIG.2) and the digital camera110is good during photographing, i.e., when there is little positional deviation of the label sheet800within the captured image, the projective transformation of S350may be omitted. In this case, the projective transformation of S382may also be omitted. S382may be omitted regardless of whether S350is omitted. Alternatively, S350may be omitted and instead the second image process S920may include the same process of projective transformation described in S382. In this case, the image645aof the label sheet800in the processed captured image645generated through the second image process S920will have the same outline as the design image800iin the original image710. (For example, the image645aof the label sheet800will not be skewed relative to the processed captured image645.) In S380the processor210generates superimposed image data670drepresenting a non-skewed image of the label sheet800. In this case, S382may be omitted. Similarly, one or more of the plurality of processes in the second image process S920(the grayscale conversion in S322, the second color adjustment in S326, the blur reduction in S334, the noise reduction in S336, and the resizing in S372) may be omitted. However, when a process forming one of the above pairs PP1-PP3is omitted from the first image process S910, the corresponding process is preferably omitted from the second image process S920, but it is also possible to omit just one of the two processes forming a pair. For example, one or both of S328(first color adjustment) and S326(second color adjustment) may be omitted.(7) The pre-processes of S340and S332(FIG.7) may be implemented in various ways. For example, one of the noise addition of S342and S344and the blurring of S346may be omitted from the pre-process of S340. When the noise addition of S342and S344is omitted, it is preferable also to omit the noise reduction of S336from the pre-process of S332, which targets the same perspective. When the blurring of S346is omitted, it is preferable also to omit the blur reduction of S344from the pre-process of S332, which targets the same quality. Further, one or both of the pre-processes of S340and S332may be omitted.(8) The first process S810in the first image process S910(FIGS.7and8) may be configured of various processes not included in the second image process S920. For example, one or more of the first color adjustment of S328, the noise addition of S342and S344, the blurring of S346, and the projective transformation of S350may be omitted. Similarly, the second process S820in the second image process S920may be configured of various processes not included in the first image process S910. For example, one or more of the second color adjustment of S326, the blur reduction of S334, the noise reduction of S336, and the resizing of S372may be omitted.(9) The color adjustment processes of S328and S326(FIG.7) may be implemented in various ways. For example, the image data to undergo the first color adjustment process of S328is not limited to the gray original image data720dbut may be various image data based on the original image data710d. Here, image data based on the original image data710dis either the original image data710ditself or image data obtained by performing an image process on the original image data710d. The first color adjustment process of S328may be performed on image data generated from the pre-process of S340, for example. Further, the reference color distribution used in the first color adjustment process of S328is not limited to the color distribution in the gray captured image620represented by the gray captured image data620dbut may be the color distribution of various image represented by image data based on the captured image data610d. Here, image data based on the captured image data610dis either the captured image data610ditself or image data obtained by performing an image process on the captured image data610d. The reference color distribution used in the first color adjustment process of S328may also be the color distribution in an image represented by image data generated in the pre-process of S332, for example. The color space of the reference color distribution is preferably the same as the color space of the image data being processed. When the color space is represented by a plurality of color components, histogram matching may be performed for each color component.

Similarly, image data to be subjected to the second color adjustment process of S326may be various image data based on the captured image data610din place of the gray captured image data620d. The reference color distribution used in the second color adjustment process of S326is also not limited to the color distribution in the gray original image720represented by the gray original image data720dbut may be the color distribution of various image represented by image data based on the original image data710d. In the second color adjustment process of S326, the color space of the reference color distribution is also preferably the same as the color space of the image data being processed.

As in the example ofFIG.7, the image data to be subjected to the first color adjustment process of S328preferably indicates the reference color distribution used in the second color adjustment process of S326, and the image data to be subjected to the second color adjustment process of S326preferably indicates the reference color distribution to be used in the first color adjustment process of S328. This configuration improves consistency between the first color adjustment process of S328and second color adjustment process of S326.

Instead of the process described inFIG.11A, various other processes may be used to adjust the color distribution in the image represented by the image data being processed to be closer to the reference color distribution. For example, a tone curve adjustment process may be performed to adjust representative color values of the image data being processed (e.g., median values of a plurality of color values for a plurality of pixels) to be closer to the representative color values of image data representing the reference color distribution.(10) The process for generating training image data, i.e., the training image data generation process may be various processes and is not limited to the process described inFIG.5. For example, various processes known as data augmentation, such as an image rotation process and an image moving process, may be executed. In a rotating process, the rotation angle may be randomly set for each set of training image data I1td. In a moving process, the direction and amount of movement may be randomly set for each set of training image data I1td. Further, the training image data generation process may include processes randomly set for each set of training image data I1td. For example, processes may be randomly selected from among a plurality of processes including a noise adding process, a blurring process, an image rotating process, and an image moving process. One or both of the noise adding process and blurring process are preferably included in the training image data generation process and the first process S810(FIG.7). Further, the training image data may be generated using captured image data of the label sheet800instead of the gray original image data720d. The processor210may generate a plurality of sets of training image data through data augmentation using the captured image data. One or both of the noise adding process and the blurring process on the captured image data may be omitted.(11) The label sheet800(seeFIG.2) may be provided on any product and not just the MFP900, such as a sewing machine, a cutting machine, or a portable terminal. Further, the object for inspection need not be the label sheet800but may be any product, such as a multifunction peripheral, a sewing machine, a cutting machine, or a portable terminal. Here, the entire product or a portion of the product may be inspected.

The object image represented by the original image data is not limited to an image to be printed but may be any suitable image depicting the appearance of the object for inspection. For example, the object image may be engineering drawing of the object or a photographed image of the appearance of the object. The object image is preferably an image to be printed or a design image of a target object, such as engineering drawing. With this configuration, the processor210can generate suitable output data related to defects in the appearance of an object.(12) The image generation model300is preferably trained to generate an image of an object with no defects (defect-free object image) based on captured images of the object. With this configuration, the processor210can use image data inputted into the image generation model300, image data generated by the image generation model300, and processed original image data generated through the first image process to generate suitable output data related to defects.

Instead of a variational autoencoder, the image generation model300may be any of various other models that generate image data using image data, such as an autoencoder or generative adversarial network. A variety of processes suited to the image generation model300may be used in the training process for training the image generation model300. For example, the plurality of model parameters in the image generation model300may be adjusted as follows. Specifically, the image generation model300may be trained to reduce the difference between generated image data produced by inputting image data of a defect-free object into the image generation model300and the image data inputted into the image generation model300. With this configuration, the trained image generation model300can generate an image of a defect-free object (defect-free object image) when an image of an object having defects is inputted into the image generation model300.

The image data inputted into the image generation model300may be color image data. The image generation model300may be configured to generate color image data from color image data. The image generation model300may also be configured to generate grayscale image data from color image data. Note that the process using the image generation model300(S374ofFIG.8) may be omitted. For example, the processor210may detect an object in the processed captured image645represented by the processed captured image data645dthrough pattern matching using a reference object image, which is a reference image of a defect-free object. In S378the processor210may then generate difference image data showing the differences between the reference object image arranged at the position of the detected object, and the image represented by the processed captured image data645d.(13) The data process for inspection is not limited to the processes in the embodiment and modifications described above, and may be any of various processes. For example, the data process for inspection may be performed using color image data and not grayscale image data. Alternatively, the data process for inspection may be performed using grayscale image data and not color image data. Further, the third image process for generating output data may be any of various processes in place of the third image process S990ofFIG.8. For example, a resizing process may be performed on image data based on the original image data710d(e.g., the processed original image data750d) instead of the difference image data660d(S379) in order to superimpose the difference image on the transformed original image750. The output data690dmay also be any data that specifies defects possessed by an object when the object possesses such defects rather than image data representing the output image690(FIG.10). For example, the output data may represent image data depicting defective areas in color and other areas in grayscale. However, the output data is not limited to image data and may be data indicating any information about defects, such as the total number of defects.

In any case, the processor210may perform a process to output data related to defects using the output data. The method of outputting data related to defects is not limited to the method of S386(FIG.8) and may be any method. For example, the processor210may output output data specifying inspection results to a storage device (e.g., the nonvolatile storage device230or an external storage device not illustrated in the drawings that is connected to the data processing apparatus200). In this case, the data processing apparatus200stores data specifying the inspection results in the storage device. Alternatively, the processor210may output an image represented by the output data to the display unit240, in which case the inspection results are displayed on the display unit240.(14) The data processing apparatus200inFIG.1may be various devices other than a personal computer, such as a digital camera or a smartphone. Further, a plurality of devices that can communicate over a network (e.g., computers) may each implement some of the functions in the data process performed by the data processing apparatus so that the devices as a whole can provide the functions required for the data process. (Here, the system comprising these devices corresponds to the data processing apparatus.)

Part of the configuration implemented in hardware in the embodiment described above may be replaced with software and, conversely, all or part of the configuration implemented in software may be replaced with hardware. For example, the functions of the color adjustment processes inFIG.7(S326and S328) may be implemented with a dedicated hardware circuit.

When some or all of the functions of the present disclosure are implemented with computer programs, the programs may be stored on a computer-readable storage medium (e.g., a non-transitory storage medium). The programs may be used on the same storage medium (a computer-readable storage medium) on which they have been supplied or may be transferred to and used on a different storage medium (a computer-readable storage medium). A “computer-readable storage medium” may be a portable storage medium, such as a memory card or CD-ROM; an internal storage device built into the computer, such as any of various ROM; or an external storage device, such as a hard disk drive, connected to the computer.