Patent ID: 12214601

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

Digital printing systems comprise parts, such as nozzles, for jetting printing fluids onto a substrate, so as to produce an image thereon. In some cases, a nozzle may jet the printing fluid incorrectly, due to clogging or any other defect therein, resulting in forming a distortion in the printed image. In principle, it is possible to print a testing job for detecting a defective nozzle, but such testing jobs reduce the production time of the printing system, involve in redundant cost of printing consumables, and in waste production. Moreover, such testing jobs can be performed periodically, and therefore, cannot detect a defective nozzle in real time.

Embodiments of the present invention that are described hereinbelow provide methods and system for detecting, during production, a defective nozzle in a digital printing system (DPS) comprising an array of such nozzles.

In some embodiments, the DPS comprises a processor which is configured to detect the defective nozzle using a convolutional neural network (CNN) applied to a first digital image (FDI) to be printed by the DPS, and a second digital image (SDI) acquired from a printed image produced by the DPS. At least one the FDI and SDI is a product image printed during production, typically both.

In some embodiments, in a training phase of the CNN, the processor is configured to produce, for selected regions in the FDI, a first set of synthetic images (SIs), each SI is associated with a respective selected region and having a simulated missing nozzle fault (MNF) caused by a respective defective nozzle of the array. The processor is configured to train the CNN to detect the MNF using at least some of the SIs of the first set.

In some embodiments, in the detection phase that is subsequent to the training phase, the processor is configured to apply the trained CNN for identifying, in the SDI, one or more regions suspected of containing the MNF. Based on the printing plan of the SDI, the processor holds a list of nozzles participating in the printing of the suspected regions. The processor is configured to produce, for each of the suspected regions, a second set of SIs having one or more MNFs produced by one or more respective defective nozzles of the list.

In some embodiments, the processor is configured to identify at least the defective nozzle by comparing, in each of the suspected regions, between the SDI and the respective SIs of the second set.

The disclosed techniques improve the quality of printed digital images by real-time identification of defective nozzles, and thereby preventing MNFs in subsequent printed images. In the context of the present disclosure and in the claims, the term real-time identification refers to identifying defective nozzle by detecting the MNF immediately after being printed on the substrate. Moreover, by detecting MNFs and identifying defective nozzles during production, the disclosed techniques improve the utilization of a DPS for production, and reduce waste of substrates and printing fluids.

System Description

FIG.1is a schematic side view of a digital printing system10, in accordance with an embodiment of the present invention. In some embodiments, system10comprises a rolling flexible blanket44that cycles through an image forming station60, a drying station64, an impression station84and a blanket treatment station52. In the context of the present invention and in the claims, the terms “blanket” and “intermediate transfer member (ITM)” are used interchangeably and refer to a flexible member comprising one or more layers used as an intermediate member configured to receive an ink image and to transfer the ink image to a target substrate, as will be described in detail below. Moreover, embodiments of the present invention that are described below, are also applicable to printing systems using one or more drum as ITM, instead of or in addition to blanket44.

In an operative mode, image forming station60is configured to form a mirror ink image, also referred to herein as “an ink image” (not shown) or as an “image” for brevity, of a digital image (DI)42on an upper run of a surface of blanket44. Subsequently the ink image is transferred to a target substrate, (e.g., a paper, a folding carton, a multilayered polymer, or any suitable flexible package in a form of sheets or continuous web) located under a lower run of blanket44.

In the context of the present invention, the term “run” refers to a length or segment of blanket44between any two given rollers over which blanket44is guided.

In some embodiments, during installation blanket44may be adhered edge to edge to form a continuous blanket loop (not shown). An example of a method and a system for the installation of the seam is described in detail in U.S. Provisional Application 62/532,400, whose disclosure is incorporated herein by reference.

In some embodiments, image forming station60typically comprises multiple print bars62, each mounted (e.g., using a slider) on a frame (not shown) positioned at a fixed height above the surface of the upper run of blanket44.

Reference is now made to an inset11, showing print bar62. In some embodiments, each print bar62comprises a strip of print heads (not shown) as wide as the printing area on blanket44and an array of individually controllable print nozzles99, each of which configured to apply (e.g., by jetting and/or directing) a printing fluid toward a predefined position on blanket44that is moved by system10.

Reference is now made back to the general view ofFIG.1. In some embodiments, image forming station60may comprise any suitable number of print bars62, each print bar62may contain the printing fluid, such as an aqueous ink of a different color. The ink typically has visible colors, such as but not limited to cyan, magenta, red, green, blue, yellow, black and white. In the example ofFIG.1, image forming station60comprises seven print bars62, but may comprise, for example, four print bars62having any selected colors such as cyan (C), magenta (M), yellow (Y) and black (K).

In some embodiments, the print heads are configured to jet ink droplets of the different colors onto the surface of blanket44so as to form the ink image (not shown) on the surface of blanket44.

In some embodiments, different print bars62are spaced from one another along the movement axis, also referred to herein as moving direction of blanket44, represented by an arrow94. In this configuration, accurate spacing between print bars62, and synchronization between directing the droplets of the ink of each print bar62and moving blanket44are essential for enabling correct placement of the image pattern.

In the context of the present disclosure and in the claims, the terms “inter-color pattern placement,” “pattern placement accuracy,” color-to-color registration,” “C2C registration” “bar to bar registration,” and “color registration” are used interchangeably and refer to any placement accuracy of two or more colors relative to one another.

In some embodiments, system10comprises heaters, such as hot gas or air blowers66, which are positioned in between print bars62, and are configured to partially dry the ink droplets deposited on the surface of blanket44. This hot air flow between the print bars may assist, for example, in reducing condensation at the surface of the print heads and/or in handling satellites (e.g., residues or small droplets distributed around the main ink droplet), and/or in preventing blockage of the inkjet nozzles of the print heads, and/or in preventing the droplets of different color inks on blanket44from undesirably merging into one another. In some embodiments, system10comprises drying station64, configured to blow hot air (or another gas) onto the surface of blanket44. In some embodiments, drying station comprises air blowers68or any other suitable drying apparatus. Additionally or alternatively, system10may comprise one or more illumination assemblies, which are configured to emit infrared (IR) radiation for drying the printing fluids (e.g., ink) applied to blanket44. Such IR-emitting assemblies may be implemented, for example, in image forming station60(instead of or in addition to air blowers66), and/or in drying station64(instead of or in addition to air blowers68), and/or in other locations along blanket44.

In drying station64, the ink image formed on blanket44is exposed to radiation and/or to hot air in order to dry the ink more thoroughly, evaporating most or all of the liquid carrier and leaving behind only a layer of resin and coloring agent which is heated to the point of being rendered tacky ink film.

In some embodiments, system10comprises a blanket module70comprising a rolling ITM, such as a blanket44. In some embodiments, blanket module70comprises one or more rollers78, wherein at least one of rollers78comprises an encoder (not shown), which is configured to record the position of blanket44, so as to control the position of a section of blanket44relative to a respective print bar62. In some embodiments, the encoder of roller78typically comprises a rotary encoder configured to produce rotary-based position signals indicative of an angular displacement of the respective roller. Note that in the context of the present invention and in the claims, the terms “indicative of” and “indication” are used interchangeably.

Additionally or alternatively, blanket44may comprise an integrated encoder (not shown) for controlling the operation of various modules of system10. One implementation of the integrated encoder is described in detail, for example, in U.S. Provisional Application 62/689,852, whose disclosure is incorporated herein by reference.

In some embodiments, blanket44is guided over rollers76and78and a powered tensioning roller, also referred to herein as a dancer assembly74. Dancer assembly74is configured to control the length of slack in blanket44and its movement is schematically represented by a double sided arrow. Furthermore, any stretching of blanket44with aging would not affect the ink image placement performance of system10and would merely require the taking up of more slack by tensioning dancer assembly74.

In some embodiments, dancer assembly74may be motorized. The configuration and operation of rollers76and78are described in further detail, for example, in U.S. Patent Application Publication 2017/0008272 and in the above-mentioned PCT International Publication WO 2013/132424, whose disclosures are all incorporated herein by reference.

In some embodiments, system10may comprise one or more tension sensors (not shown) disposed at one or more positions along blanket44. The tension sensors may be integrated in blanket44or may comprise sensors external to blanket44using any other suitable technique to acquire signals indicative of the mechanical tension applied to blanket44. In some embodiments, processor20and additional controllers of system10are configured to receive the signals produced by the tension sensors, so as to monitor the tension applied to blanket44and to control the operation of dancer assembly74.

In impression station84, blanket44passes between an impression cylinder82and a pressure cylinder90, which is configured to carry a compressible blanket.

In some embodiments, system10comprises a control console12, which is configured to control multiple modules of system10, such as blanket module70, image forming station60located above blanket module70, and a substrate transport module80, which is located below blanket module70and comprises one or more impression stations as will be described below.

In some embodiments, console12comprises a processor20, typically a general-purpose processor, with suitable front end and interface circuits for interfacing with controllers of dancer assembly74and with a controller54, via a cable57, and for receiving signals therefrom. Additionally or alternatively, console12may comprise any suitable type of an application-specific integrated circuit (ASIC) and/or a digital signal processor (DSP) and/or any other suitable sort of processing unit configured to carry out any sort of processing for data processed in system10.

In some embodiments, controller54, which is schematically shown as a single device, may comprise one or more electronic modules mounted on system10at predefined locations. At least one of the electronic modules of controller54may comprise an electronic device, such as control circuitry or a processor (not shown), which is configured to control various modules and stations of system10. In some embodiments, processor20and the control circuitry may be programmed in software to carry out the functions that are used by the printing system, and store data for the software in a memory22. The software may be downloaded to processor20and to the control circuitry in electronic form, over a network, for example, or it may be provided on non-transitory tangible media, such as optical, magnetic or electronic memory media.

In some embodiments, console12comprises a display34, which is configured to display data and images received from processor20, or inputs inserted by a user (not shown) using input devices40. In some embodiments, console12may have any other suitable configuration, for example, an alternative configuration of console12and display34is described in detail in U.S. Pat. No. 9,229,664, whose disclosure is incorporated herein by reference.

In some embodiments, console12comprises a digital front-end module (DFEM)100, which is configured to carry out various computation processes of system10. DFEM100may comprise one or more processing and memory devices such as but not limited to a Raster Image Processor (RIP) and interface circuits (not shown) for interfacing with processor20and/or with other components of system10. In the configuration presented inFIG.1, DFEM100is integrated into console12and interfaces with an operator (not shown) of system10using input devices40and display34.

In other embodiments, DFEM100may comprise a standalone computer having input/output (I/O) devices for interfacing with the operator and with console12. In alternative embodiments, DFEM100may have any other suitable configuration.

In some embodiments, processor20is configured to display on display34, DI42comprising one or more segments (not shown) of DI42and/or various types of test patterns that may be stored in memory22.

In some embodiments, blanket treatment station52, also referred to herein as a cooling station, is configured to treat the blanket by, for example, cooling it and/or applying a treatment fluid to the outer surface of blanket44, and/or cleaning the outer surface of blanket44. At blanket treatment station52, the temperature of blanket44can be reduced to a desired value before blanket44enters image forming station60. The treatment may be carried out by passing blanket44over one or more rollers or blades configured for applying cooling and/or cleaning and/or treatment fluid on the outer surface of the blanket.

In some embodiments, blanket treatment station52may be positioned adjacent to image forming station60, in addition to or instead of the position of blanket treatment station52shown inFIG.1. In such embodiments, the blanket treatment station may comprise one or more bars, adjacent to print bars62, and the treatment fluid is applied to blanket44by jetting.

In some embodiments, processor20is configured to receive, e.g., from temperature sensors (not shown), signals indicative of the surface temperature of blanket44, so as to monitor the temperature of blanket44and to control the operation of blanket treatment station52. Examples of such treatment stations are described, for example, in PCT International Publications WO 2013/132424 and WO 2017/208152, whose disclosures are all incorporated herein by reference.

Additionally or alternatively, treatment fluid may be applied to blanket44, by jetting, prior to the ink jetting at the image forming station.

In the example ofFIG.1, station52is mounted between impression station84and image forming station60, yet, station52may be mounted adjacent to blanket44at any other or additional one or more suitable locations between impression station84and image forming station60. As described above, station52may additionally or alternatively be mounted on a bar adjacent to image forming station60.

In the example ofFIG.1, impression cylinder82impresses the ink image onto the target flexible substrate, such as an individual sheet50, conveyed by substrate transport module80from an input stack86to an output stack88via impression cylinder82.

In some embodiments, the lower run of blanket44selectively interacts at impression station84with impression cylinder82to impress the image pattern onto the target flexible substrate compressed between blanket44and impression cylinder82by the action of pressure of pressure cylinder90. In the case of a simplex printer (i.e., printing on one side of sheet50) shown inFIG.1, only one impression station84is needed.

In other embodiments, module80may comprise two or more impression cylinders (not shown) so as to permit one or more duplex printing. The configuration of two impression cylinders also enables conducting single sided prints at twice the speed of printing double sided prints. In addition, mixed lots of single and double sided prints can also be printed. In alternative embodiments, a different configuration of module80may be used for printing on a continuous web substrate. Detailed descriptions and various configurations of duplex printing systems and of systems for printing on continuous web substrates are provided, for example, in U.S. Pat. Nos. 9,914,316 and 9,186,884, in PCT International Publication WO 2013/132424, in U.S. Patent Application Publication 2015/0054865, and in U.S. Provisional Application 62/596,926, whose disclosures are all incorporated herein by reference.

As briefly described above, sheets50or continuous web substrate (not shown) are carried by module80from input stack86and pass through the nip (not shown) located between impression cylinder82and pressure cylinder90. Within the nip, the surface of blanket44carrying the ink image is pressed firmly, e.g., by compressible blanket (not shown), of pressure cylinder90against sheet50(or other suitable substrate) so that the ink image is impressed onto the surface of sheet50and separated neatly from the surface of blanket44. Subsequently, sheet50is transported to output stack88.

In the example ofFIG.1, rollers78are positioned at the upper run of blanket44and are configured to maintain blanket44taut when passing adjacent to image forming station60. Furthermore, it is particularly important to control the speed of blanket44below image forming station60so as to obtain accurate jetting and deposition of the ink droplets, thereby placement of the ink image, by forming station60, on the surface of blanket44.

In some embodiments, impression cylinder82is periodically engaged to and disengaged from blanket44to transfer the ink images from moving blanket44to the target substrate passing between blanket44and impression cylinder82. In some embodiments, system10is configured to apply torque to blanket44using the aforementioned rollers and dancer assemblies, so as to maintain the upper run taut and to substantially isolate the upper run of blanket44from being affected by mechanical vibrations occurring in the lower run.

In some embodiments, system10comprises an image quality control station55, also referred to herein as an automatic quality management (AQM) system, which serves as a closed loop inspection system integrated in system10. In some embodiments, image quality control station55may be positioned adjacent to impression cylinder82, as shown inFIG.1, or at any other suitable location in system10.

In some embodiments, image quality control station55comprises a camera (not shown), which is configured to acquire one or more digital images of the aforementioned ink image printed on sheet50. In some embodiments, the camera may comprises any suitable image sensor, such as a Contact Image Sensor (CIS) or a Complementary metal oxide semiconductor (CMOS) image sensor, and a scanner comprising a slit having a width of about one meter or any other suitable width.

In the context of the present disclosure and in the claims, the terms “about” or “approximately” for any numerical values or ranges indicate a suitable dimensional tolerance that allows the part or collection of components to function for its intended purpose as described herein. For example, “about” or “approximately” may refer to the range of values ±20% of the recited value, e.g. “about 90%” may refer to the range of values from 72% to 100%.

In some embodiments, station55may comprise a spectrophotometer (not shown) configured to monitor the quality of the ink printed on sheet50.

In some embodiments, the digital images acquired by station55are transmitted to a processor, such as processor20or any other processor of station55, which is configured to assess the quality of the respective printed images. Based on the assessment and signals received from controller54, processor20is configured to control the operation of the modules and stations of system10. In the context of the present invention and in the claims, the term “processor” refers to any processing unit, such as processor20or any other processor or controller connected to or integrated with station55, which is configured to process signals received from the camera and/or the spectrophotometer of station55. Note that the signal processing operations, control-related instructions, and other computational operations described herein may be carried out by a single processor, or shared between multiple processors of one or more respective computers.

In some embodiments, station55is configured to inspect the quality of the printed images and test pattern so as to monitor various attributes, such as but not limited to full image registration with sheet50, color-to-color (CTC) registration, printed geometry, image uniformity, profile and linearity of colors, and functionality of the print nozzles. In some embodiments, processor20is configured to automatically detect geometrical distortions or other errors in one or more of the aforementioned attributes. For example, processor20is configured to compare between a design version (also referred to herein as a “master” or a “source image” of a given digital image and a digital image of the printed version of the given image, which is acquired by the camera.

In other embodiments, processor20may apply any suitable type image processing software, e.g., to a test pattern, for detecting distortions indicative of the aforementioned errors. In some embodiments, processor20is configured to analyze the detected distortion in order to apply a corrective action to the malfunctioning module, and/or to feed instructions to another module or station of system10, so as to compensate for the detected distortion.

In some embodiments, system10may print testing marks (not shown), for example at the bevels or margins of sheet50. By acquiring images of the testing marks, station55is configured to measure various types of distortions, such as C2C registration error, image-to-substrate registration, different width between colors referred to herein as “bar to bar width delta” or as “color to color width difference”, various types of local distortions, and front-to-back registration errors (in duplex printing). In some embodiments, processor20is configured to: (i) sort out, e.g., to a rejection tray (not shown), sheets50having a distortion above a first predefined set of thresholds, (ii) initiate corrective actions for sheets50having a distortion above a second, lower, predefined set of threshold, and (iii) output sheets50having minor distortions, e.g., below the second set of thresholds, to output stack88.

In some embodiments, processor20is further configured to detect, e.g., by analyzing a pattern of the printed inspection marks, additional geometric distortion such as scaling up or down, skew, or a wave distortion formed in at least one of an axis parallel to and an axis orthogonal to the movement axis of blanket44.

In some embodiments, processor20is configured to detect, based on signals received from the spectrophotometer of station55, deviations in the profile and linearity of the printed colors.

In some embodiments, processor20is configured to detect, based on the signals acquired by station55, various types of defects: (i) in the substrate (e.g., blanket44and/or sheet50), such as a scratch, a pin hole, and a broken edge, and (ii) printing-related defects, such as irregular color spots, satellites, and splashes.

In some embodiments, processor20is configured to detect these defects by comparing between a section of the printed and a respective reference section of the original design, also referred to herein as a master. Processor20is further configured to classify the defects, and, based on the classification and predefined one or more criteria, to reject sheets50having defects that are not within the specified predefined criteria.

In some embodiments, system10comprises one or more suitable types of neural networks, which may be implemented in processor20and/or in DFEM100and/or in any other suitable processing device or module of system10. One implementation of an exemplary sort of neural network is described in detail inFIG.4below, and methods for applying the neural network are described in detail inFIGS.2A,2B,3and5below.

In some embodiments, the processor of station55is configured to decide whether to stop the operation of system10, for example, in case the defect density is above a specified threshold. The processor of station55is further configured to initiate a corrective action in one or more of the modules and stations of system10, as described above. The corrective action may be carried out on-the-fly (while system10continue the printing process), or offline, by stopping the printing operation and fixing the problem in a respective modules and/or station of system10. In other embodiments, any other processor or controller of system10(e.g., processor20or controller54) is configured to start a corrective action or to stop the operation of system10in case the defect density is above a specified threshold.

Additionally or alternatively, processor20is configured to receive, e.g., from station55, signals indicative of additional types of defects and problems in the printing process of system10. Based on these signals processor20is configured to automatically estimate the level of pattern placement accuracy and additional types of defects not mentioned above. In other embodiments, any other suitable method for examining the pattern printed on sheets50(or on any other substrate described above), can also be used, for example, using an external (e.g., offline) inspection system, or any type of measurements jig and/or scanner. In these embodiments, based on information received from the external inspection system, processor20is configured to initiate any suitable corrective action and/or to stop the operation of system10.

The configuration of system10is simplified and provided purely by way of example for the sake of clarifying the present invention. The components, modules and stations described in printing system10hereinabove and additional components and configurations are described in detail, for example, in U.S. Pat. Nos. 9,327,496 and 9,186,884, in PCT International Publications WO 2013/132438, WO 2013/132424 and WO 2017/208152, in U.S. Patent Application Publications 2015/0118503 and 2017/0008272, whose disclosures are all incorporated herein by reference.

The particular configurations of system10is shown by way of example, in order to illustrate certain problems that are addressed by embodiments of the present invention and to demonstrate the application of these embodiments in enhancing the performance of such systems. Embodiments of the present invention, however, are by no means limited to this specific sort of example systems, and the principles described herein may similarly be applied to any other sorts of printing systems.

Training Neural Network for Detecting Missing Nozzle Fault in Digital Printing

FIG.2Ais a schematic pictorial illustration showing a training phase of a convolutional neural network (CNN) configured to detect a missing nozzle fault (MNF), in accordance with an embodiment of the present invention.

In some cases, a defective part (DP) in system10may cause a defect, such as a distortion, in a printed image. In the present example, a defective nozzle (DN) from among nozzles99may cause a missing nozzle fault (MNF) in the image formed on blanket44, and therefore, typically on the corresponding sheet50(shown inFIG.1above) or on any other target substrate, as well as on blanket44.

In some embodiments, as described inFIG.5below, a neural network (NN) may be used for detecting one or more MNFs and for identifying one or more DNs causing the MNFs. The method comprises two phases: (i) a training phase, in which the NN is trained using known input and output data, and (ii) a detection phase that is subsequent to and is based on the training phase. In the present example a CNN, whose structure is described in detail inFIG.4below, may be used for this task, however, any other suitable type of neural network may be used, mutatis mutandis, for detecting the MNF and for identifying the DN causing the MNF. Note that the architecture of the CNN was selected and optimized based on simulations and experiments carried out by the inventors.

In some embodiments, processor20is configured to receive, e.g., from the RIP of DFEM100, a digital image, referred to herein as a RIP image or an image101produced on blanket44and transferred to sheet50as described inFIG.1above.

In the example ofFIG.2A, image101is formed on moving blanket44, so that when blanket44passes below print bars62, ink droplets of colors selected for image101(e.g., cyan, magenta, yellow and black, also referred to herein as CMYK for brevity) are directed by print bars62onto predefined regions111of blanket44so as to form image101. As shown inFIG.2A, when blanket44passes below a cyan print bar62, the cyan ink is directed by nozzles99A and99B, onto predefined regions of blanket44, referred to herein as patches111A and111B. The same process is carried out for the other print bars62(e.g., magenta (M) and yellow (Y) print bars62).

In the context of the present disclosure and in the claims, the terms “region” and “patch” are used interchangeably and refer to a section on the digital image (e.g. image101or any other image described herein).

Reference is now made to an inset102, showing an RGB (red, green, blue) palette104comprising a red color105, a green color106, a blue color107, and combinations thereof. Also shown in inset102, a CMYK palette103comprising a cyan ink (C)108, a magenta ink (M)109, a yellow ink (Y)110and combinations thereof. Note that CMYK palette103contains all the colors of RGB palette104. Moreover, the term “K” in the CMYK palette refers to a black color formed when mixing the C, M and Y ink, as shown at the center of CMYK palette103. For example, green color106is formed in the CMYK palette by mixing C108with Y110, blue color107is formed in the CMYK palette by mixing C108with M109, and red color105is formed in the CMYK palette by mixing M109with Y110.

The following description of a method for producing a first set of one or more synthetic images (SIs) for training the CNN to detect MNFs, is implemented by processor20. However, the method may be implemented, mutatis mutandis, using the RIP of DFEM100or any other suitable processing device or module of system10, such as but not limited to a processor of quality control station55.

In some embodiments, processor20is configured to produce (i) SIs112A and112B at patch111A, (ii) SIs114A and114B at patch111B, and (iii) SIs116A and116B at patch111C. In some embodiments, processor20is configured to produce a synthetic MNF in each of the SIs. For example, when simulating a defective (e.g., blocked) nozzle99A in SI112A, cyan ink108is not applied to the substrate (e.g., blanket44) at patch111A, thus SI112A has a column118having only yellow color Similarly, when simulating a blocked nozzle99B in SI114B, cyan ink108is not applied to blanket44at patch111B, thus SI114B has a column120having only magenta color. Note that the columns are produced when blanket44moves in the moving direction relative to print bars62and the column position within a given patch is derived from the position of the respective nozzle relative to the position of the given patch.

In some embodiments, processor20is configured to select the positions of patches111using any set of one or more predefined criteria. For example, each patch111may have 32 by 32 pixels, wherein at least a given amount (e.g., percent) of the pixels within the selected patch have a grey level smaller than 253 (in a 0-255 scale of gray levels). Note that the position of each patch111corresponds to one or more simulated defective (e.g., blocked) nozzles99intended to direct the ink onto the surface of blanket44at the position of the respective patch111.

In some embodiments, a printed pixel size may be about 21 μm using a printing resolution of 1200 dots per inch (DPI), or any other suitable printing resolution. In such embodiments, a 32-by-32-pixel patch111may have a size of about 0.672 mm by 0.672 mm.

In some embodiments, processor20is configured to determine the number and positions of patches111distributed along and across image101, based on any set predefined criteria. For example, processor20may determine about 5000 patches111distributed within image101for covering about 80% of the width of image101, e.g., orthogonal to the moving direction of blanket44, represented by arrow94.

In some embodiments, processor20is configured to select patches111for training the CNN using any suitable criterion or set of criteria. For example, if, in patch111A (i) an original synthetic image (e.g., without the simulated MNF), and (ii) SI112A (having the simulated MNF), have at least ten pairs of pixels with a grey level difference larger than about 15 gray levels along the column of Y110, then patch111A can be used for training the CNN.

In some embodiments, the number of SIs is derived, inter alia, based on the number of patches and the average number of colors (e.g. RGB transformed into CMYK or any other suitable combination of ink colors). For example, each patch111may have three different colors in average, and therefore four SIs corresponding to a synthetic image for each CMYK color. In such example embodiments, for 5000 patches processor20produces about 20,000 SIs, and in case only 80% of the patches are qualified for training the CNN, processor20can use 16,000 SIs (such as SIs112A-116B) for training the CNN to detect one or more MNFs.

In case 16,000 SIs are insufficient for obtaining the requested level of training, processor20may increase the number of patches111or select patches having more colors (and therefore more SIs).

FIG.2Bis a schematic pictorial illustration showing the impact of a partially-clogged nozzle99on the printing, and a training technique of the CNN for detecting the partially-clogged nozzle, in accordance with an embodiment of the present invention.

In some embodiments, the technique described inFIG.2Aabove may be used for training the CNN and for detecting additional faults, such as but not limited to a partially-clogged nozzle, which may occur in system10.

In the example ofFIG.2B, after applying (e.g., jetting) droplets of ink from nozzles99of print bar62(shown inFIG.1above), residues of the ink may remain on the surface of a nozzle99and may coagulate to produce an undesired cluster72of dried ink. Cluster72may partially clog an orifice85of nozzle99, which may result in a registration error in the printed image as will be described herein.

In some embodiments, nozzle99is configured to jet droplets75A and75B of ink, in the present example along a Z-axis of an XYZ coordinate system, toward an intended position79on the surface of blanket44. In some case, the formation of cluster72may cause deflection in the steering angle of the droplets jetted toward blanket44. As shown in the example ofFIG.2B, droplet75A is deflected by cluster72and droplet75B is expected to have a similar deflection while passing through orifice85. Note that the deflection of the steering angle depends, inter alia, on the size and hardness of cluster72, and on the position of cluster72on the surface of nozzle99. The deflection in the steering angle causes droplet75A to land on the surface of blanket44at a position located at a distance81from intended position79. In other words, cluster72causes a registration error, which is measured by distance81between the intended and the actual landing positions of droplet75A on the surface of blanket44.

In the context of the present disclosure and in the claims, the terms “partially-clogged” and “partially-blocked” are used interchangeably and refer to a nozzle99having a cluster72that does not completely block orifice85of nozzle99, but deflects droplets75A and75B as described above.

In some embodiments, during the training phase of the CNN, processor20is configured to select patches, such as patches111shown inFIG.2Aabove, which are suitable for detecting a partially-clogged nozzle99in print bar62. Moreover, processor20is configured to produce synthetic images indicative of the simulated landing position of the deflected ink droplet. For example, for partially-clogged nozzle99, processor20is configured to produce a set of SIs in which the simulated landing position is determined in a spherical coordinate system comprising: (i) a radial distance of the landing position from the partially-clogged nozzle, also referred to herein as r, (ii) a polar angle measured from a fixed zenith direction (e.g., parallel to Z-axis and typically orthogonal to the surface of blanket44) relative to orifice85of partially-clogged nozzle99, also referred to herein as θ, and (iii) an azimuthal angle of its orthogonal projection on a reference plane that passes through the origin of the XYZ coordinate system, and is orthogonal to the zenith. The azimuthal angle is measured from a fixed reference direction on that plane, and is also referred to herein as φ.

In some embodiments, processor20is configured to estimate for each SI, a distance between the intended landing position of the ink droplet (i.e., without partial clogging), and the actual landing position of the droplet (due to the partial clogging). In the example ofFIG.2B, a distance81measured between intended position79and the actual landing position of droplet75A. The intended and actual landing positions may be calculated in r,θ,φ coordinates of the spherical coordinate system, and may be used by processor20for estimating the size and orientation of distance81.

In some embodiments, processor20is configured to select patches111for detecting partially-clogged nozzles99using any suitable one or more criteria, such as an irregular variance in the gray level within patches having an array of repetitive structures (e.g., a lattice of lines and spaces).

In some embodiments, processor20is configured to produce, for each selected patch111, a set of synthetic images comprising the simulated landing position of the ink droplets deflected by the partially-clogged nozzle, based on the spherical coordinate system described above. The number of SIs depends, inter alia, on the number of discrete points selected within the spherical coordinate system, and may be limited to use-cases of interest and to meet the computational power constrains. For example, processor20may select the discrete points having a common radial distance estimated by the jetting force applied to droplet75A and75B, and a selected predefined number of polar angles and azimuthal angles.

Detecting Missing Nozzle Faults and Identifying Defective Nozzles Using a Trained CNN

FIG.3is a schematic pictorial illustration showing a method for detecting MNFs using a trained CNN, in accordance with an embodiment of the present invention. The embodiments below describe the method implementation using processor20. However, the method may be implemented, mutatis mutandis, using DFEM100or a device thereof, or using any other suitable processing device or module of system10.

In some embodiments, processor20receives DI42, which is acquired by image quality control station55, from a printed version of a digital image received from DFEM100. Note that image101ofFIG.2Aabove has not been printed yet, and DI42is a digital image acquired from a printed image, and therefore may comprise one or more distortions and/or defects, such as one or more MNFs caused by one or more defective nozzles. Moreover, the CNN training described inFIG.2Aabove, was carried out using image101of a first digital image, whereas DI42has a second digital image, which may be similar to or different from the first digital image. In other words, the CNN training may be carried out on a given digital image, and the detection of one or more MNFs and DNs described herein may be carried out on a digital image acquired from a printed version of the given digital image, or from a printed version of a different image. For example, image101may comprise a digital image of a dog (not shown) and DI42may comprise the same dog image (with or without defects caused during the printing process), or from a printed version of a digital image of an elephant, as shown in DI42ofFIGS.1and2above.

In some embodiments, processor20is configured to apply the training dataset shown inFIG.2Aabove (and/or the training dataset described inFIG.2Babove) to a suitable CNN architecture for identifying, in DI42, one or more regions (e.g., regions135and145) suspected of having a defect (e.g., the MNF and/or the C2C registration error cause by partially-clogged nozzle99) that the CNN was trained to detect, as described inFIG.2Aabove.

Note that by detecting potential MNFs in regions135and145, the trained CNN reduces the number of suspected defective nozzles99in system10. Therefore, region135comprises a known set of color pixels printed by known nozzles99of known print bars62of image forming station60of system10.

In the example ofFIG.3, a DI136shows a higher magnification of a section of the elephant ear shown in region135of DI42. In some embodiments, processor20is configured to produce, for region135, a set of one or more synthetic images, referred to herein as a set137. Each SI of set137comprises a simulation of one or more defective nozzles99selected from the known print bars62and nozzles99used for applying the one or more colors of ink to region135.

In some embodiments, the SIs of set137comprise all combinations of nozzles99of print bars62participating in the formation of DI136.

For example, DI136may be formed using four nozzles99of cyan print bar62, five nozzles99of magenta print bar62, and three nozzles99of yellow print bar62. Therefore, set137may comprise up to (4·5·3=) 60 SIs, one SI for each nozzle99. In some embodiments, the number of nozzles suspected of having a defect, and therefore, the number of SIs of set137, may be reduced. For example, in case the MNF is visible and appears to happen in a specific color, e.g., cyan. In this example, set137may comprise only four SIs, each SI simulates one suspected defective nozzle99of cyan print bar62.

In some embodiments, processor20is configured to identify at least one defective nozzle99from among the nozzles participating in the formation of region135. In an embodiment, processor20is configured to compare between DI136and each SI of set137. The comparison may be carried out using the trained CNN or using any other suitable image comparison technique.

In some embodiments, by comparing between DI136of region135, and each SI of set137, processor20is configured to: (i) identify one or more MNFs in suspected region135, and (ii) associate each MNF with one or more respective defective nozzles99from among the nozzles participating in the formation of region135.

A process for detecting a defective nozzle, based on the embodiments described in one or more ofFIGS.2A,2B and3above, may be summarized using the following example. In the example, nozzle99A (shown inFIG.2Aabove) of cyan print bar62may be defective, e.g., blocked, and therefore cannot apply droplets of cyan ink to blanket44in region135of DI42. During the training phase shown and described inFIG.2Aabove, processor20is configured to produce SI112A, showing a simulation of blocked nozzle99A in patch111A. The CNN is trained, based on SI112A and other SIs ofFIG.2Aabove, to detect, in digital images printed by system10, regions suspected for having one or more MNFs caused by one or more respective defective nozzles99. In a detection phase (shown inFIG.3) that is subsequent to the training phase, the trained CNN is configured to detect in DI42, region135printed using nozzle99A and additional nozzles99, wherein at least one of these nozzles is suspected for being a defective nozzle.

In some embodiments, processor20is configured to produce in set137, an SI for simulating a respective nozzle99, which is associated with region135and is suspected for being defective, as described above.

In some embodiments, processor20is configured to compare between DI136and each SI of set137, and to detect blocked nozzle99A, by finding correlation between DI136and the SI of set137, simulating the MNF caused by blocked nozzle99A.

Similarly, a DI146shows a higher magnification of a section of a tip of the elephant task shown in region145of DI42. In some embodiments, processor20is configured to produce, for region145, a set of one or more synthetic images, referred to herein as a set147. Each SI of set147comprises a simulation of one or more defective nozzles99selected from the known print bars62and nozzles99used for applying the one or more colors of ink to suspected region145. Processor20is configured to apply the process described above for region135, so as to (i) detect one or more MNFs in region145, and (ii) identify, within one or more respective print bars62of system10, one or more defective nozzles99causing the detected MNFs.

In other embodiments, processor20is configured to apply the techniques described inFIG.2Aabove to the trained CNN for detecting regions that are suspected of having partially-clogged nozzles99. It will be understood by a person skilled in the art of digital printing that typically a similar mechanism may cause a partially-clogged nozzle and a fully-blocked nozzle. Thus, in some cases a given print bar62may cause both the MNF defect and the registration error depicted above inFIGS.2A and2B, respectively. In the present example, the CNN may output that region145is suspected for comprising a registration error that may be caused by cluster72that partially-blocks orifice85of nozzle99, as shown and described in detail inFIG.2Babove.

In some embodiments, processor20receives, e.g., from image quality control station55, DI146showing a higher magnification of the aforementioned section of the tip of the elephant task shown in region145of DI42.

In some embodiments, processor20is configured to produce, for region145, in addition to or instead of set147, a supplementary set of one or more synthetic images. Each SI of the supplementary set comprises a simulation of one or more registration errors caused by deflected droplets jetted by a suspected partially-clogged nozzles99, which are intended for applying the droplets of one or more colors of ink to suspected region145.

In some embodiments, processor20is configured to apply the process described above for detecting the MNF defects in regions135and145, so as to (i) detect one or more registration errors in region145, and (ii) identify, within one or more respective print bars62of system10, one or more partially-clogged nozzles99that jet deflected droplets (such as droplet75A ofFIG.2Babove) causing the detected registration error.

In the present example, the deflection of droplets may occur in specific nozzles99of a cyan print bar62(shown inFIG.2Aabove), which are intended to apply droplets of cyan ink to predefined sections of blanket44. Therefore, the aforementioned partially-clogged nozzles99may cause a C2C registration error between the cyan color and the other colors applied to these predefined sections.

In other embodiments, processor20is configured to use the CNN for detecting suspected regions135and145, e.g., by applying the technique described inFIG.2Aabove. Subsequently, at the detection phase shown inFIG.3, processor20is configured to apply the technique depicted inFIG.2Babove, for detecting within regions135and145, defects indicative of one or more partially-clogged nozzles99. In such embodiments, processor20may apply the CNN for producing SIs indicative of simulated partially-clogged nozzles, only to regions135and145.

FIGS.2A,2B and3illustrate, by way of example, a method for identifying defective nozzles99(e.g., fully blocked or partially clogged) in print bars62of system10. The techniques described herein, however, can be used, mutatis mutandis, for identifying other defective parts of system10or in any other suitable technique involving identifying one or more defective parts in any sort of system, used for printing or for any other sorts of production processes.

FIG.4is a schematic pictorial illustration of a convolutional neural network (CNN)150used for detecting MNFs in DI42, in accordance with an embodiment of the present invention. In some embodiments, the elements of CNN150may be implemented in hardware or software, or using in any suitable combination thereof.

In some embodiments, processor20is configured to train CNN150for detecting one or more MNFs in DI42. Processor20is further configured to use the one or more detected MNFs for identifying defective nozzles99(and/or other parts) of system10, as described inFIGS.2A,2B and3above, and using a method that will be described inFIG.5below.

In some embodiments, CNN150has an inception-v3 architecture provided by Google (Mountain View, Calif. 94043), but may have any other suitable type of neural network.

In some embodiments, CNN150may comprise multiple sections and modules described below.

In some embodiments, CNN150may comprise (e.g., in the aforementioned modules) a multi-layered convolutional neural network, each of the layers having an array of neurons.

In some embodiments, each neuron in CNN150computes an output value by applying a specific function to the input values coming from the receptive field in the previous layer. The function that is applied to the input values is determined by a vector of weights and a bias (typically real numbers). The learning process in CNN150, progresses by making iterative adjustments to these biases and weights.

The vector of weights and the bias are referred to herein as filters of the layers and are defined by a particular size and shape of the input (e.g., a particular shape). A distinguishing feature of CNNs is that many neurons can share the same filter.

In some embodiments, CNN150is configured to receive an input160comprising an array of 299 by 299 by 3 weights, corresponding to the 299 by 299 pixels of a respective digital image, each of the pixels having three colors (e.g., RGB as described inFIGS.2A and3above).

Reference is now made to an inset151showing a legend of layers and other elements used in the architecture of CNN150. In some embodiments, CNN150comprises convolutional layers, labeled in inset151as “convolution,” and referred to herein as CLs152. CNN comprises average pooling layers153, labeled in inset151as “AvgPool,” and max pooling layers154, labeled in inset151as “MaxPool.” Pooling layers are configured to reduce the dimensions of the data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer.

The term “Max pooling” refers to pooling that uses the maximum value from each of a cluster of neurons at the prior layer. The average pooling uses the average value from each of a cluster of neurons at the prior layer, so that average pooling is configured to convert the output tensor of a convolutional layer to a vector of weights having, for example, any suitable number of scalar numbers. In the context of the present invention and in neural networks, the term “flatten” refers to conversion of a multi-dimensional tensor into a one-dimensional vector.

In some embodiments, CNN150comprises multiple concatenations, configured to concatenate between adjacent layers and/or module and/or sections of CNN150. Each of the concatenations is labeled in inset151as “concat” and is referred to herein as a concat155.

In some embodiments, CNN150comprises one or more dropout layers, referred to herein as a dropout156, which may be used at multiple layers as will be described herein. A softmax activation function may be used in one or more layers (e.g., in a classifier layer described below), and is referred to herein as a softmax158. The dropout layer and softmax activation function are labeled in inset151as “Dropout” and “Softmax,” respectively.

In some embodiments, CNN150comprises one or more fully-connected layers (FCL)157, labeled in inset151as “Fully connected.” Note that term “fully connected layer” refers to a neural network layer that connects every neuron in one layer to every neuron in another layer.

Reference is now made back to the general view ofFIG.4. In some embodiments, CNN150comprises a module162having multiple CLs152and max pooling layers154arranged sequentially. Module162is configured to receive input160and to prepare the weights of input to be inserted into a module164, also referred to herein as a “5X Inception Module A” comprising multiple CLs152and average pooling layers153arranged in suitable structures and having concats155for concatenating between the structures of layers.

In some embodiments, CNN150comprises a grid size reduction module166, which is configured to convert the array of 299 by 299 by 3 weights of input160, to an output array of 8 by 8 by 2048 weights, referred to herein as an output180. In the present example, grid size reduction module166comprises multiple CLs152, a max pooling layer154, and a concat155.

In other embodiments, module166may have a different structure than the structure shown inFIG.4.

In some embodiments, CNN150comprises a module168, also referred to herein as a “4X Inception Module B” comprising multiple CLs152and average pooling layers153arranged in suitable structures and having concats155for concatenating between the structures of layers.

In some embodiments, CNN150comprises an auxiliary classifier174, which is coupled to a concat172, which is the right-most concat of module168. Auxiliary classifier174comprises multiple layers, such as one average pooling layer153, two CLs152, one FCL157and one softmax158.

In some embodiments, CNN150comprises a grid size reduction module170comprising multiple CLs152, a max pooling layer154, and a concat155. In some embodiments, CNN150comprises a module176, also referred to herein as a “2X Inception Module C” comprising multiple CLs152and average pooling layers153arranged in suitable structures and having concats155for concatenating between the structures of layers.

In some embodiments, CNN150comprises a tensor178, comprising multiple layers, such as one average pooling layer153, one dropout156, one FCL157and one softmax158. Tensor178, which is the output of CNN150, has the structure described above for output180(an array of 8 by 8 by 2048 weights).

The configuration of the inception-v3 CNN architecture and particularly that of CNN150, and use cases thereof are described in detail, for example by Szegedy et al., in “Rethinking the Inception Architecture for Computer Vision,” computer vision and pattern recognition (CVPR) conference of the Computer Vision Foundation (CVF) pages 2818-2826 (June 2016); and by Sik-Ho Tsang, in “Review: Inception-v3—1st Runner Up (Image Classification) in ILSVRC 2015,” (Sep. 10, 2018), which are all incorporated herein by reference.

This particular configuration of CNN150is shown by way of example, in order to illustrate certain problems that are addressed by embodiments of the present invention and to demonstrate the application of these embodiments in enhancing the performance of system10using CNN150. Embodiments of the present invention, however, are by no means limited to this specific sort of example CNN configuration, and the principles described herein may similarly be applied to other sorts of neural networks used for enhancing the performance of such digital printing systems.

FIG.5is a flow chart that schematically illustrates a method for detecting a defective nozzle in system10, in accordance with an embodiment of the present invention. The method may be implemented, as described below, using processor20. However, the method may be implemented, mutatis mutandis, using DFEM100or a device thereof, or using any other suitable processing device or module of system10.

The method begins at a first raster image receiving step200, with processor20receiving image101, e.g., from the RIP of DFEM100, as described inFIG.2Aabove. After concluding step200, the method has two phases: a training phase, and a detection phase that is subsequent to the training phase.

The detection phase begins at a patch selection step202, with processor20selecting, based on a predefined criterion, one or more (e.g., about 5000) patches (e.g., patches111,111A,111B and111C) comprising features of image101, as described inFIG.2Aabove.

At a first synthetic images (SIs) production step204, processor20produces, for each patch produced in step202, a first set of one or more synthetic images (e.g., SIs112A,112B,114A,114B,116A and116B ofFIG.2Aabove) having a simulated missing nozzle fault (MNF) in the respective patch (e.g., patches111A,111B and111C), as described inFIG.2Aabove. Additionally or alternatively, processor20produces, for each patch produced in step202, a first additional set of one or more synthetic images having a simulated registration error that may be caused (by partially-clogged nozzles99) in one or more respective patches111, as described inFIG.2Babove.

At a SI selection step206, processor20selects, based on a second predefined criterion, one or more of the SIs of the first set (and/or the first additional set) produced in step204above. The selected SIs are suitable for training a convolutional neural network (CNN), such as CNN150, as described inFIGS.2A and2Babove.

At a CNN training step208, processor20trains CNN150using the synthetic images of the first set (and/or the first additional set) selected in step206, as described inFIGS.2A and2Babove. Step208concludes the training phase and thereafter CNN150is trained for detecting defects, such as MNFs and/or registration errors, in images printed by system10.

At a digital image receiving step210, processor20receives, e.g., from image quality control station55, a digital image acquired from an image printed by system10. In some embodiments, processor produces DI42based on the image received from image quality control station55. In other embodiments, processor20receives DI42, which is produced by image quality control station55.

After concluding step210, the method begins the detection phase at a regions identification step212, with processor20applying CNN150for identifying, in DI42, regions135and145suspected for having MNFs and/or registration errors, as described in detail inFIG.3above.

At a second SIs production step214, processor20produces, for each of regions135and145detected in step212, a second set of one or more synthetic images (e.g., sets137and147ofFIG.3above) having a simulated missing nozzle fault (MNF) and/or registration errors in the respective regions (e.g., regions135and145). As described inFIG.3above, the number of SIs in sets137and147corresponds to the number of nozzles99participating in the image forming at regions135and145. For example, set137may comprise up to 60 SIs for simulating, respectively, a defect in each of the 60 nozzles99used by system10in the image formation of region135. In other words, each SI of set137has one simulated defective nozzle99, so as to incorporate all nozzles99used for applying ink to region135of DI42.

At a MNF detection step216, processor20detects one or more MNFs by comparing the second set of SIs (e.g., set137) with the acquired DI (e.g., DI136) of the corresponding suspected region (e.g., region135). In other embodiments, in step216processor20may detect (in addition to or instead of the MNFs) one or more C2C registration errors by comparing the second set of SIs (e.g., the supplementary set of one or more synthetic images described inFIG.3above) with the acquired DI (e.g., DI146) of the corresponding suspected region (e.g., region145). Note that processor20may: (i) use the training phase for training the CNN to detect both the MNF and the C2C registration error (caused by partially-clogged nozzle(s)99), (ii) apply the CNN to detect regions suspected of having the MNF (e.g., regions135and145), and (iii) apply the trained CNN to the suspected regions for detecting both the MNF and the C2C registration error.

At a defective nozzle identification step218, which concludes the detection phase and terminates the method, processor20identifies one or more defective nozzles99, by associating each MNF detected in step216, with a respective defective nozzle99of system10. As described inFIG.3above, processor20may compare between DI136and each SI of set137, so as to detect a defective nozzle (such as blocked nozzle99A), by finding correlation between DI136and the corresponding SI of set137, which is simulating the MNF caused by the defective nozzle (e.g., blocked nozzle99A). Additionally or alternatively, at step218, processor20identifies one or more partially-clogged nozzles99, by associating each C2C registration error detected in step216, with a respective partially-clogged nozzle99of system10.

In some embodiments, processor20may apply the trained CNN, only to regions135and145that were detected based on the MNF training of the CNN, for detecting the C2C registration error (as described in step216above) and for identifying one or more partially-clogged nozzles99(as described above in step218).

It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.