Patent Description:
Digital inpainting is the process of applying algorithms to images to replace lost, deteriorated, or corrupted parts of the image data. Inpainting may be used to replace lost blocks in the coding and transmission of images and to remove logos or text from images and videos and to fill in the missing details of the original image. There are many applications of image inpainting ranging from restoration of photographs, films, removal of occlusions such as text, subtitle, logos, stamps, scratches, etc..

Sometimes an image may contain embedded text that is to be removed for aesthetic, translation, or other reasons. In such cases, the goal is to remove the undesired text, to recover the background of the image where the text was located, and to inpaint the text regions with the image background. However, the process becomes complicated when the text area is not horizontal and is rotated at an angle. Also, the characters in the text region may have any text size, artistic font, or color, further complicating the text removal and inpainting process. Existing processes do not effectively remove such text and inpaint multiple text regions with various kinds of text style and angles with high efficiency and low power consumption for application of text inpainting on mobile devices, for example.

As just noted, one of the applications for inpainting is the removal of text or objects such as text-subtitle and logos from an image. <FIG> shows a further example where the text "NOODLES" in image <NUM> is removed and the background is inpainted in image <NUM>. As illustrated, the inpainted image <NUM> is inconsistent. In text translation applications, it is desired to remove the original text and to replace the original text with the translated text. <FIG>illustrates a translator application where inpainting is used to remove the original characters in image <NUM> to obtain a clean background <NUM> before rendering the translated text <NUM>, in this case, "Emergency refuge.

A general inpainting framework for rendering images as shown in <FIG>is shown in <FIG>. As illustrated, there are two steps:.

This approach relies on an initialization of a start point on the contour which is not robust. It also adopts a recursion scheme to update the inpainting result which will cause a speed issue. Existing approaches for finding the text mask include the use of image binarization segmentation to convert a pixel image <NUM> into a binary image <NUM> having values of <NUM> and <NUM> as shown, for example, in <FIG>. Binarization is used to find a threshold locally or globally to segment the image. If a pixel value is greater than the threshold, the pixel value is set to <NUM>, while if the pixel value is less than the threshold, the pixel value is set to <NUM>. However, most binarization algorithms are limited due to the low illumination or contrast in the image leading to incorrect separation of texts from the background. For example, Otsu's binarization method automatically performs clustering-based image thresholding to find the threshold globally but is not robust enough to handle low illumination and contrast in parts of the image.

The binarization process may cause loss of text information especially for the pixels on the edge. Usually, the edge between text and background is not ideally sharp where there is a blurred transition area as shown by the closeup of image <NUM> at <NUM> in <FIG>, for example. Due to this limitation, the results <NUM> of existing binarization algorithms such as the Niblack, Sauvola et al. and Wolf et al. algorithms depicted in <FIG> are sub-optimal as text pixels are labelled incorrectly or the background is mislabeled as text. Descriptions of these algorithms may be found at <NPL>; <NPL>; and <NPL>. Moreover, binarization is the process that segments the image but does not determine the foreground and background. If it wrongly labels text as background, the real background will be inpainted with text color.

<NPL>, that resizes the input image to <NUM> by <NUM> pixels and applies the Niblack method for local threshold binarization. Then they use the connected component method to find the possible text area and to set two thresholds to remove non-text regions. The binary image of text regions after dilation is the mask for inpainting. However, this algorithm is limited in that the Niblack method is a local threshold binarization method and has limitations on its usage. Also, the procedure of resizing the input image to <NUM> by <NUM> pixels causes text information loss, especially for text in a long line of text. In addition, the criteria for setting the threshold to remove macros or large connected regions is not robust. The resulting binary mask for inpainting the text region is not sufficient for many cases, especially where the image quality is low or the illumination is non-uniform. The document "<NPL>, discloses an approach for text detection, extraction and inpainting in color images. The algorithm includes three stages. In the first stage, several gradient based operators and image corners are utilized to localize text blocks. An SVM based text verification algorithm is then employed with a new set of features to reject non-text blocks. In the third stage, the inpainting algorithm is applied to restore initial image contents.

Various examples are now described to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. The Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

The systems and methods described herein address the limitations of the prior art by improving the accuracy of the mask of a text area of the original image for labelling of the text area for inpainting. The systems and methods described herein addresses the problems noted above with using binarization to extract the text area masks. In a sample embodiment, three masks based on a normalization of black background and white characters are combined into a fusion mask: a binary mask, an edge mask, and a contour mask. Alternatively, only two of the three masks may be combined into a fusion mask (i.e. a contour mask and an edge mask or a binary mask). The advantages provided by each type of mask guarantee that the text pixels in the original image are fully covered by the inpainting. The whole background is retrieved by applying the fusion mask with the inpainting algorithm on the input image. Normalization of the image to a black background and white characters decides the text binary mask as foreground for inpainting, while the fusion mask includes the edge feature of the text as well as the inside text pixels, which guarantees the accuracy of segmentation for the text. Also, the processing is not complex, which enables the use of the system and method to implement text inpainting on a mobile device with relatively low processing power and impact on battery life.

According to a first aspect of the present disclosure, there is provided a computer-implemented method of finding a text mask in an original image for extracting text for inpainting according to independent claim <NUM>. The method includes the steps of applying, with one or more processors, edge detection to detect an edge of text in the original image as an edge mask, and using the edge mask to find a set of hierarchical closed edge contours of the text and to fill in the closed edge contours with pixels labeled as possible text as a contour mask. A binarization method is applied to the original image to convert the original image into a binary image and to detect a stroke edge of the text in the binary image, and the binary image is partitioned based on the detected stroke edge of the text in the binary image. At least one binarization method is applied on each image partition to obtain at least one binary mask candidate as foreground for inpainting. The contour mask and at least one of the edge mask and binary mask are combined into a fusion mask and the fusion mask is applied to the original image to extract the text in the original image and to obtain an original background of the original image without the text. Portions of the original image are inpainted where the text has been extracted. The advantages provided by each type of mask combined into the fusion mask guarantee that the text pixels in the original image are fully covered by the inpainting.

According to a second aspect of the present disclosure, there is provided an image processing device as defined in independent claim <NUM>. As with the first aspect, the advantages provided by each type of mask combined into the fusion mask guarantee that the text pixels in the original image are fully covered by the inpainting.

According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable medium as defined in independent claim <NUM>. As with the first and second aspects, the advantages provided by each type of mask combined into the fusion mask guarantee that the text pixels in the original image are fully covered by the inpainting.

In a first implementation of any of the preceding aspects, the original image is converted to a grayscale image prior to applying the edge detection.

In a second implementation of any of the preceding aspects, applying the edge detection comprises applying at least one of a morphological gradient edge detection algorithm and a Sobel operator edge detection algorithm to the original image to detect the edge of text in the original image.

In a third implementation of any of the preceding aspects, using the edge mask to find the set of hierarchical closed edge contours of the text comprises applying the morphological gradient edge detection algorithm to the original image to create a morphological gradient contour mask, applying the Sobel operator edge detection algorithm to the grayscale image to create a Sobel contour mask, and combining the morphological gradient contour mask with the Sobel contour mask to create the contour mask.

In a fourth implementation of any of the preceding aspects, a connected components method is applied to fill in the pixels bounded by the contour mask to fill in the closed edge contour.

In a fifth implementation of any of the preceding aspects, applying the binarization method to the original image to convert the original image into the binary image and to detect the stroke edge of the text in the binary image comprises at least one of binarizing a lightness channel in a hue, lightness, and saturation color space for the original image to enhance a contrast of the original image, binarizing a contrast limited adaptive histogram equalization of a LAB color space binarization to enhance the contrast of the original image, binarizing the lightness channel of the LAB color space by applying a Principal Component Analysis binarization to enhance the contrast of the original image, and choosing a best binary image from binary images generated by each binarization method.

In a sixth implementation of any of the preceding aspects, applying the binarization method to the original image to convert the original image into the binary image and to detect the stroke edge of the text in the binary image comprises at least one of taking an inverse of the binarized lightness channel in the hue, lightness, and saturation color space for the original image, taking an inverse of the binarized contrast limited adaptive histogram equalization of the LAB color space binarization, taking an inverse of the Principal Component Analysis binarization, and choosing the best binary image from the binary images generated by each binarization method and by each binarization method inverse.

In a seventh implementation of any of the preceding aspects, partitioning the binary image based on the detected stroke edge of text in the binary image comprises partitioning the binary image into sub-images by horizontal and vertical histogram projection and binarizing the sub-images.

In an eighth implementation of any of the preceding aspects, normalizing the binary image comprises setting the text as pixels of a first color and setting the background as pixels of a second color and choosing a best binary mask from the at least one binary mask candidate.

The method is performed and the instructions on the computer readable media may be processed by the device, and further features of the method and instructions on the computer readable media result from the functionality of the device. The different embodiments may be implemented in hardware, software, or any combination thereof. Also, any one of the foregoing examples may be combined with any one or more of the other foregoing examples to create a new embodiment within the scope of the present disclosure.

It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and/or methods described with respect to <FIG> may be implemented using any number of techniques, whether currently known or in existence.

In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the methods described herein, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present disclosure. The following description of example embodiments is, therefore, not to be taken in a limited sense, and the scope of the present disclosure is defined by the appended claims.

The functions or algorithms described herein may be implemented in software in one embodiment. The software may consist of computer executable instructions stored on computer readable media or computer readable storage device such as one or more non-transitory memories or other type of hardware-based storage devices, either local or networked. Further, such functions correspond to modules, which may be software, hardware, firmware or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the embodiments described are merely examples. The software may be executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system, turning such computer system into a specifically programmed machine.

As noted above, the system and method described herein extract accurate masks of text areas for inpainting by using a binary mask, edge mask, and contour mask based on a normalization of the background to pixels of one color (e.g., black) and pixels of the foreground to pixels of another color (e.g., white). The contour mask and at least one of the edge mask and the binary mask are combined into a fusion mask that is a union of the mask set. The fusion mask is a binary image with a pixel value of <NUM> or <NUM> at each pixel, where a pixel at a particular row and column will have a value of <NUM> if it is <NUM> in any mask. A value is <NUM> only if it is <NUM> in all of the fused masks. The fusion mask is applied to the input image to retrieve the original background for application of an inpainting algorithm. Such an approach works even for images with low quality (e.g., handwritten text) or illumination or with non-uniform (e.g., fonts outlined with different colors or background of more than one color) or angled text.

<FIG> illustrates a sample embodiment for generating a fusion mask for inpainting. In this example, edge detection <NUM> is applied to original image <NUM> to convert the color image to grayscale and to detect the edge of the text to form an edge mask. Based on the edge mask, a closed edge contour is found at <NUM> and the closed edge contour is filled in with white pixels labeled as possible text. A binarization method <NUM> is used and a projection histogram is chosen to detect the stroke edge in the binary image. The image is partitioned based on the stroke edge, and several binarization methods may be applied at <NUM> on each subimage to obtain a binary mask. A normalization step is applied at <NUM> to set the text portion as white pixels and the background as black pixels for the binarization result. The best binary mask is chosen from the candidate binary images. Of course, the normalization may alternatively set the text portion as black pixels and the background as white pixels for the binarization result. The contour mask and at least one of the edge mask and the binary mask are combined into a fusion mask at <NUM> and the respective masks function to complement each other's advantages and disadvantages. The fusion mask is then used with an inpainting algorithm at <NUM>.

<FIG> illustrates a detailed flow diagram for generating a fusion mask for use in creating an inpainted image in a sample embodiment. As illustrated in <FIG>, the fusion mask may have three components: an edge mask, a contour mask, and a binary mask, although the contour mask and at least one of the edge mask and the binary mask are combined into the fusion mask. The process includes taking an input image at <NUM> and converting the input image into a grayscale image at <NUM>. The system then generates the edge mask, contour mask, and binary mask to generate a fusion mask as described below.

In a sample embodiment, two edge detectors are implemented: a morphological gradient and a Sobel operator. The morphological gradient finds the edge of the text at <NUM> by determining the difference between the dilation image <NUM> (expanded shape in input image <NUM>) and the erosion image <NUM> (reduced shape in input image <NUM>). As shown in <FIG>, the dilation image <NUM> may comprise additional pixels around the edge of the text while the erosion image <NUM> comprises a reduction in pixels around the edge of the text. The difference between the dilation image and erosion image is a text border image <NUM> that is binarized to provide the morphological gradient <NUM> as illustrated in.

On the other hand, the Sobel operator is applied to the grayscale image at <NUM> to detect the edge both in the horizontal and the vertical directions. The Sobel operator is a discrete differentiation operator that computes an approximation of the gradient of the image intensity function. The Sobel operator is typically used in edge detection algorithms to create an image emphasizing edges.

Both of these two edge detectors (morphological gradient <NUM> and Sobel operator <NUM>) are robust enough to use on images with low illumination and measurable noise. As exemplified in <FIG>, the morphological gradient <NUM> detects text edge pixels from the input image <NUM> more accurately as shown by image <NUM> but fails when the text and background are in low contrast. However, as illustrated by the image <NUM>, the Sobel operator <NUM> is a good compliment in this case.

As explained below, both the contour mask and normalization are based on the edge mask.

<FIG> illustrates the steps to achieve a contour mask in a sample embodiment. As illustrated, the contour mask process starts with edge detection <NUM>. In a sample embodiment, morphological gradient edge detection <NUM> and Sobel edge detection <NUM> are applied to the grayscale image as described above. Each hierarchical closed text contour is retrieved from the morphological gradient edge image and Sobel edge image by the findContour() algorithm in OpenCV. This algorithm implements topological structural analysis of digitized binary images by following the border. Then, a connected components method is used to fills the pixels bounded by the contours of the morphological gradient edge image and Sobel edge image by the algorithm drawContour() in OpenCV. These processes result in a morphological gradient contour mask <NUM> and Sobel contour mask <NUM>. The two types of contour masks <NUM> and <NUM> are combined into a final contour mask at <NUM>.

Contour is a shape descriptor and useful for text detection. Both the text contours are found and the pixels inside the contour are filled as shown in <FIG> to generate contour mask <NUM> from the edge image <NUM>. Contour mask <NUM> enables inpainting for the artificial font characters. For example, in <FIG> the binarization or the edge detection of the input image <NUM> only segments the boundary of the text as shown at <NUM>. However, filling the contour as shown at <NUM> resolves this issue. In the contour mask <NUM>, the text pixels are labeled as white and the background pixels are labeled as black, and the contour mask <NUM> finds most of the text pixels.

Referring back to <FIG>, the contour mask is generated from the outputs of the morphological gradient process <NUM> and Sobel edge process <NUM> by binarization of the respective edge masks at <NUM> and <NUM> and filling of the respective contours at <NUM> and <NUM>. The respective filled contours are combined into a contour mask at <NUM>. In a sample embodiment, the contour mask is normalized to a black background with white text.

Most binarization methods have limits on threshold. It is desirable to provide a threshold that may be used in various environments for processing under a variety of different circumstances as there are occasions where one binarization generates better results than others and vice-versa. Accordingly, three different binarization methods based on Otsu and a list of candidates are provided for consideration as the final binary mask in sample embodiments. Of course, more than three binarization methods may also be implemented as desired for handling all input image cases.

<FIG> illustrates the procedure for generating a binary mask in a sample embodiment. As illustrated in <FIG>, the input image <NUM> is provided to one or more binarization algorithms at <NUM>. In a sample embodiment, three different binarization options are provided for use under different conditions.

A first binarization method includes application of a Binarization of Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm that is applied to the L channel of a LAB color space (L refers to lightness, A refers to green-red color components, and B refers to blue-yellow color components). The CLAHE process uses the image enhancement for pre-processing purposes. First, histogram equalization is applied only on the L channel to enhance the contrast of the color image and to avoid amplification of noise, while preserving the brightness. Then the enhanced color image is thresholded by the Otsu global binarization algorithm. <FIG> illustrates the binarization image <NUM> resulting from processing of the gray-scale image <NUM> using the CLAHE algorithm.

A second binarization method includes binarization of the L channel of hue, lightness, and saturation (HLS) color space. Since the L channel represents the black and white in a color, it is sensitive to light changes or shadows in the image. In this way, this binarization method is very robust with low contrast images. <FIG> illustrates the input image <NUM> after binarization of L in HLS at <NUM> as compared with other methods at <NUM>.

A third binarization method provides binarization using PCA (Principal Component Analysis) of LAB color space. The Principal Component Analysis finds the primary and secondary axis in the LAB color space and the mean value of the image as well. PCA is quite sensitive to the color space rather than luminance. The PCA binarization approach is particularly useful for those characters in a fancy style. <FIG> illustrates an example where the input image <NUM> is binarized using PCA at <NUM>. Other methods set the threshold only and segments the boundary of text as shown at <NUM>, while PCA choose the primary value and segments text pixels correctly.

Referring back to <FIG>, once the input image has been binarized at <NUM>, stroke edge detection via horizontal and vertical histogram projection is applied at <NUM>. Though the binarization works in most cases, when the background is complex or a part of the input image has low contrast, the binarization processes may still fail. <FIG> illustrates an example of low contrast or illumination at <NUM> and <NUM> and the resulting binarized images at <NUM> and <NUM>. The stroke edge detection at <NUM> aims to detect the edge of obvious illumination or contrast changes.

As illustrated in <FIG>, vertical histogram projection projects the change of black and white of each column in image <NUM> and finds the starting column of the stroke edge <NUM> in the histogram <NUM>. On the other hand, as illustrated in <FIG>, horizontal histogram projection projects the sum of pixel values for each row in the input image <NUM> and finds the starting row of the stroke edge <NUM> in the histogram <NUM>.

After detecting the stroke edge of the image at <NUM>, the original image is partitioned into sub images and the respective binarization methods are applied separately at <NUM>. In this way, the result benefits from the advantages of the local binarization, though it works with global binarization methods. Moreover, this method sets the window size automatically which is robust enough to adjust for different circumstances. <FIG> is an example showing the benefits of binarization after partitioning. As illustrated, the input images <NUM> and <NUM> when binarized before partitioning as shown at <NUM> and <NUM> do not provide results as good as when binarization is applied after partitioning as shown at <NUM> and <NUM>.

Each binarization result and its inverse binary image are provided in a list of binary mask candidates for selection at <NUM>. The best binary image from the list of candidates is evaluated and selected as the binary mask using the techniques described below.

Referring back to <FIG>, the stroke edges are detected and the partitioning by horizontal and vertical projection of the L channel in the HLS binary image is performed at <NUM> and <NUM> binarization candidates are calculated at <NUM> to generate the binary image <NUM>. The <NUM> binarization candidates in a sample embodiment include CLAHE in LAB color space binarization and its inverse calculated at <NUM>, PCA in LAB color space binarization and its inverse calculated at <NUM>, and L channel of HLS color space binarization and its inverse calculated at <NUM>. Each binary image and its inverse binary image is compared with the contour mask at <NUM> to identify the binary image with the smallest difference. The binary image with the smallest different is normalized at <NUM> to provide a binary image with white characters and black background.

The fusion mask is achieved by the combination at <NUM> of the contour mask and at least one of the edge mask and the binary mask determined as described above. <FIG> illustrates the combination of the calculated edge mask <NUM>, contour mask <NUM>, and binary mask <NUM> to provide the fusion mask <NUM>. The advantages provided by each type of mask guarantee that the text pixels in the original image are fully covered. The resulting fusion mask is then dilated and applied to the original image for application of an inpainting algorithm, such as the afore-mentioned Telea inpainting algorithm, at <NUM>. The inpainting algorithm searches for pixels on the outside edge and then fills the background image into the text area. The resultant image <NUM> is free of all text and the text portions of the image are filled in with the background image data.

Overall, the embodiment described above with respect to <FIG> is robust enough to handle a variety of different original images including handwritten images, fonts with extra outlines of different font colors, and backgrounds of more than one color. As shown in <FIG>, the fusion mask labels all possible text pixels for inpainting. On the other hand, the traditional binary mask <NUM> may mislabel the text pixels of the original image <NUM>, especially along the edge. It leads the inpainting that fills the background with a mixture of background and text color as shown at <NUM>. With the fusion mask <NUM> generated as described above with respect to <FIG>, the text is completely removed and the background is recovered as shown at <NUM>.

<FIG> shows other examples of the technical benefits from application of the fusion mask to images <NUM> and <NUM>. As indicated at <NUM> and <NUM>, the fusion mask is robust for removal of all types of text.

As mentioned above, the normalization of binarization results is the process that calculates the similarity between a contour mask and a binary mask candidate and then elects the best binary image. As shown in <FIG>, this process entails using each of the binary mask candidates determined at <NUM> and comparing the calculated binary mask <NUM> and inverse binary mask <NUM> to the contour mask <NUM> to identify the most similar mask for selection at <NUM>. This normalization not only elects the binary mask but also decides the text as foreground labelled white and background labelled black since the contour mask is the standard. If the text pixel is labelled incorrectly, the inpainting would perform in a reverse way. For example, as shown in <FIG>, when the text in the original image <NUM> is mislabeled as background at <NUM>, the background is filled with text color at <NUM>. However, when the text in the original image <NUM> is correctly labeled as foreground at <NUM>, the background is filled with background color at <NUM>.

<FIG> illustrates a comparison of the text removal and inpainting techniques as described herein to a conventional text removal and inpainting technique used in a conventional translator application. The examples <NUM> and <NUM> each include artificial characters. The results of the conventional translator application still have text residuals as shown at <NUM> and <NUM>, while the results using the text removal and inpainting technique described herein do not have text residuals (<NUM> and <NUM>).

The techniques described herein are designed to extract the mask for inpainting by applying multiple features of the text image and selecting the best binary segmentation over a set of candidates. A fusion mask is used that is the combination of the contour mask and at least one of the edge mask and the binary mask to accurately label pixels for inpainting. The normalization process chooses the best binary mask that is most similar to the contour mask and guarantees that the text pixels are labelled as foreground. The resulting mask for inpainting the text region is appropriate for many cases, especially those cases where the image quality is low or the illumination is non-uniform. The resulting image inpainting may have applications for removing unwanted areas or recovering useful background information in images. Such applications include translator applications, image restoration related applications such as healing image tools on a mobile platform, and other applications of video inpainting or diminished reality. Since the processing is fast, it is particularly useful for real-time mobile platforms.

<FIG> is a block diagram illustrating circuitry for generating the fusion mask for image inpainting as described above with respect to <FIG> according to example embodiments. All components need not be used in various embodiments. One example computing device in the form of a computer <NUM> may include a processing unit <NUM>, memory <NUM>, removable storage <NUM>, and non-removable storage <NUM>. Although the example computing device is illustrated and described as computer <NUM>, the computing device <NUM> may be in different forms in different embodiments. For example, the computing device <NUM> may instead be a smartphone, a tablet, smartwatch, or other computing device. Devices, such as smartphones, tablets, and smartwatches, are generally collectively referred to as mobile devices or user equipment. Further, although the various data storage elements are illustrated as part of the computer <NUM>, the storage may also or alternatively include cloud-based storage accessible via a network, such as the Internet or server-based storage. Also, the method described herein may be implemented in a pipeline on one processing thread or may use multiple processors and/or multiple processing threads as appropriate.

Memory <NUM> may include volatile memory <NUM> and non-volatile memory <NUM>. Computer <NUM> also may include - or have access to a computing environment that includes - a variety of computer-readable media, such as volatile memory <NUM> and non-volatile memory <NUM>, removable storage <NUM> and non-removable storage <NUM>. Computer storage includes random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

Computer <NUM> may include or have access to a computing environment that includes input interface <NUM> that receives the input image for processing, output interface <NUM> that provides the processed image to display <NUM>, and a communication interface <NUM>. Display <NUM> may include a display device, such as a touchscreen, that also may serve as an input device. The input interface <NUM> may include one or more of a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated within or coupled via wired or wireless data connections to the computer <NUM>, and other input devices. The computer <NUM> may operate in a networked environment using a communication connection to connect to one or more remote computers, such as database servers. The remote computer may include a personal computer (PC), server, router, network PC, a peer device or other common DFD network switch, or the like. The communication connection may include a Local Area Network (LAN), a Wide Area Network (WAN), cellular, Wi-Fi, Bluetooth, or other networks. According to one embodiment, the various components of computer <NUM> are connected with a system bus <NUM>.

Computer-readable instructions stored on a computer-readable medium are executable by the processing unit <NUM> of the computer <NUM>, such as a program <NUM>. The program <NUM> in some embodiments comprises software that, when executed by the processing unit <NUM>, performs the image processing operations according to any of the embodiments included herein. A hard drive, CD-ROM, and RAM are some examples of articles including a non-transitory computer-readable medium such as a storage device. The terms computer-readable medium and storage device do not include carrier waves to the extent carrier waves are deemed too transitory. Storage can also include networked storage, such as a storage area network (SAN). Computer program <NUM> may include instruction modules that when processed to cause processing unit <NUM> to perform one or more methods or algorithms described herein.

In an example embodiment, the computer <NUM> includes an edge detector module applying edge detection to detect an edge of text in the original image as an edge mask, an edge mask module using the edge mask to find a set of hierarchical closed edge contours of the text and to fill in the closed edge contours with pixels labeled as possible text as a contour mask, a binarization module applying a binarization method to the original image to convert the original image into a binary image and to detect a stroke edge of text in the binary image, a partition module partitioning the binary image based on the detected stroke edge of text in the binary image and applying at least one binarization method on each image partition to obtain at least one binary mask candidate as foreground for inpainting, a combination module combining the contour mask and at least one of the edge mask and the binary mask into a fusion mask and applying the fusion mask to the original image to extract the text in the original image and to obtain an original background of the original image without text, and an inpainting module inpainting portions of the original image where the text has been extracted. In some embodiments, the computer <NUM> may include other or additional modules for performing any one of or combination of steps described in the embodiments. Further, any of the additional or alternative embodiments or aspects of the method, as shown in any of the figures or recited in any of the claims, are also contemplated to include similar modules.

Although a few embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems within the scope of the appended claims.

It should be further understood that software including one or more computer-executable instructions that facilitate processing and operations as described above with reference to any one or all of steps of the disclosure can be installed in and sold with one or more computing devices consistent with the disclosure. Alternatively, the software can be obtained and loaded into one or more computing devices, including obtaining the software through physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example.

Also, it will be understood by one skilled in the art that this disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the description or illustrated in the drawings. The embodiments herein are capable of other embodiments, and capable of being practiced or carried out in various ways. Also, it will be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. In addition, the terms "connected" and "coupled" and variations thereof are not restricted to physical or mechanical connections or couplings. Further, terms such as up, down, bottom, and top are relative, and are employed to aid illustration, but are not limiting.

The components of the illustrative devices, systems and methods employed in accordance with the illustrated embodiments can be implemented, at least in part, in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. These components can be implemented, for example, as a computer program product such as a computer program, program code or computer instructions tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers.

Also, functional programs, codes, and code segments for accomplishing the techniques described herein can be easily construed as within the scope of the claims by programmers skilled in the art to which the techniques described herein pertain. Method steps associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program, code or instructions to perform functions (e.g., by operating on input data and/or generating an output). Method steps can also be performed by, and apparatus for performing the methods can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), for example.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, a FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

The required elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example, semiconductor memory devices, e.g., electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks). The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.

Those of skill in the art understand that information and signals may be represented using any of a variety of different technologies and techniques.

As used herein, "machine-readable medium" means a device able to store instructions and data temporarily or permanently and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)), and/or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store processor instructions. The term "machine-readable medium" shall also be taken to include any medium, or combination of multiple media, that is capable of storing instructions for execution by one or more processors <NUM>, such that the instructions, when executed by one or more processors <NUM> cause the one or more processors <NUM> to perform any one or more of the methodologies described herein. Accordingly, a "machine-readable medium" refers to a single storage apparatus or device, as well as "cloud-based" storage systems or storage networks that include multiple storage apparatus or devices. The term "machine-readable medium" as used herein excludes signals per se to the extent such signals are deemed to be too transitory.

In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods within the scope of the appended claims. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise.

Claim 1:
A computer-implemented method of finding a text mask in an original image (<NUM>) for extracting text for inpainting, comprising:
applying, with one or more processors (<NUM>), edge detection (<NUM>, <NUM>, <NUM>) to detect an edge of text in the original image as an edge mask (<NUM>, <NUM>);
the one or more processors using the edge mask to find a set of hierarchical closed edge contours of the text and to fill in the closed edge contours with pixels labeled as possible text as a contour mask (<NUM>, <NUM>, <NUM>);
applying, with the one or more processors, a binarization method (<NUM>) to the original image (<NUM>) to convert the original image into a binary image and to detect a stroke edge of the text in the binary image (<NUM>);
partitioning (<NUM>), with the one or more processors, the binary image based on the detected stroke edge of the text in the binary image and applying at least one binarization method on each image partition to obtain at least one binary mask candidate as foreground for inpainting;
combining, with the one or more processors, the contour mask and at least one of the edge mask and the binary mask into a fusion mask (<NUM>) and applying the fusion mask to the original image to extract the text in the original image and to obtain an original background of the original image without the text; and
inpainting (<NUM>), with the one or more processors, portions of the original image where the text has been extracted.