Patent Description:
In the course of using a mobile application, it is sometimes useful to capture an image of a document. For example, a user of a financial management application may capture an image of a receipt related to a financial transaction for use within the application. In some cases, however, due to the limitations of cameras, such as restricted fields of view, a document may be too long to capture in a single image of sufficient quality to identify the document's textual content.

<CIT> discloses "Systems and methods for generating composite images of long documents using mobile video data".

<CIT> discloses a "Method and system for photographing long text subjects that extend beyond borders of a viewfinder".

Any embodiments or examples of the following description which are not within the scope of the appended claims are provided for illustrative purposes.

Optical character recognition (OCR) techniques are generally used to convert images of text into computer-encoded text. OCR results tend to be more accurate when used to evaluate high-resolution, low-noise images of typed, black text against a white background. However, in practice, text in digital images is often noisy, obscured, or otherwise less than ideal. In some cases, for example, a physical document may be relatively obscured or deteriorated as a result of decomposition, excessive use, folding, fingerprints, water damage, or mildew at the time an image of the document is captured. Of course, an image of a document may be of poor-quality for a variety of other reasons (e.g., if the document is no longer extant and better images therefore cannot be obtained). Poor image quality tends to increase OCR processing time and decrease final accuracy. Thus, OCR techniques often fail to produce satisfactory results on poor-quality images.

In order to make information more readily accessible and searchable, individuals, businesses, and governmental agencies often digitize paper forms. For example, the Internal Revenue Service (IRS) may wish to digitize tax forms (e.g., <NUM>, W2, <NUM>-T, or <NUM>-MISC) submitted on paper so that information from the tax forms can be inspected for errors by an automated process. In another example, a law firm may digitize a large number of paper forms received in response to a discovery request so that the documents can be electronically searched for certain keywords. In another example, a web-based genealogical research company may wish to digitize a large number of death certificates in order to make information from the death certificates electronically searchable for customers. In yet another example, a consumer may wish to digitize a large number of receipts to keep track of how much that consumer is spending.

In some cases, individuals may wish to use commercially available software, capable of performing OCR on images of digitized documents (e.g., receipts, pay stubs, etc.), to keep track of their financials. For example, in some cases, a user of the software may capture an image of a receipt related to a financial transaction for use within the software. However, in some cases, due to the limitations of cameras, such as restricted fields of view, a document may be too long to capture in a single image of sufficient quality to identify the document's textual content. Further, without a reference point, it may be difficult for a computer executing the software to determine whether an object, such as a document, is long/tall in an image. For example, it is possible to capture the world's largest building in a single image, and it is also possible to capture a single image of a toy house. However, the computer may not readily be able to determine which of the world's largest building or the toy house is actually taller in real life.

Accordingly, aspects of the present disclosure generally propose techniques for solving the above-identified problem related to not knowing a height difference between two objects without a reference point, and specifically for detecting whether a document captured in an image is a long document. Doing so may improve OCR performance, reduce processing time, and prevent certain errors that occur when an individual is trying to digitize a long document. For example, techniques presented herein may improve the functioning of a computer by reducing power consumption and processing resources (e.g., time using a processor) waste at the computer by allowing the computer to stop processing the image of the document when it is determined that the document is a long document or when the document is determined to be out of bounds of the image. In other words, the computer does not have to waste power and processing resources on processing an image of a document that is of poor quality (e.g., the document is out of bounds of the image) or an image of a document that is incomplete.

As discussed below, identifying whether a document in an image is a long document may be based on various factors. For example, identifying whether a document is a long document may be based on factors such as the size of font detected on the document, bounding information associated with edges of the document, and/or dimensions/aspect ratio of the document appearing in the image, as described in greater detail below.

In some cases, when the detects a long document in a user-captured image, the software may stop processing the user-captured image and alert the user that the document is too long to be captured in a single image. The software may then direct the user to capture multiple images of the document that cover different portions of the document. The software may then process the multiple images (e.g., by stitching the multiple images together) and perform OCR on the document.

<FIG> illustrates a computing environment <NUM> that may be used to perform techniques described in the present disclosure. A computing device <NUM> and a server <NUM> communicate via a network <NUM>. As shown, the computing device <NUM> includes a camera <NUM> used to capture images of documents. In addition, the computing device <NUM> may be used to execute applications <NUM> (e.g., financial management software). In some cases, a user of the computing device <NUM> captures a digital image <NUM> (e.g., as illustrated in <FIG>) of a document using the camera <NUM>. One of the applications <NUM> may send the digital image <NUM> of the document to the server <NUM>. In an alternative embodiment, a scanner may be used in place of the camera <NUM>.

As noted above, in some cases, the document captured in the digital image <NUM> may be too long to capture in a single image of sufficient quality to identify the document's textual content. According to certain aspects, in such a case, a long document detector <NUM> in the computing device <NUM> may determine that the document in the digital image <NUM> is a long document. The long document detector <NUM> may make this determination based on various factors such as the size of font detected on the document, bounding information associated with edges of the document, and/or dimensions/aspect ratio of the document appearing in the image, as described in greater detail below. Further, while only a single digital image <NUM> is illustrated in <FIG>, it should be understood that the long document detector <NUM> may make the determination of whether a document is a long document over a plurality of digital images (e.g., a live video stream).

According to aspects, the computing device <NUM> may also include an out-of-bounds detector <NUM> that is configured to determine bounding information associated with the document in the digital image <NUM>. For example, the out-of-bounds detector <NUM> may be configured to determine whether one or more portions (e.g., edges) of the document in the digital image <NUM> are out of bounds of the digital image <NUM> (e.g., the one or more portions of the document are not contained within the digital image <NUM>). As noted, this bounding information may be used by the long document detector <NUM> in determining whether the document in the digital image <NUM> is a long document. According to certain aspects, while the out-of-bounds detector <NUM> is illustrated as a separate component from the long document detector <NUM>, it should be understood that the out-of-bounds detector <NUM> and the long document detector <NUM> may comprise a single component.

The computing device <NUM> also includes a user alert module <NUM> that may direct a user to capture multiple digital images of the document, focused on different portions of the document, for example, if it is determined that the document in the digital image <NUM> is a long document. In some cases, if the document is determined to be a long document, the user alert module <NUM> may direct the user of the computing device <NUM> to scan the document at close range using the camera <NUM> in a video capture mode. Accordingly, once the user of the computing device <NUM> has captured images of the long document with sufficient quality (e.g., textual content on the document is discernable), the computing device may perform optical character recognition (OCR) on the document to determine the documents textual content and store a digitalized version of the document in a searchable database.

Additionally, in some cases, the server <NUM> may include a long document detector <NUM> and an out-of-bounds detector <NUM> that can perform the same functions as the long document detector <NUM> and out-of-bounds detector <NUM>. For example, instead of the long document detector <NUM> determining whether a document in the digital image <NUM> is a long document, one or more applications <NUM> in the computing device <NUM> may transmit the digital image <NUM> to the server <NUM> (e.g., via the Network <NUM>) and the long document detector <NUM> may determine that the document in the digital image <NUM> is a long document. According to aspects of the present disclosure, if the long document detector <NUM> determines the document to be a long document, the server <NUM> may direct the user alert module <NUM> in the computing device <NUM> to alert the user to capture multiple, focused images of the document, as described above.

While the server <NUM> is depicted as a single server, it should be understood that techniques of the present disclosure can be applied in a cloud-based scheme using multiple physical or virtual computing resources. The long document detector <NUM> and the out-of-bounds detector <NUM> may be distributed across different computing resources as part of a cloud-based computing system. Further, the computing device <NUM> may be considered to be representative of a variety of devices, such as a mobile device, a cellular phone, a smart phone, a tablet, a laptop computer, a desktop computer, a personal digital assistant (PDA), or any computing system that may execute software applications.

<FIG> illustrates a more detailed view of various components that make up the long document detector <NUM>. As illustrated, the long document detector <NUM> comprises a document type classification module <NUM> which determines the type of a document (e.g., a grocery receipt, a tax form, a pay stub, and invoice, etc.) in a user-captured digital image (e.g., digital image <NUM>). For example, a user of the computing device <NUM> may capture a digital image <NUM> (or multiple images, e.g., a video stream) of the form. An example digital image <NUM> is illustrated in <FIG>. The digital image <NUM> may then be sent to and received at the document type classification module <NUM>, which may process the digital image <NUM> to determine the type of document contained within the digital image <NUM>. The document type classification module <NUM> performs optical character recognition (OCR) on the digital image <NUM> to determine the textual content of the document.

The document type classification module <NUM> may then compare the textual content of the document with a pre-defined dictionary of words. According to aspects, the pre-defined dictionary comprises a list of words that are indicative of certain types of documents. For example, in some cases, the pre-defined dictionary may comprise a list of words associated with receipts, such as store names (e.g., Target®, Best Buy®, Costco®, etc.) or names of items typically sold at these stores. The pre-defined dictionary may also comprise words associated with tax forms, such as "W2", "<NUM>", "Tax return", "IRS", etc. Thus, for example, the document type classification module <NUM> may recognize that the document in the digital image <NUM> comprises the word "Costco" and determines that the type of the document in the digital image <NUM> is a receipt, for example, as illustrated in <FIG>.

The document type classification module <NUM> may then load pre-defined metrics (e.g., from memory in the computing device <NUM>, not shown) associated with the detected type of document. For example, assuming that the type of document is a receipt, the associated metrics may include a font size range (e.g., between <NUM> and <NUM> pixels), an aspect ratio threshold (e.g., <NUM>-to-<NUM>), and a minimum height percentage of the document as compared to the whole digital image <NUM> (e.g., <NUM>%). According to aspects, these pre-defined metrics may be used to determine whether or not the document in the digital image <NUM> is a long document, for example, as described in greater detail below.

In some cases, if the document type classification module <NUM> is unable to determine the type of the document in the digital image <NUM>, the document type classification module <NUM> may load default pre-defined metrics or may provide an indication to the user of the computing device <NUM> (e.g., via the user alert module <NUM>) to take a more close-up picture of the document (e.g., if the document type classification module <NUM> is unable to recognize the textual content on the document, for example, due to the text size being recognized as too small).

According to aspects, once the type of the document in the digital image <NUM> is determined and the pre-defined metrics are loaded, the digital image <NUM> may be sent to and received at a document characteristics detection module <NUM>. According to aspects, the document characteristics detection module <NUM> analyzes the digital image <NUM> and determines additional characteristics associated with the document in the digital image <NUM>, for example, based on the analysis. For example, as illustrated, the document characteristics detection module <NUM> may analyze the digital image <NUM> with a font size detection module <NUM> to determine a text size of text appearing on the document. The document characteristics detection module <NUM> may also analyze the digital image <NUM> to determine bounding information associated with edges of the document in the digital image <NUM> using an out-of-bounds detection module <NUM>. Further, the document characteristics detection module <NUM> may also analyze the digital image <NUM> to determine the dimensions and aspect ratio of the document in the digital image <NUM>, for example, using a document dimension and aspect ratio detection module <NUM>. According to aspects, these analyses may then be used to determine whether the document in the digital image <NUM> is a long document, as described in greater detail below.

As noted, the document characteristics detection module <NUM> includes a font size detection module <NUM> for determining the font size of the text on the document in the digital image <NUM>. According to aspects, the font size detection module <NUM> may determine the font size of the text based, for example, on an analysis of text blocks. For example, the font size detection module <NUM> may determine a number of text blocks on the document in the digital image <NUM> by drawing bounding rectangles around lines of text on the document. For example, given a document with three lines of text, the font size detection module <NUM> determines that there are three text blocks by drawing bounding rectangles around each line of text. The font size detection module <NUM> may then determine the average height of the text blocks. According to aspects, based on the average height of the text blocks, the font size detection module <NUM> may approximate the size of the text on the document in the digital image <NUM>. That is, the average height of the text blocks may be an estimator of text size.

Additionally, as noted above, the document characteristics detection module <NUM> includes an out-of-bounds detection module <NUM> for determining bounding information associated with edges of the document in the digital image <NUM>. For example, the out-of-bounds detection module <NUM> may be used for determining whether any part of the document in the digital image <NUM> is out of bounds of the digital image <NUM>. Further, as will be described in greater detail below, the bounding information may be used when determining whether the document in the digital image <NUM> is a long document.

<FIG> illustrates a more detailed view of the out-of-bounds detection module <NUM>. As illustrated, the out-of-bounds detection module <NUM> comprises an image processing module <NUM> that prepares the digital image <NUM> for contour analysis by the contour analyzer <NUM>. Based on the contour analysis, described in greater detail below, an out-of-bounds decision module <NUM> may determine that one or more edges of the document in the digital image <NUM> are out of bounds of the digital image <NUM>.

As noted, the image processing module <NUM> processes the digital image <NUM> to prepare the digital image <NUM> for contour analysis. For example, the image processing module <NUM> may receive the digital image <NUM>. The image processing module <NUM> may then scale the digital image <NUM> down (e.g., reduce the size of the digital image <NUM>). The scaled-down digital image <NUM> may then be processed by a clustering algorithm (e.g., OpenCV kmeans method) to separate the pixels in the scaled-down digital image <NUM> into two clusters: one cluster representing the foreground (e.g., the document in the digital image <NUM>), and another cluster representing the background. According to aspects, the image processing module <NUM> may then create a segmented image such that pixels belonging to the foreground cluster of the scaled-down digital image <NUM> are black and pixels belonging to the background cluster of the scaled-down digital image <NUM> are white. According to aspects, creating this segmented image allows the contour analyzer <NUM> to more-easily determine/find the contours in the segmented image (i.e., corresponding to the digital image <NUM>).

Accordingly, once the image processing module <NUM> has created the segmented image, the contour analyzer <NUM> may process the segmented image to determine contours in the segmented image. The contour analyzer <NUM> may then analyze a hierarchy of the contours (e.g., the nesting of contours) found in the segmented image and determine which contours are open or closed. A simple example of a closed contour is a rectangle whose four sides are all contained within the segmented image, whereas an open contour is, for example, a rectangle with one or more sides of the rectangle outside of the segmented image. More specifically, an open contour is one that does not have any child contour in the hierarchy (e.g., the contour does not bound another contour). According to aspects, for each continuous contour (e.g., whether open or closed) found in the segmented image, the contour analyzer <NUM> is configured to create a bounding rectangle that encompasses that contour. The contour analyzer <NUM> may then compute the area (e.g., pixels squared) of each bounding rectangle and determine the bounding rectangle with the largest area.

According to aspects, if the bounding rectangle with the largest area bounds an open contour and if one or more sides of this bounding rectangle touches one or more of the edges of the segmented image, the out-of-bounds decision module <NUM> may conclude that part of the document in the digital image <NUM> is out of bounds. Otherwise, if the bounding rectangle with the largest area bounds an open contour and if none of the sides of this bounding rectangle touches the edges of the segmented image, the out-of-bounds decision module <NUM> may conclude that the document in the digital image <NUM> is not out of bounds of the digital image <NUM>. Likewise, if the bounding rectangle with the largest area bounds a closed contour, the out-of-bounds decision module <NUM> may conclude that the document in the digital image <NUM> is not out of bounds of the digital image <NUM>.

According to aspects, if the out-of-bounds decision module <NUM> determines that part of the document is out of bounds of the digital image <NUM>, out-of-bounds decision module <NUM> may direct the user alert module <NUM> to inform the user of the computing device <NUM> that the document in the digital image <NUM> is out of bounds and direct the user to capture an additional digital image fully encompassing the document.

Additionally, in some cases, if the out-of-bounds decision module <NUM> determines that part of the document is out of bounds of the digital image <NUM> (e.g., the bounding rectangle with the largest area bounds an open contour and one or more sides of this bounding rectangle touches one or more edges of the digital image <NUM>), the contour analyzer <NUM> may determine the sides of the document that are out of bounds of the digital image <NUM>. According to aspects, the contour analyzer <NUM> may determine which sides of the document are out of bounds of the digital image <NUM> based on an analysis of the open contour that is bounded by the bounding rectangle with the largest area. For example, the contour analyzer <NUM> may determine which side of the bounding rectangle with the largest area touches an edge of the digital image <NUM>, and may deduce which corners of this bounding rectangle are out of bounds. For example, if the top side of this bounding rectangle touches an edge of the digital image <NUM>, the contour analyzer <NUM> may determine that the top left and top right corners are missing According to aspects, based on the missing corners, the contour analyzer <NUM> may deduce which sides of the document are out of bounds of the digital image <NUM>.

Further, once the contour analyzer <NUM> determines the sides of the document that are out of bounds of the digital image <NUM>, the contour analyzer <NUM> may supply the user alert module <NUM> with this information, which may, in turn, inform the user of the computing device <NUM> of the sides of the document that are out of bounds. This information may be used by the user of the computing device to capture an additional image of the document, for example, as described above. According to aspects, this process of determining the bounding information of the document in the digital image <NUM> will be described in greater detail below, with reference to <FIG>.

Returning to <FIG>, as noted above, the document characteristics detection module <NUM> includes a document dimension and aspect ratio detection module <NUM> for determining the dimensions and aspect ratio of the document in the digital image <NUM>. According to aspects, the document dimension and aspect ratio detection module <NUM> may be configured to determine the dimensions and aspect ratio of the document in the digital image <NUM> by again analyzing the bounding rectangle with the largest area (e.g., as described above in relation to the out-of-bounds detection module <NUM>). In some cases, the document dimension and aspect ratio detection module <NUM> may independently determine the contours in the digital image <NUM> (e.g., using the same or similar techniques described above) or may reuse contour information determined by the contour analyzer <NUM>. According to aspects, the document dimension and aspect ratio detection module <NUM> may determine the height and width of the bounding rectangle with the largest area (e.g., the rectangle that bounds the document in the digital image <NUM>) and, using the determined height and width, determine the aspect ratio (e.g., width divided by height).

According to aspects, the long document detector <NUM> may determine the characteristics of the document described above (e.g., font size, bounding information, and dimensions/aspect ratio) over multiple consecutive digital images and may keep track of the results. These results (i.e., the document characteristics), in addition to the pre-defined metrics corresponding to the determined document type described above and one or more conditions (e.g., decision rules), may then be used by the long document decision module <NUM> to determine whether the document in the digital image <NUM> is a long document or not, as described below.

For example, in some cases, the long document decision module <NUM> may determine the document in the digital image <NUM> to be a long document if the font size of the text on the document (e.g., as determined by the font size detection module <NUM>) is less than a lower bound of the loaded pre-defined font size range over the multiple consecutive images and if the bounding information of the document in the multiple consecutive images indicates that one or more sides of the document are out-of-bounds. In such case, it may be assumed that the camera <NUM> is being held far away from the document such that the text on the document is small and also that the user of the computing device <NUM> has difficulty fitting the document within a single camera frame.

In other cases, the long document decision module <NUM> may determine the document in the digital image <NUM> to be a long document if, over the multiple consecutive digital images, a height dimension of the document (e.g., as determined by the document dimension and aspect ratio detection module <NUM>) is greater than the pre-defined minimum height percentage threshold of each of the multiple consecutive digital images (e.g., the document's height is greater than 'X' percent of the digital image <NUM>), and if the aspect ratio of the document (e.g., as determined by the document dimension and aspect ratio detection module <NUM>) is greater than the pre-defined aspect ratio threshold for the multiple consecutive images. In such a case, it may be assumed that the user of the computing device <NUM> held the camera <NUM> high enough so that the whole document would fit inside one camera frame, and since the camera was held high, the text on the document is small. Additionally, by checking whether the document height is greater than the defined minimum height percentage threshold of a digital image, this reduces the possibility of a misdetermination where the document is actually a short document but the user just happened to hold the camera at a height such that the text size appears small.

In yet other cases, the long document decision module <NUM> may determine the document in the digital image <NUM> to be a long document if the font size of the text on the document in the digital image <NUM> is within the pre-defined font size range and if the bounding information indicates that one or more sides of the document is consistently out of bounds over the multiple consecutive images.

Otherwise, if the font size of the text on the document in the digital image <NUM> is at the top of the font size range or above and if the document is completely inside the first digital image <NUM> (e.g., the bounding information indicates that no sides of the document are out of bounds of the digital image <NUM>), then the long document decision module <NUM> may conclude that the document in the digital image <NUM> is not a long document.

According to aspects, if the long document decision module <NUM> determines that the document in the digital image <NUM> is a long document (e.g., based on the techniques described above), the long document decision module <NUM> may direct the user alert module <NUM> to alert the user that the document is too long to be captured in a single image and direct the user to capture multiple digital images of the document, each digital image of the multiple digital images focusing on a different portion of the document such that a combination of each digital image of the multiple digital images of the document captures the entire document. For example, based on an identification that a first image of a document includes missing edges at the top and bottom of the document, computing device <NUM> can request that the user capture additional images representing a portion of the document above the portion captured in the first image and a portion of the document below the portion captured in the first image. The computing device <NUM> may then stitch together the multiple digital images to generate a single digital image of the document and may feed the single digital image of the document into an OCR module for text extraction and analysis.

<FIG> is a flow diagram illustrating an exemplary method <NUM> for processing digital images of a document, for example, to determine if the document is a long document, according to certain aspects of the present disclosure. The method <NUM> may be performed, for example, by a computing device <NUM> and/or a server <NUM>.

The method <NUM> begins at <NUM> by obtaining a plurality of digital images of the document. As noted above, this may involve a user capturing the plurality of images (e.g., digital image <NUM>) of the document using a camera (e.g., camera <NUM>). The plurality of digital images may then be forwarded to and obtained by, for example, a component configured to detect whether the document is a long document (e.g., long document detector <NUM>). According to aspects, a long document may be a document that is too long to capture in a single image of sufficient quality to identify the document's textual content.

At <NUM>, a type of the document in the plurality of images is determined. For example, the document type classification module <NUM> may receive the plurality of digital images of the document and perform optical character recognition (OCR) on the plurality of digital images to determine the textual content of the document. The document type classification module <NUM> may then compare the textual content of the document with a dictionary of words that are indicative of certain types of documents. For example, in some cases, the document may include a store name such as "Groceries-R-Us". The document type classification module <NUM> may search the dictionary of words for "Groceries-R-Us", and may determine that "Groceries-R-Us" is listed in the dictionary of words as being associated with the document type: receipts. Thus, the document type classification module may determine the type of the document in the plurality of documents to be a receipt.

At <NUM>, the computing device <NUM> loads one or more pre-defined metrics associated with documents of the determined type. As noted above, pre-defined metrics includes a font size range for text appearing on documents of the determined type, an aspect ratio threshold, and, optionally, a minimum height percentage of documents of the determined type as compared to a whole digital image of that type of document. For example, assuming that the document captured in the plurality of images is a receipt, the pre-defined metrics may include a font range between <NUM> and <NUM> pixels, an aspect ratio threshold of <NUM>-to-<NUM>, and a minimum height percentage of the document as compared to the whole digital image <NUM> of <NUM>%. According to aspects, these pre-defined metrics represent average values for these three categories of metrics based on typical, non-long receipts. That is, a typical, non-long receipt generally has a font size of <NUM> to <NUM> pixels, an aspect ratio of <NUM>-to-<NUM>, and usually takes up <NUM>% of the height of an image capturing this non-long receipt.

At <NUM>, the computing device <NUM> determines one or more characteristics of the document in the plurality of digital images, based on one or more analyses performed on the plurality of digital images of the document.

For example, the computing device <NUM> may determine a font size of textual content on the document, bounding information associated with one or more sides of the document, dimensions of the document, and/or an aspect ratio of the document. As noted above, the computing device <NUM> may determine the font size of the textual content on the document based on an average text block size, for example, as described above.

Additionally, the computing device <NUM> may determine the bounding information by, for at least one image of the plurality of digital images, scaling down the plurality of images of the document, segmenting the scaled down image into groups of pixels corresponding to the foreground of the scaled down image (e.g., representing the document) and another group of pixels corresponding to the background of the image. The computing device may then find the contours in the segmented image and draw bounding rectangles around the found contours. According to aspects, the computing device <NUM> may then deduce that at least one side of the document is out of bounds of the segmented image if the rectangle with the largest area in the segmented image bounds an open contour and if one or more sides of this rectangle touch an edge of the segmented image. Additionally, the computing device may determine the dimensions and aspect ratio of the document by measuring the height and width of the rectangle with the largest area, as described above.

At <NUM>, the computing device <NUM> compares the one or more characteristics of the document with the one or more pre-defined metrics. For example, the computing device <NUM> compares the determined font size of the textual content on the document with the font size range. According to aspects, the computing device <NUM> may compare each of the determined characteristics of the document with their corresponding pre-defined metrics.

At <NUM>, the computing device <NUM> determines the document to be a long document based, at least in part, on the comparison. For example, the computing device <NUM> may determine that the document in the plurality of images is a long document when the font size of the textual content on the document is less than a lower bound of the loaded pre-defined font size range over multiple consecutive images of the plurality of images (e.g., a configurable number of images) and when the bounding information of the document in the multiple consecutive images indicates that one or more sides of the document are out-of-bounds. Additionally, the computing device <NUM> may determine the document in the plurality of images to be a long document when, over the multiple consecutive digital images, a height dimension of the document is greater than the pre-defined minimum height percentage threshold of each of the multiple consecutive digital images and when the aspect ratio of the document is greater than the pre-defined aspect ratio threshold for the multiple consecutive images. Further, the computing device <NUM> may determine that the document in the plurality of images is a long document when the font size of the text on the document in the plurality of images is within the pre-defined font size range and when the bounding information indicates that one or more sides of the document is consistently out of bounds over the multiple consecutive images.

At <NUM>, if the document in the plurality of documents is not a long document, the method <NUM> ends. In some cases, the computing device <NUM> may determine that the document in the plurality of images is not a long document if the font size of the text on the document in the digital image <NUM> is at or above the top of the font size range and if the document is completely inside the first digital image <NUM> (e.g., the bounding information indicates that no sides of the document are out of bounds of the digital image <NUM>). In such a case, the user of the computing device <NUM> may be allowed to proceed and store a digital copy of the document.

If, however, at <NUM>, the document in the plurality of documents is a long document, the method continues to <NUM> where the computing device <NUM> directs the user to capture multiple images of the document, each covering a different portion of the document. In some cases, the computing device <NUM> may direct the user to scan the document with the camera <NUM> (e.g., using a video capture mode). In response, the computing device may stitch together these images and store a digital copy of the document. In some cases, the computing device may perform OCR on the document and store textual content associated with the document in a searchable database.

<FIG> illustrates example operations <NUM> for determining bounding information of a document in a digital image, according to certain aspects of the present disclosure. According to certain aspects, example operations <NUM> may be performed by one or more components capable of processing a digital image, such as the out-of-bounds detection module <NUM> of the computing device <NUM>.

Operations <NUM> begin at <NUM> with the out-of-bounds detection module <NUM> obtaining a first digital image of a document. An example of a first digital image (e.g., digital image <NUM>) is illustrated in <FIG>. Additionally, while not illustrated in <FIG>, the out-of-bounds detection module <NUM> may scale down the first digital image, for example, to make it easier and faster to determine the contours of the first digital image.

According to aspects, at <NUM>, the out-of-bounds detection module <NUM> segments the first digital image into two groups of pixels: pixels associated with a foreground of the first digital image (e.g., colored white, not shown) and pixels associated with a background of the first digital image (e.g., colored black, not shown). In some cases, the out-of-bounds detection module <NUM> segments the first digital image by processing the first digital image with a clustering algorithm (e.g., OpenCV kmeans algorithm). According to aspects, creating this segmented image allows the out-of-bounds detection module <NUM> to more-easily determine/find the contours in the segmented image (i.e., corresponding to the digital image <NUM>).

According to aspects, once out-of-bounds detection module <NUM> has created the segmented first digital image, at <NUM>, the out-of-bounds detection module <NUM> detects contours in the segmented first digital image.

At <NUM>, the out-of-bounds detection module <NUM> analyzes a hierarchy of the contours (e.g., the nesting of contours) found in the segmented first digital image and decides which contours are open or closed. According to aspects, and as noted above an example of a closed contour is a rectangle whose four sides are all contained within the segmented image. An example closed contour that may be found when processing the segmented first digital image is illustrated at 702A in <FIG>. According to aspects, an open contour is, for example, a rectangle with one or more sides of the rectangle outside of the segmented image. More specifically, an open contour is one that does not have any child contour in the hierarchy (e.g., the contour does not bound another contour). <FIG> illustrates an example of an open contour that may be found when processing the segmented first digital image, which may comprise lines 702B, 702C, and 702D.

At <NUM>, for each continuous contour (e.g., whether open or closed) found in the segmented image, the out-of-bounds detection module <NUM> is configured to create a bounding rectangle that encompasses that contour. For example, as illustrated in <FIG>, the out-of-bounds detection module <NUM> may create a first bounding rectangle 802A around the closed contour 702A and a second bounding rectangle 802B around the open contour corresponding to lines 702B, 702C, and 702D.

At <NUM>, the out-of-bounds detection module <NUM> determines the area (e.g., pixels squared) of each bounding rectangle. For example, with reference to <FIG>, the out-of-bounds detection module <NUM> determines the area of the first bounding rectangle 802A and the area of the second bounding rectangle 802B. The out-of-bounds detection module <NUM> may then determine which of the first bounding rectangle 802A or the second bounding rectangle comprises the most area. In the example illustrated in <FIG>, the out-of-bounds detection module <NUM> may determine that the second bounding rectangle 802B (e.g., which bounds the open contour comprising lines 702B-702D) comprises the most area.

At <NUM>, the out-of-bounds detection module <NUM> determines whether one or more sides of the document in the segmented first digital image are out of bounds. For example, if the bounding rectangle with the largest area in the first digital image bounds an open contour and if one or more sides of this bounding rectangle touches one or more of the edges of the segmented image, the out-of-bounds detection module <NUM> may conclude that part of the document in the first digital image is out of bounds. For example, with reference to <FIG>, since the second bounding rectangle 802B bounds an open contour (e.g., lines 702B-702D) and since the second bounding rectangle 802B touches one of the sides of the first digital image (e.g., at <NUM>), the out-of-bounds detection module <NUM> determines that a portion of the document in the first digital image is out of bounds.

Otherwise, as noted above, if the bounding rectangle with the largest area bounds an open contour and if none of the sides of this bounding rectangle touches the edges of the segmented image, the out-of-bounds detection module <NUM> may conclude that the document in the first digital image is not out of bounds of the first digital image. Likewise, if the bounding rectangle with the largest area bounds a closed contour, the out-of-bounds detection module <NUM> may conclude that the document in the first digital image is not out of bounds of the first digital image.

According to aspects, if the out-of-bounds detection module <NUM> determines that one or more sides of the document are out of bounds of the first digital image, at <NUM>, the out-of-bounds detection module <NUM> also determines which sides of the document are out of bounds. For example, the out-of-bounds detection module <NUM> may analyze the second bounding rectangle 802B and determine which corners of the second bounding rectangle 802B are captured in the first digital image, as well as which corners of the bounding rectangle <NUM> are out of bounds of the first digital image. For example, with reference to <FIG>, the out-of-bounds detection module <NUM> may determine that the bottom left and bottom right corners are contained within the first digital image and that the top left and top right corners are out of bounds of the first digital image. According to aspects, based on the top left and top right corners being out of bounds, the out-of-bounds detection module <NUM> may deduce that the top side of the document in the first digital image is out of bounds. In some cases, the out-of-bounds detection module <NUM> may determine which corners are out of bounds (and deduce which side of the document is out of bounds) based on an analysis of which side of the second bounding rectangle 802B touches an edge of the first digital image. For example, as illustrated in <FIG>, the out-of-bounds detection module <NUM> will determine that the top-side of the second bounding rectangle 802B touches the edge of the first digital image at <NUM>, and deduce that the top side of the document is out of bounds.

According to aspects and as noted above, if the out-of-bounds detection module <NUM> determines that one or more sides of the document in the first digital image are out of bounds, the out-of-bounds detection module <NUM> may direct the user alert module <NUM> to notify the user of the computing device <NUM> to capture the at least one additional digital image of the document.

According to aspects, when a font size on the document (e.g., as determined by the font size detection module <NUM>) is greater than or equal to an upper font size threshold, the user alert module <NUM> alerts the user to capture an image of the entire document at a further distance than the first digital image. In such a case, the computing device <NUM> may conclude that, since the font size of the document is greater than or equal to an upper font size threshold/bound, allowing the user to take an image of the document at a further distance will still result in a digital image where the textual content of the document is still discernable.

However, when the font size of text on the document (e.g., as determined by the font size detection module <NUM>) is less than or equal to a lower font size threshold/bound, the user alert module <NUM> alerts the user to capture multiple images of the document, each image of the multiple images focusing on a different portion of the document such that a combination of each image of the multiple images of the document captures the entire document. Thereafter, the computing device <NUM> may stitch together the multiple images to generate a single image of the document.

<FIG> illustrates an example image processing system <NUM> that determines, among other things, whether a document in a digital image is a long document, according to certain aspects of the present disclosure. As shown, the image processing system <NUM> includes, without limitation, a central processing unit (CPU) <NUM>, one or more I/O device interfaces <NUM> which may allow for the connection of various I/O devices <NUM> (e.g., keyboards, displays, mouse devices, pen input, etc.) and camera <NUM> to the image processing system <NUM>, network interface <NUM>, a memory <NUM>, storage <NUM>, and an interconnect <NUM>.

CPU <NUM> may retrieve and execute programming instructions stored in the memory <NUM>. Similarly, the CPU <NUM> may retrieve and store application data residing in the memory <NUM>. The interconnect <NUM> transmits programming instructions and application data, among the CPU <NUM>, I/O device interface <NUM>, network interface <NUM>, memory <NUM>, and storage <NUM>. CPU <NUM> can represent a single CPU, multiple CPUs, a single CPU having multiple processing cores, and the like. Additionally, the memory <NUM> represents random access memory. Furthermore, the storage <NUM> may be a disk drive. Although shown as a single unit, the storage <NUM> may be a combination of fixed or removable storage devices, such as fixed disc drives, removable memory cards or optical storage, network attached storage (NAS), or a storage area-network (SAN).

As shown, memory <NUM> includes a long document detector <NUM> and a user alert module <NUM>. The long document detector <NUM> comprises a document type classification module <NUM>, a document characteristics detection module <NUM>, and a long document decision module <NUM>. A digital image of a document can be sent to the long document detector <NUM> from the I/O devices <NUM>, camera <NUM>, or from another source, such as the network <NUM>. The document type classification module <NUM> can determine a type of the document in the digital image and load pre-defined metrics (e.g., pre-defined metrics <NUM>) associated with documents of the determined type. The document characteristics detection module <NUM> can determine one or more characteristics associated with the documents, such as a font size of textual content on the document, bounding information associated with one or more sides of the document, dimensions of the document, and/or an aspect ratio of the document. The long document decision module <NUM> can decide whether the document in the digital image is a long document, for example, based on the loaded pre-defined metrics, the one or more characteristics of the document, and one or more conditions (e.g., decision rules).

As shown, storage <NUM> includes the digital image <NUM> (e.g., captured by the user via the camera <NUM>), pre-defined metrics <NUM>, and decision rules <NUM>. According to aspects, the pre-defined metrics <NUM> may include a font size range for text appearing on documents of the determined type, an aspect ratio threshold, and/or a minimum height percentage of documents of the determined type as compared to a whole digital image of that type of document. Additionally, according to certain aspects, the decision rules <NUM> may be used by the long document detector <NUM> to decide whether the document in the digital image <NUM> is a long document, for example, as explained above.

According to certain aspects, if it is determined that the document in the digital image <NUM> is a long document, the long document decision component may generate a notification and provide it to the user alert module <NUM>. Upon receiving the notification, the user alert module <NUM> may inform the user of the computing device <NUM> to capture additional digital images of the document, each focusing on a different portion of the document. According to aspects, the computing device <NUM> may stitch together the additional digital images, perform OCR on the stitched together digital images, and store textual content of the document, recognized as a result of OCR, as a searchable digital document <NUM>. In some cases, the computing device <NUM> may transmit the digital image <NUM> and/or the searchable digital document <NUM> via the network interface <NUM> for storage in a cloud database.

Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

More specific examples a computer readable storage medium include: an electrical connection having one or more wires, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the current context, a computer readable storage medium may be any tangible medium that can contain, or store a program.

Claim 1:
A computer-implemented method for processing digital images of a document, comprising:
obtaining a plurality of digital images of the document;
determining a type of the document by performing optical character recognition on the plurality of digital images to determine textual content of the document;
loading one or more pre-defined metrics associated with the document based on the determined type of the document, wherein the one or more pre-defined metrics comprise a font size range associated with documents of the determined type and an aspect ratio threshold associated with documents of the determined type;
determining one or more characteristics of the document based on one or more analyses performed on the plurality of digital images of the document, wherein the one or more characteristics comprises a font size of text on the document and an aspect ratio of the document;
determining bounding information associated with one or more sides of the document;
comparing the one or more characteristics of the document with the one or more pre-defined metrics, wherein the comparing comprises comparing the determined font size with the loaded font size range and comparing the aspect ratio of the document with the aspect ratio threshold; and
determining whether the document is a long document based, at least in part, on the comparison and the determined bounding information.