Generating searchable text for documents portrayed in a repository of digital images utilizing orientation and text prediction neural networks

The present disclosure relates to generating computer searchable text from digital images that depict documents utilizing an orientation neural network and/or text prediction neural network. For example, one or more embodiments detect digital images that depict documents, identify the orientation of the depicted documents, and generate computer searchable text from the depicted documents in the detected digital images. In particular, one or more embodiments train an orientation neural network to identify the orientation of a depicted document in a digital image. Additionally, one or more embodiments train a text prediction neural network to analyze a depicted document in a digital image to generate computer searchable text from the depicted document. By utilizing the identified orientation of the depicted document before analyzing the depicted document with a text prediction neural network, the disclosed systems can efficiently and accurately generate computer searchable text for a digital image that depicts a document.

BACKGROUND

Online or “cloud” storage systems have become an increasingly popular avenue for storing and managing electronic media generated via client devices. For example, some conventional online storage systems allow users to capture digital photographs and videos on a smart phone, and then store the captured media items on one or more remote servers for later retrieval and use. Similarly, conventional online storage systems also allow users to save and access electronic document files (e.g., word processor files, spreadsheet files, and/or other electronic files) or other digital data.

In some cases, users choose to store documents by capturing digital photos of the documents (e.g., paper receipts, invoices, pay stubs, bills, reports, and so forth) and storing the digital photos online via one or more remote servers. Although conventional systems allow users to store and access these digital photos, these conventional systems have several shortcomings.

For example, conventional online storage systems are inflexible in managing digital image repositories that include digital photos of documents. To illustrate, conventional online storage systems generally store, search, and access digital photos utilizing rigid meta-data such as a user given title, date of creation, and/or technical specifications. As a result, conventional systems cannot flexibly search and utilize digital photos portraying documents beyond the scope of these rigid categories.

In addition, conventional systems are also inefficient in relation to managing digital images that portray documents. For instance, users frequently utilize conventional online storage systems to search for and utilize text portrayed in documents, but conventional systems lack efficient means for identifying pertinent digital images portraying documents with text. Accordingly, conventional online storage systems waste significant time and computing resources as users inefficiently search through digital images stored on remote servers to identify digital text in the digital images. For example, many conventional systems generate and provide thousands of thumbnails (or other digital image representations) for users to review and search in attempting to identify text portrayed in digital images.

These shortcomings are often exacerbated by inaccuracies of digital images within conventional online storage systems. For example, digital photos of documents in conventional systems are frequently skewed, blurred, shaded, rotated or otherwise distorted. These inaccuracies only increase the difficulty of searching for and utilizing documents portrayed in digital images utilizing conventional systems. For instance, some conventional systems have utilized optical character recognition algorithms for identifying text from scanned documents. Although these optical character recognition algorithms can identify text in relatively sterile scanned documents, they fail to accurately, efficiently, or flexibly generate searchable text from documents portrayed in user-captured digital images such as digital images captured with a smartphone. Imperfections, distortions, rotations, and other digital content (e.g., images of people or places) in digital images generally stored in online digital image repositories undermine the accuracy and efficiency of such conventional systems.

SUMMARY

One or more embodiments of the present disclosure provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, methods, and non-transitory computer readable storage media that generate searchable text for documents portrayed in a repository of digital images utilizing an orientation neural network and/or text prediction neural network. For example, disclosed systems can efficiently and accurately identify a digital image depicting a document within a repository of digital images. Moreover, the disclosed systems can automatically modify the digital image by identifying document boundaries, rectifying the document, and rotating the digital image such that text of the document is properly oriented. Furthermore, the disclosed systems can accurately identify the contents of the documents depicted in the digital image utilizing a text prediction neural network efficiently trained utilizing synthetically generated text training data. The disclosed systems can then utilize the searchable text generated from documents portrayed in the digital images to flexibly search, organize, and manage the digital images.

To illustrate, in at least one embodiment, disclosed systems use a document detection neural network to identify a digital image depicting a document within a repository of digital images. Additionally, disclosed systems can apply an orientation neural network to the identified digital image to detect the orientation of the depicted document within the digital image. Moreover, disclosed systems can crop image portions (e.g., word boxes) from the depicted document within the digital image and utilize a text prediction neural network to generate computer searchable text. In particular, the disclosed systems can utilize the text prediction neural network trained with synthetic training data to generate computer searchable text based on the image portions (e.g., word boxes) from the depicted document and the detected orientation of the document.

Additional features and advantages of the present invention will be set forth in the description that follows, and in part will be obvious from the description, or may be learned by the practice of the various embodiments described. The features and advantages of such embodiments may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features will become more fully apparent from the following description and appended claims.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure include a digital image character recognition system that identifies documents portrayed in digital images and generates searchable text from the documents utilizing an orientation neural network and/or text prediction neural network. In particular, in one or more embodiments, the digital image character recognition system automatically detects a digital image that depicts a document within a repository of digital images (e.g., utilizing a document detection neural network) and converts the document to searchable text. For instance, upon identifying a digital image that depicts a document, the digital image character recognition system can determine a document boundary, rectify the document, and utilize an orientation neural network to identify (and modify) the orientation of the depicted document. Furthermore, the digital image character recognition system can then generate searchable text from the document utilizing an optical character recognition algorithm that includes a text prediction neural network.

To illustrate, in one or more embodiments, the digital image character recognition system detects a digital image that includes a depiction of a document. The digital image character recognition system then utilizes an orientation neural network to detect an orientation of the document within the digital image. The digital image character recognition system can then crop a word box from the digital image (i.e., a word box from the depiction of the document in the digital image). The digital image character recognition system can then utilize a text prediction neural network trained with synthetic training data to generate computer searchable text for the portion of the depiction of the document based on the word box and the detected orientation of the document.

As just mentioned, in one or more embodiments, the digital image character recognition system utilizes a document detection neural network to identify a digital image depicting a document. In particular, the digital image character recognition system can utilize a deep neural network trained based on training digital images portraying training documents to identify digital images in an image repository that depict documents. In this manner, the digital image character recognition system can classify digital images that do (or do not) portray documents.

Upon identifying digital images that portray documents, the digital image character recognition system can modify the digital images (e.g., to prepare the digital image for transformation into searchable text). For example, in some embodiments, the document detection neural network identifies document boundaries and corners. Utilizing the boundaries and corners, the digital image character recognition system can then crop and rectify the detected depiction of the document. Additionally, the digital image character recognition system can correct visual imperfections in the depiction of the document in the digital image (e.g., colors, shadows, contrast, skews, etc.).

Moreover, as mentioned above, in one or more embodiments the digital image character recognition system utilizes an orientation neural network to identify the orientation of the depicted document and correct the orientation of the document. For instance, the digital image character recognition system can train an orientation neural network based on training digital images at a variety of different orientations. The digital image character recognition system can then utilize the trained orientation neural network to classify orientation of identified documents in digital images (e.g., classify as rotated 0 degrees, 90 degrees, 180 degrees, and/or 270 degrees).

Upon identifying orientation of the document portrayed within the digital image, the digital image character recognition system can apply an optical character recognition algorithm that includes both a word detector and a text prediction neural network to generate searchable text. For example, in some embodiments, the digital image character recognition system utilizes a word detector that identifies depictions of text in the digital image and crops word boxes for the depictions of text. Furthermore, in one or more embodiments, the digital image character recognition system provides the word boxes (and/or the identified orientation) to a text prediction neural network. The text prediction neural network analyzes the word boxes (in light of the identified orientation) and generates computer searchable text for the word boxes.

In one or more embodiments, the digital image character recognition system also trains the text prediction neural network. For instance, the digital image character recognition system can train the text prediction neural network utilizing training text (e.g., training text boxes illustrating distorted words) and ground truth text labels for the training text. Specifically, the digital image character recognition system can train the text prediction neural network by analyzing the training text, predicting a text label, and then comparing the predicted text label with the ground truth text labels.

As mentioned above, the digital image character recognition system can also train the text prediction neural network utilizing synthetic training data. To illustrate, the digital image character recognition system can identify a corpus of words, a set of fonts, and a set of font distortions. The digital image character recognition system can generate synthetic training data by sampling from the corpus of words and applying a font (from the set of fonts) and distortion (from the set of distortions). The digital image character recognition system can then utilize the resulting synthetic training word(s) in training the text prediction neural network.

Upon generating searchable text from documents portrayed in a digital image, the digital image character recognition system can also utilize the searchable text. For example, in some embodiments, the digital image character recognition system indexes searchable text so users can search a repository of digital images based on words that are depicted in individual digital images. Furthermore, in some embodiments, the digital image character recognition system can utilize the generated computer searchable text to generate a digital overlay to the digital image that includes the searchable text. For instance, the digital image character recognition system can generate a digital overlay to provide visual search results on a digital image (e.g., highlight text on a digital image when showing search results). In addition, the digital image character recognition system can utilize a digital overlay so that users can copy and paste directly from documents portrayed in digital images. Moreover, in some embodiments, the digital image character recognition system can also categorize the digital images based on the depicted documents in the digital image and provide the digital images to a user associated with a particular category.

The digital image character recognition system provides several advantages over conventional systems. For example, the digital image character recognition system can flexibly store, search, and access digital photos based on contents of documents portrayed in the digital images. For instance, as just mentioned, the digital image character recognition system can analyze a repository of digital images, generate searchable text based on documents portrayed in the digital images, and then search the repository of digital images based on the searchable text. Similarly, the digital image character recognition system can flexibly organize, classify, and manage digital images based on contents of documents portrayed in the digital images.

Additionally, the digital image character recognition system also improves efficiency. As an initial matter, the digital image character recognition system avoids unnecessary and time-consuming browsing of digital images to identify pertinent content. Moreover, the digital image character recognition system can identify content of documents portrayed in digital images, avoiding exorbitant computer costs associated with generating and providing thumbnail images for a repository of digital images for browsing pertinent content.

Furthermore, the digital image character recognition system implements a variety of processes to reduce burden on computing resources in generating searchable text. For example, by utilizing the orientation neural network to determine the orientation of documents depicted in digital images the digital image character recognition system can reduce processing power required to generate searchable text. Additionally, utilizing the orientation of the document depicted in the digital image, the digital image character recognition system can avoid analyzing word boxes in multiple orientations (e.g., in all four rotations), thus requiring a fraction of the computer processing power. Similarly, the digital image character recognition system can avoid time and computational costs associated with generating training data for one or more neural networks. For instance, as discussed above, the digital image character recognition system can generate synthetic training data that allows the digital image character recognition system to generate a trained neural network without the cost of observing and gathering training samples with corresponding ground truth labels.

Furthermore, the digital image character recognition system is able to accurately analyze and recognize text within digital images that depict documents. For example, by utilizing a document detection neural network, an orientation neural network, and optical character recognition algorithm (that itself utilizes an accurate word detector and text prediction neural network), the digital image character recognition can generate searchable text that accurately reflects documents portrayed in digital images. For instance, the digital image character recognition system can accurately generate searchable text, even with digital images captured from smartphones that often include imperfections, distortions, rotations and extraneous content.

As illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to described features and benefits of the event document management system. Additional detail is now provided regarding the meaning of these terms. As used herein, the term “digital image” refers to any digital visual representation. In particular, the term “digital image” includes a digital photograph, symbol, or icon. Furthermore, a digital image includes an image or frame from a digital video. For example, the term “digital image” includes digital files with the following file formats: JPEG, TIFF, PNG, BMP, RAW, CR2, NEF, or PDF. Furthermore, as used herein, the term “repository of digital images” refers to a collection of digital images. For example, the term “repository of digital images” includes a collection of digital images stored on a remote server for one or more users.

As used herein the term “depiction of a document” (sometimes referred to as “document,” “displayed document,” or “portrayed document” herein) refers to a visual representation of a document within a digital image. Moreover, as used herein, the term “document” refers to any depiction, representation, or portrayal of text. In particular, the term “document” refers to a physical object comprising text, such as numbers, letters, or other characters. For example, the term “document” includes paper with text, hand written notes, signs with text, boards with text, posters with text, or a display screen with text. Additionally, a document can include hand written, printed, or computer-generated text.

Furthermore, as used herein, the term “text” refers to any visual depiction, representation, or portrayal of an element of speech or writing. In particular, the term “text” refers to hand written or printed elements of speech. Specifically, the term text includes hand written or printed characters of any language. For example, text includes characters from languages such as, but not limited to, English, Spanish, Mandarin, Hindi, Gujarati, Japanese, Russian, Arabic, and French.

Additionally, as used herein, the term “neural network” refers to a machine learning algorithm that can be tuned (e.g., trained) based on training inputs to estimate an unknown function. In particular, the term “neural network” can include a plurality of interconnected artificial neurons that transmit data to other artificial neurons that generate outputs based on one or more inputs. More specifically, the plurality of interconnected neurons learns to estimate complex elements by utilizing prior estimations and other training data. For example, the term “neural network” can include deep neural networks, convolutional neural networks (“CNN”), fully convolutional neural networks (“FCN”), or recurrent neural networks (“RNN”).

The digital image character recognition system can utilize a variety of neural networks, including document detection neural networks, orientation neural networks, and/or text prediction neural networks. Additional detail regarding exemplary architectures of these neural networks is provided below. However, as used herein, the term “orientation neural network” refers to a neural network trained to identify orientation. In particular, an orientation neural network includes a neural network trained to classify (and/or correct) orientation of a document portrayed in a digital image. In addition, as used herein, the term “text prediction neural network” refers to a neural network trained to predict text. In particular, a text prediction neural network includes a neural network trained to identify text from a document portrayed in a digital image (e.g., from a word box cropped from the document). Moreover, “document detection neural network” trained to identify digital images portraying documents. In particular, the document neural network includes a document detection neural network trained to classify digital images that depict documents (from digital images that do not depict documents).

As used herein, the term “orientation” refers to angular alignment of an object. In particular, the term “orientation” refers to the angular alignment of a document (or text within a document) portrayed within a digital image (e.g., alignment relative to horizontal or some other reference orientation). For example, the orientation can include an angle that represents the direction in which a document is positioned relative to a reference orientation.

Moreover, as used herein, the term “ground truth” refers to a known value in relation to a training sample. For example, a “ground truth orientation” refers to a known orientation of a training document for the purposes of training an orientation neural network. Similarly, as used herein, the term “training” is used as a modifier to indicate information utilized to train a neural network. Thus, for instance, a “training document” refers to a document utilized to train a neural network.

As used herein, the term “word box” refers to a visual representation of text portrayed in a digital image. For example, a word box may include a cropped digital image that represents one or more text objects from a document that are detected as a grouping of text (e.g., text characters that do not have spacing between them).

As used herein, the term “synthetic training data” refers to data that is generated for training a neural network. In particular, the term “synthetic training data” includes data that is generated to imitate data that is organically observed or measured in real world situations and, also, includes corresponding ground truth labels for training a neural network. For example, synthetic training data may include synthetic training digital images. As used herein, the term “synthetic training digital image” refers to a digital image that is generated for training a neural network. Moreover, “synthetic training digital image” refers to a digital image generated to imitate images that are organically created in real world situations, and, also include corresponding ground truth labels for training a neural network. For example, a synthetic training digital image can be an image of a word or text that is generated by combining different characteristics such as words from a corpus of words, fonts, and distortions. Furthermore, the synthetic training digital image can include a ground truth label (a “text label”) that identifies the text that is portrayed in the digital image of the word, text, or document. Moreover, as used herein, the term “text label” refers to a label, tag, identifier, list, array, set of strings, or other collections of data that represent the computer searchable text that is represented in the digital image of the word, text, or document.

As used herein, the term “computer searchable text” (or “searchable text”) refers to digital text that can be indexed and/or searched by a computing device. In particular, the term “computer searchable text” includes text structures such as strings, integers, characters, HTML mark up, and ASCII that can be searched by a computing device.

As used herein, the term “corpus of words” refers to a collection of text. In particular, the term “corpus of words” includes a collection of words from a single language (e.g., from one or more from works of literature, dictionaries, and/or online databases). More specifically, a “corpus of words” can be represented as a data set, list, data table, array, database or other collections of data that represent text. For example, a corpus of words includes a collection of words from the English language from one or more English dictionaries, novels, and/or articles.

As used herein, the term “set of fonts” refers to a collection of fonts. In particular, a set of fonts includes a data set, list, data table, array, database, or other collections of data that represent fonts. Furthermore, as used herein, the term “font” refers to a set of text characters in a specific style. In particular, the term “font” refers to a particular size, weight, and style for characters in text to create text in unique visual forms. For example, font can include computer-based fonts, such as, Times New Roman, Helvetica, Courier, Calibri, and Arial. Furthermore, font can include different machine-printed or handwritten fonts.

As used herein, the term “a set of distortions” refers to a collection of alterations to text. In particular, a set of distortions includes a data set, list, data table, array, database or other collection of data that represent modifications to text. Furthermore, as used herein, the term “distortion” refers to modifications or transformations that affect the visual appearance of a digital image. In particular, a distortion includes a geometric and/or photometric transformation. For example, distortions include, but are not limited to, image blurring, rotations, underlines, noises, applying lens filters, creases, textures, and/or shadows.

As used herein, the term “modified word” refers to a word from a corpus of words that has been altered by applying fonts and/or distortions to the word. In particular, a modified word is a word that has been visually altered to reflect an applied font and/or an applied distortion. For example, a modified word includes a word that has been altered to be represented in Helvetica and also altered to have a shadow effect on the boundary of the word.

As used herein, the term “token” refers to a data element. A token can be an entry in a data set, list, data table, array, database, index or other collections of data. In particular, “token” refers to an entry in an index utilized to search digital images. More specifically, a “token” refers to a data element that represents a word or a set of words in a database utilized to search digital images.

As used herein, the term “document category” refers to a label that represents a particular type of document. For example, document category includes, but is not limited to, receipts, memos, invoices, essays, and forms.

Additional details of various example embodiments of the digital image character recognition system will now be discussed with respect to the figures.FIG. 1is a schematic diagram illustrating an example embodiment of exemplary system environment (“environment”)100in which the digital image character recognition system can operate. As illustrated inFIG. 1, environment100can include server(s)102, client device106, and network110.

Although,FIG. 1illustrates a singular client device106, it will be appreciated that environment100can include any number of client devices (few or greater than shown). Similarly, althoughFIG. 1illustrates a particular arrangement of server(s)102, client device106, and network110, various additional arrangements are possible.

Server(s)102, client device106, and network110may be communicatively coupled with each other either directly or indirectly (e.g., through network110, discussed in greater detail below in relation toFIG. 12). Moreover, server(s)102and client device106may include any type of computing device (including one or more computing devices as discussed in greater detail below in relation toFIG. 12).

As mentioned above, environment100includes server(s)102. Server(s)102can generate, store, receive, and/or transmit any type of data. For example, server(s)102may receive data from client device106and send data to client device106. In one or more embodiments, server(s)102may comprise a data server. Server(s)102can also comprise a communication server or a web-hosting server.

As shown inFIG. 1, server(s)102can include online content management system104. In particular, online content management system104provides functionality by which a user (not shown inFIG. 1) can generate, manage, and/or store digital content. For example, a user can generate new digital content using client device106. Subsequently, a user utilizes client device106to send the digital content to online content management system104hosted on server(s)102via network110. Online content management system104then provides many options that a user may utilize to store the digital content, organize the digital content, and subsequently search for, access, and view the digital content. Additional detail regarding online content management system104is provided below (e.g., in relation toFIG. 13and online content management system1302).

Additionally, server(s)102can include digital image character recognition system112. In particular, in one or more embodiments, digital image character recognition system112uses server(s)102to generate computer searchable text for digital content (e.g., a digital image). For example, digital image character recognition system112can use server(s)102to detect a digital image that depicts a document, enhance the displayed document in the digital image, identify the orientation of the displayed document, and generate computer searchable text for the digital image of the displayed document. Furthermore, server(s)102can store the generated computer searchable text for a digital image that depicts a document on server(s)102. Additionally, client device106can access the generated computer searchable text and the digital image corresponding to the generated computer searchable text via network110.

In one or more embodiments, client device106includes computer devices that allow users of the device to access and interact with digital content, such as digital images that depict documents. For example, client device106can include a smartphone, tablet, desktop computer, laptop computer, or another electronic device. Client device106can include one or more applications (e.g., online content management system application108) that allows a user (through client device106) to access and interact with digital content such as digital images that depict documents. For example, online content management system application108can include a software application installed on client device106. Additionally, or alternatively, online content management system application108can include a software application hosted on server(s)102, which may be accessed by client device106through another application, such as a web browser. In at least one embodiment, online content management system application108also interfaces with a camera associated to client device106in order to capture digital images (e.g., digital photographs).

Moreover, digital image character recognition system112may be implemented in whole, or in part, by the individual elements of environment100. AlthoughFIG. 1illustrates digital image character recognition system112implemented with regard to server(s)102, it will be appreciated that components of digital image character recognition system112can be implemented by any of the components of environment100(e.g., by the client device106). The components of digital image character recognition system112will be discussed in more detail with regard toFIG. 10below.

As just mentioned, digital image character recognition system112can generate computer searchable text for a digital image. For example,FIGS. 2A-2Dillustrate digital image character recognition system112detecting digital images that depict documents, enhancing the depicted documents, identifying the orientation of the depicted documents, and generating computer searchable text for the digital images that depict documents. In particular,FIG. 2Aillustrates digital image character recognition system112detecting digital images that depict documents from a repository of digital images (e.g., a repository managed via online content management system104). Furthermore,FIG. 2Billustrates digital image character recognition system112enhancing a depicted document in a digital image. Additionally,FIG. 2Cshows digital image character recognition system112utilizing an orientation neural network to detect the orientation of a depicted document in the digital image. Moreover,FIG. 2Dillustrates digital image character recognition system112utilizing a word detector and a text prediction neural network to generate computer searchable text for a depicted document in a digital image.

As shown inFIG. 2A, digital image character recognition system112utilizes document detection neural network206to detect digital images204b-dthat include a depiction of documents from repository of digital images202. For instance, in one or more embodiments, digital image character recognition system112accesses a repository of digital content to identify digital images204a-204d(from repository of digital images202). Furthermore, in some embodiments, digital image character recognition system112utilizes document detection neural network206on digital images202to detect digital images that depict documents (“detected digital images”). Moreover, in one or more embodiments, document detection neural network206outputs detected digital images208.

As shown inFIG. 2A, digital image character recognition system112accesses a repository of digital content to identify digital images portraying physical documents. In some embodiments, the repository of digital content is stored on online content management system104. Digital image character recognition system112accesses online content management system104to analyze the repository of digital content. In one or more embodiments, digital image character recognition system112accesses and analyzes a repository of digital content on a client device (e.g., client device106).

As shown inFIG. 2A, repository of digital images202can include any digital image (e.g., digital images that include depictions of documents and digital images that do not include depictions of documents). For instance, digital image204adoes not include a depiction of a document whereas digital images204b-204dinclude a depiction of a digital document (e.g., a building sign in digital image204b, a document on a table in digital image204c, and a document accompanied by other items in digital image204d).

Moreover, as shown inFIG. 2A, after accessing repository of digital images202, digital image character recognition system112utilizes document detection neural network206on digital images204a-dfrom repository of digital images202.

For instance, in some embodiments, document detection neural network206can determine that digital images204b,204c, and204dall include a displayed document. Thus, document detection neural network206outputs a “positive” determination in connection with digital images204b,204c, and204dto indicate that a displayed document is present. Furthermore, document detection neural network206can determine that digital image204adoes not include a displayed document and output a “negative” determination in connection with digital image204a. Moreover, digital image character recognition system112can associate metadata with each of the digital images204a-d, such as a tag or line item, that indicates digital images204a-dinclude a displayed document or do not include a displayed document.

In relation toFIG. 2A, digital image character recognition system112trains document detection neural network206to detect documents within digital images. More specifically, in the embodiment illustrated inFIG. 2A, digital image character recognition system112comprises a convolutional neural network image classifier. Digital image character recognition system112trains document detection neural network206to determine whether a digital image depicts a document by utilizing positive training documents (that portray documents), negative training documents (that do not portray documents), and ground truth classifications (indicating a classification of the training documents). The digital image character recognition system112trains document detection neural network206by analyzing positive training documents and negative training documents, predicting classifications for the positive training documents and negative training documents, and comparing the predicted classifications with the ground truth classifications. After digital image character recognition system112repeatedly trains document detection neural network206with positive and negative training images, document detection neural network206learns to accurately classify digital images and identify digital images portraying documents. In one or more embodiments, digital image character recognition system112trains and utilizes a document classification neural network as described in U.S. patent application Ser. No. 15/658,289 and U.S. patent application Ser. No. 15/658,291, which are incorporated herein by reference.

Furthermore, digital image character recognition system112can create a set of document digital images208. In particular, as just discussed, digital image character recognition system112determines, via document detection neural network206, that digital images204b,204c, and204ddepict physical documents. Thus, digital image character recognition system112includes digital images204b,204c, and204din set of document digital images208. As mentioned above, digital image character recognition system112can associate metadata with each of digital images204b-dto indicate that digital images204b-dinclude a displayed document.

As mentioned above, upon identifying digital images portraying documents, digital image character recognition system112can generate enhanced digital images. In particular, digital image character recognition system112can generate enhanced digital images to prepare the digital images for more accurate and efficient analysis in generating searchable text. For example, inFIG. 2B, digital image character recognition system112modifies displayed document212in digital image204dto generate enhanced digital image216. For instance, digital image character recognition system112identifies a boundary and corners of displayed document212, removes blemishes, adjusts skew, and modifies displayed document212in digital image204dto generate enhanced digital image216.

For example, as shown inFIG. 2B, digital image character recognition system112generates enhanced digital image216by identifying the boundaries and corners of displayed document212within digital image204d. Digital image character recognition system112can identify a boundary and corners of displayed document212in a variety of ways. For instance, in one or more embodiments, digital image character recognition system112utilizes computer vision techniques, machine learning, and/or image space conversion to identify displayed document212within digital image204dand determine a boundary and/or corners for displayed document212.

To illustrate, digital image character recognition system112can identify boundaries by performing edge detection, line identification, corner identification, and quadrilateral identification. Furthermore, in one or more embodiments, digital image character recognition system112utilizes the Canny edge detection technique to identify all edges of displayed document212in digital image204d. Moreover, in some embodiments, digital image character recognition system112utilizes a machine learning-based edge detection technique to detect the boundaries of displayed document212. For example, one such machine learning-based technique includes a machine learning model based on a Structured Edge Detector algorithm. Moreover, in some embodiments, digital image character recognition system112utilizes a Hough Transformation technique to identify lines in a digital image to detect edges (e.g., boundaries).

Furthermore, digital image character recognition system112can also utilize a variety of approaches to identify corners of a document within a digital image. For instance, in some embodiments, digital image character recognition system112identifies intersections among identified edges or lines in digital image204dto identify possible corners of displayed document212.

Also, in some embodiments, digital image character recognition system112utilizes geometric constraints to rule out corners that do not correspond to a displayed document boundary. For instance, the geometric constraints can include the measurement of angles between two lines, line length, line length relative to other lines of digital image204d, ending points of a line, number of total lines of digital image204d, instances of perpendicular and/or parallel lines with respect to a given line, and other geometric constraints.

Furthermore, digital image character recognition system112can indicate the boundary of a displayed document by identifying a quadrilateral from the identified edges and corners of the displayed document. For instance, digital image character recognition system112identifies quadrilaterals that represent displayed document212and scores the identified quadrilaterals to accurately predict the boundary of displayed document212. In some embodiments, digital image character recognition system112adjusts the identified quadrilateral to ensure that the quadrilateral aligns with the edges of displayed document212.

In addition to identifying edges and corners of displayed documents to identify a boundary for the displayed document, digital image character recognition system112can also modify a digital image of a displayed document. For example, some modifications include cropping, rectifying, altering colors, removing shadows, correcting background and foreground elements in a displayed document, and so forth. For instance, digital image character recognition system112utilizes identified boundaries and corners of a displayed document to crop a digital image to remove portions of the digital image outside the boundary of the displayed document. As a result, as shown inFIG. 2B, digital image character recognition system112modifies digital image204dinto enhanced digital image216, where enhanced digital image216is cropped in relation to the identified boundaries and corners.

Furthermore, in some embodiments, digital image character recognition system112can rectify a displayed document within a digital image to remove or correct skewing or other image warping features. In some embodiments, digital image character recognition system112attempts to generate a digital image that represents a displayed document in a square or rectangular shape. For example, in one or more embodiments, digital image character recognition system112rectifies displayed document212within digital image204dto remove any skew or warping features in digital image204d. Furthermore, digital image character recognition system112removes warping features to generate a rectangular displayed document in enhanced digital image216.

Additionally, in some embodiments, digital image character recognition system112can rotate enhanced digital image216. For instance, digital image character recognition system112can rotate a digital image to position the edges of the displayed document to a particular alignment. To illustrate, digital image character recognition system112can rotate a digital image such that one or more edges of enhanced digital image216are vertical or horizontal.

Moreover, digital image character recognition system112can modify digital image204din order to correct other imperfections such as uneven lighting, shadows, glares, grainy or unclear foreground, or other visual abnormalities that may affect the visual clarity of displayed document212. In some embodiments, digital image character recognition system112converts displayed document212to grayscale (e.g., removing any color within the displayed document) and denoises displayed document212(e.g., removing background of displayed document212of excess variation). Additionally, in some embodiments, digital image character recognition system112can subsample displayed document212utilizing a Poisson equation to produce a displayed document that includes a crisp foreground and visibility with high contrast, and a background that is nearly uniform white (or uniform gradient).

Furthermore, in one or more embodiments, digital image character recognition system112upsamples the displayed document212to restore lost resolution. For example, in some embodiments, digital image character recognition system112determines whether each pixel in the subsampled version of displayed document212is foreground or background. Then, digital image character recognition system112creates a tri-map version of displayed document212with each pixel of displayed document212labeled as “foreground”, “background”, or “unknown”. Moreover, digital image character recognition system112assumes pixels are background pixels if they are labeled as background or unknown. Additionally, digital image character recognition system112creates contrast between the pixels labeled as foreground pixels and creates a gradient for the background and unknown pixels. As a result, digital image character recognition system112produces displayed document212in a high-resolution format that includes a crisp and visible foreground and a background that is nearly uniform in gradient.

Accordingly, as shown inFIG. 2B, digital image character recognition system112outputs enhanced digital image216, which represents a cropped version of displayed document212(e.g., no other objects and backgrounds from digital image204dare present). Furthermore, enhanced digital image216does not contain imperfections from digital image204d, such as the shadow in digital image204d. In some embodiments, enhanced digital image216also includes indicators for the boundary220and corners218of displayed document212. For example, in one or more embodiments, digital image character recognition system112generates enhanced digital image216utilizing one or more approaches as described in U.S. patent application Ser. No. 15/658,289, which is incorporated by reference herein.

AlthoughFIG. 2Bonly illustrates modifications to digital image204d, in some embodiments, digital image character recognition system112enhances a plurality of digital images (e.g., each digital image from set of document digital images208fromFIG. 2A). Moreover, although digital image204dinFIG. 2Billustrates particular objects (e.g., paper and pencil) or imperfections (e.g., shadow), digital image character recognition system112can modify a digital image that portrays any variety of different objects or imperfections (e.g., blurring, skew, underlines, noises, lens filters, creases, and discoloration).

As mentioned above, in addition to identifying and enhancing digital images of displayed documents, digital image character recognition system112can utilize an orientation neural network to identify and/or correct the orientation of a displayed document in a digital image. For example,FIG. 2Cillustrates utilizing an orientation neural network to identify the orientation of a displayed document (and provide the displayed document in a corrected orientation) in accordance with one or more embodiments. For instance,FIG. 2Cshows digital image character recognition system112providing enhanced digital image216(fromFIG. 2B) as input to orientation neural network222. Digital image character recognition system112utilizes orientation neural network222to analyze enhanced digital image216and identify orientation of the document portrayed in enhanced digital image216, (e.g., to output digital image224in a corrected orientation such that text of the document is aligned horizontally with characters arranged vertically, right-side-up, so that they are legible).

As illustrated inFIG. 2C, enhanced digital image216is incorrectly orientated (i.e., text within the displayed document within enhanced digital image216is upside down). As mentioned above, this can decrease the accuracy and efficiency of generating computer searchable text within the digital image. Accordingly, as shown inFIG. 2C, digital image character recognition system112utilizes orientation neural network222to analyze displayed documents (i.e., digital images of displayed documents) to identify the orientation of the displayed document.

Specifically, in the embodiment ofFIG. 2C, orientation neural network222is a deep convolutional neural network that is trained to analyze and classify a document based on orientation. For instance, the orientation neural network is a deep CNN trained to classify documents into one of four orientation categories (e.g., 0, 90, 180, 270 degree orientation). Furthermore, where digital image character recognition system112has already aligned the edge of a document to a particular orientation (as discussed inFIG. 2B), digital image character recognition system112can utilize orientation neural network222to accurately and efficiently identify one of four orientation categories.

To illustrate, in relation toFIG. 2C, orientation neural network222generates an integer classification representing orientation of the displayed document. For example, orientation neural network222analyzes enhanced digital image216and identifies current orientation223as an integer value of “2” (which represents that the displayed document in enhanced digital image216was in a position that is currently 180 degrees from a horizontal alignment of text with characters arranged right-side-up). In one or more embodiments, orientation neural network222can classify a digital image of a displayed document with an integer of “0”, “1”, “2”, or “3” to represent a current orientation as 0, 90, 180, and 270 degree rotations from a correct orientation of the displayed document. Additional detail regarding training an orientation neural network to generate an orientation classification is provided below (e.g., in relation toFIG. 3).

Moreover, in some embodiments, digital image character recognition system112can rotate enhanced digital image216into a corrected orientation after identifying the orientation of displayed document in enhanced digital image216. For instance, digital image character recognition system112can rotate enhanced digital image216by 180 degrees to generate modified digital image224in a correctly orientated position.

Although the foregoing example utilizes an integer to describe a particular orientation classification, orientation neural network222can also output an orientation classification in a variety of forms. For example, orientation neural network222can identify and represent the orientation of enhanced digital image216in the form of an angle (so that the output is a number between 0 and 360) or some other form (e.g., text or vector). Moreover, although the foregoing example describes four classification categories, in some embodiments, orientation neural network222digital image character recognition system112can include a different number of classifications (e.g., 365 angles).

As mentioned above, in addition to utilizing an orientation neural network to identify and/or correct the orientation of a displayed document in a digital image, digital image character recognition system112also utilizes a word detector and text prediction neural network to generate computer searchable text. For example,FIG. 2Dillustrates word detector226and text prediction neural network230. Word detector226identifies depictions of text and crops word boxes228of those depictions of text from the document portrayed in modified digital image224. Moreover, text prediction neural network230utilizes word boxes228as input to determine text in word boxes228and generate computer searchable text232.

More specifically, as shown inFIG. 2D, digital image character recognition system112utilizes modified digital image224(that was previously enhanced by digital image character recognition system112and rotated into a correct orientation by utilizing orientation neural network222as described inFIGS. 2B-2C). In one or more embodiments, digital image character recognition system112does not utilize an enhanced (or otherwise modified) digital image in utilizing word detector226and text prediction neural network230. In some embodiments, digital image character recognition system112utilizes a digital image with or without one or more of the enhancements/modifications described above in relation toFIGS. 2B-2C.

As illustrated inFIG. 2D, however, digital image character recognition system112provides modified digital image224of a displayed document to word detector226. Word detector226analyzes the modified digital image224to identify and/or extract word boxes. Word detector226can utilize a variety of approaches to identify word boxes. To illustrate, in relation toFIG. 2D, word detector226utilizes a computer vision approach to identify word boxes. Specifically, word detector226utilizes Maximally Stable Extremal Regions (“MSER”).

MSER can locate connected regions at different thresholds, or levels, of the digital image. For example, MSER detects blobs in images, where each blob is a region in a digital image that differs in properties (such as brightness, contrast, and color) compared to surrounding regions in the digital image. For example, a region (e.g., blob) might include regions of a digital image that have a higher contrast or consist of colored pixels compared to a uniform gradient surrounding (e.g., pixels that represent text on a uniform background). The MSER algorithm detects such regions in a digital image and strings them together into word and line detections. InFIG. 2D, word detector226crops these identified regions into separate word boxes. Furthermore, word detector226utilizes the MSER regions to detect spacing between regions/blobs.

Furthermore, in some embodiments, word detector226can utilize MSERs to detect regions where text is bright (e.g., white text) and the background is dark (e.g., colored or black background). Furthermore, in some embodiments, word detector226can utilize MSERs to detect regions of text where the background is not uniform (e.g., a textured background) by looking for regions that differ from that textured background. Thus, the word detector226can identify word boxes comprising white text on a dark (or textured) background as well as dark text on a white background.

For instance, word detector226can utilize MSERs to locate connected regions in modified digital image224. Word detector226, utilizing MSERs, detects regions of modified digital image224wherein pixel properties are different from that of surrounding regions within modified digital image224. As a result, word detector226generates bounding boxes in such regions and crops the regions to generate word boxes228. For example, word detector226detects the region portraying “Jan” and “14” and crops those regions to provide word boxes228for “Jan” and “14”. Ultimately, in some embodiments, word detector226generates word boxes228for all detected text in modified digital image224.

In one or more embodiments, word detector226can be an object detection system that includes a Region-based Convolutional Neural Network (“RCNN”) that detects locations of objects in digital images. For example, the RCNN can be trained to locate objects such as text in digital images. Furthermore, in one or more embodiments, the RCNN locates objects and provides bounding boxes for the located objects.

As shown inFIG. 2D, upon identifying word boxes228, digital image character recognition system112provides word boxes228as input to text prediction neural network230. In some embodiments, text prediction neural network230only accepts fixed size digital images. Therefore, in some embodiments, word detector226generates word boxes in a fixed size digital image. For example, word detector226may include more than one word (e.g., more than one region) for each word box. Furthermore, in some embodiments, word detector226may separate a word (e.g., a region) into more than one word boxes if the word/region is too large to fit the fixed size input requirement of text prediction neural network230.

Furthermore, as mentioned above, digital image character recognition system112provides word boxes228as input for text prediction neural network230to generate computer searchable text232from word boxes228. In particular, digital image character recognition system112utilizes a text prediction neural network230trained to identify text from word boxes (e.g., images of one or more words) based on training word boxes and corresponding ground truth labels. In one or more embodiments, digital image character recognition system112generates synthetic training data to efficiently train the text prediction neural network230to identify text. Specifically, in one or more embodiments, text prediction neural network230comprises a stack of convolutional layers, a stock of bidirectional long short term memory layers, and a connectionist temporal classification layer trained to recognize and generate computer searchable text232from word boxes228. Additional detail regarding the architecture and training of text prediction neural networks is provided below (e.g., in relation toFIGS. 5-6).

As shown inFIG. 2D, text prediction neural network230generates computer searchable text232from word boxes228as a data set of strings representing words depicted in word boxes228. For example, word box228for “Jan” is recognized and converted to computer searchable text232for “Jan”. Furthermore, text prediction neural network230also generates computer searchable text232for all other word boxes228.

Upon generating computer searchable text, digital image character recognition system112also associates computer searchable text232with digital image204d(the original digital image of modified digital image224). For example, in relation to the embodiment ofFIG. 2, digital image character recognition system112associates computer searchable text232by adding the text as meta-data to digital image204d. Moreover, digital image character recognition system112creates a digital layer on digital image204dthat maps/overlays computer searchable text232onto digital image204d(e.g., maps the searchable text in the same location of word boxes228within the digital image204d). Furthermore, digital image character recognition system112associates computer searchable text232as a token in an index corresponding to the digital image204d. Additionally, digital image character recognition system112stores modified digital image224(e.g., on server(s)102via on online content management system104) as a digital image.

Moreover, in one or more embodiments, digital image character recognition system112can also utilize the generated computer searchable text232for various functionalities such as, but not limited to, searching the content of digital images, selecting words within digital images to copy and paste, and organizing/distributing digital images based on computer searchable text within a digital image. Examples of such functionalities are discussed in further detail below (e.g., in relation toFIGS. 8 and 9).

As just mentioned, digital image character recognition system112can train an orientation neural network to identify the orientation of displayed documents in digital images. For example, as shown inFIG. 3, digital image character recognition system112trains an orientation neural network to generate a trained orientation neural network in accordance with one or more embodiments. In particular, as illustrated inFIG. 3, digital image character recognition system112utilizes training documents302, predicted orientations306, ground truth orientations310, and calculated loss312to train orientation neural network304and generate trained orientation neural network314.

For example, as shown inFIG. 3, digital image character recognition system112accesses training documents302. In one or more embodiments, digital image character recognition system112can access training documents302from storage in server(s)102. Furthermore, training documents302, in one or more embodiments, are documents with associated ground truth orientations310. As shown, training documents302include documents having multiple orientations. In some embodiments, training documents302are in either a 0, 90, 180, or 270 degree orientation (relative to horizontal, upright orientation of text in the documents), however, in some embodiments, training documents302can have a different orientation.

In one or more embodiments, training documents302are generated by digital image character recognition system112. In particular, digital image character recognition system112can modify orientation of existing documents to generate training documents302and corresponding ground truth orientations. Additional detail regarding generating training documents302is described below (e.g., in relation toFIG. 4). Furthermore, training documents302, in some embodiments, can be documents obtained from other third-party repositories or from online content management system104where the documents are annotated with the ground truth orientation of the document.

Furthermore, as illustrated inFIG. 3, digital image character recognition system112utilizes training documents302to train orientation neural network304. In particular, digital image character recognition system112provides training documents302as input to the orientation neural network304. In one or more embodiments, orientation neural network304analyzes training documents302to predict the orientation of training documents302.

In relation toFIG. 3, orientation neural network304comprises a deep convolutional neural network. Specifically, the orientation neural network304comprises a deep CNN that includes convolutional layers, pooling layers, fully connected layers, ReLu layers, and normalization layers, that feed to an output layer that produces a predicted orientation.

In one or more embodiments, digital image character recognition system112utilizes orientation neural network304that is a deep neural network based on the Inception Resnet v2 architecture. The deep neural network based on the Inception Resnet v2 architecture can be utilized for image classification. However, in some embodiments, digital image character recognition system112modifies the last layer in the deep neural network based on the Inception Resnet v2 architecture to classify orientation instead of classifying an image into content categories. Additionally, the deep neural network can be trained using the ImageNet database. In some embodiments, digital image character recognition system112tunes the ImageNet database to train a deep neural network to be biased towards orientation detection.

More specifically, as shown inFIG. 3, orientation neural network304generates predicted orientations306after analyzing training documents302. In one or more embodiments, predicted orientations306can be in the form of an integer, a string, a number (e.g., an angle), or other data that represents the direction in which one or more training documents302are positioned.

Additionally, as shown inFIG. 3, orientation neural network304utilizes training documents302, predicted orientations306, and ground truth orientations310to learn to accurately predict the orientation of displayed documents in digital images. For example, digital image character recognition system112compares predicted orientations306and ground truth orientation310(e.g., labeled orientation integers of training documents) to train orientation neural network304. In particular, digital image character recognition system112compares predicted orientations306and ground truth orientations310utilizing loss function308, which generates calculated loss312. In particular, loss function308can determine if predicted orientations306from orientation neural network304accurately reflect the ground truth orientations310of the training documents302. Digital image character recognition system112can utilize a variety of loss functions, including squared error loss function, 0-1 indicator function, or cross entropy loss function to determine calculated loss312.

Moreover, orientation neural network304then utilizes calculated loss312to train (e.g., tune) orientation neural network304in predicting the orientation of displayed documents. In particular, digital image character recognition system112provides information from calculated loss312to orientation neural network304(e.g., back-propagates calculated loss312) to adjust parameters of orientation neural network304. In particular, the orientation neural network304can modify internal parameters (e.g., weighting parameters for analyzing features within layers of the neural network) to minimize calculated loss312from loss function308.

In one or more embodiments, digital image character recognition system112utilizes the orientation neural network304to repeatedly generate predicted orientations306, compares predicted orientations306with ground truth orientations310, and modifies internal parameters to minimize calculated loss312. Digital image character recognition system112repeats this process until orientation neural network304is sufficiently accurate (e.g., until convergence or until analyzing a threshold number of training documents). In this manner, digital image character recognition system112generates trained orientation neural network314.

As mentioned above, digital image character recognition system112can generate training documents to train an orientation neural network. For example,FIG. 4illustrates digital image character recognition system112generating training documents to train an orientation neural network in accordance with one or more embodiments. Specifically,FIG. 4illustrates digital image character recognition system112utilizing initial documents402to generate training documents406and corresponding ground truth orientations404.

More specifically, as shown inFIG. 4, digital image character recognition system112utilizes initial documents402at a known orientation to generate training documents406to train an orientation neural network. Digital image character recognition system112can identify initial documents402from a variety of sources. For example, in one or more embodiments, digital image character recognition system112identifies initial documents402from a third party resource or from documents stored by online content management system104. Moreover, digital image character recognition system112can identify initial documents402together with a known orientation. For instance, digital image character recognition system112can capture documents previously reviewed or annotated for correct orientation (e.g., capture documents that users have already manually rotated to a correct orientation in interacting with online content management system104). As shown inFIG. 4, initial documents402aand402bare both labeled with an orientation of 0 (indicating that text in the documents are positioned horizontal with upright characters).

As shown inFIG. 4, digital image character recognition system112can generate training documents406and corresponding ground truth orientations404by rotating initial documents402. In particular, digital image character recognition system112can rotate initial documents402by increments of 90 degrees and generate a training document with corresponding ground truth label at each rotation. For example, as shown inFIG. 4, digital image character recognition system112generates ground truth training documents406afrom initial document402a. In particular, digital image character recognition system112generated ground truth training documents406aby rotating initial document402ain increments of 90 degrees and generating ground truth orientations404at each rotation (e.g., initial document402aat a known orientation was at a position of “0” and digital image character recognition system112generates four training documents406awith ground truth orientations of “0”, “1”, “2”, and “3”). Similarly, digital image character recognition system112generates ground truth training documents406bby rotating and labeling initial document402b. AlthoughFIG. 4illustrates a particular number of training documents at particular orientations, ground truth training documents406can include any number of documents and any variety of orientations.

As mentioned above, digital image character recognition system112utilizes text prediction neural network230(e.g., word deep net) to generate computer searchable text from a digital image.FIG. 5provides additional detail regarding exemplary architecture of text prediction neural network500utilized to generate computer searchable text from a digital image depicting words (e.g., a word box). More specifically, as shown inFIG. 5, digital image character recognition system112provides word box502to text prediction neural network500comprising stack of convolutional layers504, stack of bidirectional long short term memory (“LSTM”) layers506, and connectionist temporal classification output layer508to generate computer searchable text510from word box502.

For example, as shown inFIG. 5, digital image character recognition system112provides word box502to text prediction neural network500(e.g., a word box generated as described inFIG. 2D). In one or more embodiments, digital image character recognition system112provides word box502in a fixed digital image size. However, in some embodiments, text prediction neural network500can accept word box502in different digital image sizes as input. Furthermore, in one or more embodiments, if word box502includes more than one word (e.g., “Made in America”), digital image character recognition system112can utilize word detector226to further separate the digital image into individual word boxes of single words (e.g., a word box for “Made”, “in”, and “America”). In some embodiments, digital image character recognition system112can resize the individual word boxes of single words into the appropriate input size for text prediction neural network500. As shown inFIG. 5, word box502is a digital image portraying the word “America” (e.g., a digital image of a word from a document portrayed in a digital image).

As just mentioned, text prediction neural network500, in some embodiments, can provide word box502as input into stack of convolutional layers504that analyze features of the word box502. For instance, in one or more embodiments, stack of convolutional layers504includes convolutional layers that produce feature maps from a digital image (e.g., word box502). Furthermore, stack of convolutional layers504, in some embodiments, can filter (e.g., weight) and analyze features at different levels of abstraction. The stack of convolutional layers504can generate a set of visual features for further analysis (e.g., by the bidirectional LSTM layers506).

Additionally, in some embodiments, stack of convolutional layers504also includes other layers such as pooling layers, fully connected layers, ReLu layers, normalization layers, and other layers. For instance, in one or more embodiments, stack of convolutional layers504can provide output from a layer to a pooling layer. In some embodiments, a pooling layer down samples a feature map or layer to lessen spatial dimensions and to control overfitting. Furthermore, in one or more embodiments, stack of convolutional layers504can include a ReLu layer. In particular, a ReLu layer, in some embodiments, can increase non-linear properties in the neural network/machine learning model. Furthermore, in alternate embodiments, stack of convolutional layers504can include other layers such as, but not limited to, normalization layers and loss layers.

Moreover, in some embodiments, stack of convolutional layers504also includes one or more fully connected layers. In particular, in some embodiments, the fully connected layer receives input from all layers and outputs probabilities for different classifications in word box502. For example, in some embodiments, a fully connected layer can output a vector of classes with probabilities. More specifically, in some embodiments, the vector of classes can include classes such as, but not limited to, characters in an ASCII table, letters of the alphabet, or numbers. Furthermore, the fully connected layer in stack of convolutional layers504can produce a predicted probability of the presence of the above mentioned classes in word box502. In one or more embodiments, stack of convolutional layers504determines a classification for text depicted within word box502.

Additionally, in one or more embodiments, text prediction neural network500can utilize stack of bidirectional LSTM layers506. For example, in some embodiments, text prediction neural network500can provide output from stack of convolutional layers504to stack of bidirectional LSTM layers506. In one or more embodiments, text prediction neural network500utilizes stack of bidirectional LSTM layers506to, but not limited to, recognize/correct for fragmented text characters, recognize/correct for grammar, and recognize/correct for sentence structure. In some embodiments, stack of bidirectional LSTM layers506includes an implementation of an RNN that is capable of learning long-term dependencies. For instance, in some embodiments, word detector226may generate word boxes of more than one word or may fragment words in separate word boxes (e.g., generating two word boxes for a digital image portraying “Hello, I live in America”, wherein the first word box includes “Hello, I liv” and the second word box includes “e in America”). As a result, in some embodiments, stack of bidirectional LSTM layers506can recognize/correct the word boxes (or the classification output from stack of convolutional layers504) including “Hello, I liv” and “e in America” to “Hello, I live in America”.

More specifically, in one or more embodiments, stack of bidirectional LSTM layers506includes a memory cell, an input gate, an output gate, and a forget gate. In some embodiments, the memory cell can store values (e.g., information such as a classification output from stack of convolutional layers504or word box502). In some embodiments, the input gate decides values that are added or updated in the memory cell. Moreover, in one or more embodiments, the forget gate controls whether or not to keep certain values within the memory cell (i.e., the forget gate can decide to keep or remove values/information from the memory cell). Furthermore, in some embodiments, the output gate decides what values/information bidirectional LSTM layers506will output. In some embodiments, each gate and memory cell are connected to each other and each gate and memory cell contain separate parameters. In one or more embodiments, stack of bidirectional LSTM layers506can include one or more gates to identify separate features such as verbs, nouns, context of sentences, and so forth.

Moreover, in some embodiments, text prediction neural network500can utilize connectionist temporal classification output layer508. In particular, in some embodiments, connectionist temporal classification output layer508includes a neural network that outputs a conditional probability distribution over label sequences from input of other neural networks such as an RNN or stack of bidirectional LSTM layers506. For instance, in one or more embodiments, connectionist temporal classification output layer508can predict the sequence of text portrayed in word boxes502. Moreover, in some embodiments, if word box502consisted of two word boxes representing “America” and “made in”, connectionist temporal classification output layer508can predict the sequence of text and output “made in America”.

Furthermore, in some embodiments, text prediction neural network500utilizes a modified connectionist temporal classification output layer508. In particular, in one or more embodiments, connectionist temporal classification output layer508provides a confidence score in addition to the generated computer searchable text510. For example, in some embodiments, text prediction neural network500can utilize the confidence score to utilize generated predictions, filter out predictions, or utilize a lexicon for better accuracy. More specifically, in some embodiments, if the confidence score is above a threshold, text prediction neural network500will utilize computer searchable text510. Moreover, in some embodiments, if the confidence score is below a threshold, text prediction neural network500filters out computer searchable text510and assumes that computer searchable text510was a noise prediction. Furthermore, in one or more embodiments, if the confidence score was not above or below a threshold amount, text prediction neural network500utilizes a lexicon generated from a dictionary such as, but not limited to, the Oxford English Dictionary to evaluate computer searchable text510. For example, text prediction neural network500, in some embodiments, transforms (e.g., combining or splitting) words in computer searchable text510to find a matching word in the lexicon.

Ultimately, as shown inFIG. 5, digital image character recognition system112can utilize text prediction neural network500to generate computer searchable text from word boxes (e.g., digital images portraying text). As illustrated inFIG. 5, text prediction neural network500provides word box502to stack of convolutional layers504, stack of bidirectional LSTM layers506, and connectionist temporal classification output layer508to generate computer searchable text510. In some embodiments, computer searchable text510is a string representing the predicted text from word box502(e.g., “America”). In one or more alternate embodiments, text prediction neural network500can provide more than one word boxes502to the stack of convolutional layers504, stack of bidirectional LSTM layers506, and connectionist temporal classification output layers508. Moreover, in some embodiments, text prediction neural network500can predict and sequence the more than one word boxes502to generate computer searchable text510represented as multiple strings, one string of data, or any other combination of text data.

Alternatively, in one or more embodiments, text prediction neural network500can generate computer searchable text510from word box502while utilizing any combination of stack of convolutional layers504, stack of bidirectional LSTM layers506, and/or connectionist temporal classification output layer508. For example, text prediction neural network500can generate computer searchable text510from word box502by utilizing only stack of convolutional layers504and stack of bidirectional LSTM layers506.

As just mentioned, digital image character recognition system112can also train a text prediction neural network to generate computer searchable text from digital images. For example, as shown inFIG. 6, digital image character recognition system112trains a text prediction neural network to generate a trained text prediction neural network in accordance with one or more embodiments. In particular, as illustrated inFIG. 6, digital image character recognition system112utilizes synthetic training data602, predicted text608, ground truth text labels612, and calculated loss616to train text prediction neural network606to generate trained text prediction neural network618.

As shown inFIG. 6, digital image character recognition system112accesses synthetic training data602. In one or more embodiments, digital image character recognition system112can access synthetic training data602from storage in server(s)102. Furthermore, synthetic training data602, in some embodiments, are digital images portraying text (e.g., word boxes) with an associated ground truth text label (e.g., ground truth text label612). Additionally, in one or more embodiments, synthetic training data602can include many (e.g., hundreds or millions) digital images with unique variations. Additional detail regarding generating synthetic training data602is provided below (e.g., in relation toFIG. 7).

AlthoughFIG. 6illustrates utilizing synthetic training data602, digital image character recognition system112can also utilize observed (e.g., non-synthetic) training data. For example, digital image character recognition system112can access training data (e.g., digital images) from a third-party repository or from online content management system104where the obtained digital images are annotated with the ground truth text label for the corresponding digital images.

Furthermore, as illustrated inFIG. 6, digital image character recognition system112utilizes synthetic training data602to train text prediction neural network606. In one or more embodiments, digital image character recognition system112provides synthetic training data602(or observed training data) as input to text prediction neural network606. Furthermore, in one or more embodiments, text prediction neural network606analyzes synthetic training data602to generate predicted text608(e.g., computer searchable text) in accordance with the methods described inFIG. 5.

Moreover, as shown inFIG. 6, text prediction neural network606can utilize synthetic training data602, predicted text608, and ground truth text labels612to learn to accurately recognize and generate computer searchable text from digital images that depict documents. For instance, digital image character recognition system112compares predicted text608and ground truth text labels612for corresponding synthetic training data602utilizing loss function610to generate calculated loss616. More specifically, loss function610can determine a measure of error between predicted text608generated by the text prediction neural network606and the ground truth text label612for synthetic training data602.

Furthermore, text prediction neural network606then utilizes calculated loss616to train (e.g., tune) text prediction neural network606in generating computer searchable text from digital images that include text. In particular, digital image character recognition system112provides information from calculated loss616(e.g., back-propagates) to text prediction neural network606to adjust parameters (e.g., weighting parameters for analyzing features within layers of the neural network) of text prediction neural network606.

In some embodiments, digital image character recognition system112repeatedly utilizes text prediction neural network606to generate predicted text608from text prediction neural network606, compare predicted text608with ground truth text labels612, and tune parameters to minimize calculated loss616. Digital image character recognition system112repeats this process until text prediction neural network606is sufficiently accurate (e.g., until convergence or through a threshold number of iterations).

Additionally, in one or more embodiments, as shown inFIG. 6, digital image character recognition system112can also train text prediction neural network606with negative synthetic training data604and ground truth negative text labels614. In particular, digital image character recognition system112, in some embodiments, can utilize both synthetic training data602and negative synthetic training data604. For example, in some embodiments, negative synthetic training data604includes images of textures that do not portray text. Similarly, ground truth negative text labels614are labeled to indicate that there is no text portrayed in negative synthetic training data604(e.g., NULL). Digital image character recognition system112, in some embodiments, utilizes negative synthetic training data604to train text prediction neural network606to recognize when there is no portrayed text in a word box.

For example, digital image character recognition system112can provide negative synthetic training data to text prediction neural network606. Text prediction neural network606can generate predicted text608(e.g., a text prediction or a null prediction). Digital image character recognition system112can compare ground truth negative text labels614with predicted text608(via loss function610) and train text prediction neural network606(e.g., by modifying parameters to reduce calculated loss616). In this manner, digital image character recognition system112can generate trained text prediction neural network618trained to identify text (or identify non-text) portrayed in word boxes.

As mentioned above, digital image character recognition system112can generate synthetic training data to train a text prediction neural network. For example,FIG. 7illustrates digital image character recognition system112generating synthetic training data for a text prediction neural network in accordance with one or more embodiments. As shown inFIG. 7, digital image character recognition system112utilizes corpus of words702, set of fonts704, and set of distortions706to generate synthetic training data712.

As just mentioned, in relation toFIG. 7, digital image character recognition system112utilizes corpus of words702. As shown, corpus of words702includes words from the English language. In particular, corpus of words702includes words from the English language and other text characters incidental to the English language (e.g., numbers, punctuation, and symbols). In one or more alternate embodiments, corpus of words702can also include abbreviations, acronyms, or other word and/or character modifications. For instance, in one or more embodiments, digital image character recognition system112utilizes the Uniform Product Code (UPC) in order to generate synthetic training data712that resembles receipts to train the text prediction neural network to recognize receipts and receipt contents. Furthermore, corpus of words702, in some embodiments, can include numerous word entries from multiple sources (e.g., literature, dictionaries, and online databases).

As shown inFIG. 7, digital image character recognition system112also utilizes set of fonts704. As illustrated, set of fonts704includes a variety of fonts (e.g., computer fonts, machine printed fonts, or handwritten fonts). For example, in some embodiments, set of fonts704includes more common fonts such as, but not limited to, Times New Roman and Helvetica and also includes less common or unique fonts such as ornate logo fonts and different sets of handwritten calligraphy font styles. Also, in one or more embodiments and as shown in FIG.7, set of fonts704includes fonts that are commonly utilized by receipts or thermal printers (e.g., the font “pixel” in set of fonts704).

Furthermore, as illustrated inFIG. 7, digital image character recognition system112also utilizes set of distortions706. Set of distortions706includes both geometric and photometric transformations applicable to a digital image. For example, as shown inFIG. 7, set of distortions706includes transformations such as blur, warp, shadow, skew, and creases. In one or more embodiments, set of distortions706may include numerous types of distortions from multiple sources. Moreover, in some embodiments, set of distortions706include distortions that simulate real world distortions on digital images. For example, the digital image character recognition system112can apply a set of distortions to a digital image to cause a shadow on the digital image (and the words depicted in the digital image). Furthermore, the digital image character recognition system112can apply a set of distortions to a digital image to cause the glare on the digital image. In some embodiments, the digital image character recognition system112can cause numerous distortions to simulate real world distortions to digital images depicting a document and/or words such as, but not limited to, water damaged paper, rust, ink smear, blurry digital images, wrinkles, and/or fading ink.

As mentioned above, digital image character recognition system112utilizes corpus of words702, set of fonts704, and set of distortions706to generate word-font-distortion sample sets708. For example, as shown inFIG. 7, digital image character recognition system112selects words from corpus of words702, fonts from set of fonts704, and distortions from set of distortions706to generate word-font-distortion sample sets708. To illustrate, as shown inFIG. 7, in some embodiments, digital image character recognition system112can select the word “Tree”, the font “Cooper Black”, and the distortion “shadow” to generate word-font-distortion sample set710a.

Furthermore, in one or more embodiments, digital image character recognition system112can generate word-font-distortion sample sets708with any combination of one or more words, one or more fonts, and/or one or more distortions from corpus of words702, set of fonts704, and set of distortions706. For instance, in some embodiments, digital image character recognition system112can select the word “Ohio” and the font “Pixel” without selecting a distortion to generate word-font-distortion sample set710b. Similarly, digital image character recognition system112can select the word “Rugby”, the font “Berlin Sans”, and the distortions “warp” and “shadow” to generate word-font-distortion sample set710c. In one or more embodiments, digital image character recognition system112can generate word-font-distortion sample sets708without selecting words from corpus of words702in order to generate negative synthetic training data (e.g., a digital image without a word).

Additionally, in some embodiments, digital image character recognition system112can utilize a frequency system to generate word-font-distortion sample sets708. In particular, digital image character recognition system112, in one or more embodiments, utilizes the frequency system to determine the frequency of words, fonts, and distortions from corpus of words702, set of fonts704, and set of distortions706occurring in real world settings. For example, in some embodiments, the frequency of words, fonts, and distortions includes, but is not limited to, a percentage of times a word, font, or distortion occurs in a real world setting. In particular, digital image character recognition system112can determine that certain fonts, such as Helvetica or Times New Roman occur more frequently in real world settings in comparison to other fonts (e.g., a higher percentage of known occurrences where Helvetica or Times New Roman is utilized). Similarly, digital image character recognition system112can determine words and distortions that occur more frequently in real world settings in comparison to other words and distortions. Moreover, digital image character recognition system112can utilize the determined frequencies of words, fonts, and distortions to generate word-font-distortion sample sets708that more often include words, fonts, or distortions that are above a determined threshold frequency (e.g., weight sampling of words, fonts, or distortions based on frequency of utilization).

Moreover, digital image character recognition system112utilizes generated word-font-distortion sample sets708to generate synthetic training data712. For example, in some embodiments, digital image character recognition system112utilizes word-font-distortion sample sets710a,710b, and710cto generate synthetic training digital images714a,714b, and714c. For example, a word-font-distortion sample set can include any combination of one or more words, one or more fonts, and/or one or more distortions. In particular, in some embodiments, digital image character recognition system112generates synthetic training data by applying a selected font and a distortion to a selected word. For instance, as shown inFIG. 7, digital image character recognition system112applies the selected font “Cooper Black” and the distortion “shadow” (in word-font-distortion sample set710a) to the selected word “Tree” (from word-font-distortion sample set710a) to generate synthetic training digital image714a. Similarly, digital image character recognition system112applies the selected font “Pixel” in word-font-distortion sample set710bon the selected word “Ohio” to generate synthetic training digital image714b.

Additionally, in one or more embodiments, digital image character recognition system112can apply distortions from set of distortions706directly on synthetic training data712(e.g., directly on a digital image to transform the digital image as a whole). For example, digital image character recognition system112applies selected font “Berlin Sans” and a selected distortion “warp” in word-font-distortion sample set710con the selected word “Rugby” to generate the modified word in synthetic training digital image714c. Furthermore, digital image character recognition system112can also apply the selected distortion “shadow” in word-font-distortion sample set710cdirectly onto synthetic training digital image714cto give synthetic training digital image714cthe shadow effect (as shown inFIG. 7). In one or more embodiments, digital image character recognition system112can apply any combination of words, fonts, and/or distortions to generate synthetic training data (e.g., hundreds or millions of training word boxes).

Furthermore, in alternate embodiments, digital image character recognition system112can utilize an alternative corpus of words in a different language to generate synthetic training data for other languages. For example, althoughFIG. 7illustrates generating synthetic training data in English, digital image character recognition system112can quickly and efficiently generate training data for other languages (and thus generate trained text prediction neural networks for alternative languages). Moreover, by swapping corpus of words702for a second corpus of words in a second language (e.g., German or French), digital image character recognition system112can generate synthetic training data for the second language. In particular, digital image character recognition system112can generate word-font-distortion sample sets from the second corpus of words702, set of fonts704, and set of distortions706. Digital image character recognition system112can then apply the word-font-distortion sample sets to generate synthetic training data in the second language.

Additionally, as mentioned above, in some embodiments, digital image character recognition system112can generate word-font-distortion sample sets708without selecting words from corpus of words702in order to generate negative synthetic training data. For example, in some embodiments, digital image character recognition system112can generate negative synthetic training data by generating word-font-distortion sample sets708that include distortions from set of distortions706without selecting words or fonts. Moreover, in some embodiments, digital image character recognition system112generates negative synthetic training data by applying selected distortions from generated word-font-distortion sample sets708directly on a digital image. For example, in one or more embodiments, digital image character recognition system112can generate negative synthetic training data by applying a distortion such as textures (e.g., wood, marble countertops, carpet designs, and so forth) to a digital image (e.g., a digital image without text).

As mentioned above, digital image character recognition system112can utilize generated computer searchable text from digital images in a variety of ways to improve computing systems for managing, organizing, and searching digital images. For example, as shown inFIG. 8A, digital image character recognition system112can utilize generated computer searchable text from digital images to search text portrayed within the digital image. Furthermore, as shown inFIG. 8B, digital image character recognition system112can utilize generated computer searchable text from digital images that display documents to provide copy and paste functionality for text portrayed within the digital image.

In particular,FIG. 8A, illustrates client device106with display802showing user interface810associated with the online content management system application108. As shown, user interface810includes search bar804and search results808determined based on computer searchable text generated from documents portrayed in digital images. Specifically, in response to search query806entered in search bar804, digital image character recognition system112searches computer generated searchable text identified from documents portrayed in digital images and identifies those digital images with searchable text corresponding to search query806.

As mentioned above, digital image character recognition system112can associate generated computer searchable text for a digital image to the digital image. For instance, in some embodiments, digital image character recognition system112can index digital images within online content management system104by generating a token comprising the computer searchable text and associating the tokens to digital images that contain the search text represented in the token.

More specifically, digital image character recognition system112, in some embodiments, can create tokens for all of the words in the computer searchable text of a digital image and store the tokens in an index. Furthermore, in one or more embodiments, digital image character recognition system112can link the digital image (and any other digital image containing those tokens) to the stored token in the index. For instance, in some embodiments, digital image character recognition system112can utilize a bag of words model to create a list of computer searchable text for a digital image. Furthermore, in some embodiments, digital image character recognition system112can associate tokens with keywords (e.g., other words that are determined to be associated with the token word).

Moreover, in one or more embodiments, digital image character recognition system112can associate the generated computer searchable text to a digital image by including the computer searchable text of the digital image in the meta-data of the digital image. Digital image character recognition system112can then search the meta-data of digital images.

As shown inFIG. 8A, in one or more embodiments, digital image character recognition system112utilizes search query806provided in search bar804to search for digital images that have computer searchable text and/or are tokenized as described above. In one or more embodiments, digital image character recognition system112displays digital images, that are associated with search query806in search bar804, as a collection of files in search results808. For example, as shown inFIG. 8A, a user enters search query806for “Store” in search bar804. Moreover, digital image character recognition system112identifies digital images that are associated with the query “store” based on the generated computer searchable text and displays the identified digital images in search result808. For example, as shown inFIG. 8A, digital image character recognition system112lists digital images that have generated computer searchable text for “store” in search results808. In one or more embodiments, digital image character recognition system112can utilize any combination of search queries and any combination of search results. Furthermore, in some embodiments, digital image character recognition system112can refine search results based on other search criteria (e.g., filters).

As mentioned, digital image character recognition system112can also generate a digital overlay for a digital image, allowing users to copy and paste text from the digital overlay. For example,FIG. 8Billustrates client device106with user interface824(corresponding to online content management system application108) displaying digital image812. Digital image character recognition system112generates computer searchable text (e.g., “STORE”) based on a document portrayed in digital image812. Digital image character recognition system112then maps/overlays the computer searchable text onto digital image812. Furthermore, digital image character recognition system112can create digital image formats (e.g., overlays) that are capable of mapping computer searchable text onto a digital image. Additionally, digital image character recognition system112can utilize the computer searchable text in a digital image to provide text copy and paste functionality.

For example, as shown inFIG. 8B, after digital image character recognition system112maps/overlays computer searchable text onto digital image812. Based on user interaction with the user interface824, a user can highlight text814(“STORE”) in digital image812. Furthermore, online content management system application108can provide copy option816upon detecting a highlighting action. In one or more embodiments, digital image character recognition system112stores computer searchable text associated with text814(“store”) in copy/paste memory818when text814is highlighted and a user selects copy option816.

Additionally, as shown inFIG. 8B, digital image character recognition system112can provide copy/paste memory818containing computer searchable text associated with text814(“STORE”) from digital image812to other applications (e.g., messaging apps, text editors, document suites, and so forth). For example, as shown inFIG. 8B, a user can utilize text editor820and select paste option822to paste generated computer searchable text associated with text814(“STORE”) from digital image812contained in copy/paste memory818to text editor820. In one or more alternate embodiments, digital image character recognition system112can highlight multiple text elements in a digital image and can also copy and paste multiple words from a digital image to other applications.

Digital image character recognition system112can map/overlay computer searchable text to corresponding regions of a digital image. For example, inFIG. 8B, digital image character recognition system112generates an overlay and provides searchable text within digital image812in a region where “STORE” is reflected in the digital image. Digital image character recognition system112can generate the overlay in a particular region based on the location of word boxes utilized to generate computer searchable text. For instance, digital image character recognition system112can determine that word box for “STORE” originated from a first region of the digital image812. Digital image character recognition system112can generate computer searchable text and then generate an overlay corresponding to the first region.

As mentioned above, digital image character recognition system112can categorize digital images based on the depicted documents in the digital image and also provide the digital images to a user associated with a category. For example, as shown inFIG. 9, digital image character recognition system112utilizes computer searchable text associated with digital images904a-din repository of digital images902and user data table908to categorize digital images based on displayed documents in the digital image and also to distribute digital images to users910a, bassociated with certain categories.

More specifically, as shown inFIG. 9, digital image character recognition system112accesses repository of digital images902comprising digital images904a-d. As shown, digital image character recognition system112generates computer searchable text based on documents portrayed in digital images904a-904d. For instance, digital image character recognition system112associates digital image904awith associated computer searchable text “receipt”, digital image904bwith associated computer searchable text “store”, digital image904cwith associated computer searchable text “hello”, and digital image904dwith associated computer searchable text “memo”. In other embodiments, digital images904a-dcan include multiple words as computer searchable text. Digital image character recognition system112can categorize digital images904a-dbased on computer searchable text or distribute digital images904a-dbased on computer searchable text.

For instance, as shown inFIG. 9, digital image character recognition system112provides digital images to a user associated with a document category from repository of digital images902. More specifically, digital image character recognition system112can generate and utilize user data table908, which includes users and document categories associated with users to provide a digital image to a user associated with a document category. For instance, as shown inFIG. 9, digital image character recognition system112can determine that user Bob is associated with a document category of “Receipts” from user data table908. Thus, as illustrated inFIG. 9, digital image character recognition system112provides digital image904a, which contains computer searchable text for “Receipt”, to digital image repository associated to user Bob910a. Additionally, digital image character recognition system112can determine that user Susan is associated with a document category of “Memos” from user data table908. Thus, digital image character recognition system112provides digital image904d, which contains computer searchable text for “Memo”, to digital image repository associated to user Susan910b.

Accordingly, digital image character recognition system112can categorize digital images904a-dinto document categories on online content management system104based on computer searchable text associated with digital images904a-d. Furthermore, digital image character recognition system112can categorize digital images904a-dinto other objects with or without user associations (e.g., category folders associated with document categories on an online content management system). Additionally, digital image character recognition system112can provide a digital image to a user associated with a document category via email, text, and other forms of electronic communication.

In addition, digital image character recognition system112can generate user data table908based on a variety of factors. For example, in one or more embodiments, digital image character recognition system112generates user data table908(e.g., correspondence between a user and document category) based on user input (e.g., user request to receive a particular category). In other embodiments, digital image character recognition system112generates a correspondence between a user and document category based on position (e.g., company office manager receives receipts), based on access history (e.g., individual that repeatedly accesses memos receives memos), or based on demographic information.

Turning now toFIG. 10, additional detail will be provided regarding various components and capabilities of digital image character recognition system112. In particular,FIG. 10illustrates digital image character recognition system112implemented by server(s)102. Additionally, digital image character recognition system112is also part of online content management system104. As shown, digital image character recognition system112can include, but is not limited to, document detection engine1002, document enhancement engine1004, orientation detection engine1006, optical character recognition engine1008, synthetic training data generator1010, training document generator1012, and digital content manager1014.

As just mentioned, and as illustrated inFIG. 10, digital image character recognition system112includes document detection engine1002. More specifically, document detection engine1002detects, identifies, determines, and/or accesses digital content (e.g., digital images) that depict documents. For example, document detection engine1002utilizes document detection neural network206neural network to detect digital images that depict documents as discussed with reference toFIG. 2A. In one or more embodiments, document detection engine1002also labels the detected digital images or provides the digital images to a repository of digital images that depict documents.

As shown inFIG. 10, digital image character recognition system112also includes document enhancement engine1004. In particular, document enhancement engine1004modifies, transforms, and/or enhances digital images that depict documents. For example, document enhancement engine1004utilizes computer vision techniques to determine the boundaries and corners of a displayed document and then modifies the digital image based on the displayed document. For example, document enhancement engine1004can identify a depicted document and crop, rectify, and enhance the depicted document in the digital image as discussed with reference toFIG. 2B.

Moreover, as shown inFIG. 10, digital image character recognition system112includes orientation detection engine1006. More specifically, orientation detection engine1006detects, identifies, and/or identifies the orientation of a displayed document in a digital image. For example, orientation detection engine1006utilizes orientation neural network222to classify the orientation of a displayed document within a digital image as discussed with reference toFIG. 2C. In one or more embodiments, orientation detection engine1006also trains orientation neural network222with training documents generated by training document generator1012.

Additionally, as illustrated inFIG. 10, digital image character recognition system112includes optical character recognition engine1008. In particular, optical character recognition engine1008generates, creates, determines, and/or identifies computer searchable text for a digital image that depicts a document. For example, optical character recognition engine1008utilizes word detector226and text prediction neural network230as discussed with reference toFIG. 2D. More specifically, optical character recognition engine1008utilizes word detector226to detect and crop word boxes from a displayed document in a digital image to generate word boxes. Furthermore, optical character recognition engine1008utilizes text prediction neural network230to predict the text in word boxes and to generate computer searchable text from the text displayed in the word boxes. Furthermore, optical character recognition engine1008, in some embodiments, associates the generated computer searchable text with the digital image that depicts the document. Additionally, in some embodiments, optical character recognition engine1008also trains text prediction neural network with synthetic training data generated by synthetic training data generator1010.

Moreover, as shown inFIG. 10, digital image character recognition system112includes synthetic training data generator1010. More specifically, synthetic training data generator1010generates, creates, and/or forms synthetic training data such as digital images of words. For example, synthetic training data generator1010utilizes corpus of words702, set of fonts704, and set of distortions706to generate synthetic training digital images712as discussed with reference toFIG. 7. In one or more embodiments, synthetic training data generator1010also generates negative synthetic training data (e.g., digital images with textures and without words) as discussed with reference toFIGS. 6 and 7.

Furthermore, as shown inFIG. 10, digital image character recognition system112includes training document generator1012. In particular, training document generator1012generates training documents from a document by generating multiple rotations of the document. For example, training document generator1012utilizes initial documents at a known orientation to generate one or more ground truth orientations as discussed with reference toFIG. 4.

Further, as shown inFIG. 10, digital image character recognition system112includes digital content manager1014. More specifically, digital content manager1014manages digital content such as digital images in online content management system104. For example, digital content manager1014associates generated computer searchable text to digital images that depict documents and provides implementations to make digital images searchable based on computer searchable text associated with the digital images as discussed with reference toFIG. 8A. Also, digital content manager1014also provides additional functionalities to digital images associated with computer searchable text such as copy and paste functionalities as discussed with reference toFIG. 8B. Furthermore, digital content manager1014also categorizes digital images associated with computer searchable text based on the computer searchable text and provides users the categorized digital images as discussed with reference toFIG. 9.

Additionally, as shown inFIG. 10, server(s)102include data storage1022. In particular, data storage1022includes digital content item data1024and training data1026. Digital content item data1024includes all data stored, managed, edited, shared with, or created by users of online content management system104. More specifically, digital content item data1024can include items such as, but not limited to, user documents, digital image files, and user folders. Moreover, training data1026can store models utilized by digital image character recognition system112. For example, training data1026can store machine learning models and other models utilized by document detection engine1002, document enhancement engine1004, orientation detection engine1006, optical character recognition engine1008, synthetic training data generator1010, training document generator1012, and digital content manager1014. Furthermore, training data1026can also store synthetic training data generated by synthetic training data generator1010and training documents generated by training document generator1012.

Furthermore,FIG. 10also illustrates online content management system application108implemented by client device106. As shown inFIG. 10, online content management system application108can include, but is not limited to, digital content manager1028. In particular, digital content manager1028can manage and access digital content such as digital images on server(s)102and client device106. For example, digital content manager1028can search for and organize digital images that are searchable based on computer searchable text as discussed with reference toFIG. 8A. Furthermore, digital content manager1028can also utilize functionalities of digital images associated with computer searchable text such as copy and paste functionalities as discussed with reference toFIG. 8B. Moreover, digital content manager1028can also access or receive digital images associated with computer searchable text based on categories as discussed with reference toFIG. 9.

Additionally, as shown inFIG. 10, client device106includes data storage1030. In particular, data storage1030includes digital content item data1032. Digital content item data1032includes all data stored, managed, edited, shared with, or created by users of online content management system application108. More specifically, digital content item data1032can include items such as, but not limited to, user documents, digital image files, and user folders. In one or more embodiments, digital content item data1032can be provided to online content management system104.

The components1002-1032and their corresponding elements can comprise software, hardware, or both. For example, the components1002-1032and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. The components1002-1032and their corresponding elements can comprise hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally, or alternatively, the components1002-1032and their corresponding elements can comprise a combination of computer-executable instructions and hardware.

Furthermore, the components1002-1032of digital image character recognition system112may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components1002-1032of digital image character recognition system112may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components1002-1032of digital image character recognition system112may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of digital image character recognition system112may be implemented in a suit of mobile device applications or “apps.”

Turning now toFIG. 11, this figure illustrates a series of acts1100to generate computer searchable text from a digital image that depicts a document. WhileFIG. 11illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIG. 11. The acts ofFIG. 11can be performed as part of a method. In one or more embodiments, a non-transitory computer readable storage medium can comprise instructions that, when executed by one or more processors, cause a computing device to perform acts ofFIG. 11. In still further embodiments, a system can perform the acts ofFIG. 11.

The series of acts1100includes an act1110of identifying a digital image comprising a depiction of a document. One or more embodiments involve identifying the digital image comprising the depiction of the document utilizing a neural network. Additionally, one or more embodiments involve identifying a digital image comprising a depiction of a document from a repository of digital images. For example, the digital image character recognition system can analyze each digital image in a repository of digital images utilizing a document detection neural network trained to identify digital images portraying documents comprising text.

The series of acts1100also includes an act1120of utilizing an orientation neural network to determine an orientation for the digital image. For example, act1120involves utilizing an orientation neural network to detect an orientation of the document within the digital image. One or more embodiments involve training the orientation neural network. For example, one or more embodiments involve analyzing a training document utilizing the orientation neural network to predict an orientation of the training document and comparing the predicted orientation of the training document with a ground truth orientation of the training document. Additionally, one or more embodiments involve generating the training document. For example, one or more embodiments involve identifying an initial document at a known orientation and rotating the initial document to generate the training document and the ground truth orientation of the training document.

The series of acts1100further includes act1130of cropping a word box from the digital image. For example, act1130involves cropping a word box, that comprises a portion of the depiction of the document, from the digital image.

The series of acts1100also includes act1140of generating computer searchable text for the digital image utilizing a text prediction neural network. For example, act1140involves utilizing a text prediction neural network trained with synthetic training data to generate computer searchable text for the portion of the depiction of the document based on the word box and the detected orientation of the document. Additionally, one or more embodiments involve training the text prediction neural network with synthetic data that comprises a synthetic training digital image comprising a ground truth text label corresponding to the synthetic training digital image. For example, one or more embodiments involve training the text prediction neural network with the synthetic data by receiving the synthetic training data (e.g., the synthetic training digital image), utilizing the text prediction neural network on the synthetic training digital image to predict text depicted on the synthetic training digital image, and comparing the predicted text depicted on the synthetic training digital image with the ground truth text label corresponding to the synthetic training digital image.

Furthermore, in one or more embodiments, act1140also involves generating a synthetic training digital image. For example, one or more embodiments involve generating the synthetic training digital image by identifying a corpus of words and a set of fonts (and/or a set of distortions). Additionally, one or more embodiments involve generating the synthetic training digital image by selecting a word from the corpus of words and a font from the set of fonts (and/or a distortion from the set of distortions) and applying the font (and/or the distortion) to the word to generate a modified word. One or more embodiments involve generating the synthetic training digital image such that the synthetic training digital image portrays the modified word. Additionally, in one or more embodiments, the set of distortions comprise at least two of rotations, underlines, blurs, noises, shadows, creases, textures, or applying lens filters. One or more embodiments involve generating a synthetic training digital image in an additional language. For example, in one or more embodiments the corpus of words and the synthetic training digital image corresponds to a first language. Additionally, one or more embodiments involve generating an additional synthetic training digital image corresponding to an additional language by utilizing an additional corpus of words corresponding to the additional language.

In one or more embodiments, the digital image character recognition system further indexes the digital image by associating a token with the digital image. Additionally, in one or more embodiments, the token comprises the computer searchable text. In one or more embodiments, the digital image character recognition system further utilizes the computer searchable text to identify a document category corresponding to the digital image comprising the depiction of the document and provides the digital image to a user associated with the document category.

FIG. 12illustrates a block diagram of exemplary computing device1200that may be configured to perform one or more of the processes described above. One will appreciate that server(s)102and/or client device106may comprise one or more computing devices such as computing device1200. As shown byFIG. 12, computing device1200can comprise processor1202, memory1204, storage device1206, I/O interface1208, and communication interface1210, which may be communicatively coupled by way of communication infrastructure1212. While an exemplary computing device1200is shown inFIG. 12, the components illustrated inFIG. 12are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, computing device1200can include fewer components than those shown inFIG. 12. Components of computing device1200shown inFIG. 12will now be described in additional detail.

In particular embodiments, processor1202includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor1202may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1204, or storage device1206and decode and execute them. In particular embodiments, processor1202may include one or more internal caches for data, instructions, or addresses. As an example and not by way of limitation, processor1202may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory1204or storage device1206.

Memory1204may be used for storing data, metadata, and programs for execution by the processor(s). Memory1204may include one or more of volatile and non-volatile memories, such as Random Access Memory (“RAM”), Read Only Memory (“ROM”), a solid state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. Memory1204may be internal or distributed memory.

Storage device1206includes storage for storing data or instructions. As an example and not by way of limitation, storage device1206can comprise a non-transitory storage medium described above. Storage device1206may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage device1206may include removable or non-removable (or fixed) media, where appropriate. Storage device1206may be internal or external to computing device1200. In particular embodiments, storage device1206is non-volatile, solid-state memory. In other embodiments, Storage device1206includes read-only memory (ROM). Where appropriate, this ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these.

Communication interface1210can include hardware, software, or both. In any event, communication interface1210can provide one or more interfaces for communication (such as, for example, packet-based communication) between computing device1200and one or more other computing devices or networks. As an example and not by way of limitation, communication interface1210may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI.

FIG. 13is a schematic diagram illustrating environment1300within which one or more embodiments of online content management system104can be implemented. Online content management system1302may generate, store, manage, receive, and send digital content (such as digital videos). For example, online content management system1302may send and receive digital content to and from client devices1306by way of network1304. In particular, online content management system1302can store and manage a collection of digital content. Online content management system1302can manage the sharing of digital content between computing devices associated with a plurality of users. For instance, online content management system1302can facilitate a user sharing a digital content with another user of online content management system1302.

In particular, online content management system1302can manage synchronizing digital content across multiple client devices1306associated with one or more users. For example, a user may edit digital content using client device1306. The online content management system1302can cause client device1306to send the edited digital content to online content management system1302. Online content management system1302then synchronizes the edited digital content on one or more additional computing devices.

In addition to synchronizing digital content across multiple devices, one or more embodiments of online content management system1302can provide an efficient storage option for users that have large collections of digital content. For example, online content management system1302can store a collection of digital content on online content management system1302, while the client device1306only stores reduced-sized versions of the digital content. A user can navigate and browse the reduced-sized versions (e.g., a thumbnail of a digital image) of the digital content on client device1306. In particular, one way in which a user can experience digital content is to browse the reduced-sized versions of the digital content on client device1306.

Another way in which a user can experience digital content is to select a reduced-size version of digital content to request the full- or high-resolution version of digital content from online content management system1302. In particular, upon a user selecting a reduced-sized version of digital content, client device1306sends a request to online content management system1302requesting the digital content associated with the reduced-sized version of the digital content. Online content management system1302can respond to the request by sending the digital content to client device1306. Client device1306, upon receiving the digital content, can then present the digital content to the user. In this way, a user can have access to large collections of digital content while minimizing the amount of resources used on client device1306.

Client device1306may be a desktop computer, a laptop computer, a tablet computer, a personal digital assistant (PDA), an in- or out-of-car navigation system, a handheld device, a smart phone or other cellular or mobile phone, or a mobile gaming device, other mobile device, or other suitable computing devices. Client device1306may execute one or more client applications, such as a web browser (e.g., Microsoft Windows Internet Explorer, Mozilla Firefox, Apple Safari, Google Chrome, Opera, etc.) or a native or special-purpose client application (e.g., Dropbox Paper for iPhone or iPad, Dropbox Paper for Android, etc.), to access and view content over network1304.

Network1304may represent a network or collection of networks (such as the Internet, a corporate intranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks) over which client devices1306may access online content management system1302.