Patent Publication Number: US-11036976-B2

Title: Methods and systems of handwriting recognition in virtualized-mail services

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to U.S. provisional patent application No. 62/672,753, titled METHODS AND SYSTEMS OF VIRTUALIZED-MAIL SERVICES and filed on 17 May 2018. This application is hereby incorporated by reference in its entirety. 
    
    
     BACKGROUND 
     1. Field 
     This application relates generally to optical character recognition, and more particularly to a system, method and article of manufacture of hand-writing recognition in virtualized-mail services. 
     2. Related Art 
     Users and companies can receive enormous amounts of mail. Various mail management services have been created to help manage user incoming mail. Incoming mail can include both typed and/or handwritten information on the envelope/box. Handwritten address information can be reviewed by humans in order to be correctly forwarded to the recipient. This can increase the cost of mail management and decrease the competitiveness of the mail management service. Accordingly, improvements to automatically handling review and analysis of handwritten mail information is desired. 
     BRIEF SUMMARY OF THE INVENTION 
     In one aspect, a computerized method useful for handwriting recognition (HWR) on physical mail envelopes addressed to a user includes the step of scanning a physical mail item to obtain a digital image of the address-side of the physical mail item. The method includes the step of identifying that at least one of a return address region or a recipient address region of the address-side of the physical mail item. The method includes the step of determining that the at least one of the return address region or a recipient address region comprises a handwritten text. The method includes the step of providing a data store of known senders to the recipient address. The data store of known senders comprises a data store of return address information in a known sender handwriting samples and a data store of receiver address information in the known sender handwriting samples. The method includes the step of providing a data store of a receiver&#39;s identity and address. The method includes the step of creating a first training set including the historical data store of known senders. The method includes the step of training the neural network in a first stage using the first training set. The method includes the step of creating a second training set for a second stage of training including the data store of a receiver&#39;s identity and address. The method includes the step of training the neural network in a second stage using the second training set. The method includes the step of integrating the neural network into an HWR functionality. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an example system for implementing virtualized mail services, according to some embodiments. 
         FIG. 2  depicts an exemplary computing system that can be configured to perform any one of the processes provided herein. 
         FIG. 3  is a block diagram of a sample computing environment that can be utilized to implement various embodiments. 
         FIG. 4  illustrates an example process for a mail entry recognition system for typed or handwritten envelopes, according to some embodiments. 
         FIG. 5  illustrates an example process for grouping and/or batching a virtualized-mail entry with historical memory, according to some embodiments. 
         FIG. 6  illustrates an example process for file merging on scan requests, according to some embodiments. 
         FIG. 7  illustrates an example process for scanning double-sided documents, according to some embodiments. 
         FIG. 8  illustrates an example process for HWR of physical mail envelopes, according to some embodiments. 
         FIG. 9  illustrates an example process for handwriting recognition (HWR) on physical mail envelopes addressed to a user, according to some embodiments. 
     
    
    
     The Figures described above are a representative set, and are not an exhaustive with respect to embodying the invention. 
     DESCRIPTION 
     Disclosed are a system, method, and article of a virtualized-mail services. The following description is presented to enable a person of ordinary skill in the art to make and use the various embodiments. Descriptions of specific devices, techniques, and applications are provided only as examples. Various modifications to the examples described herein can be readily apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other examples and applications without departing from the spirit and scope of the various embodiments. 
     Reference throughout this specification to “one embodiment,” “an embodiment,” ‘one example,’ or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment. 
     Furthermore, the described features, structures, or characteristics of the invention may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art can recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention. 
     The schematic flow chart diagrams included herein are generally set forth as logical flow chart diagrams. As such, the depicted order and labeled steps are indicative of one embodiment of the presented method. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more steps, or portions thereof, of the illustrated method. Additionally, the format and symbols employed are provided to explain the logical steps of the method and are understood not to limit the scope of the method. Although various arrow types and line types may be employed in the flow chart diagrams, and they are understood not to limit the scope of the corresponding method. Indeed, some arrows or other connectors may be used to indicate only the logical flow of the method. For instance, an arrow may indicate a waiting or monitoring period of unspecified duration between enumerated steps of the depicted method. Additionally, the order in which a particular method occurs may or may not strictly adhere to the order of the corresponding steps shown. 
     Definitions 
     Example definitions for some embodiments are now provided. 
     Application programming interface (API) can specify how software components of various systems interact with each other. 
     Cloud computing can involve deploying groups of remote servers and/or software networks that allow centralized data storage and online access to computer services or resources. These groups of remote serves and/or software networks can be a collection of remote computing services. 
     Handwriting recognition (HWR) can include functionalities that enable a computer to receive and interpret intelligible handwritten input from sources such as paper documents, photographs, touchscreens and other devices. The image of the written text can be sensed from a piece of paper by optical scanning (OCR) or intelligent word recognition. HWR can handle formatting, segmentation into characters and location the most plausible words. Both off-line and on-line HWR techniques can be utilized. 
     Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data. Example machine learning techniques that can be used herein include, inter alia: decision tree learning, association rule learning, artificial neural networks, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, and/or sparse dictionary learning. 
     Random forests (RF) (e.g. random decision forests) are an ensemble learning method for classification, regression and other tasks, that operate by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (e.g. classification) or mean prediction (e.g. regression) of the individual trees. RFs can correct for decision trees&#39; habit of overfitting to their training set. 
     Optical character recognition (OCR) can include the electronic conversion of images of typed, handwritten or printed text into machine-encoded text from a scanned document or digital photograph of a document. 
     Exemplary Systems 
       FIG. 1  illustrates an example system for implementing virtualized mail services, according to some embodiments. System  100  can provide virtualized mail services to a plurality of users. System  100  can implement the various processes provided infra, including, processes  400 - 800 . 
     Networks  104  can include the Internet, text messaging networks (e.g. short messaging service (SMS) networks, multimedia messaging service (MMS) networks, proprietary messaging networks, instant messaging service networks, email systems, etc. Networks  104  can be used to communicate messages and/or other information (e.g. videos, forms, text files, livestreams, push notifications, etc.) from the various entities of system  100 . 
     System  100  can include a virtualized mail server(s)  106 . User-computing devices  102  can be any computing device used by a user to access/consume/manage virtualized mail content provided by system  100  (e.g. via virtualized mail server(s)  106 ). Example user-computing devices can include, inter alia: personal computers, mobile devices, augmented reality devices, virtual reality devices, tablet computers, etc. User-computing devices  102  can access virtualized mail content via a website, local mobile device application and the like. User-computing devices  102  can include an application for managing virtualized mail settings. User-computing devices  102  can enable a user to upload registration and other virtualized mail related content to virtualized mail server(s)  106 . User-computing devices  102  can enable a download virtualized mail applications from virtualized mail server(s)  106 . 
     Virtual mail server(s)  106  can manage and provide virtual mail services to user-computing devices  102 . Virtual mail server(s)  106  can include functionalities for implementing, inter alia: user notifications, OCR/HWR operations, manage printers and label printing, robot managers for opening and scanning physical mail and/or turning pages of physical mail, etc. Virtual mail server(s)  106  can include web servers, email servers, IM servers, text messaging systems, computer vision systems, machine learning systems, database management systems, etc. 
     Virtual mail server(s)  106  can utilize machine learning techniques (e.g. artificial neural networks, etc.). Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data. Example machine learning techniques that can be used herein include, inter alia: decision tree learning, association rule learning, artificial neural networks, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, and/or sparse dictionary learning. Machine learning techniques can be used to automatically improve OCR techniques, junk mail identification, and the like. 
     Virtual mail server(s)  106  can implement various specific virtualized mail services such the following. It is noted that an expedited scan request can be implemented when the user requests expedite scanning. An administrator (e.g. a human or AI-based administrator) can be notified via phone notifier or email of the expedited request. An application can then be set to scan the order and the scan job can be processed. Virtual mail server(s)  106  can enable multiple forward addresses and support unlimited forwarding addresses. For example, a user can specify each mailing address to be forwarded to a different location. When processed, the virtual mail server(s)  106  can attach each mail item to the forwarding address. Virtual mail server(s)  106  can batch/group mail shipping. For example, a customer may request that their mail to be grouped forwarded one weekly, bi-weekly or monthly basis. Virtual mail server(s)  106  can then automatically notify an administrator via SMS, email and/or push notification. Virtual mail server(s)  106  can notifies a shipping courier for pick-up based on the duration set. Virtual mail server(s)  106  can forward mail based on the location of the user. Virtual mail server(s)  106  can verify the location of the user (e.g. using geolocation techniques based on the user&#39;s mobile device location, etc.). Virtual mail server(s)  106  can then query the user to determine that the new location is where mail needs to be shipped. Virtual mail server(s)  106  can receive a confirmation from the user. Virtual mail server(s)  106  can then forward the user&#39;s physical mail to said location and the physical mail can processed to the new address (e.g. for a specified period of time as indicated by the user in the confirmation, while it is detected that the user remains at the location, etc.). It is noted that a customer may add more than one forwarding address. Customer can also request forwarding of physical mail to any of a set of pre-stored forwarding addresses (e.g. based on an identifier of the sender, date of receipt, etc.). Virtual mail server(s)  106  can automatically attach a separate forwarding address to each physical mail item. An administrator can complete the forward by clicking on the mail. Shipping postage can be calculated based on the mail item type and address to be shipped to. Virtual mail server(s)  106  can automatically generate postage labels for printing. Virtual mail server(s)  106  can implement automated page counts. Virtual mail server(s)  106  can enable users to request scans of all or specified portions of physical mail. Virtual mail server(s)  106  can then scan the specified pages. Virtual mail server(s)  106  can perform an automatic page count but discards count based on the following conditions: when a page is blank, when a page has only header or footer text (e.g. page m of page set n, one-line header or footer, etc.). Virtual mail server(s)  106  can perform shipping carrier notifications for pickup with pre-paid shipping labels. 
     Virtual mail server(s)  106  can enable a text-to-speech option for providing a user&#39;s mail content via an audible source. Virtual mail server(s)  106  can use various AI/machine learning methods to optimize the processing of user requests. For example, a user can request a scan, a shredding operation, forwarding of specified mail based on sender, etc. Initially, virtual mail server(s)  106  can automatically set any of requested status based on mail type, request history with a verify that the action is to be taken automatically if not manually changed. For example, when a mail item arrives virtual mail server(s)  106  can enable the user to request a text-to-speech method for accessing the mail content (e.g. the user is driving or busy with other tasks while wants to read the mail content). In this way, the mail content can be audibly provided to the user (e.g. as a podcast format with specified access and security protocols to listen to). Machine learning, computer vision methods and image recognition can be utilized. For example, if the user has made a previous request, the Virtual mail server(s)  106  can obtain a digital image of the mail content (e.g. in a typed or handwritten or any type of font face, images or files type, etc.) and automatically learn the behavior(s) therein. Virtual mail server(s)  106  can convert the image file to a binary file (e.g. digital text) or re-renders an image to a text, a speech or a drawing, etc. In this way, virtual mail server(s)  106  can learn each users&#39; behavior/preferences and then can act accordingly with all available functions with or without human intervention. 
       FIG. 2  depicts an exemplary computing system  200  that can be configured to perform any one of the processes provided herein. In this context, computing system  200  may include, for example, a processor, memory, storage, and I/O devices (e.g., monitor, keyboard, disk drive, Internet connection, etc.). However, computing system  200  may include circuitry or other specialized hardware for carrying out some or all aspects of the processes. In some operational settings, computing system  200  may be configured as a system that includes one or more units, each of which is configured to carry out some aspects of the processes either in software, hardware, or some combination thereof. 
       FIG. 2  depicts computing system  200  with a number of components that may be used to perform any of the processes described herein. The main system  202  includes a motherboard  204  having an I/O section  206 , one or more central processing units (CPU)  208 , and a memory section  210 , which may have a flash memory card  212  related to it. The I/O section  206  can be connected to a display  214 , a keyboard and/or other user input (not shown), a disk storage unit  216 , and a media drive unit  218 . The media drive unit  218  can read/write a computer-readable medium  220 , which can contain programs  222  and/or data. Computing system  200  can include a web browser. Moreover, it is noted that computing system  200  can be configured to include additional systems in order to fulfill various functionalities. Computing system  200  can communicate with other computing devices based on various computer communication protocols such a Wi-Fi, Bluetooth® (and/or other standards for exchanging data over short distances includes those using short-wavelength radio transmissions), USB, Ethernet, cellular, an ultrasonic local area communication protocol, etc. 
       FIG. 3  is a block diagram of a sample computing environment  300  that can be utilized to implement various embodiments. The system  300  further illustrates a system that includes one or more client(s)  302 . The client(s)  302  can be hardware and/or software (e.g., threads, processes, computing devices). The system  300  also includes one or more server(s)  304 . The server(s)  304  can also be hardware and/or software (e.g., threads, processes, computing devices). One possible communication between a client  302  and a server  304  may be in the form of a data packet adapted to be transmitted between two or more computer processes. The system  300  includes a communication framework  310  that can be employed to facilitate communications between the client(s)  302  and the server(s)  304 . The client(s)  302  are connected to one or more client data store(s)  306  that can be employed to store information local to the client(s)  302 . Similarly, the server(s)  304  are connected to one or more server data store(s)  308  that can be employed to store information local to the server(s)  304 . In some embodiments, system  300  can be a collection of remote computing services constituting a cloud-computing platform. 
     Example Methods 
       FIG. 4  illustrates an example process  400  for a mail entry recognition system for typed or handwritten envelopes, according to some embodiments. In step  402 , a postal mail is received. In step  404 , the physical mail envelopes contents are obtained by taking a digital photograph of the envelope using a mobile device. An OCR operation can be implemented on envelopes contents. A sender, recipient and other postal information can be determined. Process  400  can read the label(s) and automatically save the information to a database. This information can then be used to trigger other relevant processes in the mail virtualization system (e.g. notifications, virtualized mail delivery to a virtual inbox, etc.). 
       FIG. 5  illustrates an example process  500  for grouping and/or batching a virtualized-mail entry with historical memory, according to some embodiments. In step  502 , a postal mail is received. In step  504 , the postal mail item is entered (e.g. scanned with OCR system, digital photograph(s) taken, etc.) and either manually through a web page and/or with a digital photograph using a mobile device. In step  506 , process  500  saves the output of step  504  (e.g. a data store) and remembers the type, senders name, receiver&#39;s name shape. In step  508 , a background process scans similar physical mail items (e.g. to a same recipient as output of step  504 ) in a same manner and merges all the similar as one virtualized mail item. Similarity can be determined by a substantial identity of the recipient (e.g. same name, same address, etc.). 
     In one example of process  500 , a similar mail item arrives. Process  500  can obtain a digital photo of the letter or parcel shipping label. Process  5000  can scan a similar postal mail item and automatically enters the postal mail item into the system with the same type and other already captured data from previous entries. Process  500  can enter the sender&#39;s information. Process  500  then scans one or more similar mail items and automatically saves these with the other already captured data from previous entries. In some examples, process  500  can include an automatic junk mail recognition functionality that can identify junk mail. Process  500  can notify the user of archived junk mail. Junk mail can be stored in a separate file/location as other virtual mail. It is noted that a user can also mark virtual mail as junk mail. Process  500  can use various techniques to identify junk mail such as, inter alia: information on the label, post card, sender identity, etc. 
       FIG. 6  illustrates an example process  600  for file merging on scan requests, according to some embodiments. In step  602 , process  600  receives a user requests to scan a physical mail item. In step  604 , process  600  opens the physical mail item that contains more than one page. In step  606 , obtains a digital image of the plurality of pages of the physical mail. In step  608 , process  600  merges the files of the digital image into one file with a page count. It is noted that process  600  supports various digital image file formats. 
       FIG. 7  illustrates an example process  700  for scanning double-sided documents, according to some embodiments. In step  702 , process  700  receives a user requests to scan a physical mail item. In step  704 , process  700  opens the physical mail item that contains more than one page. In step  706 , process  700  can obtain a digital images of the plurality of pages of the physical mail by turning said pages. Turning can be implemented by a robotic page turner in one example. In step  708 , process  700  can merge the files into one file with page count and support all file formats by knowing the sequencing of the pages or re-order them if they contain page numbers. 
       FIG. 8  illustrates an example process  800  for HWR of physical mail envelopes, according to some embodiments. In step  802 , process  800  can receive a user request(s) to scan a physical mail item. Process  800  can scan the physical envelope in step  804 . 
     In step  806 , implement HWR/OCR analysis to identify recipient and sender. In one example, region(s) of the envelope associated with a sender&#39;s identity and address data (e.g. upper left-hand corner of an envelope) can be scanned and the data stored. The region(s) of the envelope associated with the receiver&#39;s identity and address data can be scanned and the data stored. It can be determined that the part or all of the stored data is from a hand-written information. This hand-written information can be automatically converted to a text and/or letter codes which are usable within computer and text-processing applications. The data obtained by this form is a static representation of handwriting. Data from datastores  808 - 812  can be used for the HWR analysis (e.g. for problem domain reduction, etc.). For example, it is known that the receiver&#39;s identity and address is limited to an identity and address in data store of customers  810 . Accordingly, data store of customers  810  can be used for problem domain reduction. Additionally, data store of identified handwriting  812  can be used to match past known handwriting samples with present handwriting under analysis, as well as, problem domain reduction. Furthermore, machine learning can be used to refine HWR functionalities based on past handwritten physical mail samples. A frequency of sender identity to receiver identity can be used to increase accuracy of the HWR functionality as well. Process  800  can be used to identity of one of several users associated with a specified address (e.g. a specified family member&#39;s names, a specific employees name, etc.). 
     In one example, neural network recognizers learn from an initial image training set. The trained network then makes the character identifications. Each neural network uniquely learns the properties that differentiate training images. It then looks for similar properties in the target image to be identified. Feature extraction can be used by the neural network recognizers. Various example properties can include, inter alia: aspect ratio; percent of pixels above horizontal half point; percent of pixels to right of vertical half point; number of strokes; average distance from image center; is reflected y axis; is reflected x axis; etc. This approach can provide the recognizer algorithm more control over the properties used in identification. Historical handwriting samples from previous senders can be used to train and continuously improve the neural network recognizers. Human curation can also be used to train and continuously improve the neural network recognizers. 
     In one example, the system can automatically recognize the receiver&#39;s information by: receiver&#39;s name; receiver&#39;s mailbox number; receiver&#39;s company name; any additional recipient names instead of the account holder name. 
       FIG. 9  illustrates an example process  900  for handwriting recognition (HWR) on physical mail envelopes addressed to a user, according to some embodiments. In step  902 , process  900  can scan a physical mail item to obtain a digital image of the address-side of the physical mail item. In step  904 , process  900  can identify that at least one of a return address region or a recipient address region of the address-side of the physical mail item. In step  906 , process  900  can determine that the at least one of the return address region or a recipient address region comprises a handwritten text. In step  908 , process  900  can provide a data store of known senders to the recipient address. The data store of known senders includes a data store of return address information in a known sender handwriting samples and a data store of receiver address information in the known sender handwriting samples. In step  910 , process  900  can provide a data store of a receiver&#39;s identity and address. In step  912 , process  900  can create a first training set comprising the historical data store of known senders. In step  914 , process  900  can train the neural network in a first stage using the first training set. In step  916 , process  900  can create a second training set for a second stage of training comprising the data store of a receiver&#39;s identity and address. In step  918 , process  900  can train the neural network in a second stage using the second training set. In step  920 , process  900  can integrate the neural network into an HWR functionality. 
     CONCLUSION 
     Although the present embodiments have been described with reference to specific example embodiments, various modifications and changes can be made to these embodiments without departing from the broader spirit and scope of the various embodiments. For example, the various devices, modules, etc. described herein can be enabled and operated using hardware circuitry, firmware, software or any combination of hardware, firmware, and software (e.g., embodied in a machine-readable medium). 
     In addition, it can be appreciated that the various operations, processes, and methods disclosed herein can be embodied in a machine-readable medium and/or a machine accessible medium compatible with a data processing system (e.g., a computer system), and can be performed in any order (e.g., including using means for achieving the various operations). Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. In some embodiments, the machine-readable medium can be a nontransitory form of machine-readable medium.