Patent ID: 12229734

DETAILED DESCRIPTION

The present disclosure relates to remote check deposit using a mobile device, but a person of skill in the art will recognize how aspects described herein can be applied to other context (e.g., with documents other than checks). The mobile check deposit system and methods described herein include a mobile banking application operating on a mobile device associated with a user. The mobile device is often a personal computing device of a user, such as a smart phone or tablet. An image capture device, such as a camera, on the mobile device is configured to be activated at particular stages in the execution of the mobile banking application to capture image(s) of a check. In some embodiments, the mobile banking application, when executed by the mobile device, receives an actuation instruction from the user, which causes the image capture device to capture an image of a check. For simplicity, an image of a check is referred to herein as a check image.

When capturing the check image, in some examples, the user is required to manually actuate capturing of the image, such as by selecting a capture button presented via a graphical user interface of the mobile device or providing a voice command via another input interface of the mobile device (e.g., by uttering an image capture command), rather than using the mobile deposit application to employ automatic image capture mechanisms to automatically capture the check image. That is, rather than using an automatic image capture mechanism to automatically capture the check image when, for example, the check is properly in the frame of the camera and in focus, the embodiments described herein require manual actuation of the image capture device to capture a check image. In response to the manual user actuation, a check image is obtained from the image capture device and, in turn, processed by the mobile banking application. Aspects of the embodiments described herein can implement manual actuation to capture the check image in a manner that provides more control to the user, minimizes user frustration caused by automatic image capture techniques, improves user privacy (e.g., by placing the capture of images in user control by not having an image feed automatically processed), decreases the use of computing resources needed to automatically analyze a camera feed, and reduces disclosure of certain information that is typically communicated over networks when using automatic image capture techniques.

In some embodiments, the captured check image is not presented (e.g., displayed via a user interface of the mobile device) to the user to verify the check image meets any capture requirements. Rather, the check image is verified by the mobile banking application to determine whether the captured image is usable for completing the remote check deposit process. Verifying aspects of the check image on the mobile banking application executed by the mobile device, referred to sometimes simply as local check image verification, can have several advantages over typical check image verification mechanism. For instance, showing the user an image of the check at this stage is relatively unlikely to add additional benefit because the user already took the image thinking that it was sufficient. Further, keeping sensitive financial information, such as check images, on the mobile device until the check is locally verified reduces the risk of data exposure during transmission of an unusable check image to a remote server. This enhances user privacy and reduces the potential for unauthorized access to sensitive data. In addition, processing aspects of the check image locally on the mobile device eliminates the need to send the image to a remote server, which can lead to faster processing times. This is especially beneficial for users in areas with limited or unstable network connectivity. Local image processing allows for some of the check deposit steps to be performed even when the mobile device is offline. Still further, processing the check image on the mobile device enables immediate feedback to users regarding the quality of the image. This feedback can guide users in capturing clearer images, reducing the likelihood of future errors and rejected deposits. Sending images to a remote server can consume significant network bandwidth, especially in cases of multiple users simultaneously submitting check images. Processing locally minimizes this bandwidth usage. Relatedly, by offloading certain image processing to the mobile device, the load on the remote server is reduced. This can contribute to improved server performance, scalability, and response times.

In some embodiments, the mobile banking application causes the mobile device to perform a check image verification process on-device. For simplicity, the check image verification process on-device is also referred to herein as an on-device check image verification process. The on-device check image verification process includes performing optical character recognition (OCR) on the check image locally at the mobile device, and determining whether required check data (e.g., MICR line data, check amount data) is usably obtainable from the check image. The on-device check image verification process further includes verifying that a signature is present on a signature line. In some embodiments, the on-device check image verification process further verifies that the check image meets requirements of the Check 21 Act. By verifying that the check image meets requirements of the Check 21 Act locally at the mobile device, a higher resolution check image is used for verification, as opposed to using a lower resolution TIFF image at a financial institution server to assess Check 21 Act requirements at the financial institution server. Thus, the accuracy and ease at which the check image's compliance with the Check 21 Act is improved by performing aspects of Check 21 verification at the mobile device with the check image rather than at the financial institution server.

The OCR data can be presented to a user on a user interface of the mobile device. In an example implementation, the OCR data is presented to a user by generating a check representation image and displaying the check representation image via the user interface of the mobile device. The check representation image, in some embodiments, is an illustration version of a check (e.g., cartoon-style), rather than the actual check image that was captured. Presenting an illustrated version of a check (which can be referred to as a check representation image) rather than the actual check image is beneficial in situations where the check image is defective. Where the check image is defective, the software on the mobile device may be unable to sufficiently understand the check image to provide feedback regarding the check image. For instance, portions of the check image may be out of frame or too blurry and so cannot be referenced to show lacking check data. In such situations, the user interface of the mobile device can inform the user of the problem. In some examples, feedback regarding the problem is directly shown to the user on the check image (e.g., annotations can be made on or near a displayed check image). In other examples, a check representation image, rather than the image of the check itself, can be used to show the user a clearer representation of the lacking information or the way in which the check image is defective. Further, showing the user what the check image ought to look like can be beneficial in understanding how to improve imaging.

In some instances, the OCR data is presented on the illustrated version of a check even where the software determines that the check image is sufficient. This can be useful to the user understanding how the software perceives the check image. This can give the user confidence in the system. Further, it can help the user identify problems with false negatives, such as where the software has false confidence that the check image is acceptable and its interpretation accurate.

If the check image is determined to be unacceptable, for example, because required check data is lacking (e.g., MICR line data does not include an account number), check data is inaccurate (e.g., the data fails a checksum, has too many digits, or too few digits), the check representation image includes an indicator, such as a circle or highlighting, that indicates a region of the check image that corresponds with the lacking check data that is required to deposit the check. In some embodiments, the mobile banking application, when executed by the mobile device, causes the mobile device to receive input corresponding to the lacking check data, for example, via an input interface of the mobile device, to remediate the lacking check data. In some embodiments, the mobile banking application, when executed by the mobile device, causes the mobile device to prompt a user via the user interface of the mobile device to recapture the check image to obtain a check image that includes the lacking check data.

When an acceptable check image is received by the mobile device, the OCR data is sent to a server associated with a financial institution for a first level of processing. The financial institution server completes a validation process to validate the received OCR data. In some instances, the financial institution server completes a duplication detection process and a fraud detection process. The server can perform other processes, such as validating deposit limits and presenting information back to the user. If the OCR data is validated, the financial institution server provides a notification to the mobile device indicating that the OCR data from the check image is valid to complete the remote check deposit process. In response to receiving a notification that the OCR data has been validated at the mobile device, a confirmation request is presented on the user interface for confirmation to process the check for remote deposit. Upon receiving confirmation to process the check for remote deposit, the captured check image is sent to the financial institution server to complete the remote check deposit. After the deposit is complete, the mobile device receives a deposit receipt notification.

Example Environment

FIG.1illustrates a diagram of an example environment100for remote check deposit using a mobile device104. In some embodiments, user U has a check102to deposit remotely into a bank account associated with a financial institution via mobile check deposit processing using the mobile device104. The mobile device104includes a user input device106, a display device108, a data communication device110, a processing device112, an image capture device116, and a memory device118. Aspects of the remote check deposit process are performed locally at the mobile device104via the mobile application120, and aspects of the remote check deposit process are performed at the financial institution server124. In some embodiments, the mobile application120executed on the mobile device104is in communication with the financial institution server124over network122throughout the remote check deposit process. In some embodiments, the mobile application120executed on the mobile device104is in communication with the financial institution server124over network122at certain stages of the remote check deposit process. While the mobile device104, network122, and financial institution server124are briefly described in this section, in addition or instead they may also include aspects described below in relation to the computing system ofFIG.16.

A user input device106of mobile device104operates to receive a user input from a user for controlling the mobile device104. The user input can include a manual input and/or a voice input. In some embodiments, the user input device106includes one or more buttons, keys, touch levers, switches, and/or other mechanical input devices for receiving the input101. The user input device106can include a touch screen or a gesture input device. In some embodiments, the user input device106can detect sounds including the voice input such as a voice of a user (e.g., an utterance) for controlling aspects of a remote check deposit process via the mobile device104.

In some embodiments, a display device108is provided that operates to display a graphical user interface that displays information for interacting with the mobile device104. Examples of such information include check data information, notifications, and other information. In some embodiments, the display device108is configured as a touch sensitive display and includes the user input device106for receiving the input101from a selector (e.g., a finger, stylus etc.) controlled by the user U. In some embodiments, therefore, the display device108operates as both a display device and a user input device.

The data communication device110operates to enable the mobile device104to communicate with one or more computing devices over one or more networks, such as the network122. For example, the data communication device110is configured to communicate with the financial institution server124and receive notifications from the financial institution server124at least partially via the network122. The data communication device110can be a network interface of various types which connects the mobile device104to the network122.

The processing device112, in some embodiments, comprises one or more central processing units (CPU). In other embodiments, the processing device112additionally or alternatively includes one or more digital signal processors, graphical processing units (GPUs), field-programmable gate arrays, or other electronic circuits.

The image capture device116, in some embodiments, is one or more cameras integrated with the mobile device104. The image capture device116is configured to capture an image or images of the check102. When the image capture device is activated, user actuation is required to capture the check image rather than permitting automatic image capture techniques that capture an image automatically without requiring any manually input by the user U. Check images captured by the image capture device116are stored in the memory device118and received by the mobile application120executed on the mobile device104for remote deposit of the check102.

The memory device118typically includes at least some form of non-transitory computer-readable media. Non-transitory computer-readable media includes any available media that can be accessed by the mobile device104, such as volatile and nonvolatile, removable and non-removable media implemented in any device configured to store information such as computer readable instructions, data structures, program modules, or other data. Memory device118can also include, but is not limited to, random access memory, read only memory, electrically erasable programmable read only memory, flash memory and other memory technology, compact disc read only memory, blue ray discs, digital versatile discs or other optical storage, magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the mobile device104in a non-transitory manner.

The memory device118operates to store data and instructions. In some embodiments, the memory device118stores instructions for a mobile application120. In some embodiments, the mobile application120is a mobile banking application hosted by a financial institution. In some examples, one or more methods described herein can be implemented as instructions stored in one or more memory devices on one or more computers that, when executed by one or more processors, cause the processors to perform one or more operations described herein.

In some examples, as described herein, network122includes a computer network, an enterprise intranet, the Internet, a LAN, a Wide Area Network (WAN), wireless transmission mediums, wired transmission mediums, other networks, and combinations thereof. Although network122is shown as a single network inFIG.1, this is shown as an example and the various communications described herein can occur over the same network or a number of different networks.

FIG.2illustrates an example user interface200of a remote check deposit system for capturing a check image. A display device108of mobile device104operates to display the user interface200. One or more image capture devices operating on mobile device104capture one or more images of a check102for remote deposit.

In some embodiments, the image capture device is activated to capture a check image without specifying any camera settings for the image capture device, and default settings of the image capture device are used to capture the check image. In some embodiments, specific settings of the image capture device (e.g., brightness, zoom) are enabled when the image capture device is activated. In some embodiments, guidance is provided to aid in capturing the check image. For example, automatic edge detection techniques can be used to determine boundaries of the check or regions therein. In some embodiments, a visible alignment guide206is presented on the user interface200such that the check102is aligned within the visible alignment guides206and user actuation is received to capture the check image once the check102is aligned within the visible alignment guide206. Visible alignment guides206can remain static or can be adjusted to the boundaries of the check as the image capture device is moved in relation to the check102.

In some examples, the user interface200displays a live preview of what will be captured when the capture button is pressed but also includes annotations over the live preview. The annotations can include feedback on whether the image as-is would be acceptable (e.g., using a same or similar process described below in method300). In some examples, in addition to classification of the check, a machine learning model can then perform object detection and object segmentation on the frame of the video or live preview. The object detection can then be used to identify aspects of the front of the check and the back of the check. Further processing of the check objects is also possible; the processing could include handwriting recognition, OCR, other processing, or combinations thereof. The front of the check can then reveal information, such as date, payee name, payor name and address, the amount, check number, financial establishment associated with the check, whether the check is signed, and whether the check is endorsed. In some embodiments this information can then be digitally superimposed in typed text at the correct location (e.g., the amount and date can be shown at appropriate location on the check). An endorsement can be digitally superimposed on the back of the check if an endorsement is lacking. In other embodiments, a separate check sample can be shown to the user for confirmation of capturing the correct content.

To capture the check image, user actuation can be required. In some examples, user actuation is the selection of a capture button210presented on the user interface200causing the image capture device to capture the check image. In other examples, user actuation is the selection of a physical button on the mobile device104itself causing the image capture device to capture the check image (e.g., a volume button of the mobile device104). Check images may be captured of the front side of the check and the back side of the check. Once a check image or images are captured, the check image(s) are received by the mobile application executed on the mobile device104for remote deposit of the check.

In some examples, user actuation is required but actuation of the capture button210is prevented until a process determines that the image to be taken will be sufficiently usable. For instance, the capture button210is unavailable (e.g., locked or grayed out) unless a usability metric is higher than a predetermined threshold. The usability metric can be correlated with how well the check can be used for further processing. The usability metric can be determined based on the angle of skew of the check object, the amount of light on the object (e.g., too dark or very bright), the amount of contrast, the quality of the check image itself, other characteristics, or combinations thereof. In some instances, the capture button210is prevented from being actuated until required check data is present and detectable in the live preview.

In an example, the check image is obtained using a document mosaicking technique. Document mosaicking is a traditional technique in which multiple frames are combined to form a composite image that uses information from the multiple frames to form a more suitable image (e.g., in terms of area covered, image resolution, sharpness, or other factors) than before. Example implementations include image stitching techniques, such as those provided by the OPENCV library. In some examples, the actuation of the capture button210causes the mobile device104to take the multiple frames, apply mosaicking, and produce a final check image as a result. In other examples, while the user interface200is active, the camera continuously obtains and discards images until the capture button210is actuated and then previous x frames are retained as well as future y frames are obtained. The x and y frames can then serve as the multiple frames for use with mosaicking. In addition or instead, a frame is captured in response to the user actuating the capture button210, and then n frames before and after the capture button being actuated are used to enhance the image captured in response to the user selecting the capture button210.

FIGS.3aand3billustrates an example flowchart of a method300for remote check deposit using a mobile device104. The operations withinFIG.3aoccur at the mobile device104, and the operations withinFIG.3boccur at a financial institution server. In an example, the mobile device104activates a mobile application120for mobile check deposit in response to receiving user input. The mobile application120may be, for example, a dedicated application for depositing checks, a full featured banking application, another kind of application, or combinations thereof. The method300for remote check deposit can begin with operation302.

Operation302includes a mobile application120operating on the mobile device104receiving a check image captured by an image capture device116of the mobile device104. For instance, the check image can be received in response to receiving a user actuation causing the image capture device116to capture the check image. In some embodiments, the check image is captured in response to a user selecting a selectable element (e.g., capture button210) presented on a user interface200of the mobile device104as described above with reference toFIG.2. The check image can but need not be captured automatically using automatic image capture techniques. In some examples, the mobile application120executed on the mobile device104receives a check image of the front of the check102and a check image of the back of the check102. In some examples, a single actuation of the capture button210causes multiple check images to be obtained. In some instances, the images can be obtained with different settings (e.g., from different cameras or with different aperture or shutter speed settings). The obtaining an image can include using all of these images or selecting a subset of these images that meet a selection criteria (e.g., having a usability score higher than a threshold). Many mobile devices104include operating systems with application programming interfaces that facilitate capturing images. For instance, on IOS and macOS, the AVCaptureSession class is used for media capture. Following operation302, the flow of the method300can move to operation304.

At operation304, the mobile application120executed on the mobile device104causes the mobile device104to perform OCR on the check image. The mobile application includes an OCR engine configured to extract text from the check image and recognize characters of the extracted text to generate OCR data. The OCR engine can take any of a variety of forms. OCR and in particular handwriting recognition is a relatively popular area of research and many different people and organizations have proposed solutions. In some instances, the mobile device104causes OCR to be performed by a software OCR process hosted by a third party (e.g., via AMAZON TEXTRACT, GOOGLE CLOUD VISION API, MICROSOFT AZURE AI SERVICES), a manual OCR process hosted by a third party or locally (e.g., the user of the mobile device104is asked to manually enter all characters in the image), or using a library stored and run locally. In an example, OPENCV can be used to recognize characters. In some examples, there is a machine learning framework trained to receive an image (e.g., part or all of the check image) and provide recognized characters as output.

In an example, the OCR implementation includes one or more aspects of US 2022/0319216, which was filed as U.S. patent application Ser. No. 17/569,121 on Jan. 5, 2022, and which is hereby incorporated herein by reference in its entirety for any and all purposes. That application describes performing geometric extraction on a document (e.g., a check) by recognizing text blocks of the unstructured document, generating bounding boxes that surround and correspond to the textual blocks, determining search paths having coordinates of two endpoints and connecting at least two bounding boxes, and generating a graph representation of the at least a portion of the page, the graph representation including the plurality of textual blocks, the coordinates of the vertices of each bounding box and the coordinates of the two endpoints of each search path.

In some instances, the OCR engine is specifically configured for (e.g., trained for) use on check data. A general purpose OCR engine may be configured with a very wide vocabulary of potential words, but that may create issues for processing check data. For example, an entry in a date field could be “MAR 10”, which a general purpose OCR engine may plausibly (but incorrectly) interpret as “MARIO”, which would not make sense in the context of a date field on a check image. Rather, the OCR engine applied to the date field may be configured to favor outputting dates. As an example, an OCR engine may be configured to produce the n highest confidence outputs and then select the highest confidence one that is a date. As another example, there may be a separate OCR engine trained on the date fields to more accurately interpret what is written in that field. Similar configurations may be used for other fields on the check image.

Depending on the implementation, the operation304can include one or more sub-steps or functions, such as a preprocessing operation304a, a field identification operation304b, a template matching operation304c, a side identification operation304d, other steps, or combinations thereof.

In the preprocessing operation304a, one or more preprocessing techniques can be applied to the check image. Such preprocessing techniques can improve the reliability, consistency, or results of the following steps.

Example preprocessing includes identifying check boundaries relative to the remainder of the check image. This can be performed using any of a variety of techniques such as edge detection (e.g., as implemented in OPENCV) and then setting the boundaries of the check within the check image to a polygon that has an expected check aspect ratio. This can be performed or enhanced by using color, contrast, brightness or other kinds of identification. For instance, a check may have different visual characteristics compared to a table on which it is resting when the picture is taken. Such visual characteristics can be used to identify the check within the image. In another example, the boundaries can be determined by identifying the corners of the check and then connecting the corners with lines to form the boundaries.

Further example preprocessing includes normalizing the check within the image. This normalization processing can include normalizing the position or proportions of the check by, for example, skewing, deskewing, rotating, scaling, stretching, moving, or applying other transformations to the check within the check image to bring the check closer to an expected or standard format on which other steps may depend. This can include cropping the image so substantially only the check is within the check image. The normalizing can be based on the check boundaries identified above. This can also include normalizing the check to have a predetermined aspect ratio. Normalizing the check within the check image can further include adjusting image quality of the check image, such as modifying brightness, contrast, levels, sharpness, or other modifications.

In still further examples, preprocessing can include removing backgrounds from the check image. If the check was placed on a table or surface by the user to capture the check image, the background containing the table or surface can be removed, blurred, filled, or otherwise modified. The brightness of the check image can also be adjusted to obtain an improved image brightness for performing OCR on the check image. Defects detected in the check image can be removed.

Preprocessing can include saving multiple versions of the check image at the mobile device104. For instance, an original color image can be stored in addition to a bi-tonal TIFF (Tag Image File Format) formatted image. Thus, the mobile application can include a TIFF image generation module configured to convert the original color check image into a grayscale TIFF image. The color image can be saved in a specified format, such as TIFF, JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), PDF (Portable Document Format), or another format.

In the field identification operation304b, the OCR engine or another engine analyzes regions within the check image to identify check data fields such that the generated OCR data includes an indication of check fields and data associated with each check field. For example, the engine is configured to recognize a region of the check as an amount box, so the engine can determine that the text within that region corresponds with the check amount. In another example, expected fields can be used to determine which side of a check is depicted in the image (see preprocessing operation304a, above). In a further example, whether all expected fields can be identified can be used to determine whether the required check data is present. Example front check image expected fields can include a payor information field, a payee name field, a check number field, a date field, a check amount in words field, a memo field, and a MICR line data field (e.g., including a routing number field, an account number field, and a check number field). Example back check image expected fields can include an endorsement field. These fields can be identified using various techniques including OCR. For instance, a field can be identified and a bounding box can be drawn around the field. The bounding box can then be used as a region in which OCR will be performed to determine a value associated with that field. Because identifiers or labels of these fields are often typed, they may be more reliably identified than handwritten areas of a check image. In some examples, the fields can be identified using known check templates, such as described below in template matching operation304c.

In the template matching operation304c, known check templates are used to determine regions on a check image where check data is be expected. For instance, because many checks may have similar formats (e.g., the relative positioning of fields and other content) that can be predetermined and labeled (e.g., manually or with an automated process), identifying that the check image contains an image of a check having a predetermined pattern can be used to identify or know that expected fields are present (e.g., because otherwise the template would not match). The check image can be compared to the known check templates to determine a format that the check follows. When a match is found between a known check template and the check image, the matching known check template can be used during the OCR process to correlate the extracted text with the corresponding check field based on the location of the extracted text on the check. This improves the accuracy of the OCR process to generate OCR check data. If there is no known check template that matches the check image, this might be used as an indication that the check image is defective and there is a problem with the check image. If there is no known check template that matches the check image, in some embodiments, the image capture device is activated with instructions for the user to capture another check image. In an example implementation, a template matching function can be used (e.g., the cv.matchTemplate( ) function from OPENCV) to determine the location and position of a known check template (e.g., the check template can be samples of known check design patterns) within the check image to determine which template the check follows. If none matches, then the application can infer that the image is bad). If there is a match, then the application can take known positions of check elements from the template (e.g., bounding boxes of where that data is expected to be) and then perform OCR or other recognition of the data within that region to increase accuracy of the OCR process.

In the side identification operation304d, the check image can be analyzed to determine whether the check image depicts the front or the back of the check. In some examples, the output of prior steps can be used for this determination, such as whether the image include fields associated with the front or back or whether the image matches a template of a front of a check or a back of a check (e.g., if the check image contains an endorsement field, then it is considered the back of the check and if the check image contains a memo field, then it is considered the front of the check). In one example implementation, a machine learning model is trained to determine whether an image is of a front of a check or a back of the check.

Following operation304, the flow of the method300can move to operation306.

At operation306, OCR data generated from the check image is presented on the user interface of the mobile device104. In some embodiments, the presented OCR data includes one or more of a check amount, a date, a payee name, an account number, a routing number, and a check number. In some embodiments, the OCR data is presented on a check representation image, wherein the check representation image is an illustration representing the check image (e.g., cartoon-style) and not the check image itself. The check representation image can include standard check fields (e.g., payee line, date field, amount box, MICR line), and the OCR data is displayed in the appropriate corresponding location on the check representation image. For example, if the OCR data includes a check amount of $100.00, then “100.00” is displayed in the amount box on the check representation image. Presenting the OCR data on the check representation image (operation306) is described in more detail below with respect toFIG.4a.

Continuing operation306, in examples, the OCR data is presented on the user interface of the mobile device104in a tabular or list format in addition or instead of the check representation image. An example tabular or list format is described in more detail with respect toFIG.5aandFIG.5b, below.

Following operation306, the flow of the method300can move to operation308.

At operation308, on-device check image verification processing occurs at the mobile device104. For example, the check image verification processing can verify that the OCR data generated from the check includes required check data needed to process the check for remote deposit (operation308). In order to process a check for deposit, certain information must be obtained from the check so that a financial institution can determine, for example, where the funds for deposit are coming from and an amount of funds to deposit. The verification operation308can verify whether the OCR data generated from the check image includes required check data to process the check for remote deposit. This can include comparing the generated OCR data to predetermined required check data fields. In some examples, required check data comprises an amount and MICR line data, wherein the MICR line data includes an account number and a routing number, and a check number. In some examples, the verification operation308includes determining whether any value is present in one of the fields. For instance, if there is OCR data corresponding to the date field, then it can be determined that the required data for that field is present. In some examples, the operation can include determining not only that such data is present but also determining whether that data is believable. For instance, the mobile application120can determine whether the data in those fields contains the correct amount or kind of data (e.g., correct number of digits in a routing number or a plausible date is in the date field). For instance, if the check image is slightly blurry, then the OCR engine may produce OCR data for a field, but the OCR data may not make sense in context or at all (e.g., alphabet characters in the numeric amount field). In addition or instead, the mobile application120can determine whether the numeric amount and the written-out amount match. In some embodiments, other required check data can include a payee name and a date. In some embodiments, the verification operation308further includes verifying that the check image complies with Check 21 standards based on the generated OCR data.

In some examples, the verification operation308on the mobile device104includes duplicate detection. For example, at the mobile device and prior to providing the OCR data to the financial institution server124for validation processing, the mobile application120can determine a similarity value between the second data and prior data sent to the financial institution server124. For instance, the mobile application120can store OCR data of all checks that have been sent to the financial institution server124. If the mobile application120determines that the OCR data is sufficiently similar to previously submitted OCR, then the mobile application120can alert the user that this may be a duplicate check. In some examples, the OCR data is only provided to the financial institution server124for validation processing responsive to the similarity failing to pass a threshold (e.g., it is not too similar to prior checks so is unlikely to be a duplicate check).

In some examples, the verification operation308can be performed using a trained machine learning model. For instance, the machine learning model can be trained to verify a check as having proper data or parts thereof for processing. The check image or portions thereof can be provided to the machine learning model and the output of which can be used for verification. The machine learning model can be local to or remote from the mobile device104.

In some embodiments, at operation308, the mobile device104performs additional on-device check image verification processing. For example, the mobile device104can use visual recognition techniques to determine whether a signature or endorsement is present on the check image. A signature can be detected if a pixel count is above a predetermined threshold in a region of the check image where a signature line is located. In some examples, machine learning is used to identify the presence or absence of a signature in a specific region of the check image. In some embodiments, an additional verification operation is performed by the mobile device104to detect whether an endorsement signature is present on a check image of the back side of the check,

If the required check data is not present on the check image, then the method300can take the “NO” branch from operation308to operation312.

At operation312, an indication is presented on the check representation image of a region of the check image that is lacking required check data. For example, if the OCR processing could not determine OCR data for the check amount (e.g., because the check image was blurry around the check amount), so no check amount is included in the generated OCR data, then the check amount box is circled or highlighted on the check representation image. This provides an indication to the user as to why the check image is not useable to process for remote check deposit. By providing an indication on the user interface of the precise region or regions of the check image that are problematic or lacking required data, the user receives informative feedback to enable remediation of issues with the check image.

In the example user interface400ashown inFIG.4aand described below, the OCR data is lacking the check amount in the check amount box426a. Thus, the mobile application executed on the mobile device104determines that the required check data is not present and takes the “NO” branch from operation308to operation312. An indication is then provided on the check representation image of the region lacking the required check data, as described in more detail below with respect toFIGS.4b-4c.

After an indication is provided on the user interface identifying a region of the check image that is lacking required data at operation312, the problem with the check image can be remediated. In examples, the problem is remediated by receiving a new check image captured by the image capture device. Thus, following operation312the flow of the method can move to operation316.

At operation316, the image capture device is activated with instructions for the user to capture another check image. Knowing the problematic region of the original check image, the user can proceed to capture a second check image used to generate OCR data including the required check data that was lacking from the first captured check image.

In some implementations, the flow of the method300moves from operation312to operation314.

At operation314, user input is received that remediates the problem with the check image. In an example, user input is received that edits the OCR data to input the lacking required check data. For example, if an indication was provided on the check representation image that a check amount could not be determined from the check image so the OCR data is lacking the check amount, input can be received via the user interface to input the check amount. In some embodiments, after the lacking required check data has been received, the mobile device104repeats the verification operation to verify that the OCR data generated from the check includes required check data needed to process the check for remote deposit (operation308).

If the required check data is present on the check image, the flow of the method300can take the “YES” branch from operation308to operation310.

At operation310, input may be received via the user interface to edit the OCR data. In some examples, input to edit the OCR data is received to correct a mistake in the OCR data that nonetheless passes the test for whether the required check data is present.

At operation318, the OCR data is provided to a financial institution server124for a first level of validation processing. This can be performed after the OCR data is verified at the mobile device104as including required check data. Following operation318, the flow of the method300can move to operation352, as shown inFIG.3b, which occurs at the financial institution server124.

In some embodiments, during the first level of validation processing by the financial institution server124, the financial institution server124does not have the check image at this stage. In some embodiments the financial institution server124only has the OCR data. The financial institution server124may also include additional metadata regarding the check, such as whether and to what extent the OCR data has been edited by the user (e.g., either to correct the OCR data or to enter missing required check data). The OCR data provided to the financial institution server124can be processed along with instructions for manual review of the OCR data if needed. During the first level of validation processing by the financial institution server124, the authenticity and legitimacy of the check is validated by analyzing the OCR data. For example, the OCR data generated from the MICR line on the check image is validated for accuracy and authenticity to ensure the check is legitimate and corresponds with a legitimate account number and routing number.

At operation352, shown inFIG.3b, the financial institution server124receives the OCR data and performs processing. In an example, this operation includes performing duplicate detection processing. This can include comparing the OCR data with previously submitted OCR data or the information of previously deposited checks to determine whether the same or substantially the same check has already been deposited. If so, then the process may end (e.g., with an error message being sent to the user) or may continue under modified conditions (e.g., with a flag indicating manual review is necessary before depositing). Before, during, or after operation352, the method300can include operation354.

At operation354, shown inFIG.3b, fraud detection processing can be performed using the OCR data. For instance, the OCR data may be analyzed to determine a confidence that the OCR data is associated with fraud. In some examples, the operation354can include providing the OCR data to a machine learning model trained to detect fraud from OCR data. In some examples, fraud detection can determine whether the OCR data is in line or out of line from prior checks from that payee or payor. It can also include determining whether other data about the check indicates a level of fraud. For instance, in addition to the OCR data, the financial institution server124may receive other data regarding the check, such as whether and to what extent a check matches a template of a check. This can include what background the check had, what font the check had, other features, or combinations thereof. In some examples, the OCR data includes a vector representation of some data of the check, such as handwriting or signatures. Such data can be compared (e.g., by measuring a distance in vector space) to known true data to determine whether fraud may be present.

In some examples, the fraud detection can include one or more steps or operations described in U.S. patent application Ser. No. 18/052,081, filed Nov. 2, 2022, which is hereby incorporated herein by reference in its entirety for any and all purposes.

In operation356, shown inFIG.3b, following the duplicate detection and fraud detection processing the financial institution server124sends a notification back to the mobile device104regarding the processing. The notification can include information regarding the relative success or failure of the processing. In some examples, prior to sending the notification, other processing can be performed by the financial institution server124. Such processing can include determining whether processing the check would violate rules or protocols, such as whether depositing the check would result in violating deposit limits.

Returning toFIG.3a, if the financial institution server124is not able to validate the OCR data to confirm the accuracy and authenticity of the OCR data (e.g., taking the “NO” branch from operation320to operation322), this is indicative of a problem with the OCR data, which, in examples, is caused by a problem with the check image. Thus, in examples, the image capture device on the mobile device104is activated with instructions for the user to capture another check image (operation322).

If the financial institution server124is able to validate the OCR data to confirm the accuracy and authenticity of the OCR data, a confirmation is received by the mobile device104from the financial institution confirming that the OCR data has been validated. If a confirmation is received that the OCR data has been validated by the financial institution server124, then the “YES” branch can be taken from operation320to operation324.

In operation324, a request for confirmation to process the check for deposit is presented on the user interface of the mobile device104. In some embodiments, the request for confirmation presented on the user interface includes presenting a selectable element on the user interface that, if selected by the user, indicates that a confirmation has been received from the user to process the check for deposit. Following operation324, the flow of the method300can move to operation326.

If no confirmation from the user is received by the mobile device104to process the check for remote deposit or there is an explicit cancelation by the user, then the flow of the method300can take the “NO” branch from operation326to operation328.

In operation328, in some embodiments, the remote check deposit is canceled. In some instances, this can include sending a notification to the financial institution server124that the remote deposit is canceled.

If confirmation is received by the mobile device104to process the check for remote deposit, then the flow of the method300can take the “YES” branch from operation326to operation330.

In operation330, the mobile device104sends the check image to the financial institution server124with instructions to process the check for remote deposit. In some implementations, prior to operation330, the financial institution server124has not received the check image. Thus, in some embodiments, the check image is sent from the mobile device104to the financial institution server124in a separate communication file from the file that sent the OCR data to the financial institution server124. Following operation330, the flow of the method can move to operation358, as shown inFIG.3b, which occurs at the financial institution server124.

At operation358, shown inFIG.3b, the financial institution server124receives the check images from the mobile device104. The flow of the method300can move from operation358to operation360. In examples, before receiving confirmation at the mobile device104to process the check for remote deposit, an account identification number associated with the check is not provided to the financial institution server124. Rather, account labels or aliases are used, in some examples, for account identification prior to receiving confirmation from the user to process the check for remote deposit.

At operation360, shown inFIG.3b, the financial institution server124initiates the deposit process, such as by performing a traditional mobile check deposit process. In some examples, this operation can include performing additional verification on the check image and the OCR data, such as additional fraud detection verification. This additional processing can include analyzing handwriting, ink color, date style, signature, endorsement signature, etc. to detect if the check was modified after it was drafted by the payor. After additional verification of the check image and OCR data by the financial institution server124, a memo is posted on the user's account showing the check deposit amount. The financial institution server124then writes a log record of the check to capture the details of check being processed for remote check deposit. The flow of the method300can move to operation362.

At operation362, shown inFIG.3b, the financial institution server124sends a notification to the mobile device104confirming the deposit process. This confirmation can be a confirmation that the deposit process has started or is completed. Following operation362the flow of the method can return to the mobile device104as operation332, as shown inFIG.3a.

Returning toFIG.3a, at operation332, the mobile device104receives the confirmation from the financial institution server124. In an example, the financial institution server124has completed the remote check deposit process and the confirmation indicates as much. In some embodiments, the deposit receipt notification is a push notification presented on the user interface of the mobile device104. Upon completion of the remote check deposit process, funds will appear in the user's account. Funds may not appear in the user's account immediately, and may take a few days to become available to the user.

FIG.4aillustrates an example user interface400aof a remote check deposit system for presenting check data. The user interface400ais an example of the user interface on which the OCR data is presented in operation306of method300described above with reference toFIG.3a. User interface400ais presented on a display device108of a mobile device104associated with a user U. In some examples, after a captured check image has been received and the mobile application has caused the mobile device104to perform OCR on the check image, the OCR data is presented on a check representation image406aand presented on the user interface400a.

The check representation image406ais not the actual check image or directly derived from the check image (e.g., arrived at by modifying pixels of the check image until the check representation image406ais achieved), rather, the check representation image406acan be an illustration or “cartoon-style” representation of a check. In some examples, the check representation image406ais a photograph of another check (i.e., not the check102). The representation image406can be ersatz or a simulacrum of the check102. For instance, the check representation image406acan violate one or more standards of check appearance (e.g., ANSI standards for checks), such as by having a non-standard aspect ratio or by replacing MICR fonts with non-MICR fonts. One or more MICR symbols (e.g., transit, on-us, amount, and dash symbols) can be omitted or replaced with non-MICR symbols. The check representation image406acan include watermarks, words, or other features that convey that the check representation image406ais not a real check. The check representation image406acan include one or more watermarks, words, or other features that convey that the check representation image406ais not a receipt, is not the check image, it not an indication that the check102was successfully deposited, and is not meant as a substitute for the check102for any purpose other than facilitating the mobile check deposit workflow. While the above has been described in the context of such indications being displayed on the check representation image406a, they may instead or in addition be provided elsewhere on the user interface displaying the check representation image406a.

In some instances, the check representation image406acan be selected from among multiple different check representation images406aor templates for creating the check representation image406a. For instance, a check representation image406acan be selected from among multiple check representation templates, such as to increase or decrease a similarity between the check representation image406aand the check image. In certain instances, the check representation image406acan be made to appear very different from or very similar to the check image (e.g., different layout, color, symbols, formatting, etc.) depending on user or system preferences. In some examples, the differences from the check image can be increased or decreased depending on how much information is correctly available (e.g., how close the check image is to meeting requirements). For instance, the check representation image406acan be in black and white or grayscale if one or more requirements are not met and then in color once all requirements are met. In some examples, an average or other representative color can be determined from the check image (e.g., the entire image or the check portion thereof) and then used as a background or primary color of the representation image. In another example, an opposite or contrasting color can be used.

In some instances, the check representation image406acan be created by compositing the check image with a check representation template. For instance, one or more regions (e.g., the date region) of the check image can be extracted from then check image and superimposed over the template. Such superimposition can be such that the composite image is representative of not only what the content was on the check image but also how the content in the check image appeared (e.g., as influenced by handwriting, font, ink color, number formatting, date formatting, position, other contributors to appearance, or combinations thereof).

In other instances, the check representation image406alacks stylistic content from the check image. Such a check representation image406acan be representative of what the substantive content was on the check image but not how the content in the check image appeared. In other words, given a check representation image406athat lacks stylistic content from the check, one would be expected to pick the original check image from images of checks having the same substantive content (e.g., date, amount, memo, etc.) but written by different people (e.g., thereby having different stylistic content such as handwriting and other factors contributing to appearance) at a rate no better than random chance.

In still other instances, the check representation image406acan preserve stylistic content without directly importing stylistic elements from the check image. For instance, elements created for the check representation image406acan be constructed with stylistic elements (e.g., font, color, scaling, position, etc.) selected based on a similarity to stylistic elements of the check image. As described in more detail below, such techniques can include selecting a font based on handwriting in the check image, selecting a color based on ink in the check image, selecting element scale/position based, other techniques, or combinations thereof.

Preserving stylistic content without directly importing stylistic elements from the check image can include selecting a font based on handwriting within the check image. For instance, at least some of the OCR data (e.g., a date) can be rendered in several different fonts and then one font can be selected that is most similar to the handwriting in the check image. A machine learning algorithm can be used to determine similarity (e.g., a machine learning model trained to determine similarities between fonts). In addition or instead, similarity can be determined by overlaying each of the rendered fonts over the original and selecting the one that has the most pixels in common. In addition or instead, the renderings and the handwriting can be embedded into a vector space and the rendered font that has the least distance from the handwriting can be selected.

Preserving stylistic content without directly importing stylistic elements from the check image can include selecting a color based on an ink in the check image. For instance, the color can be an average color within a region or at a point believed to be within a handwritten ink line. In addition to or instead of ink color used in handwriting the check, the ink color can be the color of ink used to print the background, text, or other features on the check.

Preserving stylistic content without directly importing stylistic elements from the check image can include selecting a scale and/or position of an element in the check representation based on a scale of a corresponding element in the check image. In an example, scale and position can be determined by identifying locations of regions of handwritten content and then scaling and/or positioning the generated content to approximate the regions. For instance, bounding boxes can be created to encompass a field in the check image and corresponding content in the representation image template can be scaled and positioned (including rotated) to more closely align with the bounding box identified in the check image.

For instance, where the check has cursive handwriting in blue ink, stylistic elements can be selected to mimic that handwriting, such as by selecting a specific script font from among multiple font choices based on a similarity to the handwriting or selecting a font color configured to match the ink color of the pen used in the check.

The OCR data generated from the check image is superimposed on or integrated with the check representation image406ain the appropriate check field location (e.g., OCR data for a check amount is presented in a check amount box on the check representation image). For instance, the content can be placed within a bounding box of a corresponding field (e.g., date content is put in a date bounding box located relative to the date field). In some embodiments, the OCR data includes a payee name408a, a check number410a, a date412a, a check amount in words414a, a memo416a, MICR line data418aincluding a routing number420a, an account number422a, and a check number424a, and a check amount in a check amount box426a.

In some embodiments, the user interface400afurther includes a selectable element432aconfigured to receive user input to edit the OCR data. In examples, the user interface400ais configured such that the display device108is a touch screen and if user U selects (e.g., taps, touches) OCR data displayed on the user interface400a, the user U is able to edit the OCR data. For example, if user U reviews the OCR data displayed on the user interface400aand sees that the OCR data includes a mistake (e.g., OCR data does not match text on check), in some embodiments, input will be received via the user interface to edit the OCR data. Edits to the OCR data, in some examples, can include edits to the spelling of the payee's name or editing the check amount if the OCR data does not match what was written on the check. The edits can appear live on the check representation image.

In some instances, the user input can be compared with the OCR data and determine a relative similarity or difference. For instance, the difference can be determined based on a string metric (e.g., Levenshtein distance or another string metric). If the difference is larger than a threshold, then the system may provide an alert indicating that the correction is meant for correcting OCR errors rather than errors in writing the actual check (in other words errors in the system understanding what is written on the check rather than correcting what should be written on the check instead). In another example, the OCR process can provide an output of the most likely correct answers and it can be determined whether the proposed correction is a top N most likely correct answer. If so, then the correction may be permitted otherwise the correction may be flagged or prohibited.

In some instances, the user input is prohibited from being received when information is lacking. For example, if the OCR data lacks date data, then it may be because the check image was inadequate (e.g., the date field was cut off), rather than from an error in the OCR process. So to solve the problem, the user may be required to recapture the check image rather than manually specify the information. In certain examples, the system may permit the receipt of input even if the data lacking for certain fields, such as the memo field or endorsement field.

In some instances, the mobile application120requests user input to resolve conflicts or uncertainties. For instance, the mobile application120can compare the OCR data for the written out dollar amount and the numerical dollar amount and determine that the two amounts are different (and they should be the same). The mobile application120can then present the two OCR data values and ask the user to choose which one is correct. In addition or instead, the mobile application120can have the user specify the correct amount.

FIG.4billustrates an example user interface400bof a remote check deposit system for presenting check data. The user interface400bis presented on a display device108of a mobile device104operated by a user U. The user interface400bpresents a check representation image406bincluding OCR data generated (e.g., payee name408b, check number410b, date412b, check amount in words414b, memo416b, MICR line data418bincluding a routing number420b, account number422b, and check number424b.

As mentioned briefly above with respect toFIG.4a, the check representation image406a,406bis lacking the check amount in the check amount box426a,426b. In examples, the lacking check data is a result of the check image being of poor quality such that the OCR processing could not extract the text from the image. An indication430bis provided on the check representation image406bof the region lacking the required check data. In the illustrated example, the indication430bshown on user interface400bis an oval around the check amount box426b, which is lacking required check data. In other examples, the indication of the check region lacking required check data can take the form of highlighting, or another type of visual cue provided on the user interface to draw attention to the check region lacking required check data. This indication on the user interface400bprovides feedback to user U regarding a problematic region of the check image. By providing specific feedback to user U regarding the lacking required check data, the problem with the check image can be remediated more easily and the lacking required check data can be obtained more quickly than if no feedback or less specific feedback regarding the problem was provided.

In examples, the lacking required check data is remediated by receiving another picture of the check image. Thus, the image capture device can be activated, and instructions presented to the user U on the user interface400bto capture a second check image of the check that includes the lacking required check data. In some embodiments, the lacking required check data is remediated by receiving input to edit the OCR data to add the lacking required check data. Thus, in some embodiments, a selectable element432bis presented on the user interface400b, and when a selection of the selectable element432bis received, the OCR data presented on the check representation image406bcan be edited. In some embodiments, the user interface400bis configured to receive a selection of the OCR data or of the check region where check data is lacking, and the selection enables OCR data to be edited or added. For example, if the payee name presented on the check representation image does not match the name on the check itself, a selection can be received in response to a user tapping on payee name408b, which in this example use case is “John Doe”, and an option to edit the OCR data is provided. To add lacking check amount data, a selection can be received in response to a user tapping on the check amount box426band an option to edit the OCR data to add the check amount written on the check is provided.

In some examples, the system only permits edits to be made to pre-filled amounts on an enter details screen rather than any other fields. In some examples, the system only permits edits to a memo field or a numerical amount field. In some instances, the OCR data is used to determine the quality of the check image, so if the OCR data is not correct, then that may be used as an indication that the check image quality is not sufficiently high. However, there may be instances where the check image is quality is sufficiently high for processing, but the OCR data is nonetheless incorrect. For example, the handwriting may be poor such that the OCR produces an output that is incorrect. If the OCR data does not match the check image, then the check image may be flagged for manual processing.

FIG.4cillustrates another example user interface400cof a remote check deposit system for presenting check data. The user interface400cis presented on a display device108of a mobile device104operated by a user U. The user interface400cpresents a check representation image406cin the form of a check illustration having some OCR data filled in relevant locations. The user interface400cfurther indicates that certain OCR data is lacking: check amount, routing number, account number, and check number. Those elements are identified on the check representation image406cwith indications430c. In the illustrated example, the routing number, account number, and check number are filled in with dummy data (e.g., numbers that do not correspond to areal routing or accounting numbers), which can be beneficial in giving a user a sense of what the information to capture looks like. By contrast the check amount is not filled in in the check representation image. Because the user of the system may have a strong reaction to the actual check amount they see on the real check differing from the amount shown in the check representation image, it may be beneficial to leave the check amount empty rather than filled with dummy data. In an alternative example, the numeric check amount may be filled with data from the textual check amount, if such can be detected. In this illustrated example, a selectable element432cis provided that, upon actuation discards the previously obtained check image and permits the user U to take another image.

FIG.4dillustrates a further example user interface400dof a remote check deposit system for presenting check data. The user interface400dis presented on a display device108of a mobile device104operated by a user U. The user interface400dpresents a check representation image406din the form of a check illustration having no OCR data filled in. Rather, fields are represented as illustrated blocks or other shapes lacking alphanumeric symbols. The user interface400dfurther indicates that certain OCR data is lacking: check amount, routing number, account number, and check number. Those elements are identified on the check representation image406dwith indications430din the form of boxes drawn around the blocks corresponding to the fields lacking data. In other instances, the blocks corresponding to the fields lacking data can be highlighted with color or content indicating that the block is lacking detectable data. In this illustrated example, a selectable element432dis provided that, upon actuation discards the previously obtained check image and permits the user U to take another image.

FIG.4eillustrates a further example user interface400eof a remote check deposit system for presenting check data. The user interface400eis presented on a display device108of a mobile device104operated by a user U. The user interface400epresents a check representation image406ein the form of a check illustration having no OCR data filled in. The user interface400efurther indicates that certain OCR data is lacking by placing an indication directly over and covering those portions of the check that are not usefully detectable. In this illustrated example, a selectable element432eis provided that, upon actuation discards the previously obtained check image and permits the user U to take another image.

FIG.5aillustrates an example user interface500aof a remote check deposit system for presenting check data. The user interface500ais presented on a display device108of a mobile device104operated by a user U. In examples, the OCR data generated from the check image is presented on the user interface500ain a list format. The list format includes a listing of check fields506a, such as a listing of required check fields needed to process a check for remote deposit, along with the OCR data508afor each check field. If any required check data is lacking, an indication of the lacking data is presented on the user interface500a. For example, if the check amount could not be read from the check image during OCR processing, there will be no OCR data corresponding to the check amount. Thus, this field in the list format510apresenting the OCR data can highlighted, an error message can be presented, or some other type of indication is presented indicating that required check data is lacking.

In some embodiments, the user interface500afurther includes a selectable element532aconfigured to receive user input to edit the OCR data. In some embodiments, the OCR data displayed on the user interface500ais configured to be selected, such that the user U can select OCR data to edit by tapping on the OCR data. The OCR data can be edited to correct a mistake in the OCR data or to add lacking required check data. In examples, both a check representation image and a listing of the OCR data are displayed on a user interface.

FIG.5billustrates an example user interface500bof a remote check deposit system for presenting check data. The user interface500bis presented on a display device108of a mobile device104operated by a user U. In examples, the OCR data generated from the check image is presented on the user interface500bin a list format, wherein the list format includes check fields506band the OCR data508bcorresponding to the check fields. Example user interface500billustrates user interface500adiscussed with reference toFIG.5aafter user U has edited the OCR data to add lacking check amount data510b.

FIG.6illustrates an example method600for remote check deposit using a mobile device104. In some embodiments, the method600is performed by a mobile banking application executed on a mobile device104, such as mobile device104and mobile application120described above with reference toFIG.1. The method600begins by receiving, at the mobile application executed on the mobile device104, a check image captured by an image capture device of the mobile device104in response to receiving a user actuation causing the image capture device to capture the check image at operation602. In examples, receiving a check image in operation602includes receiving a check image of a front side of the check and a check image of the back side of the check. The capturing of a check image is described in more detail above with respect toFIG.2.

At operation604, the mobile application causes the mobile device104to perform OCR on the received check image to generate OCR data, such as is described above in operation304.

At operation606, the OCR data is verified locally at the mobile device104during on-device check image verification processing. The on-device check image verification processing confirms that the OCR data generated from the check image includes required check data necessary for processing the check for remote deposit. This can be performed in a manner at least similar to that described above in relation to operation308.

At operation608, in addition to verification of the OCR data, the mobile application executed on the mobile device104can verify that a signature is present on the check image, such as is described above in relation to operation308.

At operation610, the OCR data is provided to a financial institution server124for validation processing at operation610. The OCR data is provided to the financial institution server124at operation610without providing the check image to the financial institution server124. The validation processing at operation610is a first level of validation processing by the financial institution server124based only the received OCR data. If the OCR data has been edited by the user, either to correct the OCR data or to enter missing required check data, the OCR data is provided to the financial institution server124at operation610along with instructions for manual review of the OCR data. The providing the data and the verification processing can be as described above.

In response to receiving a confirmation notification from the financial institution server124at the mobile device104that the OCR data has been validated, a request for confirmation is presented on the user interface of the mobile device104at operation612. The request for confirmation requests that a user either confirm or cancel the remote check deposit process. If no confirmation is received confirming that the check should continue to be processed for remote deposit (e.g., input is received via the user interface to cancel the remote deposit), then the remote deposit process will be canceled, and the check will not be remotely deposited. If confirmation is received confirming that that check should be processed for remote deposit (e.g., input is received via the user interface to confirm remote deposit), then the check will be processed for remote deposit.

At operation614, in response to receiving a confirmation via the user interface to process the check for remote deposit, the check image is provided to the financial institution server124with instructions to process the check for remote deposit. In some embodiments, the check image is provided to the financial institution server124at operation614in a different transmission file than the OCR data that was provided to the financial institution server124at operation610.

The financial institution server124then continues processing the check for deposit using the check image. The financial institution performs additional verification on the check image, such as additional fraud detection verification, which can include analyzing handwriting, ink color, date style, signature, endorsement signature, etc. to detect if the check has been modified after it was drafted by the payor. After additional verification of the check image and OCR data by the financial institution server124, a memo is posted on the user's bank account showing the check deposit amount. The financial institution server124then writes a log record of the check to capture the details of check being processed for remote check deposit.

When the financial institution server124has completed the remote check deposit process, the mobile device104receives a deposit receipt notification from the financial institution server124at operation616. In some embodiments, the deposit receipt notification is a push notification presented on the user interface of the mobile device104. In addition or instead, an email or instant message notification can be sent as well as posted transaction appears in activity history

FIG.7illustrates an example method700for remote check deposit using a mobile device104. The method700is performed by a mobile application executing on a mobile device104, such as mobile device104and mobile application120described above with reference toFIG.1. In examples during remote check deposit, after a check image has been received and OCR has been performed on the check image, the OCR data is presented on the user interface of the mobile device702. In some embodiments, the OCR data is presented on a check representation image, wherein the check representation image is an illustration representing the check image and not the check image itself. Example user interfaces presenting OCR data on a check representation image are described in greater detail with respect toFIGS.4a-4e. In some embodiments, the OCR data is presented in a list format. In some examples, the OCR data is not user editable. In other examples, the OCR data is editable. Example user interfaces presenting OCR data in a list format are described in greater detail with respect toFIG.5aandFIG.5b.

During on-device check image verification processing, when verifying that the OCR data generated from the check image includes required check data to process the check for remote deposit, in some embodiments, the mobile application executed on the mobile device determines that the OCR data generated from the check image does not include the required check data to process the check for remote deposit at operation704. This determination is made by comparing the generated OCR data to predetermined required check data fields (e.g., check amount, account number, routing number). For example, if a check amount is a required check data field but no check amount data is included in the OCR data generated from the check image, then the mobile application determines at operation704that the OCR data does not include all the required check data to process the check for remote deposit.

At operation,706, based on the determination that the OCR data generated from the check image does not include the required predetermined check data to process the check for remote deposit at operation704, an indication is provided on the user interface of a region of the check image or a check field that is lacking the required check data to process the check for remote deposit. For example, a region on a check representation image lacking required check data can be circled or highlighted, or a listing of OCR data can show an error where a check field is lacking data. Example user interfaces displaying an indication of a region or check field lacking required check data are shown inFIGS.4b-4candFIG.5b.

In some embodiments, the lacking check data is remediated by receiving another check image that includes the lacking data, or by receiving edits to the OCR to input the lacking data. At operation708, the image capture device of the mobile device104is activated with instructions to capture second check image of check, wherein the second image includes the lacking data. At operation710, user input is received to input the lacking required check data. For example, if the check amount is lacking from the data, input is received caused by the user manually entering in the check amount written on the check. If user input is received to enter the lacking required check data at operation710, the mobile device104repeats the verification step (operation606) described with reference toFIG.6to verify that the OCR data includes the required check data to process the check for remote deposit. If the OCR data is verified, the remote check deposit process can continue with providing the OCR data to a financial institution server124for validation processing, presenting a request for confirmation from a user to process the check for remote deposit, providing the check image to the financial institution server124with instructions to process the check for remote deposit, and receiving, at the mobile device104, a deposit receipt notification from the financial institution server124after the check is deposited.

Example User Interface Workflow

FIGS.8-16illustrate example user interfaces on a mobile device104running an example implementation of the mobile application120that implements a modified version of method300.

FIG.8illustrates a mobile device104running an example implementation of the mobile application120showing a first user interface800. The first user interface800is arrived at after receiving an indication that the user wants to deposit a check. The first user interface800asks the user to select where the user wants to deposit a check. The first user interface800includes a set of user-actuatable user interface elements802that correspond to accounts of the user. Following actuation of one of the user-actuatable user interface elements802, the mobile device104can display the second user interface900as shown inFIG.9.

FIG.9illustrates the mobile device104running the example implementation of the mobile application120showing a second user interface900. The second user interface900includes advice to the user for capturing a check image and has user actuatable button902to open a camera of the mobile device104for obtaining the check image. In some examples, user consent to access the camera of the mobile device104is obtained. Following actuation of the user actuatable button902, the mobile device104displays the third user interface1000as shown inFIG.10.

FIG.10illustrates the mobile device104running the example implementation of the mobile application120showing a third user interface1000. The third user interface1000includes a live preview1002of the camera viewfinder, a capture button210, and instructions1004to the user to take a picture of a front of the check102. advice to the user for capturing a check image and has user actuatable button to open a camera of the mobile device104for obtaining the check image. Following actuation of the capture button210, the mobile device104may execute operations302and304as described above. Following actuation of the capture button210, the mobile device104displays the fourth user interface1100as shown inFIG.11.

FIG.11illustrates the mobile device104running the example implementation of the mobile application120showing a fourth user interface1100. The fourth user interface1100includes a prompt1102indicating that the check image of the front of the check102was successfully obtained. The user may choose a continue button1104or a retake button1106. Behind the prompt, the live preview1002of the camera viewfinder continues. In other implementations, the live preview1002may be paused or absent until the user presses the continue button1104.

In this instance, the check image of the front of the check was successfully obtained. For instance, as described above, the required check data may be determined to be present (see operation308, above). Further, in this instance, the OCR data was not presented on a check representation image. In other instances, notwithstanding the successful obtaining of the check image, the mobile application120may provide the check representation image with the correct data (e.g., a check representation as shown inFIG.4abut with all required data present) or a tabular representation with the correct data as shown inFIG.5b. The mobile application120may provide the user with an opportunity to edit such data. In other scenarios, however, the image might not have been successfully obtained. For instance, the flow of the method300may follow the “NO” path from operation308to operation312and proceed accordingly.

In response to the user actuating the continue button1104, the mobile application120causes the mobile device104to display the fifth user interface1200as shown inFIG.12.

FIG.12illustrates the mobile device104running the example implementation of the mobile application120showing a fifth user interface1200. The fifth user interface1200includes a live preview1002of the camera viewfinder, a capture button210, and instructions to the user to take a picture of a back of the check102. Following actuation of the capture button210, the mobile device104may execute operations302and304as described above. If the application determines that the resulting check image does not include the required data, then the flow of the method300can move to operation312and proceed accordingly. If the mobile application120determines that resulting check image includes required data, then the mobile device104displays the sixth user interface1300as shown inFIG.13. In some instances, prior to moving to the sixth user interface as shown inFIG.13, a confirmation screen similar to that shown inFIG.11can be shown.

FIG.13illustrates the mobile device104running the example implementation of the mobile application120showing a sixth user interface1300. The sixth user interface1300includes a field1302indicating the amount of the check102(e.g., as determined from the OCR data) and having a deposit button actuatable to initiate deposit of the check. Further, the sixth user interface1300includes indications1304that photos of the front of the check and the back of the check were successfully obtained (e.g., with a green check mark), but lacks a preview of the images and lacks a way for a user to view the images. The sixth user interface1300further includes a field1306in which the user can specify a personal note regarding the check102. The user may actuate the deposit button1308to cause the mobile application120to perform operation318as described above to provide OCR data to the financial institution server for validation processing. Responsive to receiving confirmation that the server validated the OCR data (operation320), the mobile device104displays the seventh user interface1400as shown inFIG.14.

FIG.14illustrates the mobile device104running the example implementation of the mobile application120showing a seventh user interface1400. The seventh user interface1400includes a prompt1402to confirm whether the user is ready to deposit by hitting a confirm deposit button1404or wants to cancel by pressing a cancel button1406. This can correspond to operation324as described above and responsive to failing to receive the confirmation, the flow of the method can move to operation328and responsive to receiving the confirmation via the confirm deposit button1404, the flow can move to operations330and332as shown in the eighth user interface1500as shown inFIG.15.

FIG.15illustrates the mobile device104running the example implementation of the mobile application120showing an eighth user interface1500. The eighth user interface1500includes an indication that the check has been received by the server. The eighth user interface1500further indicates that although the check has been received, the deposit will not be fully available until after it is approved.

Computing System

FIG.16illustrates an example block diagram of a virtual or physical computing system1600. One or more aspects of the computing system1600can be used to implement remote check deposit systems described herein.

In the embodiment shown, the computing system1600includes one or more processors1602, a system memory1608, and a system bus1622that couples the system memory1608to the one or more processors1602.

The one or more processors1602are components that execute instructions, such as instructions that obtain data, process the data, and provide output based on the processing. The one or more processors1602often obtain instructions and data stored in the memory1608. The one or more processors1602can take any of a variety of forms, such as central processing units, graphics processing units, coprocessors, tensor processing units, artificial intelligence accelerators, microcontrollers, microprocessors, application-specific integrated circuits, field programmable gate arrays, other processors, or combinations thereof. Example providers processors1602include INTEL, AMD, QUALCOMM, TEXAS INSTRUMENTS, and APPLE.

The system memory1608includes RAM (Random Access Memory)1610and ROM (Read-Only Memory)1612. The computing system1600further includes a mass storage device1614. The mass storage device1614is able to store software instructions and data, such as those that, when executed by the one or more processors1602cause the one or more processors to perform operations described herein.

The mass storage device1614is connected to the one or more processors1602through a mass storage controller (not shown) connected to the system bus1622. The mass storage device1614and its associated computer-readable data storage media provide non-volatile, non-transitory storage for the computing system1600. Although the description of computer-readable data storage media contained herein refers to a mass storage device, such as a hard disk or solid state disk, it should be appreciated by those skilled in the art that computer-readable data storage media can be any available non-transitory, physical device or article of manufacture from which the central display station can read data and/or instructions.

Computer-readable data storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable software instructions, data structures, program modules or other data. Example types of computer-readable data storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROMs, DVD (Digital Versatile Discs), other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computing system1600.

According to various embodiments described herein, the computing system1600operates in a networked environment using logical connections to remote network devices through the network1620. The network1620is a computer network, such as an enterprise intranet and/or the Internet. The network1620can include a LAN, a Wide Area Network (WAN), the Internet, wireless transmission mediums, wired transmission mediums, other networks, and combinations thereof. In some embodiments, the computing system1600connects to the network1620through a network interface unit1604connected to the system bus1622. It should be appreciated that the network interface unit1604can also be utilized to connect to other types of networks and remote computing systems.

The computing system1600also includes an input/output controller1606for receiving and processing input from a number of other devices, including a touch user interface display screen, or another type of input device. Similarly, in some embodiments, the input/output controller1606provides output to a touch user interface display screen or other type of output device. Examples of interfaces that the input/output controller1606can facilitate interaction with include components that facilitate receiving input from and providing output to something external to the computing system1600, such as visual output components (e.g., displays or lights), audio output components (e.g., speakers), haptic output components (e.g., vibratory components), visual input components (e.g., cameras), auditory input components (e.g., microphones), haptic input components (e.g., touch or vibration sensitive components), motion input components (e.g., mice, gesture controllers, finger trackers, eye trackers, or movement sensors), buttons (e.g., keyboards or mouse buttons), position sensors (e.g., terrestrial or satellite-based position sensors such as those using the Global Positioning System), other input components, or combinations thereof (e.g., a touch sensitive display).

As mentioned briefly above, the mass storage device1614and the RAM1610of the computing system1600can store software instructions and data. The software instructions can include an operating system1618suitable for controlling the operation of the computing system1600. In addition, the memory1608or mass storage device1614can include a basic input/output system that contains the basic routines that help to transfer information between elements within the computing system1600, such as during startup. The mass storage device1614and/or the RAM1610also store software instructions, that when executed by the one or more processors1602, cause one or more of the systems, devices, or components described herein to provide functionality described herein. For example, the mass storage device1614and/or the RAM1610can store software instructions that, when executed by the one or more processors1602, cause the computing system1600to receive and execute managing network access control and build system processes.

The computing system1600can include any of a variety of other components to facilitate performance of operations described herein. Example components include one or more power units (e.g., batteries, capacitors, power harvesters, or power supplies) that provide operational power, one or more busses to provide intra-device communication, one or more cases or housings to encase one or more components, other components, or combinations thereof.

A person of skill in the art, having benefit of this disclosure, may recognize various ways for implementing technology described herein, such as by using any of a variety of programming languages (e.g., a C-family programming language, PYTHON, JAVA, RUST, HASKELL, other languages, or combinations thereof), libraries (e.g., libraries that provide functions for obtaining, processing, and presenting data), compilers, and interpreters to implement aspects described herein. Example libraries include NLTK (Natural Language Toolkit) by Team NLTK (providing natural language functionality), PYTORCH by META (providing machine learning functionality), NUMPY by the NUMPY Developers (providing mathematical functions), and BOOST by the Boost Community (providing various data structures and functions) among others. Operating systems (e.g., WINDOWS, LINUX, MACOS, IOS, and ANDROID) may provide their own libraries or application programming interfaces useful for implementing aspects described herein, including user interfaces and interacting with hardware or software components. Web applications can also be used, such as those implemented using JAVASCRIPT or another language. A person of skill in the art, with the benefit of the disclosure herein, can use programming tools to assist in the creation of software or hardware to achieve techniques described herein. Such tools can include intelligent code completion tools (e.g., INTELLISENSE) and artificial intelligence tools (e.g., GITHUB COPILOT).

One or more techniques described herein can benefit from or be implemented using a machine learning framework. A machine learning framework is a collection of software and data that implements artificial intelligence trained to provide output based on input. Examples of artificial intelligence that can be implemented in a trainable way include neural networks (including recurrent neural networks), language models (including so-called “large language models”), generative models, natural language processing models, adversarial networks, decision trees, Markov models, support vector machines, genetic algorithms, others, or combinations thereof. Machine learning frameworks or components thereof are often built or refined from existing frameworks, such as TENSORFLOW by GOOGLE, INC. or PYTORCH by the PYTORCH community. The machine learning framework β00 can include one or more models that are the structured representation of learning and an interface that supports use of the model.

The model can take any of a variety of forms. In many examples, the model includes representations of nodes (e.g., neural network nodes, decision tree nodes, Markov model nodes, other nodes, or combinations thereof) and connections between nodes (e.g., weighted or unweighted unidirectional or bidirectional connections). In certain implementations, the model β02 can include a representation of memory (e.g., providing long short-term memory functionality). Where the set includes more than one model, the models can be linked, cooperate, or compete to provide output.

The interface can include software procedures (e.g., defined in a library) that facilitate the use of the model, such as by providing a way to interact with the model (e.g., receive and prepare input, processing the input with the model and provide output). The interface can define a vector embedding technique for creating a representation of data usable as input into the model. Example embedding techniques include Word2Vec and BERT. The software can further provide the ability to create, customize, fine tune, and train the model.

In an example implementation, interface can provide a training method that includes initializing a model, obtaining training data, providing a portion of the training data to the model to produce an actual output, comparing the expected output with the actual output, updating the model based on the result of the comparison (e.g., updating weights of the model, such as using backpropagation), continuing providing training data and updating the model until a stopping criterion has been reached, and deploying the trained model for use in production.

While particular uses of the technology have been illustrated and discussed above, the disclosed technology can be used with a variety of data structures and processes in accordance with many examples of the technology. The above discussion is not meant to suggest that the disclosed technology is only suitable for implementation with the data structures shown and described above.

This disclosure described some aspects of the present technology with reference to the accompanying drawings, in which only some of the possible aspects were shown. Other aspects can, however, be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects were provided so that this disclosure was thorough and complete and fully conveyed the scope of the possible aspects to those skilled in the art.

As should be appreciated, the various aspects (e.g., operations, memory arrangements, etc.) described with respect to the figures herein are not intended to limit the technology to the particular aspects described. Accordingly, additional configurations can be used to practice the technology herein and/or some aspects described can be excluded without departing from the methods and systems disclosed herein.

Similarly, where operations of a process are disclosed, those operations are described for purposes of illustrating the present technology and are not intended to limit the disclosure to a particular sequence of operations. For example, the operations can be performed in differing order, two or more operations can be performed concurrently, additional operations can be performed, and disclosed operations can be excluded without departing from the present disclosure. Further, each operation can be accomplished via one or more sub-operations. The disclosed processes can be repeated.

Although specific aspects were described herein, the scope of the technology is not limited to those specific aspects. One skilled in the art will recognize other aspects or improvements that are within the scope of the present technology. Therefore, the specific structure, acts, or media are disclosed only as illustrative aspects. The scope of the technology is defined by the following claims and any equivalents therein.

This application incorporates certain other applications by reference. To the extent that there is a conflict between the material in this specification and the material that is incorporated by reference, the subject matter of this specification (as opposed to the incorporated one) controls for the purposes of resolving that conflict.

Various embodiments are described in detail with reference to the drawings, wherein like reference numerals represent like parts and assemblies throughout the several views. Reference to various embodiments does not limit the scope of the claims attached hereto. Additionally, any examples set forth in this specification are not intended to be limiting and merely set forth some of the many possible embodiments for the appended claims.

Whenever appropriate, terms used in the singular also will include the plural and vice versa. The use of “a” herein means “one or more” unless stated otherwise or where the use of “one or more” is clearly inappropriate. The use of “or” means “and/or” unless stated otherwise. The use of “comprise,” “comprises,” “comprising,” “include,” “includes,” and “including” are interchangeable and not intended to be limiting. The term “such as” also is not intended to be limiting. For example, the term “including” shall mean “including, but not limited to.”