In-scanner document image processing

Various embodiments herein each include at least one of systems, methods, data structures, and software for document image processing within a device such as a retail scanner. One such embodiment, in the form of a method, includes receiving a document image from an imaging device of a scanner and determining a type of the document represented in the document image. The document image may then be processed to obtain document metadata and the document image and metadata representative of the document metadata and document type are added to a data structure. This method may further include adding transaction-related metadata to the data structure. The data structure may then be stored on the scanner, transmitted via local communications link to a host device, transmitted via a network to a database for storage, or transmitted via a network to another system for one or both of processing and storage.

BACKGROUND INFORMATION

During a checkout process in retail or other environments, it is common for customers to present one or more of various document types. Such documents may include checks, identification cards, bankcards such as debit or credit cards, membership cards, library cards, and the like. One or more of these documents may be presented for a variety of purposes ranging from making a payment and validating a customer identity to confirming date of birth and address verification. These documents are typically presented at a Point of Sale (POS) terminal, which may be a cashier-assisted lane, a Self-Service Terminal (SST) or other kiosk terminal that may or may not involve payment. At later times, transactions performed at such terminals may be audited for security, compliance, training, and other purposes. However, only some of the presented documents may be available, such as a presented check or data magnetically read from a bankcard. This may present a security or compliance exposure for an operating entity, especially when an identification card is needed in completing a transaction for compliance with laws and regulations applicable to purchase or otherwise obtaining certain controlled items (e.g., alcohol, tobacco, and pharmaceutical products).

SUMMARY

Various embodiments herein each include at least one of systems, methods, data structures, and software for in-scanner document image processing.

One such embodiment, in the form of a method, includes receiving a document image from an imaging device of a scanner and determining a type of the document of the document image. The document image may then be processed to obtain document metadata and the document image, and metadata representative of the document metadata and document type are added to a data structure. This method may further include adding transaction-related metadata to the data structure. The data structure may then be stored on the scanner, transmitted via local communications link to a host POS terminal or kiosk, transmitted via a network to a database for storage, or transmitted via a network to another system for one or both of processing and storage.

Another embodiment in the form of a method includes scanning each of a plurality of documents presented on a scanning surface of a scanner to obtain document images. This method then determines a type of each document for each document image and processes each document image to obtain a set of respective document metadata. Each document image and each respective set of document metadata including data representative of the respective document type is then added to a data structure and the data structure is stored.

One device embodiment is a scanner. The scanner of such embodiments includes at least one processor, at least one memory device, at least one imaging device, and a communication interface device. The at least one memory device stores instructions that are executable by the at least one processor to perform data processing activities. These data processing activities include receiving a document image from a first imaging device of the at least one imaging devices and determining a type of the document of the document image. The data processing activities further include processing the document image to obtain document metadata and adding the document image and metadata representative of the document metadata and document type to a data structure in the at least one memory device.

DETAILED DESCRIPTION

Various embodiments herein each include at least one of systems, methods, data structures, and software for in-scanner document image processing. In some embodiments, the document image processing is performed within a scanner, such as a product scanner that may be present at a POS terminal or other kiosk terminal including a scanner such as a product scanner.

Some embodiments leverage one or more imaging devices present in a scanner to capture an image of each of one or more documents presented as part of a transaction, such as an identification card, a check, bankcard (e.g., credit card, debit card, prepaid card), membership card, library card, loyalty card, etc.). Each captured image may then be processed on the scanner to identify a document type and to extract or generate certain data from the document included in the image. In some embodiments, the data extracted or generated from the document may then be used for various forms of validation, such as validating a person is at least a certain age, that identification card information matches account holder information on a check or credit card, and the like. In some embodiments, each document image, data of the document, and a record of any validation performed are bundled into a single data structure that may be stored locally to the scanner, provided to a terminal coupled to the scanner, transmitted from the scanner via a network for storage elsewhere such as in a database, or is otherwise stored. The single data structure may also include a token, such as a transaction identifier, a date time stamp, or other data to assist in later retrieval. The single data structure, in some embodiments, is encrypted prior to storing or transmission.

In some such embodiments, the scanner may also be coupled to one or more other imaging devices external to the scanner, such as cameras. The one or more other imaging devices may be coupled to the scanner either physically, via a network, or via a wired or wireless connection. The one or more imaging devices may each be directed in different directions. For example, one imaging device may be directed toward a customer presenting the one or more documents while another imaging device may be directed toward a clerk stations, such as that of a cashier. Images may be captured from the imaging devices and added to the data structure in the scanner to add information to the data structure of the people that participated in the transaction. Such data may be useful to audit a particular transaction to verify the people involved, such as when a transaction is later determined to involve fraud or a violation of law or policy. In some embodiments, the validation performed may include facial recognition processing between a facial image included on a presented document such as a driver's license and a face of a customer captured by an external imaging device.

These and other embodiments are described herein with reference to the figures.

The functions or algorithms described herein are implemented in hardware, software or a combination of software and hardware in one embodiment. The software comprises computer executable instructions stored on computer readable media such as memory or other type of storage devices. Further, described functions may correspond to modules, which may be software, hardware, firmware, or any combination thereof. Multiple functions are performed in one or more modules as desired, and the embodiments described are merely examples. The software is executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a system, such as a personal computer, server, a router, peripheral device, or other device capable of processing data including network interconnection devices.

FIG. 1is a logical block diagram of a system100including a scanner, according to an example embodiment. The system100is an example of deployment in a retail environment, although the various embodiments herein are also applicable to other environments, such as customer service kiosks and library checkouts, travel check-in kiosks, among others.

The system100includes a scanner102coupled to a terminal104. The scanner102may be a product scanner or similar scanning device in some embodiments. An example of such a scanner is the REALSCAN 79 BI-OPTIC IMAGER available from NCR Corporation of Duluth, Ga. Such modern scanners utilize one or more cameras in performing their scanning functions. As such, these cameras may be utilized to capture images of presented documents.

The terminal104may be a modern cash register serviced by a checkout clerk or cashier, or it can be a self-checkout terminal or other Self-Service Terminal (SST), customer service kiosk, and the like. In some embodiments, the scanner102is coupled, via either wired or wireless connections, to one or more additional cameras, such as cameras106,108. However, various embodiments may include no cameras, one camera, and more than two cameras. In some embodiments, the camera106may be pointed in the direction of a customer presenting items to purchase and documents to complete a purchase transaction at the terminal104. Similarly, the camera108may be pointed in the direction of a terminal104clerk.

The system100also typically includes a network110that connects one or both of the scanner102and the terminal104to other computing devices. The other computing devices may include one or more transaction processing systems114and one or more databases112that may store transaction, product, and other related data, depending on the particular embodiment.

As products are presented for purchase, the products may be scanned by the scanner102, which then provides a product identifier, such as a Universal Product Code (UPC) to the terminal104. When all products have been presented, the customer then typically pays for presented items by tendering a payment. When the payment is tendered, one or more documents may be provided to complete the transaction. The documents to be presented may depend in part on the products being purchased. In some embodiments, the terminal104may send a signal to the scanner102indicating a document is to be scanned and may identify the particular document type. The document may then be presented to the scanner102and the scanner102captures an image thereof either via a product scanning camera included in the scanner102or an external camera such as one of cameras106and108].

For example, when a controlled item is being purchased, such as tobacco, alcohol, certain pharmaceutical products, and the like, an identification card such as a driver's license may be required to verify the age or identity of the customer. Similarly, when certain forms of payment are provided, the identity of the customer may also or alternatively need to be verified. In such circumstances where a document is required or otherwise provided, the scanner102may be utilized to capture an image of the presented document. While a person may view a document and verify the data contained thereon, capturing an image of the presented document(s) provides an auditable record to aid in legal and policy compliance and monitoring, fraud investigation, and increased transactional security. Further, captured document images may be processed to identify a type of document presented and to perform document-type specific processing and validation, as well as validation between imaged documents and even identity validation between images on identification cards and a customer image as may be captured by a customer-facing camera106, when present.

When the scanner102scans and processes a document, data may be transmitted by the scanner102to the terminal104in some embodiments. For example, when the terminal104has sent a message to the scanner102to scan an identification card, the scanner102may capture an image of a presented identification card, process the image to validate an age or date of birth, and pass a validating signal back to the terminal104indicating validation success or failure. In some embodiments, the captured images, data captured from and validated with regard to images and documents included therein, is maintained securely within the scanner. In such embodiments, the images and data are not shared by the scanner102with the terminal104. Instead, when such data is needed, it typically is not the raw data. Instead, the terminal104may only need to know whether the age of the customer is sufficient to purchase controlled items. In some other embodiments, only certain data may be shared, such as a name and address of the customer as may be required by law when the customer is purchasing a tracked item, such as certain pharmaceutical products that may be utilized in the manufacture of illegal drugs.

As generally described above, as documents images and images from one or more of the cameras are acquired, the images, in some embodiments, are processed to extract or recognize data therein, which forms metadata related to the respective documents. Validation may be performed against that data and between metadata and various images. The images, metadata, and records of the performed validation activities may then be bundled into one or more data structures and stored on the scanner102that performs the processing. The one or more data structures may also, or alternatively, be transmitted to the host terminal104over a local communications link, or over the network110to one or both of the database112and the transaction processing system114. The one or more data structures, prior to transmission over the network110, may be encrypted, in whole or in part. This data may then be utilized at a later time to audit or review activities performed at the terminal, such as to verify an identification card was checked before closing the transaction.

FIG. 2is a diagram illustrating components of a scanner208that may be deployed at a checkout station such as a POS terminal, self-service terminal (SST), or other kiosk system including a scanner208, according to an example embodiment. It is to be noted that the scanner208is shown schematically in greatly simplified form, with example components relevant to understanding various embodiments herein. Note that the scanner208may include more or fewer components in some embodiments.

Furthermore, the various components included in theFIG. 2as illustrated and arranged are provided for illustration purposes only. It is to be noted that other arrangements with more or fewer components are possible without departing from the contributions herein, in particular with regard to in-scanner document image processing.

Moreover, the methods and scanner208presented herein and below may include all or some combination of the components described and shown in the various contexts herein. Further, although the scanner208may be paired with a POS terminal, the scanner208may be a standalone element or an element of other systems, devices, and terminals in other embodiments. Examples of other terminal-types that may include a scanner208are SSTs, clerk operated and self-service library checkout stations, time-keeping terminals, and the like. Additionally, although a scanner is illustrated inFIG. 2according to some embodiments, the in-scanner document image processing described herein may also be implemented with regard to images and video streams generated by other devices, such as stand-alone cameras, such as one or more of cameras240,242that may be coupled either directly or indirectly via a network to the scanner208.

The methods of some embodiments are programmed as executable instructions in memory and/or non-transitory computer-readable storage media and executed on one or more processors associated with the components and devices herein. For example, some embodiments may be deployed as firmware present within the scanner208, such as firmware226. In other embodiments, the in-scanner document image processing functions may be deployed as software on the scanner or network deployed service accessible by the scanner.

The scanner208may be referred to herein as a product scanner or barcode scanner as that is the task most commonly associated with such devices. During operation, items are placed within a scan field of the scanner208. One or more imaging devices218of the scanner208, such as one or more cameras, then scan a barcode and information read therefrom is communicated to a POS system. The POS system then uses that data to identify the item placed within the scan field of the scanner208and performs an additional function. The additional function may include a price lookup and addition of the item to a list of items to be purchased, which may be presented on the one or more POS displays. The additional function may also include identifying a scanned product as a controlled item for which an age, identity, and/or address validation must be performed. A message may be presented on the one or more POS displays and a message may also be sent to the scanner208indicating an identification card or other document is to be scanned.

The scanner208may include one or more scan fields, such as two scan fields of bi-optic scanners that are commonly seen in grocery and discount retail outlets. In addition to the imaging devices218, the scanner208may include various other components. The various other components may include an integrated scale210such as may be used in a grocery outlet to weigh produce and one or both of a speaker212and display lighting216to output audio a visual signals such as signals of (un)successful scans. The scanner208may also include scan field lighting220that may be turned on and off and adjusted based on a detected presence of an item to be scanned.

During typical operation, the scanner208is operated according to instructions executed on a processor222. The processor222may be an application specific integrated circuit (ASIC), digital signal processor, microprocessor, or other type of processor. The instructions may be firmware226or software230stored in one or more memories224. The one or more memories224may be or include volatile and non-volatile memories, write-protected memories, write-once memories, random access memory (RAM), read only memories (ROM), secure memory, and other memory and data storage types and devices.

The instructions as may be stored in firmware226or as software230in memory224are executed according configuration settings stored in the memory224. The configuration settings228configure operation of the scanner208and the various components therein. For example, the configuration settings208may configure speaker212volume, display lighting216outputs, and scan field lighting220brightness. The configuration settings may also configure decoding algorithms of the imaging device(s)218and the instructions, one or more communication protocols used to communicate data from the scanner208to a POS system or other system via wired or wireless connections, and scale210operating parameters (e.g., unit of measure as pounds or kilograms). The configuration settings may also configure other settings of the particular scanner208an embodiment may include. In some embodiments, the configuration settings228may include a firmware version, a software version, and the like. Thus, when a configuration is set or updated, the setting or updating of the configuration settings228may include population and updates of any of the configuration settings228of a particular embodiment, including an update to firmware and software present on the scanner.

The scanner208may include one or more communication interfaces214,232that enable the scanner208to communicate via one or both of wired and wireless connections over a network and with other computing devices. In some embodiments, the communication interface232may provide a virtualized communication connection on the scanner208that enables the scanner to communicate over a network leveraging network connectivity of a terminal or other computing device with which the scanner is paired, such as a POS terminal.

During operation, the one or more imaging devices218of the scanner208capture images, such as still images and frames of video. These still images and video may be stored within the memory224or other data storage device of the scanner208, a terminal or other device paired therewith, or transmitted via one of the communication interface214,232over a network. In other embodiments, the video is simply captured and processed on the scanner208and discarded. However, in some circumstances, certain images are needed, such as images of documents presented for payment, identity verification, and the like. In such instances, these images are processed according to instructions stored in software230, firmware226, or by passing such images to a network service via one of the communication interface214,232for appropriate processing.

For example, such processing in some embodiments includes receiving a document image from an imaging device218of the scanner208and determining a type of the document of the document image.

To identify a document-type of a document included in a document image, document-type profiles are defined. A document-type profile231is stored in the memory224and includes data defining image features or parameters utilized in identifying a document-type. A document-type item profile may further include data defining processing parameters of actions to be performed by an image processing function or module with regard to an image once a particular document-type is identified.

The document-type defining data generally includes parameters used to define the document-type for an image processing function. The parameter may define shapes, text, symbols, alphanumeric patterns, relative proportions of two or more features, colors, and the like. For example, bankcards, such as credit and debit cards, have a standard shape and typically include card network logos and numbers presented in certain patterns, such as fifteen or sixteen numbers of a bankcard number. Further, driver's licenses and other government issued identification cards include certain patterns of information presentation. Similarly, checks include standardized features such as account and bank routing numbers located in certain locations and in a certain font.

The processing parameters of a document-type profile generally include one or more instructions that instruct the image processing function or module with regard to one or more actions that are to be performed on an image including a document of a particular document-type. Such actions may include recognizing and reading text or numbers from certain locations in the document image and/or obscuring a portion of an image including certain portions of sensitive information, such as an address, bankcard number, account numbers and a signature on a check, and the like. With regard to certain document-types, portions of a document image may also be extracted, such as an image of a person from a driver's license. The obscuring may include removing portions of the image including the sensitive information, changing a color of a portion of the image to all the same color such as black or grey, encrypting the image, and even removing the image from a still image or an image stream of video. Thus, in some embodiments, document content may not only be read and extracted, but certain portions of a document may also be obscured to increase security of sensitive data.

The processing of the image to determine the document type and the subsequent processing of the document image is performed to obtain data from which document metadata may be generated. The processing of the document image further includes adding the document image and metadata representative of the document metadata and document type to a data structure in the memory224. In some embodiments, transaction related metadata is also added to the data structure, such as data that may be received from a terminal such as a transaction identifier or a date time stamp and a scanner208or terminal identifier.

FIG. 3illustrates a document that may be presented according to an example embodiment. The document illustrated inFIG. 3is a check302. The check302may be imaged by a scanner, such as scanner102ofFIG. 1or scanner208ofFIG. 2. An image of the check may first be processed in view of a plurality of document-type profiles, such as document type profiles231ofFIG. 2. Each of the document-type profiles includes data defining image features that define properties of the particular document-type and processing parameters of actions to be performed by an image processing function or module upon identification of the respective document type.

A document-type profile for identifying checks, such as check302, may include data defining features of checks. Such features may include a number pattern304along a long-edge of a rectangular document and text-pattern of a name and address in an upper left-hand portion306of the document. Another feature may be a rectangular box308within which a number is present.

With regard to other document types, such as a driver's license, other features may be defined such as an image of a person in a particular area and various text patterns. For a document type of a bankcard, a fifteen or sixteen digit number or a month/year date field may be defined features.

Thus, once an image of a document is obtained by a scanner, an image processing function attempts to identify the document type in view of one or more document-type profiles. Once the document-type is identified according to a document-type profile, processing parameters of the respective profile may be executed against the image. The processing parameters may execute to obtain specific data from the image, such as by performing an optical character recognition function against specific document image portions or an entirety of document image. Certain data items may then be the subject of additional processing, such as with regard to the check302, data read from the check may be submitted via a network to electronically process a payment with regard to the check302amount and the account from which the check302is drawn. Another example is when the document-type is a driver's license that includes an image of the person to whom the driver's license is issued. In some embodiments, the image of the person may be extracted from the document image, the scanner may capture an image from a camera coupled to the scanner directed toward the customer. The extracted image and the captured image may then be processed according to a facial recognition algorithm to determine if the images match to verify the person presenting the driver's license is the person to whom the driver's license was issued. Other document-types may be subject to similar or different processing parameters.

FIG. 4is a block flow diagram of a method400, according to an example embodiment. The method400is an example of a method that may be performed by a scanner in performing in-scanner document image processing according to some embodiment.

The method400includes receiving402a command from a host system, such as POS terminal, to scan one or more documents. The scanner then scans404the document to obtain a document image. The document type is identified406in view of one or more document-type profiles and processed408in accordance therewith to obtain document metadata. The document image and metadata are then added410to a data structure. When412there are more documents to scan, the method again scans404a document and proceeds from there to add410the document and metadata therefrom to the data structure. When412there are no more documents to scan, the method400proceeds in view of configuration settings, document-type profile processing parameters, and details of the particular embodiment to perform optional functions or to simply store422the data structure.

The optional functions may include one or more of capturing414one or more images from one or more other cameras that may be coupled to scanner to obtain images of one or more of a customer, a cashier or other clerk, and the like. The optional functions may also include performing416validation of one or more documents from document images and captured images, which may also be performed in view of one or more rules associated with one or more products the customer is attempting to purchase, such as controlled products as described above. The optional processing may further include adding418images and validation metadata to the data structure and encrypting420at least a portion of the data structure prior to storing422the data structure.

In some embodiments, storing422the data structure includes transmitting the data to a database for storage, to a transaction processing system or other system associated with storing transaction related data.

Another embodiment in the form of a method includes receiving a document image from an imaging device of a scanner and determining a type of the document of the document image. The document image may then be processed to obtain document metadata and the document image and metadata representative of the document metadata and document type are added to a data structure. This method may further include adding transaction-related metadata to the data structure. The data structure may then be stored on the scanner, transmitted via a network to a database for storage, or transmitted via a network to another system for one or both of processing and storage.

In some embodiments, this method includes receiving a second image from a second imaging device of or coupled to the scanner and adding the second image to the data structure. In such embodiments, when the determined type of the document of the document image is an identification document including a picture of an individual, such as a driver's license, the processing of the document image to obtain document metadata includes extracting the picture. The method in such embodiments may then proceed by comparing the extracted picture to an individual present in the second image to verify the individual present in the second image is the same individual represented in the extracted picture. When the comparing identifies a match, the method may store data representative of the comparing match to the data structure. However, when the comparing fails to identify a match, the method may output a signal from the scanner indicating the failure to identify a match.

A further embodiment in the form of a method includes scanning each of a plurality of documents presented on a scanning surface of a scanner to obtain document images. This method then determines a type of each document of each document image and processes each document image to obtain a set of respective document metadata. Each document image and each respective set of document metadata including data representative of the respective document type is then added to a data structure and the data structure is stored.