Patent ID: 12248522

DETAILED DESCRIPTION

The present invention is a system and method for creating a digital and searchable inventory of physically stored, items, objects, or documents. Methods and apparatus of the present disclosure may image physical items as a person removes them from a container. This person may also provide verbal descriptions of items being imaged. Artificial intelligence, such as object recognition, optical character recognition, or natural language processing, may then applied to the captured data to classify the items into a searchable hierarchy. The digital classification of the items and their attributes may be linked to a machine-readable ID, such as a barcode or other identifier, that may allow the items scanned to be associated with a specific physical storage device. This may include using identifiers that were originally on a container and by associating new identifiers with the container. In certain instances, users may only image portions of individual documents such that the documents can be quickly characterized.

FIG.1illustrates a rapid box indexing system. This system100may be include computing device105, sensors & input/output devices130of a capture station, and a number of boxes170. Computing device105includes memory110, processor115, input/output communication interfaces120, and data capture & classification databases125. The instructions stored in memory110may include instructions that when executed allow a processor to capture data and classify that captured data. Receptacles or boxes170may be constructed of any material, such as plastic, fiberboard, cardboard etc. Here boxes170include contents175that need to be classified or characterized. A box may be any form of receptacle, such as a container, a file folder, a crate, a jar, a tank, a tub, or a package, essentially any receptacle that is capable of holding items or documents. Capture station130includes sensors & input/output (I/O) devices. These sensors or I/O devices may include a camera135, a barcode reader140, a microphone145, a radio frequency identifier (RFID) reader145, a scale155, a magnetometer165, and output devices165. Computer105may be communicatively coupled to a capture station130via a direct network connection, such as Ethernet, a wireless connection, such as Wi-Fi, or integrated into one or more of the hardware pieces at the capture station130. Boxes170may be part of a project in which the owners of the boxes170are legally required to maintain records of contents included in specific boxes. The term “scan” may include single or multiple forms of imaging, such as optical, acoustic, laser scanning, radar, thermal, and seismic-based imaging. A data capture database of databases125may store data collected at the capture station135. Data from the capture station130may include video data from one or more cameras135. Barcode reader140may be used to collect box identifying information, audio data may be captured by one or more microphones145, and data from an RFID chip may be captured by an RFID reader150. The data capture database may be populated with data collected from based on execution of instructions of a capture software module. Boxes may include or be associated with more than one identifier. For example, the box identifiers may be a series of stickers with a barcode printed on them or they may include text written on a box. The data capture database may store a table of data for each box associated with a project.

When a box identifier is scanned using the barcode reader140, all data captured by the camera135, barcode reader140, microphone145, RFID reader150, scale155and magnetometer160, may be stored in databases125and this data may be associated with a particular box identifier. A classification database of databases125may store data after that data has been classified.

Instructions of a data capture software module may allow a processor to collect data from the capture station130and associate that data with a box identifier and then the processor115may store that associated data in a data capture database of databases125. The first scan of a given box identifier by the barcode reader140may begin the association of content175with a specific box170. A second scan of the same box identifier may indicate that a particular box170has had all of its contents175inventoried (i.e. scanned and classified or characterized). This process may include the processor115of computing device105processing the data in the data capture database, when a digital searchable inventory of a set of boxes and their contents are created.

In one instance, optical character recognition (OCR) may be used to identify written words on documents, objects, or in images. Speech to text translation of a verbal description of the contents175of box170that were uttered by a user may be captured by microphone145. An RFID or other similar near field communications devices may have their data ingested by the RFID reader150. Scale155may be used to record a weight of box170. Obtaining a box weight may be used to determining how heavy box170is at start point and at an end point of a data collection and classification process. An initial box weight may be used to identify whether a box is empty before it has been opened by a user or may be used to validate that a box has been completely emptied. In certain instances, a magnetometer160may be used to record the magnetic field of the contents of a box. Magnetometer data may be used to identify whether magnetic material is included in box170.

Artificial intelligence (AI) may be applied to an object recognition process on any image of an object. As such, AI may be used to identify some of the contents175included in box170. Natural language processing (NPL) may be applied to text identified through OCR to determine a sentiment or tone to associate with contents of a box. Such sentiments may be related to a positive sentiment, a negative sentiment, or a neutral sentiment of a given item.

A hierarchical structure may be applied to the collected data. For example, a name and a mailing address may be interpreted as the name being at the top of a hierarchy and a street address may be designated as the ‘child’ of the ‘parent’ name. In another example, a hierarchy may include three levels, with the addressee being the first level or parent, the street address being a second level or child, and the city, state, and postal code, may be a third tier or grandchild. Text, objects, or images may be identified as having a designated place in the hierarchy. For example, a law firm may have their client list input as having a predefined place in the hierarchy. They may be automatically assigned to the second tier in the hierarchy below the law firm partner who handles their cases. Some or all of the classifications made by processor115executing instructions of a data classification software module may have a confidence level associated with them. For example, an object recognition process may identify an elephant in a picture and indicate that it is 92% confident in that identification.

Capture station130may allow a user to scan the contents175of a number of boxes170such that captured data may be stored in data capture database of databases125. A data capture station may include, at least one camera114, barcode reader116, microphone118, and RFID reader120. In certain instances, a full scanning area of capture station130may be visible to a field of view of camera153. The user may place the box170on one side of the scanning area within capture station130, scan a box identifier of the present box170, and begin placing the contents175in the scanning area one at a time. As each item in box170are passed into the field of view of camera135, a current document or object being viewed may be described verbally by a user while the object is under the camera135. Once all contents175have been scanned, they may be returned to the present box170. The user may then scan the box identifier for the present box170a second time to indicate that data collection for the present box170has been completed.

Camera135may be the rear-facing camera on a tablet, smartphone, or other computing device. Barcode reader140or other optical scanners may be capable of ingesting data represented in a visual, machine-readable form such as a barcode, QR code, or universal product code (UPC). In one instance, barcode reader140may be a wireless handheld scanner. Microphone145may be integrated into the tablet or computer that contains the camera135. Output devices165may be a display or a speaker that may provide commands to a user. Such a command may include instructing the user to scan (show to the camera) additional portions of a particular document. For example, in an instance when processor115identifies that a particular document (e.g. a file folder, notebook, or report) is of a type of interest, the user may be asked to scan a table of contents or some other portion of the document. This additional information may allow the processor to better classify a document.

Table 1 illustrates data that may be stored data capture database. Each data capture project may begin with the creation of a number of box identifiers. In one instance, the box identifiers are barcodes that may include or encode a name assigned by the box owner and an identifier associated with that box identifier. The identifier may vary in style or number of digits but is designed to uniquely identify a specific box. The box identifiers may each have or be associated with a table that may populate the contents of the box with. The box identifier may be printed on a sticker that is then affixed to the box. The data included in the capture database may cross-reference various types of information. This process may associate video data captured by a camera with a box identifier.

Data associated with a box, or its contents, may be captured by a microphone, RFID reader, barcode reader, scale, or magnetometer. Each data point captured may have a timestamp associated with it so a processor may associate data captured with a corresponding time in a set of video data. This may allow a user to capture data with the barcode reader before they scan a box identifier. In one instance, the processor may store some number of barcodes captured in a blank table in a data capture database. A scale may be used twice at a start point and at an end point of a data collection and classification process.

Once a box identifier has been scanned, all barcode data may be stored in association with the scanned box identifier. In certain instances, descriptive data may be captured from a microphone or an RFID reader before the box identifier is scanned. Data captured after the box identifier is scanned may have a timestamp associated with it. An initial scanning of a box identifier may prompt the processor to initiate recording video data associated with the present box identifier. Data captured before the box identifier is scanned may list a sequence of identifiers in a “queue” entries in place of a timestamp. Note that table 1 cross-references a type of data (video, audio, RFID, barcode, weight, magnetic field) with a type of data (MP3, MP4, RFID data, bar code data, weight data, or magnetic field strength data).

TABLE 1Data Capture Database DataXYZ Corp - Box 12345QueueTIMEVideoBOX12345.mp4AudioAudio 1.mp3Audio 2.mp3Audio 3.mp3Audio 4.mp3Audio 5.mp3Audio 6.mp3Audio 7.mp3RFIDRFID1.datBarcodesBC1.datBC2.datBC3.datWeightWeight.datMagnetic FieldFieldStrength.dat

Table 2 illustrates data that may be stored in a database that may include classification data. This classified data may be stored as part of a classification process. In one instance a database may store a set of tabulated data that associates a box identifier with each object included in a box. All of the objects that make up the contents of a box may have a unique identifier assigned to them. In one instance, an object identifier may be a combination of the box identifier, a timestamp in a video indicating a time when an object was identified. An index of information related to an object may be generated. For example, OCR may identify words on the page.

In certain instances, AI may be applied to identify objects or images. For example, an image may contain a commonly identifiable object, such as two dogs. A processor executing instruction of a set of AI software may identify that there are two dogs in the picture, that one is a husky, and the other a dalmatian. This processor may also identify that there is snow in the background behind the dogs. In some instances, AI software may be trained to recognize objects and images specific to a field of art. For example, an image recognition system may be trained to identify different valve types symbols that relate to oil and gas application. The sentiment, such as positive, negative, or neutral, of an object may be indicated in a set of classification data.

Text gathered from execution of an OCR process or a speech to text recognition process, may be examined for sentiment. Facial expressions of a person in an image may be examined for sentiment, such as happy or sad. Furthermore, an audio description provided by the user may be stored as text. Each piece of information characterized may have a confidence level associated with its identification. Each piece of information in an index of information may have an identifier associated with it.

TABLE 2Classified Database DataXYZ Corp-Box 12345Object IDXYZ12345_0_0.12728770804918646Object typeCommercial_itemSentimentNeutralUser descriptionGEM-89-137 5 I Tab Products Co Spacefinder Systems Model 1189-5Object indexconfidenceidTypeparent_idlevel_0OUT 1614O8 OUT86.190078740LINEem76.1170654310WORD1GEM-89-13794.5681991611WORD20Tab Products Co Spacefinder Systems95.386413573LINE3Model 1189-50% 186.489234924LINEWhite. 2 Blue. Green 4 Gray Canary92.314407355LINE6 Orange Pink, 8 Red Lavender Goldenrod91.672111516LINEOUT97.804260259WORD01614O866.186531078WORD0Tab95.6234130912WORD3Products95.6998977713WORD3Co92.6742858914WORD3Spacefinder95.5237426815WORD3Systems97.4107055716WORD3Model95.1530685417WORD41189-50% 177.8254089418WORD4White. 286.1060104419WORD5Blue.92.8930816720WORD5Green94.247352621WORD54 Gray92.6896057122WORD5Canary95.6359863323WORD56 Orange96.1198654224WORD6Pink,90.4617309625WORD6890.8123779326WORD6Red88.9364471427WORD6Lavender92.6462783828WORD6Goldenrod91.0559539829WORD651I2Model 1189-5(Å°)4(*) 1 White 2 Blue, 3 Green, 4 Gray 5 Canary56 Orange. 7 Pink, 8 Red. 9 Lavender, o Goldenrod6

FIG.2illustrates a series of steps that may be performed when data included in a receptacle is collected and classified. First, in step205bar code data or other data unique to the receptacle are collected. This may include the scanning scan of a barcode or other identifier (e.g. an RFID chip). Next, determination step210may identify whether the bard code or other box identifying data has been received before. In an instance when the barcode data is not associated with a box identifier, a temporary record may be created, at step215ofFIG.2and this data may be associated with the barcode. A user may scan multiple barcodes that already appear on the box before a box identifier is scanned. In certain instances, a box identifier may be an identifier associated with a customer or may be generated from data that a customer uses to identify the box. After step215, program flow may move to step220described below.

Alternatively, when the barcode or other identifier has previously been scanned, previously collected data may be retrieved, at step220ofFIG.2. A context associated with a set of data may be identified in step225. The context of the scanning may include but is not limited to, the time of day, location, subject, etc. For example, the context of the current scan may be 12:45 PM on Friday, Feb. 19, 2021. This is the 32nd box 122 being scanned for XZY Oil & Gas Corp. at their Oklahoma City Oklahoma storage facility. This context data may be used to categorize or distribute data. A timer may be started at step230. Video capture may begin, at step235. Data from sensing devices (microphone, RFID, scale, or magnetometer) may be polled as discussed in respect toFIG.1.

In this manner, data may be captured on each and all items that make up the contents of a box. Step245may include classifying the box content based on execution of instructions that evaluate data or that perform an AI function. Step245may be performed after the scanning of the contents of the present box is complete. Next, determination step250may identify whether the scanning of a set of boxes content are complete. When the scanning of the set of boxes are not complete, program flow may move back to step205where barcode data of another box is received. When the scanning of the set of boxes is complete, program flow may move to step255whereFIG.2ends.

FIG.3illustrates a set of steps that may be performed when data associated with a receptacle is classified. Step305ofFIG.3includes retrieving data of a box record or file. This data may be associated with a box identifier that includes content that was previously scanned. Image or video data may be segmented into a series frames from which still images of content may be generated. Each video frame in a set of video data may be analyzed. In one instance, an indication may be received to identify a subset of frames that are to be analyzed. For example, only frames that have an audio file associated with the same timestamp may be examined.

In certain instances, the hands of a user may be tracked and the subset of frames in which their hands are not present may be examined. In one instance, background detection will be carried out in step315to determine what the video static background is. Background frames may be identified based on the appearance of a frame that does not include anything else but a known background image. For example, a background may include a black color with white vertical and horizontal lines. In certain instances, sets of lines may be separated by a distance such that a scale may be identified. Respective frames may be compared to a background frame in step320. Determination step325may then identify whether a particular frame is a background frame. Step325may also identify frames that include a user's hands. When a frame includes only the background or includes the user's hands, that frame may be removed in step330. After step330, program flow may move back to determination step325. When determination step identifies that particular frames should not be removed, program flow may move to step335where the frame(s) is/are stored as an indexable snapshot of frame data. These indexed frame snapshots may then be processed in step340ofFIG.3. This processing may include OCR, NLP, and/or sentiment analysis. Objects may then be detected in step345ofFIG.3.

In certain instances, the video being processed may be have the individual frames saved at a database. The system may take frames 2 to 50 and overlay them on each other to create an average, for example. The first 2 seconds of the video may be an empty background as users may be instructed not to place anything on the background until they start the video. The frames 2 to 50 may then be averaged to account for camera shake and shadows that may distort the appearance of an object. The average of these frames may be used to identify the background to which all other frames are compared. This means that a background image may be identified via an AI process.

In an instance when a frame displays a difference above a threshold, for example 5× different, when compared to the background, it is kept as a candidate for a frame that may be relevant. Adjoining frames are compared for velocity as motion can also affect the quality of a particular frame. This may include determining whether anything is moving across the image for some number of frames. For example, until 5 consecutive frames are associated with a zero velocity. A difference plot may be made of the background vs the image to remove the background from the relevant image. A halo identifier may then be created around the image. For example, a halo may span approximately 10 pixels may be used as a delineator between an area of relevance and a background area. This may allow for the creation of a “clean edge” or the identification that a still image is clean. The image may then re-sized and or rotated for OCR extraction. OCR may be performed, at step340on any identified text. A hierarchy may be identified, this hierarchy may be based on the character's position on or within an object. This may help identify a header or a title of a document. This header or title may be designated as being the parent of a hierarchy and other text may be designated as a child of the hierarchy. A hierarchy may also be based on the characteristics of the content. For example, a company logo appearing on the page may automatically be at or near the top of the categorization hierarchy.

Object recognition may be performed, at step345. Object recognition may be used to identify physical objects or images. There are many object recognition systems known in the art. In addition to items that are commonly known, some embodiments may have object recognition systems that may be trained to recognize objects that are specific to a given application. For example, an oil and gas company may need an object recognition system that can differentiate between several similar-looking valves in a series of pictures. It may then be determined, at step350whether there is audio data associated with the frame being examined. When there is audio data present, speech to text conversion may be performed at step355. In one example, a user may provide a verbal description of the object being imaged.

This process may include identifying a sentiment at step360. This sentiment may have been identified based on use of NLP. Alternatively, or additionally, the sentiment of people in an image may be determined through AI based on facial cues. The identified information about a frame may be stored at a database that stores classified data in step360. When determination step identifies that there no audio to process, program flow may move to step360where classified data is stored at the database.

FIG.4illustrates an example of content that may be included in a receptacle. The image ofFIG.4may be a frame from a video captured by a camera. Optical character recognition may be used to identify a geographical location or locations of “southwest,” “Wowai” and “Papau New Guinea” from text included in the image. Optical character recognition may be used to identify the recording instrument used of “SERCEL SN 348.” Note that “Pecten Niuginig Company, LTD” may be identified as a parent in a hierarchy and details like “RNA Stack” or other text may be identified as being a child in this hierarchy.

FIG.5illustrates a computing system that may be used to implement an embodiment of the present invention. The computing system500ofFIG.5includes one or more processors510and main memory520. Main memory520stores, in part, instructions and data for execution by processor510. Main memory520can store the executable code when in operation. The system500ofFIG.5further includes a mass storage device530, portable storage medium drive(s)540, output devices550, user input devices560, a graphics display570, peripheral devices580, and network interface595.

The components shown inFIG.5are depicted as being connected via a single bus590. However, the components may be connected through one or more data transport means. For example, processor unit510and main memory520may be connected via a local microprocessor bus, and the mass storage device530, peripheral device(s)580, portable storage device540, and display system570may be connected via one or more input/output (I/O) buses.

Mass storage device530, which may be implemented with a magnetic disk drive or an optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor unit510. Mass storage device530can store the system software for implementing embodiments of the present invention for purposes of loading that software into main memory520.

Portable storage device540operates in conjunction with a portable non-volatile storage medium, such as a FLASH memory, compact disk or Digital video disc, to input and output data and code to and from the computer system500ofFIG.5. The system software for implementing embodiments of the present invention may be stored on such a portable medium and input to the computer system500via the portable storage device540.

Input devices560provide a portion of a user interface. Input devices560may include an alpha-numeric keypad, such as a keyboard, for inputting alpha-numeric and other information, or a pointing device, such as a mouse, a trackball, stylus, or cursor direction keys. Additionally, the system500as shown inFIG.5includes output devices550. Examples of suitable output devices include speakers, printers, network interfaces, and monitors.

Display system570may include a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, an electronic ink display, a projector-based display, a holographic display, or another suitable display device. Display system570receives textual and graphical information, and processes the information for output to the display device. The display system570may include multiple-touch touchscreen input capabilities, such as capacitive touch detection, resistive touch detection, surface acoustic wave touch detection, or infrared touch detection. Such touchscreen input capabilities may or may not allow for variable pressure or force detection.

Peripherals580may include any type of computer support device to add additional functionality to the computer system. For example, peripheral device(s)580may include a modem or a router.

Network interface595may include any form of computer interface of a computer, whether that be a wired network or a wireless interface. As such, network interface595may be an Ethernet network interface, a BlueTooth™ wireless interface, an 802.11 interface, or a cellular phone interface.

The components contained in the computer system500ofFIG.5are those typically found in computer systems that may be suitable for use with embodiments of the present invention and are intended to represent a broad category of such computer components that are well known in the art. Thus, the computer system500ofFIG.5can be a personal computer, a hand held computing device, a telephone (“smart” or otherwise), a mobile computing device, a workstation, a server (on a server rack or otherwise), a minicomputer, a mainframe computer, a tablet computing device, a wearable device (such as a watch, a ring, a pair of glasses, or another type of jewelry/clothing/accessory), a video game console (portable or otherwise), an e-book reader, a media player device (portable or otherwise), a vehicle-based computer, some combination thereof, or any other computing device. The computer can also include different bus configurations, networked platforms, multi-processor platforms, etc. The computer system500may in some cases be a virtual computer system executed by another computer system. Various operating systems can be used including Unix, Linux, Windows, Macintosh OS, Palm OS, Android, iOS, and other suitable operating systems.

The present invention may be implemented in an application that may be operable using a variety of devices. Non-transitory computer-readable storage media refer to any medium or media that participate in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, including, but not limited to, non-volatile and volatile media such as optical or magnetic disks and dynamic memory, respectively. Common forms of non-transitory computer-readable media include, for example, a FLASH memory/disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM disk, digital video disk (DVD), any other optical medium, RAM, PROM, EPROM, a FLASH EPROM, and any other memory chip or cartridge.

While various flow diagrams provided and described above may show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments can perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen in order to best explain the principles of the technology and its practical application to thereby enable others skilled in the art to best utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the technology be defined by the claim.