System and method for object counting and tracking

Disclosed systems and methods for detecting and tracking a quantity of items in a particular location by optical means. The system includes an imager having a field of view directed over a region of interest where the items to be tracked are located, the imager being operable to acquire images of the items. The system further includes a controller in operative communication with the imager, where the controller acquires depth data from the images and determines volume measurements based on the depth data. Based on the determined volume measurements, the system is capable of counting and tracking the items present in the region of interest using optical means to avoid relying on barcodes or other identifier information affixed to the items.

BACKGROUND

The field of the disclosure relates generally to object recognition systems, and in particular, to such systems capable of detecting and tracking a quantity of items in a particular location using optical means. In some embodiments, such systems may be useful to track items and improve inventory techniques in the grocery and retail environment.

In many conventional grocery and retail establishments, items for sale typically include identifiers printed on or otherwise affixed to the items, such as optical codes, barcodes, RFID tags, or other suitable machine-readable indicia. These identifiers carry a variety of item-specific information, such as item identification, price, SKU information, etc. In many instances, these identifiers are used not only during a retail checkout process to tally items for payment, but also as a means for managing inventory by tracking sales, movement, and/or removal of items from the retail establishment.

For certain items, such as produce, it may be fairly expensive and/or time-consuming to affix an identifier to each individual item, or it may be challenging in some instances to do so based on the size, shape, and characteristics of the items (e.g., wet or waxy skins, irregular shapes, etc.). Moreover, in many instances where the identifiers (e.g., stickers and labels) remain affixed to the item, they are usually small and difficult to read with current scanning equipment or other related methods. Accordingly, it is generally difficult to precisely count and track produce and other similar items with conventional tools and methods.

The present inventor has therefore recognized a need for improved object recognition systems and methods, such systems and methods providing improved counting and tracking capabilities without reliance on barcodes or other identifiers affixed to the object. Additional aspects and advantages will be apparent from the following detailed description of example embodiments, which proceeds with reference to the accompanying drawings.

Understanding that the drawings depict only certain embodiments and are not, therefore, to be considered limiting in nature, these embodiments will be described and explained with additional specificity and detail with reference to the drawings.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

With reference to the drawings, this section describes particular embodiments and their detailed construction and operation. The embodiments described herein are set forth by way of illustration only and not limitation. The described features, structures, characteristics, and methods of operation may be combined in any suitable manner in one or more embodiments. In view of the disclosure herein, those skilled in the art will recognize that the various embodiments can be practiced without one or more of the specific details or with other methods, components, materials, or the like. For the sake of clarity and conciseness, certain aspects of components or steps of certain embodiments are presented without undue detail where such detail would be apparent to those skilled in the art in light of the teachings herein and/or where such detail would obfuscate an understanding of more pertinent aspects of the embodiments.

In the following description of the figures and any example embodiments, the system may be referred to in conjunction with use at a retail establishment. It should be understood that such use is merely one example use for such a system. Other uses for a system with the characteristics and features described herein may be implemented, for example, in an industrial location for processing inventory, as well as other suitable environments for tracking and counting inventory items.

Collectively,FIGS. 1-9illustrate embodiments of an object detection or recognition system10that may be used in a grocery (or other retail) establishment, such as in a supermarket, to detect a quantity of items in a particular location by optical means. In some embodiments, the object recognition system10may be used to count and track grocery items22, such as produce, for inventory purposes without reliance on a barcode or other identifier affixed to the item22.

For example, with general reference toFIG. 1, the object recognition system10includes a camera (or other suitable imager)12arranged with its field of view20overlapping a region of interest14where the items22are located. In an example operation, camera12obtains or otherwise acquires images of the region of interest14. A processor (or other subsystem)24analyzes the acquired images and calculates a volume measurement for the items22located in the region of interest14. The volume measurement may be obtained in a variety of methods, such as by calculating the sum of the depth pixels over the region of interest14. Thereafter, the measured volume may be compared to a measured baseline measurement (e.g., a baseline volume for a known quantity of items22in the region of interest14) to determine a variance or change in volume over time, where the variance may be indicative of changes in the number of items22located within the region of interest14. Using a conversion calculation described below, the variance in volume may be directly correlated with a change in the quantity of items22present in the region of interest14. Additional details of these and other embodiments of the object recognition system10are described below with reference to the figures.

FIGS. 1 and 2are schematic illustrations of an object recognition system10operable for tracking and counting items22located within a region of interest14. With collective reference toFIGS. 1 and 2, the object recognition system10includes a camera12operable for acquiring one or more images of the items22located on a display stand or unit18. In some embodiments, the camera12may employ range imaging techniques, such that the images acquired by the camera12have pixel values that correspond to distance dimensions, which may be used in the object recognition techniques as further explained in detail below.

In some embodiments, the items22may be grouped with similar items, such as by placing like items22in a box or other container16. This arrangement may help the object recognition system10identify movement of particular items22located within a region of interest14that may include multiple different items22. As is further detailed below with reference toFIGS. 4 and 6-9, and the accompanying discussion, the images may be analyzed to create a depth profile or map of the items22in the region of interest, where the depth profile is used to determine changes in volumetric measurements over time to track a change in the quantity of items22.

As illustrated inFIGS. 1 and 2, in one embodiment, the camera12may be arranged in an overhead configuration with a field of view20directed generally downwardly toward the region of interest14. In other embodiments, the display18may be arranged differently. For example, with reference toFIG. 3, the display18may be an upright shelving unit with a variety of different grocery items22arranged in separate compartment. The camera12(not shown) may be positioned overhead as inFIGS. 1 and 2while still being able to capture images of all the items22, or may be arranged sidewardly with its field of view overlapping the items22. In some embodiments, such as those including a variety of different items22, visual cues/features for the items22and/or their respective location on the display18may be used by the object recognition system10to identify and track the particular items22as is further explained with reference toFIG. 4below. It should be understood that regardless of the arrangement of the items22and the display18, the camera12may be correspondingly aligned such that its field of view20sufficiently overlaps with the region of interest14whereat the items22to be tracked and counted are located to ensure that the acquired images properly capture the items22.

The cameras12may operate in accordance with a number of different ranging techniques. For example, the depth information may be measured using a standard time-of-flight (ToF) technique, where the camera12captures a scene in three dimensions. With this technique, a short laser pulse illuminates a scene, and the intensified CCD camera opens its high speed shutter for a short time (e.g., a few hundred picoseconds). The three-dimensional information is calculated from a two-dimensional image series that was gathered with increasing delay between the laser pulse and the shutter opening. Other range-camera operating techniques may be employed, such as stereo triangulation, where the depth data of the pixels is determined from data acquired using a stereo or multiple-camera setup system; interferometry, where the depth data may be obtained by illuminating points with coherent light and measuring the phase shift of the reflected light relative to the light source; or other suitable techniques not particularly mentioned herein.

In some embodiments, the camera12includes a processor24(or is otherwise in operable communication with a remote controller/processor) which, among other functions, is programmed to: (1) control operating parameters of the camera12; (2) to analyze the acquired images of the items22; and (3) to determine/compare the volumetric measurements in the region of interest14as discussed in further detail below. The processor24may comprise any suitable digital processor, such as a low-power DSP core or ARM core processor. In some embodiments, processor24comprises an ARM9 processor AT91SAM9G20 sold by Atmel of San Jose, Calif., USA, or OMAP processor sold by Texas Instruments of Dallas, Tex., USA or an i.MX1 series processor (such as the MC9328MX1 processor) sold by Freescale Semiconductor, Inc. of Austin, Tex., USA. Alternately, multiple processors, micro-processors, sub-processors or other types of processor electronics such as comparators or other specific function circuits may be used alone or in combination. For the purposes of this description, the term processor is meant to include any of these combinations.

As noted previously, in some embodiments, the camera12may obtain or acquire images of the items12based on an interval cycle, such as after a predetermined amount of time has elapsed (e.g., every five minutes, or every 15 minutes, or every hour, etc.). In other embodiments, to conserve power, the camera12may instead be in an inactive state (e.g., a standby mode) or may simply not run any image-acquisition protocols until the camera12receives a signal that a person is at or near the display18. For example, returning toFIG. 1, the object recognition system10may include a sensor system26in communication with the camera12(and/or the processor24or other subsystem of the camera12), where the sensor system26is operable to detect the presence of a person28within its activation field30. In some embodiments, the sensor system26may include one or more sensors positioned adjacent the display18. The one or more sensors of the first sensor system26may include optical sensors, infrared sensors, acoustics sensors, pressure sensors or any other suitable sensor configured to detect the presence of person28. When the sensor system26detects the person28, the sensor system26may generate an activation signal, which may signal the camera12to initiate an image-capture sequence. In other embodiments, the sensor system26may instead be incorporated into the camera12, where the field of view20of the camera12essentially overlaps the activation field30of the sensor system26.

In still other embodiments, the camera12may delay the image-capture sequence until receiving a second signal from the sensor system26indicating that the person is no longer within the activation field30. This arrangement may allow the camera12to obtain an unobstructed image of the region of interest14and avoid having the person28potential obscure the items22, and/or may avoid having the camera12inadvertently capture and account for the person28or an item belonging to the person28(e.g., a cup, purse, or other personal item) left near the items22as part of the items22to be tracked.

In some embodiments, the camera12may include a memory module32, which may be implemented using one or more suitable memory devices, such as RAM and ROM devices, secure digital (SD) cards, or other similar devices. In one embodiment, any number of protocols/instructions may be stored in the memory unit32, including operating systems, application programs, and volumetric calculations or calibration protocols. The memory module32may also store the images acquired by the camera12of the items22and/or may store inventory information of the items22as determined using volumetric measurement techniques described in further detail below.

In some embodiments, the camera12may also include a network interface to facilitate communication with one or more peripheral devices or systems34, such as a database/server, a mobile device, a computer, or any other suitable device. Connection with the peripheral devices or systems34may be used to communicate with and/or receive information from the camera12. For example, in some embodiments, the camera12may regularly push inventory information related to the items22to a computer system or handheld device to communicate such information with store or other personnel.

The network interface may facilitate wired or wireless communication with other devices over a short distance (e.g., Bluetooth™) or nearly unlimited distances (e.g., via the Internet). Preferably, the camera12uses a wireless connection, which may use low or high powered electromagnetic waves to transmit data using any wireless protocol, such as Bluetooth™, IEEE 802.11b (or other WiFi standards), infrared data association (IrDa), and radio frequency identification (RFID). In the case of a wired connection, a data bus may be provided using any suitable protocol, such as IEEE 802.3 (Ethernet), advanced technology attachment (ATA), personal computer memory card international association (PCMCIA), and USB.

FIG. 4is a flow chart illustrating a method400for counting and tracking objects using the object recognition system10as described with reference toFIGS. 1-3. In conjunction withFIG. 4,FIGS. 6-9provide additional details for calculations and analysis methods performed by the object recognition system10(e.g., such as via the processor24, the peripheral device32, or any other suitable subsystem) to count items22.

Before proceeding with details relating to particular calculation and analysis methods, the following provides a brief overview of the general concept of volumetric calculations using the 3D camera. As a general matter, one challenge with using an overhead 3D camera configuration as illustrated inFIG. 1is that the camera is measuring the distance from itself to the objects. The total volume measured by the camera is actually the volume of empty space between the camera and the target items, specifically along a generally pyramid-shaped trajectory that defines the field of view of the camera (seeFIGS. 6-9). The total volume may be calculating using a depth profile, a mapping of depth pixels in an x-y coordinate system as measured by the camera. This total volume, when calculated from the depth maps, is typically measured in cubic pixels, with each pixel “seeing” an area of 1×1 pixels. The depth may be measured in any one of variety of units, such as meters, least significant bits (LSBs), or other suitable units. The total volume is the sum of the individual distance measurements across a desired portion of the image, referred to as the region of interest in the context of the written description. The following provides a more detailed description of the calculation and analysis methods used by the object recognition system.

With reference toFIG. 4, at step402, a region of interest is identified, the region including the items/objects to be counted, such as a box of apples. Once the region of interest is identified, at step404, the camera is positioned with its field of view arranged to overlap the identified region of interest and the items/objects. As noted previously, although much of the description herein references the camera positioned in an overhead configuration, the camera may be arranged at different viewing configurations other than in an overhead arrangement in other embodiments.

At step406, to calibrate the object recognition system and determine a baseline volume measurement for comparison to the total volume calculated at various points in time, the camera may first acquire an image of a baseline region of interest, that is, an image of the region of interest having a known quantity of items (e.g., apples). As a frame of reference, the following description specifies the baseline region of interest as an “empty” region of interest having no apples, but it should be understood that in other embodiments the baseline measurement may be determined when the region of interest is full, or otherwise has a known quantity of items present. With particular reference toFIGS. 6 and 7, the following describes additional detail for determining a baseline volume measurement of the “empty” region of interest.

FIG. 6is a schematic of the object recognition system100arranged in a similar fashion as illustrated inFIG. 1, with the camera12positioned overhead and having a pyramid-shaped field of view20projecting downwardly therefrom. With reference toFIG. 6, the camera12obtains an image of the region of interest from which is generated a depth profile or map, D(x,y) (seeFIG. 7) measuring the depth from the camera12to any object located in the region of interest, where x and y represent pixel coordinates in the map. For example, with reference toFIGS. 6 and 7, the “empty” region of interest includes a display stand or table18on a floor36. Suppose the floor is 1000 cm away from the camera12and the table is 100 cm tall, which equates to the table being 900 cm away from the camera12. In this scenario, the generated depth profile or map would result in the mapping illustrated inFIG. 7, with the “1000” value representing the distance from the camera12to the floor36, and the “900” value representing the distance from the camera12to the table18.

From the depth profile, the object recognition system10(or a processor24thereof) calculates the total volume for the baseline or “empty” region of interest using the following equation:
Vbase=ΣxΣyD(x,y)  (1)
where the total volume of the baseline region of interest, Vbase, is measured as the sum of all the depths from the all the x-y pixel coordinates. In some embodiments, the summation may be computed as follows:

sum=0for x=0 to max x in imagefor y=0 to max y in imagesum = sum + D(x,y)endendvolume = sum

After the Vbaseis measured, the camera, at step408, next determines a second volumetric measurement for the region of interest having a second known quantity of items. For reference purposes, the following example refers to this as a “full” region of interest, that is, the display being full of apples. It should be understood, however, the in other embodiments, the region of interest may not be entirely full, but rather contain a second known quantity of items different from the number of items in the baseline measurement. At step408, the camera acquires an image of a “full” region of interest to determine a volume measurement Vfullin a similar fashion as described previously with respect to step406.

Briefly, with reference toFIG. 8, the camera12obtains an image of the region of interest from which is generated a depth profile or map, D(x,y) (seeFIG. 9). Supposing that each of the items22is approximately 10 cm tall, the depth map will reflect the position of the items22in the x-y coordinate system as being 890 cm away from the camera12. Accordingly, the depth map reflects a “1000” value to represent the distance from the camera12to the floor36, a “900” value to represent the distance from the camera12to the table18, and “890” to represent the distance from the camera12to the apples22.

From the depth profile, the object recognition system10calculates the total volume for the “full” region of interest using the following equation:
Vfull=ΣxΣyD(x,y)  (2)
where the total volume of the “full” region of interest, Vfull, is measured as the sum of all the depths from the all the x-y pixel coordinates in a similar fashion as described previously.

With these baseline volume measurements determined at steps406and408, at step410, the object recognition system determines a value for a change in volume, ΔV, per item using the following relationship:

Δ⁢⁢V=Vfull-VbaseN(3)
where N is the number of items in the “full” region of interest, that is, when the box of apples is full in the above example. Based on equation (3), the object recognition system is able to associate a measured change in volume as determined using the 3D camera and image analysis to an actual quantity of items, i.e., the number of apples remaining in the box of apples. Essentially, equation (3) embodies the concept that changes in the measured total volume of the region of interest as captured in the images indicates that a certain quantity of items may have been removed. By quantifying such changes into volumetric measurements, and associating such volumetric measurements to a known quantity of items, it is possible to monitor and track the current quantity of items in the region of interest by optical means as described in further detail below.

With the volume measurements determined from steps406,408, and410, at step412, the camera monitors the region of interest and periodically acquires images of the items. In some embodiments, the images may be acquired in accordance with a programmed protocol, such as a predetermined time cycle (e.g., every ten minutes, or every half hour). In other embodiments, the camera may acquire images in response to detection of a person near the region of interest, such as by a sensor system as described previously, or at any other desired intervals.

Once the images are acquired, at step414, the images are analyzed and a real-time volume measurement, VRT, is determined therefrom. The VRTis determined using a similar depth map analysis and summation calculation described previously with reference toFIGS. 6-9. The calculation may be determined as follows:
VRT=ΣxΣyD(x,y)  (4)

Based on the VRT, at step416, the object recognition system determines a current quantity of items, n, remaining in the region of interest based on the volumetric measurements derived at steps406,408, and410with the following relationship:

The following provides a brief example illustration of the calculation process described by method400. For example, assume Vbase=10,000 and Vfull=8,800, where Vfullwas calculated with three apples in the region of interest. Then, using equation (3), the ΔV per item is −400, with the negative number simply reflecting the point of the view of the overhead camera. With the calibration process complete, the camera periodically obtains and analyzes images of the region of interest. Assuming at a first time, t1, the camera obtains a first image and the VRTis calculated to be 9200 using equation (4). Substituting these values into equation (5), we determine that n=2, meaning that two items remain in the region of interest at the first time, t1, based on the measured VRT.

In some embodiments, equation (5) may be rounded to the nearest whole number to account for an assumption that all items are substantially the same volume and to account for potential measurement error. For example, assuming the calculation in equation (5) returns an n=2.3. The calculation may be rounded to the nearest whole number, n=2. Similarly, if the calculation returned n=1.7, then the calculation may be rounded upward to n=2.

The method400illustrates an example embodiment of an object recognition system for detecting and tracking a quantity of items in a region of interest. In such embodiments, the items in the region of interest may all be the same. For example, in the embodiment described above, the items were all apples. Accordingly, the object recognition system is able to rather easily associate a change in the number of items (as calculated by equation (5)) as reflecting a change in the number of apples that the retail establishment may have in its inventory. In other embodiments, however, the region of interest may include a plurality of different items (e.g., apples, pears, oranges, lettuce, onions, etc.) that are being simultaneously tracked by the object recognition system, such as the embodiment illustrated inFIG. 3. In such embodiments, a method500may be used to detect and track not only whether an item has been removed from the display, but may also identify which item has been removed as further described in detail below.

FIG. 5is a flow chart illustrating a method500for counting and tracking objects using an object recognition system. Many of the steps in the method500may be the same or substantially similar as the corresponding steps in the method400. Accordingly, inFIG. 5, reference numbers having the same final two digits as those inFIG. 4, identify analogous or similar steps. For example, method step402may be the same or substantially similar as method step502. To avoid repetition and potentially obscuring more pertinent aspects of the embodiment, some detail of these steps may not be further described with the understanding that a similar process or method may occur as described with the analogous step described inFIG. 4.

In step502, a region of interest is identified, the region including the items/objects to be counted. Once the region of interest is identified, at step504, the camera is positioned with its field of view arranged to overlap the identified region of interest and the objects. At step506, the object recognition system is calibrated by acquiring an image of a baseline or “empty” region of interest. In some embodiments, this step may be performed separately for each discrete type of item from among the different items to obtain the baseline volume measurement of the “empty” region of interest, Vbase, for each item. Similarly, at step508, the camera acquires images of a “full” region of interest, which may be performed separately for each discrete type of item to obtain a volume measurement of the “empty” region of interest, Vfull. The calculations for Vbaseand Vfullmay be obtained in a similar fashion as described previously with respect to equations (1) and (2) of method400.

With these baseline volume measurements determined at steps506and508, at step510, the object recognition system determines a value for a change in volume, ΔV, per item using the following relationship:

Δ⁢⁢V=Vfull-VbaseN(6)
where N is the number of items in the “full” region of interest. The system may determine this relationship for each of the different items.

At step512, the system associates the calculated ΔV measurement from equation (6) with each of the distinct items so that the system is able to track each of the distinct groups of items separately. In some embodiments, the object recognition system may associate the ΔV measurement with particular items based on one or a combination of factors, such as by using visual features and shape of the item and/or by using a known shelf location for the item to segment the items. In other embodiments, shelf location and other identifying information may instead be manually programmed into the system.

With the volume measurements determined from steps506,508,510, and512, at step514the camera monitors the region of interest and periodically acquires images of the items. Once the images are acquired, at step518, the images are analyzed and a real-time volume measurement, VRT, is determined therefrom for each of the items in a similar fashion as described previously with relation to equation (4). Using known location information and/or other identifiers for the items as described previously, the system is able to track quantities of the various items on the display.

In some embodiments, the object recognition system may be in further communication with a weigh scale, where the system monitors the scale data for items sold by weight. In some embodiments, the scale data may be used to calibrate the item quantity calculation as determined by the volumetric measurements described with relation toFIGS. 4 and 5. In other embodiments, the scale data may be used to track items sold by weight so that store personnel may be notified when inventory is low for such items.

FIGS. 1-9illustrate embodiments for an object recognition system configured to track items on display based on image analysis and volumetric measurements without a need to manually count the items and/or use a barcode scanner. It should be understood that the volumetric measurements and calculations provided herein are for illustration purposes only and not meant to be limiting. In other embodiments, different relationships or mathematical tools may be used to determine the volumetric measures from the images without departing from the principles of the disclosed subject matter.

It is intended that subject matter disclosed in any one portion herein can be combined with the subject matter of one or more other portions herein as long as such combinations are not mutually exclusive or inoperable. In addition, many variations, enhancements and modifications of the imager-based optical code reader concepts described herein are possible.

The terms and descriptions used above are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations can be made to the details of the above-described embodiments without departing from the underlying principles of the invention.