Patent ID: 12217441

DETAILED DESCRIPTION

System Overview

FIG.1is a schematic diagram of an embodiment of an item tracking system100that is configured to employ digital image processing. The item tracking system100may employ digital image processing to identify items204that are placed on a platform202of an imaging device102and to assign the items204to a particular user. This process allows the user to obtain items204from a space without requiring the user to scan or otherwise manually identify the items204they would like to take. In one embodiment, the item tracking system100may be installed in a space (e.g. a store) so that shoppers need not engage in the conventional checkout process. Although the example of a store is used in this disclosure, this disclosure contemplates that the item tracking system100may be installed and used in any type of physical space (e.g. a room, an office, an outdoor stand, a mall, a supermarket, a convenience store, a pop-up store, a warehouse, a storage center, an amusement park, an airport, an office building, etc.). As an example, the space may be a store that comprises a plurality of items204that are available for purchase. The item tracking system100may be installed in the store so that shoppers need not engage in the conventional checkout process to purchase items from the store. In this example, the store may be a convenience store or a grocery store. In other examples, the store may not be a physical building, but a physical space or environment where shoppers may shop. For example, the store may be a “grab-and-go” pantry at an airport, a kiosk in an office building, an outdoor market at a park, etc. As another example, the space may be a warehouse or supply room that comprises a plurality of items204that are available for a user to use or borrow. In this example, the item tracking system100may be installed to allow users to checkout parts or supplies by themselves. In other examples, the item tracking system100may be employed for any other suitable application.

In one embodiment, the item tracking system100comprises one or more imaging devices102and an item tracking device104that are in signal communication with each other over a network106. The network106allows communication between and amongst the various components of the item tracking system100. This disclosure contemplates the network106being any suitable network operable to facilitate communication between the components of the item tracking system100. The network106may include any interconnecting system capable of transmitting audio, video, signals, data, messages, or any combination of the preceding. The network106may include all or a portion of a local area network (LAN), a wide area network (WAN), an overlay network, a software-defined network (SDN), a virtual private network (VPN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., cellular networks, such as 4G or 5G), a Plain Old Telephone (POT) network, a wireless data network (e.g., WiFi, WiGig, WiMax, etc.), a Long Term Evolution (LTE) network, a Universal Mobile Telecommunications System (UMTS) network, a peer-to-peer (P2P) network, a Bluetooth network, a Near Field Communication (NFC) network, a Zigbee network, and/or any other suitable network.

Imaging Devices

The imaging device102is generally configured to capture images122and depth images124of items204that are placed on a platform202of the imaging device102. In one embodiment, the imaging device102comprises one or more cameras108, one or more three-dimensional (3D) sensors110, and one or more weight sensors112. Additional information about the hardware configuration of the imaging device102is described inFIGS.2A-2C.

The cameras108and the 3D sensors110are each configured to capture images122and depth images124of at least a portion of the platform202. The cameras108are configured to capture images122(e.g. RGB images) of items204. Examples of cameras108include, but are not limited to, cameras, video cameras, web cameras, and printed circuit board (PCB) cameras. The 3D sensors110are configured to capture depth images124such as depth maps or point cloud data for items204. A depth image124comprises a plurality of pixels. Each pixel in the depth image124comprises depth information identifying a distance between the 3D sensor110and a surface in the depth image124. Examples of 3D sensors110include, but are not limited to, depth-sensing cameras, time-of-flight sensors, LiDARs, structured light cameras, or any other suitable type of depth sensing device. In some embodiments, a camera108and a 3D sensor110be integrated within a single device. In other embodiments, a camera108and a 3D sensor110be distinct devices.

The weight sensors112are configured to measure the weight of items204that are placed on the platform202of the imaging device102. For example, a weight sensor112may comprise a transducer that converts an input mechanical force (e.g. weight, tension, compression, pressure, or torque) into an output electrical signal (e.g. current or voltage). As the input force increases, the output electrical signal may increase proportionally. The item tracking device104is configured to analyze the output electrical signal to determine an overall weight for the items204on the weight sensor112. Examples of weight sensors112include, but are not limited to, a piezoelectric load cell or a pressure sensor. For example, a weight sensor112may comprise one or more load cells that are configured to communicate electrical signals that indicate a weight experienced by the load cells. For instance, the load cells may produce an electrical current that varies depending on the weight or force experienced by the load cells. The load cells are configured to communicate the produced electrical signals to item tracking device104for processing.

Item Tracking Device

Examples of the item tracking device104include, but are not limited to, a server, a computer, a laptop, a tablet, or any other suitable type of device. InFIG.1, the imaging device102and the item tracking device104are shown as two devices. In some embodiments, the imaging device102and the item tracking device104may be integrated within a single device. In one embodiment, the item tracking device104comprises an item tracking engine114and a memory116. Additional details about the hardware configuration of the item tracking device104are described inFIG.6. The memory116is configured to store item information118, user account information120, a machine learning model126, an encoded vector library128, and/or any other suitable type of data.

In one embodiment, the item tracking engine114is generally configured to process images122and depth images124to identify items204that are placed on the platform202of the imaging device102and to associate the identified items204with a user. An example of the item tracking engine114in operation is described in more detail below inFIGS.3and7-26.

The item information118generally comprises information that is associated with a plurality of items. Examples of item information118include, but are not limited to, prices, weights, barcodes, item identifiers, item numbers, features of items, or any other suitable information that is associated with an item204. Examples of features of an item include, but are not limited to, text, logos, branding, colors, barcodes, patterns, a shape, or any other suitable type of attributes of an item204. The user account information120comprises information for one or more accounts that are associated with a user. Examples of accounts include, but are not limited to, a customer account, an employee account, a school account, a business account, a financial account, a digital cart, or any other suitable type of account. The user account information120may be configured to associate user information with accounts that are associated with a user. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, reward membership information, or any other suitable type of information that is associated with the user. In some embodiments, the item information118and/or the user account information120may be stored in a device (e.g. a cloud server) that is external from the item tracking device104.

Examples of machine learning models126include, but are not limited to, a multi-layer perceptron, a recurrent neural network (RNN), an RNN long short-term memory (LSTM), a convolution neural network (CNN), a transformer, or any other suitable type of neural network model. In one embodiment, the machine learning model126is generally configured to receive an image122as an input and to output an item identifier based on the provided image122. The machine learning model126is trained using supervised learning training data that comprises different images122of items204with their corresponding labels (e.g. item identifiers). During the training process, the machine learning model126determines weights and bias values that allow the machine learning model126to map images122of items204to different item identifiers. Through this process, the machine learning model126is able to identify items204within an image122. The item tracking engine114may be configured to train the machine learning models126using any suitable technique as would be appreciated by one of ordinary skill in the art. In some embodiments, the machine learning model126may be stored and/or trained by a device that is external from the item tracking device104.

The encoded vector library128generally comprises information for items204that can be identified by the item tracking device104. An example of an encoded vector library128is shown inFIG.16. In one embodiment, the encoded vector library128comprises a plurality of entries1602. Each entry1602corresponds with a different item204that can be identified by the item tracking device104. Referring toFIG.16as an example, each entry1602may comprise an encoded vector1606that is linked with an item identifier1604and a plurality of feature descriptors1608. An encoded vector1606comprises an array of numerical values. Each numerical value corresponds with and describes a physical attribute (e.g. item type, size, shape, color, etc.) of an item204. An encoded vector1606may be any suitable length. For example, an encoded vector1606may have a size of 1×256, 1×512, 1×1024, or any other suitable length. The item identifier1604uniquely identifies an item204. Examples of item identifiers1604include, but are not limited to, a product name, a stock-keeping unit (SKU) number, an alphanumeric code, a graphical code (e.g. a barcode), or any other suitable type of identifier. Each of the feature descriptors1608describes a physical characteristic of an item204. Examples of feature descriptors1608include, but are not limited to, an item type1610, a dominant color1612, dimensions1614, weight1616, or any other suitable type of descriptor that describes the physical attributes of an item204. An item type1610identifies a classification for the item204. For instance, an item type1610may indicate whether an item204is a can, a bottle, a box, a fruit, a bag, etc. A dominant color1612identifies one or more colors that appear on the surface (e.g. packaging) of an item204. The dimensions1614may identify the length, width, and height of an item204. In some embodiments, the dimensions1614may be listed in ascending order. The weight1616identifies the weight of an item204. The weight1616may be shown in pounds, ounces, litters, or any other suitable units.

Hardware Configuration for the Imaging Device

FIG.2Ais a perspective view of an embodiment of an imaging device102. In this example, the imaging device102comprises a platform202, a frame structure206, a plurality of cameras108, a plurality of 3D sensors110, and a weight sensor112. The imaging device102may be configured as shown inFIG.2Aor in any other suitable configuration. In some embodiments, the imaging device102may further comprise additional components including, but not limited to, light, displays, and graphical user interfaces.

The platform202comprises a surface208that is configured to hold a plurality of items204. In some embodiments, the platform202may be integrated with the weight sensor112. For example, the platform202may be positioned on the weight sensor112which allows the weight sensor112to measure the weight of items204that are placed on the platform202. As another example, the weight sensor112may be disposed within the platform202to measure the weight of items204that are placed on the platform202. In some embodiments, at least a portion of the surface208may be transparent. In this case, a camera108or scanner (e.g. a barcode scanner) may be disposed below the surface208of the platform202and configured to capture images122or scan the bottoms of items204placed on the platform202. For instance, a camera108or scanner may be configured to identify and read product labels and/or barcodes (e.g. SKUs) of items204through the transparent surface208of the platform202. The platform202may be formed of aluminum, metal, wood, plastic, glass, or any other suitable material.

The frame structure206is generally configured to support and position cameras108and 3D sensors110. InFIG.2A, the frame structure206is configured to position a first camera108A and a second camera108C on the sides of the imaging device102with a perspective view of the items204on the platform202. The frame structure206is further configured to position a third camera108D on the back side of the imaging device102with a perspective view of the items204on the platform202. In some embodiments, the frame structure206may further comprise a fourth camera108(not shown) on the front side of the imaging device102with a perspective view of items204on the platform202. The frame structure206may be configured to use any number and combination of the side cameras108A and108C, the back side camera108D, and the front side camera108. For example, one or more of the identified cameras108may be optional and omitted. A perspective image122or depth image124is configured to capture the side-facing surfaces of items204placed on the platform202. The frame structure206is further configured to position a third camera108B and a 3D sensor110with a top view or overhead view of the items204on the platform202. An overhead image122or depth image124is configured to capture upward-facing surfaces of items204placed on the platform202. In other examples, the frame structure206may be configured to support and position any other suitable number and combination of cameras108and 3D sensors110. The frame structure206may be formed of aluminum, metal, wood, plastic, or any other suitable material.

FIG.2Bis a perspective view of another embodiment of an imaging device102with an enclosure210. In this configuration, the enclosure210is configured to at least partially encapsulate the frame structure206, the cameras108, the 3D sensors110, and the platform202of the imaging device102. The frame structure206, the cameras108, the 3D sensors110, and the platform202may be configured similar to as described inFIG.2A. In one embodiment, the frame structure206may further comprise rails or tracks212that are configured to allow the cameras108and the 3D sensors110to be repositionable within the enclosure210. For example, the cameras108A,108C, and108D may be repositionable along a vertical axis with respect to the platform202using the rails212. Similarly, camera108B and 3D sensor110may be repositionable along a horizontal axis with respect to the platform202using the rails212.

FIG.2Cis a perspective view of another embodiment of an imaging device102with an open enclosure214. In this configuration, the enclosure214is configured to at least partially cover the frame structure206, the cameras108, the 3D sensors110, and the platform202of the imaging device102. The frame structure206, the cameras108, the 3D sensors110, and the platform202may be configured similar to as described inFIG.2A. In one embodiment, the frame structure206may be integrated within the enclosure214. For example, the enclosure214may comprise openings216that are configured to house the cameras108and the 3D sensors110. InFIG.2C, the enclosure214has a rectangular cross section with rounded edges. In other embodiments, the enclosure214may be configured with any other suitable shape cross section.

An Item Tracking Process

FIG.3is a flowchart of an embodiment of an item tracking process300for the item tracking system100. The item tracking system100may employ process300to identify items204that are placed on the platform202of an imaging device102and to assign the items204to a particular user. As an example, the item tracking system100may employ process300within a store to add items204to a user's digital cart for purchase. As another example, the item tracking system100may employ process300within a warehouse or supply room to check out items to a user. In other examples, the item tracking system100may employ process300in any other suitable type of application where items204are assigned or associated with a particular user. This process allows the user to obtain items204from a space without having the user scan or otherwise identify the items204they would like to take.

At operation302, the item tracking device104performs auto-exclusion for the imaging device102. During an initial calibration period, the platform202may not have any items204placed on the platform202. During this period of time, the item tracking device104may use one or more cameras108and 3D sensors110to capture reference images122and reference depth images124of the platform without any items204placed on the platform202. The item tracking device104can then use the captured images122and depth images124as reference images to detect when an item is placed on the platform202. For example, the item tracking device104may use a 3D sensor110that is configured with a top view or overhead view of the platform202to capture a reference depth image124of the platform202when no items204are placed on the platform202. In this example, the captured depth image124may comprise a substantially constant depth value throughout the depth image124that corresponds with the surface208of the platform202. At a later time, the item tracking device104can detect that an item204has been placed on the surface208of the platform202based on differences in depth values between subsequent depth images124and the reference depth image124. As another example, the item tracking device104may use a camera108that is configured with a top view or a perspective view of the platform202to capture a reference image122of the platform when no items204are placed on the platform202. In this example, the captured image122comprises pixel values that correspond with a scene of the platform when no items204are present on the platform202. At a later time, the item tracking device104can detect that an item204has been placed on the platform202based on differences in the pixel values between subsequent images122and the reference image122.

At operation304, the item tracking device104determines whether a triggering event has been detected. A triggering event corresponds with an event that indicates that a user is interacting with the imaging device102. For instance, a triggering event may occur when a user approaches the imaging device102or places an item204on the imaging device102. As an example, the item tracking device104may determine that a triggering event has occurred in response to detecting motion using a 3D sensor110or based on changes in depths images124captured by a 3D sensor110. For example, the item tracking device104can detect that an item204has been placed on the surface208of the platform202based on differences in depth values between depth images124captured by a 3D sensor110and the reference depth image124. Referring toFIG.4as an example,FIG.4shows an example of a comparison between depth images124from an overhead view of the platform202of the imaging device102before and after placing items204shown inFIG.2Aon the platform202. Depth image124A corresponds with a reference depth image124that is captured when no items204are placed on the platform202. Depth image124B corresponds with a depth image124that is captured after items204are placed on the platform202. In this example, the colors or pixel values within the depth images124represent different depth values. In depth image124A, the depth values in the depth image124A are substantially constant which means that there are no items204on the platform202. In depth image124B, the different depth values correspond with the items204(i.e. items204A,204B, and204C) that are placed on the platform202. In this example, the item tracking device104detects a triggering event in response to detecting the presence of the items204on the platform202based on differences between depth image124A and depth image124B. The item tracking device104may also use an image122or depth image124to count the number of items204that are on the platform202. In this example, the item tracking device104determines that there are three items204placed on the platform202based on the depth image124B. The item tracking device104may use the determined number of items204later to confirm whether all of the items204have been identified. This process is discussed in more detail below in operation312.

As another example, the item tracking device104may determine that a triggering event has occurred in response to detecting motion using a camera108or based on changes in images122captured by a camera108. For example, the item tracking device104can detect that an item204has been placed on the platform202based on differences in the pixel values between subsequent images122and the reference image122. As another example, the item tracking device104may determine that a triggering event has occurred in response to a weight increase on the weight sensor112of the imaging device102. In this case, the increase in weight measured by the weight sensor112indicates that one or more items204have been placed on the platform202. In other examples, the item tracking device104may use any other suitable type of sensor or technique for detecting when a user approaches the imaging device102or places an item204on the imaging device102.

The item tracking device104remains at operation304in response to determining that a triggering event has not been detected. In this case, the item tracking device104determines that a user has not interacted with the imaging device102yet. The item tracking device104will remain at operation304to continue to check for triggering events until a user begins interacting with the imaging device102. The item tracking device104proceeds to operation306in response to determining that a triggering event has been detected. In this case, the item tracking device104determines that a user has begun interacting with the imaging device102. The item tracking device104proceeds to operation306to begin identifying items that are placed on the platform202of the imaging device102.

At operation306, the item tracking device104identifies one or more cameras108for capturing images122of the items204on the platform202of the imaging device102. The item tracking device104may identify cameras108for capturing images122of the items204based at least in part upon the pose (e.g. location and orientation) of the items204on the platform202. The pose of an item204corresponds with the location the item204and how the item204is positioned with respect to the platform202. Referring to the example inFIG.2A, a first item204A and a second item204C are positioned in a vertical orientation with respect to the platform202. In the vertical orientation, the identifiable features of an item204are primarily in the vertical orientation. Cameras108with a perspective view, such as cameras108A and108C, may be better suited for capturing images122of the identifiable features of item204that are in a vertical orientation. For instance, the item tracking device104may select camera108A to capture images122of item204A since most of the identifiable features of item204A, such as branding, text, and barcodes, are located on the sides of the item204A and are most visible using a perspective view of the item204. Similarly, the item tracking device104may then select camera108C to capture images122of item204C. In this example, a third item204B is positioned in a horizontal orientation with respect to the platform202. In the horizontal orientation, the identifiable features of an item204are primarily in the horizontal orientation. Cameras108with a top view or overhead view, such as camera108B, may be better suited for capturing images122of the identifiable features of item204that are in a horizontal orientation. In this case, the item tracking device104may select camera108B to capture images122of item204B since most of the identifiable features of item204B are located on the top of the item204B and are most visible from using an overhead view of the item204B.

In one embodiment, the item tracking device104may determine the pose of items204on the platform202using depth images124. Referring toFIG.4as an example, the depth image124B corresponds with an overhead depth image124that is captured after the items204shown inFIG.2A(i.e. items204A,204B, and204C) are placed on the platform202. In this example, the item tracking device104may use areas in the depth image124B that correspond with each item204to determine the pose of the items204. For example, the item tracking device104may determine the area402within the depth image124B that corresponds with item204A. The item tracking device104compares the determined area402to a predetermined area threshold value614. The item tracking device104determines that an item204is in a vertical orientation when the determined area402for the item204is less than or equal to the predetermined area threshold value614. Otherwise, the item tracking device104determines that the item204is in a horizontal orientation when the determined area402for the item204is greater than the predetermined area threshold value614. In this example, the item tracking device104determines that items204A and204C are in a vertical orientation because their areas402and406, respectively, are less than or equal to the area threshold value614. The item tracking device104determines that item204B is in a horizontal orientation because its area404is greater than the area threshold value614. This determination means that the item tracking device104will select cameras108(e.g. cameras108A and108C) with a perspective view of the platform202to capture images122of items204A and204C. The item tracking device104will select a camera108(e.g. camera108B) with a top view or overhead view of the platform202to capture images122of item204B.

In one embodiment, the item tracking device104may identify a camera108for capturing images122of an item204based at least in part on the distance between the item204and the camera108. For example, the item tracking device104may generate homographies608between the cameras108and/or the 3D sensors110of the imaging device102. By generating a homography608the item tracking device104is able to use the location of an item204within an image122to determine the physical location of the item204with respect to the platform202, the cameras108, and the 3D sensors110. This allows the item tracking device104to use the physical location of the item204to determine distances between the item204and each of the cameras108and 3D sensors110. A homography608comprises coefficients that are configured to translate between pixel locations in an image122or depth image124and (x,y) coordinates in a global plane (i.e. physical locations on the platform202). The item tracking device104uses homographies608to correlate between a pixel location in a particular camera108or 3D sensor110with a physical location on the platform202. In other words, the item tracking device104uses homographies608to determine where an item204is physically located on the platform202based on their pixel location within an image122or depth image124from a camera108or a 3D sensor110, respectively. Since the item tracking device104uses multiple cameras108and 3D sensors110to monitor the platform202, each camera108and 3D sensor110is uniquely associated with a different homography608based on the camera's108or 3D sensor's110physical location on the imaging device102. This configuration allows the item tracking device104to determine where an item204is physically located on the platform202based on which camera108or 3D sensor110it appears in and its location within an image122or depth image124that is captured by that camera108or 3D sensor110. Additional information about generating a homography608and using a homography608is disclosed in U.S. Pat. No. 11,023,741 entitled, “DRAW WIRE ENCODER BASED HOMOGRAPHY” which is hereby incorporated by reference herein as if reproduced in its entirety.

As an example, the item tracking device104may use an image122or a depth image124from a camera108or 3D sensor110, respectively, with a top view or overhead view of the platform202to determine the physical location of an item on the platform202. In this example, the item tracking device104may determine a pixel location for the item204within the image122or depth image124. The item tracking device104may then use a homography608to determine the physical location for the item204with respect to the platform202based on its pixel location. After determining the physical location of the item204on the platform202, the item tracking device104may then identify which camera108is physically located closest to the item204and select the identified camera108. Returning to the example inFIG.2A, the item tracking device104may select camera108A to capture images122of item204A since camera108A is closer to item204A than camera108C. Similarly, the item tracking device104may select camera108C to capture images122of item204C since camera108C is closer to item204C than camera108A. This process ensures that the camera108with the best view of an item204is selected to capture an image122of the item204.

At operation308, the item tracking device104captures images122of the items204on the platform202using the identified cameras108. Here, the item tracking device104uses the identified cameras108to capture images of the items204. Referring toFIGS.5A,5B, and5Cas examples, the item tracking device104may capture a first image122A of the item204A, a second image122B of item204B, and a third image122C of item204C using cameras108A,108B, and108C, respectively. The item tracking device104may collect one or more images122of each item204for processing. By using a subset of the cameras108available on the imaging device102to capture images of the items204, the item tracking device104is able to reduce the number of images122that will be captured and processed to identify the items204on the platform202. This process reduces the search space for identifying items204and improves the efficiency and hardware utilization of the item tracking device104by allowing the item tracking device104to process fewer images122to identify the item204instead of processing images122from all of the cameras108on the imaging device102, which may include multiple images122of the same items204. In addition, the item tracking device104also selects cameras108that are positioned to capture features that are the most useful for identifying the items204based on the orientation and location of the items204, as discussed in operation306. Examples of features include, but are not limited to, text, logos, branding, colors, barcodes, patterns, a shape, or any other suitable type of attributes of an item204.

Returning toFIG.3at operation310, the item tracking device104identifies the items204on the platform202based on the captured images122. Here, the item tracking device104identifies an item204within each image122based on the features of the item204in the image122. As an example, the machine learning model126may be a CNN. In this example, the machine learning model126includes an input layer, an output layer, and one or more hidden layers. The hidden layers include at least one convolution layer. For example, the machine learning model126may include the following sequence of layers: input layer, convolution layer, pooling layer, convolution layer, pooling layer, one or more fully connected layers, output layer. Each convolution layer of machine learning model126uses a set of convolution kernels to extract features from the pixels that form an image122. In certain embodiments, the convolution layers of machine learning model126are implemented in the frequency domain, and the convolution process is accomplished using discrete Fourier transforms. This may be desirable to reduce the computational time associated with training and using machine learning model126for image classification purposes. For example, by converting to the frequency domain, the fast Fourier transform algorithm (FFT) may be implemented to perform the discrete Fourier transforms associated with the convolutions. Not only does the use of the FFT algorithm alone greatly reduce computational times when implemented on a single CPU (as compared with applying convolution kernels in the spatial domain), the FFT algorithm may be parallelized using one or more graphics processing units (GPUs), thereby further reducing computational times. Converting to the frequency domain may also be desirable to help ensure that the machine learning model126is translation and rotation invariant (e.g., the assignment made by machine learning model126of an image122to an item identifier, based on the presence of an item204in the image122, should not depend on the position and/or orientation of the item204within image122).

As another example, the machine learning model126may be a supervised learning algorithm. Accordingly, in certain embodiments, item tracking device104is configured to train the machine learning model126to assign input images122to any of a set of predetermined item identifiers. The item tracking device104may train the machine learning model126in any suitable manner. For example, in certain embodiments, the item tracking device104trains the machine learning model126by providing the machine learning model126with training data (e.g. images122) that includes a set of labels (e.g. item identifiers) attached to the input images122. As another example, the machine learning model126may be an unsupervised learning algorithm. In such embodiments, the item tracking device104is configured to train machine learning model126by providing the machine learning model126with a collection of images122and instructing the machine learning model126to classify these images122with item identifiers identified by the item tracking device104, based on common features extracted from the images122. The item tracking device104may train the machine learning model126any time before inputting the captured images122into the machine learning model126.

After training the machine learning model126, the item tracking device104may input each of the captured images122into the machine learning model126. In response to inputting an image122in the machine learning model126, the item tracking device104receives an item identifier for an item204from the machine learning model126. The item identifier corresponds with an item204that was identified within the image122. Examples of item identifiers include, but are not limited to, an item name, a barcode, an item number, a serial number, or any other suitable type of identifier that uniquely identifies an item204.

In some embodiments, the item tracking device104may employ one or more image processing techniques without using the machine learning model126to identify an item204within an image122. For example, the item tracking device104may employ object detection and/or optical character recognition (OCR) to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. In this case, the item tracking device104may process pixels within an image122to identify text, colors, barcodes, patterns, or any other characteristics of an item204. The item tracking device104may then compare the identified features of the item204to a set of features that correspond with different items204. For instance, the item tracking device104may extract text (e.g. a product name) from an image122and may compare the text to a set of text that is associated with different items204. As another example, the item tracking device104may determine a dominant color within an image122and may compare the dominant color to a set of colors that are associated with different items204. As another example, the item tracking device104may identify a barcode within an image122and may compare the barcode to a set of barcodes that are associated with different items204. As another example, the item tracking device104may identify logos or patterns within the image122and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items204. In other examples, the item tracking device104may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items204. After comparing the identified features from an image122to the set of features that are associated with different items204, the item tracking device104then determines whether a match is found. The item tracking device104may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item204. In response to determining that a meaningful portion of features within an image122match the features of an item204, the item tracking device104may output an item identifier that corresponds with the matching item204. In other embodiments, the item tracking device104may employ one or more image processing techniques in conjunction with the machine learning model126to identify an item204within an image122using any combination of the techniques discussed above.

In some embodiments, the item tracking device104is configured to output a confidence score610that indicates a probability that an item204has been correctly identified. For example, the item tracking device104may obtain an confidence score610from the machine learning model126with the determined item identifier. In this example, the machine learning model126outputs a confidence score610that is proportional to the number of features that were used or matched when determining the item identifier. As another example, the item tracking device104may determine a confidence score610based on how well identified features match the features of the identified item204. For instance, the item tracking device104may obtain a confidence score610of 50% when half of the text identified within an image122matches the text associated with identified item204. As another example, the item tracking device104may determine obtain a confidence score610of 100% when a barcode within an image122matches a barcode of the identified item204. As another example, the item tracking device104may obtain a confidence score610of 25% when the dominant color within an image122matches a dominant color of the identified item204. In other examples, the item tracking device104may obtain a confidence score610that is based on how well any other suitable type or combination of features matches the features of the identified item204. Other information that can impact a confidence score610include, but are not limited to, the orientation of the object, the number of items on the platform202(e.g., a fewer number of items on the platform202are easier to identify than a greater number of items on the platform202); the relative distance between items on the platform (e.g., spaced apart items on the platform202are easier to identify than crowded items on the platform202); and the like. The item tracking device104may compare the confidence score610for an identified item204to a confidence score threshold value612to determine whether the item204has been identified. The item tracking device104may determine that an item204has not been identified when the confidence score610for the item204is less than the confidence score threshold value612. The item tracking device104determines that the item204has been identified when the confidence score610for the item204is greater than or equal to the confidence score threshold value612. The confidence score threshold value612may be set to 90%, 80%, 75%, or any other suitable value.

At operation312, the item tracking device104determines whether all of the items204on the platform202have been identified. For example, the item tracking device104may compare the number of identified items204from the captured images122to the number of items204on the platform202that was determined in operation304. The item tracking device104determines that all of the items204on the platform202have been identified when the number of items204identified items204from the captured images122matches the determined number of items204on the platform202. Otherwise, the item tracking device104determines that at least one of the items204has not been identified when the number of items204identified items204from the captured images122does not match the determined number of items204on the platform202.

The item tracking device104proceeds to operation314in response to determining that one or more of the items204on the platform202have not been identified. In this case, the item tracking device104may output a request for the user to reposition one or more items204on the platform202to assist the item tracking device104with identifying some of the items204on the platform. At operation314, the item tracking device104outputs a prompt to rearrange one or more items204on the platform202. As an example, one or more items204may be obscuring the view of an item204for one of the cameras108. In this example, the item tracking device104may output a message on a graphical user interface that is located at the imaging device102with instructions for the user to rearrange the position of the items204on the platform202. In some embodiments, the item tracking device104may also identify the locations of the one or more items204on the platform202that were not identified. For example, the item tracking device104may activate a light source above or below the platform202that illuminates an item204that was not recognized. In one embodiment, after outputting the message to rearrange the items204on the platform202, the item tracking device104returns to operation306to restart the process of identifying the items204on the platform202. This process prevents the item tracking device104from double counting items204after the items204have been rearranged on the platform202.

Returning to operation312, the item tracking device104proceeds to operation316in response to determining that all of the items204on the platform202have been identified. In some embodiments, the item tracking device104may validate the accuracy of detecting the identified items204based on the weight of the items204on the platform202. For example, the item tracking device104may determine a first weight that is associated with the weight of the identified items204based on item information118that is associated with the identified items204. For instance, the item tracking device104may use item identifiers for the identified items204to determine a weight that corresponds with each of the identified items204. The item tracking device104may sum the individual weights for the identified items204to determine the first weight. The item tracking device104may also receive a second weight for the items204on the platform202from the weight sensor112. The item tracking device104then determines a weight difference between the first weight and the second weight and compares the weight difference to a weight difference threshold value. The weight difference threshold value corresponds with a maximum weight difference between the first weight and the second weight. When the weight difference exceeds the weight difference threshold value, the item tracking device104may determine that there is a mismatch between the weight of the items204on the platform202of the imaging device102and the expected weight of the identified items204. In this case, the item tracking device104may output an error message and/or return to operation306to restart the item tracking process. When the weight difference is less than or equal to the weight difference threshold value, the item tracking device104may determine that there is a match between the weight of the items204on the platform202of the imaging device102and the expected weight of the identified items204. In this case, the item tracking device104may proceed to operation316.

At operation316, the item tracking device104checks whether any prohibited or restricted item204are present on the platform202. A prohibited or restricted item204is an item204that the user is not authorized to obtain due to permission restrictions, age restrictions, or any other type of restrictions. The item tracking device104may compare item identifiers for the identified items204to a list of item identifiers for restricted or prohibited items616. In response to determining that an item204matches one of the items on the list of restricted or prohibited items616, the item tracking device104proceeds to operation318to output an alert or notification that indicates that the user is prohibited from obtaining one of the items204that is on the platform202. For example, the item tracking device104may output an alert message that identifies the prohibited item204and asks the user to remove the prohibited item204from the platform202using a graphical user interface that is located at the imaging device102. As another example, the item tracking device104may output an alert message that identifies the prohibited item204to another user (e.g. an employee) that is associated with the space. In other examples, the item tracking device104may output any other suitable type of alert message in response to detecting a prohibited item204on the platform202.

At operation320, the item tracking device104determines whether the prohibited item204has been removed from the platform202. For example, the item tracking device104may use the weight sensors112to determine whether the measured weight of the item204on the platform202has decreased by an amount that corresponds with the weight of the prohibited item204. As another example, the item tracking device104may use the cameras108and/or 3D sensors110to determine whether the prohibited item204is still present on the platform202. In response to determining that the prohibited item204is still present on the platform202, the item tracking device104may pause process300and remain at operation320until the prohibited item204has been removed from the platform202. This process prevents the user from obtaining the prohibited item204. The item tracking device104may proceed to operation322after the prohibited item204has been removed from the platform202.

Otherwise, the item tracking device104proceeds to operation322in response to determining that no prohibited items204are present on the platform202. At operation322, the item tracking device104associates the items204with the user. In one embodiment, the item tracking device104may identify the user that is associated with the items204on the platform202. For example, the user may identify themselves using a scanner or card reader that is located at the imaging device102. Examples of a scanner include, but are not limited to, a QR code scanner, a barcode scanner, a near-field communication (NFC) scanner, or any other suitable type of scanner that can receive an electronic code embedded with information that uniquely identifies a person. In other examples, the user may identify themselves by providing user information on a graphical user interface that is located at the imaging device102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or any other suitable type of information that is associated with the user.

The item tracking device104uses the information provided by the user to identify an account that is associated with the user and then to add the identified items204to the user's account. For example, the item tracking device104may use the information provided by the user to identify an account within the user account information120that is associated with the user. As an example, the item tracking device104may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items204that the user has placed on the platform202to purchase. The item tracking device104may add the items204to the user's digital cart by adding the item identifiers for the identified items204to the digital cart. The item tracking device104may also add other information to the digital cart that is related to the items204. For example, the item tracking device104may use the item identifiers to look up pricing information for the identified items204from the stored item information118. The item tracking device104may then add pricing information that corresponds with each of the identified items204to the user's digital cart.

After the item tracking device104adds the items204to the user's digital cart, the item tracking device104may trigger or initiate a transaction for the items204. In one embodiment, the item tracking device104may use previously stored information (e.g. payment card information) to complete the transaction for the items204. In this case, the user may be automatically charged for the items204in their digital cart when they leave the space. In other embodiments, the item tracking device104may collect information from the user using a scanner or card reader that is located at the imaging device102to complete the transaction for the items204. This process allows the items204to be automatically added to the user's account (e.g. digital cart) without having the user scan or otherwise identify the items204they would like to take. After adding the items204to the user's account, the item tracking device104may output a notification or summary to the user with information about the items204that were added to the user's account. For example, the item tracking device104may output a summary on a graphical user interface that is located at the imaging device102. As another example, the item tracking device104may output a summary by sending the summary to an email address or a user device that is associated with the user.

Hardware Configuration for the Item Tracking Device

FIG.6is an embodiment of an item tracking device104for the item tracking system100. In one embodiment, the item tracking device104may comprise a processor602, a memory116, and a network interface604. The item tracking device104may be configured as shown or in any other suitable configuration.

Processor

The processor602comprises one or more processors operably coupled to the memory116. The processor602is any electronic circuitry including, but not limited to, state machines, one or more central processing unit (CPU) chips, logic units, cores (e.g. a multi-core processor), field-programmable gate array (FPGAs), application-specific integrated circuits (ASICs), or digital signal processors (DSPs). The processor602may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The processor602is communicatively coupled to and in signal communication with the memory116and the network interface604. The one or more processors are configured to process data and may be implemented in hardware or software. For example, the processor602may be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The processor602may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components.

The one or more processors are configured to implement various instructions. For example, the one or more processors are configured to execute item tracking instructions606to implement the item tracking engine114. In this way, processor602may be a special-purpose computer designed to implement the functions disclosed herein. In an embodiment, the item tracking engine114is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware. The item tracking engine114is configured to operate as described inFIGS.1and3. For example, the item tracking engine114may be configured to perform the operations s of process300as described inFIG.3.

Memory

The memory116is operable to store any of the information described above with respect toFIGS.1and3along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by the processor602. The memory116comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory116may be volatile or non-volatile and may comprise a read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), dynamic random-access memory (DRAM), and static random-access memory (SRAM).

The memory116is operable to store item tracking instructions606, item information118, user account information120, machine learning models126, images122, depth images124, homographies608, confidence scores610, confidence score threshold values612, area threshold values614, a list of restricted or prohibited items616, encoded vector libraries128, and/or any other data or instructions. The item tracking instructions606may comprise any suitable set of instructions, logic, rules, or code operable to execute the item tracking engine114. The item information118, the user account information120, the machine learning models126, images122, depth images124, homographies608, confidence scores610, confidence score threshold values612, area threshold values614, the list of restricted or prohibited items616, and encoded vector libraries128are configured similar to the item information118, the user account information120, the machine learning models126, images122, depth images124, homographies608, confidence scores610, confidence score threshold values612, area threshold values614, the list of restricted or prohibited items616, and encoded vector libraries128described inFIGS.1-26, respectively.

Network Interface

The network interface604is configured to enable wired and/or wireless communications. The network interface604is configured to communicate data between the imaging device102and other devices, systems, or domains. For example, the network interface604may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, a radio-frequency identification (RFID) interface, a WIFI interface, a LAN interface, a WAN interface, a PAN interface, a modem, a switch, or a router. The processor602is configured to send and receive data using the network interface604. The network interface604may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

Hand Detection Process for Triggering Item Identification

FIG.7is a flowchart of an embodiment of a hand detection process700for triggering an item identification process for the item tracking system100. The item tracking system100may employ process700to detect a triggering event that corresponds with when a user puts their hand above the platform202to place an item204on the platform202. This process allows the item tracking device104to detect the presence of a user interacting with the platform202which can be used to initiate an item detection process such as processes300and2300described inFIGS.3and23, respectively.

At operation702, the item tracking device104captures a first overhead depth image124using a 3D sensor110at a first time instance. Here, the item tracking device104first captures an overhead depth image124of the platform202to ensure that there are no items204placed on the platform202and that there are no hands present above the platform202before periodically checking for the presence of a user's hand above the platform202. The overhead depth image124captures any upward-facing surfaces of objects and the platform202. Referring toFIG.8Aas an example, the item tracking device104may employ a 3D sensor110that is positioned above the platform202to capture an overhead depth image124of the platform202. Within the overhead depth images124of the platform202, the item tracking device104defines a region-of-interest802for the platform202. The region-of-interest802(outlined with bold lines inFIGS.8A-8C) identifies a predetermined range of pixels in an overhead depth image124that corresponds with the surface of the platform202. The item tracking device104uses the defined region-of-interest802to determine whether any item204has been placed on the platform202or whether a user has their hand positioned above the platform202. The region-of interest802is the same predetermined range of pixels for all of the depth images124captured by the 3D sensor110.

Returning toFIG.7at operation704, the item tracking device104captures a second overhead depth image124using the same 3D sensor110at a second time instance. After capturing the first overhead depth image124, the item tracking device104begins periodically capturing additional overhead depth images124of the platform202to check whether a user's hand has entered the region-of-interest802for the platform202. The item tracking device104may capture additional overhead depth images124every second, every ten seconds, every thirty seconds, or at any other suitable time interval. In some embodiments, the item tracking device104may capture the second overhead depth image124in response to detecting motion near the platform202. For example, the item tracking device104may employ a proximity sensor that is configured to detect motion near the platform202before capturing the second overhead depth image124. As another example, the item tracking device104may periodically capture additional overhead depth image124to detect motion. In this example, the item tracking device104compares the first overhead depth image124to subsequently captured overhead depth images124and detects motion based on differences, for example, the presence of an object, between the overhead depth images124.

At operation706, the item tracking device104determines whether an object is present within the region-of-interest802in the second overhead depth image124. In one embodiment, the item tracking device104determines an object is present within the region-of-interest802based on differences between the first overhead depth image124and the second overhead depth image124. Referring toFIG.8Bas an example, the item tracking device104compares the second overhead depth image124(shown inFIG.8B) to the first overhead depth image124(shown inFIG.8A) to identify differences between the first overhead depth image124and the second overhead depth image124. In this example, the item tracking device104detects an object804within the in region-of-interest802in the second overhead depth image124that corresponds with the hand of a user.FIG.8Cshows a corresponding image122of the object804that is present in the second overhead depth image124.

Returning toFIG.7at operation706, the item tracking device104returns to operation704in response to determining that there is not an object present within the region-of-interest802in the second overhead depth image124. In this case, the item tracking device104returns to operation704to continue periodically capturing overhead depth image124of the platform202to check where a user's hand has entered the region-of-interest802of the platform202. The item tracking device104proceeds to operation708in response to determining that an object is present within the region-of-interest802in the second overhead depth image124. In this case, the item tracking device104proceeds to operation708to confirm whether the object in the second overhead depth image124corresponds with the hand of a user.

The item tracking device104is configured to distinguish between an item204that is placed on the platform202and the hand of a user. When a user's hand is above the platform202, the user's hand will typically be within the region-of-interest802in the second overhead depth image124while the user's arm remains outside of the region-of-interest802in the second overhead depth image124. The item tracking device104uses these characteristics to confirm that a user's hand is above the platform202, for example, when the user places an item204on the platform202.

At operation708, the item tracking device104determines that a first portion806of a first object (e.g. a user's hand and arm) is within the region-of-interest802in the second overhead depth image124. Here, the item tracking device104confirms that a first portion806of the detected object which corresponds with the user's hand is within the region-of-interest802in the second overhead depth image124. Returning to the example inFIG.8B, the user's hand (shown as portion806of the object804) is at least partially within the region-of-interest802in the second overhead depth image124.

Returning toFIG.7at operation710, the item tracking device104determines that a second portion808of the first object (e.g. a user's wrist or arm) is outside of the region-of-interest802while the first portion806of the first object (e.g. a user's hand) is within the region-of-interest802in the second overhead depth image124. Returning to the example inFIG.8B, the user's wrist and arm (shown as portion808of the object804) is at least partially outside of the region-of-interest802while the user's hand (shown as portion806of the object804) is within the region-of-interest802in the second overhead depth image124. These characteristics allow the item tracking device104to confirm that a user's hand has been detected in the second overhead depth image124.

After detecting the user's hand, the item tracking device104begins periodically capturing additional overhead depth images124of the platform202to check whether a user's hand has exited the region-of-interest802for the platform202. At operation712, the item tracking device104captures a third overhead depth image124using the 3D sensor110at a third time instance. The item tracking device104may capture additional overhead depth images124every second, every ten seconds, every thirty seconds, or at any other suitable time interval. In some embodiments, the item tracking device104may capture the third overhead depth image124in response to a weight change or difference on the platform202. For example, the item tracking device104may use a weight sensor112to determine a first weight value at the first time instance when no items204are placed on the platform202. The item tracking device104may then use the weight sensor122to determine a second weight value at a later time after the user places an item204on the platform202. In this example, the item tracking device104detects a weight difference between the first weight value and the second weight value and then captures the third overhead depth image124in response to detecting the weight difference.

At operation714, the item tracking device104determines whether the first object (i.e. the user's hand) is still present within the region-of-interest802in the third overhead depth image124. Here, the item tracking device104may determine whether the first object is present still within the region-of-interest802based on differences between the second overhead depth image124and the third overhead depth image124. Referring to the example inFIG.8D, the item tracking device104compares the third overhead depth image124(shown inFIG.8D) to the second overhead depth image124(shown inFIG.8B) to identify differences between the third overhead depth image124and the second overhead depth image124. In this example, the item tracking device104detects the first object804corresponding with the user's hand is no longer present within the in region-of-interest802in the third overhead depth image124.

Returning toFIG.7at operation714, the item tracking device104returns to operation712in response to determining that the first object804is still present within the region-of-interest802in the third overhead depth image124. In this case, the item tracking device104returns to operation712to continue periodically checking for when the user's hand exits the region-of-interest802for the platform202. The item tracking device104proceeds to operation716in response to determining that the first object804is no longer present within the region-of-interest802in the third overhead depth image124. In this case, the item tracking device104begins checking for any items204that the user placed onto the platform202.

At operation716, the item tracking device104determines whether an item204is within the region-of-interest802in the third overhead depth image124. When an item204is placed on the platform202, the item204will typically be completely within the region-of-interest802in the third overhead depth image124. The item tracking device104uses this characteristic to distinguish between an item204that is placed on the platform202and the hand of a user. Returning to the example inFIG.8D, the item tracking device104detects that there is an item204within the region-of-interest802in the third overhead depth image124.

Returning toFIG.7at operation716, the item tracking device104returns to operation704in response to determining that an item204is not present within the region-of-interest802in the third overhead depth image124. In this case, the item tracking device104determines that the user did not place any items204onto the platform202. The item tracking device104returns to operation704to repeat the hand detection process to detect when the user's hand reenters the region-of-interest802for the platform202. The item tracking device104proceeds to operation718in response to determining that an item204is present within the region-of-interest802in the third overhead depth image124. In this case, the item tracking device104proceeds to operation718to begin capturing images122and/or depth images124of the item204for additional processing such as item identification.

At operation718, the item tracking device104captures an image122of the item204in response to determining that the first object804is no longer present within the region-of-interest802in the third overhead depth image124and that an item204is present within the region-of-interest802in the third overhead depth image124. The item tracking device104may use one or more cameras108and/or 3D sensors110to capture images122or depth images124, respectively, of the item204that is placed on the platform202.

In some embodiments, the item tracking device104may capture an image122in response to detecting a weight change or difference on the platform202. For example, the item tracking device104may use a weight sensor112to determine a first weight value at the first time instance when no items204are placed on the platform202. The item tracking device104may then use the weight sensor122to determine a second weight value at a later time after the user places the item204on the platform202. In this example, the item tracking device104detects a weight difference between the first weight value and the second weight value and then captures image122in response to detecting the weight difference.

After capturing the image122of the item204, the item tracking device104may use a process similar to processes300and2300that are described inFIGS.3and23, respectively, to identify items204that are placed on the platform202based on physical attributes of the item204that are present in the captured image122.

Image Cropping Process for Item Identification

FIG.9is a flowchart of an embodiment of an image cropping process900for item identification by the item tracking system100. The item tracking system100may employ process900to isolate items204within an image122. For example, when a camera108captures an image122of the platform202, the image122may contain multiple items204that are placed on the platform202. To improve the accuracy when identifying an item204, the item tracking device104first crops the image122to isolate each item204within the image122. Cropping the image122generates a new image122(i.e. a cropped image122) that comprises pixels from the original image122that correspond with an item204. The item tracking device104repeats the process to create a set of cropped images122that each correspond with an item204.

At operation902, the item tracking device104captures a first image122of an item204on the platform202using a camera108. The item tracking device104may use a camera108with an overhead, perspective, or side profile view to capture the first image122of the item204on the platform202. As an example, the camera108may be configured with an overhead view to capture upward-facing surfaces of the item204. As another example, the camera108may be configured with a perspective or side profile view to capture the side-facing surfaces of the item204.

At operation904, the item tracking device104identifies a region-of-interest1002for the item204in the first image122. The region-of-interest1002comprises a plurality of pixels that correspond with an item204in the first image122. An example of a region-of-interest1002is a bounding box. In some embodiments, the item tracking device104may employ one or more image processing techniques to identify a region-of-interest1002for an item204within the first image122. For example, the item tracking device104may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. In this case, the item tracking device104may process the pixels within the first image122to identify text, colors, barcodes, patterns, or any other characteristics of an item204. The item tracking device104may then compare the identified features of the item204to a set of features that correspond with different items204. For instance, the item tracking device104may extract text (e.g. a product name) from the first image122and may compare the text to a set of text that is associated with different items204. As another example, the item tracking device104may determine a dominant color within the first image122and may compare the dominant color to a set of colors that are associated with different items204. As another example, the item tracking device104may identify a barcode within the first image122and may compare the barcode to a set of barcodes that are associated with different items204. As another example, the item tracking device104may identify logos or patterns within the first image122and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items204. In other examples, the item tracking device104may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items204.

After comparing the identified features from the first image122to the set of features that are associated with different items204, the item tracking device104then determines whether a match is found. The item tracking device104may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item204. In response to determining that a meaningful portion of features within the first image122matches the features of an item204, the item tracking device104identifies a region-of-interest1002that corresponds with the matching item204. In other embodiments, the item tracking device104may employ any other suitable type of image processing techniques to identify a region-of-interest1002.FIGS.10A,10B,10C, and10Dillustrate examples of region-of-interest1002for the item204.

At operation906, the item tracking device104determines a first number of pixels in the region-of-interest1002that correspond with the item204in the first image122. Here, the item tracking device104counts the number of pixels within the plurality of pixels in the identified region-of-interest1002. The number of pixels within the region-of-interest1002is proportional to how much of the first item204was detected within the first image122. For example, a greater number of pixels within the region-of-interest1002indicates that a larger portion of the item204was detected within the first image122. Alternatively, a fewer number of pixels within the region-of-interest1002indicates that a smaller portion of the item204was detected within the first image122. In some instances, a small number of pixels within the region-of-interest1002may indicate that only a small portion of the item204was visible to the selected camera108or that the region-of-interest1002was incorrectly identified. The item tracking device104proceeds to operation908to determine whether the region-of-interest1002was correctly identified.

At operation908, the item tracking device104captures a first depth image124of the item204on the platform using a 3D sensor110. Here, the item tracking device104uses a 3D sensor110to capture a first depth image124with a similar view of the item204that was captured by the camera108in operation902. For example, the item tracking device104may use a 3D sensor110that is configured with an overhead view of the item204when a camera108with an overhead view of the item204is used to capture the first image122. As another example, the item tracking device104may use a 3D sensor110that is configured with a perspective or side profile view of the item204when a camera108with a perspective or side profile view of the item204is used to capture the first image122. In other examples, the item tracking device104may use a 3D sensor110that has any other type of view of the item204that is similar the view captured in the first image122.FIGS.10A,10B,10C, and10Dillustrate examples of the first depth image124.

At operation910, the item tracking device104determines a second number of pixels in the first depth image124corresponding with the item204. Here, the item tracking device104counts the number of pixels within the first depth image124that correspond with the item204. In some embodiments, the item tracking device104may use a depth threshold value to distinguish between pixels corresponding with the item204and other items204or the platform202. For example, the item tracking device104may set a depth threshold value that is behind the surface of the item204that is facing the 3D sensor110. After applying the depth threshold value, the remaining pixels in the first depth image124correspond with the item204. The item tracking device104may then count the remaining number of pixels within the first depth image124after applying the depth threshold value to the first depth image124.

At operation912, the item tracking device104determines a difference between the first number of pixels and the second number of pixels. Here, the item tracking device104the difference between the number of pixels for the item204from the region-of-interest1002and the number of pixels for the item204from the first depth image124to determine how similar the two values are to each other. For example, the item tracking device104may subtract the first number of pixels from the second number of pixels to determine the difference between the two values. In this example, the item tracking device104may use the absolute value of the difference between the two values.

At operation914, the item tracking device104determines whether the difference is less than or equal to a difference threshold value. The distance threshold value is a user-defined value that identifies a maximum pixel difference for the identified region-of-interest1002to be considered valid for additional processing. An invalid region-of-interest1002means that the difference between the number of pixels for the item204in the region-of-interest1002and the number of pixels for the item204in the first depth image124is too great. An invalid region-of-interest1002indicates that the region-of-interest1002captures a smaller portion of the item204than is visible from the camera108and the 3D sensor110. Since an invalid region-of-interest1002only captures a small portion of the item204, the region-of-interest1002may not be suitable for subsequent image processing after cropping the first image122using the region-of-interest1002. Referring toFIG.10Aas an example of an invalid region-of-interest1002, the item tracking device104identifies a first region-of-interest1002A and the first depth image124of the item204. In this example, the difference between the number of pixels for the item204in the region-of-interest1002and the number of pixels for the item204in the first depth image124is greater than the difference threshold value. An example of the first region-of-interest1002A overlaid with the item204in the first depth image124is shown inFIG.10B.

A valid region-of-interest1002means that the difference between the number of pixels for the item204in the region-of-interest1002and the number of pixels for the item204in the first depth image124is within a predetermined tolerance level (i.e. the difference threshold value). Referring toFIG.10Cas an example of a valid region-of-interest1002, the item tracking device104identifies a second region-of-interest1002B and the first depth image124of the item204. In this example, the difference between the number of pixels for the item204in the region-of-interest1002and the number of pixels for the item204in the first depth image124is less than or equal to the difference threshold value. An example of the second region-of-interest1002B overlaid with the item204in the first depth image124is shown inFIG.10D.

Returning toFIG.9, the item tracking device104returns to operation904in response to determining that the difference is greater than the difference threshold value. In this case, the item tracking device104discards the current region-of-interest1002and returns to operation904to obtain a new region-of-interest1002for the item204. The item tracking device104proceeds to operation916in response to determining that the difference is less than or equal to the difference threshold value. In this case, the item tracking device104proceeds to operation916to crop the first image122using the identified region-of-interest1002.

At operation916, the item tracking device104crops the first image122based on the region-of-interest1002. After determining that the region-of-interest1002is valid additional processing, the item tracking device104crops the first image122by extracting the pixels within the region-of-interest1002from the first image122. By cropping the first image122, the item tracking device104generates a second image122that comprises the extracted pixels within the region-of-interest1002of the first image122.

At operation918, the item tracking device104outputs the second image122. After generating the second image122, the item tracking device104may output the second image122for additional processing. For example, the item tracking device104may output the second image122by inputting or loading the second image122into a machine learning model126to identify the item204using a process similar to process2300that is described inFIG.23. As another example, the item tracking device104may associate the second image122with feature descriptors1608(e.g. an item type1610, dominant color1612, dimensions1614, weight1616) for the item204using a process similar to process2300that is described inFIG.23.

Item Location Detection Process

FIG.11is a flowchart of an embodiment of an item location detection process1100for the item tracking system100. The item tracking system100may employ process1100to identify groups of images122that correspond with the same item204. The item tracking device104typically uses multiple cameras108to capture images122of the items204on the platform202from multiple perspectives. This process allows the item tracking device104to use redundancy to ensure that all of the items204are visible in at least one of the captured images122. Since each camera108has a different physical location and perspective of the platform202, the items204will appear in different locations in each of the captured images122. To resolve this issue, the item tracking device104uses homographies608to cluster together images122of the same item204based on each item's204physical location on the platform202. This process allows the item tracking device104to generate a set of images122for each item204that is on the platform202using the captured images122from the multiple camera perspectives.

Generating a Homography

The item tracking device104is configured to generate and use homographies608to map pixels from the cameras108and 3D sensors110to the platform202. An example of a homography608is described below inFIGS.12A and12B. By generating a homography608the item tracking device104is able to use the location of an item204within an image122to determine the physical location of the item204with respect to the platform202, the cameras108, and the 3D sensors110. This allows the item tracking device104to use the physical location of the item204to cluster images122and depth images124of an item204together for processing. Each homography608comprises coefficients that are configured to translate between pixel locations in an image122or depth image124and (x,y) coordinates in a global plane (i.e. physical locations on the platform202). Each image122and depth image124comprises a plurality of pixels. The location of each pixel within an image122or depth image124is described by its pixel location1202which identifies a pixel row and a pixel column for a pixel where the pixel is located within an image122or depth image124.

The item tracking device104uses homographies608to correlate between a pixel location in a particular camera108or 3D sensor110with a physical location on the platform202. In other words, the item tracking device104uses homographies608to determine where an item204is physically located on the platform202based on their pixel location1202within an image122or depth image124from a camera108or a 3D sensor110, respectively. Since the item tracking device104uses multiple cameras108and 3D sensors110to monitor the platform202, each camera108and 3D sensor110is uniquely associated with a different homography608based on the camera's108or 3D sensor's110physical location on the imaging device102. This configuration allows the item tracking device104to determine where an item204is physically located on the platform202based on which camera108or 3D sensor110it appears in and its location within an image122or depth image124that is captured by that camera108or 3D sensor110. In this configuration, the cameras108and the 3D sensors110are configured to capture images122and depth images124, respectively, of at least partially overlapping portions of the platform202.

Referring toFIG.12A, a homography608comprises a plurality of coefficients configured to translate between pixel locations1202in an image122or a depth image124and physical locations (e.g. (x,y) coordinates1204) in a global plane that corresponds with the top surface of the platform202. In this example, the homography608is configured as a matrix and the coefficients of the homography608are represented as H11, H12, H13, H14, H21, H22, H23, H24, H31, H32, H33, H34, H41, H42, H43, and H44. The item tracking device104may generate the homography608by defining a relationship or function between pixel locations1202in an image122or a depth image124and physical locations (e.g. (x,y) coordinates1204) in the global plane using the coefficients. For example, the item tracking device104may define one or more functions using the coefficients and may perform a regression (e.g. least squares regression) to solve for values for the coefficients that project pixel locations1202of an image122or a depth image124to (x,y) coordinates1204in the global plane. Each (x,y) coordinate1204identifies an x-value and a y-value in the global plane where an item is located on the platform202. In other examples, the item tracking device104may solve for coefficients of the homography608using any other suitable technique. In the example shown inFIG.5A, the z-value at the pixel location1202may correspond with a pixel value that represents a distance, depth, elevation, or height. In this case, the homography608is further configured to translate between pixel values in a depth image124and z-coordinates (e.g. heights or elevations) in the global plane.

The item tracking device104may use the inverse of the homography608to project from (x,y) coordinates1204in the global plane to pixel locations1202in an image122or depth image124. For example, the item tracking device104receives an (x,y) coordinate1204in the global plane for an object. The item tracking device104identifies a homography608that is associated with a camera108or 3D sensor110where the object is seen. The item tracking device104may then apply the inverse homography608to the (x,y) coordinate1204to determine a pixel location1202where the object is located in the image122or depth image124. The item tracking device104may compute the matrix inverse of the homograph608when the homography608is represented as a matrix. Referring toFIG.12Bas an example, the item tracking device104may perform matrix multiplication between an (x,y) coordinates1204in the global plane and the inverse homography608to determine a corresponding pixel location1202in the image122or depth image124.

Additional information about generating a homography608and using a homography608is disclosed in U.S. Pat. No. 11,023,741 entitled, “DRAW WIRE ENCODER BASED HOMOGRAPHY” which is hereby incorporated by reference herein as if reproduced in its entirety.

Using Homographies to Determine an Item's Location

Returning toFIG.11, after generating homographies608for the cameras108and/or 3D sensors110, the item tracking device104may then use the homographies608to cluster images122and depth images124of items204together for processing. At operation1102, the item tracking device104captures a first image122of an item204using a first camera108. The first camera108may be configured upward-facing surfaces and/or side surfaces of the items204on the platform202. Referring toFIG.13A, the item tracking device104uses a first camera108to capture a first image1302of items204A and204B that are on the platform202.

Returning toFIG.11at operation1104, the item tracking device104identifies a first region-of-interest1304for an item204in the first image122. The first region-of-interest1304comprises a plurality of pixels that correspond with the item204in the first image122. An example of a region-of-interest1304is a bounding box. In some embodiments, the item tracking device104may employ one or more image processing techniques to identify a region-of-interest1304for an item204within the first image122. For example, the item tracking device104may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. In this case, the item tracking device104may process pixels within an image122to identify text, colors, barcodes, patterns, or any other characteristics of an item204. The item tracking device104may then compare the identified features of the item204to a set of features that correspond with different items204. For instance, the item tracking device104may extract text (e.g. a product name) from an image122and may compare the text to a set of text that is associated with different items204. As another example, the item tracking device104may determine a dominant color within an image122and may compare the dominant color to a set of colors that are associated with different items204. As another example, the item tracking device104may identify a barcode within an image122and may compare the barcode to a set of barcodes that are associated with different items204. As another example, the item tracking device104may identify logos or patterns within the image122and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items204. In other examples, the item tracking device104may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items204.

After comparing the identified features from an image122to the set of features that are associated with different items204, the item tracking device104then determines whether a match is found. The item tracking device104may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item204. In response to determining that a meaningful portion of features within an image122match the features of an item204, the item tracking device104may identify a region-of-interest1304that corresponds with the matching item204. In other embodiments, the item tracking device104may employ any other suitable type of image processing techniques to identify a region-of-interest1304. Returning to the example inFIG.13A, the item tracking device104identifies a first region-of-interest1304A corresponding with the first item204A and a second region-of-interest1304B corresponding with the second item204B in the first image1302.

Returning toFIG.11at operation1106, the item tracking device104identifies a first pixel location1202within the first region-of-interest1304. The pixel location1202may be any pixel within the first region-of-interest1304. In some embodiments, the item tracking device104may identify a pixel location1202that is closest to the platform202. For example, the item tracking device104may identify a pixel location1202at a midpoint on a lower edge of the region-of-interest1304. Returning to the example inFIG.13A, the item tracking device104may identify a pixel location1202A within the first region-of-interest1304A for the first item204A and a pixel location1202B within the second region-of-interest1304B for the second item204B.

Returning toFIG.11at operation1108, the item tracking device104applies a first homography608to the first pixel location1202to determine a first (x,y) coordinate1204on the platform202for the item204. For example, the item tracking device104identifies a homography608that is associated with the first camera108and then applies the identified homography608to the pixel location1202for each item204to determine their corresponding (x,y) coordinate1204on the platform202.

At operation1110, the item tracking device104captures a second image122of the item204using a second camera108. Here, the item tracking device104uses a different camera108to capture a different view of the items204on the platform202. The second camera108may be configured upward-facing surfaces and/or side surfaces of the items204on the platform202. Referring to the example inFIG.13B, the item tracking device104uses a second camera108to capture a second image1306of the items204A and204B that are on the platform202. In this example, the second camera108is on the opposite side of the platform202from the first camera108. In this example, the first camera108captures a first side of the items204on the platform202and the second camera108captures an opposing side of the items204on the platform202. In other examples, the second camera108may be in any other suitable location.

Returning toFIG.11at operation1112, the item tracking device104identifies a second region-of-interest1304for the item204in the second image122. The second region-of-interest1304comprises a second plurality of pixels that correspond with the item204in the second image122. The item tracking device104may repeat the process described in operation1104to identify the second region-of-interest1304. Returning to the example inFIG.13B, the item tracking device104identifies a third region-of-interest1304C corresponding with the first item204A and a fourth region-of-interest1304D corresponding with the second item204B in the second image1306.

Returning toFIG.11at operation1114, the item tracking device104identifies a second pixel location1202within the second region-of-interest1304. Returning to the example inFIG.13B, the item tracking device104may identify a pixel location1202C within the third region-of-interest1304C for the first item204A and a pixel location1202D within the fourth region-of-interest1304D for the second item204B.

Returning toFIG.11at operation1116, the item tracking device104applies a second homography608to the second pixel location1202to determine a second (x, y) coordinate1204on the platform202for the item204. Here, the item tracking device104identifies a homography608that is associated with the second camera108and then applies the identified homography608to the pixel location1202for each item204to determine their corresponding (x,y) coordinate1204on the platform202.

The item tracking device104may repeat this process for any other suitable number of cameras108. Referring toFIG.13Cas another example, the item tracking device104may use third camera108to capture a third image1308of the items204on the platform202. The item tracking device104may then identify regions-of-interest1304and pixel locations1202for each item204. In this example, the item tracking device104identifies a region-of-interest1304E and a pixel location1202E for the first item204A and a region-of-interest1304F and a pixel location1202F for the second item204B. After determining the pixel locations1202for the items204, the item tracking device104then identifies a homography608that is associated with the third camera108and applies the identified homography608to the pixel location1202for each item204to determine their corresponding (x,y) coordinate1204on the platform202.

Returning toFIG.11at operation1118, the item tracking device104determines a distance1402between the first (x,y) coordinate1204and the second (x,y) coordinate1204. Referring toFIG.14as an example,FIG.14shows an overhead view of the platform202with the (x,y) coordinates1204for each item204projected onto the platform202. In this example, (x,y) coordinates1204A,1204B, and1204C are associated with the first item204A and (x,y) coordinates1204D,1204E, and1204F are associated with the second item204B. The item tracking device104is configured to iteratively select pairs of (x,y) coordinates1204and to determine a distance1402between a pair of (x,y) coordinates1204. In one embodiment, the item tracking device104is configured to determine a Euclidian distance between a pair of (x,y) coordinates1204.

Returning toFIG.11at operation1120, the item tracking device104determines whether the distance1402is less than or equal to a distance threshold value. The distance threshold value identifies a maximum distance between a pair of (x,y) coordinates1204to be considered members of the same cluster1404for an item204. The distance threshold value is a user-defined value that may be set to any suitable value. The distance threshold value may be in units of inches, centimeters, millimeters, or any other suitable units. The item tracking device104compares the distance1402between a pair of (x,y) coordinates1204and the distance threshold value and determines whether the distance1402between the pair of (x,y) coordinates1204is less than the distance threshold value.

The item tracking device104terminates process1100in response to determining that the distance1402is greater than the distance threshold value. In this case, the item tracking device104determines that the pair of (x,y) coordinates1204are not members of the same cluster1404for an item204. In some embodiments, the item tracking device104may not terminate process1100, but instead will select another pair of (x,y) coordinates1204when additional (x,y) coordinates1204are available to compare to the distance threshold value.

The item tracking device104proceeds to operation1122in response to determining that the distance1402is less than or equal to the distance threshold value. In this case, the item tracking device104determines that the pair of (x,y) coordinates1204are members of the same cluster1404for an item204. At operation1122, the item tracking device104associates the pixels within the first region-of-interest1304from the first image122and the pixels within the second region-of-interest1304from the second image122with a cluster1404for the item204. Referring toFIG.14as an example, the item tracking device104may identify a first cluster1404A for the first item204A and a second cluster1404B for the second item204B. The first cluster1404A is associated with (x,y) coordinates1204A,1024B, and1204C and region-of-interest1304A,1304C, and1304E. The second cluster1404B is associated with (x,y) coordinates1204D,1024E, and1204F and region-of-interest1304B,1304D, and1304F.

Returning toFIG.11at operation1124, the item tracking device104outputs the pixels within the first region-of-interest1304from the first image122and the pixels within the second region-of-interest1304from the second image122. In one embodiment, the item tracking device104will crop the captured images122by extracting the pixels within identified regions-of-interest1304from the images122. By cropping an image122, the item tracking device104generates a new image122that comprises the extracted pixels within a region-of-interest1304of the original image122. This process allows the item tracking device104to generate a new set of images122for an item204that each comprise the extracted pixels from the identified regions-of-interest1304that were associated with the item204. The item tracking device104may output the new images122for the item204for additional processing. For example, the item tracking device104may output the images122by inputting or loading them into a machine learning model126to identify the item204based on the physical attributes of the item204in the images122using a process similar to process2300that is described inFIG.23.

In some embodiments, the item tracking device104may also associate any identified feature descriptors with the images122for the item204and output the feature descriptors with the images122of the item204. For example, while determining the region-of-interest1304for an item204, the item tracking device104may identify an item type for the item204. In this example, the item tracking device104may associate the item type with the region-of-interest1304and output the item type with the image122of the item204that is generated based on the region-of-interest1304. As another example, the item tracking device104may obtain a weight for the item204using the weight sensor122. In this example, the item tracking device104may associate the weight with the region-of-interest1304and output the weight with the image122of the item204that is generated based on the region-of-interest1304. In other examples, the item tracking device104may be configured to identify and associate any other suitable type of feature descriptors with a region-of-interest1304before outputting the region-of-interest1304.

Search Space Reduction Process for an Encoded Vector Library

FIG.15is a flowchart of an embodiment of a search space reduction process1500for an encoded vector library128. The item tracking system100may employ process1500to filter the entries1602in the encoded vector library128to reduce the amount of items204that are considered when attempting to identify an item204that is placed on the platform202. This process reduces the amount of time required to search for a corresponding entry1602in the encoded vector library128as well as improves the accuracy of the results from identifying an entry1602in the encoded vector library128.

At operation1502, the item tracking device104obtains feature descriptors1608for an item204. Each of the feature descriptors1608describes the physical characteristics or attributes of an item204. Examples of feature descriptors1608include, but are not limited to, an item type1610, a dominant color1612, dimensions1614, weight1616, or any other suitable type of descriptor that describes an item204. In one embodiment, the item tracking device104may obtain feature descriptors using a process similar to the process described in operation1104ofFIG.11. For example, the item tracking device104may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. In some embodiments, the item tracking device104may determine the dimensions of the item204using a process similar to process1800that is described inFIG.18. The item tracking device104may determine the weight of the item204using a weight sensor112. In other embodiments, the item tracking device104may use any other suitable process for determining feature descriptors for the item204.

At operation1504, the item tracking device104determines whether the feature descriptors1608identify an item type1610for the item204. Here, the item tracking device104determines whether any information associated with an item type1610for the item204is available. An item type1610identifies a classification for the item204. For instance, an item type1610may indicate whether an item204is a can, a bottle, a box, a fruit, a bag, etc. The item tracking device104proceeds to operation1506in response to determining that the feature descriptors1608identify an item type1610for the item204. In this case, the item tracking device104uses the item type1610to filter the encoded vector library128to reduce the number of entries1602in the encoded vector library128before attempting to identify the item204.

At operation1506, the item tracking device104filters the encoded vector library128based on the item type1610. Referring toFIG.16as an example, the encoded vector library128comprises a plurality of entries1602. Each entry1602corresponds with a different item204that can be identified by the item tracking device104. Each entry1602may comprise an encoded vector1606that is linked with an item identifier1604and a plurality of feature descriptors1608. An encoded vector1606comprises an array of numerical values. Each numerical value corresponds with and describes an attribute (e.g. item type, size, shape, color, etc.) of an item204. An encoded vector1606may be any suitable length. For example, an encoded vector1606may have a size of 1×256, 1×512, 1×1024, or any other suitable length. The item identifier1604uniquely identifies an item204. Examples of item identifiers1604include, but are not limited to, a product name, an SKU number, an alphanumeric code, a graphical code (e.g. a barcode), or any other suitable type of identifier. In this example, the item tracking device104uses the item type1610to filter out or remove any entries1602in the encoded vector library128that do not contain the same item type1610. This process reduces the number of entries1602in the encoded vector library128that will be considered when identifying the item204.

Returning toFIG.15at operation1504, the item tracking device104proceeds to operation1508in response to determining that the feature descriptors1608do not identify an item type1610. In this case, the item tracking device104checks for other types of feature descriptors1608that can be used to filter the entries1602in the encoded vector library128. At operation1508, the item tracking device104determines whether the feature descriptors1608identify a dominant color1612for the item204. A dominant color1612identifies one or more colors that appear on the surface (e.g. packaging) of an item204.

The item tracking device104proceeds to operation1510in response to determining that the feature descriptors1608identify a dominant color1612for the item204. In this case, the item tracking device104proceeds to operation1510to reduce the number of entries1602in the encoded vector library128based on the dominant color1612of the item204. At operation1510, the item tracking device104filters the encoded vector library128based on the dominant color1612of the item204. Here, the item tracking device104uses the dominant color1612to filter out or remove any entries1602in the encoded vector library128that do not contain the same dominant color1612.

Returning to operation1508, the item tracking device104proceeds to operation1512in response to determining that the feature descriptors1608do not identify a dominant color1612for the item204. At operation1512, the item tracking device104determines whether the feature descriptors1608identify dimensions1614for the item204. The dimensions1614may identify the length, width, and height of an item204. In some embodiments, the dimensions1614may be listed in ascending order.

The item tracking device104proceeds to operation1514in response to determining that the feature descriptors1608identify dimensions1614for the item204. In this case, the item tracking device104proceeds to operation1514to reduce the number of entries1602in the encoded vector library128based on the dimensions1614of the item204. At operation1514, the item tracking device104filters the encoded vector library128based on the dimensions1614of the item204. Here, the item tracking device104uses the dimensions1614to filter out or remove any entries1602in the encoded vector library128that do not contain the same dimensions1614as the item204or within a predetermined tolerance of the dimensions1614of the item204. In some embodiments, this dimensions1614of the item204may be listed in ascending order to make the comparison easier between the dimensions1614of the item204and the dimensions1614of the item204in the encoded vector library128.

Returning to operation1512, the item tracking device104proceeds to operation1516in response to determining that the feature descriptors1608do not identify dimensions1614for the item204. At operation1516, the item tracking device104determines whether the feature descriptors1608identify a weight1616for the item204. The weight1616identifies the weight of an item204. The weight1616may be in pounds, ounces, litters, or any other suitable units.

The item tracking device104proceeds to operation1518in response to determining that the feature descriptors1608identify a weight1616for the item204. In this case, the item tracking device104proceeds to operation1518to reduce the number of entries1602in the encoded vector library128based on the weight1616of the item204.

At operation1518, the item tracking device104filters the encoded vector library128based on the weight of the item204. Here, the item tracking device104uses the weight1616to filter out or remove any entries1602in the encoded vector library128that do not contain the same weight1616as the item204or within a predetermined tolerance of the weight1616of the item204.

In some embodiments, the item tracking device104may repeat a similar process to filter or reduce the number of entries1602in the encoded vector library128based on any other suitable type or combination of feature descriptors1608.

Similarity Vectors

After filtering the encoded vector library128based on the feature descriptors1608of the item204, the item tracking device104may generate a similarity vector1704for a received encoded vector1702. A similarity vector1704comprises an array of numerical values1710where each numerical value1710indicates how similar the values in the received encoded vector1702are to the values in an encoded vector1606in the encoded vector library128. In one embodiment, the item tracking device104may generate the similarity vector1704by using matrix multiplication between the received encoded vector1702and the encoded vectors1606in the encoded library128. Referring toFIG.17as an example, the dimensions of the encoded vectors1606in the encoded vector library128may be M-by-N, where M is the number of entries1602in the encoded vector library128, for example, after filtering the encoded vector library128, and N is the length of each encoded vector1606, which corresponds with the number of numerical values1706in an encoded vector1606. The encoded vector1702for an unidentified item204may have the dimensions of N-by-1 where is N is the length of the encoded vector1702, which corresponds with the number of numerical values1708in the encoded vector1702. In this example, the item tracking device104may generate the similarity vector1704by performing matrix multiplication between the encoded vector1702and the encoded vectors1606in the encoded vector library128. The resulting similarity vector1704has the dimensions of N-by-1 where N is the length of the similarity vector1704which is the same length as the encoded vector1702. Each numerical value1710in the similarity vector1704corresponds with an entry1602in the encoded vector library128. For example, the first numerical value1710in the similarity vector1704indicates how similar the values in the encoded vector1702are to the values in the encoded vector1606in the first entry1602of the encoded vector library128, the second numerical value1710in the similarity vector1704indicates how similar the values in the encoded vector1702are to the values in the encoded vector1606in the second entry1602of the encoded vector library128, and so on.

After generating the similarity vector1704, the item tracking device104can identify which entry1602, or entries1602, in the encoded vector library128most closely matches the encoded vector1702for the identified item204. In one embodiment, the entry1602that is associated with the highest numerical value1710in the similarity vector1704corresponds is the entry1602that closest matches the encoded vector1702for the item204. After identifying the entry1602from the encoded vector library128that most closely matches the encoded vector1702for the identified item204, the item tracking device104may then identify the item identifier1604that is associated with the identified entry1602. Through this process, the item tracking device104is able to determine which item204from the encoded vector library128corresponds with the unidentified item204based on its encoded vector1702. The item tracking device104then output or use the identified item identifier1604for other processes such as process2300that is described inFIG.23.

Item Dimensioning Process Using Point Cloud Information

FIG.18is a flowchart of an embodiment of an item dimensioning process1800using point cloud information. The item tracking system100may employ process1800to determine the dimensions1614of an item204that is placed on the platform202. This process generally involves first capturing 3D point cloud data for an item204using multiple 3D sensors110and then combining the 3D point cloud data from all of the 3D sensors110to generate a more complete point cloud representation of the item204. After combining the point cloud data from the 3D sensors110, the item tracking device104then determines the dimensions1614of the item204based on the new point cloud data representation. This process allows the item tracking device104to determine the dimensions1614of an item204without having a user take physical measurements of the item204.

At operation1802, the item tracking device104captures point cloud data1902of items204on the platform202using an overhead 3D sensor110. The point cloud data1902comprises a plurality of data points1901within a 3D space. Each data point1901is associated with an (x, y, z) coordinate that identifies the location of the data point1901within the 3D space. In general, the point cloud data1902corresponds with the surfaces of objects that are visible to the 3D sensor110. Referring toFIG.19as an example,FIG.19illustrates an example of point cloud data1902that is captured using an overhead 3D sensor110. In this example, the 3D sensor110is positioned directly above the platform202and is configured to capture point cloud data1902that represents upward-facing surfaces of the items204on the platform202. The 3D sensor110captures point cloud data1902A that corresponds with a first item204and point cloud data1902B that corresponds with a second item204.

Returning toFIG.18at operation1804, the item tracking device104segments the point cloud data1902based on clusters1904within the point cloud data1902. In one embodiment, the item tracking device104may identify clusters1904within the point cloud data1902based on the distance between the data points1901in the point cloud data1902. For example, the item tracking device104may use a distance threshold value to identify data points1901that are members of the same cluster1904. In this example, the item tracking device104may compute the Euclidian distance between pairs of data points1901to determine whether the data points1901should be members of the same cluster1904. For instance, when a pair of data points1901are within the distance threshold value from each other, the item tracking device104may associate the data points1901with the same cluster1904. When the distance between a pair of data points1901is greater than the distance threshold value, the item tracking device104determines that the data points1901are not members of the same cluster1904. The item tracking device104may repeat this process until one or more clusters1904have been identified within the point cloud data1902. In other examples, the item tracking device104may cluster the data points1901using k-means clustering or any other suitable clustering technique. After identifying clusters1904within the point cloud data1902, the item tracking device104segments the point cloud data1902based on the identified clusters1904. Segmenting the point cloud data1902splits the data points1901in the point cloud data1902into smaller groups of point cloud data1902based on the identified clusters1904. Each cluster1904of data points1901corresponds with a different item204that is placed on the platform202.

At operation1806, the item tracking device104selects a first item204from the segmented point cloud data1902. Here, the item tracking device104identifies one of the items204on the platform202to begin aggregating the point cloud data1902from other 3D sensors110that are associated with the first item204. The item tracking device104may iteratively select each item204from the platform202. Returning to the example inFIG.19, the item tracking device104may select a first item204that corresponds with cluster1904A.

Returning toFIG.18at operation1808, the item tracking device104identifies a region-of-interest1906for the first item204within the point cloud data1902. The region-of-interest1906identifies a region within the 3D space. For example, the region-of-interest1906may define a range of x-values, y-values, and/or z-values within the 3D space. Returning to the example inFIG.19, the item tracking device104may identify a region-of-interest1906A that contains the point cloud data1902A for the first item204. In this example, the item tracking device104identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data1902A.

Returning toFIG.18at operation1810, the item tracking device104extracts point cloud data1902from the identified region-of-interest1906. Here, the item tracking device104identifies and extracts the point cloud data1902from within the region-of-interest1906for the first item204. By extracting the point cloud data1902within the region-of-interest1906, the item tracking device104is able to isolate the data points1901for the first item204in the point cloud data1902from the data points1901that are associated with other items204on the platform202. Returning to the example inFIG.19, the item tracking device104may extract the data points1901(i.e. point cloud data1902A) within the region-of-interest1906A from the point cloud data1902for all the items204on the platform202.

Returning toFIG.18at operation1812, the item tracking device104selects another 3D sensor110. After extracting point cloud data1902for the first item204from the overhead 3D sensor110, the item tracking device104may repeat the same process to extract additional point cloud data1902for the first item204from the perspective of other 3D sensors110. Each 3D sensor110is only able to capture point cloud data1902for the portion of the first item204that is visible to the 3D sensor110. By capturing point cloud data1902from multiple 3D sensors110with different views of the first item204, the item tracking device104is able to generate a more complete point cloud data representation of the first item204. The item tracking device104may iteratively select a different 3D sensor110from among the 3D sensors110of the imaging device102.

At operation1814, the item tracking device104captures point cloud data1902using the selected 3D sensor110. Here, the item tracking device104uses a process similar to the process described in operation1802to capture point cloud data1902using the selected 3D sensor110. Referring toFIG.20as an example, the item tracking device104may select a 3D sensor110that has a side perspective view of the items204on the platform202. In other words, the selected 3D sensor110captures point cloud data1902that represents side-facing surfaces of the items204on the platform202. In this example, the 3D sensor110captures point cloud data1902C that corresponds with the first item204and point cloud data1902D that corresponds with the second item204.

Returning toFIG.18at operation1816, the item tracking device104identifies a region-of-interest1906corresponding with the first item204for the selected 3D sensor110. In one embodiment, the item tracking device104may use a homography608to determine the region-of-interest1906for the selected 3D sensor110based on the region-of-interest1906identified by the overhead 3D sensor110. In this case, the item tracking device104may identify a homography608that is associated with the selected 3D sensor110. The homography608is configured similarly to as described inFIGS.12A and12B. After identifying the homography608that is associated with the 3D sensor110, the item tracking device104uses the homography608to convert the range of x-values, y-values, and z-values within the 3D space that are associated with the region-of-interest1906for the overhead 3D sensor110to a corresponding range of x-values, y-values, and z-values within the 3D space that are associated with the selected 3D sensor110. In other examples, the item tracking device104may use any other suitable technique for identifying a region-of-interest1906for the first item204. For example, the item tracking device104may use a process similar to the process described in operation1808. Returning to the example inFIG.20, the item tracking device104identifies a region-of-interest1906B that contains the point cloud data1902C for the first item204. In this example, the item tracking device104identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data1902C.

Returning toFIG.18at operation1818, the item tracking device104extracts point cloud data1902from the region-of-interest1906corresponding with the first item204. Here, the item tracking device104identifies and extracts the point cloud data1902from within the identified region-of-interest1906for the first item204. Returning to the example inFIG.20, the item tracking device104may extract the data points1901(i.e. point cloud data1902C) within the region-of-interest1906B from the point cloud data1902for all the items204on the platform202.

Returning toFIG.18at operation1820, the item tracking device104determines whether to select another 3D sensor110. Here, the item tracking device104determines whether to collect additional point cloud data1902for the first item204. In one embodiment, the item tracking device104may determine whether to select another 3D sensor110based on the amount of point cloud data1902that has been collected. For example, the item tracking device104may be configured to collect point cloud data1902from a predetermined number (e.g. three) of 3D sensors110. In this example, the item tracking device104may keep track of how many sets of point cloud data1902have been collected. Each set of collected point cloud data1902corresponds with point cloud data1902that has been obtained from a 3D sensor110. The item tracking device104then compares the number of collected sets of point cloud data1902to the predetermined number of 3D sensors110. The item tracking device104determines to select another 3D sensor110when the number of collected sets of point cloud data1902is less than the predetermined number of 3D sensors110.

As another example, the item tracking device104may determine whether to select another 3D sensor110to collect additional point cloud data1902based on the number of data points1901that have been collected for the first item204. In this example, the item tracking device104may determine the number of data points1901that have been obtained from all of the extracted point cloud data1902for the first item204. The item tracking device104compares the number of obtained data points1901to a predetermined data point threshold value. The data threshold value identifies a minimum number of data points1901that should be collected for the first item204. The item tracking device104determines to select another 3D sensor110when the number of collected data points1901is less than the predetermined data point threshold value. In other examples, the item tracking device104may determine whether to select another 3D sensor110to collect additional point cloud data1902based on any other suitable type of criteria.

The item tracking device104returns to operation1812in response to determining to select another 3D sensor. In this case, the item tracking device104returns to operation1812to select another 3D sensor110and to obtain additional point cloud data1902for the first item204. Referring toFIG.21as an example, the item tracking device104may determine to select another 3D sensor110that has a side perspective view of the items204on the platform202. In this example, the 3D sensor110captures point cloud data1902E that corresponds with the first item204and point cloud data1902F that corresponds with the second item204. The item tracking device104then identifies a region-of-interest1906C that contains the point cloud data1902E for the first item204. In this example, the item tracking device104identifies the range of x-values, y-values, and z-values within the 3D space that contains the point cloud data1902E. After identifying the region-of-interest1906C, the item tracking device104extracts the data points1901(i.e. point cloud data1902E) within the region-of-interest1906C from the point cloud data1902for all the items204on the platform202. The item tracking device104may repeat this process for any other selected 3D sensors110.

Returning toFIG.18at operation1820, the item tracking device104proceeds to operation1822in response to determining to not select another 3D sensor110. At operation1822, the item tracking device104combines the extracted point cloud data1902for the first item204. Here, the item tracking device104merges all of the collected point cloud data1902into a single set of point cloud data1902. By combining the point cloud data1902from multiple 3D sensors110, the item tracking device104can generate a more complete point cloud data representation of the first item204that can be used for determining the dimensions1614of the first item204. Referring toFIG.22as an example, the item tracking device104may combine point cloud data1902A,1902C, and1902E into a single set of point cloud data1902G. The combined point cloud data1902G contains all of the data points1901from point cloud data1902A,1902C, and1902E.

Returning toFIG.18at operation1824, the item tracking device104determines the dimensions1614of the first item204based on the combined point cloud data1902. In one embodiment, the item tracking device104may determine the dimensions1614of the item204by determining the distance between data points1901at the edges of the combined point cloud data1902. For example, the item tracking device104may identify a pair of data points1901on opposing ends of the combined point cloud data1902and then compute the distance (e.g. Euclidean distance) between the pair of data points1901. In this example, the distance between the data points1901may correspond with the length2202, width2204, or height2206of the first item204. In other examples, the item tracking device104may determine the dimensions1614of the first item204using any other suitable technique. Returning to the example inFIG.22, the item tracking device104may determine a length2202, a width2204, and a height2206for the first item204based on the combined point cloud data1902G.

Returning toFIG.18at operation1826, the item tracking device104determines whether to determine the dimensions1614for another item204. In one embodiment, the item tracking device104may be configured to determine the dimensions1614for all of the items204that are on the platform202. In this case, the item tracking device104may determine whether the dimensions1614for all of the items204on the platform202have been determined. The item tracking device104will determine the dimensions1614for another item204when the dimensions1614of an item204are still unknown and have not yet been determined. In other examples, the item tracking device104may determine whether to determine the dimensions1614for another item204based on any other suitable criteria.

The item tracking device104returns to operation1806in response to determining to find the dimensions1614for another item204. In this case, the item tracking device104returns to operation1806to collect point cloud data1902for a different item204. The item tracking device104may then repeat the same process of aggregating point cloud data1902from multiple 3D sensors110, combining the point cloud data1902, and then determining the dimensions1614of the item204based on the combined point cloud data1902.

In response to determining not to determine the dimensions1614for another item204, the item tracking device104may store the dimensions1614for the first item204. For example, the item tracking device104may obtain an item identifier1604for the first item204and then generate an entry1602in the encoded vector library128that associates the determined length2202, width2204, and height2206with the first item204as feature descriptors1608. In some embodiments, the item tracking device104may store the length2202, width2204, and height2206for the first item204in ascending order when generating the entry1602.

In other embodiments, the item tracking device104may output or store the determined length2202, width2204, and height2206for the first item204as feature descriptors1608for other processes such as item identification. For instance, the item tracking device104may use the feature descriptors1608to help identify the first item204using a process similar to process2300that is described inFIG.23.

An Item Tracking Process Using Encoded Vectors

FIG.23is a flowchart of an embodiment of an item tracking process2300for using encoded vectors1606for the item tracking system100. The item tracking system100may employ process2300to identify items204that are placed on the platform202of an imaging device102and to assign the items204to a particular user. As an example, the item tracking system100may employ process2300within a store to add items204to a user's digital cart for purchase. As another example, the item tracking system100may employ process2300within a warehouse or supply room to check out items to a user. In other examples, the item tracking system100may employ process2300in any other suitable type of application where items204are assigned or associated with a particular user. This process allows the user to obtain items204from a space without having the user scan or otherwise identify the items204they would like to take.

At operation2302, the item tracking device104performs auto-exclusion for the imaging device102. The item tracking device104may perform auto-exclusion using a process similar to the process described in operation302ofFIG.3. For example, during an initial calibration period, the platform202may not have any items204placed on the platform202. During this period of time, the item tracking device104may use one or more cameras108and/or 3D sensors110to capture reference images122and reference depth images124, respectively, of the platform202without any items204placed on the platform202. The item tracking device104can then use the captured images122and depth images124as reference images to detect when an item204is placed on the platform202. At a later time, the item tracking device104can detect that an item204has been placed on the surface208of the platform202based on differences in depth values between subsequent depth images124and the reference depth image124and/or differences in the pixel values between subsequent images122and the reference image122.

At operation2304, the item tracking device104determines whether a hand has been detected above the platform202. In one embodiment, the item tracking device104may use a process similar to process700that is described inFIG.7for detecting a triggering event that corresponds with a user's hand being detected above the platform202. For example, the item tracking device104may check for differences between a reference depth image124and a subsequent depth image124to detect the presence of an object above the platform202. The item tracking device104then checks whether the object corresponds with a user's hand or an item204that is placed on the platform202. The item tracking device104determines that the object is a user's hand when a first portion of the object (e.g. a user's wrist or arm) is outside a region-of-interest802for the platform202and a second portion of the object (e.g. a user's hand) is within the region-of-interest802for the platform202. When this condition is met, the item tracking device104determines that a user's hand has been detected above the platform202. In other examples, the item tracking device104may use proximity sensors, motion sensors, or any other suitable technique for detecting whether a user's hand has been detected above the platform202.

The item tracking device104remains at operation2304in response to determining that a user's hand has not been detected above the platform202. In this case, the item tracking device104remains at operation2304to keep checking for the presence of a user's hand as a triggering event. The item tracking device104proceeds to operation2306in response to determining that a user's hand has been detected. In this case, the item tracking device104uses the presence of a user's hand as a triggering event and proceeds to operation2306to begin identifying any items204that the user has placed on the platform202.

At operation2306, the item tracking device104performs segmentation using an overhead view of the platform202. In one embodiment, the item tracking device104may perform segmentation using a depth image124from a 3D sensor110that is configured with overhead or perspective view of the items204on the platform202. In this example, the item tracking device104captures an overhead depth image124of the items204that are placed on the platform202. The item tracking device104may then use a depth threshold value to distinguish between the platform202and items204that are placed on the platform202in the captured depth image124. For instance, the item tracking device104may set a depth threshold value that is just above the surface of the platform202. This depth threshold value may be determined based on the pixel values corresponding with the surface of the platform202in the reference depth images124that were captured during the auto-exclusion process in operation2302. After setting the depth threshold value, the item tracking device104may apply the depth threshold value to the captured depth image124to filter out or remove the platform202from the depth image124. After filtering the depth image124, the remaining clusters of pixels correspond with items204that are placed on the platform202. Each cluster of pixels corresponds with a different item204. After identifying the clusters of pixels for each item204, the item tracking device104then counts the number of items204that are placed on the platform202based on the number of pixel clusters that are present in the depth image124. This number of items204is used later to determine whether all of items204on the platform202have been identified.

At operation2308, the item tracking device104captures images122of the items204on the platform202. Here, the item tracking device104captures multiple images122of the items204on the platform202using multiple cameras108. For example, the item tracking device104may capture images122with an overhead view, a perspective view, and/or a side view of the items204on the platform202. The item tracking device104may also capture multiple depth images124of the items204on the platform202using one or more 3D sensors110.

At operation2310, the item tracking device104generates cropped images122of the items204in each image122. In one embodiment, the item tracking device104generates a cropped image122of an item204based on the features of the item204that are present in an image122. The item tracking device104may first identify a region-of-interest (e.g. a bounding box) for an item204based on the detected features of the item204that are present in an image122and then may crop the image122based on the identified region-of-interest. The region-of-interest comprises a plurality of pixels that correspond with the item204in a captured image122or depth image124of the item204on the platform202. The item tracking device104may employ one or more image processing techniques to identify a region-of-interest for an item204within an image122based on the features and physical attributes of the item204. For example, the item tracking device104may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. In this case, the item tracking device104may process pixels within an image122to identify text, colors, barcodes, patterns, or any other characteristics of an item204. The item tracking device104may then compare the identified features of the item204to a set of features that correspond with different items204. For instance, the item tracking device104may extract text (e.g. a product name) from an image122and may compare the text to a set of text that is associated with different items204. As another example, the item tracking device104may determine a dominant color within an image122and may compare the dominant color to a set of colors that are associated with different items204. As another example, the item tracking device104may identify a barcode within an image122and may compare the barcode to a set of barcodes that are associated with different items204. As another example, the item tracking device104may identify logos or patterns within the image122and may compare the identified logos or patterns to a set of logos or patterns that are associated with different items204. In other examples, the item tracking device104may identify any other suitable type or combination of features and compare the identified features to features that are associated with different items204.

After comparing the identified features of the item204to the set of features that are associated with different items204, the item tracking device104then determines whether a match is found. The item tracking device104may determine that a match is found when at least a meaningful portion of the identified features match features that correspond with an item204. In response to determining that a meaningful portion of features within an image122match the features of an item204, the item tracking device104may identify a region-of-interest that corresponds with the matching item204.

After identifying a region-of-interest for the item204, the item tracking device104crops the image122by extracting the pixels within the region-of-interest for the item204from the image122. By cropping the image122, the item tracking device104generates a second image122that comprises the extracted pixels within the region-of-interest for the item204from the original image122. This process allows the item tracking device104to generate a new image122that contains an item204that is on the platform202. The item tracking device104repeats this process for all of the items204within a captured image122and all of the captured images122of the items204on the platform202. The result of this process is a set of cropped images122that each correspond with an item204that is placed on the platform202.

In some embodiments, the item tracking device104may use a process similar to process900inFIG.9to generate the cropped images122of the items204. In some embodiments, operation2310may be optional and omitted. For example, operation2310may be omitted when the item tracking device104detects that only one item204is placed on the platform202.

At operation2312, the item tracking device104obtains an encoded vector1606for each item204. An encoded vector1606comprises an array of numerical values. Each numerical value in the encoded vector1606corresponds with and describes an attribute (e.g. item type, size, shape, color, etc.) of an item204. An encoded vector1606may be any suitable length. The item tracking device104obtains an encoded vector1606for each item204by inputting each of the images122(e.g. cropped images122) from operation2310into the machine learning model126. The machine learning model126is configured to output an encoded vector1606for an item204based on the features or physical attributes of the item204that are present in the image122of the item204. Examples of physical attributes include, but are not limited to, an item type, a size, shape, color, or any other suitable type of attribute of the item204. After inputting the image122of the item204into the machine learning model126, the item tracking device104receives an encoded vector1606for the item204. The item tracking device104repeats this process to obtain an encoded vector1606for each item204on the platform202.

At operation2314, the item tracking device104identifies each item204in the encoded vector library128based on their corresponding encoded vector1606. Here, the item tracking device104uses the encoded vector1606for each item204to identify the closest matching encoded vector1606in the encoded vector library128. In some embodiments, the item tracking device104may first reduce the search space within the encoded vector library128before attempting to identify an item204. In this case, the item tracking device104may obtain or identify feature descriptors1608for the item204using a process similar to the process described in operation1104ofFIG.11. Each of the feature descriptors1608describes the physical characteristics of an item204. Examples of feature descriptors1608include, but are not limited to, an item type1610, a dominant color1612, dimensions1614, weight1616, or any other suitable type of descriptor that describes an item204. The item tracking device104may employ object detection and/or OCR to identify text, logos, branding, colors, barcodes, or any other features of an item204that can be used to identify the item204. The item tracking device104may determine the dimensions of the item204using a process similar to process1800that is described inFIG.18. The item tracking device104may determine the weight of the item204using a weight sensor112. In other embodiments, the item tracking device104may use any other suitable process for determining feature descriptors1608for the item204. After obtaining feature descriptor1608for an item204, the item tracking device104may filter or remove the entries1602from consideration in the encoded vector library128using a process similar to process1500inFIG.15. After filtering the entries1602in the encoded vector library128, the item tracking device104may then identify the closest matching encoded vector1606in the encoded vector library128to the encoded vector1606for an unidentified item204. This process reduces the amount of time required to search for a corresponding entry1602in the encoded vector library128as well as improves the accuracy of the results from identifying an entry1602in the encoded vector library128.

In one embodiment, the item tracking device104identifies the closest matching encoded vector1606in the encoded vector library128by generating a similarity vector1704between the encoded vector1606for an unidentified item204and the remaining encoded vectors1606in the encoded vector library128. The similarity vector1704comprises an array of numerical values1710where each numerical value1710indicates how similar the values in the encoded vector1606for the item204are to the values in an encoded vector1606in the encoded vector library128. In one embodiment, the item tracking device104may generate the similarity vector1704by using a process similar to the process described inFIG.17. In this example, the item tracking device104uses matrix multiplication between the encoded vector1606for the item204and the encoded vectors1606in the encoded vector library128. Each numerical value1710in the similarity vector1704corresponds with an entry1602in the encoded vector library128. For example, the first numerical value1710in the similarity vector1704indicates how similar the values in the encoded vector1702are to the values in the encoded vector1606in the first entry1602of the encoded vector library128, the second numerical value1710in the similarity vector1704indicates how similar the values in the encoded vector1702are to the values in the encoded vector1606in the second entry1602of the encoded vector library128, and so on.

After generating the similarity vector1704, the item tracking device104can identify which entry1602, or entries1602, in the encoded vector library128most closely matches the encoded vector1606for the item204. In one embodiment, the entry1602that is associated with the highest numerical value1710in the similarity vector1704corresponds is the entry1602that most closely matches the encoded vector1606for the item204. After identifying the entry1602from the encoded vector library128that most closely matches the encoded vector1606for the item204, the item tracking device104may then identify the item identifier1604from the encoded vector library128that is associated with the identified entry1602. Through this process, the item tracking device104is able to which item204from the encoded vector library128corresponds with the item204based on its encoded vector1606. The item tracking device104then outputs the identified item identifier1604for the identified item204. For example, the item tracking device104may output the identified item identifier1604for the identified item204by adding the item identifier1604to a list of identified items204that is on a graphical user interface. The item tracking device104repeats this process for all of the encoded vectors1606that were obtained in operation2312.

At operation2316, the item tracking device104determines whether all of the items204have been identified. Here, the item tracking device104determines whether the number of identified items204matches the number of items204that were detected on the platform202in operation2306. The item tracking device104determines that all of the items204have been identified when the number of identified items204matches the number of items204that were detected on the platform202. Otherwise, the item tracking device104determines that one or more items204have not been identified when the number of identified items204does not match the number of items204that were detected on the platform202.

The item tracking device104proceeds to operation2318in response to determining that one or more items204have not been identified. In this case, the item tracking device104proceeds to operation2318to ask the user to identify the one or more items204that have not been identified. At operation2318, the item tracking device104outputs a prompt requesting the user to identify one or more items204on the platform202. In one embodiment, the item tracking device104may request for the user to identify an item204from among a set of similar items204. Referring toFIG.24as an example, the item tracking device104may output a screen2400that displays items204that were detected (shown as display elements2402) as well as any items204that were not identified. In this example, the screen2400displays the recommendations (shown as display elements2404) for other similar items204in the event that an item204is not identified. In one embodiment, the item recommendations may correspond with other items204that were identified using the similarity vector1704. For example, the item recommendations may comprise items204that are associated with the second and third highest values in the similarity vector1704. The user may provide a user input to select the any items204that were not identified.

In some embodiments, the item tracking device104may prompt the user scan any items204that were not identified. For example, the item tracking device104may provide instructions for the user to scan a barcode of an item204using a barcode scanner. In this case, the item tracking device104may use the graphical user interface to display a combination of items204that were detected on the platform202as well as items204that were manually scanned by the user. Referring toFIG.25as an example, the item tracking device104may output a screen2500that displays items204(shown as display elements2502) that were detected on the platform202and items204(shown as display elements2504) that were manually scanned by the user.

Returning toFIG.23at operation2316, the item tracking device104proceeds to operation2320in response to determining that all of the items204have been identified. At operation2320, the item tracking device104determines whether there are any additional items204to detect for the user. In some embodiments, the user may provide a user input that indicates that the user would like to add additional items204to the platform202. In other embodiments, the item tracking device104may use the presence of the user's hand removing and adding items204from the platform202to determine whether there are additional items204to detect for the user. The item tracking device104returns to operation2304in response to determining that there are additional items204to detect. In this case, the item tracking device104returns to operation2304to begin detecting additional items204that the user places on the platform202. The item tracking device104proceeds to operation2322in response to determining that there are no additional items204to detect for the user. In this case, the item tracking device104proceeds to operation2322to associate the detected items204with the user.

Before associating the items204with the user, the item tracking device104may allow the user to remove one or more items204from the list of identified items204by selecting the items204on the graphical user interface. Referring toFIG.26as an example, the item tracking device104may receive a user input that identifies an item204to remove from the list of identified items204and output a screen2600that confirms that the user would like to remove the item204. This feature allows the user to edit and finalize the list of detected items204that they would like to purchase.

Returning toFIG.23at operation2322, the item tracking device104associates the items204with the user. In one embodiment, the item tracking device104may identify the user that placed the items204on the platform202. For example, the user may identify themselves using a scanner or card reader that is located at the imaging device102. Examples of a scanner include, but are not limited to, a QR code scanner, a barcode scanner, an NFC scanner, or any other suitable type of scanner that can receive an electronic code embedded with information that uniquely identifies a person. In other examples, the user may identify themselves by providing user information on a graphical user interface that is located at the imaging device102. Examples of user information include, but are not limited to, a name, a phone number, an email address, an identification number, an employee number, an alphanumeric code, or any other suitable type of information that is associated with the user.

The item tracking device104uses the information provided by the user to identify an account that is associated with the user and then to add the identified items204to the user's account. For example, the item tracking device104may use the information provided by the user to identify an account within the user account information120that is associated with the user. As an example, the item tracking device104may identify a digital cart that is associated with the user. In this example, the digital cart comprises information about items204that the user has placed on the platform202to purchase. The item tracking device104may add the items204to the user's digital cart by adding the item identifiers1604for the identified items204to the digital cart. The item tracking device104may also add other information to the digital cart that is related to the items204. For example, the item tracking device104may use the item identifiers1604to look up pricing information for the identified items204from the stored item information118. The item tracking device104may then add pricing information that corresponds with each of the identified items204to the user's digital cart.

After the item tracking device104adds the items204to the user's digital cart, the item tracking device104may trigger or initiate a transaction for the items204. In one embodiment, the item tracking device104may use previously stored information (e.g. payment card information) to complete the transaction for the items204. In this case, the user may be automatically charged for the items204in their digital cart when they leave the space. In other embodiments, the item tracking device104may collect information from the user using a scanner or card reader that is located at the imaging device102to complete the transaction for the items204. This process allows the items204to be automatically added to the user's account (e.g. digital cart) without having the user scan or otherwise identify the items204they would like to take. After adding the items204to the user's account, the item tracking device104may output a notification or summary to the user with information about the items204that were added to the user's account. For example, the item tracking device104may output a summary on a graphical user interface that is located at the imaging device102. As another example, the item tracking device104may output a summary by sending the summary to an email address or a user device that is associated with the user.

While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated with another system or certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.

To aid the Patent Office, and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants note that they do not intend any of the appended claims to invoke 35 U.S.C. § 112(f) as it exists on the date of filing hereof unless the words “means for” or “step for” are explicitly used in the particular claim.