Patent ID: 12235932

DETAILED DESCRIPTION

As described above, previous technologies fail to provide efficient and reliable solutions to 1) update a training dataset of an item identification model; 2) capture images for training an item identification model; 3) identify items based on aggregated metadata; and 4) refine an item identification model based on feedback. This disclosure provides various systems and methods that provide technical solutions to the technical problems described herein.

Example System for Updating a Training Dataset of an Item Identification Model

FIG.1illustrates one embodiment of a system100that is configured to update a training dataset154of an item identification model152. In one embodiment, system100comprises a server140communicatively coupled to an imaging device120using a network110. Network110enables the communication between components of the system100. Server140comprises a processor142in signal communication with a memory148. Memory148stores software instructions150that when executed by the processor142, cause the processor142to perform one or more functions described herein. For example, when the software instructions150are executed, the processor142executes an item tracking engine144to detect one or more items102placed on a platform128of the imaging device120, and add a new entry130for each detected item102to the training dataset154. In other embodiments, system100may not have all of the components listed and/or may have other elements instead of, or in addition to, those listed above.

System Components

Network

Network110may be any suitable type of wireless and/or wired network, including, but not limited to, all or a portion of the Internet, an Intranet, a private network, a public network, a peer-to-peer network, the public switched telephone network, a cellular network, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and a satellite network. The network110may be configured to support any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

Example Imaging Device

Imaging device120is generally configured to capture images104and depth images106of items102that are placed on the platform128of the imaging device120. In one embodiment, the imaging device120comprises one or more cameras122, one or more three-dimensional (3D) sensors124, one or more weight sensors126, and a platform128. Additional information about the hardware configuration of the imaging device120is described inFIG.2.

Each camera122is configured to capture images104of at least a portion of the platform128. For example, when an item102is placed on the platform128, the cameras122are configured to capture images104(e.g., RGB images) of the item102. Examples of cameras122include, but are not limited to, cameras, 3D cameras, 2D cameras, video cameras, web cameras, and printed circuit board (PCB) cameras.

Each 3D sensor124is configured to capture depth images106of at least a portion of the platform128. For example, when an item102is placed on the platform128, the 3D sensors124are configured to capture depth images106(e.g., depth maps or point clouds) of the item102. Examples of 3D sensors124include, but are not limited to, depth-sensing cameras, time-of-flight sensors, LiDARs, structured light cameras, or any other suitable type of depth sensing device. In some embodiments, a camera122and a 3D sensor124may be integrated within a single device. In other embodiments, a camera122and a 3D sensor124may be distinct devices.

Each weight sensor126is configured to measure the weight of items102that are placed on the platform128of the imaging device120. For example, a weight sensor126may comprise a transducer that converts an input mechanical force (e.g., weight, tension, compression, pressure, or torque) into an output electrical signal (e.g., current or voltage). As the input force increases, the output electrical signal may increase proportionally. The item tracking engine144is configured to analyze the output electrical signal to determine an overall weight162for the items102on the weight sensor126. Examples of weight sensors126include, but are not limited to, a piezoelectric load cell or a pressure sensor. For example, a weight sensor126may comprise one or more load cells that are configured to communicate electrical signals that indicate a weight162experienced by the load cells. For instance, the load cells may produce an electrical current that varies depending on the weight or force experienced by the load cells. The load cells are configured to communicate the produced electrical signals to the server140(and consequently to the item tracking engine144) for processing.

The platform128comprises a flat surface on which items102may be placed. Details of the platform128are described inFIG.2.

Server

Server140is generally any device that is configured to process data and communicate with other computing devices, databases, systems, etc., via the network110. The server140may also be referred to as an item tracking device. Examples of the server140include, but are not limited to, a server, a computer, a laptop, a tablet, or any other suitable type of device. InFIG.1, the imaging device120and the server140are shown as two devices. In some embodiments, the imaging device120and the server140may be integrated within a single device. The server140is generally configured to oversee the operations of the item tracking engine144, as described further below in conjunction with the operational flow of the system100and method500described inFIG.5.

Processor142comprises one or more processors operably coupled to the memory148. The processor142is any electronic circuitry including, but not limited to, state machines, one or more central processing unit (CPU) chips, logic units, cores (e.g., a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or digital signal processors (DSPs). The processor142may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The processor142is communicatively coupled to and in signal communication with the memory148and the network interface146. The one or more processors are configured to process data and may be implemented in hardware or software. For example, the processor142may be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The processor142may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors are configured to implement various instructions. For example, the one or more processors are configured to execute software instructions150to implement the item tracking engine144. In this way, processor142may be a special-purpose computer designed to implement the functions disclosed herein. In an embodiment, the item tracking engine144is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware. The item tracking engine144is configured to operate as described inFIGS.1-5. For example, the item tracking engine144may be configured to perform the operations of method500as described inFIG.5.

Memory148is operable to store any of the information described above with respect toFIGS.1-15along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by the processor142. The memory148comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory148may be volatile or non-volatile and may comprise a read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), dynamic random-access memory (DRAM), and static random-access memory (SRAM).

The memory148is operable to store the software instructions150, item identification model152, item images104, depth images106, training dataset154, item identifier132, features158, machine learning algorithm156, triggering event108, confidence scores160, weights162, threshold percentage164, number166, threshold percentage168, and/or any other data or instructions. The software instructions150may comprise any suitable set of instructions, logic, rules, or code operable to execute the item tracking engine144. The number166may represent a particular number of dominant colors of an item102, such as one, two, three, four, five, etc.

Network interface146is configured to enable wired and/or wireless communications. The network interface146is configured to communicate data between the server140and other devices, systems, or domains. For example, the network interface146may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, a radio-frequency identification (RFID) interface, a WIFI interface, a LAN interface, a WAN interface, a PAN interface, a modem, a switch, or a router. The processor142is configured to send and receive data using the network interface146. The network interface146may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

Item Tracking Engine

Item tracking engine144may be implemented by the processor142executing the software instructions150, and is generally configured to process images104and depth images106to identify items102that are placed on the platform128of the imaging device120. In the present disclosure, an image104of an item102may be interchangeably referred to as an item image104. Operations of the item tracking engine144are described in detail further below in conjunction with the operational flow of the system100and method500described inFIG.5. The corresponding description below includes a brief description of certain operations of the item tracking engine144.

In one embodiment, the item tracking engine144is implemented by a machine learning algorithm156to process item images104and depth images106. For example, the machine learning algorithms156may include, but are not limited to, a support vector machine, neural network, random forest, k-means clustering, etc. In other examples, the machine learning algorithms156may include, but are not limited to, a multi-layer perceptron, a recurrent neural network (RNN), an RNN long short-term memory (LSTM), a convolution neural network (CNN), a transformer, or any other suitable type of neural network model. The item tracking engine144may implement the machine learning algorithm156to implement and execute the item identification model152.

In one embodiment, the machine learning algorithm156is generally configured to receive an image104of an item102as an input and extract a set of features158from the item image104. Similarly, the item tracking engine144may receive a depth image106of an item102and extract the set of features158from the depth image106. Each feature158may correspond to and/or describe a physical attribute of the item102.

The set of features158may be represented by a feature vector134that comprises a set of numerical values. For example, the set of features158may include, but not limited to: 1) one or more dominant colors of the item102; 2) a dimension of the item102; 3) a bounding box around the item102; 4) a mask that defines a contour around the item102; 5) a shape of the item102; 6) edges of the item102; and 7) a logo displayed on the item102. Each of these features158of an item102is described in greater detail below.

Each dominant color of the item102is determined based on determining colors of pixels that illustrate the item102in the item image104and/or depth image106, determining percentages of the numbers of pixels that have different colors, and determining one or more colors that have percentages of number of pixels more than a threshold percentage164.

In one embodiment, the item tracking engine144may be configured to detect a particular number166(e.g., three, five, or any other number) of dominant colors of the item102in the image item104and/or depth image106. The item tracking engine144(e.g., via the machine learning algorithm156) may determine percentages of numbers of pixels that illustrate the item102and rank them in descending order. The item tracking engine144(e.g., via the machine learning algorithm156) may detect the top particular number166of dominant colors in the ranked list of colors of the item102. The item tracking engine144may determine a percentage of a particular dominant color of an item102in an item image104by determining a ratio of a number of pixels that have the particular dominant color in relation to the total number of pixels illustrating the item102in the item image104.

For example, assume that the particular number166of dominant colors is three. Also, assume that the item tracking engine144detects that 40% of pixels that illustrate the item102in the image104are blue, 35% of pixels that illustrate the item102in the image104are red, 32% of pixels that illustrate the item102in the image104are green, and the rest of the colors have smaller percentages of numbers of pixels. In this example, the item tracking engine144determines that the top three dominant colors of the item102in the image104are blue, red, and green.

In one embodiment, the item tracking engine144may be configured to detect dominant colors of the item102in the image104that have percentages of numbers of pixels more than a threshold percentage164, such as 40%, 42%, etc. Each dominant color may be determined based on determining that a number of pixels that have the dominant color is more than a threshold number. In this case, the item tracking engine144(via the machine learning algorithm156) may determine percentages of numbers of pixels that illustrate the item102in the image104, rank them in descending order, and determine the top dominant colors that have percentages of a number of pixels more than the threshold percentage164.

The dimension of the item102in the image104may be represented by a length, a weight, and a height of the item102.

The bounding box around the item102may correspond to a shape (e.g., a rectangular, a square, any other geometry) that forms a boundary around the item102.

The mask of the item102may define a contour around the item102. For example, the mask of the item102may have a higher resolution compared to the bounding box, meaning that the mask around the item102may represent a more accurate representation of edges and lines that form the item102.

In one embodiment, the machine learning algorithm156may include a supervised machine learning algorithm, where the machine learning algorithm156may be trained using training dataset154that comprises item images104and depth images106of items102with their corresponding labels, e.g., item identifiers132, feature vectors134, features158, annotations136, etc.

Details of the training dataset154are described inFIG.4. In brief, the training dataset154comprises multiple entries130for each item102. Each entry130may be associated with one image104of an item102. Each image104of an item102may be associated with a set of features158represented by a feature vector134. Each image104of an item102may be associated with a corresponding identifier132of the item102. For example, an identifier132of the item102may include a label, a barcode, a Quick Response (QR) code, and/or the like.

Each entry130may be associated with one or more annotations136. In one embodiment, an annotation136may be used to reduce a search space during identifying an item102placed on the platform128. For example, the one or more annotations136may include a dimension (e.g., a length, a height, a weight), a dimension range (e.g., a length range, a height range, a weight range), one or more dominant colors, an item category (e.g., a type of an item, such as a can, a bottle, a candy, etc.), a logo, a brand, a shape, a weight, a weight range, among other aspects of the item102. For example, if the item tracking engine144determines that an annotation136of an item102placed on the platform128of the imaging device120comprises an item category of bottle, the item tracking engine144may search among those entries130that are associated with the same item category for identifying the item102, hence, reducing the search space. This provides practical applications of reducing computational complexity and utilizing processing and memory resources for identifying the item102more efficiently.

In the example ofFIG.1, the training dataset154comprises entries130a-1,130a-2, and130a-nfor item102a. The training dataset154may include other entries130for other items102. With respect to item102a, entry130a-1is associated with an image104a-1of the item102a. The entry130a-1is associated with identifier132a-1, feature vectors134a-1, features158a-1, and annotations136a-1. The entry130a-2is associated with identifier132a-2, feature vectors134a-2, features158a-2, and annotations136a-2. Similarly, each entry130in the training dataset154may be associated with one depth image106of an item102. Each depth image106of the item102may be associated with a set of features158represented by a feature vector134. Each depth image106of the item102may be associated with a corresponding identifier132of the item102and annotations136.

During the training process of the machine learning algorithm156, the machine learning algorithm156determines weights and bias values of the neural network layers of the machine learning algorithm156that allow the machine learning algorithm156to map images104of items102to different labels, e.g., item identifiers132, features158, feature vectors134, annotations136, etc. Through this process, the machine learning algorithm156is able to identify items102within an image104. The item tracking engine144may be configured to train the machine learning algorithm156using any suitable technique. In some embodiments, the machine learning algorithm156may be stored and/or trained by a device that is external from the server140. Similarly, the machine learning algorithm156may be trained to map depth images106of items102to their corresponding labels, e.g., item identifiers132, features158, feature vectors134, and annotations136.

In an example operation, assume that an item102is placed on the platform128. The imaging device120may capture one or more images104of the item102. The imaging device120may send the captured images104to the server140for processing. The item tracking engine144(e.g., via the machine learning algorithm156) may extract a set of features158from an image104of the item102, where the set of features158is represented by a feature vector134.

The item tracking engine144may compare the captured feature vector134with each feature vector134previously stored in the training dataset154. In this process, the item tracking engine144may perform a dot product between the captured feature vector134and each feature vector134previously stored in the training dataset154. By this process, the item tracking engine144may determine a confidence score160for each comparison, where the confidence score160may represent the similarity between a first feature vector134(extracted from the image104of the item102on the platform128) and a second feature vector134associated with an item102stored in the training dataset154. The confidence score160may be represented by a percentage, e.g., 80%, 85%, etc.

The item tracking engine144identifies an item102in the training dataset154that is associated with the highest confidence score160from among the confidence scores160. The item tracking engine144may determine that the item102(placed on the platform128) corresponds to the identified item102in the training dataset154that is associated with the highest confidence score160.

In one embodiment, the item tracking engine144may determine that the first item102placed on the platform128corresponds to a second item102stored in the training dataset154, if more than a threshold percentage (e.g., 80%, 85%, etc.) of the set of features158extracted from the image104of the first item102corresponds to counterpart features from the set of features158associated with the second item102stored in the training dataset154.

Similarly, the imaging device120may capture one or more depth images106of the item102, send the captured depth images106to the server140, and the item tracking engine144may extract the set of features158from a depth image106of the item102. The item tracking engine144may compare the extracted set of features158with each set of features158previously stored in the training dataset154by calculating a Euclidian distance between a first feature vector134extracted from a depth image106of the item placed on the platform128and a second feature vector134previously stored in the training dataset154. The Euclidian distance may correspond to the similarity between the first feature vector134and the second feature vector134. If the Euclidian distance is less than a threshold distance (e.g., 1%, 2%, 3%, etc.), the item tracking engine144may determine that a first item102associated with the first feature vector134corresponds to the second item102associated with the second feature vector134stored in the training dataset154.

Operational Flow for Updating a Training Dataset of an Item Identification Model

In one embodiment, the operational flow of the system100may include operations to determine that an item102is not included in the training dataset154, and in response, add a new entry130for the new item102in the training dataset154. For example, assume that a new item102is added to a physical store. The machine learning algorithm156may need to be configured to identify the new item102.

In one potential approach, a machine learning model is retrained to be able to identify the new item102. In the retraining process, weight and bias values of perceptron of neural network layers of the machine learning model are revised to be able to detect the new item102. However, retraining a model may be time-consuming and consume a lot of computational resources. The present disclosure discloses a technology that enables the machine learning algorithm156to identify new items102without retraining the machine learning algorithm156, thereby saving time and computational resources. This process is described below.

The machine learning algorithm156may include an input layer, one or more hidden layers, and an output layer. The input layer is the first layer of the machine learning algorithm156that receives an image104of an item102. The one or more hidden layers may include at least one convolution layer to extract features158of the item102from pixels of the image104.

Conventionally, the machine learning algorithm156may be trained to output an identifier of an item102detected in the image104. For example, the output layer may include a plurality of perceptrons, where each perceptron outputs a different identifier of an item102, e.g., a particular bottle, a particular candy, etc. Thus, if a new item102is added, a new perceptron may need to be added to the output layer of the machine learning algorithm156and the machine learning algorithm156may need to be retrained to be able to identify the new item102. However, if the output layer of the machine learning algorithm156is configured to represent extracted features158of items102, adding new items102may not cause retraining the machine learning algorithm156. This technique may obviate retraining the machine learning algorithm156, reduce computational complexity caused by retraining the machine learning algorithm156, and optimize processing and memory resource efficiency. Thus, in one embodiment, the machine learning algorithm156may be configured to output features158of items102in the output layer.

Determining that an Item is not Included in a Training Dataset

In one embodiment, the operational flow of the system100may begin when the item tracking engine144determines that an item102is not included in the training dataset154. For example, the item tracking engine144may determine that the item102is not included in the training dataset154if the item tracking engine144receives an image104of the item102, extracts features158of the item102from the image104, and determines that no image104in the training dataset154has corresponding (or matching) features158.

In response to determining that the item102is not included in the training dataset154, the item tracking engine144may perform operations described below to add a new entry130representing the item102to the training dataset154without retraining the training dataset154.

The item tracking engine144may obtain an identifier132associated with the item102. In this process, the item tracking engine144may obtain a scan of a barcode associated with the item102. For example, the item tracking engine144may obtain the scan of the barcode associated with the item102when a user scans the barcode of the item102, for example, using a barcode scanner. In other examples, the item tracking engine144may obtain a scan of a QR code, a label, or any other identifier that uniquely identifies the item102.

Detecting a Triggering Event at the Platform

The item tracking engine144detects a triggering event108at the platform128(illustrated inFIG.2). The triggering event108may correspond to a user placing the item102on the platform128.

In one embodiment, the item tracking engine144may detect the triggering event108at the platform128based on the images104captured by the cameras122.

To this end, the imaging device120may capture a reference image104of the platform128when no item102is placed on the platform128. The imaging device120may send the reference image104to the server140. When an item102is placed on the platform128, the imaging device120may capture an image104of the item102on the platform128. The imaging device120may send the image104to the server140. The item tracking engine144may compare the reference image104with the image104. The item tracking engine144may determine that the item102is placed on the platform128based on the differences between the reference image104and the image104.

In one embodiment, the item tracking engine144may detect the triggering event108at the platform128based on depth images106captured by 3D sensors124, similar to that described inFIGS.3A and3B. To this end, the imaging device120may capture a reference depth image106of the platform128when no item102is placed on the platform128. The imaging device120may send the reference depth image106to the server140. The imaging device120may capture a depth image106of an item102on the platform128when the item102is placed on the platform128. The imaging device120may send the depth image106to the server140. The item tracking engine144may compare the reference depth image106with the depth image106. The item tracking engine144may detect that the item102is placed on the platform128based on the differences between the reference depth image106and the depth image106.

In one embodiment, the item tracking engine144may detect the triggering event108at the platform128based on weight changes at the platform128detected by the weight sensor126. In this process, when no item102is placed on the platform128, the weight sensor126may detect that there is no item102is on the platform128because no pressure or weight is sensed by the weight sensor126. When an item102is placed on the platform128, the weight sensor126may detect a weight162of the item102, e.g., a weight change. The imaging device120may send the detected weight162of the item102to the server140. The item tracking engine144may detect the triggering event108based on the detected weight162of the item102.

In one embodiment, the item tracking engine144may detect the triggering event108at the platform128based on detecting that an object has entered a virtual curtain or boundary around the platform128. The object may include an item102, a hand of a user, etc. For example, the item tracking engine144may define a virtual curtain around the platform128, e.g., by implementing image processing.

In certain embodiments, the item tracking engine144may detect the triggering event108by aggregating one or more indications detected from differences between images104and the reference image104of the platform128, differences between depth images106and reference depth image106of the platform128, weight change162on the platform128, and/or an object entering the virtual curtain around the platform128.

Capturing Image(s) of the Item and Extracting Features of the Item

The imaging device120may capture one or more images104of the item102using the cameras122. The cameras122may be placed at different locations with respect to the platform128. An example configuration of arrangements of the cameras122is described inFIG.2. The one or more images104may be captured from one or more angles. Example images104are illustrated inFIG.4. The imaging device120may send the one or more images104to the server140. The item tracking engine144may perform the following operations for each image104of the item102.

The item tracking engine144may extract a set of features158associated with the item102from the image104, e.g., by feeding the image104to the machine learning algorithm156, similar to that described above. The item tracking engine144may associate the item102to the identifier132and the set of features158.

The item tracking engine144may add a new entry130to the training dataset154, where the new entry130may represent the item102labeled with the identifier132and the set of features158.

In some embodiments, the item tracking engine144may add a new entry130for each captured image104of the new item102to the training dataset154, where each new entry130is associated with a set of features158, identifier132, feature vector134, and/or annotations136, similar to that described above. The item tracking engine144may perform a similar operation for one or more depth images106of the item102placed on the platform128.

Identifying the New Item

Now that the new item102is added to the training dataset154, it can be identified by the item tracking engine144, as described below.

For example, assume that the new item102is placed on the platform128. The item tracking engine144may detect a second triggering event108at the platform128, similar to that described above. The imaging device120may capture one or more second images104of the item102using the cameras122. The imaging device120may send the one or more second images104to the server140.

The item tracking engine144may extract a second set of features158associated with the item102from each of the one or more second images104. The item tracking engine144may compare the extracted second set of features158with the set of features158previously extracted and stored in the training dataset154.

In one embodiment, the item tracking engine144may determine that the new item102corresponds to the item102previously stored in the training dataset154if it is determined that more than a threshold percentage168(e.g., more than 80%, 85%, etc.) of the second set features158corresponds to counterpart features158of the previously extracted set of features158, similar to that described above.

In certain embodiments, the item tracking engine144may perform a similar operation for depth images106of the item102. For example, the item tracking engine144may receive one or more depth images106of the item102, extract features158from each of depth images106, and add a new entry130for each depth image106of the item102to the training dataset154. The item tracking engine144may identify the new item102by comparing a captured depth image106and depth images106stored in the training dataset154, similar to that described above.

Example Imaging Device

FIG.2illustrates a perspective view of an embodiment of an imaging device120. In this example, the imaging device120comprises a plurality of cameras122, a plurality of 3D sensors124, a weight sensor126, a platform128, and a frame structure210. The imaging device120may be configured as shown inFIG.2or in any other suitable configuration. In some embodiments, the imaging device120may further comprise additional components, including, but not limited to, light, displays, and graphical user interfaces.

The platform128comprises a surface212that is configured to hold a plurality of items102. In some embodiments, the weight sensor126may be a distinct device from the imaging device126. In some embodiments, the platform128may be integrated with the weight sensor126. For example, the platform128may be positioned on the weight sensor126which allows the weight sensor126to measure the weight of items102that are placed on the platform128. As another example, the weight sensor126may be disposed within the platform128(such that the weight sensor126is integrated with the platform128) to measure the weight of items102that are placed on the platform128. In some embodiments, at least a portion of the surface212may be transparent. In this case, a camera122or scanner (e.g., a barcode scanner, a QR code scanner) may be disposed below the surface212of the platform218and configured to capture images104or scan the bottoms of items102placed on the platform128. For instance, a camera122or scanner may be configured to identify and read product labels, barcodes, and/or QR codes of items102through the transparent surface212of the platform128. The platform128may be formed of aluminum, metal, wood, plastic, glass, or any other suitable material.

The frame structure210may comprise a set of rails that are assembled to hold the cameras122and 3D sensors124. The frame structure210is generally configured to support and position cameras122and 3D sensors124. In the example ofFIG.2, the frame structure210is configured to position cameras122aand122bon one side of the platform128, a camera122con another side of the platform128, and cameras122dand122eon another side of the platform128. The cameras122ato122ehave perspective views of the platform128. The cameras122ato122eare configured to capture side or perspective images104of items102placed on the platform128. An example of a perspective image104of an item102is illustrated inFIG.3B.

In some embodiments, the frame structure128may further comprise one or more other cameras122(not shown) positioned on one or more other sides of the platform128. The frame structure210may be configured to use any number and combination of cameras122ato122e. For example, one or more of the identified cameras122may be optional and omitted.

The frame structure210is further configured to position a camera122fabove the platform128. The cameras122fmay be configured to capture top-view images104of the platform128. In some embodiments, the frame structure210may further comprise one or more other cameras122(not shown) above the platform128to capture top-view images104of items102placed on the platform128.

Similarly, the frame structure210may comprise 3D sensors124ato124fpositioned on sides and above of the platform128as illustrated inFIG.2. In the example ofFIG.2, the frame structure210is configured to position 3D sensors124aand124bon one side of the platform128, a 3D sensor124con another side of the platform128, and 3D sensors124dand124eon another side of the platform128. A 3D sensor124may be integrated with a camera122or be separate.

Each of the 3D sensors124ato124eis configured to capture side depth images106of items102placed on the platform128. The 3D sensor124fmay be configured to capture top-view depth image106of items102placed on the platform128.

Each of a perspective image104and a perspective depth image106is configured to capture the side-facing surfaces of items102placed on the platform128. An example of a top-view depth image106of an item102is described in conjunction withFIGS.3A and3B. Each of a top-view or overhead image104or depth image106is configured to capture upward-facing surfaces of items102placed on the platform128. An example of a perspective image104of an item102is described in conjunction withFIG.3B.

In other examples, the frame structure210may be configured to support and position any other suitable number and combination of cameras122and 3D sensors124on any position with respect to the platform128. The frame structure210may be formed of aluminum, metal, wood, plastic, or any other suitable material.

Additional details of the imaging device120are disclosed in U.S. patent Ser. No. 17/362,261 entitled, “ITEM IDENTIFICATION USING DIGITAL IMAGE PROCESSING” which is hereby incorporated by reference herein as if reproduced in its entirety.

FIGS.3A and3Billustrate example top-view depth images106of the platform128before and after an item102is placed on the platform128.FIG.3Aillustrates a top-view depth image106aof the platform128captured by the 3D sensor124f(seeFIG.2) before an item102is placed on the platform128.

The depth image106ashows a substantially constant point cloud indicating that there are no items102on the platform128. Substantially constant point cloud means that there no, minimal, or less than a threshold difference between values that represent colors of the cloud of points in the depth image106a. The depth image106acorresponds to a reference depth image106that is captured with no items102are placed on the platform128. The item tracking engine144may use the reference depth image106to compare with subsequent depth images106and determine whether an item102is placed on the platform128.

FIG.3Aillustrates a top-view depth image106bof the platform128captured by the 3D sensor124f(seeFIG.2) after an item102is placed o the platform128. In this example, the colors or pixel values within the depth images106represent different depth values. In depth image106b, the different depth values correspond with the item102that is placed on the platform128.

FIG.3Billustrates an example perspective image104of an item102detected on the platform128. The image104may be captured by any of the cameras122described inFIG.2. The item tracking engine144may implement a neural network, e.g., the machine learning algorithm156to crop the image104such that the background of the image104is suppressed or minimized. This process is described in detail further below in conjunction with the operational flow1400described inFIG.14.

FIG.4illustrates an example embodiment of the training dataset154. Aspects of the training dataset154are described inFIG.1, and additional aspects are described below. In the example ofFIG.4, assume that an item102ais placed on the platform128of the imaging device120. The imaging device120capture images104of the item102ausing the cameras122. The imaging device120sends the images104to the server140for processing. The item tracking engine144implements the machine learning algorithm156to extract features158from each image104. An image104captured from each camera122may be added in a new entry130in the training dataset154. In the example ofFIG.4, the item tracking engine144extracts features158a-1from the image104a-1. The features158a-1may be represented by the feature vector134a-1that comprises a set of numerical values. The item tracking engine144extracts features158a-2from the image104a-2. The features158a-2may be represented by the feature vector134a-2that comprises a set of numerical values. The item tracking engine144extracts features158a-nfrom the image104a-n. The features158a-nmay be represented by the feature vector134a-nthat comprises a set of numerical values. Each image104may be labeled or associated with one or more annotations136, similar to that described inFIG.1.

Example Method for Adding Items to the Training Dataset of an Item Identification Model

FIG.5illustrates an example flowchart of a method500for adding items102to the training dataset154of an item identification model152. Modifications, additions, or omissions may be made to method500. Method500may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system100, processor142, item tracking engine144, imaging device120or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method500. For example, one or more operations of method500may be implemented, at least in part, in the form of software instructions150ofFIG.1, stored on non-transitory, tangible, machine-readable media (e.g., memory148ofFIG.1) that when run by one or more processors (e.g., processor142ofFIG.1) may cause the one or more processors to perform operations502-514.

Method500may begin at502where the item tracking engine144may determine that an item102is not included in the training dataset154of the item identification model152. For example, the item tracking engine144may determine that the item102is not included in the training dataset154if it is determined that no images104of the item102are included in the training dataset154, similar to that described inFIG.1.

At502, the item tracking engine144obtains an identifier132associated with the item102. For example, the item tracking engine144may obtain a scan of a barcode of the item102, similar to that described inFIG.1.

At504, the item tracking engine144determines whether a triggering event108is detected. The triggering event108may correspond to a user placing the item102on the platform128. Various embodiments of determining whether a triggering event108is detected are described inFIG.1. If the item tracking engine144determines that the triggering event108is detected, method500proceeds to508. Otherwise, method500remains at506until it is determined that the triggering event108is detected.

At508, the imaging device120captures images104of the item102, e.g., using the cameras122. For example, the item tracking engine144may send a signal to the imaging device120to capture images104of the item102. The imaging device120may send the images104to the server140.

At510, the item tracking engine144extracts a set of features158associated with the item102from the images104. In this process, the item tracking engine144may feed each image104to the machine learning algorithm156to extract features158associated with the item102, similar to that described inFIG.1. Similarly, the item tracking engine144may extract the set of features158from depth images106of the item102.

At512, the item tracking engine144associates the item102to the identifier132and the set of features158.

At514, the item tracking engine144adds a new entry130for the item102to the training dataset154.

In certain embodiments, the item tracking engine144may be configured to remove an item102from the training dataset154. For example, if an item102is removed from a physical store, the item102may be removed from the training dataset154.

Example System for Capturing Images for Training an Item Identification Model

FIG.6illustrates one embodiment of a system600that is configured to capture images104and/or depth images106for training an item identification model152. In one embodiment, system600comprises the server140. In some embodiments, system600further comprises the network110, an imaging device620, and a weight sensor626. In other embodiments, system600may not have all of the components listed and/or may have other elements instead of, or in addition to, those listed above. Aspects of certain components of the system600are described above inFIGS.1-5, and additional aspects are described below. The network110enabled communication between components of the system600. Server140comprises the processor142in signal communication with the memory148. Memory148stores software instructions610that when executed by the processor142, cause the processor142to perform one or more functions described herein. For example, when the software instructions610are executed, the processor142executes the item tracking engine144to detect one or more items102placed on the platform628, and add a new entry for each detected item102to the training dataset154. This operation is described further below in conjunction with an operational flow of the system600and method900described inFIG.9.

The system600may further be configured to aggregate corresponding features158of an item102extracted from different images104of the item102and add the aggregated value for the feature158to a training dataset154of the item identification model152. The system600may perform a similar operation for each corresponding feature158such as: 1) one or more dominant colors of an item102; 2) a dimension of an item102; 3) a weight of an item102; and 4) any other feature158of an item102described inFIG.1. This operation is described further below in conjunction with an operational flow1000of the system600described inFIG.10and method1100described inFIG.11.

System Components

Example Imaging Device

Imaging device620is generally configured to capture images104and depth images106of items102that are placed on the platform628of the imaging device620. In one embodiment, the imaging device620comprises one or more cameras622, one or more 3D sensors624, and a platform628. Example embodiments of hardware configurations of the imaging device620are described inFIGS.7and8.

In certain embodiments, each of the cameras622and 3D sensors624may correspond to and/or be an instance of camera122and 3D sensor124described inFIG.1, respectively.

The platform628comprises a surface on which items102can be placed. In certain embodiments, the platform628may comprise a surface that is configured to rotate, such as a turntable.

In certain embodiments, the imaging device620may further include a weight sensor626. The weight sensor626may be integrated within the platform628, similar to that described inFIGS.1and2with respect to the weight sensor126. In certain embodiments, the weight sensor626may be a distinct device from the imaging device620. The weight sensor626may correspond to and/or be an instance of the weight sensor126described inFIGS.1and2.

In an embodiment where the weight sensor626is distinct from the imaging device620, the weight sensor626may be placed underneath a board, platform, or a surface where items102can be placed.

The items102can be weighted by the weight sensor626. The weight sensor626is configured to detect a weight162of an item102. The weight sensor626sends the detected weight162to the server140.

Aspects of the server140are described inFIG.1, and additional aspects are described below. The memory148is further configured to store the software instructions610, images104, depth images106, item identification model152, training dataset154, identifier132, features158, machine learning algorithm156, image capturing operation630, triggering event108, weights162, threshold area632, signal634, values1002a,1002b, and1002n, threshold percentage636, and particular number638. The particular number638may represent a number of degrees, such as two, five, ten, or any other number.

Operational Flow for Capturing Images for Training an Item Identification Model

In an example operation, the operational flow of system600may include operations to capture one or more images104and/or depth images106of an item102for training the item identification model152.

In one embodiment, the operational flow of system600may begin when the item tracking engine144obtains an identifier132associated with the item102. The identifier132associated with the item102may include a barcode, a QR code, a product label of the item102. For example, the item tracking engine144may obtain the identifier132of the item102when a user scans the barcode of the item102by using a barcode scanner, similar to that described inFIG.1.

The item tracking engine144may detect a triggering event108at the platform628. The triggering event108may correspond to a user placing the item102on the platform628. Various embodiments of detecting the triggering event108are described above inFIG.1.

Capturing Image(s) of the Item

The item tracking engine144may execute an image capturing operation630to capture image(s)104and/or depth image(s)106of the item102. In this operation, the item tracking engine144may cause the platform628to rotate (as illustrated inFIG.7).

For example, by executing the image capturing operation630, the item tracking engine144may send a signal634to the imaging device620, where the signal634includes instructions to rotate the platform628. In one embodiment, the platform628may rotate in an x-y plane. In certain embodiments, the platform628may rotate one degree at a time until the platform628is fully rotated once.

Further, by executing the image capturing operation630, a signal may be sent to cameras622to capture images104of the item102while the platform628is rotating.

In one embodiment, each camera622may capture one image104of the item102at each degree of rotation of the platform628. For example, at degree=0, each camera622may capture one image104of the item102; at degree=1, each camera622may capture one image104of the item102; and so on until one full turn of the platform628. Thus, in one embodiments, each camera622may capture three hundred sixty images104of the item102.

In another embodiment, each camera622may capture one image104of the item102at each plurality of degrees of rotation of the platform628, e.g., every two degrees, every five degrees, or any suitable number of degrees. In certain embodiments, one or more captured images104may be optional and omitted.

In one embodiment, the platform628may rotate a particular number of degrees at a time. The particular number638of degrees may be two, five, ten, or any other number. In one embodiment, one or more cameras622may not be triggered to capture an image104of the item102.

The item tracking engine144may perform a similar operation for 3D sensors624. Thus, the image capturing operation630may include capturing depth images106of the item102while the platform628is rotating.

For example, by executing the image capturing operation630, a signal may be sent to 3D sensors624to capture depth images106of the item102while the platform628is rotating.

Each 3D sensor624may capture one depth image106of the item102at each degree of the rotation of the platform628.

Thus, in one embodiment, each 3D sensor624may capture three hundred sixty depth images106of the item102. In another embodiment, each 3D sensor624may capture one depth image106of the item102at each plurality of degrees of rotation of the platform628, e.g., every two degrees, every five degrees, or any suitable number of degrees. In certain embodiments, one or more captured depth images106may be optional and omitted.

Determining an Orientation of the Item

In one embodiment, the item tracking engine144may be configured to determine an orientation of the item102with respect to the platform628.

In this process, the item tracking engine144may cause a 3D sensor624to capture a depth image106of the item102while the platform628is turning, similar to that described above. For example, the item tracking engine144may cause the 3D sensor624f(seeFIG.7) to capture an overhead depth image106of the item102. The overhead depth image106may be configured to capture upward-facing surfaces of the item102on the platform628. The 3D sensor624may capture the depth image106of the item102. The imaging device620may send the depth image106to the server140for processing.

The item tracking engine144may determine an orientation of the item102with respect to the platform628based on the depth image106, as described below.

The orientation of the item102may be vertical or horizontal with respect to the platform628. For example, the item tracking engine144may determine whether the item102is positioned in a vertical orientation (e.g., standing position) or in a horizontal orientation with respect to the platform628. In the vertical orientation, features158of an item102are primarily in the vertical orientation. In the horizontal orientation, features158of an item102are primarily in the horizontal orientation. Thus, cameras622with top-views of the platform628may be better suited for capturing images104of the item102.

If the item tracking engine144determines that the item102is positioned in a horizontal orientation with respect to the platform628, the item tracking engine144may determine that the orientation of the item102is longitudinal with respect to the platform628. In response, the item tracking engine144may cause a subset of cameras622that are on top of the platform628to capture overhead images104of the item102on the platform628.

In one embodiment, the item tracking engine144may determine the orientation of an item102based on a pose of the item detected from the depth image106, e.g., standing or laid down.

The item tracking engine144may use an area of the item102to determine the orientation of the item102. Referring toFIG.3Aas an example, the item tracking engine144may determine the area302of the item102. The item tracking engine144may compare the determined area302with a threshold area632(seeFIG.6). The item tracking engine144may determine that the item102is in vertical orientation if it is determined that the determined area302is less than or equal to the threshold area632(seeFIG.6). Otherwise, the item tracking engine144may determine that the item102is in a horizontal orientation when the determined area302is more than the threshold area632(seeFIG.6). In the example ofFIG.3A, the item tracking engine144determines that the item102is in vertical orientation because the area302is less than the threshold area632(seeFIG.6).

Extracting Features of the Item from Each Image and Adding a New Entry for Each Image

Referring back toFIG.6, The item tracking engine144may extract a set of features158from each image104of the item102, where each feature158corresponds to a physical attribute of the item102, similar to that described inFIG.1. The item tracking engine144associates the item102to the identifier132and the set of features158. The item tracking engine144adds a new entry130to the training dataset154, where the new entry130may represent the item102labeled with the identifier132and the set of features158.

In some embodiments, the item102in the new entry130may further be labeled with a feature vector134and/or annotations136, similar to that described inFIG.1.

In one embodiment, the item tracking engine144may be configured to associate the item102with a weight162. In this operation, the item tracking engine144may receive a plurality of weights162of multiple instances of the item102. For example, multiple instances of the item102may be placed on the weight sensor626and weighed by the weight sensor626.

The item tracking engine144may determine a mean of the weights162of the multiple instances of the item102. The item tracking engine144may associate the mean of the weights162of the multiple instances of the item102to the item102. The item tracking engine144may add the mean of the weights162of the item102to the new entry130in the training dataset154, e.g., in the annotations136.

Example Imaging Device

FIG.7illustrates a perspective view of an embodiment of an imaging device620. In this example, the imaging device620comprises a plurality of cameras622, a plurality of 3D sensors624, a platform628, and a frame structure710. The imaging device620may be configured as shown inFIG.7, or in any other suitable configuration. In some embodiments, the imaging device620may further comprise additional components, including, but not limited to, light, displays, and graphical user interfaces.

The platform628comprises a surface712that is configured to hold one or more items102. In some embodiments, the platform628may be configured to rotate. For example, the platform628may rotate in an x-y plane around the z-axis at its center point. The platform628may be operably coupled to a circuit board714. The circuit board714may comprise a hardware processor (e.g., a microprocessor) in signal communication with a memory, and/or circuitry (not shown) configured to perform any of the functions or actions of the circuit board714described herein. For example, the circuit board714may be configured to rotate the platform628in response to receiving a signal634(seeFIG.6) from the item tracking engine144. The circuit board714may be communicatively coupled to the server140, for example, wirelessly (e.g., via WiFi, Bluetooth, other wireless communication protocols) and/or through wires. The platform628may receive a signal634(seeFIG.6) from the item tracking engine144, where the signal634may include electrical signals to cause the platform628to rotate.

In one embodiment, the platform628may rotate one degree at a time until the platform628is fully rotated once. In one embodiment, at least one camera622may be triggered to capture one image104of the item102on the platform628at each degree of rotation of the platform628.

In another embodiment, the platform628may rotate a particular number638of degrees at a time, e.g., every two degrees, every five degrees, or any other suitable number of degrees. In one embodiment, at least one camera622may be triggered to capture one image104of the item102on the platform628at each of a plurality of degrees of rotation of the platform628, e.g., every two degrees, every five degrees, or any other suitable number of degrees, similar to that described inFIG.6.

In one embodiment, at least one 3D sensor624may be triggered to capture one depth image106of the item102on the platform628at each degree of rotation of the platform628.

In another embodiment, at least one 3D sensor624may be triggered to capture one depth image106of the item102on the platform628at each of a plurality of degrees of rotation of the platform628, e.g., every two degrees, every five degrees, or any other suitable number of degrees, similar to that described inFIG.6.

In some embodiments, at least a portion of the surface712may be transparent. In this case, a camera622may be disposed below the surface712of the platform628and configured to capture images104of the bottom(s) of item(s) on the platform628. Similarly, a scanner (e.g., a barcode scanner, a QR code scanner) may be disposed below the surface712of the platform628and configured to scan the bottom(s) of the item(s)102on the platform628. For instance, a camera622and/or scanner may be configured to identify and read product labels, barcodes, and/or QR codes of items102through the transparent surface712of the platform628. The platform628may be formed of aluminum, metal, wood, plastic, glass, or any other suitable material.

The frame710may comprise a set of rails that are assembled to hold the cameras622and 3D sensors624. The frame710is generally configured to support and position cameras622and 3D sensors624. In the example ofFIG.7, the frame structure710is configured to position cameras622ato622f.

A first subset of cameras622may be positioned at one or more heights with respect to the platform628on a side of the platform628. In the example ofFIG.7, cameras622ato622care positioned at three different heights with respect to the platform628. The cameras622ato622care arranged vertically on a rail716. The rail716is on a side of the platform628adjacent to the platform628. The cameras622ato622chave perspective views of the platform628. Thus, the cameras622ato622care configured to capture perspective images104of item102placed on the platform628. In some embodiments, any number of cameras622may be placed on one or more rails716.

A second subset of cameras622may be positioned above the platform628. In the example ofFIG.7, cameras622dto622fare positioned above the platform628. The cameras622dto622fare arranged to form a triangle.

The cameras622dto622fhave top-views of the platform628. Thus, the cameras622dto622fare configured to capture overhead images104of item102placed on the platform628.

In some embodiments, any number and/or combination of cameras622may be positioned above the platform628.

The frame structure710may be configured to position 3D sensors624. In certain embodiments, any number and/or any combination of cameras622may be integrated with a 3D sensor624. In certain embodiments, a camera622and a 3D sensor624may be distinct devices.

In certain embodiments, the frame structure710may be configured to position 3D sensors624ato624f. A first subset of 3D sensors624may be positioned at one or more heights with respect to the platform628on a side of the platform628.

The first subset of 3D sensors624may have perspective views of the platform628. Thus, the first subset of 3D sensors624may be configured to capture perspective depth images106of item102placed on the platform628. In some embodiments, any number of 3D sensors624may be placed on one or more rail716.

A second subset of 3D sensors624may be positioned above the platform628. In the example ofFIG.7, 3D sensors624dto624fmay be positioned above the platform628. The second subset of 3D sensors624is arranged to form a triangle. The second subset of 3D sensors624have top-views of the platform628. Thus, the second subset of 3D sensors624may be configured to capture overhead depth images106of item102placed on the platform628. In some embodiments, any number and/or combination of 3D sensors624may be positioned above the platform628.

In other examples, the frame structure710may be configured to support and position any other suitable number and combination of cameras622and 3D sensors624. The frame structure710may be formed of aluminum, metal, wood, plastic, or any other suitable material.

FIG.8illustrates a perspective view of another embodiment of an imaging device620with an enclosure810. In this configuration, the enclosure810is configured to at least partially encapsulate the frame structure710, the cameras622, the 3D sensors624, and the platform628of the imaging device620. The frame structure710, the cameras622, the 3D sensors624, and the platform628may be similar to that described inFIGS.6and7.

In some embodiments, the enclosure810may be formed from a cloth material, a fabric, plastic alloys, and/or any other suitable material. The enclosure810is configured to provide a lighting condition for the interior of the imaging device620that is more than a threshold lighting condition quality. For example, the enclosure810may provide a brightness that is more than a threshold brightness level.

Example Method for Capturing Images for Training an Item Identification Model

FIG.9illustrates an example flowchart of a method900for capturing images104and/or depth images106for training an item identification model152. Modifications, additions, or omissions may be made to method900. Method900may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system600, processor142, item tracking engine144, imaging device620or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method900. For example, one or more operations of method900may be implemented, at least in part, in the form of software instructions610ofFIG.6, stored on non-transitory, tangible, machine-readable media (e.g., memory148ofFIG.6) that when run by one or more processors (e.g., processor142ofFIG.6) may cause the one or more processors to perform operations902-914.

Method900begins at902where the item tracking engine144obtains an identifier132associated with the item102. For example, the item tracking engine144may obtain a scan of a barcode of the item102, similar to that described inFIGS.1and6.

At904, the item tracking engine144determines whether a triggering event108is detected. The triggering event108may correspond to a user placing the item102on the platform128. Various embodiments of determining whether a triggering event108is detected are described inFIGS.1and6. If the item tracking engine144determines that the triggering event108is detected, method900proceeds to906. Otherwise, method900remains at904until it is determined that the triggering event108is detected.

At906, the item tracking engine144causes the platform628to rotate. For example, the item tracking engine144may transmit a signal634to the circuit board714of the platform628, where the signal634includes electrical signals to rotate the platform628, similar to that described inFIGS.6and7. In one example, the signal634may include instructions to rotate the platform628one degree at a time. In response, the platform628may rotate one degree at a time until one full rotation. In another example, the signal634may include instructions to rotate the platform628a particular number638of degrees at a time, e.g., every two degrees, every five degree, or any other suitable number of degrees. In response, the platform628may rotate the particular number638of degrees at a time until one full rotation.

At908, the item tracking engine144causes one or more cameras622to capture one or more images104of the item102placed on the platform628. In one example, one or more cameras622may be triggered to capture one image104of the item102on the platform628at each degree of the rotation of the platform628, based on the instructions included in the signal634. Similarly, one or more 3D sensors624may be triggered to capture one depth image106of the item on the platform628at each degree of the rotation of the platform628. In another example, one or more cameras622may be triggered to capture one image104of the item102on the platform628at each of a plurality of degrees of rotation of the platform628based on the instructions included in the signal634. Similarly, one or more 3D sensors624may be triggered to capture one depth image106of the item on the platform628at each of the plurality of degrees of rotation of the platform628.

At910, the item tracking engine144extracts a set of features158associated with the item102from the one or more images104. For example, the item tracking engine144may feed the one or more images104to the machine learning algorithm158to extract the set of features158of the item102, similar to that described inFIGS.1to5. Similarly, the item tracking engine144may extract the set of features158from depth images106of the item102. Examples of the set of features158are described inFIGS.1to5.

At912, the item tracking engine144adds a new entry130for the item102to the training dataset154of the item identification model152. The new entry130may be used to later identify the item102, similar to that described inFIGS.1to5.

Operational Flow for Identifying Items Based on Aggregated Metadata

FIG.10illustrates an example of an operational flow1000of the system600ofFIG.6for identifying items102based on aggregated metadata. As discussed inFIG.6, system600may be configured to identify items102based on aggregated metadata. The aggregated metadata may include aggregated features158captured from different images104of an item102placed on the platform628.

As described inFIGS.6to9, multiple images104may be captured of the item102placed on the platform628while the platform628is rotating. Each image104of the item102may be from a different angle and show a different side of the item102. Thus, the item tracking engine144may extract a different set of features158from each image104of the item102. Thus, system600may be configured to aggregate features158from the different sets of features158to produce a more accurate representation and description of the item102. This operation is described below in conjunction with the operational flow1000of the system600described inFIG.6and method1100described inFIG.11.

The operational flow1000begins when the item tracking engine144obtains a plurality of images104of an item102(e.g., item102a).

Extracting a Set of Features from Each Image of the Item

The item tracking engine144may obtain the plurality of images104of the item102afrom the imaging device520. In the example ofFIG.10, the item tracking engine144obtains images104a,104b,104n, among other images104of the item102a.

The item tracking engine144may feed each image104of the item102ato the machine learning algorithm156to extract a set of features158associated with the item102afrom the image104. For example, the item tracking engine144may extract a first set of features158a-1from the first image104aof the item102a, where the first set of features158a-1may be represented by a first feature vector134a-1. Similarly, the item tracking engine144may extract a second set of features158a-2from the second image104bof the item102b, where the second set of features158a-2may be represented by a second feature vector134a-2; and extract an n-th set of features158a-nfrom the n-th image104nof the item102a, where the n-th set of features158a-nmay be represented by an n-th feature vector134a-n.

Aggregating Corresponding Features from Different Feature Vectors

The item tracking engine144may perform the following operations for each feature158of the item102a. The item tracking engine144may identify a first feature158of the item102ain each feature vector134a-1,134a-2, and134a-n. For example, the first feature158of the item102amay be one or more dominant colors, a dimension, a weight, a shape, a logo, or any other feature158described inFIG.1.

The item tracking engine144may identify a first value1002aof the first feature158of the item102afrom the first image104a. The first value1002aof the first feature158may be represented by an array of numerical values, such as [a, . . . , n], where “a” and “n” represent numerical values.

Similarly, the item tracking engine144may identify a second value1002bof the first feature158of the item102afrom the second image104b. The second value1002bof the first feature158may be represented by an array of numerical values, such as [b, . . . , m], where “b” and “m” represent numerical values.

Similarly, the item tracking engine144may identify an n-th value1002nof the first feature158of the item102afrom the n-th image104n. The n-th value1002nof the first feature158of the item102amay be represented by an array of numerical values, such as [c, . . . , o], where “c” and “o” represent numerical values. The item tracking engine144may identify other values1002of the first feature158from other images104of the item102.

The item tracking engine144may determine an aggregated value1004for the first feature158of the item102aby aggregating two or more of the values1002a,1002b,1002n, and other values1002of the first feature158. The item tracking engine144may associate the item102awith the aggregated value1004for the first feature158.

The item tracking engine144may add a new entry130for each image104to the training dataset154(seeFIG.6), similar to that described inFIGS.1,5,6, and9. The item tracking engine144may add the aggregated value1004for the first feature158to the new entry130. The item tracking engine144may perform a similar operation for each feature158of the item102a.

For example, with respect to a second feature158of the item102a, the item tracking engine144may identify a first value1002aof the second feature158of the item102ain the first feature vector134a-1, a second value1002bof the second feature158of the item102ain the second feature vector134a-2, an n-th value1002nof the second feature158of the item102ain the n-th feature vector134a-n, among other values1002of the second feature158of the item102ain other feature vectors134extracted from other images104of the item102a. The item tracking engine144may determine an aggregated value1004for the second feature158by aggregating two or more values1002of the second feature158of the item102a.

The item tracking engine144may add the aggregated value1004for the second feature158to the new entry130in the training dataset154. This information may be used for identifying the item102a.

The operation of aggregating the values1002of a feature158may vary depending on the feature158. Various use cases of aggregating the values1002of a feature158are described below.

Case where the Feature is One or More Dominant Colors of the Item

In a case where the feature158is one or more dominant colors of the item102a, the item tracking engine144may perform one or more operations below to aggregate the one or more dominant colors detected from different images104of the item102a.

The item tracking engine144may identify one or more first dominant colors of the item102afrom the first image104aof the item102a. Each dominant color may be determined based on determining a number of pixels (with the dominant color) that is higher than other pixels (with other colors).

In one embodiment, the item tracking engine144may identify a particular number166of dominant colors, e.g., three, five, or any suitable number of dominant colors, by implementing the machine learning algorithm156. To this end, the item tracking engine144may determine pixel colors that illustrate the item102ain the first image104a, determine percentages of numbers of pixels based on their colors, rank them in descending order, and determine the top particular number166of dominant colors, similar to that described inFIG.1.

The item tracking engine144may determine a percentage of a particular dominant color of the item102ain the image104aby determining a ratio of a number of pixels that have the particular dominant color in relation to the total number of pixels illustrating the item102ain the image104a.

In one embodiment, the item tracking engine144may identify one or more dominant colors that have percentages of a number of pixels more than a threshold percentage164, for example, by implementing the machine learning algorithm156, similar to that described inFIG.1.

In this process, the item tracking engine144may determine pixel colors that illustrate the item102ain the first image104a, determine percentages of numbers of pixels based on their colors, rank them in descending order, and determine one or more dominant colors of the item102athat have percentages of a number of pixels more than a threshold percentage164, e.g., more than 40%, 45%, etc.

The item tracking engine144may perform a similar operation for determining one or more dominant colors of the item102afrom the second image104a, n-th image104n, and other images104of the item102a.

The item tracking engine144may cluster the dominant colors detected in the images104a,104b,104n, and other images104of the item102a. In one embodiment, the item tracking engine144may determine the one or more dominant colors of the item102aby determining which dominant colors from among the dominant colors detected in the images104have percentages more than a threshold percentage636, e.g., more than 40%, 45%, etc.

In an example scenario, assume that the item tracking engine144determines one or more first dominant colors of the item102afrom the first image104aof the item102a, and one or more second dominant colors of the item102afrom the second image104bof the item102a. The item tracking engine144may determine which dominant colors from among the one or more first dominant colors and the one or more second dominant colors have percentages more than the threshold percentage636. The item tracking engine144may perform a similar operation for dominant colors detected in other images104of the item102a.

In one embodiment, the item tracking engine144may determine a particular number166of dominant colors of the item102aby determining the top particular number of dominant colors from among the dominant colors detected in the images104.

In this manner, the item tracking engine144may determine the one or more overall dominant colors of the item102adetected in different images104of the item102aby clustering the dominant colors detected in different images104of the item102a. The item tracking engine144may associate the one or more detected dominant colors to the item102a. The item tracking engine144may add the one or more detected dominant colors to the new entry130. This information may be used for identifying the item102a.

Case where the Feature is a Weight of the Item In a case where the feature158is a weight162of the item102a, the item tracking engine144may perform one or more operations below to aggregate multiple weights162of multiple instances of the item102a.

The item tracking engine144may receive a plurality of weights162of multiple instances of the item102a. For example, the item tracking engine144may receive a plurality of weights162of multiple instances of the item102awhen a user places the multiple instances of the item102a(e.g., five, six, or any number of instances of the item102a) on the weight sensor626(seeFIG.60) and the weight sensor626(seeFIG.6) measure the overall weights162of the multiple instances of the item102a.

The weight sensor626(seeFIG.6) transmits the measured weights162of the multiple instances of the item102ato the server140. The item tracking engine144may determine a mean of the plurality of weights162of the multiple instances of item102a.

The item tracking engine144may associate the mean of the plurality of weights162of the multiple instances of the item102ato the item102a. The item tracking engine144may add the mean of the plurality of weights162of the multiple instances of the item102ato the new entry130. This information may be used for identifying the item102a.

Case where the Feature is the Dimension of the Item

In a case where the feature158is a dimension of the item102a, the item tracking engine144may perform one or more operations below to aggregate multiple dimensions of the item102adetected from multiple images104.

As discussed inFIG.1, the dimension of the item102amay be represented by a length, a width, and a height of the item102a. Since different images104of the item102ashow different sides of the item102a, multiple dimensions of the item102amay be measured from multiple images104of the item102a. For example, the item tracking engine144(e.g., via the machine learning algorithm156) may measure a first dimension of the item102afrom the first image104a, a second dimension of the item102afrom the second image104b, an n-th dimension of the item102afrom the n-th image104n, and other dimensions of the item102afrom other images104.

The item tracking engine144may determine the dimension of the item102aby determining a mean of the multiple dimensions of the item102ameasured from multiple images104of the item102a. The item tracking engine144may associate the mean of multiple dimensions of the item102ato the item102a. The item tracking engine144may add the mean of the multiple dimensions of the item102ato the new entry130. This information may be used for identifying the item102a.

Case where the Feature is a Mask Around the Item

In a case where the feature158is a mask that defines a contour around the item102a, the item tracking engine144may perform one or more operations below to aggregate masks of the item102adetected in multiple images104of the item102a.

The item tracking engine144may identify multiple masks around the item102afrom multiple images104of the item102a. For example, the item tracking engine144may identify a first mask that defines a first contour around the item102ain the first image104a, a second mask that defines a second contour around the item102a, and other masks around the item102afrom other images104.

The item tracking engine144may compare the first mask with the second mask. The item tracking engine144may determine differences between the first mask (detected in the first image104a) and the second mask (detected in the second image104b).

Based on the determined differences between the first mask and second mask, the item tracking engine144may determine at least a portion of a three-dimensional mask around the item102a.

The item tracking engine144may perform a similar operation for every two adjacent images104. For example, the item tracking engine144may determine a first set of differences between the first mask (detected in the first image104a) and the second mask (detected in the second image104b); a second set of differences between the second mask (detected in the second image104b) and a third mask (detected in a third image104); and so on. The item tracking engine144may combine the multiple masks of the item102adetected from different images104.

The item tracking engine144may determine a three-dimensional mask around the item102abased on the differences between the multiple masks of the item102a, and the combined masks of the item102a. The item tracking engine144may associate the three-dimensional mask of the item102ato the item102a. The item tracking engine144may add the three-dimensional mask of the item102ato the new entry130. This information may be used for identifying the item102a. The item tracking engine144may identify the item102abased on the features158associated with the item102a, similar to that described inFIG.1.

In one embodiment, the item tracking engine144may determine the three-dimensional mask around the item102aif the item tracking engine144fails to identify the item102ausing one or more two-dimensional masks. In other words, determining the three-dimensional mask around the item102ais in response to determining that the item102ais not identified based on the two-dimensional mask of the item102a.

Example Method for Identifying Items Based on Aggregated Metadata

FIG.11illustrates an example flowchart of a method1100for identifying items102based on aggregated metadata. Modifications, additions, or omissions may be made to method1100. Method1100may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system600, processor142, item tracking engine144, imaging device620, or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method1100. For example, one or more operations of method1100may be implemented, at least in part, in the form of software instructions610ofFIG.6, stored on non-transitory, tangible, machine-readable media (e.g., memory148ofFIG.6) that when run by one or more processors (e.g., processor142ofFIG.6) may cause the one or more processors to perform operations1102-1116.

Method1100begins at1102where the item tracking engine144obtains a plurality of images104of an item102. The item tracking engine144may obtain the plurality of images104of the item102from the imaging device520, similar to that described inFIGS.6and10.

At1104, the item tracking engine144extracts a set of feature158associated with the item102from each image of the plurality of images104. For example, the item tracking engine144may feed each image104to the machine learning algorithm156to extract a set of features158, similar to that described inFIGS.1and10. Similarly, the item tracking engine144may extract the set of features158from depth images106of the item102, similar to that described inFIGS.1and10. Examples of the set of features158are described inFIGS.1and10.

At1106, the item tracking engine144selects a feature158from among the set of features158. The item tracking engine144may iteratively select a feature158until no feature158is left for evaluation.

At1108, the item tracking engine144identifies a plurality of values1002that represent the feature158from each image104of the item102. For example, the item tracking engine144may identify a first value1002athat represents the feature158from the first image104a, a second value1002bthat represents the feature158from the second image104b, and so on, similar to that described inFIG.10.

At1110, the item tracking engine144aggregates the plurality of values1002that represents the feature158. The operation of aggregating the plurality of values1002of a feature158may vary depending on the feature158. Various use cases of aggregating the values1002of a feature158are described inFIG.10.

At1112, the item tracking engine144associates the item102with the aggregated plurality of values1002.

At1114, the item tracking engine144determines whether to select another feature158. The item tracking engine144may determine to select another feature158if at least one feature158is left for evaluation. If the item tracking engine144determines to select another feature158, method1100may return to1106. Otherwise, method1100may proceed to1116.

At1116, the item tracking engine144adds a new entry130for each image104to the training dataset154associated with the item identification model152. In this manner, the item tracking engine144may use aggregated metadata to identify the item102.

Example System for Refining an Item Identification Model Based on Feedback

FIG.12illustrates one embodiment of a system1200that is configured to refine an item identification model152based on feedback1220. In one embodiment, system1200comprises the network110, the imaging device120, the server140, and a computing device1210. Aspects of the network110, the imaging device120, and the server140are described inFIGS.1-5, additional aspects are described below. Network110enables the communication between components of the system1200. Server140comprises the processor142in signal communication with the memory148. Memory148stores software instructions1250that when executed by the processor142, cause the processor142to perform one or more functions described herein. For example, when the software instructions1250are executed, the processor142executes the item tracking engine144to refine the item identification model152based on feedback1220. In other embodiments, system1200may not have all of the components listed and/or may have other elements instead of, or in addition to, those listed above.

In an example scenario, assume that a user1202is adding an item102to a shopping cart at a store. The user1202may place the item102on the platform128of the imaging device120so the cameras122of the imaging device120can capture images104of the item102. The cameras122of the imaging device120capture images104of the item102. The imaging device120transmits the images104to the item tracking engine144. The item tracking engine144may feed the images104to the machine learning algorithm156of the item identification model152to identify the item102. In some cases, the item102in the captured images104may be obstructed by other items102. In some cases, the item102may not be completely shown in the images104. In such cases, the item102may be identified incorrectly by the item tracking engine144, for example, because features158of the item102extracted from the images104may not accurately describe the item102. Thus, the system1200may be configured to refine the item identification model152based on feedback1220. This operation is described in conjunction with the operational flow1300of the system1200described inFIG.13and method1500described inFIG.15.

In some cases, a captured image104of an item102may include a background portion that shows the area beside the item102. The background portion in the image104may cause the item tracking engine144to not be able to extract accurate features158of the item102. For example, additional information that is extracted from the background portion may reduce the accuracy of item identification. Thus, system1200may be configured to suppress or minimize the background section in an image104by performing a background suppression operation1402. This process is described in conjunction with the operational flow1400of the system1200described inFIG.14.

System Components

Aspects of the server140are described inFIGS.1-5, additional aspects are described below. The memory148is further configured to store the software instructions1250, feedback1220, background suppression operation1402, triggering event108, signal1214, percentages1414, and threshold values1416.

Computing device1210is generally any device that is configured to process data and interact with users. Examples of the computing device1210include, but are not limited to, a personal computer, a desktop computer, a workstation, a server, a laptop, a tablet computer, etc. The computing device1210may include a user interface, such as a display, a microphone, keypad, or other appropriate terminal equipment usable by a user. The computing device1210may include a hardware processor, memory, and/or circuitry configured to perform any of the functions or actions of the computing device1210described herein. For example, a software application designed using software code may be stored in the memory and executed by the processor to perform the functions of the computing device1210.

A graphical user interface1212may be accessed from the computing device1210. When one or more items102are placed on the platform128, the imaging device120may capture one or more images104and/or depth images106from the one or more items102. The imaging device120may transmit the captured images104and depth images106to the server140. The item tracking engine144may identify the one or more items102by feeding the captured images104and/or the depth images106to the machine learning algorithm156. The item tracking engine144may present the identified items102on the graphical user interface1212. A user1202can view the identified items102on the graphical user interface1212. The user1202may indicate, on the graphical user interface1212, whether each item102is identified correctly, for example, by pressing a button on the graphical user interface1212. Thus, the user1202can provide feedback1220indicating whether each item102is identified correctly. The feedback1220is transmitted to the server140from the computing device1210. The item tracking engine144may use the provided feedback1220to refine the item identification model152. This process is described in conjunction with the operational flow1300of system1200described inFIG.13and method1500described inFIG.15.

Operational Flow for Refining an Item Identification Model Based on Feedback

FIG.13illustrates an example of an operational flow1300of the system1200ofFIG.12for refining an item identification model152based on feedback1220.

Capturing Images of an Item

The operational flow1300may begin when the item tracking engine144detects a triggering event108at the platform128, similar to that described inFIG.1. In response, the imaging device120may capture one or more images104of one or more items102that are placed on the platform128of the imaging device120. As noted above, an item102may be obstructed by other items102in an image104or otherwise not fully visible in the image104. The imaging device120transmits the one or more images104of one or more items102to the server140.

The item tracking engine144may perform one or more operations below for each of the one or more images104. The item tracking engine144may feed the image104of the item102to the machine learning algorithm156of the item identification model152. The item tracking engine144may extract a set of features158associated with the item102from the image104.

Similarly, the imaging device120may capture one or more depth images106of the one or more items102placed on the platform128of the imaging device120. The imaging device120may transmit the one or more depth images106to the server140. The item tracking engine144may feed each of the one or more depth images106to the machine learning algorithm156, and extract the set of features158associated with the item102from each depth image106. The process of extracting a set of features158associated with the item102is described inFIG.1. The item tracking engine144may identify the item102based on the extracted set of features158, similar to that described inFIG.1.

Determining Whether the Item is Identified Correctly

The item tracking engine144may determine whether the item102is identified correctly. In this process, the item tracking engine144may present the identified item102on the graphical user interface1212. If the item tracking engine144receives a signal1214from the graphical user interface1212indicating that the item102is not identified correctly, the item tracking engine144determines that the item102is not identified correctly. If the item tracking engine144receives a signal1214from the graphical user interface1212indicating that the item102is identified correctly, the item tracking engine144determines that the item102is identified correctly.

For example, the graphical user interface1212may include a first button1216athat a user1202can press to indicate that the item102is identified correctly. In another example, the graphical user interface1212may include a second button1216bthat a user1202can press to indicate that the item102is not identified correctly.

If the item tracking engine144determines that the item102is identified correctly, the item tracking engine144may associate the item102to the user1202, for example, by adding the item102to the shopping cart associated with the user1202.

If the item tracking engine144determines that the item102is not identified correctly, the item tracking engine144may refine the item identification model152based on feedback1220, as described below.

Refining an Item Identification Model Based on Feedback

In a case where the item102is not identified correctly, the user1202can scan an identifier132of the item102. For example, the user1202can scan a barcode, a QR code, a label associated with the item102by a barcode scanner, a QR code scanner, or any other suitable type of scanner. The item tracking engine144may receive the identifier132of the item102.

The item tracking engine144may identify the item102based on the identifier132of the item102. The identifier132of the item102may be included in the feedback1220. The item tracking engine144may feed the identifier132of the item102and the one or more captured images104of the item102to the machine learning algorithm156of the item identification model152.

The item tracking engine144may retrain the machine learning algorithm156of the item identification model152to learn to associate the item102to the one or more captured images104of the item102. In this process, the item tracking engine144may update weight and bias values of perceptrons in neural network layers of the machine learning algorithm156. By doing so, the set of features158extracted from the one or more images104may be updated to present a more accurate representation of the item102even from images104where the item102is not fully visible, e.g., where at least a portion of the item102is obstructed by other items102and/or at least a portion of the item102is not captured in an image104.

Thus, the item tracking engine144may update the set of features158associated with the item102based on the determined association between the item102and the one or more images104.

Suppressing Background in an Image of an Item

FIG.14illustrates an example image104of an item102on which the item tracking engine144performs a background suppression operation1402by performing the operational flow1400. In some cases, a captured image104of an item102may show a background1408in addition to the item102. For a more optimal identification of the item102, it may be desired to reduce or minimize a portion of the image104where the background is shown. To this end, the item tracking engine144may perform a background suppression operation1402, as described below.

In this process, the item tracking engine144may determine a first number of pixels1410that illustrate the item102in the image104. In other words, the item tracking engine144may determine an area in the image104that shows the item102. Similarly, the item tracking engine144may determine an overall number of pixels1412that form the image104. Thus, the item tracking engine144may determine a second number of pixels (e.g., an area) where the background1408is shown.

The item tracking engine144may determine a percentage1414of the first number of pixels1410based on a ratio of the first number of pixels1410in relation to the overall number of pixels1412. The item tracking engine144may determine whether the percentage1414of the first number of pixels1410is less than a threshold percentage1416. The threshold percentage1416may be 80%, 85%, or any other suitable percentage.

If the item tracking engine144determines that the percentage1414of the first number of pixels1410is less than a threshold percentage1416, the item tracking engine144may crop at least a portion of the background1408in the image104until the percentage1414of the first number of pixels1410in relation to the overall number of pixels1412is more than the threshold percentage1416. In other words, the item tracking engine144may suppress the background1408until the percentage1414of the first number of pixels1410that illustrate the item102is more than the threshold percentage1416. Otherwise, the item tracking engine144may not need to further crop the image104.

Example Method for Refining an Item Identification Model Based on Feedback

FIG.15illustrates an example flowchart of a method1500for refining an item identification model152based on feedback1220. Modifications, additions, or omissions may be made to method1500. Method1500may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system1200, processor142, item tracking engine144, imaging device120or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method1500. For example, one or more operations of method1500may be implemented, at least in part, in the form of software instructions1650ofFIG.12, stored on non-transitory, tangible, machine-readable media (e.g., memory148ofFIG.12) that when run by one or more processors (e.g., processor142ofFIG.12) may cause the one or more processors to perform operations1502-1514.

Method1500begins at1502where the item tracking engine144determines whether a triggering event108is detected. The triggering event108may correspond to a user placing an item102on the platform128. Various embodiments of determining whether a triggering event108is detected are described inFIGS.1and6. If the item tracking engine144determines that the triggering event108is detected, method1500proceeds to1504. Otherwise, method1500remains at1502until it is determined that the triggering event108is detected.

At1504, the imaging device120captures one or more images104from an item102that is placed on the platform128of the imaging device120using the cameras122. Similarly, the imaging device120may capture one or more depth images106of the item102using 3D sensors124.

At1506, the item tracking engine144extracts a set of features158associated with the item102from the one or more images104. In this process, the item tracking engine144may feed each image104to the machine learning algorithm156to extract features158associated with the item102, similar to that described inFIG.1. Similarly, the item tracking engine144may extract the set of features158from depth images106of the item102. Examples of the set of features158are described inFIG.1.

At1508, the item tracking engine144identifies the item102based on the set of features158, similar to that described inFIG.1.

At1510, the item tracking engine144determines whether the item102is identified correctly. For example, the item tracking engine144may determine whether the item102is identified correctly based on a signal1214received from a graphical user interface1212, similar to that described inFIGS.12and13. For example, if the item tracking engine144receives a signal1214from the graphical user interface1212indicating that the item102is not identified correctly, the item tracking engine144determines that the item102is not identified correctly. Otherwise, if the item tracking engine144receives a signal1214from the graphical user interface1212indicating that the item102is identified correctly, the item tracking engine144determines that the item102is identified correctly. If it is determined that the item102is identified correctly, method1500proceeds to1512. Otherwise, method1500proceeds to1514.

At1512, the item tracking engine144associates the item102to the user1202. For example, the item tracking engine144may add the item102to a shopping cart associated with the user1202.

At1514, the item tracking engine144receives an identifier132of the item102. The identifier132of the item102may include a barcode, a QR code, a label associated with the item102. For example, the item tracking engine144may receive the identifier132of the item102when the user1202scans the identifier132of the item102by a barcode scanner, a QR code scanner, etc., communicatively coupled with the imaging device120and the server140, similar to that described inFIG.13.

At1516, the item tracking engine144feeds the identifier132and the one or more images106to the item identification model152. For example, the item tracking engine144may feed the identifier132and the one or more images106to the machine learning algorithm156of the item identification model152.

At1518, the item tracking engine144retrains the item identification model152to lean to associate the item102to the one or more images104. The item tracking engine144may also retrain the item identification model152to lean to associate the item102to one or more depth images106of the item102.

At1520, the item tracking engine144updates the set of features158based on the determined association between the item102and the one or more images104. Similarly, the item tracking engine144may update the set of features158based on the determined association between the item102and the one or more depth images106. In certain embodiments, method1500may further include operations to perform the background suppression operation1402, similar to that described inFIG.14.

While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated with another system or certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.

To aid the Patent Office, and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants note that they do not intend any of the appended claims to invoke 35 U.S.C. § 112(f) as it exists on the date of filing hereof unless the words “means for” or “step for” are explicitly used in the particular claim.