Patent Description:
For example, the products may be cases of beverage containers (e.g. cartons of cans and beverage crates containing bottles or cans, etc). There are many different permutations of flavors, sizes, and types of beverage containers delivered to each store. When building pallets, missing or mis-picked product can account for significant additional operating costs.

The loaded pallet(s) are then loaded on a truck, along with pallets for other stores. Misloaded pallets cause significant time delays to the delivery route since the driver will have to rearrange the pallets during the delivery process with potentially limited space in the trailer to maneuver. Extra pallets on trucks can also cause additional loading times to find the errant pallet and re-load it on the correct trailer.

At the store, the driver unloads the pallet(s) designated for that location. Drivers often spend a significant amount of time waiting in the store for a clerk to become available to check in the delivered product by physically counting it. During this process the clerk ensures that all product ordered is being delivered. The driver and clerk often break down the pallet and open each case to scan one UPC from every unique flavor and size. After the unique flavor and size is scanned, both the clerk and driver count the number of cases or bottles for that UPC. This continues until all product is accounted for on all the pallets. Clerks are typically busy.

<CIT> discloses generating an image-based identifier for a stretch wrapped loaded pallet based on images captured in association with application of stretch wrap to the loaded pallet.

<CIT> discloses an apparatus and method for acquiring an image of a pallet load.

<CIT> discloses systems and methods of tracking products during transport.

The improved delivery system provides improvements to several phases of the delivery process. Although these improvements work well when practiced together, fewer than all, or even any one of these improvements could be practiced alone to some benefit.

The improved delivery system facilitates order accuracy from the warehouse to the store by combining machine learning and computer vision software with a serialized (RFID/Barcode) shipping pallet. Pallet packing algorithms are based on the product mix and warehouse layout.

Electronic order accuracy checks are done while building pallets, loading pallets onto trailers and delivering pallets to the store. When building pallets, the delivery system validates the build to ensure the correct product SKUs are being loaded on the correct pallet according to the pick list. Once the pallet is built the overall computer vision sku count for that specific pallet is compared against the pick list for that specific pallet to ensure the pallet is built correctly. This may be done prior to the pallet being stretch wrapped thus mitigating the cost of unwrapping of the pallet to audit and correct. This also prevents shortages and overages at the delivery point thus preventing the driver from having to bring back excess or make additional trips to deliver missing product.

An optimized queue system may then be used to queue and load pallets onto the trailer in the correct reverse-stop sequence (last stop is loaded onto the trailer first). An electronic visual control showing which pallet is to be loaded on which trailer will be visible to the loader, e. g: Loading pallet #<NUM> on Dock #<NUM>.

The system will also decrease the time for the receiver at the delivery point (e.g. store) to check-in the product through a combination of checks that build trust at the delivery point. This is done through conveyance of the computer vision images of the validated SKUs on the pallet before it left the warehouse and upon delivery to the store. This can be a comparison of single images or a deep machine learning by having the image at the store also electronically identify the product SKUs. Delivery benefits include significantly reducing costs associated with waiting and checking product in at the store level and a verifiable electronic ledger of what was delivered for future audit.

The delivery system will utilize a mobile device that the driver or receiver will have that takes one or more still images of the pallet (for example, <NUM>, i.e. <NUM> on each side). The image(s) can then be compared electronically to the control picture from the warehouse and physically by the clerk. The clerk can electronically sign off that all product SKUs are there against their pick list. Different levels of receipt will be available for the clerk to approve. Validation at the store can be simple pallet serial scan via RFID/Barcode and GPS coordinates against the delivery, pallet image compare and/or sku validation through a machine learning computer vision algorithm called from the mobile device.

<FIG> is a high level view of a delivery system <NUM> including one or more distribution centers <NUM>, a central server <NUM> (e.g. cloud computer), and a plurality of stores <NUM>. A plurality of trucks <NUM> or other delivery vehicles each transport the products <NUM> on pallets <NUM> from one of the distribution centers <NUM> to a plurality of stores <NUM>. Each truck <NUM> carries a plurality of pallets <NUM> which may be half pallets, each loaded with a plurality of goods <NUM> for delivery to one of the stores <NUM>. A wheeled sled <NUM> is on each truck <NUM> to facilitate delivery of one of more pallets <NUM> of goods <NUM> to each store <NUM>. Generally, the goods <NUM> could be loaded on the half pallets <NUM>, full-size pallets, carts, or hand carts, or dollies - all considered "platforms" herein.

Each distribution center <NUM> includes one or more pick stations <NUM> each associated with a validation station <NUM>. Each validation station <NUM> is associated with a loading station <NUM>, such as a loading dock for loading the trucks <NUM>.

Each distribution center <NUM> may have a plurality of loading stations <NUM>. Each distribution center <NUM> includes a DC computer <NUM>. The DC computer <NUM> receives orders <NUM> from the stores <NUM> and communicates with a central server <NUM>. Each DC computer <NUM> receives orders and generates pick sheets <NUM>, each of which stores SKUs and associates them with pallet ids. Alternatively, the orders <NUM> can be sent from the DC computer <NUM> to the central server <NUM> for generation of the pick sheets <NUM>, which are synched back to the DC computer <NUM>.

Some or all of the distribution centers <NUM> may include a training station <NUM> for generating image information and other information about new products <NUM> which can be transmitted to the central server <NUM> for analysis and future use.

The central server <NUM> may include a plurality of distribution center accounts <NUM>, including DC1-DCn, each associated with a distribution center <NUM>. Each DC account <NUM> includes a plurality of store accounts <NUM>, including store <NUM>-store n. The orders <NUM> and pick sheets <NUM> for each store are stored in the associated store account <NUM>. The central server <NUM> further includes a machine learning model including a plurality of SKU files <NUM>, including SKU <NUM>-SKUn. The model is periodically synched to the DC computers <NUM>.

The SKU files <NUM> each contain information for a SKU. A "SKU" may be a single variation of a product that is available from the distribution center <NUM> and can be delivered to one of the stores <NUM>. For example, each SKU may be associated with a particular number of containers (e.g. 12pack) in a particular form (e.g. can v bottle), with particular packaging (cardboard vs reusuable plastic crate, etc), having a particular flavor, and a particular size (e.g. <NUM> ounces). This information is contained in each SKU file <NUM> along with the name of the product, a description of the product, dimensions of the product, and image information for the product. Each SKU file <NUM> may also include the weight of the product. Image information may be further decomposed into text and color information. It is also possible that more than one variation of a product may share a single SKU, such as where only the packaging, aesthetics, and outward appearance of the product varies, but the content and quantity is the same. For example, sometimes promotional packaging may be utilized, which would have different image information for a particular SKU. In general, all the SKU files <NUM> including their associated image information, may be generated through the training module <NUM>.

Referring also to the flowchart in <FIG>, an order <NUM> may be received from a store <NUM> in step <NUM>. As an example, an order <NUM> may be placed by a store employee using an app or mobile device <NUM>. The order <NUM> is sent to the distribution center computer <NUM> (or alternatively to the server <NUM>, and then relayed to the proper (e.g. closest) distribution center computer <NUM>). The distribution center computer <NUM> analyzes the order <NUM> and creates a pick sheet <NUM> associated with that order <NUM> in step <NUM>. The pick sheet <NUM> assigns each of the SKUs (including the quantity of each SKU) from the order. The pick sheet <NUM> specifies how many pallets <NUM> will be necessary for that order (as determined by the DC computer <NUM>). The DC computer <NUM> may also determine which SKUs should be loaded near one another on the same pallet <NUM>, or if more than one pallet <NUM> will be required, which SKUs should be loaded on the same pallet <NUM>. For example, SKUs that go in the cooler may be together on the same pallet (or near one another on the same pallet), while SKUs that go on the shelf may be on another part of the pallet (or on another pallet, if there is more than one). If the pick sheet <NUM> is created on the DC computer <NUM>, it is copied to the server <NUM>. If it is created on the server <NUM>, it is copied to the DC computer <NUM>.

<FIG> shows the pick station <NUM> of <FIG>. Referring to <FIG> and <FIG>, workers at the distribution center read the palled id (e.g. via rfid, barcode, etc) on the pallet(s) <NUM> on a pallet jack 24a (see screenshot of <FIG>), such as with a mobile device or a reader on the pallet jack 24a. Shelves may contain a variety of items <NUM> for each SKU, such as first product 20a of a first SKU and a second product 20b of a second SKU (collectively "products <NUM>"). A worker reading a computer screen or mobile device screen displaying from the pick sheet <NUM> retrieves each product <NUM> and places that product <NUM> on the pallet <NUM>. Alternatively, the pallet <NUM> may be loaded by automated handling equipment.

Workers place items <NUM> on the pallets <NUM> according to the pick sheets <NUM>, and report the palled ids to the DC computer <NUM> in step <NUM>. The DC computer <NUM> dictates merchandizing groups and sub groups for loading items 20a, b on the pallets <NUM> in order to make unloading easier at the store. In the example shown, the pick sheets <NUM> dictate that products 20a are on one pallet <NUM> while products 20b are on another pallet <NUM>. For example, cooler items should be grouped, and dry items should be grouped. Splitting of package groups is also minimized to make unloading easer. This makes pallets <NUM> more stable too.

After one pallet <NUM> is loaded, the next pallet <NUM> is brought to the pick station <NUM>, until all of the SKUs required by the pick sheet <NUM> are loaded onto as many pallets <NUM> as required by that pick sheet <NUM>. Pallets <NUM> are then loaded for the next pick sheet <NUM>. The DC computer <NUM> records the pallet ids of the pallet(s) <NUM> that have been loaded with particular SKUs for each pick sheet <NUM>. The pick sheet <NUM> may associate each pallet id with each SKU.

After being loaded, each loaded pallet <NUM> is validated at the validation station <NUM>, which may be adjacent to or part of the pick station <NUM>. As will be described in more detail below, at least one still image, and preferably several still images or video, of the products <NUM> on the pallet <NUM> is taken at the validation station <NUM> in step <NUM>. The pallet id of the pallet <NUM> is also read. The images are analyzed to determine the SKUS of the products <NUM> that are currently on the identified pallet <NUM> in step <NUM>. The SKUs of the products <NUM> on the pallet <NUM> are compared to the pick sheet <NUM> by the DC computer <NUM> in step <NUM>, to ensure that all the SKUs associated with the pallet id of the pallet <NUM> on the pick sheet <NUM> are present on the correct pallet <NUM>, and that no additional SKUs are present. Several ways are of performing the aforementioned steps are disclosed below.

First, referring to <FIG>, the validation station may include a CV/RFID semi-automated wrapper 66a with turntable <NUM> may be specially fitted with a camera <NUM> and rfid reader <NUM> (and/or barcode reader). The wrapper 66a holds a roll of translucent, flexible, plastic wrap or stretch wrap <NUM>. As is known, a loaded pallet <NUM> can be placed on the turntable <NUM>, which rotates the loaded pallet <NUM> as stretch wrap <NUM> is applied. The camera <NUM> may be a depth camera. In this wrapper 66a, the camera <NUM> takes at least one image of the loaded pallet <NUM> while the turntable <NUM> is rotating the loaded pallet <NUM>, prior to or while wrapping the stretch wrap <NUM> around the loaded pallet <NUM>. Images/video of the loaded pallet <NUM> after wrapping may also be generated. As used herein, "image" or "images" refers broadly to any combination of still images and/or video, and "imaging" means capturing any combination of still images and/or video. Again, preferably <NUM> to <NUM> still images, or video, are taken.

In one implementation, the camera <NUM> is recording video (or a continuously changing image), while the turntable <NUM> is rotating. When the camera <NUM> detects that the two outer ends of the pallet <NUM> are equidistant (or otherwise that the side of the pallet <NUM> facing the camera <NUM> is perpendicular to the camera <NUM> view), the camera <NUM> records a still image. The camera <NUM> can record four still images in this manner, one of each side of the pallet <NUM>.

The rfid reader <NUM> (or barcode reader, or the like) reads the pallet id (a unique serial number) from the pallet <NUM>. The wrapper 66a includes a local computer <NUM> in communication with the camera <NUM> and rfid reader <NUM>. The computer <NUM> can communicate with the DC computer <NUM> (and/or server <NUM>) via a wireless network card <NUM>. The image(s) and the pallet id are sent to the server <NUM> via the network card <NUM> and associated with the pick list <NUM> (<FIG>). Optionally, a weight sensor can be added to the turntable <NUM> and the known total weight of the products <NUM> and pallet <NUM> can be compared to the measured weight on the turntable <NUM> for confirmation. An alert is generated if the total weight on the turntable <NUM> does not match the expected weight.

As an alternative, the turntable <NUM>, camera <NUM>, rfid reader <NUM>, and computer <NUM> of <FIG> can be used without the wrapper. The loaded pallet <NUM> can be placed on the turntable <NUM> for validation only and can be subsequently wrapped either manually or at another station.

Alternatively, referring to <FIG> and <FIG>, the validation station can include the camera <NUM> and rfid reader <NUM> (or barcode reader, or the like) mounted to a robo wrapper 66b. As is known, instead of holding the stretch wrap <NUM> stationary and rotating the pallet <NUM>, the robo wrapper 66b travels around the loaded pallet <NUM> with the stretch wrap <NUM> to wrap the loaded pallet <NUM>. The robo wrapper 66b includes the camera, <NUM>, rfid reader <NUM>, computer <NUM> and wireless network card <NUM>.

<FIG> shows the robo wrapper 66b wrapping the loaded pallet <NUM> and items <NUM> with stretch wrap <NUM> (as is commonly used) and generating at least one image <NUM> of the loaded pallet <NUM>. The robo wrapper 66b travels around the loaded pallet <NUM> and generates at least one image <NUM> of the loaded pallet <NUM> prior to and/or while wrapping the loaded pallet <NUM>. Images of the loaded pallet <NUM> after wrapping may also be generated. Other than the fact that the robo wrapper 66b travels around the stationary loaded pallet <NUM>, the robo wrapper 66b operates the same as the wrapper 66b of <FIG>.

Alternatively, referring to <FIG>, the validation station can include a worker with a networked camera, such as on a mobile device <NUM> (e.g. smartphone or tablet) for taking one or more images <NUM> of the loaded pallet <NUM>, prior to wrapping the loaded pallet <NUM>. <FIG> is a screenshot of the app on the mobile device <NUM> instructing the user to take two still images of the long sides of the loaded pallets <NUM> (alternatively, the user could take video while walking around the pallet <NUM>). <FIG> is a screenshot of the app on the mobile device <NUM> on which the user can approve the image the user took. <FIG> is a screenshot of the app on the mobile device <NUM> indicating the quantity of products <NUM> of each sku that has been identified on the pallet <NUM>.

Other ways can be used to gather images of the loaded pallet. In any of the methods, the image analysis and/or comparison to the pick list is performed on the DC computer <NUM>, which has a copy of the machine learning model. Alternatively, the analysis and comparison can be done on the server <NUM>, locally on a computer <NUM>, or on the mobile device <NUM>, or on another locally networked computer.

As mentioned above, the camera <NUM> (or the camera on the mobile device <NUM>) can be a depth camera, i.e. it also provides distance information correlated to the image (e.g. pixel-by-pixel distance information or distance information for regions of pixels). Depth cameras are known and utilize various technologies such as stereo vision (i.e. two cameras) or more than two cameras, time-of-flight, or lasers, etc. If a depth camera is used, then the edges of the products stacked on the pallet <NUM> are easily detected (i.e. the edges of the entire stack and possibly edges of individual adjacent products either by detecting a slight gap or difference in adjacent angled surfaces). Also, the depth camera <NUM> can more easily detect when the loaded pallet <NUM> is presenting a perpendicular face to the view of the camera <NUM> for a still image to be taken.

However the image(s) of the loaded pallet <NUM> are collected, the image(s) are then analyzed to determine the sku of every item <NUM> on the pallet <NUM> in step <NUM> (<FIG>). Images and dimensions of all sides of every possible product, including multiple versions of each SKU, if applicable, are stored in the server <NUM>. If multiple still images or video are collected, then the known dimensions of the pallet <NUM> and the items <NUM> are used to ensure that every item <NUM> is counted once and only once. For example, the multiple sides of the loaded pallet <NUM> may be identified in the images first. Then, the layers of items <NUM> are identified on each side. The individual items <NUM> are then identified on each of the four sides of the loaded pallet <NUM>.

The package type of each item <NUM> is identified by the computer, such as reusable beverage crate, corrugated tray with translucent plastic wrap, or fully enclosed cardboard or paperboard box. The branding of each item <NUM> is also identified by the computer (e.g. a specific flavor from a specific manufacturer), such as by reading the images/text on the packaging. The packaging is identified first, thus narrowing the list of possible branding options to be identified. If one technique leads to an identification with more confidence, that result could take precedence over a contrary identification. For example, if the branding is determined with low confidence and the packaging is determined with high confidence, and the identified branding is not available in the identified packaging, the identified packaging is used and the next most likely branding that is available in the identified packaging is then used.

After individual items <NUM> are identified on each of the four sides of the loaded pallet <NUM>, based upon the known dimensions of the items <NUM> and pallet <NUM>, duplicates are removed, i.e. it is determined which items are visible from more than one side and appear in more than one image. If some items are identified with less confidence from one side, but appear in another image where they are identified with more confidence, the identification with more confidence is used.

For example, if the pallet <NUM> is a half pallet, its dimensions would be approximately <NUM> to approximately <NUM> inches by approximately <NUM> to approximately <NUM> inches, including the metric <NUM> x <NUM>. Standard size beverage crates, beverage cartons, and wrapped corrugated trays would all be visible from at least one side, most would be visible from at least two sides, and some would be visible on three sides.

If the pallet <NUM> is a full-size pallet (e.g. approximately <NUM> inches by approximately <NUM> inches, or <NUM> by <NUM>), most products would be visible from one or two sides, but there may be some products that are not visible from any of the sides. The dimensions and weight of the hidden products can be determined as a rough comparison against the pick list. Optionally, stored images (from the SKU files) of SKUs not matched with visible products can be displayed to the user, who could verify the presence of the hidden products manually.

The computer vision-generated sku count for that specific pallet <NUM> is compared against the pick list <NUM> to ensure the pallet <NUM> is built correctly. This may be done prior to the loaded pallet <NUM> being wrapped thus preventing unwrapping of the pallet <NUM> to audit and correct. If the built pallet <NUM> does not match the pick list <NUM> (step <NUM>), the missing or wrong SKUs are indicated to the worker (step <NUM>), e.g. via a display (e.g. <FIG>). Then the worker can correct the items <NUM> on the pallet <NUM> (step <NUM>) and reinitiate the validation (i.e. initiate new images in step <NUM>).

If the loaded pallet <NUM> is confirmed, positive feedback is given to the worker (e.g. <FIG>), who then continues wrapping the loaded pallet <NUM> (step <NUM>). Additional images may be taken of the loaded pallet <NUM> after wrapping. For example, four image may be taken of the loaded pallet before wrapping, and four more images of the loaded pallet <NUM> may be taken after wrapping. All images are stored locally and sent to the server <NUM>. The worker then moves the validated loaded pallet <NUM> to the loading station <NUM> (step <NUM>).

After the loaded pallet <NUM> has been validated, it is moved to a loading station <NUM> (<FIG>). As explained in more detail below, at the loading station <NUM>, the distribution center computer <NUM> ensures that the loaded pallets <NUM>, as identified by each pallet id, are loaded onto the correct trucks <NUM> in the correct order. For example, pallets <NUM> that are to be delivered at the end of the route are loaded first.

A computer (DC computer <NUM>, server <NUM>, or another) determines efficient routes to be driven by each truck <NUM> to visit each store <NUM> in the most efficient sequence, the specific loaded pallets <NUM> that must go onto each truck <NUM>, and the order in which the pallets <NUM> should be loaded onto the trucks <NUM>.

As shown in <FIG>, a route for each truck <NUM> is optimized by server <NUM> so that an efficient route is plotted for the driver. As shown, the route is communicated to the driver's mobile device <NUM> (or on-board navigation system) and may be modified after the truck <NUM> has left the DC <NUM> as necessary (e.g. based upon traffic).

Referring to <FIG>, an optimized queue system is used to queue and load loaded pallets <NUM> onto the truck <NUM> in the correct reverse-stop sequence (last stop is loaded onto the truck <NUM> first) based upon the route planned for that truck <NUM>. Each truck <NUM> will be at a different loading dock doorway <NUM>.

<FIG> shows an example loading station <NUM>, such as a loading dock with a doorway <NUM>. Based upon the sequence determined by the server <NUM>, an electronic visual display <NUM> proximate the doorway <NUM> shows which pallet <NUM> is to be loaded onto that truck <NUM> next. A camera <NUM> and/or rfid reader <NUM> adjacent the doorway <NUM> identifies each loaded pallet <NUM> as it is being loaded onto the truck <NUM>. If the wrong pallet <NUM> is moved toward the doorway <NUM>, an audible and/or visual alarm alerts the workers. Optionally, the rfid reader <NUM> at the doorway <NUM> is able to determine the direction of movement of the rfid tag on the loaded pallet <NUM>, i.e. it can determine if the loaded pallet <NUM> is being moved onto the truck <NUM> or off of the truck <NUM>. This is helpful if the wrong loaded pallet <NUM> is moved onto the truck <NUM>. The worker is notified that the wrong pallet <NUM> was loaded, and the rfid reader <NUM> can confirm that the pallet was then moved back off the truck <NUM>.

When a group of loaded pallets <NUM> (two or more) is going to the same store <NUM>, the loaded pallets <NUM> within this group can be loaded onto the truck <NUM> in any order. The display <NUM> may indicate the group of loaded pallets <NUM> and the loaded pallets <NUM> within this group going to the same store <NUM> will be approved by the rfid reader <NUM> and display <NUM> in any order within the group.

Referring to <FIG>, a portal <NUM> (generated by server <NUM>) provides visibility of truck <NUM> schedules for local companies to reduce wait times.

Referring to <FIG>, the loaded truck <NUM> carries a hand truck or pallet sled <NUM>, for moving the loaded pallets <NUM> off of the truck <NUM> and into the stores <NUM> (<FIG>, step <NUM>). The driver has a mobile device <NUM> which receives the optimized route from the distribution center computer <NUM> or central server <NUM>. The driver follows the route to each of the plurality of stores <NUM> for which the truck <NUM> contains loaded pallets <NUM>.

At each store <NUM> the driver's mobile device <NUM> indicates which of the loaded pallets <NUM> (based upon their pallet ids) are to be delivered to the store <NUM> (as verified by gps on the mobile device <NUM>). The driver verifies the correct pallet(s) for that location with the mobile device <NUM> that checks the pallet id (rfid, barcode, etc). The driver moves the loaded pallet(s) <NUM> into the store <NUM> with the pallet sled <NUM>.

Referring to <FIG>, optionally, the pallet sled <NUM> can include an rfid reader <NUM> to check the pallet id of pallet <NUM> carried thereon by reading the RFID tag <NUM> secured to the pallet <NUM>. The rfid reader <NUM> may also read RFID tags <NUM> on the items <NUM>. Optionally the pallet sled <NUM> may alternatively or additionally include a camera <NUM> for imaging the loaded pallet <NUM> carried thereon for validation. A local wireless communication circuit (e.g. Bluetooth) may communicate the pallet id of the pallet <NUM> on the pallet sled <NUM> to the driver's mobile device <NUM>. The driver's mobile device <NUM> can confirm to the driver that the correct pallet <NUM> is loaded on the pallet sled <NUM> or warn the driver if the pallet <NUM> on the pallet sled <NUM> does not correspond to the store <NUM> at the current location (determined via gps on the mobile device <NUM>).

The pallet sled <NUM> can also assist in tracking the return of the pallets <NUM> and returnable packaging such as plastic beverage crates <NUM>. If the returnable packaging, such as plastic beverage crates <NUM>, have rfid tags <NUM>, the pallet sled <NUM> can count the number of crates <NUM> and the pallets <NUM> that are being returned to the truck <NUM>. Over time, this can provide asset tracking information. For example, this makes it easy to determine if the number of pallets <NUM> and crates <NUM> delivered to a particular store <NUM> consistently exceeds the number of pallets <NUM> and crates <NUM> that are returned from that store <NUM>, thus indicating that the store <NUM> is experiencing a high rate of asset loss for some reason, which can then be investigated and remedied.

One of several methods can then be followed at the store.

In the first method, the driver removes the wrapping from the loaded pallets <NUM> and uses the mobile device <NUM> in the store <NUM> to take at least one, and preferably several still images or video of the loaded pallet <NUM> (<FIG>; <FIG>; <FIG>, step <NUM>). Optionally, the driver may be able to take a single image of a corner of the unwrapped loaded pallet <NUM>, so that two sides of the loaded pallet <NUM> are captured in a single image. The image(s) <NUM> are sent from the mobile device <NUM> to the server <NUM> (or alternatively the DC computer <NUM>). In step <NUM>, the distribution central server <NUM> analyzes the images in one of the several ways described above to confirm the presence of the correct number of items <NUM> of each of the SKUs associated with the pallet id of that pallet <NUM> on the pick sheet <NUM> (step <NUM>), and then communicates a confirmation to the driver's mobile device <NUM> and/or the store employee's mobile device <NUM>, which is displayed on the screens. (<FIG> and <FIG>).

If a discrepancy is detected (step <NUM>), the system indicates the specific discrepancy and how to remedy the discrepancy to the driver in step <NUM>. The driver can correct the discrepancy by retrieving products <NUM> of the missing SKUs from the truck <NUM> or crediting the missing SKUs to the store account <NUM> (step <NUM>). Any SKUs detected that do not belong on the pallets <NUM> can be returned by the driver to the truck <NUM>. On the store worker's mobile device <NUM> (via an app), the store worker confirms the presence of the loaded pallet <NUM> and receives a list of SKUs associated with that pallet id from the distribution center computer <NUM> or the server <NUM>.

If one or more SKUs do not match, the driver is shown the screen of <FIG> which indicates specifically what is missing (step <NUM>). Optionally, not shown, the screen on his mobile device may also visually indicate on the image the SKUs that do not match, such as by drawing boxes or circles around the SKUs in the image. If necessary, he can manually identify them by clicking on them and then assigning the right SKU to it. If a SKU was actually physically missing or was legitimately not on the pick list it would also be identified here and the driver could potentially correct the order, such as by retrieving the missing items from the truck <NUM> in step <NUM>. The driver then completes the delivery in step <NUM>.

Referring to <FIG>, the store employee may receive a notification via their mobile device <NUM> that the delivery has been made. Via their mobile device <NUM>, the employee may view the image(s) of the loaded pallets <NUM> and may be asked to sign off on the delivery based upon the image(s) and/or based upon an indication from the server <NUM> that the system <NUM> has confirmed the accuracy of the delivery (i.e. after validation of the in-store image(s)).

In the second method, the driver images the loaded pallets <NUM> (again, one or more still images or video of each loaded pallet <NUM>) before unwrapping them. The images <NUM> are sent from the mobile device <NUM> to the distribution center computer <NUM> or server <NUM>. The distribution center computer <NUM> or central server <NUM> analyzes the images by identifying the SKUs through the stretch wrapping, which is translucent. Alternatively, rather than a full, fresh identification of the SKUs on the loaded pallet <NUM>, all that is needed is a confirmation that nothing on the previously-validated loaded pallet <NUM> has been changed. For example, knowing the previous arrangement of each SKU on the pallets <NUM> and the specific packaging of each SKU (for SKUs that may have more than one possible package), it is easier to identify that those SKUs are still in the same location and arrangement as they were when validated at the DC <NUM>.

Additionally, if images of the loaded pallets <NUM> were also taken after wrapping, the DC computer <NUM> and/or server <NUM> can also verify that the wrapping is relatively undisturbed. Alternatively, determining that the wrapping is undisturbed may be done without identifying the SKUs beneath, and if the wrapping is too disturbed, then the driver is notified to remove the wrapping and to image the loaded pallets <NUM> unwrapped for a full image analysis. Again, the store worker confirms the presence of the loaded pallet <NUM> and receives a list of SKUs associated with that pallet id from the distribution center computer <NUM> or the server <NUM>.

Alternatively, the image(s) can simply be compared as an image to the image(s) taken at the distribution center, without actually identifying the skus in the image(s). If the image(s) at the store are similar enough to the image(s) taken at validation, the accuracy of the delivery can be confirmed. This can be done by comparing the unwrapped images to one another or by comparing the wrapped images to one another. However, this would not enable the driver to correct the missing skus as easily. Therefore, if it is determined that the images are not similar enough to the validation images, then a new SKU identification based upon images of the unwrapped loaded pallet <NUM> at the store <NUM> could be initiated at that time.

In a third method, the store worker has gained trust in the overall system <NUM> and simply confirms that the loaded pallet <NUM> has been delivered to the store <NUM>, without taking the time to go SKU by SKU and compare each to the list that he ordered and without any revalidation/imaging by the driver. In that way, the driver can immediately begin unloading the products <NUM> from the pallet <NUM> and placing them on shelves <NUM> or in coolers <NUM>, as appropriate. This greatly reduces the time of delivery for the driver.

<FIG> shows a sample training station <NUM> including a turntable <NUM> onto which a new product <NUM> (e.g. for a new SKU or new variation of an existing SKU) can be placed to create the SKU file <NUM>. The turntable <NUM> may include an RFID reader <NUM> for reading an RFID tag <NUM> (if present) on the product <NUM> and a weight sensor <NUM> for determining the weight of the product <NUM>. A camera <NUM> takes a plurality of still images and/or video of the packaging of the product <NUM>, including any logos <NUM> or any other indicia on the packaging, as the product <NUM> is rotated on the turntable <NUM>. Preferably all sides of the packaging are imaged. The images, weight, RFID information are sent to the server <NUM> to be stored in the SKU file <NUM>. Optionally, multiple images of the product <NUM> are taken at different angles and/or with different lighting. Alternatively, or additionally, the computer files with the artwork for the packaging for the product <NUM> (i.e. files from which the packaging is made) are sent directly to the server <NUM>.

<FIG> shows an alternate training station 28a that could be used in the system of <FIG>. The training station 28a includes a support stand <NUM> onto which a new product <NUM> (e.g. for a new SKU or new variation of an existing SKU) can be placed to create the SKU file <NUM>. The support stand <NUM> may include an RFID reader <NUM> for reading an RFID tag <NUM> (if present) on the product <NUM> and an optional weight sensor <NUM> for determining the weight of the product <NUM>. One or more cameras <NUM> take a plurality of still images and/or video of the packaging of the product <NUM>, including any logos <NUM> or any other indicia on the packaging. In the example shown, three cameras <NUM> are mounted to a frame <NUM> that is secured to the support stand <NUM>. Preferably all sides of the packaging are imaged. Therefore, in the example shown, after capturing three sides with the three cameras <NUM>, a user may rotate the product <NUM> so that the remaining three sides can be captured. The images, weight, RFID information may be received by a local training computer <NUM> and sent to the server <NUM> to be stored in the SKU file <NUM>. Again, optionally, multiple sets of images may be taken with different lighting.

Each of the cameras <NUM> or <NUM> can be a depth camera, i.e. it also provides distance information correlated to the image (e.g. pixel-by-pixel distance information or distance information for regions of pixels). Depth cameras are known and utilize various technologies such as stereo vision (i.e. two cameras) or more than two cameras, time-of-flight, or lasers, etc. If a depth camera is used, then the edges of the product <NUM> are easily detected.

In one possible implementation of either training station <NUM> or 28a, shown in <FIG>, cropped images of products <NUM> from the training station <NUM> are sent from the local computer <NUM> via a portal <NUM> to sku image storage <NUM>, which may be at the server <NUM>. Alternatively, the computer files with the artwork for the packaging for the product <NUM> (i.e. files from which the packaging is made) are sent directly to the server <NUM>.

Whichever method is used to obtain the images of the items, the images of the items are received in step <NUM> of <FIG>. In step <NUM>, an API <NUM> takes the sku images and builds them into a plurality of virtual pallets, each of which shows how the products <NUM> would look on a pallet <NUM>. The virtual pallets may include four or five layers of the product <NUM> on the pallet <NUM>. Some of the virtual pallets may be made up solely of the single new product <NUM>, and some of the virtual pallets will have a mixture of images of different products <NUM> on the pallet <NUM>. The API <NUM> also automatically tags the locations and/or boundaries of the products <NUM> on the virtual pallet with the associated skus. The API creates multiple configurations of the virtual pallet to send to a machine learning model <NUM> in step <NUM> to update it with the new skus and pics.

The virtual pallets are built based upon a set of configurable rules, including, the dimensions of the pallet <NUM>, the dimensions of the products <NUM>, number of permitted layers (such as four, but it could be five or six), layer restrictions regarding which products can be on which layers (e.g. certain bottles can only be on the top layer), etc. The image of each virtual pallet is sized to be a constant size (or at least within a particular range) and placed on a virtual background, such as a warehouse scene. There may be a plurality of available virtual backgrounds from which to randomly select.

The virtual pallet images are sent to the machine learning model <NUM> along with the bounding boxes indicating the boundaries of each product on the image and the SKU associated with each product. The virtual pallet images along with the bounding boxes and associated SKUs constitute the training data for the machine learning model.

In step <NUM>, the machine learning model <NUM> analyzes the images of the virtual pallets based upon the location, boundary, and sku tag information. The machine learning model <NUM> is updated and stored. The machine learning model <NUM> is deployed and used in conjunction with the validation stations <NUM> (<FIG>) and optionally with the delivery methods described above. The machine learning model <NUM> may also receive actual images taken in the distribution center or the stores, which after identification can be added to the machine learning model. Optionally, feedback from the workers can factor into whether the images are used, e.g. the identified images are not used until a user has had an opportunity to verify or contradict the identification.

<FIG> shows another alternative validation station. A pallet <NUM> loaded with goods <NUM> is carried on a first conveyor <NUM> to a turntable <NUM>. An rfid reader <NUM> and at least one depth camera <NUM> are positioned adjacent the turntable <NUM>. When the loaded pallet <NUM> reaches the turntable <NUM>, the rfid reader <NUM> identifies the pallet <NUM> and the loaded pallet <NUM> is rotated on the turntable <NUM> so that the camera <NUM> can take images or video (as before), such as one still image of each of the four sides of the loaded pallet <NUM>. As before, the images are used to identify all of the SKUs on the pallet <NUM>, which are compared to the pick list associated with that pallet <NUM>. If the loaded pallet <NUM> is validated against the pick list, then the loaded pallet <NUM> is moved to the second conveyor <NUM>, which carries the loaded pallet <NUM> to a dedicated wrapping station, with a turntable <NUM> and stretch wrap <NUM>. The loaded pallet <NUM> is wrapped with the stretch wrap at the wrapping station. If the loaded pallet <NUM> is not validated against the pick list, the loaded pallet <NUM> is moved on a third conveyor <NUM> to an audit station <NUM>, where a worker can make the corrections to the goods <NUM> on the pallet <NUM> in the manner explained above in the other embodiments.

Claim 1:
A delivery method comprising:
a) receiving (<NUM>) an order (<NUM>) for a plurality of SKUs;
b) generating (<NUM>) a pick sheet (<NUM>) based upon the order (<NUM>) for the plurality of SKUs;
c) assembling (<NUM>) a plurality of items (<NUM>) on a platform (<NUM>) based upon the pick sheet (<NUM>);
d) imaging (<NUM>) the plurality of items (<NUM>) assembled on the platform (<NUM>) to generate at least one image (<NUM>);
e) analyzing (<NUM>) the at least one image (<NUM>) to identify the SKUs of the assembled plurality of items (<NUM>),
f) comparing (<NUM>) the SKUs identified in step e) to the SKUs on the pick sheet (<NUM>); and
g) indicating (<NUM>, <NUM>) whether the SKUs identified in step e) match the SKUs on the pick sheet (<NUM>) based upon the comparison in step f),
h) based on an indication from step g) that the SKUs identified in step e) do not match the SKUs on the pick sheet (<NUM>), correcting (<NUM>) the items (<NUM>) on the platform (<NUM>), and in that step e) includes:
i) analyzing the at least one image (<NUM>) to identify a package type of one of the plurality of items (<NUM>); characterized in that the method further comprises the steps of
j) based upon the identified package type from step i) narrowing the list of possible branding options to be identified; and
k) after step j), determining a brand of the one of the plurality of items (<NUM>) based upon the narrowed list of possible branding options.