System and method for automatic product enrollment

A system for automatic product enrollment, includes: multiple visual sensors configured to capture images of a product; multiple scales configured to measure weight distribution of the product; and a computing device in communication with the visual sensors and the scales. The computing device is configured to: determine identification of the product and construct a 3D model of the product using the captured images; retrieve warehouse information of the product based on the identification; and enroll the warehouse information, the 3D model, and the weight distribution of the product into a product database. An automatic product enrollment method using the system.

FIELD OF THE INVENTION

The invention relates generally to robot technology, and more particularly to an enrollment system that automatically captures various properties of a product for robotic manipulation.

BACKGROUND OF THE INVENTION

The background description provided herein is for the purpose of generally presenting the context of the invention. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present invention.

An e-commerce company often has as many as millions of products (SKUs). For manipulation of these products, information about the products needs to be extracted. Today, it is a common practice to manually capture the information of the product when needed. However, the manual method is not a scalable way to enroll those millions of products.

Therefore, there is a need to address the aforementioned deficiencies and inadequacies in the art.

SUMMARY OF THE INVENTION

In certain aspects, the present invention relates to a system for automatic product enrollment. In certain embodiments, the system includes: a belt conveyor system having a first end and a second end, configured to convey a product from the first end to the second end; a transparent panel, disposed in a path of the conveyor belt; a plurality of scales, disposed at the second end (or alternatively the first end), and configured to record weights of the product at different points; a plurality of RGB cameras, respectively disposed above the transparent panel, under the transparent panel, and above the scales, and configured to capturing visual images of the product; a plurality of depth cameras, respectively disposed above and under the transparent panel, and configured to capturing depth images of the product; a plurality of lights, disposed above and under the transparent panel, and configured to light up the product; and a computing device. The computer device is configured to: detect identifications of the product from the captured visual images; construct three-dimensional (3D) model with appearance using the captured visual images and depth images; calculate weight distribution of the product based on the 3D model with appearance, the captured visual images (or/and depth images) of the product when the product is located on the scales for recording weights, and the recorded weights of the product; determine manipulation parameters of the product based on the 3D model with appearance, and the weight distribution of the product; and enroll the warehouse information, the 3D model with appearance, the weight distribution, and the manipulation parameters of the product in a product database. The manipulation parameters include grasping surface(s) of the product, and grasping point(s) on the grasping surface(s).

In certain embodiments, the computing device is further configured to: send the enrolled information of the product from the product database to the warehouse management system and other devices and equipment in a warehouse.

In certain embodiments, the identification of the product comprises a tag, an Apriltag, a quick response (QR) code, an one-dimensional (1D) or a two-dimensional (2D) barcode, a watermark, or the 3D model with appearance.

In certain embodiments, the computing device is further configured to store one or more captured visual images in the product database.

In certain embodiments, the system further comprises an opaque housing enclosing the conveyor belt, the transparent panel, the scales, the RGB cameras, the depth cameras, the lights, and the computing device:

In certain embodiments, the system further comprises a rig disposed around the conveyor belts. The RGB cameras, the depth cameras, and the lights are mounted on the rig.

In certain embodiments, the rig comprises an upper layer, a middle layer, and a lower layer, the conveyor belts are located between the middle layer and the lower layer, and the RGB cameras, the depth cameras, and the lights are respectively placed on all the three layers of the rig.

In certain embodiments, the belt conveyor system comprises a first conveyor belt having the first end and a second conveyor belt having the second end, the transparent panel is placed between the first conveyor belt and the second conveyor belt, a top plate is placed on top of the scales, and top surfaces of the first conveyor belt, the second conveyor belt, and the transparent panel are at a same planes as the top plate.

In certain embodiments, the transparent panel has a first corner and a second corner at two ends of a diagonal line of the transparent panel, the depth cameras comprise a first depth camera disposed above the first corner and a second depth camera disposed under the second corner.

In certain aspects, the present invention relates to a system for automatic product enrollment. In certain embodiments, the system includes: a plurality of visual sensors, configured to capture images of a product; a plurality of scales, configured to measure weight distribution of the product; and a computing device, in communication with the visual sensors and the scales. The computing device is configured to: construct a 3D model with appearance of the product using the captured images; and enroll the 3D model, and the weight distribution of the product into a product database. In certain embodiments, the computing device is further configured to determine identification of the product based on the captured images, retrieve warehouse information of the product based on the identification, and enroll the warehouse information into the product database.

In certain embodiments, the computing device is further configured to provide manipulation suggestion for the product based on at least one of the warehouse information, the 3D model with appearance, and the weight distribution of the product. The manipulation suggestion includes at least one grasping surface of the product, and at least one grasping point on the grasping surface.

In certain embodiments, the system further includes: a belt conveyor system having a first end and a second end, configured to convey a product from the first end to the second end; and a transparent panel disposed in a path of the belt conveyor system. The scales are disposed at the second end.

In certain embodiments, the transparent panel has a first corner and a second corner at two ends of a diagonal line of the transparent panel, and the visual sensors comprise: a plurality of RGB cameras, respectively disposed above the transparent panel, under the transparent panel, and above the scales; a first depth camera disposed above the first corner; and a second depth camera disposed under the second corner.

In certain aspects, the present invention relates to a method for automatic product enrollment. In certain embodiments, the method includes: capturing images of a product by a plurality of visual sensors; measuring weight distribution of the product by a plurality of scales; constructing a 3D model of the product using the captured images by a computing device, wherein the computing device is in communication with the visual sensors and the scales; and enrolling the warehouse information, the 3D model, and the weight distribution of the product into a product database.

In certain embodiments, the capture images include images of each of the sides or side surfaces of the product, the 3D model includes appearance of all sides of the product.

In certain embodiments, the method further includes: determining identification of the product using the capture images; retrieving warehouse information of the product base on the identification, and enrolling the warehouse information of the product into the product database.

In certain embodiments, the method further includes: providing manipulation parameters of the product based on at least one of the warehouse information, the 3D model, and the weight distribution of the product. The manipulation parameters includes one or a few grasping surfaces of the product, and one or more grasping points on each grasping surface.

In certain embodiments, the method further includes: providing a belt conveyor system having a first end and a second end for conveying the product from the first end to the second end; providing a transparent panel disposed in a path of the belt conveyor system; and placing the scales at the second end.

In certain embodiments, the transparent panel has a first corner and a second corner at two ends of a diagonal line of the transparent panel, and the method further includes: respectively placing the RGB cameras above the transparent panel, under the transparent panel, and above the scales; placing a first depth camera above the first corner; and placing a second depth camera under the second corner.

In certain aspects, the present invention relates to a non-transitory computer readable medium storing computer executable code. The computer executable code, when executed at a processor of the computing device, is configured to perform the method described above.

DETAILED DESCRIPTION OF THE INVENTION

The description will be made as to the embodiments of the present invention in conjunction with the accompanying drawings. In accordance with the purposes of this invention, as embodied and broadly described herein, this invention, in certain aspects, relates to a system for automatic product enrollment, and an automatic enrollment method. The system provides an efficient way to enroll various properties of millions of products, so that the products can be manipulated by robots such as picked from a bin, stacked or packaged based on the automatically enrolled properties and manipulation suggestions based on the enrolled properties. The enrolled properties include, among other things, (1) three-dimensional (3D) model of the product; (2) images of the product from different view angles; (3) barcode location(s); and (4) weight and weight distribution of the product or the package of the product.

FIG. 1is a schematic view of an automatic product enrollment system according to certain embodiments of the present invention. As shown inFIG. 1, the enrollment system100includes a belt conveyor system110, a plurality of scales120, a visual sensor structure130, a computing device140, a warehouse management system (WMS)150, and a product database160. The computing device140is configured to communicate with the belt conveyor system110, the scales120, the visual sensor structure130, and the WMS150, and enroll product information into the product database160.

The belt conveyor system110is configured to move a product102from a first end to an opposite, second end. The belt conveyor system110includes a first conveyor belt112, a transparent panel114, a second conveyor belt116, and a driving mechanism118. The first conveyor belt112, the transparent panel114, the second conveyor belt116, and the scales120are sequentially arranged from the first end to the second end. Upper surfaces of the first conveyor belt112, the transparent panel114, the second conveyor belt116, and the scales120are flush or in other words, leveled with each other.

The first conveyor belt112and the second convey belt116may be independent conveyor systems. In certain embodiments, the first and second convey belts112and116may be synchronized with each other. In certain embodiments, there may be only one convey belt which is transparent, and the separate transparent panel114is not needed. As shown inFIG. 2, each of the first and second conveyor belts112and116may include two pulleys and a belt rotates about the two pulleys. One of the two pulleys is a drive pulley that is driven by the driving mechanism118, and the other one of the two pulleys is a drive pulley or an idler pulley. In certain embodiments, each conveyor belts112and116may also include other idler pulleys, to assist smooth operation of the conveyor belts. For brevity, other structure components of the belt conveyor system110, such as frames, gears, bearings, motor, etc., are not shown in the figures. The driving mechanism118is configured to drive the conveyor belts112and116, and may include one or more motors, gears, gear reducers, etc.

The transparent panel114is transparent, such that the visual sensors below the transparent panel114may capture images of the product102above the transparent panel114. In certain embodiments, the transparent panel114is made of fiber glass. As shown inFIG. 2andFIG. 3, the transparent panel114is placed between the first conveyor belt112and the second conveyor belt116. The distance between the transparent panel114to the first conveyor belt112and the second conveyor belt116is smaller than the minimal stable dimension of a potential product, such that the product102, such as a soft package, will not stuck between the transparent panel14and any one of the first and second conveyor belts112and116. The length of the transparent panel14along the moving direction of the first and second conveyor belts112and116is large enough such that the visual sensors below the transparent panel114detects or takes images of the product102easily when the product passes through. On the other hand, since the transparent panel114is stationary, the length of the transparent panel114should be short enough such that the product102will not stuck on the transparent panel114. In certain embodiments, the product102is moved on the transparent panel114by at least one of the conveyor belt112and the conveyor belt116at any time. In other embodiments, when the product102is small and can completely stand on the transparent panel114without contacting with any of the conveyor belts112and116, the product102may also be pushed through the transparent panel114by another product at the first end side of the product102. In certain embodiments, the length of the transparent panel114along the moving direction of the conveyor belts112and116is in a range of 0.5-100 centimeters (cm). In certain embodiments, the length of the transparent panel114is in a range of 0.5-20 cm. In one embodiment, the length of the transparent panel114is about 1-2 cm. In certain embodiments, the belt of the belt conveyor system110is transparent. In that case, only one conveyor belt is needed, and the transparent panel14is not necessary. In certain embodiments, one of the conveyor belts112and116is transparent, and the transparent panel114is not needed. Under this situation, the transparent belt may be located at the first end side or the second end side. In certain embodiments, the scales120has a transparent upper panel, visual sensors are placed above and below the scales120, the transparent panel114is not required, and only one of the conveyor belts112and116is needed. In certain embodiments, the transparent panel114may be located at the first end or second end of the conveyor belt, and thus only one of the conveyor belts112and116is needed.

The scales120are disposed at the second end of the belt conveyor system110, and configured to measure the weight and weight distribution of the product102. As shown inFIG. 3andFIG. 4A, the scales120include a top plate120-0and four scales120-1,120-2,120-3, and120-4disposed under four corners of the top plate120-0. In certain embodiments, the top plate120-0is a rigid plate that covers on the four scales. In other embodiments, the top surfaces of the scales120-1,120-2,120-3, and120-4may also be portions of the top surface of the top plate120-0. As shown inFIG. 4A, when the product102is moved onto the top plate120-0, the position of the product102can be determined by the computing device140based on the images captured by the visual sensors at real time. At the same time, the four scales120-1,120-2,120-3, and120-4respectively record the weight of the product102at four corners, and send the weight to the computing device140. The weight distribution of the product102then can be calculated by the computing device140, based on the pose of the product102, the recorded weights by the four scales, and the three-dimensional model of the product102. In this embodiment, the four scales are disposed at four corners of the scales120. In other embodiments, the scales120may also include two, three, or more than four scales. In certain embodiments, as shown inFIG. 4B, when the scales120includes two scales, the two scales may be disposed at two corners of the scales120along a diagonal line of the top plate120-0. In certain embodiments, as shown inFIG. 4C, when the scales120includes three scales, one of the scales may be disposed at a middle point of one side of the top plate120-0, and the other two scales may be disposed at two corners of the other side of the top plate120-0.

The visual sensor structure130is configured to, among other things, simultaneously capture 3D model information and appearance of the product102. Referring toFIG. 1, the visual sensor structure130includes a plurality of red, green and blue (RGB) or RGB depth (RGBD) cameras132, a plurality of depth cameras134, a plurality of lights136, and a rig138. In certain embodiments, the depth cameras134are time-of-flight (ToF) cameras. When RGBD cameras are available, the visual sensor structure130may not need to include depth cameras. In certain embodiments, the visual sensor structure130may include one or more laser scanners, and the laser scanner is configured to capture identifications, such as barcode shown in the outer surface of the product102. In certain embodiments, the laser scanner may also be a lidar, which can be used to construct 3D model of the product102.

As shown inFIG. 2, the rig138has vertical columns1380and three horizontal layered structure. The three horizontal layers are an upper layer1382, a middle layer1384, and a lower layer1386. The upper layer1382is placed above the middle layer1384, the middle layer1384is placed above the transparent panel114, and the lower layer1386is placed below the transparent panel114. In certain embodiments, based on the number of visual sensors needed, the rig138may also have less than or greater than three layers. For example, the rig138may have two layers, with one layer placed above the belt conveyor system110, and the other layer placed below the belt conveyor system110. In certain embodiments, the distance between the upper layer1382, the middle layer1384, and the lower layer1386and horizontal plane of the conveyor belts112/116and transparent panel114are respectively in a range of 40-700 cm, 20-350 cm, 5-80 cm. In certain embodiments, the distance between the upper layer1382, the middle layer1384, and the lower layer1386and the conveyor belts112,116/transparent panel114are respectively in a range of 100-400 cm, 50-200 cm, and 10-40 cm. In certain embodiments, the distance between the upper layer1382, the middle layer1384, and the lower layer1386and the conveyor belts112,116/transparent panel114are about 200 cm, 100 cm, and 20 cm. In certain embodiments, the heights of at least one of the upper layer1382, the middle layer1384and the lower layer1386are adjustable. In other words, the three layers can be moved up and down along the columns1380, such that the system is usable for different sizes of product102. In certain embodiments, the height adjustment of the layers is controlled by the computing device140automatically, and when the product102is loaded on the conveyor belt and before entering the area of the rig138, some images captured by the RGB cameras132and depth cameras134is used by the computing device140to estimate the size of the product102, so as to trigger the height adjustment of the layer138to a level that suitable for capturing images of the product102when the product102is substantially located just above the transparent panel114or located in the center of the rig138.

The RGB cameras132, the depth cameras134, and the light136are installed on the three layers or the columns of the rig138. As shown inFIG. 2, the RGB (or RGBD) cameras132are placed on all three layers. One RGB camera132is placed in the center of the upper layer1382, four RGB cameras132are respectively placed at the centers of four sides of the middle layer1384, and four RGB cameras132are respectively placed at the center of four sides, and the center of the lower layer1386. In certain embodiments, the layer1382and the middle layer1384may combined as one layer, and the arrangement of the RGB cameras and depth cameras on the combined layer is the same as that on the lower layer1386. In certain embodiments, Because the view of the product102from under the transparent panel114is difficult, and the view of the side surfaces of the product102from under the transparent panel114is not necessary, the lower layer1386may only have one center RGB camera132and one depth camera134placed at the center of the lower layer1386. In one embodiment. Three RGB cameras132are placed on the lower layer1386along a center line under the transparent panel114, where one of them is located at the center, and the other two are located at centers of two sides. In addition, an extension1388is extended from the middle layer1384toward the moving direction of the product102, and a RGB camera132is placed on the extension1388, and above the scales120, such that the images of the product102can be taken clearly when the product stands on the scales120for measuring weight. In certain embodiments, a depth camera may be placed on the extension1388and above the scales120to capture depth images of the product102when the product102is placed on the scales120for measuring weight.

In certain embodiments, the product102is shown inFIG. 5Aas a cuboid box. As shown inFIG. 5A, in order to capture surfaces of the product102from different angles, RGB cameras132are placed from the top, the bottom, and four upper sides of the product102. When the product102is located on the center of view of the RGB cameras132, such as located on the transparent panel114, the top RGB camera captures an image from above the upper surface of the product102, and the image capture direction of the top RGB camera is substantially perpendicular to the upper surface of the product102. Each of the side cameras132, on the other hand, may capture an image of the corresponding side surface, and the angle α shown inFIG. 5Ais preferably about 45 degrees. In this way, the RGB cameras132respectively capture a clear image of each of the top and four side surfaces of the product102. The bottom surface can also be captured by the RGB camera132from under the transparent panel114. Because the length of the transparent114along the moving direction of the conveyor belt is likely smaller than the length of the product102along the moving direction, the RGB camera132is designed to capture multiple images when the product102moves through the transparent panel134, and combine those images to obtain the complete image of the bottom surface of the product102.

There are at least two depth cameras134, and the two depth cameras134are substantially placed at the center of the upper layer1382and the center of the lower layer1386. In certain embodiments, as shown inFIG. 2andFIG. 5B, the projections of the two depth cameras134on the transparent panel114are basically the center of the transparent panel114. In certain embodiments, as shown inFIG. 5C, the transparent panel114has opposite corners at two ends of a diagonal line of the transparent panel114, one depth camera134is located above one corner of the transparent panel114, and the other depth camera134is located below the other corner of the transparent panel114. By this type of design, the depth cameras134are configured to capture more accurate depth images of the product102when the product102stands on the transparent panel114. The RGB images show the appearance of the six side surfaces of the product102, including but not limited to the size, the barcode and pictures on the surfaces. Further, those RGB images, optionally combined with the depth images, are used to construct a 3D model of the produce102.

The lights or light sources136are mounted on the rig138, and configured to provide consistent lighting condition and reduce shadow and glare. In certain embodiments, the light sources136preferably provide diffused light. Referring toFIG. 2, four lights136are placed at four corners of the upper layer1382, and two lights136are placed at two sides of the lower layer1386. In certain embodiments, each of the two lights of the lower layer1386is placed at1/4of the corresponding sides, so as to avoid physical interference with the RGB camera132, the depth camera134and the lights136installed on the low layer1386. In certain embodiments, an opaque box around the rig or replacing the rig may be provided to reduce or eliminate the external light sources, such that the environment within the box has a consistent light condition. In certain embodiments, the lights136are manually controlled. In certain embodiments, the lights136are controlled by a specific controller. In certain embodiments, the lights136are controlled by the computing device140, to turn certain lights136on and off, or to adjust intensity and optionally orientation of the lights136.

In certain embodiments, the structure of the rig138, and the arrangement of the RGB cameras132, the depth cameras134and the light136may vary depending on their respective characteristics.

The computing device140may be a server computer, a cluster, a general-purpose computer, a specialized computer, a tablet, a smart phone, or a cloud-based device. In certain embodiments, the computing device140is a server computer to store and processing information collected from the scales120, the visual sensors130, and the WMS150. As shown inFIG. 1, the computing device140may include, without being limited to, a processor142, a memory144, and a storage device146. In certain embodiments, the computing device140may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but not limited to, other required memory, interfaces, buses, Input/Output (I/O) modules or devices, network interfaces, and peripheral devices.

The processor142controls operation of the computing device140. In certain embodiments, the processor142may be a central processing unit (CPU). The processor142can execute an operating system (OS) or other applications of the computing device140. In some embodiments, the computing device140may have more than one CPU as the processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

The memory144can be a volatile memory, such as the random-access memory (RAM), for storing the data and information during the operation of the computing device140. In certain embodiments, the memory144may be a volatile memory array. In certain embodiments, the computing device140may run on more than one memory144.

The storage device146is a non-volatile data storage media or device for storing the OS (not shown) and other applications of the computing device140. Examples of the storage device146may include flash memory, memory cards, USB drives, hard drives, floppy disks, optical drives, or any other types of data storage devices. In certain embodiments, the computing device140may have multiple storage devices146, which may be identical storage devices or different types of storage devices, and the applications of the computing device140may be stored in one or more of the storage device146of the computing device140. The storage device146includes an automatic enrollment application148, which is configured to retrieving data of a product and process the data, and enroll the processed data in the product database160.

FIG. 6schematically depicts the structure of the automatic enrollment application148according to certain embodiments of the present invention. As shown inFIG. 6, the automatic enrollment application148may include, among other things, an image capture module1481, a product identification and warehouse information retrieval module1482, a product detection and segmentation module1483, a 3D model construction module1484, a weight distribution calculation module1485, and an enrolling module1486.

The image capture module1481is configured to control the RGB or RGBD cameras132and the depth cameras134to capture images. In certain embodiments, the image capture module1481may be further configured to, when a laser scanner is used, control the laser scanner to scan the product102. In certain embodiments, the image capture module1481may also passively receive images captured by the RGB/RGBD cameras132and the depth cameras134. After obtaining the images, the image capture module1481may further pre-process the images. The pre-process may include, among other things, synchronizing the RGB images and the depth images, adjusting light balance of the images, reformatting the images, and resizing the images. The image capture module1481then sends the processed RGB images to the product identification and WMS retrieval module1482, and sends the processed RGB images and depth images to the product detection and segmentation module1483. In certain embodiments, when a laser scanner is used, the image capture module1481will then send the scanned information to the product identification and WMS retrieval module1482.

The product identification & WMS retrieval module1482is configured to, upon receiving the RGB images or laser scanned information, identify the labels from the RGB images or scanned information. The labels include 1D or 2D barcode, Apriltags, quick response (QR) codes, watermarks, or the like. The obtained labels are referred to as the identification of the product102. In certain embodiments, the 3D model of the product102with side surface appearance information may also be used as the identification of the product102. When the identification of the product102is obtained, the product identification and WMS retrieval module1482then can retrieve the WMS information of the product102from the WMS150or database of the WMS150. The WMS information of the product102may include receiving date, inventory, weight, product category, and other information related to the product102. In certain embodiments, the identification of the product102may be used later to index the information of the product102and enroll those information in the product database160. When the identification of the product102is used for indexing, the product identification and WMS retrieval module1482may send the identification to the product detection and segmentation module1483, such that the following processing of the product information is linked to that identification index.

The product detection and segmentation module1483is configured to, upon receiving the RGB images and the depth images, detect the product and segment the product. A background depth and appearance model of the empty conveyor belt is built beforehand. By comparing the empty conveyor belt model with the current depth and appearance of the conveyor belt, the disclosed product detection and segmentation module1483detects the difference and extract the product on the conveyor belt based on analysis of the difference. Specifically, the product detection and segmentation module1483partitions each image into a set of segments, so as to extract the location, boundaries, point clouds and sent the information to the 3D model construction module1484.

The 3D model construction module1484, upon receiving the segmented RGB images and depth images, is configured to construct a 3D textured model of the product102. In certain embodiments, the 3D model is constructed by triangulation using at least two RGB images. In certain embodiments, the use of the depth images improves the speed and accuracy of constructing the 3D model of the product102. Because the 3D model is a texture model, the 3D model not only includes dimensions of the product, but also the appearance of the product, for example, each of the side surfaces of the 3D model. Therefore, the 3D model, of for example a cuboid shaped product, includes the dimensions of the product and appearance of the six surfaces of the product. Each of the six surfaces includes, if available, labels and images on that surface. In certain embodiments, the appearance in the 3D model is in color.

The weight distribution calculation module1485is configured to control the scales120to weight the product102. In certain embodiments, when the product102is placed on the top plate120-0of the scales120, the product102is maintained still for a short period of time, so that the reading of the scales120is stable. The stable readings of the scales120is then recorded. The position and orientation, i.e. pose, of the product102on the scales120is also recorded by the RGB camera132disposed above the scales120. In certain embodiments, the scales120itself may also be a conveyor belt structure, so as to move in the product102, maintain the product102still for a while, and then move the product102out. In certain embodiments, a depth camera134is available to take depth images of the product102, so as to assist the determination of the pose of the product102.

In certain embodiments, the weight distribution calculation module1485may simply read and record the scales120continuously until the recording is stable, and the pose of the product102is also calculated based on the captured images during the measuring. Upon receiving or retrieving the reading of the scales120, the weight distribution calculation module1485is configured to calculate the weight distribution of the product102. The calculation of weight distribution is based on the real-time image taken by the RGB camera132placed above the scales120, the 3D model of the product102, and the weight measured by the scales120. In certain embodiments, the scales120are preferably placed at the second end of the belt conveyor system110. Therefore, when the weight measurement and weight distribution calculation is performed, the computing device140already has the 3D model of the product120, and the pose of the product120, which are used for the calculation.

In certain embodiments, the weight distribution of the product102may be described as a relative location of the center of mass (or center of weight) to the center computed from the geometrical shape of the product. In certain embodiments, the weight distribution of the product102may be described as a relative location of the center of mass (or center of weight) to the center computed from the geometrical shape of the product when the product is on different poses such as which side faces up. The location of the true center of mass vs. the center computed from the shape of the object will be used to determine the grasping point on based on the pose of the product. For example, if a suction cup manipulator is used, the cup can grasp the product102by aiming the suction cup toward the center of mass instead of geometrical center.

The enrolling module1486is configured to, after receiving the warehouse information of the product102, the 3D model with appearance of the product102, and the weight distribution of the product102, enroll those information into the product database160. The enrollment data may be indexed using the identification of the product102. In certain embodiments, the enrolling module1486further determines robotic manipulation suggestion of the product102, based on the images, the 3D model with appearance, and the weight distribution of the product102.

The WMS150is an independent platform for a warehouse, and is in communication with the computing device140. In certain embodiments, the computing device140is a server computer, and the WMS150is part of the server computer140.

The product database160is configured to store the enrolled information of the product102, and may include WMS information, identification, varies view images, and weight distribution of the product102. In certain embodiments, the computing device140is a server computer, and the product database160is part of the storage device146. In certain embodiments, the product database160may also be stored in storage device separated or remote from the computing device140. In certain embodiments, the product database160is stored in a cloud.

FIG. 7AandFIG. 7Bschematically show a method of enrolling a product according to certain embodiments of the present invention. In certain embodiments, the method as shown inFIG. 7AandFIG. 7Bmay be implemented on an automatic product enrollment system as shown inFIG. 1. It should be particularly noted that, unless otherwise stated in the present invention, the steps of the method may be arranged in a different sequential order, and are thus not limited to the sequential order as shown inFIG. 7AandFIG. 7B.

As shown inFIG. 7A, when an automatic product enrollment system as shown inFIG. 1is provided, and a product102is moved on the first conveyor belt112, the transparent panel114, the second conveyor belt116, and onto the scales120, at procedure705, the RGB cameras132and the depth cameras134take a plurality of images of the product102. The images are taken continuously, or else, may be taken when the product102is in the field of view. Each of the images has a respective time tag. When the RGB cameras132are RGBD cameras, it may not be necessary to have the depth cameras134. In certain embodiments, the system100may further include one or more laser scanners to scan information such as barcode on the product102. The RGB cameras132, the depth cameras134, and optionally the laser scanner may be controlled and operated by the image capture module1481. In certain embodiments, the RGB cameras132, the depth cameras134, and optionally the laser scanner may also operate independently from the image capture module1481, and are configured to send the captured images to the image capture module1481.

At procedure710, the RGB cameras132and the depth cameras134sends the captured images to the image capture module1481. In certain embodiments, the image capture module1481may also retrieve the images from the RGB cameras132and the depth cameras134.

Upon receiving or retrieving the images, at procedure715, the image capture module1481pre-processes the images. In certain embodiments, the pre-processing may include synchronizing the RGB images and the depth images using their time tag, adjusting light balance of the images to compensate light variance in the images, and possibly reformatting and resizing the images when needed for different purposes.

After pre-processing the images, the image capture module1481sends certain pre-processed images to the product identification and WMS retrieval module1482at procedure720, and sends certain pre-processed images to the product detection and segmentation module1483at procedure740.

At procedure725, upon receiving the pre-processed images from the image capture module1481, the product identification and WMS retrieval module1482identifies the product102by extracting identification from the image of the product102. The identification of the product102may include 1D and 2D barcode, Apriltags, QR code, image based and 3D model based matching, etc., which could be extracted from the pre-processed images and point cloud. In certain embodiments, the identification may also be extracted by a laser scanner. In certain embodiments, the image capture module1481only sends the RGB images to the product identification and WMS retrieval module1482, but not the depth images.

Then at procedure730, the product identification and WMS retrieval module1482retrieves warehouse information corresponding to the product102from the WMS150, using the identification of the product obtained from procedure725. The warehouse information of the product102may include, but not limited to, inventory, product category, known shape, weight, location of the barcode, etc.

After retrieving the warehouse information of the product102, at procedure735, the product identification and WMS retrieval module1482sends the identification and the retrieved warehouse information to the enrolling module1486. In certain embodiments, the product identification and WMS retrieval module1482may also send the identification and the retrieved warehouse information directly to the product database160.

At procedure745, upon receiving certain pre-processed images from the image capture module1481, the product detection and segmentation module1483detects the product102from the received images. The received pre-processed images may include pre-processed RGB images and pre-processed depth images. The detected product102may be different views of a cuboid box in the different images. Then the product detection and segmentation module1483segments the detected product102in those images. Specifically, the product detection and segmentation module1483partitions each of the images into multiple segments, and locates the products102and boundaries of the products102and the environment.

After product detection and segmentation, at procedure750, the product detection and segmentation module1483sends the detected and segmented information to the 3D model construction module1484.

Upon receiving those detected and segmented information, at procedure755, the 3D model construction module1484builds a 3D model of the product102. For example, if the product102is in a cuboid box structure, the 3D model will be a cuboid box, with accurate length, wide, and height. In addition, the pose of the 3D model in the environment is also calculated, where each of the faces of the cuboid box is determined, orientation of the cuboid box relative to the conveyor belt is determined, and distance between the cuboid box and the edges of the conveyor belt is determined. In certain embodiments, the 3D model further includes appearance information. For example, each of the six side surfaces of the cuboid box structure are included in the 3D model. The appearance includes those tags or images on the side surfaces, and may be colored. Therefore, the 3D model here is also named 3D model with appearance or textured 3D model.

After constructing the 3D model, at procedure760, the 3D model construction module1484sends the 3D model to the enrolling module1486. In certain embodiments, the 3D model construction module1484may also send the 3D model directly to the product database160. In addition, at procedure765, the 3D model construction module1484also sends the 3D model and certain product images, such as the images taken when the product102is on the scales120, to the weight distribution calculation module1485.

At procedure770, when the product102moves onto the scales120, the scales120measure the weight of the product102. The measurement may be recorded by the scales120multiple times when the product102is completely standing on the scale plate120-0. The measurement by the scales120may be after or before the 3D model construction of the product120. But preferably, when the product102is on the scales120, the 3D model has already been constructed, so that the length, width, height of the product102are available for the calculation of weight distribution.

After the measurement, at procedure775, the scales120send the weight measurement to the weight distribution calculation module1485.

At this time, the weight distribution calculation module1485has the weight measurement from the scales120, and 3D model and certain images from the 3D model construction module1484. The images include those RGB images taken when the product102is standing on the scales120, so that the time of the images can be matched with the time of the weight measurement. In certain embodiments, the images may also be sent by the image capture module1481instead of the 3D model construction module1484. In other embodiments, those images used by the weight distribution calculation module1485may also be received from the cameras132/134, or the product identification and WMS retrieval module1482, or the product detection and segmentation module1483when appropriate. Using 3D models, weight measurement, and images corresponding to the time of the weight measurement, the weight distribution can be calculated at procedure780.

In certain embodiments, the weight distribution calculation is performed as shown inFIG. 8. Specifically, the product102is moved onto the top plate120-0of the scales. The scales A, B, C and D (or120-1,120-2,120-3, and120-4) are located under the product102. The weights measure by the four scales are Fa, Fb, Fc and Fd. The total weights of the four scales are the sum of Fa, Fb, Fc and Fd, and named Fabcd. The sum of Fa and Fb is named Fab, the sum of Fb and Fc is named Fbc, the sum of Fc and Fd is named Fcd, and the sum of Fd and Fa is named Fad. The length of the scales is defined L (between the line connecting the scales A and D and the line connecting scales B and C), and the width of the scale is defined W (between the line connecting the scales A and B and the line connecting the scales C and D). The length of L may be the same, greater than, or less than the length of W, depending on the space required by the scales150and the sizes of the products to be weighted. The center of mass of product102, projected on the top plate120-0, along the length direction, is calculated to be in a distance of L×Fbc/Fadcd to the line connecting A and D, or to be in a distance of L×Fad/Fadcd to the line connecting B and C. The center of mass of product102, projected on the top plate120-0, along the width direction, is calculated to be in a distance of W×Fcd/Fadcd to the line connecting A and B, or to be in a distance of W×Fab/Fadcd to the line connecting C and D. Accordingly, the center of mass of the product102projected on the top plate120-0(shown by a solid circle M) is calculated. In comparison, the geometrical center of the product102is calculated through the 3D model. The geometrical center projected on the top plate120-0is shown as empty circle G. In certain embodiments, the arrangement of the product102standing on the current bottom surface is a main position for manipulation, and the center of mass of the product102in 3D is estimated by extending the point M upward half of the height H of the product102. In certain embodiments, the product102may also be flipped three times to measure three projections of the center of mass, and the center of mass can be estimated more accurately using the three projections of the center of mass. After calculation of the weight distribution of the product102, at procedure785, the weight distribution calculation module1485sends the calculated weight distribution to the enrolling module1486. In certain embodiments, the weight distribution calculation module1485may send the calculated weight distribution directly to the product database160.

Now the enrolling module1486has the identification and the retrieved warehouse information of the product102received from the product identification and WMS retrieval module1482, the 3D model of the product102received from the 3D model construction model1484, the calculated weight distribution of the product102received from the weight distribution calculation module1485, and images of the product102. Then at procedure790, the enrollment module1486compiles those data, for example, using the identification of the product102as index. In addition, the enrolling module1486may also determines manipulation suggestions based on those data. In certain embodiments, the enrolling module1486uses the weight distribution, or the center of mass of the product102together with 3D model with appearance, and optionally warehouse information, to determine certain manipulation parameters of variety of manipulators. For example, the enrolling module1486may first determine a grasping surface based on the dimension of the produce102, or the smoothness or hardness of the surface, to determine one or more grasping surface. When the grasping surfaces are determined, the enrolling module1486further determines a grasping point on the surface based on the projection of the center of mass of the product102on that surface. In certain embodiments, the enrolling module1486determines the manipulation parameters based on the manipulators used.

After compiling and processing the received data, at procedure795, the enrolling module1486then enrolls those data in the product database160. The enrolled data are under the same entry for the product102, and can be used later. For example, when a robotic device is used to operate the product102, the robotic device can retrieve data of the product102from the product database160. The retrieved information may include the dimensions of the product102, the weight and weight distribution of the product102, the smoothness of the surfaces of the product102, the suggested manipulation operations of the product102, etc. The information is helpful to assist the manipulation of the product102by a robotic device.

FIG. 7Bis a flowchart of the method in a different format. As shown inFIG. 7B, at procedure705, the RGB cameras132and the depth cameras capture images of a product102. The computing device140receives those captured images, and pre-processes those images at procedure715. Certain pre-processed images, such as some RGB images, are used to determine the identification of the product at procedure725. The identification may be a barcode. At procedure730, the computing device140retrieves WMS information corresponding to the product using the identification.

The pre-processed images, such as the RGB images and depth images, can be used at procedure145to detect the product from the images and segment the images. Those detected objects and segmented images are then used at procedure755to construct a 3D model of the product.

At procedure770, multiple scales are used to measure the weight of the product. Based on the 3D model, the pose of the product on the scales, and the weight measurement, the weight distribution of the product can be estimated.

The computing device140then organizes those WMS information, 3D model, different image views, weight distribution, and possibly provide manipulation suggestions based on the above information, and enroll all the data into the product database160. With the data of the product available, other devices, such as a robotic device, can retrieve those data to instruct how to manipulate the product102.

In certain aspects, the present invention relates to a non-transitory computer readable medium storing computer executable code. In certain embodiments, the computer executable code may be the software stored in the storage device146as described above. The computer executable code, when being executed, may perform one of the methods described above. In certain embodiments, the non-transitory computer readable medium may include, but not limited to, the storage device146of the computing device140as described above, or any other storage media of the computing device140.

In summary, certain embodiments of the present invention provide a systematic structure and method to automatically enroll a large number of products, and improve the efficiency of a logistic solution for e-commerce companies. A transparent panel is placed between conveyor belts, and RGB cameras and depth cameras are placed above and under the transparent panel. By this type of design, the system is able to capture appearance of all six surfaces of the product without the need of flipping the product. The system also includes scales to measure weight distributions of the product. When the metric center of the product102is not the center of weight, the weight distribution information is helpful for instructing the specific manipulation of the product, for example, by a robotic device. This automatic enrollment system, among other things, provides six side surface appearance, 3D model, barcode identification, and weight distribution information, and store those information in a product database. The information can then be used by others, such as a robotic device.