# A Practical Approach for Picking Items in an Online Shopping Warehouse
## 5 Evaluation
### Failure Analysis  
Figure 14 shows that CNA with vacuum is the most used and is also the most reliable approach. So if it is deemed that vacuum is most useful for a certain object then there is over 90% chance that it will be successfully picked up. The set of pie charts provides important information as to what happened during testing with the unsuccessful grasps. The pie charts document the reasons for failure. The most notable reasons relevant to grasping that were seen during testing are listed below. * Grasp not found: The algorithm does not output a grasp based on the input point cloud. * Grasp not feasible: The executed grasp will not result in lifting the object. * Robot imprecision: The grasp is feasible and can be executed; however, the robot misses the location of grasp due to calibration or actuation errors. * Object slip/drop: Object is grasped but then slips between the fingers or escapes from the vacuum pressure. * Texture of object not consistent: The grasp from the point cloud looks potentially successful; however, a label or a bump in the object obstructs the lifting of the object. * Grasp not reachable: The restrictions to robot's wrist motion deem the grasp not executable. Figure 13: Number of items per grasping strategy. In the case of CNA with vacuum, the main reason, as seen in Figure 15a, is the inconsistency of object texture. For example, the pie plates item has a back side that is perfectly flat and has 100% chance of being picked up; however, testing was performed on all sides and the other side is curved and air pressure slips between the holes. On the other side, the pie plates item has been picked up 0/5 times; as such, the total success can only be 50%. Figure 15b shows the reason behind the failure of CNA with Pincher. The main reason is that the object slips. The material of an object such as table cloth is very thin and thus drops unexpectedly. Another major reason is that the texture is inconsistent. An object such as black fashion gloves has a label and if one of the fingers touches this label then it glides across the label without grabbing any of the material. Note that in the above case the CNA algorithm is successful despite the hardware restrictions. This is due to the preassigned angular rolls around the z-axis of the robot wrist. This technique in CNA allows for the configuration to always be comfortable for [PERSON] and the IK to always find a solution. Hence the strength of this algorithm is its independence of the orientation of the wrist. Figure 15c shows the reasons for the failure of the HAF algorithm with Pincher. This combination shows a low success rate of 25%. The reason behind this low success rate can be attributed to the hardware restrictions: a grasp can always be calculated, even if it is not the best, but it can rarely be executed. Another reason, especially relevant for the hand weight item, is robot imprecision: the Pincher opening allows about a millimeter of clearance with the center of the hand weight and the robot has 1 cm accuracy, hence the robot simply misses the correct grasp location. Figure 14: Percent success per grasping strategy with isolated objects; each success rate has been determined for the corresponding subset of objects. The total average is weighted with the percentage of items per strategy, including the 6 blacklisted not_possible items. Figure 15: Reasons of failure for the different methods. The conditions for testing have been as realistic as possible. It is very important to evaluate a grasp in this perspective because grasping criteria often only results in success with regards to the object, not to the scene. Despite the setback described above, the total success rate, as shown in Figure 14, shows a 69% success for object grasping.