Supplementary Material : Annotating Object Instances with a Polygon-RNN

We show additional results for our approach on the Cityscapes dataset [1]. Note that in all our experiments we assume to be given a ground-truth box around the object. Our goal then is to provide a polygon outlining this object as accurately as possible and with minimal number of clicks required from the annotator. While a box around the object in principle requires two additional clicks, boxes are typically much easier and cheaper to obtain using crowd-sourcing services such as AMT. On the other hand, for most major segmentation benchmarks, polygons have been collected with high quality annotators. In particular, in Table 1 we show an additional experiment in which we train the model on only one category (i.e., car) and test it on all the categories. While this result is not at the level of the model trained on all objects, it performs surprisingly well, particularly for the vehicle categories. This shows that our model essentially learns to follow boundaries, and is thus able to generalize to other classes. This is a desirable property for a generic annotation tool to facilitate labeling of various diverse, potentially not-seen-before classes. Full image annotation in prediction mode. We show full image results from our approach without any correction (i.e., 0 clicks) in Fig. 14. In the first column, we show the GT provided by the Cityscapes dataset [1], and in the second column, we show results from our approach. Below each image we show the number of clicks required to annotate the GT polygons as well as the number of bounding boxes (i.e. the number of instances) in the image. Note that in this experiment our method requires only the bounding boxes around the objects. Examples of humans-in-the-loop. In Fig. 58, we show visualizations of the instances inside the crop of the GT boxes. In the first column, we show the ground-truth polygon annotation, while in the second column, we show the output from SharpMask [3]. Note that Sharpmask predicts pixel labeling of the image (crop in this case). In our visualization, we draw the boundary based on the connectivity in the 8-neighborhood. In the third column, we report our predictions without any human intervention. Finally, in the fourth column, we show the “human-in-the-loop” results, where we provide our PolygonRNN with a correction of a point if the prediction deviates from the ground-truth vertex by 1 pixel. For each example, we show the number of vertices in the original GT annotation and the number of corrections needed in our model. We can observe that our model performs well on small instances, particularly person or bicycle (rows 1, 2 and 3 of Fig. 5). Our model also performs well on car since they have simpler shapes (row 4 of Fig. 5 and row 1 of Fig. 6). The blocky effects on some of the objects is due to the resolution of our output (our RNN operates on a 28× 28 grid). We plan to increase our output resolution in the future, requiring architectural changes to our RNN due to memory constraints. In Fig. 6 rows 2, 3, 4, we show some failure cases. For big instances (such as bus, truck or train) both SharpMask and our model achieve a lower performance. The most likely reason for this is that there are fewer big instances in the dataset. Fig. 7 row 1 shows an instance with multiple components. Our model tends to include occlusion inside the instance segmentation. By providing a few corrections, our model can recover and provide a much better labeling (column 4). Row 2 shows a trend of SharpMask in providing worse annotations for smaller instances. While overall, the predictions of SharpMask are quite good, we can still see many examples of mistakes. From the labeling perspective and the nature of SharpMask (and similar dense pixel-labeling approaches), such examples are useless since the annotator needs to re-label them from scratch. The main advantage of our method is that the interaction with the human labeler comes very naturally, and allows the annotator to obtain a good annotation with only a few clicks.