Symmetry Detection Competition

Executive Summary We completed the second round of the symmetry detection competition and presented results at a dedicated workshop hosted at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011 in Colorado Springs, Colorado. The competition was divided into three parts, each focusing on one of three types of symmetries: reflection, rotation and translation respectively. For each symmetry category, we collected images depicting objects with representative symmetry features. In order to minimize bias towards specific symmetries, we also obtained a large variety of symmetry images from professional and amateur photographers who signed up and submitted images to our Flickr photosharing website (http://www.flickr.com/groups/symmetrycompetition/). We collected a total of four training and six test sets, totaling 124 images. Each image was annotated, yielding a total of 167 reflection and rotation symmetries and over two thousand wallpaper tiles for translation symmetry. During the annotation phase we identified and specified several ambiguities that can arise during the labeling process. In all cases of ambiguities a trade off between local and global context had to be made before it could be resolved. Because this trade of is in almost all cases subjective in nature, we have marked such symmetries as ambiguous and discounted any detections by the tested algorithms. We received eight submissions for symmetry detection, three for reflection, two for rotation and three for translation symmetry. Adding one baseline algorithm to each symmetry group for comparison, we evaluated a total of eleven algorithms. The evaluation process was completely automated, counting the number of correct detections (true positives) and the number of incorrect detections (false positives). Detection performance is then reported in terms of precision and recall rates. , for which the later only considers the number of true positives vs the total number of symmetries present, while the former also considers and penalizes for the number of false positives. The quite remarkable outcome of all evaluations is that overall, none of the submitted algorithms performed better than their respective baseline algorithm (including execution speed of the algorithms). In comparison, tested algorithms performed best on reflection symmetry, second best on rotation symmetry and last translation symmetry. However, looking at sub categories of reflection symmetries, we found that one algorithm performed superior to all other tested algorithms on real images with multiple reflection axis present (judged on recall rate). A different algorithm outperforms all other algorithms on real and synthetic images depicting only a single reflection symmetry, when the number false positives has been taken into account (judged on precision rate). Overall, our efforts have been successful. We established a testbed for the evaluation of symmetry detection algorithms, devised evaluation metrics and automated the evaluation process. We tested our process on eleven algorithms and established a performance baseline that can be used as reference for future work on symmetry detection.