An Implementation of Faster RCNN with Study for Region Sampling

We adapted the join-training scheme of Faster RCNN framework from Caffe to TensorFlow as a baseline implementation for object detection. Our code is made publicly available. This report documents the simplifications made to the original pipeline, with justifications from abla tion analysis on both PASCAL VOC 2007 and COCO 2014. We further investigated the role of non-maximal suppression (NMS) in selecting regions-of-interest (RoIs) for reg ion classification, and found that a biased sampling toward small regions helps performance and can achieve on-par mAP to NMS-based sampling when converged sufficiently. 1. Baseline Faster RCNN with Simplification We adapted the join-training scheme of Faster RCNN detection framework 1  from Caffe2 to TensorFlow3 as a baseline implementation. Our code is made publicly available4. During the implementation process, several simplifications are made to the original pipeline, with observation s from ablation analysis that they are are either not affectin g or even potentially improving the performance. The ablation analysis has the following default setup: Base network. Pre-trained VGG16 . The feature map from conv5 3 are used for region proposals and fed into region-of-interest (RoI) pooling. Datasets.Both PASCAL VOC 2007  and COCO 2014 . For VOC we use thetrainval split for training, andtest for evaluation. For COCO we use train+valminusminival andminival, same as the published model. Training/Testing. The default end-to-end, single-scale training/testing scheme is copied from the original implementation. Learning rate starts with .001 and is 1https://github.com/rbgirshick/py-faster-rcnn 2https://github.com/BVLC/caffe 3https://github.com/tensorflow 4https://github.com/endernewton/tf-faster-rcnn reduced after50k/350k iterations. Training finishes at 70k/490k iterations. Following COCO challenge requirements, for each testing image, the detection pipeline provides at most 100 detection results.  We use evaluation toolkits provided by the respective dataset. The metrics are based on detection average precision/recall. The first notable change follows Huang et al. . Instead of using the RoI pooling layer, we use the crop and resize operator, which crops and resizes feature maps to14 × 14, and then max-pool them to 7 × 7 to match the input size of fc6. Second, we do not aggregate gradients from N = 2 images andR = 128 regions , instead we simply sample R = 256 regions fromN = 1 images during a single forward-backward pass. Gradient accumulation across multiple batches is slow, and requires extra operators in Tenso rFlow. Note thatR is the number of regions sampled for training the region classifier, for training region proposa l network (RPN) we still use the default 256 regions. Third, the original Faster RCNN removes small proposals (less than16 pixels in height or width in the original scale). We find this step redundant, hurting the performance especially for small objects. Other minor changes that does not seem to affect the performance include: 1) double the learning rate for bias; 2) stop weight decay on bias; 3) remove aspect-ratio grouping (introduced to save memory); 4) exclude ground-truth bounding boxes in the RoIs during training, since they are not accessible during testing and can bias the input distrib ution for region classification. For ablation analysis results on VOC 2007, please check at Table1. Performance-wise, our implementation is in general on par with the original Caffe implementation. The crop and resize pooling appears to have a slight advantage over RoI pooling. We further test the pipeline on COCO, see Table 2. We fix N = 1 and only usecrop and resize pooling, which in general gives better average recall than RoI pooling. Keeping the small region proposals also gives consis-