arXiv:1511.07763v2 [] 7 Apr 2016

LocNet: Improving Localization Accuracy for Object Detection
Spyros Gidaris
Universite Paris Est, Ecole des Ponts ParisTech

Nikos Komodakis
Universite Paris Est, Ecole des Ponts ParisTech

gidariss@imagine.enpc.fr

nikos.komodakis@enpc.fr

Abstract
We propose a novel object localization methodology with
the purpose of boosting the localization accuracy of stateof-the-art object detection systems. Our model, given a
search region, aims at returning the bounding box of an object of interest inside this region. To accomplish its goal,
it relies on assigning conditional probabilities to each row
and column of this region, where these probabilities provide
useful information regarding the location of the boundaries
of the object inside the search region and allow the accurate inference of the object bounding box under a simple
probabilistic framework.
For implementing our localization model, we make use of
a convolutional neural network architecture that is properly
adapted for this task, called LocNet. We show experimentally that LocNet achieves a very significant improvement on
the mAP for high IoU thresholds on PASCAL VOC2007 test
set and that it can be very easily coupled with recent stateof-the-art object detection systems, helping them to boost
their performance. Finally, we demonstrate that our detection approach can achieve high detection accuracy even
when it is given as input a set of sliding windows, thus proving that it is independent of box proposal methods.

1. Introduction
Object detection is a computer vision problem that has
attracted an immense amount of attention over the last
years. The localization accuracy by which a detection system is able to predict the bounding boxes of the objects of
interest is typically judged based on the Intersection over
Union (IoU) between the predicted and the ground truth
bounding box. Although in challenges such as PASCAL
VOC an IoU detection threshold of 0.5 is used for deciding
whether an object has been successfully detected, in real
life applications a higher localization accuracy (e.g. IoU
> 0.7) is normally required (e.g., consider the task of a
This work was supported by the ANR SEMAPOLIS project. Code
and trained models are available on:
https://github.com/gidariss/LocNet

Figure 1: Illustration of the basic work-flow of our localization
module. Left column: our model given a candidate box B (yellow
box) it ”looks” on a search region R (red box), which is obtained
by enlarging box B by a constant factor, in order to localize the
bounding box of an object of interest. Right column: To localize
a bounding box the model assigns one or more probabilities on
each row and independently on each column of region R. Those
probabilities can be either the probability of an element (row or
column) to be one of the four object borders (see top-right image),
or the probability for being on the inside of an objects bounding
box (see bottom-right image). In either case the predicted bounding box is drawn with blue color.

robotic arm that must grasp an object). Such a need is
also reflected in the very recently introduced COCO detection challenge [23], which uses as evaluation metric the traditional average precision (AP) measurement but averaged
over multiple IoU thresholds between 0.5 (loosely localized
object) and 1.0 (perfectly localized object) so as to reward
detectors that exhibit good localization accuracy.
Therefore, proposing detectors that exhibit highly accurate (and not loose) localization of the ground truth objects
should be one of the major future challenges in object detection. The aim of this work is to take a further step towards
addressing this challenge. In practical terms, our goal is to

boost the bounding box detection AP performance across a
wide range of IoU thresholds (i.e., not just for IoU threshold of 0.5 but also for values well above that). To that end,
a main technical contribution of this work is to propose a
novel object localization model that, given a loosely localized search region inside an image, aims to return the accurate location of an object in this region (see Figure 1).
A crucial component of this new model is that it does
not rely on the commonly used bounding box regression
paradigm, which uses a regression function to directly predict the object bounding box coordinates. Indeed, the motivation behind our work stems from the belief that trying to
directly regress to the target bounding box coordinates, constitutes a difficult learning task that cannot yield accurate
enough bounding boxes. We argue that it is far more effective to attempt to localize a bounding box by first assigning
a probability to each row and independently to each column
of the search region for being the left, right, top, or bottom
borders of the bounding box (see Fig. 1 top) or for being on
the inside of an object’s bounding box (see Fig. 1 bottom).
In addition, this type of probabilities can provide a measure
of confidence for placing the bounding box on each location
and they can also handle instances that exhibit multi-modal
distributions for the border locations. They thus yield far
more detailed and useful information than the regression
models that just predict 4 real values that correspond to estimations of the bounding box coordinates. Furthermore, as
a result of this, we argue that the task of learning to predict
these probabilities is an easier one to accomplish.
To implement the proposed localization model, we rely
on a convolutional neural network model, which we call
LocNet, whose architecture is properly adapted such that
the amount of parameters needed on the top fully connected
layers is significantly reduced, thus making our LocNet
model scalable with respect to the number of object categories.
Importantly, such a localization module can be easily incorporated into many of the current state-of-the-art object
detection systems [9, 10, 28], helping them to significantly
improve their localization performance. Here we use it in
an iterative manner as part of a detection pipeline that utilizes a recognition model for scoring candidate bounding
boxes provided by the aforementioned localization module,
and show that such an approach significantly boosts AP performance across a broad range of IoU thresholds.
Related work. Most of the recent literature on object detection, treats the object localization problem at prerecognition level by incorporating category-agnostic object
proposal algorithms [35, 40, 26, 1, 18, 19, 2, 34, 33] that
given an image, try to generate candidate boxes with high
recall of the ground truth objects that they cover. Those proposals are later classified from a category-specific recognition model in order to create the final list of detections [11].

Instead, in our work we focus on boosting the localization
accuracy at post-recognition time, at which the improvements can be complementary to those obtained by improving the pre-recognition localization. Till now, the work
on this level has been limited to the bounding box regression paradigm that was first introduced from Felzenszwalb
et al. [8] and ever-since it has been used with success on
most of the recent detection systems [11, 10, 28, 30, 14,
37, 39, 29, 24]. A regression model, given an initial candidate box that is loosely localized around an object, it tries
to predict the coordinates of its ground truth bounding box.
Lately this model is enhanced by high capacity convolutional neural networks to further improve its localization
capability [9, 10, 30, 28].
To summarize, our contributions are as follows:
• We cast the problem of localizing an object’s bounding
box as that of assigning probabilities on each row and
column of a search region. Those probabilities represent either the likelihood of each element (row or column) to belong on the inside of the bounding box or
the likelihood to be one of the four borders of the object. Both of those cases is studied and compared with
the bounding box regression model.
• To implement the above model, we propose a properly adapted convolutional neural network architecture
that has a reduced number of parameters and results
in an efficient and accurate object localization network
(LocNet).
• We extensively evaluate our approach on VOC2007 [5]
and we show that it achieves a very significant improvement over the bounding box regression with respect to the mAP for IoU threshold of 0.7 and the
COCO style of measuring the mAP. It also offers an
improvement with respect to the traditional way of
measuring the mAP (i.e., for IoU > 0.5), achieving
in this case 78.4% and 74.78% mAP on VOC2007 [5]
and VOC2012 [6] test sets, which are the state-of-theart at the time of writing this paper. Given those results we believe that our localization approach could
very well replace the existing bounding box regression
paradigm in future object detection systems.
• Finally we demonstrate that the detection accuracy of
our system remains high even when it is given as input
a set of sliding windows, which proves that it is independent of bounding box proposal methods if the extra
computational cost is ignored.
The remainder of the paper is structured as follows: We
describe our object detection methodology in §2 and then
present our localization model in §3. Implementation details and experimental results are provided in §4 and §5 respectively. Finally, we conclude in §6.

2. Object Detection Methodology
Algorithm 1: Object detection pipeline
Input : Image I, initial set of candidate boxes B1
Output: Final list of detections Y
for t ← 1 to T do
St ← Recognition(Bt |I)
if t < T then
Bt+1 ← Localization(Bt |I)
end
end
D ← ∪Tt=1 {St , Bt }
Y ← PostProcess(D)
Our detection pipeline includes two basic components,
the recognition and the localization models, integrated into
an iterative scheme (see algorithm 1). This scheme starts
from an initial set of candidate boxes B1 (which could
be, e.g., either dense sliding windows [30, 25, 27, 22] or
category-agnostic bounding box proposals [40, 35, 28]) and
on each iteration t it uses the two basic components in the
following way:
Recognition model: Given the current set of candidate
t
boxes Bt = {Bit }N
i=1 , it assigns a confidence score
N
t
to each of them {sti }i=1
that represents how likely it is
for those boxes to be localized on an object of interest.
Localization model: Given the current set of candidate
t
boxes Bt = {Bit }N
i=1 , it generates a new set of candiNt+1
date boxes Bt+1 = {Bit+1 }i=1
such that those boxes
they will be “closer” (i.e., better localized) on the objects of interest (so that they are probably scored higher
from the recognition model).
In the end, the candidate boxes that were generated
on each iteration from the localization model along with
the confidences scores that were assigned to them from
the recognition model are merged together and a postprocessing step of non-max-suppression [8] followed from
bounding box voting [9] is applied to them. The output
of this post-processing step consists the detections set produced from our pipeline. Both the recognition and the localization models are implemented as convolutional neural
networks [21] that lately have been empirically proven quite
successful on computers vision tasks and especially those
related to object recognition problems [31, 20, 13, 16, 32].
More details about our detection pipeline are provided in
appendix E.
Iterative object localization has also been explored before [3, 9, 12, 36]. Notably, Gidaris and Komodakis [9]
combine CNN-based regression with iterative localization
while Caicedo et al. [3] and Yoo et al. [36] attempt to localize an object by sequentially choosing one among a few

Figure 2: The posterior probabilities that our localization model
yields given a region R. Left Image: the in-out conditional probabilities that are assigned on each row (py ) and column (px ) of
R. They are drawn with the blues curves on the right and on the
bottom side of the search region. Right Image: the conditional
probabilities pl , pr , pt , and pb of each column or row to be the left
(l), right (r), top (t) and bottom (b) border of an object’s bounding
box. They are drawn with blue and red curves on the bottom and
on the right side of the search region.

possible actions that either transform the bounding box or
stop the searching procedure.

3. Localization model
In this paper we focus on improving the localization
model of this pipeline. The abstract work-flow that we use
for this model is that it gets as input a candidate box B in
the image, it enlarges it by a factor γ 1 to create a search region R and then it returns a new candidate box that ideally
will tightly enclose an object of interest in this region (see
right column of Figure 1).
The crucial question is, of course, what is the most effective approach for constructing a model that is able to
generate a good box prediction. One choice could be, for
instance, to learn a regression function that directly predicts
the 4 bounding box coordinates. However, we argue that
this is not the most effective solution. Instead, we opt for a
different approach, which is detailed in the next section.

3.1. Model predictions
Given a search region R and object category c, our object localization model considers a division of R in M equal
horizontal regions (rows) as well as a division of R in M
equal vertical regions (columns), and outputs for each of
them one or more conditional probabilities. Each of these
conditional probabilities is essentially a vector of the form
pR,c = {p(i|R, c)}M
i=1 (hereafter we drop the R and c conditioned variables so as to reduce notational clutter). Two
types of conditional probabilities are considered here:
In-Out probabilities: These are vectors px = {px (i)}M
i=1
and py = {py (i)}M
i=1 that represent respectively the conditional probabilities of each column and row of R to be
1 We

use γ = 1.8 in all of the experiments.

inside the bounding box of an object of category c (see left
part of Figure 2). A row or column is considered to be inside
a bounding box if at least part of the region corresponding
to this row or column is inside this box. For example, if
B gt is a ground truth bounding box with top-left coordinates (Blgt , Btgt ) and bottom-right coordinates (Brgt , Bbgt ),2
then the In-Out probabilities p = {px , py } from the localization model should ideally equal to the following target
probabilities T = {Tx , Ty }:

1, if Blgt ≤ i ≤ Brgt
∀i ∈ {1, . . . , M }, Tx (i) =
,
0, otherwise

1, if Btgt ≤ i ≤ Bbgt
∀i ∈ {1, . . . , M }, Ty (i) =
.
0, otherwise
Border probabilities: These are vectors pl = {pl (i)}M
i=1 ,
M
M
pr = {pr (i)}M
,
p
=
{p
(i)}
and
p
=
{p
(i)}
t
t
b
b
i=1
i=1
i=1
that represent respectively the conditional probability of
each column or row to be the left (l), right (r), top (t) and
bottom (b) border of the bounding box of an object of category c (see right part of Figure 2). In this case, the target probabilities T = {Tl , Tr , Tt , Tb } that should ideally
be predicted by the localization model for a ground truth
bounding box B gt = (Blgt , Btgt , Brgt , Bbgt ) are given by

1, if i = Bsgt
∀i ∈ {1, . . . , M }, Ts (i) =
,
0, otherwise
where s ∈ {l, r, t, b}. Note that we assume that the left and
right border probabilities are independent and similarly for
the top and bottom cases.
3.1.1

Bounding box inference

Given the above output conditional probabilities, we
model the inference of the bounding box location B =
(Bl , Bt , Br , Bb ) using one of the following probabilistic
models:
In-Out ML: Maximizes the likelihood of the in-out elements of B
Y
Y
Lin-out (B) =
px (i)
py (i)
i∈{Bl ,...,Br }

Y
i∈{B
/
l ,...,Br }

p̃x (i)

i∈{Bt ,...,Bb }

Y

p̃y (i),

(1)

i∈{B
/
t ,...,Bb }

where p̃x (i) = 1−px (i) and p̃y (i) = 1−py (i). The first two
terms in the right hand of the equation represent the likelihood of the rows and columns of box B (in-elements) to be
inside a ground truth bounding box and the last two terms
2 We actually assume that the ground truth bounding box is projected on

the output domain of our model where the coordinates take integer values
in the range {1, . . . , M }. This is a necessary step for the definition of the
target probabilities

the likelihood of the rows and columns that are not part of B
(out-elements) to be outside a ground truth bounding box.
Borders ML: Maximizes the likelihood of the borders of
box B:
Lborders (B) = pl (Bl ) · pt (Bt ) · pr (Br ) · pb (Bb ).

(2)

Combined ML: It uses both types of probability distributions by maximizing the likelihood for both the borders
and the in-out elements of B:
Lcombined (B) = Lborders (B) · Lin-out (B).
3.1.2

(3)

Discussion

The reason we consider that the proposed formulation of
the problem of localizing an object’s bounding box is superior is because the In-Out or Border probabilities provide
much more detailed and useful information regarding the
location of a bounding box compared to the typical bounding box regression paradigm [8]. In particular, in the later
case the model simply directly predicts real values that corresponds to estimated bounding box coordinates but it does
not provide, e.g., any confidence measure for these predictions. On the contrary, our model provides a conditional
probability for placing the four borders or the inside of an
object’s bounding box on each column and row of a search
region R. As a result, it is perfectly capable of handling
also instances that exhibit multi-modal conditional distributions (both during training and testing). During training, we
argue that this makes the per row and per column probabilities much easier to be learned from a convolutional neural network that implements the model, than the bounding
box regression task (e.g., see Figure 3), thus helping the
model to converge to a better training solution. Indeed, as
we demonstrate, e.g., in Figure 4, our CNN-based In-Out
ML localization model converges faster and on higher localization accuracy (measured with the mAR [15] metric)
than a CNN-based bounding box regression model [10, 9].
This behaviour was consistently observed in all of our proposed localization models.
Furthermore, during testing, these conditional distributions as we saw can be exploited in order to form probabilistic models for the inference of the bounding box coordinates. In addition, they can indicate the presence of a
second instance inside the region R and thus facilitate the
localization of multiple adjacent instances, which is a difficult problem on object detection. In fact, when visualizing,
e.g., the border probabilities, we observed that this could
have been possible in several cases (e.g., see Figure 5). Although in this work we did not explore the possibility of
utilizing a more advanced probabilistic model that predicts
K > 1 boxes per region R, this can certainly be an interesting future addition to our method.

Figure 3: We show the evolution during training. In the left image
the green squares indicate the two highest modes of the left border
probabilities predicted by a network trained only for a few iterations (5k). Despite the fact that the highest one is erroneous, the
network also maintains information for the correct mode. As training progresses (50k), this helps the network to correct its mistake
and recover a correct left border(right image).

Figure 5: We depict the probabilities for the left (blue) and right
(red) borders that a trained model yields for a region with two
instances of the same class (cow). The probability modes in this
case can clearly indicate the presence of two instances.

dependence is exploited in the design of the convolutional
neural network that implements our model in order to keep
the number of parameters of the prediction layers small (see
§ 3.2). This is significant for the scalability of our model
with respect to the number of object categories since we
favour category-specific object localization that has been
shown to exhibit better localization accuracy [31].

3.2. LocNet network architecture

Figure 4: mAR as a function of the training iteration for the
bounding box regression model (Bbox reg.) and the In-Out ML
localization model. In order to create this plot, we created a small
validation set of candidate boxes with a ground truth bounding box
assigned on each of them, and during training given those candidates as input to the models we measure the mAR of the predicted
boxes. We observe that the In-Out ML localization model converges faster and to a higher mAR than the bounding box regression localization model.

Alternatively to our approach, we could predict the probability of each pixel to belong on the foreground of an object, as Pinheiro et al. [26] does. However, in order to
learn such a type of model, pixel-wise instance segmentation masks are required during training, which in general is
a rather tedious task to collect. In contrary, for our model
to learn those per row and per column probabilities, only
bounding box annotations are required. Even more, this in-

Our localization model is implemented through the convolutional neural network that is visualized in Figure 6 and
which is called LocNet. The processing starts by forwarding the entire image I (of size wI × hI ), through a sequence
of convolutional layers (conv. layers of VGG16 [31]) that
outputs the AI activation maps (of size w16I × h16I × 512).
Then, the region R is projected on AI and the activations
that lay inside it are cropped and pooled with a spatially
adaptive max-pooling layer [14]. The resulting fixed size
activation maps (14 × 14 × 512) are forwarded through two
convolutional layers (of kernel size 3 × 3 × 512), each followed by ReLU non-linearities, that yield the localizationaware activation maps AR of region R (with dimensions
size 14 × 14 × 512).
At this point, given the activations AR the network yields
the probabilities that were described in section §3.1. Specifically, the network is split into two branches, the X and Y,
with each being dedicated for the predictions that correspond to the dimension (x or y respectively) that is assigned
to it. Both start with a max-pool layer that aggregates the
AR activation maps across the dimension perpendicular to
the one dedicated to them, i.e.,
AxR (i, f ) = max AR (i, j, f ),
j

(4)

activations AI
w I hI
× ×512
16 16

X Branch

Image I
w I ×h I ×3

output size
14×14×512

VGG16
Conv. Layers

Region Adaptive
Max Pooling

R

Conv. Layer + ReLU
Kernel: 3 x 3 x 512

Max Pooling
over Y Axis

activations AR

output size
14×14×512

activations AxR
14×1×512

14×14×512

Conv. Layer + ReLU
Kernel: 3 x 3 x 512
Y Branch

Fully Connected
Layer + Sigmoid

activations ARy
14×1×512

Max Pooling
over X Axis

Projecting region R

Fully Connected
Layer + Sigmoid

M border probabilities pl , pr
and/or M in-out probabilities px

M border probabilities pt , pb
and/or M in-out probabilities py

Figure 6: Visualization of the LocNet network architecture. In the input image, with yellow is drawn the candidate box B and with red the
search region R. In its output, the LocNet network yields probabilities for each of the C object categories. The parameter M that controls
the output resolution is set to the value 28 in our experiments. The convolutional layers of the VGG16-Net [31] that are being used in order
to extract the image activations AI are those from conv1 1 till conv5 3. The new layers that are not derived from the VGG16-Net [31],
are randomly initialized with a Gaussian distribution with standard deviation of 0.001 for the hidden layers and 0.01 for the final fully
connected layers.

AyR (j, f ) = max AR (i, j, f ),
i

(5)

where i,j,and f are the indices that span over the width,
height, and feature channels of AR respectively. The resulted activations AxR and AyR (both of size 14 × 512) efficiently encode the object location only across the dimension
that their branch handles. This aggregation process could
also be described as marginalizing-out localization cues irrelevant for the dimension of interest. Finally, each of those
aggregated features is fed into the final fully connected layer
that is followed from sigmoid units in order to output the
conditional probabilities of its assigned dimension. Specifically, the X branch outputs the px and/or the (pl , pr ) probability vectors whereas the Y branch outputs the py and/or
the (pt , pb ) probability vectors. Despite the fact that the last
fully connected layers output category-specific predictions,
their number of parameters remains relatively small due to
the facts that: 1) they are applied on features of which the
dimensionality has been previously drastically reduced due
to the max-pooling layers of equations 4 and 5, and 2) that
each branch yields predictions only for a single dimension.

3.3. Training
During training, the network learns to map a search regions R to the target probabilities T that are conditioned on
the object category c. Given a set of N training samples
{(Rk , Tk , ck )}N
k=1 the loss function that is minimised is
L(θ) =

1
N

N
X

l(θ|Rk , Tk , ck ),

(6)

k=1

where θ are the network parameters that are learned and
l(θ|R, T, c) is the loss for one training sample.
Both for the In-Out and the Borders probabilities we use
the sum of binary logistic regression losses per row and column. Specifically, the per sample loss of the In-Out case

is:
M
X X

Ta (i) log(pa (i)) + T̃a (i) log(p̃a (i)) ,

(7)

a∈{x,y} i=1

and for the Borders case is:
X

M
X

λ+ Ts (i) log(ps (i)) + λ− T̃s (i) log(p̃s (i)) , (8)

s∈{l,r,u,b} i=1

where p̃ = 1 − p. In objective function (8), λ+ and λ−
represent the weightings of the losses for misclassifying a
border and a non-border element respectively. These are set
as
M
λ− = 0.5 ·
, λ+ = (M − 1) · λ− ,
M −1
so as to balance the contribution on the loss of those two
cases (note that T̃s (i) will be non-zero M − 1 times more
than Ts (i)). We observed that this leads to a model that
yields more “confident” probabilities for the borders elements. For the Borders case we also tried to use as loss
function the Mean Square Error, while modifying the target
probabilities to be Gaussian distributions around the border
elements, but we did not observe an improvement in performance.

4. Implementation details
General: For the implementation code of our paper we
make use of the Caffe framework [17]. During training of
all the models (both the localization and the recognition
ones) we fine-tune only from the conv4 1 convolutional
layer and above. As training samples we use both selective
search [35] and edge box [40] proposals. Finally, both
during training and testing we use a single image scale that
is obtained after resizing the image such as its smallest
dimension to be 600 pixels.

Proposed localization models (In-Out ML, Borders
ML, Combined ML): To create the training samples we
take proposals of which the IoU with a ground truth bounding box is at least 0.4, we enlarge them by a factor of 1.8
in order to obtain the search regions R, and we assign to
them the ground truth bounding box with which the original box proposal has the highest IoU in order to obtain the
target bounding boxes and the corresponding target vectors
T . This process is performed independently for each category. The parameter M that controls the output resolution
of our networks, is set to the value 28. For optimization we
use stochastic gradient descend (SGD) with mini-batch size
of 128 training candidate boxes. To speed up the training
procedure we follow the paradigm of Fast-RCNN [10] and
those 128 training candidate boxes are coming from only
two images on each mini-batch. The weight decay is set to
0.00005 and the learning rate is set to 0.001 and is reduced
by a factor of 10 after each 60k iterations. The overall training procedure is continued for up to 150k iterations and it
takes around 1.5 days in one NVIDIA Titan Black GPU.

5. Experimental results
We empirically evaluate our localization models on PASCAL VOC detection challenge [7]. Specifically, we train all
the recognition and localization models on VOC2007+2012
trainval sets and we test them on the VOC2007 test set.
As baseline we use a CNN-based bounding box regression
model [9] (see appendices A, B, and C). The remaining
components of the detection pipeline include
Initial set of candidate boxes: We examine three alternatives for generating the initial set of candidate boxes: the
Edge Box algorithm [40] (EB), the Selective Search algorithm (SS), and a sliding windows scheme. In Table 1 we
provide the recall statistics of the those bounding box proposal methods.
Recognition model: For the recognition part of the detection system we use either the Fast-RCNN [10] or the MRCNN [9] recognition models. During implementing the latter one, we performed several simplifications on its architecture and thus we call the resulting model Reduced-MRCNN (those modifications are detailed in appendix D). The
Fast-RCNN and Reduced-MR-CNN models are trained using both selective search and edge box proposals and as top
layer they have class-specific linear SVMs [11].
First, we examine the performance of our approach with
respect to localization (§5.1) and detection (§5.2) accuracy.
Then we report the detection accuracy of our approach for
the sliding windows case (§5.3). Finally, we provide preliminary results of our approach on COCO detection challenge
in §5.4 and qualitative results in §5.5.

Initial set of
candidate boxes

Number

Sliding Windows
Edge Box
Sel. Search

around 10k
around 2k
around 2k

IoU≥0.5
0.920
0.928
0.936

Recall
IoU≥0.7
0.389
0.755
0.687

mAR
0.350
0.517
0.528

Table 1: Recall statistics on VOC2007 test set of the box proposals
methods that we use in our work in order to generate the initial set
of candidate boxes.

5.1. Localization performance
We first evaluate merely the localization performance of
our models, thus ignoring in this case the recognition aspect of the detection problem. For that purpose we report
the recall that the examined models achieve. Specifically,
in Figure 7 we provide the recall as a function of the IoU
threshold for the candidate boxes generated on the first iteration and the last iteration of our detection pipeline. Also,
in the legends of these figures we report the average recall
(AR) [15] that each model achieves. Note that, given the
set of initial candidate boxes and the recognition model, the
input to the iterative localization mechanism is exactly the
same and thus any difference on the recall is solely due to
the localization capabilities of the models. We observe that
for IoU thresholds above 0.65, the proposed models achieve
higher recall than bounding box regression and that this improvement is actually increased with more iterations of the
localization module. Also, the AR of our proposed models
is on average 6 points higher than bounding box regression.

5.2. Detection performance
Here we evaluate the detection performance of the examined localization models when plugged into the detection pipeline that was described in section §2. In Table 2 we
report the mAP on VOC2007 test set for IoU thresholds of
0.5 and 0.7 as well as the COCO style of mAP that averages
the traditional mAP over various IoU thresholds between
0.5 and 1.0. The results that are reported are obtained after
running the detection pipeline for T = 4 iterations. We observe that the proposed InOut ML, Borders ML, and Combined ML localization models offer a significant boost on
the mAP for IoU ≥ 0.7 and the COCO style mAP, relative
to the bounding box regression model (Bbox reg.) under all
the tested cases. The improvement on both of them is on
average 7 points. Our models also improve for the mAP
with IoU≥ 0.5 case but with a smaller amount (around 0.7
points). In Figure 8 we plot the mAP as a function of the
IoU threshold. We can observe that the improvement on the
detection performance thanks to the proposed localization
models starts to clearly appear on the 0.65 IoU threshold
and then grows wider till the 0.9. In Table 3 we provide
the per class AP results on VOC2007 for the best approach
on each metric. In the same table we also report the AP re-

Figure 7: Recall of ground truth bounding boxes as a function of the IoU threshold on PASCAL VOC2007 test set. Note that, because
we perform class-specific localization the recall that those plots report is obtained after averaging the per class recalls. Top-Left: Recalls
for the Reduced MR-CNN model after one iteration of the detection pipeline. Bottom-Left: Recalls for the Reduced MR-CNN model
after four iterations of the detection pipeline.Top-Right: Recalls for the Fast-RCNN model after one iteration of the detection pipeline.
Bottom-Right: Recalls for the Fast-RCNN model after four iterations of the detection pipeline.

Figure 8: mAP as a function of the IoU threshold on PASCAL VOC2007 test set. Left plot: includes the configurations with the ReducedMR-CNN recognition model. Right plot: includes the configurations with the Fast-RCNN recognition model.

Localization
–
InOut ML
Borders ML
Combined ML
Bbox reg.
–
InOut ML
Borders ML
Combined ML
Bbox reg.
–
InOut ML
Borders ML
Combined ML
Bbox reg.
–
InOut ML
Borders ML
Combined ML
Bbox reg.

Detection Pipeline
Recognition
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Reduced-MR-CNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN
Fast-RCNN

IoU ≥ 0.5
0.747
0.783
0.780
0.784
0.777
0.719
0.782
0.777
0.781
0.774
0.729
0.779
0.774
0.780
0.773
0.710
0.777
0.772
0.775
0.769

Initial Boxes
2k Edge Box
2k Edge Box
2k Edge Box
2k Edge Box
2k Edge Box
2k Sel. Search
2k Sel. Search
2k Sel. Search
2k Sel. Search
2k Sel. Search
2k Edge Box
2k Edge Box
2k Edge Box
2k Edge Box
2k Edge Box
2k Sel. Search
2k Sel. Search
2k Sel. Search
2k Sel. Search
2k Sel. Search

mAP
IoU ≥ 0.7
0.434
0.654
0.644
0.650
0.570
0.456
0.654
0.648
0.653
0.584
0.427
0.651
0.641
0.648
0.570
0.446
0.645
0.640
0.645
0.579

COCO style
0.362
0.522
0.525
0.530
0.452
0.368
0.529
0.530
0.535
0.460
0.356
0.522
0.522
0.530
0.453
0.362
0.526
0.526
0.532
0.458

Table 2: mAP results on VOC2007 test set for IoU thresholds of 0.5 and 0.7 as well as the COCO style mAP that averages the traditional
AP for various IoU thresholds between 0.5 and 1 (specifically the thresholds 0.5:0.05:95 are being used). The hyphen symbol (–) indicates
that the localization model was not used at all and that the pipeline ran only for T = 1 iteration. The rest entries are obtained after running
the detection pipeline for T = 4 iterations.
Year
2007
2007
2007
2012
2012
2012

Metric
IoU ≥ 0.5
IoU ≥ 0.7
COCO style
IoU ≥ 0.5
IoU ≥ 0.5
IoU ≥ 0.5

Approach
Reduced-MR-CNN & Combined ML & EB
Reduced-MR-CNN & In Out ML & EB
Reduced-MR-CNN & Combined ML & SS
Reduced-MR-CNN & In Out ML & EB
Reduced-MR-CNN & Borders ML & EB
Reduced-MR-CNN & Combined ML & EB

areo
0.804
0.707
0.580
0.863
0.865
0.866

bike
0.855
0.742
0.603
0.830
0.827
0.834

bird
0.776
0.622
0.500
0.761
0.755
0.765

boat
0.729
0.481
0.413
0.608
0.602
0.604

bottle
0.622
0.452
0.367
0.546
0.535
0.544

bus
0.868
0.840
0.703
0.799
0.791
0.798

car
0.875
0.747
0.631
0.790
0.785
0.786

cat
0.886
0.786
0.661
0.906
0.902
0.902

chair
0.613
0.429
0.357
0.543
0.533
0.546

cow
0.860
0.730
0.581
0.816
0.800
0.810

table
0.739
0.670
0.500
0.620
0.607
0.618

dog
0.861
0.754
0.620
0.890
0.886
0.889

horse
0.870
0.779
0.625
0.857
0.857
0.857

mbike
0.826
0.669
0.545
0.855
0.848
0.847

person
0.791
0.581
0.494
0.828
0.826
0.828

plant
0.517
0.309
0.269
0.497
0.496
0.498

sheep
0.794
0.655
0.522
0.766
0.765
0.763

sofa
0.752
0.693
0.579
0.675
0.673
0.678

train
0.866
0.736
0.602
0.832
0.831
0.830

tv
0.777
0.690
0.555
0.674
0.676
0.679

mean
0.784
0.654
0.535
0.748
0.743
0.747

Table 3: Per class AP results on VOC2007 and VO2012 test sets.

sults on VOC2012 test set but only for the IoU ≥ 0.5 case
since this is the only metric that the evaluation server provides. In this dataset we achieve mAP of 74.8% which is the
state-of-the-art at the time of writing this paper (6/11/2015).
Finally, in Figure 9 we examine the detection performance
behaviour with respect to the number of iterations used by
our pipeline. We observe that as we increase the number
of iterations, the mAP for high IoU thresholds (e.g. IoU
≥ 0.8) continues to improve while for lower thresholds the
improvements stop on the first two iterations.

5.3. Sliding windows as initial set of candidate boxes
In Table 4 we provide the detection accuracy of our
pipeline when, for generating the initial set of candidate
boxes, we use a simple sliding windows scheme (of 10k
windows per image). We observe that:
• Even in this case, our pipeline achieves very high mAP
results that are close to the ones obtained with selec-

tive search or edge box proposals. We emphasize that
this is true even for the IoU≥ 0.7 or the COCO style
of mAP that favour better localized detections, despite
the fact that in the case of sliding windows the initial
set of candidate boxes is considerably less accurately
localized than in the edge box or in the selective search
cases (see Table 1).
• In the case of sliding windows, just scoring the candidate boxes with the recognition model (hyphen (–)
case) yields much worse mAP results than the selective search or the edge box proposals case. However,
when we use the full detection pipeline that includes
localization models and re-scoring of the new better
localized candidate boxes, then this gap is significantly
reduced.
• The difference in the mAP results between the proposed localization models (In-Out ML, Borders ML,
and Combined ML) and the bounding box regression

Localization
–
InOut ML
Borders ML
Combined ML
Bbox reg.

Detection Pipeline
Recognition
Initial Boxes
Reduced-MR-CNN 10k Sliding Windows
Reduced-MR-CNN 10k Sliding Windows
Reduced-MR-CNN 10k Sliding Windows
Reduced-MR-CNN 10k Sliding Windows
Reduced-MR-CNN 10k Sliding Windows

IoU ≥ 0.5
0.617
0.770
0.764
0.773
0.761

mAP
IoU ≥ 0.7
0.174
0.633
0.626
0.639
0.550

COCO style
0.227
0.513
0.513
0.521
0.436

Table 4: mAP results on VOC2007 test set when using 10k sliding windows as initial set of candidate boxes. In order to generate the sliding
windows we use the publicly available code that accompanies the work of Hosang et al. [15] that includes a sliding window implementation
inspired by BING [4, 38]).

Detection Pipeline
Localization
Recognition
Combined ML Fast R-CNN
Bbox reg.
Fast R-CNN
Combined ML Fast R-CNN

Proposals
Sel. Search
Sel. Search
Sel. Search

Dataset
5K mini-val set
5K mini-val set
test-dev set

mAP
IoU ≥ 0.5 IoU ≥ 0.75
0.424
0.282
0.407
0.202
0.429
0.279

COCO style
0.264
0.214
0.263

Table 5 – Preliminary results on COCO. In those experiments
the Fast R-CNN recognition model uses a softmax classifier [10]
instead of class-specific linear SVMs [11] that are being used for
the PASCAL experiments.

shown in Table 5, where it is observed that the proposed
model boosts the mAP by 5 points in the COCO-style evaluation, 8 points in the IoU ≥ 0.75 case and 1.4 points in the
IoU ≥ 0.5 case.
Figure 9: Plot of the mAP as a function of the iterations number
of our detection pipeline on VOC2007 test set. To generate this
plot we used the Reduced-MR-CNN recognition model with the
In-Out ML localization model and Edge Box proposals.

model (Bbox reg.) is even greater in the case of sliding
windows.
To the best of our knowledge, the above mAP results
are considerably higher than those of any other detection
method when only sliding windows are used for the initial
bounding box proposals (similar experiments are reported
in [10, 15]). We also note that we had not experimented with
increasing the number of sliding windows. Furthermore, the
tested recognition model and localization models were not
re-trained with sliding windows in the training set. As a
result, we foresee that by exploring those two factors one
might be able to further boost the detection performance for
the sliding windows case.

5.4. Preliminary results on COCO
To obtain some preliminary results on COCO, we applied our training procedure on COCO train set. The only
modification was to use 320k iterations (no other parameter
was tuned). Therefore, LocNet results can still be significantly improved but the main goal was to show the relative
difference in performance between the Combined ML localization model and the box regression model. Results are

5.5. Qualitative results
In Figure 10 we provide sample qualitative results that
compare the newly proposed localization models (In-Out
ML, Borders ML, and Combined ML) with the current stateof-the-art bounding box regression localization model.

6. Conclusion
We proposed a novel object localization methodology
that is based on assigning probabilities related to the localization task on each row and column of the region in which
it searches the object. Those probabilities provide useful
information regarding the location of the object inside the
search region and they can be exploited in order to infer its
boundaries with high accuracy.
We implemented our model via using a convolutional
neural network architecture properly adapted for this task,
called LocNet, and we extensively evaluated it on PASCAL VOC2007 test set. We demonstrate that it outperforms CNN-based bounding box regression on all the evaluation metrics and it leads to a significant improvement on
those metrics that reward good localization. Importantly,
LocNet can be easily plugged into existing state-of-the-art
object detection methods, in which case we show that it contributes to significantly boosting their performance. Finally,
we demonstrate that our object detection methodology can
achieve very high mAP results even when the initial set of
candidate boxes is generated by a simple sliding windows
scheme.

(a) Candidate box

(b) Bbox regression

(c) In-Out ML

(d) Borders ML

(e) Combined ML

Figure 10: Qualitative results of the bounding box localization step given an initial candidate box (column (a)) from the bounding box
regression model (column (b)), the In-Out ML localization model (column (c)), the Borders ML localization model (column (d)), and the
Combined ML localization model (column (e)). The candidate box is drawn with yellow color, the predicted boxes are drawn with blue
color, and the ground truth bounding box is drawn with green color.

References
[1] B. Alexe, T. Deselaers, and V. Ferrari. Measuring the objectness of image windows. Pattern Analysis and Machine
Intelligence, IEEE Transactions on, 2012. 2
[2] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition, 2014. 2
[3] J. C. Caicedo and S. Lazebnik. Active object localization

with deep reinforcement learning. In Proceedings of the
IEEE International Conference on Computer Vision, 2015.
3
[4] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. Torr. Bing: Binarized normed gradients for objectness estimation at 300fps.
In Computer Vision and Pattern Recognition (CVPR), 2014
IEEE Conference on, 2014. 10
[5] M. Everingham, L. Van Gool, C. Williams, J. Winn, and

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]
[19]

[20]

[21]

A. Zisserman. The pascal visual object classes challenge
2007 (voc 2007) results (2007), 2008. 2
M. Everingham, L. Van Gool, C. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes challenge
2012, 2012. 2
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 2010. 7
P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained partbased models. Pattern Analysis and Machine Intelligence,
IEEE Transactions on, 2010. 2, 3, 4
S. Gidaris and N. Komodakis. Object detection via a multiregion & semantic segmentation-aware cnn model. In Proceedings of the IEEE International Conference on Computer
Vision, 2015. 2, 3, 4, 7, 13
R. Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, 2015. 2, 4, 7, 10, 13,
14
R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In Computer Vision and Pattern Recognition
(CVPR), 2014 IEEE Conference on, 2014. 2, 7, 10, 13, 14
A. Gonzalez-Garcia, A. Vezhnevets, and V. Ferrari. An active search strategy for efficient object class detection. In
Computer Vision and Pattern Recognition (CVPR), 2015
IEEE Conference on, 2015. 3
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proceedings of the IEEE International Conference on Computer Vision, 2015. 3
K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. IEEE
transactions on pattern analysis and machine intelligence,
2015. 2, 5
J. Hosang, R. Benenson, P. Dollár, and B. Schiele. What
makes for effective detection proposals? arXiv preprint
arXiv:1502.05082, 2015. 4, 7, 10
S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
arXiv preprint arXiv:1502.03167, 2015. 3
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of
the ACM International Conference on Multimedia. ACM,
2014. 6
P. Krähenbühl and V. Koltun. Geodesic object proposals. In
Computer Vision–ECCV 2014. Springer, 2014. 2
P. Krähenbühl and V. Koltun. Learning to propose objects.
In Computer Vision and Pattern Recognition (CVPR), 2015
IEEE Conference on. IEEE, 2015. 2
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in neural information processing systems, 2012. 3
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computation, 1989. 3

[22] K. Lenc and A. Vedaldi. R-cnn minus r. In Proceedings of
the British Machine Vision Conference (BMVC), 2015. 3
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014,
2014. 1
[24] W. Ouyang, P. Luo, X. Zeng, S. Qiu, Y. Tian, H. Li, S. Yang,
Z. Wang, Y. Xiong, C. Qian, et al. Deepid-net: multi-stage
and deformable deep convolutional neural networks for object detection. arXiv preprint arXiv:1409.3505, 2014. 2
[25] G. Papandreou, I. Kokkinos, and P.-A. Savalle. Modeling local and global deformations in deep learning: Epitomic convolution, multiple instance learning, and sliding window detection. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2015. 3
[26] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In Advances in Neural Information
Processing Systems, 2015. 2, 5
[27] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You
only look once: Unified, real-time object detection. arXiv
preprint arXiv:1506.02640, 2015. 3
[28] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards
real-time object detection with region proposal networks. In
Advances in Neural Information Processing Systems, 2015.
2, 3, 14
[29] S. Ren, K. He, R. Girshick, X. Zhang, and J. Sun. Object
detection networks on convolutional feature maps. arXiv
preprint arXiv:1504.06066, 2015. 2
[30] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. arXiv preprint
arXiv:1312.6229, 2013. 2, 3
[31] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014. 3, 5, 6, 14
[32] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2015. 3
[33] C. Szegedy, S. Reed, D. Erhan, and D. Anguelov.
Scalable, high-quality object detection. arXiv preprint
arXiv:1412.1441, 2014. 2
[34] C. Szegedy, A. Toshev, and D. Erhan. Deep neural networks
for object detection. In Advances in Neural Information Processing Systems, 2013. 2
[35] K. E. Van de Sande, J. R. Uijlings, T. Gevers, and A. W.
Smeulders. Segmentation as selective search for object
recognition. In Computer Vision (ICCV), 2011 IEEE International Conference on, 2011. 2, 3, 6
[36] D. Yoo, S. Park, J.-Y. Lee, A. S. Paek, and I. So Kweon. Attentionnet: Aggregating weak directions for accurate object
detection. In Proceedings of the IEEE International Conference on Computer Vision, 2015. 3
[37] Y. Zhang, K. Sohn, R. Villegas, G. Pan, and H. Lee. Improving object detection with deep convolutional networks

via bayesian optimization and structured prediction. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2015. 2
[38] Q. Zhao, Z. Liu, and B. Yin. Cracking bing and beyond.
In Proceedings of the British Machine Vision Conference.
BMVA Press, 2014. 10
[39] Y. Zhu, R. Urtasun, R. Salakhutdinov, and S. Fidler.
segdeepm: Exploiting segmentation and context in deep neural networks for object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2015. 2
[40] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In Computer Vision–ECCV 2014, 2014.
2, 3, 6, 7

A. Bounding box regression model
This localization model consists of four CNN-based
scalar regression functions fx (B, c), fy (B, c), fw (B, c),
and fh (B, c) that given a candidate box B and a category
c, they actually predict the coefficients of a geometric transformation that will ideally map the candidate box B to a
ground truth bounding box of the c object category [11].
Specifically, if B = (Bx , By , Bw , Bh ) are the coordinates
of the candidate box in form of its top-left corner (Bx , By )
and its width and height (Bw , Bh ), then the predicted candidate box B̂ is given by the following equations:
B̂x = Bw · fx (B, c) + Bx

(9)

B̂y = Bh · fy (B, c) + By

(10)

B̂w = Bw · exp(fw (B, c))

(11)

B̂h = Bh · exp(fh (B, c)).

(12)

Hence, the four scalar target regression values T =
{tx , ty , tw , th } for the ground truth bounding box B gt =
gt
, Bhgt ) are defined as:
(Bxgt , Bygt , Bw
Bxgt − Bx
Bw
B gt
tw = log( w )
Bw
tx =

Bygt − By
Bh
B gt
th = log( h )).
Bh
ty =

(13)
(14)

For the CNN architecture that implements the bounding
box regression model we adopt the one proposed in [9]. As
a loss function we use the sum of euclidean distances between the target values and the predicted values of each
training sample. The final fully connected layer is initialized from a Gaussian distribution with standard deviation
of 0.01. The rest training details (i.e. SGD, mini-batch, definition of training samples) are similar to those described
for the proposed localization models.

B. Training the localization models
As proposed in Fast-RCNN [10], when training the
bounding box regression model we simultaneously train the

Localization model
Method
Region Feat. Size
Bbox reg.
14 x 14 x 512
Bbox reg.
7 x 7 x 512
InOut ML
14 x 14 x 512
Borders ML
14 x 14 x 512
14 x 14 x 512
Combined ML

mAP VOC2007 test
IoU ≥ 0.5 IoU ≥ 0.7 COCO style
0.777
0.568
0.453
0.777
0.570
0.452
0.783
0.654
0.522
0.780
0.644
0.525
0.784
0.650
0.530

Table 6: mAP results on VOC2007 test set. All the entries use the
Reduced-MR-CNN recognition model and Edge Box proposals for
the initial set of boxes. In the second column (Region Feat. Size)
we provide the region activation maps size that the region adaptive
max-pooling layer yields.

Fast-RCNN recognition model with the two models sharing their convolutional layers. In our experiments, this way
of training improves the accuracy of both the Fast-RCNN
recognition model and the bounding box regression model.
On the contrary, the newly proposed localization models (i.e., Borders ML, In-Out ML, and Combined ML) are
not currently trained simultaneously with the recognition
model. We expect that the joint training of these models
with the recognition model can help to further improve their
overall performance, which is therefore something that we
plan to explore in the future.

C. Region activation maps size in bounding
box regression model
Early in our work we tried to boost the localization performance of the VGG16-based box regression model by increasing the size of the region activation maps from 7x7 to
14x14. The resulting network had around 422M parameters, and in order to train it successfully we had to initialize
the fc6 layer by up-sampling with bilinear interpolation the
initial 7x7x512x4096 weights (pre-trained on ImageNet) of
it to 14x14x512x4096 and then divide them by a factor of
142
72 (to leave unchanged the expected magnitude of its activations). Even with this initialization scheme, the difference w.r.t. the 7x7 case is negligible (see Table 6) and thus
this models was not used. Note that, despite the fact that the
region activation maps of LocNet are of size 14x14x512, it
contains only 26M and 34M parameters for the InOut and
Borders cases correspondingly (which is almost 15 times
less than the 14x14 regression case) and it does not need
any explicit care in the initialization of its extra layers.

D. Recognition models
Reduced MR-CNN model: We based the implementation of this model on the state-of-the-art MR-CNN detection system that was proposed in [9]. Briefly, the MR-CNN
detection system recognises a candidate box by focusing on
multiple regions of it, each with a dedicated network component termed as region adaptation module. In our implementation however, for efficiency reasons and in order to
speed up the experiments, we applied the following reduc-

tions:
• We include only six out of the ten regions proposed,
by skipping the half regions.
• We do not include the semantic segmentation-aware
CNN features.
• We reduce the total amount parameters on the region
adaptation modules.
In order to achieve the reduction of parameters on the
hidden fully connected layers fc6 and fc7 of the region
adaptation modules, each of them is decomposed on two
fully connected layers without any non-linearities between
them. Specifically, the fc6 layer with weight matrix W6 :
25088 × 4096 is decomposed on the layers fc6 1 and fc6 2
with weight matrices W6.1 : 25088 × 1024 and W6.2 :
1024 × 4096 correspondingly. The fc7 layer with weight
matrix W7 : 4096 × 4096 is decomposed on the layers
fc7 1 and fc7 2 with weight matrices W7.1 : 4096×256 and
W7.2 : 256 × 4096 correspondingly. To train the Reduced
MR-CNN network, we first train only the original candidate box region of it without reducing the parameters of the
fc6 and fc7 layers. Then, we apply the truncated SVD decomposition on the aforementioned layers (for more details
see section §3.1 of [10]) that results on the layers fc6 1,
fc6 2, fc7 1, and fc7 2. We copy the parameters of the resulting fully connected layers to the corresponding layers
of the rest region adaptation modules of the model and we
continue training.
Fast-RCNN model: We re-implemented Fast-RCNN
based on the publicly available code provided from FastRCNN [10] and Faster-RCNN [28]. Here we will describe
only the differences of our implementation with the original Fast-RCNN system [10]. In our implementation, we
have different branches for the recognition of a candidate
box and for its bounding box regression after the last convlutional layer (conv5 3 for the VGG16-Net [31]) that do
not share any weights. In contrary, the original Fast-RCNN
model splits to two branches after the last hidden layer. We
applied this modification because, in our case, the candidate box that is fed to the regression branch is enlarged by a
factor α = 1.3 while the candidate box that is fed to recognition branch is not. Also, after the fine-tuning, we remove
the softmax layer of the recognition branch and we train
linear SVMs on top of the features that the last hidden layer
of the recognition branch yields, just as R-CNN [11] does.
Finally, we do not reduce the parameters of the fully connected layers by applying the truncated SVD decomposition
on them as in the original paper. In our experiments those
changes improved the detection performance of the model.

Algorithm 2: Object detection pipeline
Input : Image I, initial set of candidate boxes
{B1c }C
c=1
Output: Final list of per category detections {Yc }C
c=1
for t ← 1 to T do
for c ← 1 to C do
Stc ← Recognition(Btc |I, c)
if t == 1 then
{Stc , Btc } ←
P runeCandidateBoxes({Stc , Btc })
end
end
if t < T then
for c ← 1 to C do
Bt+1
← Localization(Btc |I, c)
c
end
end
end
for c ← 1 to C do
Dc ← ∪Tt=1 {Stc , Btc }
Yc ← PostProcess(Dc )
end

E. Object detection pipeline
In Algorithm 1 of section §2 we provide the pseudo-code
of the object detection pipeline that we adopt. For clarity purposes, the pseudo-code that is given corresponds to
the single object category detection case and not the multiple object categories case that we are dealing with. Moreover, the actual detection algorithm, after scoring the candidate boxes for the first time t = 1, it prunes the candidate
boxes with low confidence in order to reduce the computational burden of the subsequent iterations. For that purpose, we threshold the candidate boxes of each category
such that their average number per image and per category
to be around 18 boxes. Also, during this step, non-maxsuppression with IoU of 0.95 is applied in order to remove
near duplicate candidate boxes (in the case of using sliding
windows to generate the initial set of candidate boxes this
IoU threshold is set to 0.85). A more detailed algorithm
of our detection pipeline is presented in Algorithm 2. Note
that, since the initial candidate boxes {B1c }C
c=1 are coming
from a category-agnostic bounding box proposal algorithm,
those boxes are the same for all the categories and when
applying on them (during t = 1 iteration) the recognition
module, the computation between all the categories can be
shared.

