\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{comment}
\usepackage{amsmath,amssymb} %
\usepackage{color}
\usepackage{xspace}
\usepackage{tabularx,colortbl}
\usepackage{graphicx, caption, subcaption}
\newcommand{\seenclasses}{$\mathrm{S}$}
\newcommand{\trainclasses}{${\mathrm{L}}$ }
\newcommand{\unseenclasses}{${\mathrm{U}}$ }
\newcommand{\seendataset}{${\mathrm{D_{L}}}$ }
\newcommand{\seenimages}{${\mathrm{I_{S}}}$ }
\newcommand{\oivlong}{Open Images V4\xspace}
\newcommand{\oiv}{OIV4\xspace}
\newcommand{\oivsource}{OIV4-source\xspace}
\newcommand{\oivtarget}{OIV4-target\xspace}
\newcommand{\oivsourcetrain}{OIV4-source-train\xspace}
\newcommand{\oivsourceval}{OIV4-source-val\xspace}
\newcommand{\oivtargettrain}{OIV4-target-train\xspace}
\newcommand{\oivtargetval}{OIV4-target-val\xspace}
\newcommand{\oivall}{OIV4-all\xspace}
\newcommand{\cocoall}{COCO-all\xspace}
\newcommand{\coco}{COCO\xspace}
\newcommand{\cocotarget}{COCO-target\xspace}
\newcommand{\cocosource}{COCO-source\xspace}
\newcommand{\AR}[1] {AR@#1}
\newcommand{\frcnn}{Faster R-CNN\xspace}
\newcommand{\retina}{RetinaNet\xspace}
\newcommand{\deepti}[1]{{\color{blue}{Deepti: #1}}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
\newcolumntype{C}[1]{>{\centering\arraybackslash}c{#1}}
\begin{document}
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{13}  %
\title{What leads to generalization of object proposals?} %
\author{Rui Wang \and
Dhruv Mahajan \and
Vignesh Ramanathan}
\authorrunning{R. Wang et al.}
\institute{Facebook AI \\
\email{\{ruiw, dhruvm, vigneshr\}@fb.com}}
\maketitle

\begin{abstract}
Object proposal generation is often the first step in many detection models. It is lucrative to train a good proposal model, that generalizes to unseen classes. This could help scaling detection models to larger number of classes with fewer annotations. Motivated by this, we study how a detection model trained on a small set of source classes can provide proposals that \emph{generalize} to unseen classes. We systematically study the properties of the dataset --  visual diversity and label space granularity -- required for good generalization. We show the trade-off between using fine-grained labels and coarse labels. We introduce the idea of prototypical classes: a set of sufficient and necessary classes required to train a detection model to obtain generalized proposals in a more data-efficient way. On the \oivlong dataset, we show that only $25\%$ of the classes can be selected to form such a prototypical set. The resulting proposals from a model trained with these classes is only $4.3\%$ worse than using all the classes, in terms of average recall (AR). We also demonstrate that \frcnn model leads to better generalization of proposals compared to a single-stage network like \retina.

\keywords{object proposals, object detection, generalization}
\end{abstract}
\section{Introduction}
\label{sec:intro}
Object detection systems have shown considerable improvements for fully supervised settings \cite{ren2015faster,lin2017focal,liu2016ssd,redmon2017yolo9000,dai2016r}, as well as weakly supervised settings~\cite{Gao_2019_ICCV,arun2019dissimilarity,tang2018pcl} that only use image-level labels. Both approaches typically consider detection as a combination of two tasks: (a) spatial localization of the objects using proposals and (b) classification of the proposals into correct classes.
A generalized proposal model that localizes all classes can help in scaling object detection. This could lead to the use of fewer or no bounding box annotations to only solve the classification task and development of more sophisticated classifiers, as explored in works like \cite{uijlings2018revisiting,singh2018r}.
Many detection models \cite{ren2015faster,lin2017focal} have been developed in recent years, which can be used to obtain high quality object proposals. However, an equally important aspect that determines the generalization ability of proposals is \emph{the dataset} used to train these models. As illustrated in Fig.~\ref{fig:pull_fig}, the objects and class labels in a dataset significantly impact the ability to generalize to new classes. Intuitively, to localize a fine-grained vehicle like taxi in a target dataset, it might be sufficient to train a localization model with other vehicles like cars or vans in the source dataset. For localization (unlike classification), we may not need any training data for this class. On the other hand, training with these classes will not help in localizing other vehicles like boat.

 
 
While few works leverage this intuition for weakly supervised learning~\cite{uijlings2018revisiting}, the extent to which object localization depends on the categories used to train the model has not been well quantified and studied in detail. Towards this end, we define ``generalization" as the ability of a model to localize (not classify) objects not annotated in the training dataset. In our work, we answer the question: \emph{What kind of dataset is best suited to train a model that generalizes even to unseen object classes?}
We further study the ability of popular detection models like \frcnn \cite{ren2015faster} and \retina \cite{lin2017focal} to generate proposals that generalize to unseen classes. These networks are designed to improve the detection quality for the small set of seen classes in the training dataset. We carefully study these design choices and provide a way to obtain proposals that generalize to a larger set of unseen classes.
\begin{figure}[t!]
\centering
\includegraphics[width=0.95\linewidth]{figures/rui_pull2}
\caption{Proposal models learned on seen vehicle classes can localize unseen classes which share similar localization structure like ``bus" and ``taxi". However, ``barge" and ``gondola", which are also vehicles will not be precisely localized by this model, due to lack of visual diversity in the training dataset for vehicles}
\label{fig:pull_fig}
\vspace{-0.2in}
\end{figure}
We answer several questions about dataset properties and modeling choices required for generalized proposals:
\begin{itemize}

    \item \textbf{What are the properties of object classes to ensure generalization of proposals from a model?} First, we show that it is crucial to have visual diversity to obtain generalized proposals. We need examples of different vehicles like ``car" and ``boats", even if the examples are only labelled as ``vehicle". Further, we hypothesize the existence of {\it{prototypical classes}} as a subset of leaf classes in a semantic hierarchy that are sufficient and necessary to construct a dataset to train a model for proposal generalization. We define new quantitative metrics to measure these properties for any set of classes and show that it is possible to construct a small prototypical set of object classes. This has positive implications for large taxonomies, since it is sufficient to annotate examples only for the prototypical classes.

    \item \textbf{Does the label-granularity of the dataset affect generalization? If so, what is the coarsest granularity that can be used?} Coarse-grained labels (``vehicles" instead of ``taxis") are significantly less tedious to annotate and more accurate than fine-grained labels. Past works like RFCNN-3000 \cite{singh2018r} argued that a single super class might be sufficient to obtain good proposals. However, we show that there is a trade-off between using very few coarse classes and large-number of fine-grained classes, and a middle-ground approach leads to best generalization.

    \item \textbf{What are the \emph{modeling} choices that are critical for leveraging state-of-the-art detectors to obtain generalized proposals?} We show that: (a) detections from two-stage networks like \frcnn are better for obtaining generalized proposals than a single-stage network like \retina, (b) while class-specific bounding box regression is typically used in \frcnn, it is beneficial only when considering larger number of proposals (average recall AR@1000) and class-agnostic regression is better when considering fewer proposals (AR@100) and (c) choice of NMS threshold is dependent on the number of proposals being considered (AR@100 or AR@1000).
    
    
    
\end{itemize}

On \oiv \cite{kuznetsova2018open}, we show that compared to training with all the object classes, using a prototypical subset of $25\%$ of the object classes only leads to a drop of $4.3\%$ in average recall (AR@100), while training with $50\%$ of such classes leads to a negligible drop of $0.9\%$.
We also show how the detections from \frcnn can be fused to obtain high quality proposals that have $10\%$ absolute gain in AR@100 compared to the class-agnostic proposals of the RPN from the same network and $3.5\%$ better than \retina. To stress the practical importance of generalized proposals, we also show that generalization ability is directly correlated with the performance of weakly supervised detection models.

\section{Related Work}
\label{sec:relwork}
\noindent \textbf{Generalizing localization across multiple classes: }The idea of different object classes sharing the same structure has been exploited in building detection models for a long time\cite{felzenszwalb2009object,novotny2016have,ott2011shared,salakhutdinov2011learning,torralba2004sharing}. More recently, \cite{dai2016r,ren2015faster} also have a dedicated proposal network for object localization. However these works do not measure the transferability of proposals trained on one set of classes to another.

Uijlings \textit{et al.} \cite{uijlings2018revisiting} tried to transfer information from coarse source classes to fine-grained target classes that share similar localization properties. They showed that this can help weakly supervised detection for the target classes. LSDA \cite{hoffman2014lsda} transformed classifiers into detectors by sharing knowledge between classes. Multiple works \cite{tang2016large,hoffman2016large,rochan2015weakly,guillaumin2012large} showed the benefit of sharing localization information between similar classes to improve semi supervised and weakly supervised detection.
Yang \textit{et al.} \cite{yang2019detecting} trained a large-scale detection model following similar principles. Singh \textit{et al.} \cite{singh2018r} showed that even a detector trained with one class can localize objects of different classes sufficiently well due to commonality between classes. We generalize this idea further. There has also been work on learning models \cite{yang2019detecting,redmon2017yolo9000,gao2019note} with a combination of bounding boxes for certain classes and only class labels for others. They inherently leverage the idea that localization can generalize across multiple classes. We provide systematic ways to quantify and measure this property for proposal models. 

\noindent \textbf{Object proposal generation models:}
There have been many seminal works on generating class-agnostic object proposals  \cite{uijlings2013selective,zitnick2014edge,pont2016multiscale,krahenbuhl2014geodesic}. A comprehensive study of different methods can be found in \cite{hosang2015makes} and a study of proposal evaluation metrics can be found in \cite{chavali2016object}. Proposal models have also been trained with dedicated architectures and objectives in \cite{pinheiro2015learning,kuo2015deepbox,szegedy2014scalable}.
In our work, we leverage standard models like \frcnn and focus on the dataset properties required to achieve generalization with this model. 
\section{Approach}
\vspace{-0.1in}
\label{sec:approach}
We study two important aspects involved in obtaining generalized proposals from a detection model:

(1) {\bf{Data Properties}} such as the granularity of the label space (shown in Fig.~\ref{fig:g1}), and the visual diversity of object classes under each label, required for generalization of proposals. The idea of label granularity and visual diversity is shown in Fig.~\ref{fig:g2}. We investigate how a smaller subset of ``prototypical" object classes in a dataset which is representative of all other classes can be identified.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[t]{0.56\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/label_granularity.pdf}
        \caption{Label semantic hierarchy}
        \label{fig:g1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.4\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/rui_figs_gran_v2.pdf}
        \caption{Granularity vs. Diversity}
        \label{fig:g2}
    \end{subfigure}
    \caption{We study two important dataset properties needed to train a proposal model: label granularity and visual diversity. (a) Label granularity can be represented by different levels in a semantic hierarchy as shown. (b) The difference between label granularity and visual diversity is illustrated. At the same granularity, we can either have high or low visual diversity as shown}
    \label{fig:gran_visual}
    \vspace{-0.2in}
\end{figure}
(2) {\bf{Modeling Choice}} for leveraging a detector trained on a dataset with seen classes to obtain proposals that generalize to unseen classes.
\subsection{Dataset Properties}
\label{sec:data_prop}
The choice of labels and data used to train the model is crucial for generalization. To study these properties, we assume: (a) classes are organized in a semantic tree and (b) internal nodes do not have any data of their own, that are not categorized into one of its child nodes. In practice, such a hierarchy is either already available (\oiv) or can be obtained from Wordnet~\cite{wordnet}.  These assumptions help us study the datasets under controlled settings. However, later we explore a way to identify ``prototypical" subsets even when a semantic hierarchy is unavailable.
\subsubsection{Label Space Granularity}
\label{sec:label_space}
As we noted through some examples earlier, it is intuitive that we might not need fine-grained labels to train a good localization model. To quantitatively study the effect of granularity, we construct different datasets with the same set of images and object bounding boxes, but consider classes at different levels of semantic hierarchy (Fig.~\ref{fig:g1}). We then train a model with these datasets and evaluate the generalization ability as a function of label granularity. For instance, for the coarsest root level, we assign all the bounding boxes the same ``object" label and train a detector to distinguish objects from all non-objects. This pertains to the idea of objectness used in weakly supervised algorithms~\cite{uijlings2013selective} and super-class in \cite{singh2018r}. For an intermediate level, we collapse all leaf-labels to their corresponding parent labels at that level to train the model. While a fine-grained label space provides more information, a model trained at this level also attempts to distinguish object classes with similar structure and this could affect generalization. We quantify this trade-off in Sec.~\ref{sec:exp_data}.
\subsubsection{Prototypical classes to capture visual diversity}
\label{sec:proto}
One of the main aims of our work is to see if we can identify a significantly smaller number of classes than the full object-label space, so that bounding boxes from this set of classes are sufficient to train a generalized proposal model. Note that in Sec.~\ref{sec:label_space}, we wanted to study if a small set of coarse labels are sufficient to train a generalized proposal model. However, this does not answer anything about the visual diversity of objects within each sub-category that is required for generalization. As an example (shown in Fig.~\ref{fig:gran_visual}), in order to localize different types of vehicles like ``car" or ``airplane" it might be sufficient to collapse the label for all these objects into a single label named ``vehicle", however dropping all instances of airplane during training will lead to a drop in performance for this class.
To quantitatively study this effect, we introduce the notion of ``prototypical" classes. Given a large set of leaf classes, these are the smallest subset such that a model trained only with instances from them is sufficient to localize objects from the remaining classes. Note that due to the long-tail distribution of real-world data, obtaining images for large number of semantic classes is a tedious task. If a small set of prototypical classes does exist, this makes the data collection process much easier when scaling detection to large number of classes.

\noindent{\bf{Properties: }}We identify the two properties that are required to quantify the prototypicality of a set of classes :

\textit{Sufficient set}: is a set of classes such that training a model only with examples from them should be sufficient to localize objects from all other classes. The most superfluous sufficient set would be the entire set of leaf classes themselves.

\textit{Necessary set}: is a set of classes such that dropping any class from this set will lead to a significant drop in generalization. A simple example would be a very coarse vertical like ``vehicle". Intuitively dropping all vehicles would affect their localization as they do not share localization properties with other classes.

We provide concrete ways to measure both these properties in Sec.~\ref{sec:exp_data}.

\noindent{\bf{Identifying prototypical classes: }}
Given a set of $N$ leaf classes $\mathbb{C}$, we wish to identify a set of $P$ prototypical classes $\mathbb{P} \subset \mathbb{C}$. Intuitively, this is similar to
clustering the classes that have the same localization structure and then choosing a representative class from each cluster. Below, we discuss three approaches:

\noindent(a) \textbf{Oracle visual clustering}: To get an upper bound for choosing the best $P$ prototypical classes, we assume that bounding box annotations for all the $N$ leaf classes are available. We then use these bounding boxes to compute visual similarity between classes. We note that this is not a practical approach, but is crucial to evaluate the effectiveness of proxies we introduce later.

We first train a detection model using the annotations of all the leaf classes. We then measure the visual similarity between two classes $i, j$ as
\vspace{-0.05in}
{\small
\begin{align}
    \label{eq:max_ap}
    S_{ij} = \max \left( \frac{\text{AP}^i(j)}{\text{AP}^j(j)}, \frac{\text{AP}^j(i)}{\text{AP}^i(i)}\right),
\end{align}}where $AP^i(j)$ is the detection average precision (AP) for the $j^{th}$ class when we use the detections corresponding to the $i^{th}$ class as detections of class $j$. $S_{ij}$ is a measure of how well one class can replace another class in localizing it. We then use the resulting similarity measure to hierarchically cluster the classes into $P$ clusters using agglomerative clustering. We then pick the class with the highest number of examples in each cluster to construct the set of prototypical classes. For practical reasons, we use frequency to choose the representative class, since this results in the construction of the largest dataset.
\noindent(b) \textbf{Semantic clustering based on frequency}: Semantic similarity is often viewed as a good proxy for visual similarity as shown through datasets like Imagenet \cite{deng2009imagenet} and \oiv. Hence, we use the semantic tree to cluster the classes in an hierarchical fashion starting from the leaves. At any given step, we cluster together two leaf classes that share a common parent if they jointly have the lowest number of examples. The algorithm stops when $P$ clusters are left. We then select the most frequent class from each cluster as a prototypical class. Here we assume that apriori we know the frequency of each class in a dataset. This is a very weak assumption, since a rough estimate of class distribution in a dataset can often be obtained even from weak labels like hashtags. This doesn't require any image-level label or bounding boxes and is easy to implement in practice. 

\noindent(c) \textbf{Most frequent prototypical subset}: For this baseline, we choose the top $P$ most frequently occurring classes in the dataset as the prototypical classes. Note that unlike the previous approaches, this does not require any knowledge of the semantic hierarchy.
\subsection{Modeling Choice\label{subsec:model}}
\label{sec:model_choice}
Once the dataset is fixed, the next step is to train a detection model. In our work, we explore the use of two models: \frcnn and \retina. The observations made in our work should nevertheless generalize to other two-stage and single-stage detection models as well.
In the case of a single-stage network, the detections from a model trained on a source dataset with seen classes can directly be treated as proposals. Their ability to localize novel classes in a target dataset can be evaluated to test generalization. However, for a two-stage network, another natural choice would be to use the Region Proposal Network (RPN) of the model, since it is trained in a class-agnostic fashion and aims to localize all objects in the image. However, as noted by He et al. \cite{he2017mask}, the detection part of the model is better at localizing the object due to more fine-tuned bounding box regression and better background classification. We study this more rigorously, by comparing the generalization of proposals obtained from the detection head as well as RPN. We vary different model parameters to obtain the optimal setting for proposal generalization.

\section{Experiments}
\label{sec:expts}
We evaluate the ability of the object proposal obtained from detection models learned with different settings in Section~\ref{sec:model_choice} to generalize to new unseen classes. We also explore the effects of label-space granularity and the need for semantic and visual diversity. Finally, we show that a small set of prototypical classes could be used to train an effective proposal model for all classes in the dataset.

\subsection{Experimental Setup}
\noindent \textbf{Source and target splits: } We split each dataset into two parts: (a) {\it{Source dataset}} consisting of a set of seen classes called {\it{source classes}} and (b) {\it{Target dataset}} consisting of a set of unseen classes called {\it{target classes}}. {\it{Target dataset}} is used to evaluate the generalization of proposal models trained with the {\it{Source dataset}}. Since an image can contain both source and target classes, we ensure that such images are not present in the source class dataset. However, there may be a small number of images in the target dataset that contain source classes. We use the following two datasets for our experiments:

(1) {\it{\oivlong(\oiv)~\cite{kuznetsova2018open}}} consists of $600$ classes. We retain only object classes which have more than $100$ training images. This results in a total of $482$ leaf classes. We randomly split all the leaf classes into $432$ source (\oivsource dataset) and $50$ target (\oivtarget dataset) classes. There are also annotations associated only with internal nodes (for example, "animal") and without a specific leaf label (like the type of animal). We remove such annotations and all associated images, since such images cannot be unambiguously assigned to a source or target split. This leaves us with $1.2M$ images with $7.96M$ boxes in the train split and $73k$ images with $361K$ boxes in the test split. For training proposal models, we always use the train split and for evaluation we use the test split. Wherever needed, we explicitly suffix the dataset with "train" and "test" (for example, \oivsource-train and \oivsource-test).

(2) {\it{\coco~\cite{coco}}}: We use the 2017 version of the \coco dataset and randomly split the classes in to $70$ source (\cocosource dataset) and $10$ target (\cocotarget dataset) classes. For training, we use the train split and for evaluation, we use the $5000$ images from the validation set. Wherever needed, we explicitly suffix the dataset with ``train" and ``test".

Target classes list is provided in the supplementary.

\noindent\textbf{Evaluation metrics: }
We report the standard average recall (\AR{k})~\cite{hosang2015makes} metric to evaluate the quality of proposals. One of the main motivations for building a generalized proposal model is to use the resulting proposals to train detection models for unseen classes with limited or no bounding box annotation. A typical proposal-based supervised detection model RCNN could also be used to evaluate the quality of proposals. However, the application to weakly supervised detection is more compelling since their performance is closely tied to proposals than supervised models which can correct the inaccuracies in proposals due to availability of labelled bounding boxes.  Hence, we implement a weakly supervised detector with the approach used in YOLO9000~\cite{redmon2017yolo9000}\footnote{We chose~\cite{redmon2017yolo9000} due to its simplicity. In practice, we can use other weakly supervised approaches too.}. We report the detection AP (averaged over IoU thresholds ranging from $0.5$ to $0.95$) on the test set of the target dataset. Please see the supplementary material for more details.

\noindent\textbf{Implementation details: }
We fix Imagenet pre-trained ResNet-50 with Feature Pyramid Networks \cite{lin2017feature} as the backbone for all models. We use the Detectron codebase~\cite{girshick2018detectron}. For \coco, we train the models for $90k$ iterations with an initial learning rate and the decay suggested in \cite{ren2015faster}. For \oiv, we train the models for $800k$ iterations with an initial learning rate of $0.01$ and cosine learning rate decay. When training the weakly supervised model (\cite{redmon2017yolo9000}), we use the top $100$ proposals in each image to choose pseudo ground truth at every training iteration.
\subsection{Modeling Choices}
We first identify the best detection model and setting to extract proposals that generalize to new unseen classes. We then analyze generalization ability under different settings from this model. We reiterate that in order to test generalization, evaluation is done on target classes that have no intersection with the source classes used during training.

\noindent {\textbf{Choice of detection model:}} We compare the generalization ability of a two-stage network (\frcnn) and a single-stage network (\retina) in Fig.~\ref{fig:mod1}. Since, in a two-stage model like \frcnn, the output from the RPN is class-agnostic and can be used as proposals too, we compare the performance of the RPN as well. The models are trained on \cocosource-train dataset. We report AR@100 on seen classes in the \cocosource-test dataset, as well as unseen classes in the \cocotarget-test. The difference in performance between seen and unseen classes reflects the generalization gap. We also show an upper-bound performance on \cocotarget-test obtained by models trained on the full training dataset containing both \cocosource-train and \cocotarget-train.

\begin{figure}[t]
    \begin{subfigure}[t]{0.45\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/coco_rpn_vs.pdf}
        \caption{Comparison of detection models}
        \label{fig:mod1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.45\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/coco_ar_breakdown_v2.pdf}
        \caption{RPN vs. detection head}
        \label{fig:mod2}
    \end{subfigure}
    \caption{(a) \AR{100} corresponding to different models trained on \cocosource-train and evaluated on different test splits. Upper-bound corresponds to model trained on full \coco dataset and evaluated on \cocotarget-test. (b) Average recall of RPN and detection head at different IoU thresholds, for model trained on \cocosource-train and evaluated on \cocotarget-test}
    \label{fig:rpn_vs_det}
    \vspace{-0.2in}
\end{figure}

We notice that on seen classes, \retina achieves a lower performance compared to \frcnn (drop of $2.4\%$). However, the drop is larger for unseen target classes ($3.5\%$), indicating a larger generalization gap for \retina. One reason for this is that \retina is more sensitive to missing bounding boxes corresponding to unlabelled unseen classes in the source dataset. Proposals corresponding to unseen object classes that are not annotated in the training data are treated as hard-negatives, due to the use of focal-loss. Hence, the model heavily penalizes proposals corresponding to unannotated bounding boxes, leading to overall drop in AR. Since some seen classes share visual similarity with unseen classes, this sensitivity to missing annotations affects AR for seen classes too. However, this effect is more magnified for unseen target classes. On the other hand, in \frcnn, only a small number of proposals (less than $512$) which do not intersect with annotated bounding boxes are sampled at random as negatives. The probability that a proposal corresponding to an unseen object class is chosen as a negative is lower, leading to better generalization. Hence, for the rest of the paper, we use \frcnn as the detection model.

We also notice that the detection head of \frcnn provides better overall performance \emph{without} sacrificing generalization. This can be attributed to better bounding box regression from the detection head which has additional layers, following the RPN in the model. To investigate this effect, we measure AR at different IoU thresholds for both sets of proposals for the model trained on \cocosource and evaluated on \cocotarget in Fig.~\ref{fig:mod2}. We see that the difference in \AR{1000} increases drastically at higher values of IoU threshold, and is negligible at a threshold of $0.5$. This implies that the boxes from the detection head are more fine-tuned to exactly localize objects, unlike the RPN.

\noindent {\textbf{Choice of \frcnn settings:}}
The results so far were obtained using class-specific bounding box regression (which is the standard setting in \frcnn) for the detection head. Since we want the bounding boxes to generalize to unseen classes, class agnostic regression could be a valid choice too. We study this in Fig.~\ref{fig:cls_ag} for \oiv and \coco. We see that class agnostic regression is better for small number of proposals as seen by \AR{10,20,50}. However, when we consider more proposals (\AR{1000}), class specific regression provides a significant gain ($4.5\%$ for \oiv and $7.5\%$ for \coco). It results in multiple regressed versions (one corresponding to each class) of the same proposal generated from the RPN. This helps in improving recall at higher number of proposals.

Previously, we fixed the NMS threshold to $0.5$. We study the effect of this threshold in Fig.~\ref{fig:nms_fig}. We train on \oivsource, \cocosource and test on \oivtarget, \cocotarget respectively. Intuitively, a low threshold can improve spatial coverage of objects by ensuring proposals are spatially well spread out. When considering a larger number of proposals, there are sufficient boxes to ensure spatial coverage, and having some redundancy is helpful. This is witnessed by the steeper drop in \AR{1000} at low NMS thresholds, unlike \AR{100}. 

Based on these observations, we use class-specific bounding box regression with an NMS threshold of $0.5$ for rest of the experiments.
\begin{figure}[t]
\vspace{-0.1in}
  \centering
  \begin{minipage}[t]{0.45\textwidth}
    \includegraphics[width=\textwidth]{figures/cls_specific_vs_agnostic.pdf}
    \caption{Effect of class agnostic regression vs. class specific regression}
    \label{fig:cls_ag}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.45\textwidth}
    \includegraphics[width=\textwidth]{figures/nms.pdf}
    \caption{Effect of NMS threshold on performance of proposals}
    \label{fig:nms_fig}
  \end{minipage}
  \vspace{-0.21in}
\end{figure}
\begin{table}[h]
\vspace{-0.4in}
\centering
\begin{center}
\caption{Comparing performance of proposals generated by RPN head and detection head for weakly supervised detection. We also show the \AR{100} numbers which are seen to be correlated with detection AP}\label{tab:det_map}
\begin{tabular}{l|c|c|c|c}
\hline
\multicolumn{5}{c}{Target Dataset - \oivtarget}\\
\hline
 & \multicolumn{2}{c|}{Source: \oivsource} & \multicolumn{2}{c}{Source: \oivall}\\
& Det. AP & \AR{100} & Det. AP & \AR{100} \\\hline
\frcnn RPN & 8.7 & 55.0 & 9.6 & 60.4\\
\frcnn Detection & \textbf{24.0} & \textbf{69.4} & \textbf{30.8} & \textbf{76.9} \\
\hline
\end{tabular}
\end{center}
\vspace{-0.35in}
\end{table}

\noindent {\textbf{Weakly supervised detection:}}
A strong practical utility for generalized proposals that localize all objects is that, no bounding box annotations should be needed to train a detection model for new object classes. Hence, we measure the effect of better generalized proposals on the performance of a weakly supervised detection model, trained without bounding box annotations. We show results corresponding to the RPN head and detection head of \frcnn in Tab.~\ref{tab:det_map}. The weakly supervised model is trained on \oivtarget-train and evaluated on \oivtarget-test. We also show results for proposals obtained from training with \oivsource as well as \oivall (upper-bound). We see that the performance of the weakly supervised detection model is directly correlated with the quality of the proposals being used, showing the need for good generalized proposals.

\subsection{Dataset Properties}
\label{sec:exp_data}
\noindent {\textbf{Effect of label space granularity: }} \oiv organizes object classes in a semantic hierarchy with $5$ levels. We directly leverage this hierarchy to measure the effect of label granularity (Fig.~\ref{fig:g1}). We construct a dataset at each level $L_i$ (\oivsource-$L_i$) by retaining all the images in \oivsource, but relabeling bounding boxes corresponding to leaf labels with their ancestor at $L_i$. We construct 5 datasets, one for each level with the same set of images and bounding boxes.

We report the performance of these models on \oivtarget in Tab.~\ref{tab:label_gran}. Along with \AR{100/1000}, we also report the detection AP of the weakly supervised detection models trained with the proposals obtained from the corresponding levels. The weakly supervised models are trained on \oivtarget-train and evaluated on \oivtarget-test.

\vspace*{-8mm}
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{Effect of different label space granularities on the quality of proposal for \oiv dataset. The number of classes at each level is shown in brackets. Evaluation is done on \oivtarget-eval dataset. Both AR and weakly supervised detection AP are reported}
\label{tab:label_gran}
\begin{tabular}{cccc}
\hline\noalign{\smallskip}
Source Dataset & AR@100 & AR@1000 & AP (weak)\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
\oivsource-$L_0 (1)$ & 61.7 & 72.0 & 19.5\\
\hline
\oivsource-$L_1 (86)$ & 63.4 & 73.0 & 22.6\\
\hline
\oivsource-$L_2 (270)$ & 63.7 & 75.2 & 23.1\\
\hline
\oivsource-$L_3 (398)$ & 65.2 & 77.2 & 24.3\\
\hline
\oivsource-$L_4 (432)$ & 64.2 & 76.1 & 24.0\\
\hline
\end{tabular}
\end{center}
\end{table}
  \vspace{-0.35in}
\setlength{\tabcolsep}{1.4pt}
Some past works like \cite{singh2018r} postulated that one super-class (similar to $L_0$) could be sufficient. However, we observe that both \AR{100} and \AR{1000} increase as we move from $L_0$ to $L_1$ along with a significant gain ($3.1\%$) in AP. This indicates that training with just a binary label yields lower quality proposals compared to training with at least a coarse set of labels at $L_1$. While both AP and \AR{100} increase as the granularity increases from $L_1$ to $L_3$, the difference is fairly small for both metrics ($ < 2\%$ change). However, annotating bounding boxes with labels at $L_1$ ($86$ labels) is significantly cheaper than $L_3$ ($398$ labels). Hence, $L_1$ can be seen as a good trade-off in terms of labelling cost, and training a good model.

\vspace*{0.02in}
\noindent {\textbf{Need for visual and semantic diversity: }}
We noticed that training with coarse labels can yield good proposals. It would be interesting to observe if all or only some of these coarse classes are crucial to build a good proposal model. To study this, we conduct ablation experiments where we train a model with \oivsource-train after dropping all images having a specific $L_1$ label and evaluate the proposals on the \oivsource-test images belonging to this label in Fig.~\ref{fig:drop_classes_fig}a. We repeat this experiment for a few fine-grained classes at $L_4$ in Fig.~\ref{fig:drop_classes_fig}b.
\begin{figure}[t]
\vspace{-0.1in}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{figures/drop_coarse.pdf}
        \label{fig:1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{figures/drop_fine.pdf}
        \label{fig:2}
    \end{subfigure}
    \vspace{-0.1in}
    \caption{Effect of Semantic Diversity, measured by dropping an object class during training and measuring the resulting change in AR for that class: (a) dropping L1 classes and (b) dropping L4 classes}
    \label{fig:drop_classes_fig}
    \vspace{-0.2in}
\end{figure}
We notice that certain coarse classes (like ``clothing" and ``vehicle") experience a huge drop in performance. On the other hand, ``animal" and ``food" are less affected. This can be explained from the fact that, there are many toy-animal images within the coarse label ``toy", similarly ``containers" is a coarse class in \oiv which is often depicted with food in it. These classes can act as proxies for ``animal" and ``food" respectively. However, ``clothing" and ``vehicle" do not have good proxies. More interestingly, we  make a similar observation for finer classes at $L_4$ like airplanes and helicopters. This suggests that there is a smaller set of objects that have unique localization properties in \oiv.

\noindent {\textbf{Prototypical classes: }}
Some object classes are similar to others in terms of localization, while there are classes that are unique and need to be included in training. Motivated by this observation, we try to identify a small set of classes called ``prototypical" classes which are both necessary and sufficient to train a generalizable proposal model.

We use the \oivsource dataset as before with 432 leaf classes. We use the different approaches outlined in Sec.~\ref{sec:proto} to identify a subset of ``prototypical" classes. Note that among these methods, oracle visual clustering assumes availability of bounding boxes for all classes and serves as an upper bound on how to identify a really good prototypical set. Some sample clusters of classes obtained by this method are shown in Tab.~\ref{tab:visual_clusters}. The remaining methods make weaker assumptions and are more useful in practice. In addition to these methods,we also train models with a set of randomly chosen prototypical classes.

\vspace*{-0.3in}
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{Sample clusters obtained by oracle visual clustering for $P=50$. The most frequent class in each cluster chosen as a prototypical class is highlighted}
\label{tab:visual_clusters}
\begin{tabular}{lll}
\hline\noalign{\smallskip}
\scriptsize{\textbf{Woman}, Girl, Doll} & \scriptsize{\textbf{Wheel}, Tire, Bicyclewheel} & \scriptsize{\textbf{Lobster}, Scorpion, Centipede} \\
\hline
\scriptsize{\textbf{Glasses}, Goggles} & \scriptsize{\textbf{Jeans}, Shorts, Miniskirt} & \scriptsize{\textbf{Goose}, Ostrich, Turkey} \\
\hline
\scriptsize{\textbf{Book}, Shelf, Bookcase} & \scriptsize{\textbf{Musicalkeyboard}, Piano} & \scriptsize{\textbf{Swimmingpool}, Bathtub, Jacuzzi} \\
\hline
\scriptsize{\textbf{Man}, Boy, Shirt} & \scriptsize{\textbf{Apple}, Pomegranate, Peach} & \scriptsize{\textbf{Raven}, Woodpecker, Bluejay}\\
\hline
\end{tabular}
\end{center}
\vspace{-0.3in}
\end{table}
\setlength{\tabcolsep}{1.4pt}

We introduce two ways to measure \textit{sufficiency} and \textit{necessity}. From the $432$ classes, once we pick a subset of $P$ prototypical classes, we train a proposal model and evaluate the resulting model on the $50$ target classes in \oivtarget, to measure \textit{sufficiency} and \textit{necessity}.

\noindent{\textbf{Dataset construction for fair comparison}} We ensure that the total number of images as well as bounding box annotations are kept fixed when we construct datasets for different prototypical subsets. This is important to ensure that proposals trained with different subsets are comparable. Once we chose a set of $P$ prototypical classes, we uniformly sub-sample \oivsource images having any of these prototypical classes to get a subset of $920K$ images. And within each subset, we uniformly sub-sample the bounding boxes corresponding to the prototypical classes to retain $5.2M$ bounding boxes. We do not retain any bounding boxes outside the chosen prototypical classes.

\noindent{\textbf{Training with prototypical subsets}}
For a set of prototypical classes and the corresponding dataset, we train a \frcnn with those classes as labels. We combine the detections as described in Sec.~\ref{sec:model_choice} to obtain proposals.

\noindent{\textbf{Measuring sufficiency of prototypical classes}}
A subset of classes are sufficient, if a proposal model trained with them generalizes as well as a model trained with all classes. We follow this notion and evaluate the proposals obtained from the models trained with different prototypical subsets on \oivtarget and report the average recall (\AR{100}) in Fig.~\ref{fig:proto_properties}a. Similar trends are observed with \AR{1000} as well (shown in supplementary).

Looking at the proposals obtained from oracle visual clustering, training with less than 25\% of the classes (100) leads to only a drop of $4.8\%$ in \AR{100}, compared to training with images belonging to all object classes. This gap reduces to $0.4\%$ if we train with 50\% (200) of all the classes. This provides an empirical proof for the existence of a significantly smaller number of object classes that are sufficient to train a generalizable proposal model.

Next, we look at the prototypical classes obtained from a more practical approach: semantic clustering. We notice that the proposal model trained with these prototypical classes always outperform other approaches such as choosing a random set of classes or the most frequent set of classes. Further, the performance of this method is only lower by a margin of $3\%$ compared to oracle visual clustering for different value of $P$. Selecting most frequent set of classes as the prototypical subset performs slightly worse than semantic clustering. This shows that semantic clustering can serve as a good way to identify prototypical classes for large taxonomies when the semantic hierarchy is available for the dataset, else the most frequent subset is a weaker alternative.

\begin{figure}[t]
\vspace{-0.1in}
  \begin{minipage}[t]{0.99\textwidth}
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=1.0\textwidth]{figures/sampling_method_ar100.pdf}
        \label{fig:suff_1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/necessity.pdf}
    \label{fig:necessity}
    \end{subfigure}
    \vspace{-0.2in}
    \caption{(a) Average recall \AR{100} for proposals obtained from models trained with varying number of prototypical classes chosen by different methods. We show the average recall on the \oivtarget dataset with $50$ unseen classes. $P$ denotes the number of prototypical classes. Higher value indicates higher sufficiency. (b) The relative change in AR for target classes by dropping proposals corresponding to the most similar class in the prototypical subset. Higher value indicates lower redundancy in prototypical subset and higher necessity}
    \label{fig:proto_properties}
  \end{minipage}
\vspace{-0.25in}
\end{figure}

  
\noindent{\textbf{Measuring necessity of prototypical classes}}
A set of classes are considered necessary, if there is no redundancy among the classes in terms of localization properties. For a given class in the set, there should be no equivalent class which can provide similar bounding boxes. We measure this property for a prototypical subset by evaluating the corresponding proposal model on \oivtarget dataset using the following method. For every target class in \oivtarget, we measure the relative change in \AR{100} and \AR{1000} by removing proposals corresponding to the most similar class in the prototypical subset (similarity measured by Eq.~\ref{eq:max_ap}). The change in AR would be minimal if there is another class in the prototypical subset which can localize the target class. This measure, averaged over all target classes provides a good estimate of necessity. A high value symbolizes a high degree of necessity, while a low value corresponds to redundancy among the prototypical classes. We plot this for different number of prototypical classes for oracle visual clustering and semantic clustering in Fig.~\ref{fig:proto_properties}b.
We notice that at any given number of prototypical classes, the change in average recall is higher for oracle visual clustering compared to semantic clustering. This demonstrates that visual clustering leads to prototypical classes which are less redundant (and more necessary). As expected, we see the necessity drops, as we increase the number of prototypical classes for both methods. Again, this is expected since redundancy between classes increases with more number of classes. The relative change in \AR{1000} is also seen to be lower than \AR{100}, since when considering a larger number of proposals, we expect more redundancy among the proposals. Finally, for oracle visual clustering as we move from $200$ to $300$ classes, sufficiency changes by a small amount from $73.2$ to $75.9$ ( Fig.~\ref{fig:proto_properties}a), while the necessity drops steeply in Fig.~\ref{fig:proto_properties}b. This suggests that the ideal number of prototypical classes for \oiv could be around $200$.
\section{Conclusion}
We studied the ability of detection models trained on a set of seen classes to localize unseen classes. We showed that \frcnn can be used to obtain better proposals for unseen classes than \retina, and studied the effect of model choices on generalization of proposals, like class-agnostic bounding box regression and NMS threshold. We quantitatively measured the importance of visual diversity and showed that using a very fine-grained or very coarse label-space can both affect generalization, while a middle-ground approach is best suited. We introduced the idea of prototypical classes that are sufficient and necessary to obtain generalized proposals. We demonstrated different approaches to determine small prototypical subsets for a given dataset. We believe that our work is a step forward towards learning proposals that generalize to a large number of classes and scaling up detection in a more data-efficient way.


\clearpage
\bibliographystyle{splncs04}
\bibliography{egbib}
\end{document}


Peer review:
Prototype classes can be sufficient to localize in object detection
[Summary] In 2-3 sentences, describe the key ideas, experiments, and their significance.

The authors propose to learn object localizations using only prototype classes. They explore what defines prototype classes and experiment with many ablations and hyperparameters to their method.

[Strengths] What are the strengths of the paper? Clearly explain why these aspects of the paper are valuable.

Powerful idea; clear definitions; practical modeling choices for determining prototypical classes; extensive experimentation.

[Weaknesses] What are the weaknesses of the paper? Clearly explain why these aspects of the paper are weak.

Marginally related to visual inductive priors.

[Overall rating] Paper rating: Accept
Rating: 7: Good paper, accept
Confidence: 3: The reviewer is fairly confident that the evaluation is correct