Title: DCTD: DEEP CONDITIONAL TARGET DENSITIES FOR ACCURATE REGRESSION

Abstract: While deep learning-based classification is generally addressed using standardized approaches, a wide variety of techniques are employed for regression. In computer vision, one particularly popular such technique is that of confidencebased regression, which entails predicting a confidence value for each input-target pair (x, y). While this approach has demonstrated impressive results, it requires important task-dependent design choices, and the predicted confidences often lack a natural probabilistic meaning. We address these issues by proposing Deep Conditional Target Densities (DCTD), a novel and general regression method with a clear probabilistic interpretation. DCTD models the conditional target density p(y|x) by using a neural network to directly predict the un-normalized density from (x, y). This model of p(y|x) is trained by minimizing the associated negative log-likelihood, approximated using Monte Carlo sampling. We perform comprehensive experiments on four computer vision regression tasks. Our approach outperforms direct regression, as well as other probabilistic and confidence-based methods. Notably, our regression model achieves a 1.9% AP improvement over Faster-RCNN for object detection on the COCO dataset, and sets a new state-ofthe-art on visual tracking when applied for bounding box regression.

1. INTRODUCTION:

Supervised regression entails learning a model capable of predicting a continuous target value y from an input x, given a set of paired training examples. It is a fundamental machine learning problem with many important applications within computer vision and other domains. Common regression tasks within computer vision include object detection (Jiang et al., 2018; Zhou et al., 2019) , headand body-pose estimation (Xiao et al., 2018; Yang et al., 2019) , age estimation (Rothe et al., 2016; Pan et al., 2018) , visual tracking (Danelljan et al., 2019) and medical image registration (Niethammer et al., 2011; Chou et al., 2013) , just to mention a few. While all of these tasks benefit from accurate regression of the target values, high accuracy can even be safety-critical in e.g. automotive and medical applications. Today, such regression problems are commonly tackled using Deep Neural Networks (DNNs), due to their ability to learn powerful feature representations from data.

While classification is generally addressed using standardized losses and output representations, a wide variety of different techniques are employed for regression. The most conventional strategy is to train a DNN to directly predict a target y given an input x (Lathuilière et al., 2019) . In such direct regression approaches, the model parameters of the DNN are learned by minimizing a loss function, for example the L 2 or L 1 loss, penalizing the discrepancy between the predicted and ground truth target values. From a probabilistic perspective, this approach corresponds to creating a simple parametric model of the conditional target density p(y|x), and minimizing the associated negative log-likelihood. The L 2 loss, for example, corresponds to a fixed-variance Gaussian model. More recent work (Kendall & Gal, 2017; Lakshminarayanan et al., 2017) has also explored learning more expressive models of p(y|x), by letting a DNN instead output the full set of parameters of a certain family of probability distributions. These probabilistic regression approaches however still restrict the parametric model to fairly simple distributions in most cases, such as Gaussian (Chua et al., 2018) or Laplace (Gast & Roth, 2018; Ilg et al., 2018) , limiting the expressiveness of the learned conditional target density. While these methods benefit from a clear probabilistic interpretation, they may thus not fully exploit the predictive power of the DNN. i=1 is generated by the ground truth conditional target density p(y|x). DCTD models p(y|x) by predicting the unnormalized density from (x, y). In contrast to the Gaussian model p(y|x, θ) = N (y; µ θ (x), σ 2 θ (x)), DCTD can learn complex target densities directly from data, including multi-modal ones.

The quest for improved regression accuracy has also led to the development of more specialized methods, designed for a specific set of tasks. In computer vision, one particularly popular approach is that of confidence-based regression. Here, a DNN instead predicts a scalar confidence value for input-target pairs (x, y). The confidence can then be maximized w.r.t. y to obtain a target prediction for a given input x. The approach is commonly employed for image coordinate regression tasks within e.g. human pose estimation (Cao et al., 2017; Xiao et al., 2018) and object detection (Law & Deng, 2018; Zhou et al., 2019) , where a 2D heatmap over image pixel coordinates y is predicted. Recently, the approach was also applied to the problem of bounding box regression by Jiang et al. (2018) . Their proposed method, IoU-Net, obtained state-of-the-art accuracy on object detection, and was later also successfully applied to the task of visual tracking (Danelljan et al., 2019) . The training of such confidence-based regression methods does however entail generating additional pseudo ground truth labels, by for example employing a Gaussian kernel (Xiao et al., 2018) , and selecting an appropriate loss function. This both requires numerous design choices to be made, and limits the general applicability of the methods. Moreover, confidence-based regression methods do not allow for a natural probabilistic interpretation in terms of the conditional target density p(y|x). In this work, we therefore set out to develop a method combining the general applicability and clear interpretation of probabilistic regression with the predictive power of confidence-based approaches.

Contributions:

We propose Deep Conditional Target Densities (DCTD), a novel and general regression method with a clear probabilistic interpretation. DCTD predicts the un-normalized conditional target density p(y|x) from the input-target pair (x, y). It is trained by directly minimizing the associated negative log-likelihood by exploiting Monte Carlo approximations. At test time, targets are predicted by maximizing the conditional target density p(y|x) through gradient-based refinement. Compared to confidence-based approaches, our DCTD requires no pseudo-labels and benefits from a direct probabilistic interpretation. Unlike existing probabilistic models, our approach can learn highly flexible target densities directly from data, as visualized in Figure 1 .

We evaluate the proposed method on four diverse computer vision regression tasks: object detection, age estimation, head-pose estimation and visual tracking. Our DCTD method is found to outperform both the direct regression baselines, and popular probabilistic and confidence-based alternatives. Notably, our method achieves a 1.9% AP improvement over the FPN Faster-RCNN (Lin et al., 2017) baseline on the COCO dataset (Lin et al., 2014) when applied for object detection. It also sets a new state-of-the-art on standard benchmarks (Müller et al., 2018; 2016) when applied for bounding box regression in the ATOM visual tracking algorithm (Danelljan et al., 2019) .

2. BACKGROUND & RELATED WORK:

In supervised regression, the task is to learn to predict a target value y ∈ Y from a corresponding input x ∈ X , given a training set of i.i.d. input-target examples, D = {(x i , y i )} N i=1 , (x i , y i ) ∼ p(x, y). As opposed to classification, the target space Y is a continuous set, e.g. Y = R K . In computer vision, the input space X often corresponds to the space of images, whereas the output space Y depends on the task at hand. Common examples include Y = R + in age estimation (Rothe et al., 2016) , Y = R 2 in image keypoint estimation (Xiao et al., 2018) , and Y = R 4 in object bounding box regression (Jiang et al., 2018) .

Direct regression Over the last decade, DNNs have been shown to excel at a variety of regression problems. Here, a DNN is viewed as a function f θ : X → U, parameterized by a set of learnable weights θ ∈ R P . The most conventional regression approach is to train a DNN to directly predict the targets, y = f θ (x ), called direct regression. The model parameters θ are learned by minimizing a loss (f θ (x i ), y i ) that penalizes the discrepancy between the prediction f θ (x i ) and the ground truth target value y i on training samples (x i , y i ). The most common choices are the L 2 loss, (y * , y) = y * -y 2 2 , the L 1 loss, (y * , y) = y * -y 1 , and their close relatives (Huber, 1964; Lathuilière et al., 2019) . From a probabilistic perspective, the choice of loss corresponds to minimizing the negative log-likelihoodlog p(y|x, θ) of a specific model p(y|x, θ) of the conditional target density. For example, the L 2 loss is derived from a fixed-variance Gaussian model, p(y|x, θ) = N (y; f θ (x), σ 2 ).

Probabilistic regression More recent work (Kendall & Gal, 2017; Lakshminarayanan et al., 2017; Chua et al., 2018) has explicitly taken advantage of this probabilistic perspective to achieve more flexible parametric models p(y|x, θ) = p(y|φ θ (x)), by letting the DNN output the parameters φ of a family of probability distributions p(y|φ). For example, a general 1D Gaussian model can be realized as p(y|φ θ (x)) = N y; µ θ (x), σ 2 θ (x) , where the DNN outputs the mean and log-variance as

f θ (x) = φ θ (x) = [ µ θ (x) log σ 2 θ (x) ] T ∈ R 2 .

The model parameters θ are learned by minimizing the negative log-likelihood -N i=1 log p(y i |x i , θ) over the training set D. At test time, a target estimate y is obtained by first predicting the density parameter values φ θ (x ) and then, for instance, taking the expected value of p(y|φ θ (x)). Previous work has applied Gaussian and Laplace models on computer vision tasks such as object detection (Feng et al., 2019; He et al., 2019) and optical flow estimation (Gast & Roth, 2018; Ilg et al., 2018) . The aim of such probabilistic approaches is often not only to achieve accurate predictions, but also to provide an estimate of the aleatoric uncertainty (Kendall & Gal, 2017) , which models noise and ambiguities inherent in the data itself. Our method also entails predicting a conditional target density p(y|x, θ) and minimizing the associated negative log-likelihood. However, our model p(y|x, θ) is not restricted to the functional form of any specific probability density (e.g. Gaussian or Laplace), but is instead directly defined by the DNN architecture itself, allowing for more expressive target densities.

Confidence-based regression Another category of approaches reformulate the regression problem as y = arg max y f θ (x, y), where f θ (x, y) ∈ R is a scalar confidence value predicted by the DNN. The idea is thus to predict a quantity f θ (x, y), depending on both input x and target y, that can be maximized over y to obtain the final prediction y . This maximization-based formulation is inherent in Structural SVMs (Tsochantaridis et al., 2005) , but has also been adopted for DNNs. We term this family of approaches confidence-based regression. Different from direct regression, the predicted confidence f θ (x, y) can encapsulate multiple hypotheses and other ambiguities. Confidence-based regression has been shown particularly suitable for image-coordinate regression tasks, such as hand keypoint localization (Simon et al., 2017) and body-part detection (Wei et al., 2016; Pishchulin et al., 2016; Xiao et al., 2018) . In these cases, a CNN is trained to output a 2D heatmap over the image pixel coordinates y, thus taking full advantage of the translational invariance of the problem. A similar approach has also been employed to locate the two defining corners (Law & Deng, 2018) or four extreme points (Zhou et al., 2019) of bounding boxes in object detection. In computer vision, confidence prediction has also been successfully employed for tasks other than pure image-coordinate regression. Jiang et al. (2018) proposed the IoU-Net for bounding box regression in object detection, where a bounding-box y ∈ R 4 and image x are both inputs to the DNN to predict a confidence f θ (x, y). It employs a pooling-based architecture that is differentiable w.r.t. the bounding box y, allowing gradient-based maximization to obtain the final estimate y = arg max y f θ (x, y). IoU-Net was later also applied to visual tracking (Danelljan et al., 2019) .

In general, confidence-based approaches are trained using a set of generated pseudo label confidences u x,y and by employing a loss (f θ (x, y), u x,y ). One strategy (Pishchulin et al., 2016; Law & Deng, 2018) is to treat the confidence prediction as a binary classification problem, where u x,y represents either the class, u x,y ∈ {0, 1}, or its probability, u x,y ∈ [0, 1], and employ cross-entropy based losses . The other approach is to treat the confidence prediction as a direct regression problem itself by applying standard regression losses, such as L 2 (Simon et al., 2017; Danelljan et al., 2019; Wei et al., 2016) or the Huber loss (Jiang et al., 2018) . In these cases, the pseudo label confidences u x,y can be constructed using a similarity measure S in the target value space, u xi,y = S(y, y i ), for example defined as the Intersection over Union (IoU) between two bounding boxes (Jiang et al., 2018) or simply by a Gaussian kernel (Wei et al., 2016; Xiao et al., 2018) . While these approaches have demonstrated impressive results, existing confidence-based approaches require important design choices. In particular, the strategy for constructing the pseudo labels u x,y and the choice of loss (f θ (x, y), u x,y ) are often crucial for performance and highly task-dependent. Moreover, the predicted confidence f θ (x, y) can be difficult to interpret, since it has no natural connection to the conditional target density p(y|x). In contrast, our approach is directly trained to predict p(y|x) itself, and does not require generation of pseudo label confidences or choosing a specific loss.

Regression-by-classification A regression problem can also be treated as a classification problem by first discretizing the target space Y into a finite set of C classes. Standard techniques from classification, such as softmax and the cross-entropy loss, can then be employed. Rothe et al. (2016) additionally computed the softmax expected value to obtain a more fine-grained prediction, and applied their method to the task of age estimation. Ruiz et al. (2018) applied the same method to head-pose estimation, but also added an L 2 loss term for the softmax expected value during training. Again for age estimation, Pan et al. ( 2018) then added an additional loss term penalizing the softmax variance. A hierarchical classification approach has also been proposed for both age estimation (Yang et al., 2018) and head-pose estimation (Yang et al., 2019) . The discretization of the target space Y often complicates exploiting its inherent neighborhood structure. This has been addressed by exploring ordinal regression methods for 1D problems (Cao et al., 2019; Diaz & Marathe, 2019) . Finally, classification into coarse discrete bins can be combined with direct regression, a technique often utilized in 2D (Redmon et al., 2016; Liu et al., 2016) and 3D (Shi et al., 2019; Qi et al., 2018) object detection. While our approach can be seen as a generalization of the softmax model for classification to the continuous target space Y, it does not suffer from the aforementioned drawbacks of regression-by-classification. On the contrary, our model naturally allows the network to exploit the full structure of the continuous target space Y.

3. REGRESSION USING DEEP CONDITIONAL TARGET DENSITIES:

In this work, we take the probabilistic view of regression by creating a model p(y|x, θ) of the conditional target density p(y|x). Instead of defining p(y|x, θ) by letting a DNN predict the parameters of a certain family of probability distributions (e.g. Gaussian or Laplace), we construct a versatile model that can better leverage the predictive power of DNNs. To that end, we take inspiration from confidence-based regression approaches and let a DNN predict a scalar value for any input-target pair (x, y). Unlike confidence-based methods however, this prediction has a clear probabilistic interpretation. Specifically, we view a DNN as a function f θ : X × Y → R, parameterized by θ ∈ R P , that maps an input-target pair (x, y) ∈ X × Y to a scalar value f θ (x, y) ∈ R. Then, we define the Deep Conditional Target Density (DCTD) according to,

EQUATION

where Z(x, θ) is the normalizing constant. We train our DCTD model p(y|x, θ) by minimizing the negative log-likelihood -log p({y

i } i |{x i } i , θ)= N i=1 -log p(y i |x i , θ)

, where each term is given by,

EQUATION

The training thus requires the evaluation of the normalizing constant Z(x, θ), involving the integral in equation 2. This can be achieved using effective finite approximations. In some tasks, such as image-coordinate regression, this is naturally performed by a grid approximation, utilizing the dense prediction already employed in many such methods. In this work, we however investigate a more generally applicable technique, namely Monte Carlo approximations. This procedure, when employed for training the network, is detailed in Section 3.1.

At test time, given an input x , our model in equation 1 allows evaluating the conditional target density p(y|x , θ) for any y by first approximating the constant Z(x , θ) and then predicting the scalar f θ (x , y) using the DNN. This enables the computation of, for instance, means and variances of the target value y. In this work, we focus on finding the most likely prediction, y = arg max y p(y|x , θ) = arg max y f θ (x , y), which does not require the evaluation of Z(x, θ) during inference. Thanks to the auto-differentiation capabilities of modern deep learning frameworks, we can apply gradient-based techniques to find the final prediction by simply maximizing the network output f θ (x , y) w.r.t. y. We elaborate on this procedure for prediction in Section 3.2.

3.1. TRAINING:

Our model p(y|x, θ) = e f θ (x,y) /Z(x, θ) of the conditional target density is trained by minimizing the negative log-likelihood N i=1log p(y i |x i , θ). To evaluate the integral in equation 2, we employ a Monte Carlo approximation. Specifically, each termlog p(y i |x i , θ) is approximated by sampling values y (k) from a proposal distribution q(y|y i ) that depends on the ground truth target y i ,

EQUATION

The final loss J(θ) is then obtained by averaging over all training samples (x i , y i ) in the mini-batch,

EQUATION

where {y (i,m) } M m=1 are M samples drawn from q(y|y i ). Qualitatively, minimizing J(θ) encourages the DNN to output large values f θ (x i , y i ) for the ground truth target y i , while minimizing the predicted value f θ (x i , y) at all other targets y. In ambiguous or uncertain cases, the DNN can output small values everywhere or large values at multiple hypotheses, but at a cost of a higher loss.

As seen in equation 4, the DNN f θ is applied both to the input-target pair (x i , y i ), and all inputsample pairs {(x i , y (i,m) )} M m=1 during training. While this can seem inefficient, most applications in computer vision employ network architectures that first extract a deep feature representation for the input x i . The DNN can thus be designed to combine this input feature with the target y at a late stage, meaning that the input feature extraction process, which becomes the main computational bottleneck, needs to be performed only once for each x i . In practice, we found our training strategy to not add any significant computational overhead compared to the baselines.

Compared to confidence-based regression, a significant advantage of our approach is that there is no need for generating task-dependent pseudo label confidences or choosing between different losses. The only design choice of our method is the proposal distribution q(y|y i ). Note however that since the loss (equation 4) explicitly adapts to q(y|y i ), this choice has no effect on the overall behaviour of the loss, only on the quality of the sampled approximation. We found a simple mixture of a few equally weighted Gaussian components, all centered at the target label y i , to consistently perform well in our experiments. Specifically we set,

EQUATION

where the variances σ 2 l are hyperparameters selected based on a validation set for each experiment.

3.2. PREDICTION:

Given an input x at test time, the trained DNN f θ can be used to evaluate the full target density p(y|x , θ) = e f θ (x ,y) /Z(x , θ), by employing the aforementioned techniques to approximate Z(x , θ). In many applications, the most likely prediction y = arg max y p(y|x , θ) is however the single desired output. For DCTD, this is obtained by directly maximizing the DNN output, y = arg max y f θ (x , y), thus not requiring Z(x , θ) to be evaluated. By designing the DNN f θ to be differentiable w.r.t. the target y, the gradient ∇ y f θ (x , y) can be efficiently evaluated using the auto-differentiation tools implemented in modern deep learning frameworks. We can therefore perform gradient ascent to find a local maximum of f θ (x , y). The gradient ascent refinement is performed either on a single initial estimate ŷ, or on a set of random initializations {ŷ k } K k=1 to obtain a final accurate prediction y . As noted in Section 3.1, this prediction procedure can be made highly efficient in practice by extracting the deep feature representation for x only once. Backpropagation is then performed only through a few final layers of the DNN f θ in order to evaluate the gradient ∇ y f θ (x , y). Moreover, the gradient computation for a set of target candidates {ŷ k } K k=1 can be parallelized on the GPU by simple batching, requiring no significant overhead. Please refer to Appendix B for a detailed algorithm of this prediction procedure. 

4. EXPERIMENTS:

We perform comprehensive experiments on four different computer vision tasks. Our DCTD method is compared both to baseline regression methods and to state-of-the-art models. All experiments are implemented in PyTorch (Paszke et al., 2017) . Code will be made available upon publication.

4.1. OBJECT DETECTION:

We first perform experiments for visual object detection, the problem of estimating a bounding box for each object in the image from a set of given classes. Specifically, we compare our regression method to other techniques for the task of bounding-box regression, by integrating them into an existing object detection pipeline. To this end, we use the Faster-RCNN (Ren et al., 2015) framework, which serves as a popular baseline in the object detection field due to its strong state-of-the-art performance. It uses one network head for classification and the second for regressing the bounding box using the direct method. We also compare our approach to the confidence-based IoU-Net (Jiang et al., 2018) . It extends Faster-RCNN with an additional branch that predicts the IoU overlap between a target box y and the ground truth. The IoU prediction branch uses differentiable region pooling (Jiang et al., 2018) , allowing the initial bounding box predicted by the Faster-RCNN to be refined using gradient-based maximization of the predicted IoU confidence.

For our approach, we employ an identical architecture as used in IoU-Net for a fair comparison. Instead of training the network to output the IoU, we predict the exponent f θ (x, y) in equation 1, trained by minimizing the negative log-likelihood (NLL) in equation 4. We parametrize the bounding box as y = (c x /w 0 , c y /h 0 , log w, log h) ∈ R 4 , where (c x , c y ) and (w, h) denote the center coordinate and size respectively. The reference size (w 0 , h 0 ) is set to that of the ground truth during training and the initial box during inference. For the proposal distribution (equation 5) we employ L = 3 isotropic Gaussians with standard deviation σ l = 0.05 • 2 l . In addition to the standard IoU-Net, we compare with a version (denoted IoU-Net † ) employing the same proposal distribution and inference settings as in our approach. For both our method and IoU-Net † , we set the refinement step-length using grid search on a separate validation set. We also compare with a Gaussian and a Laplace probabilistic model for bounding box regression by modifying the Faster-RCNN regression head to predict both the mean and log-variance of the distribution, and adopting the NLL loss.

Our experiments are performed on the large-scale COCO benchmark (Lin et al., 2014) The age y is processed by four fully-connected layers, generating g y ∈ R 128 . The two feature vectors are then concatenated and processed by two fully-connected layers, outputting f θ (x, y) ∈ R. We apply our DCTD to refine the age predicted by baseline models, using the gradient ascent maximization of f θ (x, y) (Section 3.2). All baseline DNN models employ a similar architecture, including an identical ResNet50 for feature extraction and the same number of fully-connected layers to output either the age y ∈ R (Direct), mean and variance parameters for Gaussian and Laplace distributions, or to output logits for C discretized classes (Softmax). The results are found in Table 2 . We observe that age refinement provided by our DCTD method consistently improves the accuracy of the predictions generated by the baseline methods. Further details are provided in Appendix D.

4.3. HEAD-POSE ESTIMATION:

In head-pose estimation, we are given an image x ∈ R h×w×3 of a person, and are tasked with predicting the orientation y ∈ R 3 of his/her head, where y is the Yaw, Pitch and Roll angles. We utilize the BIWI (Fanelli et al., 2013) dataset, specifically the processed dataset provided by Yang et al. (2019) , in which the images have been cropped to faces detected using MTCNN (Zhang et al., 2016) . We also employ protocol 2 as defined by Yang et al. (2019) , with 10 613 images for training and 5 065 images for testing. Additionally, we use 1 010 training images for validation. The methods are evaluated in terms of the average MAE for Yaw, Pitch and Roll. The network architecture of the DNN f θ defining our DCTD takes the image x ∈ R h×w×3 and orientation y ∈ R 3 as inputs, but is otherwise identical to the age estimation case (Section 4.2). Our DCTD model is again evaluated by applying the optimization-based refinement to the predicted orientation y ∈ R 3 outputted by a number of baseline models. We use the same baselines as for age estimation, and apart from minor changes required to increase the output dimension from 1 to 3, identical network architectures are also used. The results are found in Table 3 , and also in this case we observe that refinement using DCTD consistently improves upon the baselines. Further details are provided in Appendix E.

4.4. VISUAL TRACKING:

Lastly, we evaluate our approach on the problem of generic visual object tracking. The task is to estimate the bounding box of a target object in every frame of a video. The target object is defined by a given box in the first video frame. We employ the recently introduced ATOM (Danelljan et al., 2019) tracker as our baseline. Given the first-frame annotation, ATOM trains a classifier to first roughly localize the target in a new frame. The target bounding box is then determined using an IoU-Net based module, which is also conditioned on the first-frame target appearance using a modulationbased architecture. We train our network to predict the conditional target density through f θ (x, y) in equation 1, using a network architecture identical to the baseline ATOM tracker. In particular, we employ the same bounding box parameterization as for object detection (Section 4.1) and sample M = 128 boxes during training from a proposal distribution (equation 5) generated by L = 2 Gaussians with standard deviations of 0.05 and 0.5. During tracking, we follow the same procedure as in ATOM, sampling 10 boxes in each frame followed by gradient ascent to refine the estimate generated by the classification module.

We demonstrate results on two standard tracking benchmarks: TrackingNet (Müller et al., 2018) and UAV123 (Müller et al., 2016) . TrackingNet contains challenging videos sampled from YouTube, Under review as a conference paper at ICLR 2020 (Zhu et al., 2018; Li et al., 2019) . We therefore only compare with the ATOM baseline and include other recent state-of-the-art methods in the comparison. As in section 4.1, we compare with a version of the IoU-Net based ATOM (denoted ATOM † ) employing the same training and inference settings as our final approach. The results are shown in Table 4 . Our approach achieves a significant 1.6% and 1.5% absolute improvement over ATOM on the overall metric on TrackingNet and UAV123 respectively. Note that the improvements are most prominent for high-accuracy boxes, indicated by the OP 0.75 score. Moreover, our approach outperforms the recent SiamRPN++ (Li et al., 2019) , which employs anchor-based bounding box regression (Ren et al., 2015; Redmon & Farhadi, 2016 ) and a much deeper backbone network (ResNet50) compared to ours (ResNet18).

Figure 2 visualizes the conditional target density generated by our approach for tracking.

5. CONCLUSION:

We proposed Deep Conditional Target Densities (DCTD), a novel and generally applicable regression method with a clear probabilistic interpretation. It directly models the conditional target density p(y|x) by predicting the un-normalized density through a DNN f θ (x, y), taking the input-target pair (x, y) as input. The model is trained by minimizing the associated negative log-likelihood, employing a Monte Carlo approximation of the normalizing constant. At test time, targets are predicted by maximizing the DNN output f θ (x, y) w.r.t. y via gradient-based refinement. Experiments performed on four diverse computer vision applications demonstrate the high accuracy and wide applicability of our method. However, this work constitutes an initial investigation of DCTD. Future directions include exploring better architectural designs, studying other regression applications, and investigating DCTD's potential for aleatoric uncertainty estimation. Feat-Box) consisting of two fully-connected (FC) layers. The output feature vector is then passed through two parallel FC layers, one which predicts the class label (denoted FC-Cls), and another which regresses the offsets between the proposal and the ground truth box (denoted FC-BB). We use the PyTorch implementation for Faster-RCNN from Massa & Girshick (2018) . Note that we use the RoiAlign (He et al., 2017) layer instead of RoiPool in our experiments as it has been shown to achieve better performance (He et al., 2017) .

For the Gaussian and Laplace probabilistic models (Gaussian and Laplace in Table 1 ), we replace the FC-BBReg layer in Faster-RCNN with parallel two FC layers, denoted FC-BBMean and FC-BBVar, which predict the mean and the log-variance of the distribution modeling the offset between the proposal and the ground truth box for each coordinate.

For our confidence-based IoU-Net (Jiang et al., 2018 ) models (IoU-Net and IoU-Net † in Table 1 ), we use the same network architecture as employed in the original paper. That is, we add an additional branch to predict the IoU overlap between the proposal box and the ground truth. This branch uses the PrRoiPool (Jiang et al., 2018) layer to pool the features from the proposal regions. The pooled features are passed through a feature extractor (denoted Feat-Conf) consisting of two FC layers.

The output feature vector is passed through another FC layer, FC-Conf, which predicts the IoU. We use an identical architecture for our approach, but train it to output f θ (x, y) in equation 1 instead. Illustrations of the architectures are found in Figure 3 .

C.2 TRAINING:

We use the pre-trained weights for Faster-RCNN from Massa & Girshick (2018) . Note that the bounding box regression in Faster-RCNN is trained using a direct method, with an Huber loss (Huber, 1964) . We trained the other networks in Table 1 (Gaussian, Laplace, IoU-Net, IoU-Net † and DCTD) on the MS-COCO (Lin et al., 2014) training split (2017 train) using stochastic gradient descent (SGD) with a batch size of 16 for 60k iterations. The base learning rate lr base is reduced by a factor of 10 after 40k and 50k iterations, for all the networks. We also warm up the training by linearly increasing the learning rate from 1 3 lr base to lr base during the first 500 iterations. We use a weight decay of 0.0001 and a momentum of 0.9. For all the networks, we only trained the newly added layers, while keeping the backbone and the region proposal network fixed.

For the Gaussian and Laplace models, we only train the final predictors (FC-BBMean and FC-BBVar), while keeping the class predictor (FC-Cls) and the box feature extractor (Feat-Box) fixed. We also tried fine-tuning the FC-Cls and Feat-Box weights, with different learning rate settings, but obtained worse performance on the validation set. The weights for both FC-BBMean and FC-BBVar were initialized with zero mean Gaussian with standard deviation of 0.001. Both Gaussian and Laplace models were trained with a base learning rate lr base = 0.005 by minimizing the negative log-likelihood.

For the IoU-Net, IoU-Net † and our DCTD model, we only trained the newly added confidence branch. We found it beneficial to initialize the feature extractor block (Feat-Conf) with the corresponding weights from Faster-RCNN, i.e. the Feat-Box block. The weights for the predictor FC-Conf were initialized with zero mean Gaussian with standard deviation of 0.001. As mentioned in Algorithm 2 Prediction via optimization-based refinement (variation) Input: x , ŷ, T , λ, Ω 1 , Ω 2 .

1: y ← ŷ.

2: for t = 1, . . . , T do 3:

PrevValue ← f θ (x , y).

4::

y ← y + λ∇ y f θ (x , y).

5::

NewValue ← f θ (x , y).

6::

if |PrevValue -NewValue| < Ω 1 or (NewValue -PrevValue) < Ω 2 then 7:

Return y. 8: Return y.

D.4 BASELINES:

All baselines are trained for 75 epochs with a batch size of 32, using the ADAM optimizer with weight decay of 0.001. Identical data augmentation and parameter initialization as for DCTD is used.

Direct The DNN architecture of Direct first extracts ResNet50 features g x ∈ R 2048 from the input image x. The feature vector g x is then processed by two fully-connected layers (2048 → 2048, 2048 → 1), outputting the prediction ŷ ∈ R. It is trained by minimizing either the Huber or L 2 loss.

Gaussian The Gaussian model is defined using a DNN f θ (x) according to,

p(y|x, θ) = N y; µ θ (x), σ 2 θ (x) , f θ (x) = [ µ θ (x) log σ 2 θ (x) ] T ∈ R 2 .

It is trained by minimizing the negative log-likelihood, corresponding to the loss,

J(θ) = 1 N N i=1 (y i -µ θ (x i )) 2 σ 2 θ (x i ) + log σ 2 θ (x i ).

The DNN architecture of f θ (x) first extracts ResNet50 features g x ∈ R 2048 from the input image x. The feature vector g x is then processed by two heads of two fully-connected layers (2048 → 2048, 2048 → 1) to output µ θ (x) and log σ 2 θ (x). The mean µ θ (x) is taken as the prediction ŷ.

Laplace The Laplace model is defined using a DNN f θ (x) according to,

p(y|x, θ) = 1 2β θ (x) exp - |y -µ θ (x)| β θ (x) , f θ (x) = [ µ θ (x) log β θ (x) ] T ∈ R 2 .

It is trained by minimizing the negative log-likelihood, corresponding to the loss,

J(θ) = 1 N N i=1 |y i -µ θ (x i )| β θ (x i ) + log β θ (x i ).

The DNN architecture of f θ (x) first extracts ResNet50 features g x ∈ R 2048 from the input image x. The feature vector g x is then processed by two heads of two fully-connected layers (2048 → 2048, 2048 → 1) to output µ θ (x) and log β θ (x). The mean µ θ (x) is taken as the prediction ŷ. We adopt the ATOM (Danelljan et al., 2019) tracker as our baseline, and use the PyTorch implementation and pre-trained weights from Danelljan & Bhat (2019) . ATOM trains an IoU-Net based module to predict the IoU overlap between a candidate box and the ground truth, conditioned on the first-frame target appearance. The IoU predictor is trained by generating 16 candidates for each ground truth box. The candidates are generated by adding a Gaussian noise for each ground truth box coordinate, while ensuring a minimum IoU overlap of 0.1 between the candidate box and the ground truth. The network is trained by minimizing the squared error (L 2 loss) between the predicted and ground truth IoU.

Our DCTD model is instead trained by sampling 128 candidate boxes from a proposal distribution (equation 5) generated by L = 2 Gaussians with standard deviations of 0.05 and 0.5, and minimizing the negative log likelihood of the training data. We use the training splits of TrackingNet (Müller et al., 2018) , LaSOT (Fan et al., 2019) , GOT10k (Huang et al., 2018) , and MS-COCO datasets for our training. Our network is trained for 50 epochs, using the ADAM optimizer with a base learning rate of 0.001 which is reduced by a factor of 5 after every 15 epochs. The rest of the training parameters are exactly the same is in ATOM. The ATOM † model is trained by using the exact same proposal distribution, datasets and settings. It only differs by the loss, which is the same squared error between the predicted and ground truth IoU as in the original ATOM.

F.2 INFERENCE:

During tracking, the ATOM tracker first apply the classification head network, which is trained online, to coarsely localize the target object. 10 random boxes are then sampled around this prediction, to be refined by the IoU prediction network. We only alter the final bounding box refinement step of the 10 given random initial boxes, and preserve all other settings as in the original ATOM tracker. The original version performs T = 5 gradient ascent iterations with a step length of λ = 1.0. For our DCTD-based and the ATOM † version, we use T = 10 iterations, employing the bounding box parameterization described in Section 4.1. For our approach we set the step length to λ = 2 • 10 -4 for position and λ = 10 -3 for size dimensions. For ATOM † we use λ = 10 -2 for position and λ = 5 • 10 -2 for size dimensions. These parameters were set on the separate validation set. For simplicity, we adopt the vanilla gradient ascent strategy employed in ATOM, for the two other methods as well. That is, we have no decay (η = 1) and do not perform checks whether the confidence score is increasing in each iteration.

:



Figures and Tables:

Figure 1: An illustrative 1D regression problem. The training data {(x i , y i )} 2000i=1 is generated by the ground truth conditional target density p(y|x). DCTD models p(y|x) by predicting the unnormalized density from (x, y). In contrast to the Gaussian model p(y|x, θ) = N (y; µ θ (x), σ 2 θ (x)), DCTD can learn complex target densities directly from data, including multi-modal ones.

Figure 2: Visualization of the conditional target density p(y|x, θ) ∼ e f θ (x,y) predicted by our network for the task of bounding box estimation in visual tracking. Since the target space y ∈ R 4 is 4-dimensional, we visualize the density for different locations of the top-right corner as a heatmap, while the bottom-left is kept fixed at the tracker output (blue box). Our network predicts flexible densities, expressing meaningful uncertainties in challenging cases.

Figure3: Network architectures for the different detection networks used in our experiments in Section 4.1. The backbone feature extractor (ResNet50-FPN), and the region proposal network (RPN) is not shown for clarity. We do not train the blocks in blue color, using the pre-trained Faster-RCNN weights fromMassa & Girshick (2018) instead. The blocks in red are initialized with the pre-trained Faster-RCNN weights and fine-tuned. The blocks in green on the other hand are trained from scratch.

Results for the object detection task on the test-dev split of the COCO dataset. Our approach significantly outperforms the baseline Faster-RCNN and the confidence-based IoU-Net.

Table content: <table><tr><td>Formulation</td><td>Direct</td><td colspan="5">Gaussian Laplace Confidence Confidence DCTD</td></tr><tr><td>Approach</td><td>Faster-RCNN</td><td/><td/><td>IoU-Net</td><td>IoU-Net  †</td><td>Ours</td></tr><tr><td>AP (%)</td><td>37.2</td><td>36.7</td><td>37.1</td><td>38.3</td><td>38.2</td><td>39.1</td></tr><tr><td>AP 50 (%)</td><td>59.2</td><td>58.7</td><td>59.1</td><td>58.3</td><td>58.4</td><td>58.5</td></tr><tr><td>AP 75 (%)</td><td>40.3</td><td>39.6</td><td>40.2</td><td>41.4</td><td>41.4</td><td>41.8</td></tr></table>

. As per the official guideline, we use the 2017 train split (≈ 118 000 images) for training and the 2017 val split (≈ 5 000 images) as the validation set for setting the hyperparameters. The results are reported on the 2017 test-dev split (≈ 20 000 images), in terms of the standard COCO metrics AP (mean Average Precision AP T over 10 IoU thresholds T ∈ [0.5, 0.95]), AP 50 , and AP 75 . We initialize all networks in our comparison with the pre-trained Faster-RCNN weights, using the ResNet50-FPN(Lin et al., 2017) backbone and re-train only the newly added layers for a fair comparison. Further details are provided in Appendix C. The results are shown in Table1. Our DCTD approach obtains the best results, outperforming both Faster-RCNN and IoU-Net by 1.9% and 0.8% in AP, respectively.

Table content: <table><tr><td>4.2 AGE</td></tr></table>

Results for the age estimation experiments. Refinement using DCTD consistently improves MAE (lower is better) for the age predictions outputted by a number of baselines.

Table content: <table><tr><td>+DCTD Cao et al. (2019)</td><td>Direct</td><td>Gaussian</td><td>Laplace</td><td colspan="2">Softmax (CE, L 2 ) Softmax (CE, L 2 , Var)</td></tr><tr><td>5.47 ± 0.01 -</td><td colspan="3">4.81 ± 0.02 4.79 ± 0.06 4.85 ± 0.04 4.65 ± 0.02 4.66 ± 0.04 4.81 ± 0.04</td><td>4.78 ± 0.05 4.65 ± 0.04</td><td>4.81 ± 0.03 4.69 ± 0.03</td></tr></table>

Results for the head-pose estimation experiments. Refinement using DCTD consistently improves the average MAE for Yaw, Pitch and Roll for the predicted pose outputted by our baselines.He et al., 2016) features g x ∈ R 2048 from the input image x.

Table content: <table><tr><td>+DCTD Yang et al. (2019)</td><td>Direct</td><td>Gaussian</td><td>Laplace</td><td colspan="2">Softmax (CE, L 2 ) Softmax (CE, L 2 , Var)</td></tr><tr><td>3.60 -</td><td colspan="3">3.09 ± 0.07 3.12 ± 0.08 3.21 ± 0.06 3.07 ± 0.07 3.11 ± 0.07 3.19 ± 0.06</td><td>3.04 ± 0.08 3.01 ± 0.07</td><td>3.15 ± 0.07 3.11 ± 0.06</td></tr><tr><td colspan="2">of our DCTD first extracts ResNet50 (</td><td/><td/><td/></tr></table>

Results on the two tracking datasets TrackingNet and UAV123. The asterisk * indicate an approximate value (±1), taken from the plot in the corresponding paper, due to the unavailability of the raw results. Our approach outperforms the baseline ATOM and other state-of-the-art trackers. Ours Bertinetto et al. (2016) Nam & Han (2016) Zhu et al. (2018) Li et al. (2019) Danelljan et al. (2019)

Table content: <table><tr><td colspan="8">Dataset ATOM  † TrackingNet Metric SiamFC MDNet DaSiamRPN SiamRPN++ ATOM Precision (%) 53.3 56.5 59.1 69.4 64.8 66.7 Norm. Prec. (%) 66.6 70.5 73.3 80.0 77.1 78.3</td><td>68.9 79.5</td></tr><tr><td/><td>Success (%)</td><td>57.1</td><td>60.6</td><td>63.8</td><td>73.3</td><td>70.3</td><td>72.1</td><td>73.7</td></tr><tr><td/><td>OP0.50 (%)</td><td>-</td><td>-</td><td>73.6</td><td>75  *</td><td>78.9</td><td>79.6</td><td>80.1</td></tr><tr><td>UAV123</td><td>OP0.75 (%)</td><td>-</td><td>-</td><td>41.1</td><td>56  *</td><td>55.7</td><td>56.0</td><td>59.8</td></tr><tr><td/><td>AUC (%)</td><td>-</td><td>52.8</td><td>58.4</td><td>61.3</td><td>65.0</td><td>65.0</td><td>66.5</td></tr></table>

SoftmaxThe DNN architecture of Softmax first extracts ResNet50 features g x ∈ R 2048 from the input image x. The feature vector g x is then processed by two fully-connected layers (2048 → 2048, 2048 → C), outputting logits for C = 101 discretized classes {0, 1, . . . , 100} It is trained by minimizing either the cross-entropy (CE) and L 2 losses, J = J CE + 0.1J L 2 , or the CE, L 2 and variance(Pan et al., 2018) losses, J = J CE + 0.1J L 2 + 0.05J V ar . The prediction ŷ is computed as the softmax expected value.

Table content: <table/>

Full results for the age estimation experiments. Refinement using DCTD consistently improves MAE (lower is better) for the age predictions outputted by a number of baselines.

Table content: <table><tr><td>Method</td><td>MAE</td></tr><tr><td colspan="2">OR-CNN (Niu et al., 2016) CORAL-CNN (Cao et al., 2019) Direct -Huber Direct -Huber + DCTD Direct -L2 Direct -L2 + DCTD Gaussian Gaussian + DCTD Laplace Laplace + DCTD Softmax -CE &amp; L2 Softmax -CE &amp; L2 + DCTD Softmax -CE, L2 &amp; Var Softmax -CE, L2 &amp; Var + DCTD 4.69 ± 0.03 5.74 ± 0.05 5.47 ± 0.01 4.80 ± 0.06 4.74 ± 0.06 4.81 ± 0.02 4.65 ± 0.02 4.79 ± 0.06 4.66 ± 0.04 4.85 ± 0.04 4.81 ± 0.04 4.78 ± 0.05 4.65 ± 0.04 4.81 ± 0.03</td></tr></table>

Full results for the head-pose estimation experiments. Refinement using DCTD consistently improves the average MAE for Yaw, Pitch, Roll (lower is better) for the predicted poses outputted by a number of baselines. ± 0.09 3.73 ± 0.13 2.90 ± 0.09 3.14 ± 0.07 Direct -Huber + DCTD 2.75 ± 0.08 3.70 ± 0.11 2.87 ± 0.09 3.11 ± 0.06 Direct -L2 2.81 ± 0.08 3.60 ± 0.14 2.85 ± 0.08 3.09 ± 0.07 Direct -L2 + DCTD 2.78 ± 0.08 3.62 ± 0.13 2.81 ± 0.08 3.07 ± 0.07 Gaussian 2.89 ± 0.09 3.64 ± 0.13 2.83 ± 0.09 3.12 ± 0.08 Gaussian + DCTD 2.84 ± 0.08 3.67 ± 0.12 2.81 ± 0.08 3.11 ± 0.07 Laplace 2.93 ± 0.08 3.80 ± 0.15 2.90 ± 0.07 3.21 ± 0.06 Laplace + DCTD 2.89 ± 0.07 3.81 ± 0.13 2.88 ± 0.06 3.19 ± 0.06 Softmax -CE & L2 2.73 ± 0.09 3.63 ± 0.13 2.77 ± 0.11 3.04 ± 0.08 Softmax -CE & L2 + DCTD 2.67 ± 0.08 3.61 ± 0.12 2.75 ± 0.10 3.01 ± 0.07 Softmax -CE, L2 & Var 2.83 ± 0.12 3.79 ± 0.10 2.84 ± 0.11 3.15 ± 0.07 Softmax -CE, L2 & Var + DCTD 2.76 ± 0.10 3.74 ± 0.09 2.83 ± 0.10 3.11 ± 0.06 F.1 TRAINING

Table content: <table><tr><td>Method</td><td>Yaw MAE</td><td>Pitch MAE</td><td>Roll MAE</td><td>Av. MAE</td></tr><tr><td>SSR-Net-MD (Yang et al., 2018)</td><td>4.24</td><td>4.35</td><td>4.19</td><td>4.26</td></tr><tr><td>VGG16 (Gu et al., 2017)</td><td>3.91</td><td>4.03</td><td>3.03</td><td>3.66</td></tr><tr><td>FSA-Caps-F (Yang et al., 2019)</td><td>2.89</td><td>4.29</td><td>3.60</td><td>3.60</td></tr><tr><td>Direct -Huber</td><td>2.78</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary: This paper proposes defining a regression model of the form where . The normalizing constant is analytically intractable, however, and the paper proposes evaluating it via importance sampling: where is set during training according to the observed label. This model is validated on several vision tasks including object detection, age estimation, pose estimation, and visual tracking. Results show competitive to superior performance against recently-proposed baselines. Pros: Defining more expressive predictive models---ones that can handle multiple modes and heteroskedastic response noise---is an important project, and this paper does seem to make progress towards this goal, as seen in Figure 1. Despite computational downsides, the paper proposes some efficient work-arounds (e.g. importance sampling, introducing the label deep into the architecture) that allow the model / method to be scaled to realistic vision tasks. The performance in the experiments seems good (but I was not previously familiar with these tasks). Cons: I have three significant reservations about the work… (#1) Incomplete validation of the core methodology: The paper claims to propose a general framework for constructing flexible regression models: “...a novel and general regression method with a clear probabilistic interpretation” (p 2). However, the paper considers only real-valued regression---when the core method is clearly applicable to other domains---and discusses only one numerical strategy for computing the difficult normalizing constant. As the model requires a forward propagation for every sample, such a sampling strategy has clear limitations (additional to the usual difficulties with importance sampling). The authors admit this and bypass the obstacle by introducing the label late into the architecture, meaning the computation specific to the features can be re-used. Yet, if the paper is truly proposing a “general and flexible framework,” I think ablation studies and runtime experiments showing that the model is applicable to a range of architectures and dimensionalities must be done. The current draft only validates these choices for specific vision tasks. (#2) Relationship to energy-based models: I was surprised that the authors make no mention of energy-based models---this paper’s Equation 1 is essentially Equation 2 from LeCun et al.’s [2006] “A Tutorial on Energy-Based Learning.” There are a large number of models discussed in this survey, including many for prediction. Is the proposed model possibly related to one of these? As the paper has no citations to or discussion of energy-based learning, it is impossible to tell. The authors need to include some discussion of this related work and probably should revise their model name so that it doesn’t seem like a totally new class of model. (#3) No discussion or consideration of alternative training strategies: The paper only considers approximate maximum likelihood learning, and the importance sampling approximation has serious limitations due to needing a forward prop for each sample, as discussed above. However, there is much work on estimating hard-to-normalizing models via other means such as score matching and contrastive divergence. There is even recent work on scaling these methods up to models for high-dimensional images: see [Du & Mordatch; 2019] and [Song et al.; 2019]. Can these methods be used to by-pass the normalization issue? If not, there should at least be some discussion of why. References Du, Y., & Mordatch, I. (2019). Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). A Tutorial on Energy-Based Learning. Predicting Structured Data. Song, Y., Garg, S., Shi, J., & Ermon, S. (2019). Sliced Score Matching: A Scalable Approach to Density and Score Estimation. UAI 2019.