Pedestrian and Part Position Detection using a
Regression-based Multiple Task Deep Convolutional
Neural Network
Takayoshi Yamashita

Hiroshi Fukui

Yuji Yamauchi

Hironobu Fujiyoshi

Computer Science Department
Chubu University
Aichi 487–8501, Japanyamashita@cs.chubu.ac.jp

Computer Science Department
Chubu University
Aichi 487–8501, Japanfhiro@vision.cs.chubu.ac.jp

Robot Department
Chubu University
Aichi 487–8501, Japan
yuu@vision.cs.chubu.ac.jp

Robot Department
Chubu University
Aichi 487–8501, Japanhf@cs.chubu.ac.jp

Abstract—In driving support systems, it is not only necessary
to detect the position of pedestrians, but also to estimate thedistance between a pedestrian and the vehicle. In general approaches using monocular cameras, the upper and lower positionsof each pedestrian are detected using a bounding box obtained
from a pedestrian detection technique. The distance between thepedestrian and the vehicle is then estimated using these positions
and the camera parameters. This conventional framework uses
independent pedestrian detection and position detection processes
to estimate the distance. In this paper, we propose a method todetect both the pedestrian and their position simultaneously usinga regression-based deep convolutional neural network (DCNN).
This simultaneous detection method is possible to train efficientfeatures for both tasks, because it is attention to head andleg regions from given labels. In the experiments, our method
improves the performance of pedestrian detection compared with
the DCNN which detects only pedestrian. The proposed approachalso improves the detection accuracy of the head and leg positions
compared with the methods that detect only these positions. Using
the results of position detection and the camera parameters, ourmethod achieves distance estimation to within 5% error.

I.

I NTRODUCTION

Traffic accidents between cars and pedestrians are a likely
occurrence. In support systems for elderly drivers and autonomous cars, pedestrian detection is an important technology
for avoiding accidents or alerting the driver to dangerous
situations. To estimate the distance between a pedestrian and avehicle, researchers have developed monocular camera-based
approaches, stereo camera-based approaches, and LIDARbased approaches. LIDAR measures the time taken for anirradiating laser beam to return from an object. It has a
low resolution, as there is a limit to the number of laser
irradiations. In addition, the expense of LIDAR systems makes
them unfeasible in most publicly available vehicles. In contrast,
methods using stereo cameras estimate distances from theparallax of the two cameras. Although such systems are
increasingly common on public vehicles, there is a limit to
the miniaturization of the apparatus. Methods using monocular cameras consist of three processes: pedestrian detection,
position estimation of the upper and lower parts of the pedestrian, and distance estimation using this positional information
with the camera parameters. Monocular camera systems are
relatively inexpensive, and can be miniaturized. Because eachprocess is conducted separately, the performance of monocularcamera systems is worse than that of other methods. Thus,
monocular camera-based approaches require improved pedestrian detection and position detection methods. We propose
a method that detects both the pedestrian and their position
simultaneously, using a regression-based deep convolutional
neural network (DCNN). DCNN [1] have achieved state-ofthe-art performance in various fields, including pedestrian
detection [2]. In addition, DCNN make it possible to perform
multiple tasks in a single network. Zhang proposed a method
for the simultaneous detection of facial points, face poses,
glasses, smiles, and gender [3]. We apply a DCNN to detectpedestrians and their head and leg positions using a regression
technique in a single network. Our approach can also estimate
the distance between the pedestrian and the vehicle using thepositions obtained from the DCNN.
II.

R ELATED WORK

Methods to estimate the distance between pedestrians andvehicles can be categorized into approaches using monocularcameras, stereo cameras, or LIDAR. Methods based on LIDAR
obtain a point cloud from an irradiating laser and recognize
objects such as pedestrians from the shape of the point cloud.
While this method has the advantage that it is barely affectedby weather conditions, it has the critical issue of being too
expensive for publicly available vehicles.
Stereo camera approaches recognize the shape and positionof three-dimensional objects using a disparity map obtained
from the two cameras. In a vehicular environment, two cameras
are installed onboard the vehicle, and pedestrians are detectedfrom the parallax of the two cameras. Zhao proposed a methodthat extracts edge features using a disparity map and color
image, and inputs features to a multi-layer perceptron. This
allows high-accuracy pedestrian detection to be performed inreal time [4]. Because the disparity between the stereo cameras
occurs from the differences in position of a common object
in each image, it becomes difficult to recognize pedestrians
standing in front of other objects such as vehicles or buildings.
Monocular camera systems estimate distance by first detecting the pedestrian, then detecting the positions of upperand lower parts of their body, such as the head and legs.
Finally, the distance between the pedestrian and the vehicle
is estimated using this positional information. Dalal proposeda breakthrough pedestrian detection method that extracts gradient features using HOG [5], and various improved methods
have since been developed [6][7][8]. To handle variations inthe shape of pedestrians, the deformable part model detects
the whole body and partial regions simultaneously [9].
DCNN [1] have attracted attention for their ability to
extract efficient features, and have achieved state-of-the-art
performance on various benchmark datasets, including one
focused on pedestrian detection [2]. Ouyang proposed the twostage structure of joint deep learning, which obtains scores foreach body part and detects the pedestrian region from the scoreand feature map of the DCNN. Joint deep learning is robust to
various body poses as it considers information about separate
body parts. One of the great advantages of DCNN is theirability to handle multiple tasks. Zhang proposed an improved
method for facial part detection that simultaneously recognizes
face poses, whether the person is wearing glasses or smiling,
and their gender [3]. In this paper, we employ this advantage
of DCNN for pedestrian detection. Our DCNN detects thepresence of a pedestrian and simultaneously determines theirhead and leg positions.
Estimating the distance between a pedestrian and a vehicle
using a monocular camera requires the upper and lower positions of the pedestrian region. Kishino has proposed a method
for estimating this distance using a projective method [11]. We
also employ this projective method to the estimation processusing the head and leg positions obtained from the DCNN.
III.

P ROPOSED METHOD

We propose a regression-based DCNN that detects pedestrians and simultaneously regresses their head and leg positions.
First, we introduce the structure and training process of the
DCNN for this regression task.
A. DCNN for regression task
As shown in Fig. 1, a DCNN consists of three layer
types, namely a convolutional layer, pooling layer, and fully
connected layer. The convolutional layers and pooling layers
are arranged hierarchically, and the fully connected layers are
positioned after them. The DCNN can accept any type of
input image (e.g., color, grayscale, gradient, or preprocessed).
In the convolutional layer, the input image is convoluted andfiltered to size kx ×ky , and the whole input image is subjectedto the same filter. The convolved value x is input to theactivation function f (x) to give one value of the feature map.
The convolutional layer has M filters, and feature maps are
obtained from each filter. The traditional activation function isa sigmoid form, although Rectified Linear Units (ReLU) andMaxout are becoming more common, because the gradient ofthese functions does not vanish in the training process. Because
the sigmoid function f (x) outputs values of approximately 1
for large values of x, large gradient values cannot be obtained.
However, the ReLU proposed by Krizhevsky outputs the samevalue as the input when the input value is larger than 0. This
linear nature ensures that, even when x is large, the gradientis equal to 1. Maxout selects the maximum value h′ in aparticular position hk across all feature maps [12]. When there
are M filters in the convolution layer, the feature maps are split
into sets of k and the maximum value h′ is selected at eachposition, as described by Eq. (1).

h′ = max hk
k∈[1,K](1)

The feature maps are applied in the reduction process at thepooling layer. There are several pooling processes, such as
max pooling, average pooling, and Lp pooling. Max poolingselects the maximum value within a specified small region (i.e.,2 × 2), whereas average pooling calculates the mean value inthis specific region. Among the various pooling types, maxpooling is the most popular approach.
The convolutional layers and pooling layers are hierarchically layered to construct a deep network. The fully connected
layers, which are the same as in conventional neural networks,
follow this layered network, as shown in Eq. (2). The input
vector to the fully connected layer contains flattened values
from previous feature maps. The output value hi (x) obtained
from the activation function f (·) is the same as in the convolutional layer.
N∑hi (x) = f 
wij xj + bi (2)
j=1

In recognition tasks, the output layer has the same units
as the number of recognition classes. A specific unit outputs
a class probability of close to 1, whereas other units output
values close to 0. In our method, the DCNN for regression
has two units for pedestrian detection corresponding to theprobability of pedestrian and background, and six units for
position detection corresponding to the x and y coordinates of
the head and the right and left legs. The output layer in thisnetwork has a total of eight units, each of which output values
in the range [0, 1]. By multiplying the image size, we obtainthe coordinate values in the image. The differences in how
the DCNN handles recognition and regression tasks lie in thecalculation of output values. Whereas the probability of eachclass is given by the Softmax function in recognition tasks, itcomes from an identity function in regression tasks.
B. Training the DCNN
The parameters of the DCNN consist of filter values andconnection weights, along with their biases. These numerous
parameters are optimized by an iterative process using stochastic gradient descent and backpropagation. The parameters
are randomly initialized. Backpropagation calculates the error
between the output values and supervised label values of each
image Em , and accumulates these to give the total error E, as
described by Eq. (3).
1 ∑
Em
2 m=1
M

E=

(3)

where {m|1, ..., M } is the set of training samples. The
mean square error, which is the difference between the output
value yk and label tk in class k, is calculated using Eq. (4).

Em =

C
∑
k=1

(tk − yk )2(4)

Input image

Feature map
Convolution layer

Pooling layer
Convolution layer

Pooling layer

Fully connection layer
Fully connection layer
Output layer

Fig. 1. Structure of the Deep Convolutional Neural Network. It consists of three layer types, namely a convolutional layer, pooling layer, and fully connected
layer. The convolutional layers and pooling layers are arranged hierarchically, and the fully connected layers are followed.

Note that C indicates the number of units in the output layer.
Backpropagation updates the filter values and weights so as
to minimize the total error E by the gradient descent method.
The updated value of each parameter comes from the partial
derivative of E, as given by Eq. (5).
(l)

(l)

(l)

(l)

wji ← wji + ∆wji = wji − λ

∂E
(l)

where

(l) (l−1)

(6)

(l)

(l)

(7)

(l)
Vk =

∑

(l)
(l−1)
wkj ∗ yj

Pooling layer
Convolution layer

Poolinglayer

Fully connection layer
Fully connection layer
Output layer

∂wji

(l)

δk = ek f (Vk )Feature map
Convolution layer(5)

where λ is the learning rate of the training process, and(l)wji is the weight of the connection between unit i in layer l
and unit j in layer l + 1. The update amount and gradient are
obtained from Eq.(6) and Eq.(7), respectively.
∆wji = −λδk yj

Input image(8)

1 iteration

2 iterationr iteration

r+1 iteration

Fig. 2. Fully connected layer with dropout. It sets the corresponding valueof randomly selected units to 0 and continues training the remaining units ateach iteration.

update amount to propagate to lower layers, and the network
is able to recognize the connection between different parts.

jC. Estimating the distance to a pedestrian(l−1)
and yjis the output value of unit j in layer (l − 1), ek(l)is the error of unit k, and Vk is the weighted accumulation

value of unit k. The parameters are updated iteratively until
the maximum number of iterations has been reached or someconvergence condition has been satisfied. There are three ways
to input the training samples and obtain the error E: full-batch,
online, or mini-batch. Mini-batch updates the parameters according to a small subset of samples at each iteration. Thiscan obtain a sufficient change over each iteration, and the
update time is relatively short. For these reasons, Mini-batchis commonly used to train DCNN.
The dropout method improves the robustness of DCNN andprevents overfitting. This method sets the value of randomly
selected units to 0 and continues training the remaining units
at each iteration, as shown in Fig. 2. The number of units set to
0 is pre-defined, and is commonly 50%. Dropout enables the

Methods of estimating the distance between a pedestrian
and a vehicle using monocular cameras can be categorized
into approaches that use the size of the observed object andthose that use its position. In the case of the object size, it isdifficult obtain a stable distance value as errors are introduced
by differences in height. We therefore employ an approach thatuses the object position and assumes that the legs are grounded.
When the camera is fixed at a height of h and the plane of thecamera is parallel to the ground, the y coordinate of an image
is given by the following equation:
fh H
+
(9)
d
2
Note that d is the distance between the pedestrian and thecamera, f is the focal distance, and H is the height of theimage. The distance is given by Eq. (10).
y=Positive
sampleNegativesample(a) Daimler Mono Pedestrian Benchmak Dataset(b) INRIA Person Dataset

Fig. 3. Examples from the datasets. To reduce the processing time and evaluate the efficient of output structure, the test images are cropped in pedestrian andbackground regions.

1d = fh22y − H

0.9

(10)

0.8
0.7
0.6

E XPERIMENTS

We evaluated the proposed regression DCNN to confirm
the effectiveness of our method. In these experiments, weevaluated the accuracy of pedestrian detection, accuracy of
position detection, and estimation accuracy of distance in realenvironments. We used the INRIA Person Dataset and Daimler
Mono Pedestrian Dataset for pedestrian detection and position
detection. The Daimler dataset contains 31,320 positive samples and 254,356 negative samples for training, and 21,790
samples for testing. We applied a data augmentation technique
to increase the number of positive samples to 250,560. The
INRIA dataset contains 2,100 positive samples and 50,000
negative samples for training, and 1,000 positive samples
and 9,000 negative samples for testing. We again used data
augmentation to increase the number of positive samples inthe training set to 50,000.
To evaluate the pedestrian detection performance, we compared the results given by our method with those from a
detection-only DCNN. This DCNN had two units in its output
layer, corresponding to the probabilities of pedestrian andbackground, as in recognition tasks. For the position detection experiment, we compared our results with those from a
position-only DCNN. This DCNN had six units, corresponding
to the x and y coordinates of the head and the left and rightlegs. Table I summarizes the structure of each DCNN. The
proposed and comparison methods consist of three convolution
and pooling layers. The fully connected layers correspond to
the input size of each DCNN, i.e., 96×48 grayscale images forthe Daimler dataset and 128 × 64 color images for the INRIA
dataset.
To determine the accuracy of the distance estimates, we
require the camera parameters and the true distance between
the pedestrian and the camera. However, neither the Daimler
dataset nor the INRIA dataset include these data. Thus, we
examined the datasets to evaluate these distances.

Miss Rate

IV.

0.5

0.4

0.3

Regression DCNN 31.77%
Conventional DCNN 38.38%
0.2
0.0001
0.001

0.01

0.1

1

10

False Positive per Image

Fig. 4. Comparison result using the Daimler dataset. The dataset contains
21,790 grayscale images for testing.

A. Comparison of pedestrian detection
In general pedestrian detection based on DCNN, candidate
regions are obtained by another classifier during a preprocessing stage to reduce the processing time and the number of false
positives. Thus, we employed HOG- and SVM-based methods
to detect candidate regions. Fig. 4 shows the detection accuracy
using the Daimler dataset. For a false positive per image (FPPI)
rate of 0.1 the miss rate of the conventional DCNN is 38%,
whereas our method achieves a lower rate of 32%. Our DCNN
is able to focus on characteristic pedestrian regions, since thecoordinates of the head and legs are given as supervised labels.
Fig. 5 shows the detection accuracy on the INRIA dataset.
This dataset contains variations in human poses. Because the
HOG and SVM classifiers find it difficult to detect posevariations, we evaluated the pedestrian region from annotations
and a randomly cropped background. As a result, Fig. 5 showsthat the miss rate of our method was just 14% at an FPPW
of 0.1. This represents a significant improvement over theconventional DCNN, which achieved a miss rate of 39%.

TABLE I.
MethodInput

S TRUCTURE OF THE DCNN FOR EACH DATASET

Conv

Layer1Max
Pooling

MaxoutConv(a) Daimler dataset
Layer2Max
Pooling
MaxoutConv

Layer3Max
Pooling

MaxoutLayer4# ofunit

Layer5# ofunit

Layer6# ofunit

detection

96x48x1

8,5x3

2x2

2

16,5x42x2

232,5x4

2x2

2

1,000

500

100

regression

96x48x1

8,5x3

2x2

2

16,5x42x2

232,5x4

2x2

2

1,000

500

100

proposed

96x48x1

8,5x3

2x2

2

16,5x42x2

232,5x4

2x2

2

1,000

500

100

MethodInput
Conv

Layer1Max
Pooling

MaxoutConv

Layer3Max
Pooling

MaxoutLayer4# ofunit

Layer5# ofunit

Layer6# ofunit(b) INRIA dataset
Layer2Max
Pooling
MaxoutConv

detection

64x128x3

20,9x52x2

264,5x32x2

232,3x3

2x2

2

1,000

500

100

regression

128x64x3

20,9x52x2

264,5x32x2

232,3x3

2x2

2

1,000

500

100

proposed

128x64x3

20,9x52x2

264,5x32x2

232,3x3

2x2

2

1,000

500

100

TABLE III.

1

Miss rate

Softmax2
Sigmoid6
Sigmoid8

OutputSoftmax2
Sigmoid6
Sigmoid8

ACCURACY OF DISTANCE ESTIMATION
distance
5m
10m
15m

0.1

Outputestimation
4.89m
9.26m
14.12merror[%]
2.2
5.3
5.8

denote position detection results and the red points are theground truth. These results indicate that our method obtains
better position detection results for various poses.
C. Comparison of distance estimation in a real environment

Regression DCNN
Conventional DCNN
0.01
0.0001

0.001

0.01

0.1

False Positive Per Window

Fig. 5. Comparison result using the INRIA dataset. We collects1,000 positive
color images that are cropped from annotations and 9,000 negative color
images that are randomly cropped from test dataset.
TABLE II.

E RROR IN THE POSITION OF EACH PART [ PIXELS ]methods
regression DCNN
Proposed DCNN

methods
regression DCNN
Proposed DCNN(a) Daimler dataset
part
head
left legright leg
6.1
5.9
10.7
4.2
5.3
9.4
(b) INRIA dataset
partleft legright leg
8.7
9.9
8.0
9.2

head
8.1
6.5

average
7.6
6.3

average
8.9
7.9

B. Comparison of position detection
Table II indicates the accuracy of each position with theDaimler and INRIA datasets. The proposed DCNN reduces
the error compared with the regression DCNN, giving a 6%
error with respect to the image size of 96 × 48 in the Daimler
dataset. The proposed method also reduces the position error
to 7.9 pixels with the INRIA dataset, an error of 5.5% for theimage size of 128 × 64. Fig. 6 shows examples of the position
detection results using the INRIA dataset. The green points

We evaluated the distance between the pedestrian and thecamera using the position detection results. The evaluation
images were captured at distances of 5 m, 10 m, and 15
m with known camera parameters. At each distance, sevenwalking images were obtained. We then evaluated the mean
distance accuracy at each distance. The estimation accuracy
and examples of the result images are given in Table III and
Fig. 7, respectively. The green points denote position detection
results and the red points are the ground truth. Our method
is clearly able to estimate head and leg positions for various
pedestrian poses. The estimation distance is 4.89 m at the 5 m
distance, an error of 2.2%. At greater distances, although theestimation accuracy is worse, the error remains less than 5%,even at a distance of more than 10 m.
V.

C ONCLUSION

In this paper, we have proposed a regression-based DCNN
that simultaneously detects pedestrians and the positions of
their head and legs. The output layer of our DCNN has units
for the probabilities of pedestrian and background, and for thex, y coordinates of the head and legs. By performing multiple
tasks that have strong relationships, it is possible to obtaineffective features. As a result, the accuracy of both pedestrian
detection and position detection improved as compared with asingle-task DCNN. In addition, the proposed method achieveda distance estimation error of less than 5% using the position
detection results and camera parameters.

Fig. 6.

Examples of position detection using the INRIA dataset. Red points are ground truth and Green points are detection results.

 

 



Fig. 7. Examples of distance estimation. It is clearly able to estimate head and leg positions for various pedestrian poses. At the 5m distance, the estimation
distance is 4.89m, it is an error of 2.2%.

R EFERENCES[1] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-Based Learning
Applied to Document Recognition, Proceedings of the IEEE, 1998.
[2] W. Ouyang, X. Wang, Joint Deep Learning for Pedestrian Detection,
Computer Vision and Pattern Recognition, 2013.
[3] Z. Zhang, P. Luo, C. Change, T. Xiaoou, Facial Landmark Detection by
Deep Multi-task Learning, European Conference on Computer Vision,
2014.
[4] L. Zhao, C. E. Thorpe, Stereo- and Neural Network-Based Pedestrian
Detection, IEEE Transactions on Intelligent Transportation Systems,
Vol.1, No.3, pp.148-154, 2000.
[5] N.Dalal, B. Triggs, Histograms of oriented gradients for human detection, Computer Vision and Pattern Recognition, 2005.
[6] X. Wang, T. X. Han, S. Yan, An HOG-LBP Human Detection with
Partial Occlusion, International Conference on Computer Vision, 2009.
[7] J. Marin, D. Vazquez, A. Lopez, J. Amores, B. Leibe, Random Forestsof Local Experts for Pedestrian Detection, International Conference on
Computer Vision, 2012.
[8] W. Nam, B. Han, J. H. Han, Improving Object Localization Using
Macrofeature Layout Selection, International Conference on Computer
Vision Workshop on Visual Surveillance, 2011.
[9] P. Felzenszwalb, D. McAllester, D. Ramaman, A Discriminatively
Trained, Multiscale, Deformable Part Model, Computer Vision and
Pattern Recognition 2008.
[10] P. Sermanet, K. Kavukcuoglu, S. Chintala, Y. LeCun, Pedestrian Detection with Unsupervised Multi-stage Feature Learning, Computer Vision
and Pattern Recognition, pp.3626-3633, 2013.
[11] T. Kishino, Sun Zhe, R. Micheletto, A Fast and Precise HOG-Adaboost

[12]

[13]

Based Visual Support System Capable to Recognize Pedestrian andEstimate Their Distance, Lecture Notes in Computer Science, Vol. 8158,
pp.20-29, 2013.
I. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Couville, Y. Bengio, Maxout Network, International Conference on Machine Learning,
pp.1319-1327, 2013.
G. E. Hinton, N. Srivastava, A. Krizhevsky, S. Ilya, R. Salakhutdinov,
Improving neural networks by preventing co-adaptation of featuredetectors, Clinical Orthopaedics and Related Research, vol. abs/1207.0,
2012.