IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 2016

1221

PAPER

Privacy Protection for Social Video via Background Estimation and
CRF-Based Videographer’s Intention Modeling
Yuta NAKASHIMA†a) , Member, Noboru BABAGUCHI†† , Fellow, and Jianping FAN††† , Nonmember

SUMMARY
The recent popularization of social network services(SNSs), such as YouTube, Dailymotion, and Facebook, enables people toeasily publish their personal videos taken with mobile cameras. However,at the same time, such popularity has raised a new problem: video privacy.
In such social videos, the privacy of people, i.e., their appearances, must be
protected, but naively obscuring all people might spoil the video content.
To address this problem, we focus on videographers’ capture intentions.
In a social video, some persons are usually essential for the video content.
They are intentionally captured by the videographers, called intentionallycaptured persons (ICPs), and the others are accidentally framed-in (nonICPs). Videos containing the appearances of the non-ICPs might violate
their privacy. In this paper, we developed a system called BEPS, which
adopts a novel conditional random field (CRF)-based method for ICP detection, as well as a novel approach to obscure non-ICPs and preserve ICPs
using background estimation. BEPS reduces the burden of manually obscuring the appearances of the non-ICPs before uploading the video to
SNSs. Compared with conventional systems, the following are the main
advantages of BEPS: (i) it maintains the video content, and (ii) it is immune
to the failure of person detection; false positives in person detection do not
violate privacy. Our experimental results successfully validated these twoadvantages.
key words: intentionally captured person, conditional random field, background estimation, privacy protection, social video

1.

Introduction

Recently, many personal videos taken with mobile cameras
are being uploaded on such social network services (SNSs)as YouTube and Facebook, where they are widely navigatedand viewed. Generally, such social videos contain the appearances of persons, which are crucial privacy sensitive information. Since disclosing such privacy sensitive information may seriously violate the privacy rights of others, systems are required that obscure the appearances of persons.
Many systems for video privacy protection have been
proposed for applications with specific tasks, such as video
surveillance [1]–[4], Google Street View [5], [6], and lifelogging [7]. These systems basically detect all or a predetermined set of persons and obscure them using such appearance obscuration methods as blurring and blocking out. The
Manuscript received September 19, 2015.
Manuscript revised December 12, 2015.
Manuscript publicized January 13, 2016.
†
The author is with Graduate School of Information Science,
Nara Institute of Science and Technology, Ikoma-shi, 630–0192
Japan.
††
The author is with Graduate School of Engineering, Osaka
University, Suita-shi, 565–0871 Japan.
†††
The author is with Department of Information Science, University of North Carolina, USA.
a) E-mail: n-yuta@is.naist.jp
DOI: 10.1587/transinf.2015EDP7378

fundamental idea underlying these systems is that whether
the persons captured in the videos are shown to viewers issolely determined based on whether they grant permission.
Some systems strictly adhere to this idea and allow the individuals to choose whether to be shown [4], [8], [9], but thisrequires special infrastructure to identify persons.
Systems solely based on such an idea can suﬃce forvarious tasks, but they are not suitable for protecting privacy
in social videos for two reasons: (i) Social videos are usuallytaken by videographers who have a strong intention for capturing the videos, e.g., recording a child’s play or a friend’s
wedding. Such intentions determine persons who are essential for the videos, and videographers intentionally capture
them. We call them intentionally captured persons (ICPs).
Obscuring all or a predetermined set of persons without considering the ICPs significantly spoils the video content. (ii)In social videos, visual quality is an essential factor for enhancing the viewers’ experience; how persons are obscuredis thus crucial for privacy protection.
Considering these diﬀerences, we summarize the requirements for a privacy protection system for social videos,besides completely obscuring the appearances of persons
without permission, as follows:
1. It must present ICPs in privacy protected videos.
2. It must provide various methods for obscuring the appearances of persons.
Requirement 1) confirms that privacy protected videos
retain the content of their original. This requirement
might cause conflict between videographer intentions and
ICPs’ privacy because ICPs are determined solely based onvideographer intentions but the fundamental idea underlying
privacy protection systems does not allow this. Automatically resolving it is impossible without negotiation between
the videographer and the ICPs. Even if the conflict is resolved, we still need a technique to locate ICPs or non-ICPs
in the videos for automatic privacy protection. Requirement2) allows the users to choose a suitable appearance obscuration method to maintain the visual quality. This requirement
is demanding because flexible obscuration in, e.g. [4], requires background images, which are usually not available
for videos taken by mobile cameras.
In our previous work [10], [11], we developed a system that fulfills requirement 1) by selectively obscuring
non-ICPs under the assumption that no conflict exists between videographer intentions and the ICPs’ privacy. For
selective obscuration, our previous system adopts non-ICP

c 2016 The Institute of Electronics, Information and Communication Engineers
Copyright IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 20161222

detection [12] that detects all of the persons and classifies
them into ICPs or non-ICPs to obscure only the latter group(Fig. 1 (a)). Unfortunately, this system still suﬀers from alimited number of obscuration methods; it can only provide blocking out, blurring, and seam carving, all of which
can be applied without using background images. In addition, experimental results revealed the diﬃculty in detecting non-ICPs because they are not essential for the video,and videographers do not pay much attention to them. This
results in too small or partially captured non-ICPs. In ourprevious system, such detection failure directly leads to thedisclosure of appearances, making it inadequate for video
privacy protection.
In this paper, we propose a system for social video
privacy protection, called BEPS (Background Estimationbased Privacy protection system for Social videos), which
is an extension of our previous system [13]. BEPS takes anovel approach for video privacy protection that estimates
the background image of each frame, finds ICPs to be presented, and superimposes them onto the background image
to maintain the video content (Fig. 1 (b)). Superimpositionof ICPs enables BEPS to present them for requirement 1),
and background estimation oﬀers various obscuration meth-ods, as in Fig. 2, for requirement 2). BEPS also overcomesthe problem of sensitivity to detection failure in our previous system. Background estimation generates backgroundimages that contain no person; therefore, explicit detectionof non-ICPs is no longer necessary. This makes BEPS’s
privacy protection capability immune to non-ICP detection
failure.
The following are our contributions:• We develop a novel system for social video privacy
protection called BEPS assuming that the conflict between the videographer and the ICPs is resolved by negotiation. By using ICP detection and background estimation, BEPS automatically generates a privacy protected video without any additional burden on videographers. This is a new research direction toward video
privacy protection.
• We introduce new preprocessing and frame selection
stages for background estimation to make BEPS applicable to longer videos than our previous system [13].
Our experimental results demonstrate that these stages
actually achieve this without sacrificing BEPS’s performance.
• We propose a new ICP detection method using the conditional random field (CRF). Compared to our previous method [10], [12], our CRF-based model encodes
the spatial relationship among the persons. Our experimental results indicate performance gain over the previous methods.
The rest of this paper is organized as follows: The
next section introduces related work. Section 3 presents an
overview of BEPS and justifies the assumption that no conflict exists between videographer intentions and the privacyof ICPs. The details of the methods used in BEPS are described in Sects. 4–6. In Sect. 7, our experimental results areillustrated. Section 8 gives discussion, and we conclude thispaper in Sect. 9.
2.

Fig. 1 Diﬀerences between (a) our previous system [10], [11] and (b)BEPS in their approaches to privacy protection.

Related Work

Video privacy protection has been extensively studied for
fixed camera applications, especially for video surveillance.
Tansuriyavong and Hanaki [1] silhouetted persons, whosenames were displayed based on a face recognition technique. Chen et al.’s system [2] obscures a person’s appearance while preserving his/her shape and motion for surveillance purposes. Dufaux and Ebrahimi [3] provided scrambled video for unauthorized viewers, but for authorized

Fig. 2 Some Example of appearance obscuration. From left to right: original, see-through, dot, edge,
and avatar, some of which are from [4].

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1223

viewers, the original video was restored from the scrambled one. Chinomi et al.’s system [4], called PriSurv, adaptively applies various appearance obscuration, such as blurring, blocking out, and complete removal, based on the relationship between the person and the viewer. Mitsugami
et al. [14] proposed to replace persons to icons, which may
not cover the persons’ entire silhouette. For this, they synthesized a background image for a fixed camera and superimposed the icons on it. To handle diﬀerent lighting conditions, their background images are synthesized from longterm observation of the environment.
The privacy issue of Google Street View has also beenaddressed [5], [6]. Frome et al. [5] proposed a technique toincrease the recall rate of face detection to avoid disclosing
faces. Flores and Belongie [6] replaced persons with corresponding regions in other Google Street View images captured at diﬀerent positions.
For videos taken with mobile cameras, Kitahara et al.
proposed a system called Stealth Vision [15], which applies
pixelization to persons. To locate persons in a mobile camera’s frame, their system uses fixed cameras installed in thetarget environment. Brassil’s system [8] obscures individuals with a special device for locating them, and he argued
that his system was applicable to mobile cameras. It projects
the person’s position in the target environment onto the viewof a camera. Chaudhari et al. [7] blocked out faces in videos
by a wearable life-log system. YouTube obscures persons in
uploaded videos based on person detection [16].
Another interesting direction for video privacy protection research is knowledge discovery while preserving
privacy. Fan et al. [17] and Peng et al. [18] developed
methods for statistical inference from video collections distributed among multiple parties without disclosing the original videos to the other parties.
Privacy protection systems oﬀer various types of privacy obscuration for fixed cameras, e.g., blurring, showing a
dot at the person’s position, or even complete removal [19],because the background images are available. For mobile
cameras, however, the diﬃculty in obtaining the background
prevents privacy protection systems from using such obscuration as complete removal. Thus, BEPS adopts backgroundestimation for mobile cameras to gain more flexibility.
In addition, most existing systems for video privacy
protection preliminarily determine those to be obscured.
Some obscure all persons (e.g. [5]–[7]), and others use a
database and such devices as RFID readers and tags for
identification (e.g. [4], [8], [15]); this approach completely
ignores videographer intentions. Using such devices makesthese systems impractical, and ignoring intentions can detract from the video content (Fig. 3). In contrast, BEPS uses
ICP detection [12] for determining the individuals to be presented, which makes BEPS practical and suitable for social
videos.
The existing systems for video privacy protection detect persons to be obscured and apply appearance obscuration methods. With this approach, the detection failure
immediately discloses their appearances. By defining a cri-Fig. 3 Examples of original frame (left) and blocked out video frames
without considering videographer’s intention (right).

terion to measure privacy loss in videos, Saini et al. experimentally demonstrated that obscuring persons based on theirdetection is not reliable due to detection failure [20]. Therefore, research has addressed the reduction of detection failure [5]. Since BEPS takes a very diﬀerent approach that superimposes those to be presented on estimated backgroundimages, the generated privacy protected videos are immune
to detection failure. In other words, BEPS can be viewed asa system that makes a privacy protected background imageof the entire video frame regardless of the persons’ presence
and selectively reveals the ICPs by superimposing them on
the estimated background. This approach basically shares
the same idea as the conclusion of [20], where global obscuration is preferable to obscuration based on person detectionin terms of privacy loss.
BEPS can be deemed an extension of our previous system [10], [11], as it provides various privacy obscuration(Fig. 2) as well as complete removal. BEPS is also an extension of the system [13]. To improve ICP detection accuracy,
we developed an ICP model, which leverages ICPs’ temporal and spatial relationship using the CRF. This new model
is similar to one in previous work [21] for real-time applications, but we introduce richer features and a model at thecost of computational complexity to improve the accuracy.
In addition, the system [13] is not applicable to long video
because of its severe memory consumption during background estimation. Our preprocessing and frame selection
stages relax this problem by reducing the number of frames
used in background estimation.
Another extension of the system [10], [11] is real-time
privacy protection [22], which is crucial for live streamingfrom mobile cameras. To achieve real-time processing, it isconstrained from various aspects, including applicable features for distinguishing ICPs from non-ICPs and privacy obscuration methods. In contrast, since BEPS’s target is videos
to be uploaded to SNSs, the real-time requirement is not imposed on BEPS.
3.

System Overview

Most existing systems for video privacy protection (e.g.,[4], [8]) obscure all persons in a video or provide each person in it an opportunity to consent to be presented in the privacy protected video based on the fundamental idea for privacy protection mentioned in Sect. 1. This idea potentiallyconflicts with videographer intention. However, especiallyregarding videos uploaded to SNSs, we make the following

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 20161224

observations:• In most cases, ICPs are the videographer’s friends orfamily. For them, the videographer can easily obtain
permission to capture and upload the video.
• Since non-ICPs are generally passers-by, obtaining
their permission is practically impossible.
This means the potential conflict between videographer intention and the fundamental idea for privacy protection canbe resolved easily by negotiation between videographersand ICPs, where there is no conflict between videographersand non-ICPs because non-ICPs are obscured. BEPS generates privacy protected videos based on ICP detection, assuming no conflict.
Thus, BEPS is not designed to be a fully automatic system that strictly follows the idea behind privacy protection.
Rather, it is a tool for supporting videographers, who arecompliant to that idea, by automatically generating privacy
protected videos. The following shows the flow that videographers go through to upload their video: (i) Obtain permission to upload it to SNSs from the ICPs who appear in it.
(ii) Obscure all non-ICPs (i.e., people irrelevant to the video
content) using BEPS so that they can upload the video without permission from the non-ICPs. (iii) Upload the video
only when all ICPs in the video allow them to do so.
Figure 4 shows an overview of BEPS. First, BEPS estimates the background of frames in an input video using
the method [23]. Since it is designed for a small numberof images and computationally expensive, we perform preprocessing and frame selection stages to extract the frames
suitable for background estimation. For each frame in the
input video, we synthesize its background image from theestimated background for selected frames. After detecting
the ICPs, BEPS extracts the ICPs using the graph cuts algorithm [24]. Finally, a privacy protected video is generated

Fig. 4 Overview of BEPS. Privacy protected video frame is generated
for the frame in input frames enclosed by red.

by superimposing the ICPs onto the synthesized backgroundimage. The following provides their details.
4.

Background Estimation

In BEPS, we adopt Kim et al.’s method for its applicability to the images from a moving camera; however, their
method is computationally infeasible when it is applied tolong videos. Thus, observing that successive frames in avideo resemble each other and hardly contribute to the background estimation, we introduce preprocessing and frame
selection stages to reduce the number of frames used forbackground estimation.
Figure 5 shows an overview of our background estimation. The preprocessing stage clusters the frames and extracts a representative frame for each cluster. Then, givena target frame for which the background is estimated, theframe selection stage further extracts a subset of representative frames that are suitable for it. In the estimation stage,
the algorithm based on [23] estimates the target frame’s
background using that subset. We further reduce the computational burden by applying background estimation only
to representative frames and synthesizing the other frames
from them.
Preprocessing stage. Although there are many clustering algorithms, e.g., k-means [25] and aﬃnity propagation [26], we develop a simple clustering algorithm toconfirm that each resulting cluster consists of consecutive frames to simplify the synthesis of non-representative
frames.
Our clustering algorithm is provided in Algorithm 1.
Given set of frames in a video {Ft |t = 1, 2, . . . , NFrame },the algorithm outputs set of clusters S = {Ck |k =1, 2, . . . , NClusters }, where Ck is the k-th cluster. NFrame andNCluster are the numbers of frames in the video and clusters,
respectively. The basic idea is building a cluster that consists of consecutive frames that resemble the representative
frame. Let Ck and t be a current set of consecutive frames

Fig. 5 Overview of our background estimation. Target frame and frames
selected for it are indicated by red and green rectangles, respectively.

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1225
Algorithm 1. Clustering algorithm.
input {Ft |t = 1, 2, . . . , NFrame }S ← ∅, t ← 1, k ← 1
while t < NFrame do
Ck ← ∅
repeatCk ← Ck ∪ {Ft }Calculate criterion crit for set Ck = Ck ∪ {Ft+1 }Increment t
until crit > THcrit and |Ck | < THNS ← S ∪ {Ck }Increment k
end while
return S

and its largest time index, respectively. The algorithm calculates criterion crit for Ck = Ck ∪ {Ft+1 } by
1crit(Ck ) = maxs(Ft , Fτ ) |Ck | ,(1)FτFt ∈Ck /Fτ

where Ck /Fτ is Ck but excludes Fτ and |Ck | is the numberof frames in Ck . s(Ft , Fτ ) is the similarity between Ft andFτ based on the number of matched feature points, of which
detail is given in our previous paper [13]. The maximization
is calculated over Ck = Ck ∪ {Ft+1 }. This finds a set Ck offrames whose geometric mean of the similarities over Ck /Fτis high. If crit ≥ THcrit , the algorithm adds Ft+1 to Ck andrepeats this process until crit < THcrit holds true. Representative frame Fk∗ for Ck is defined as Fτ that maximizes crit.
In addition, the number of frames in a cluster is limited to at
most THN because too many frames in a few clusters excessively suppress their contribution to background estimation.
Threshold THcrit controls the balance between the computational cost and the accuracy of background estimation
with THN . A large value results in many small clusters andraises the computational cost because the number of representative frames increases. In contrast, a small value results in large clusters that decrease the accuracy of the estimated background of non-representative frames, because
the synthesized non-representative frames become diﬀerent
from the actual ones. We used 100 for THcrit , which gave agood balance (|Ck | averages 7.5 frames). The value of THN
was empirically set to 20.
Frame selection stage. Since the number of representative frames remains large for the graph cuts-based background estimation and may include frames that do not sharea field of view, we further extract frames that are suitable forthe background estimation of a given target frame. Observing that the suitable frames uniformly overlap with a major
portion of the target frame, i.e., the number of frames thatcover each pixel in the target frame is uniform, we extract
them as follows.
Let R = {Fk∗ |k = 1, 2, . . . , NCluster } be the set of all representative frames. As we estimate their backgrounds andthose of other frames are synthesized, target frame FT∗ is anelement of R. We reinterpret the above observation as thefollowing maximization problem:

Fig. 6(a) Original frame and (b) estimated background labels ln .

RT = arg max

R ⊂R


Fk∗ ∈RZT (Fk∗ ) + γ

D(Fk∗ , Fk∗ ),(2)

Fk∗ ,Fk∗ ∈R

where ZT (Fk∗ ) is the proportion of the area of FT∗ covered by
Fk∗ to the area of FT∗ and D(Fk∗ , Fk∗ ) is the distance between
the frame centers of Fk∗ and Fk∗ projected to FT∗ . Parameterγ determines the contribution of each term. In this equation, the first term rewards frames with large overlapping
areas, and the second term rewards frames that give large
distances from the other frames in R . The second term confirms that suitable frames are uniformly distributed over thetarget frame. Since this criterion monotonically increases
as |R | increases, we set the maximum number of frames inR to 20, considering the computational burden of the graphcuts algorithm. This maximization problem is computationally expensive; therefore, a greedy algorithm finds a suboptimal solution by adding one frame to R at a time, where
the added frame gives the maximum value of the criterion.
Estimation stage. To construct the background of FT∗
using RT , we employ a method used in our previous system’s background estimation [13], which is originally basedon [23]. The method assigns to each pixel of the target frame
FT∗ a label, ln , indicating the frame whose corresponding
pixel’s color is most likely to be the background. The likelihood of being background color is measured based on theassumptions that persons in video frames move while capturing and that the pixel’s background color appears more
frequently than the moving persons (see [13] for more detail).
Figure 6 (b) shows example labels obtained for theframe shown in Fig. 6 (a). Black represents the regions
where ln = LT , and the other colors represent ln  LT , where
LT represents the target frame, i.e., colors of the pixels withln = LT remain the same. The resulting labels are referred
to as background labels.
5.

ICP Detection and Extraction

To detect ICPs, we focus on our observation that a videographer pays much attention to them but less to the non-ICPs.
This diﬀerence is reflected in a videographer’s behavior,namely, how she moves her camera. Based on this observation, we detect all the persons and classify them as ICPs
or non-ICPs using features that encode the videographer’s
behavior.
Assuming that videographers suﬃciently capture the
ICPs so that at least their upper bodies are visible in theframe, we first detect and track the upper bodies of all the
persons. There are many methods for detection and tracking

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 20161226

the probability of y given X is obtained byp(y|X) =
1
exp EICP (X, y) ,Z

(4)where Z is a normalization term, and EICP is the energy function defined as
f (xt,k , yt,k ) +ht,k,k (yt,k , yt,k )EICP (X, y)=t,kFig. 7 Vertical positions uYt of ICPs (surrounded by green rectangles)
in the same frame tend to have similar values. Their sizes (uSt ) are alsosimilar. Those values of ICPs and non-ICP (surrounded by red rectangle)
are diﬀerent.

Fig. 8 Graphical representation of our CRF-based model. Lines connecting xt,k and yt,k represent local classifier terms f . Dashed lines are temporal consistency terms g, which depend on yt,k and yt−1,k . Spatial terms hhare represented by lines connecting yt,k and yt,k .

of persons, e.g., [27]–[29], but we assume that the detection
and tracking results are given so we focus on our goal ofvideo privacy protection.
Our ICP classification uses as features the trajectories
Y S
x = {(uXτ , uτ , uτ )|τ = 1, . . . , NT }(3)of tracked person in NT consecutive frames, where uX
τ and
S
are
the
positionofthedetection
window,anduuY
ττ is its
size, which can describe the videographer’s behavior withrespect to that person. For a person in the t-th frame, x consists of his trajectory centered at that frame. Each feature isnormalized so that its mean and variance over the training
dataset can be 0 and 1, respectively.
For classifying each person as an ICP or a non-ICP, we
use a statistical model. To make viewers understand what
the videographers want to present, the videographers tend
to capture the same ICPs for a while. Also, some featuresfrom a pair of ICPs in a single frame have correlation. For
example, the vertical position uY
t ’s of ICPs in the t-th frame
tend to have similar values as shown in Fig. 7. Taking these
into account, we adopt a CRF-based model. Let xt,k denotethe features extracted from the k-th person in the t-th frame,and X the set of features from all persons in all frames. The
problem is to estimate label yt,k ∈ {−1, 1} for xt,k , whereyt,k = −1 means the non-ICP and yt,k = 1 means the ICP. We
denote a vector of all labels by y. Figure 8 shows a graphicalrepresentation of our CRF-based model. Under this model,+ g(yt,k , yt−1,k ),t,k,k

(5)t,kconsisting of local classifier terms f , spatial relationshipterms h, and temporal consistency terms g.
A local classifier term indicates how likely the personwith xt,k is an ICP. Basically, our CRF-based ICP model
is linear, i.e., all terms are linear combinations of features;
therefore, to improve the discrimination power, we employ
a two-step approach. First, we learn a classifier f˜(xt,k ) for
which we employ a support vector machine (SVM) with allfeatures in xt,k . Second, as f , we use a linear function with abias term that takes the decision value f˜(xt,k ) as well as uX
t,k ,
S
uY
and
u
as
a
feature.
Tosummarize,
our
local
classifiert,kt,kterm isf (xt,k , yt,k ) = yt,k w uFt,k ,

(6)

Y
S
˜
where the feature vector uFt,k = (1, uXt,k , ut,k , ut,k , f (xt,k )) andthe parameter vector w = (w0 , w1 , w2 , w3 , w4 ) . This local
classifier term uses the position and size of the t-th frame’s
detection window multiple times to emphasize their importance in classification.
A temporal consistency term describes the relationshipbetween the labels yt,k and yt−1,k , defined as
a if yt,k = yt−1,k.
(7)g(yt,k , yt−1,k ) =0 otherwise

Negative a rewards consistent labels while positive a penalizes them.
Our spatial relationship term makes use of the correlation among ICPs. We design six features that encode
such correlation between a pair of persons in a frame, i.e.,
m
m
m
|um
t,k − ut,k | and |ut,k + ut,k | (m ∈ {X, Y, S}). We denote a vector containing them together with 1 for a bias parameter byuH
t,k,k . The spatial relationship term is
ht,k,k (yt,k , yt,k ) = −yt,k yt,k v uHt,k,k ,(8)

where v = (v0 , v1 , v2 , v3 , v4 , v5 , v6 ).
All parameters w, v, and a can be determined via learning. However, we learn w and v but not a to demonstrate theeﬀect of temporal consistency term. For learning, we employ contrastive divergence learning [30], which is repeatedly applied for various values of a. The feature vectors
FuHt,k and ut,k are normalized so that the mean and varianceof each element over the training dataset can be 0 and 1,
respectively.
To present ICPs in privacy protected videos, they are

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1227

Bt = ωk,k (t)Hk,t (Bk ) + [1 − ωk,k (t)]Hk ,t (Bk ),(10)where ωk,k (t) = (t − tk )/(tk − tk ) is a weight. tk and tk are
the indices of the k-th and k -th representative frames.
7.

Fig. 9 (a) Shape prior constructed using a dataset of manually labeledimages. (b) ICP detection result. (c) Obtained ICP labels l¯n . (d) Combinedlabels l˜n .

Experimental Results

We first evaluate the performance of ICP detection using ourdataset, comparing it with several baselines including [10].
Then we demonstrate the advantage of BEPS over our previous system [10] using several example videos. We also
show the impact of the frame selection stage on the risk ofprivacy disclosure by comparing the performance of BEPS
with and without it.
7.1 Performance Evaluation of ICP Detection

extracted from the original frames again using the graph cuts
algorithm as in [13]. More specifically, it infers ICP labelsl¯i ∈ {0, 1}, where l¯i = 0 represents that the i-th pixel belongs
to an ICP and l¯i = 1 otherwise. To maintain the detail of
ICPs’ shape, the algorithm works on pixels in all frames,
not grids consisting of 5 × 5 pixels in representative frames
as in background estimation. Figure 9 (c) shows an exampleof the obtained labels. The blue region represents l¯i = 0, andthe black region represents l¯i = 1. In this example, the ICP isaccurately extracted. Our previous paper [13] provides moredetail.
6.

Privacy Protected Video Generation

Privacy protected video frames are generated by superimposing the extracted ICPs onto the estimated backgroundimage. Although BEPS potentially generates videos shownin Fig. 2, we implement only non-ICP removal. Other typesof obscuration can easily be implemented after non-ICP removal.
Since BEPS only applies background estimation to
the representative frames, it synthesizes the background of
the non-representative frames based on the representative
frames. First, l¯i in Ft is projected to the two nearest representative frames. Combined label l˜i for each representative
frame is then generated:LT if l¯i = 1
l˜i =
,
(9)
ln otherwise
where ln is the label for Ωn that contains the i-th pixel.
An example of l˜n is shown in Fig. 9 (d). The regions withl˜n  LT are replaced by the pixels from the representative
frames corresponding to l˜n . Since pixel values from diﬀerent frames may diﬀer due to illumination change, for example, to prevent visual artifacts, we use Poisson blending [31].
After reconstructing two frames from l˜n and two representative frames, we again project them to target frame Ft andadd them with a weight. That is, by letting Hk,t (Bk ) andHk ,t (Bk ) be the projection of the two reconstructed frames,Bk and Bk , we generate privacy protected video frame Bt by

To evaluate ICP detection, we used a video dataset consisting of 20 videos. We could not use publicly available
video datasets because they did not contain ground truth labels of ICPs and non-ICPs, which should be specified by thevideographers who captured them. Instead, we collected thedataset ourselves† . We employed three videographers, andthey captured the videos from various situations including
both indoor and outdoor scenes. The main subjects weremoving and static people. The videographers then manually assigned to their video a label identifying each person
as either ICP or non-ICP in each frame. The frame size ofthese videos was 854 × 480 pixels, and the frame rate was
30 frames per second. The total number of frames in thevideo dataset was 32,725. As mentioned in Sect. 5, we did
not apply person detection but instead manually specified
them. After manually specifying the individuals in the video
dataset, videographers assigned to each person a label representing ICP or non-ICP. The cumulative total number of persons was 56,067, where the numbers of ICPs and non-ICPs
were 38,122 and 17,945, respectively. We manually tracked
the persons in the videos, which resulted in 361 tracked persons, where tracking was terminated if the person’s upper
body was disappear or occluded. Among them, 251 persons were tracked more than 30 frames, 151 persons werelabeled as ICP in more than one frame. We adopted fivefold cross-validation, where 16 videos were used for SVM
training and CRF model parameter learning, and five othersfor evaluation. We employed false positive rate (FPR) andtrue positive rate (TRP) as performance measures.
Figure 10 (left) shows the AUC values for various a. To
generate ROC curves from y∗ , we calculated the probabilityof yt = 1 given X and y∗ byp(yt = 1, X, y∗\t )p(yt = 1|X, y∗\t ) = ∗ ,yt ∈0,1 p(yt , X, y\t )(11)where y∗\t is y∗ excluding the t-th element y∗t , and we applied thresholding to this probability. The result indicated†We will provide our dataset with ground truth labels upon request. Please contact the first author.

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 2016

1228

Fig. 11 Definitions: (a) ΩT consisting of black pixels and (b) ΩICP andΩNON-ICP .

Fig. 10 Left: AUC values for various a. Right: ROCs for our proposed
ICP detection (OURS) and various baselines.

that a = 0.05 gave the best performance, which means that
our temporal consistency term is not very significant in ourmodel. This can be explained by our new features fed to the
SVM: They are a trajectory of detection window’s positionand size, and is temporally redundant. Therefore, temporalconsistency is already counted in the features.
To show the superiority of our ICP detection (OURS),
we compared it with three baselines: our previous ICP detection method used in [10] (PREV), the raw SVM decision
value-based method [12] (RAW-SVM), and human annotators (HUMAN). In RAW-SVM, we directly applied thresholding to decision values. PREV improved the detection accuracy using ICPs’ temporal consistency. We used the same
parameters as [10] for PREV. To evaluate the performanceof the human annotators, six human annotators† individuallyinferred whether a person in the videos was an ICP. For this,
we assigned specific IDs to all the persons in the videos.
For each ID, each human annotator specified the frames inwhich she/he inferred that the corresponding person was an
ICP. A person in a frame was judged to be an ICP if four ormore human annotators agreed. Figure 10 (right) shows the
ROC curves. The human annotators outperformed the others. OURS and PREV worked better than RAW-SVM, indicating that ICPs’ temporal consistency improved the performance. Our ICP detection outperformed PREV.
7.2

Comparison with Our Previous System

We evaluated BEPS’s capability for privacy protection andvideo content preservation by comparing it with our previous system (BASELINE) [10]. The parameter γ for BEPS
was empirically set to 1. An example frame of BASELINE
is shown in Fig. 1 (a). To show the potential performance of
BEPS and BASELINE, we used the ground truths of ICPs
and non-ICPs specified by the videographers of our video
dataset instead of person detection and the ICP classification
results. In some videos, persons were not manually specified because they were too small or were only partially captured. For fair comparison, we clarified whether such non†The human annotators were students in our laboratory, excluding the authors and videographers who captured the videos in
our dataset.

ICPs were included in each video. In addition, for demonstrating BEPS’s immunity to the failure of person detection
and ICP classification, we randomly dropped the manuallyspecified persons at the rate of  to simulate failure, where 
was set to either 0 or 0.1.
As evaluation measures for BEPS, we defined removal
rate (RR) and preservation rate (PR) as follows:
RR =|ΩT ∩ ΩICP ||Ω̄T ∩ ΩNON-ICP |, PR =,|ΩNON-ICP ||ΩICP |

(12)

where ΩT is the set of pixels with l¯n = LT , as in Fig. 11 (a),and Ω̄T is its complementary set. ΩICP and ΩNON-ICP are
the sets of the pixels in ICPs and non-ICPs, respectively,
and were manually labeled for all representative frames(Fig. 11 (b)). The RR and PR stand for how accurately BEPS
removes the pixels belonging to non-ICP and how accurately BEPS retains the pixels belonging to ICPs, respectively. For BASELINE, we deemed the blocked out regions
to be Ω̄T and used the same definitions as BEPS. We empirically determined the regions to be blocked out based on
the manually specified detection windows so that most of anon-ICP region was blocked. Some ICP regions may also be
partially blocked out if their positions are close to the nonICPs. Since the performances of the background estimation
and ICP extraction largely depend on the video content, suchas camera motion and person motions, we show RR and PR
for each representative frame instead of the statistics of ourvideo dataset. In addition, we applied BEPS to four videos
in our dataset, but the evaluation measures are presented fortheir excerpts with textual descriptions of their content.
Figure 12 shows RR (left) and PR (right) for each representative frame of VID1–VID4, where the black dots on
the horizontal axes are the representative frames. Figures
13–16 are examples of the original and generated framesin VID1–VID4 for  = 0. In these figures, the red rectangles indicate ICPs. Brief textual descriptions of each excerpt(about 10 s) with the length of the original video follow:• VID1 (64.9 s, Fig. 13): The videographer first captured
a sitting person as an ICP until around the 100-th frame.
Then two persons were captured as non-ICPs, corresponding to the interval without RR in the first row ofFig. 12. Finally, the person who was an ICP at the beginning became a non-ICP, and the other person was
captured as an ICP.
• VID2 (63.9 s, Fig. 14): This video contained two persons. First, one person was an ICP and the other was a

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1229
Table 1
# frames# representative frames
Background estimation
ICP detection & extraction
Privacy protected video gen.
Total

Fig. 12 Comparision: RR (left) and PR (right) between BEPS and
BASELINE for VID1–VID4 (from top to bottom).

non-ICP who was partially out of the frame. Then thevideographer captured both as ICPs.
• VID3 (9.93 s, Fig. 15): VID3 kept capturing two ICPs.
After the 200-th frame, some non-ICPs occluded thetwo ICPs. Since the non-ICPs intersected the camera’s
field of view, they were partially captured when they
were framed in and out.
• VID4 (40.9 s, Fig. 16): There were four persons, one ofwhom was an ICP. One of the non-ICPs in this video
was too small, and another was only partially captured.
For  = 0, the RR values gave comparable results for
BEPS and BASELINE in most videos. However, in somevideos, BEPS’s RR values dropped, which was significant
after the 100-th frame in VID1. This RR degradation wascaused by background estimation failure. Our backgroundestimation, which is analogous to median-based methods,
basically fails to estimate the background when non-ICPs
do not move, as in the third to fifth frames in Fig. 13. The

Timing results.
VID1
1947
196
5289 s757 s1552 s7598 s

VID2
1917
308
6488 s780 s1001 s8270 sVID3298
141067 s109 s
303 s
1479 s

VID4
1228
63
4863 s461 s1632 s6957 sslight degradation in the RRs of VID2, VID3, and VID4 wasalso caused by the failure in the background estimation.
The PR of BEPS was mostly lower than BASELINE.
This was caused by our inflexible shape prior. To present
ICPs in privacy protected videos, BEPS extracts them fromthe original video frames and superimposes them on an estimated background. In this process, our shape prior for
ICP extraction gave a weak likelihood of being included
in an ICP region when its actual shape was diﬀerent fromour shape prior shown in Fig. 9 (a). This resulted in thefailed extraction of the exact shapes of ICPs, especially forfeet and arms. The inflexibility of our shape prior also
led to blurring-like visual artifacts due to Poisson blending (Fig. 16). Employing a human pose estimation technique (e.g., [28]) can relax this inflexibility by deforming
the shape prior based on a person’s pose, although its implementation is beyond the scope of this paper.
For  = 0.1, BASELINE’s RR values significantly degraded in some frames because some manually detected
persons were randomly dropped. This result implies that
BASELINE and all other conventional systems that detectpersons to be obscured are vulnerable to detection failureof non-ICPs, resulting in their complete disclosure. On theother hand, the RR values did not change for BEPS, andthus, it was immune to the detection failure. Such insensitivity is desirable for privacy protection systems because
failure in person detection does not lead the disclosure of
the privacy sensitive information, although it in turn depends
on the performance of background estimation. In contrast,
BEPS’s PR values fell due to randomly dropped persons,and BASELINE was insensitive to detection failure of ICPs.
Table 1 shows the timing results on Windows 7 PC withIntel Core i7 CPU at 3.4 GHz and 16 GB memory. We implemented ICP detection in Python and all other components
in C++. Background estimation for representative frames
took processing time the most, although it was applied only
to representative frames. It took roughly 5 s per frame on
average. The graph cuts algorithm, which is used in all processes, was the bottleneck.
7.3 Validation of Frame Selection StageTo investigate the influence of the frame selection stage on
BEPS’s capability in privacy protection and video content
preservation, we compared BEPS with and without it, where
BEPS without the frame selection stage simulates our system in [13]. We used the same videos and parameters as in
Sect. 7.2, and  was set to 0. As mentioned above, handlingIEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 2016

1230

Fig. 13 Examples of original frames (top) and resulting frames (bottom) for VID1. ICPs are indicatedby red rectangles.

Fig. 14 Examples of original frames (top) and resulting frames (bottom) for VID2. ICPs are indicatedby red rectangles.

Fig. 15 Examples of original frames (top) and resulting frames (bottom) for VID3. ICPs are indicatedby red rectangles.

Fig. 16 Examples of original frames (top) and resulting frames (bottom) for VID4. ICPs are indicatedby red rectangles.

long videos is practically infeasible without the frame selection stage because such videos have many representative
frames and require a large amount of memory for processing. Therefore, we extracted an excerpt from each video
and applied BEPS without the frame selection stage to it,
while we applied BEPS with the frame selection stage tothe entire video and extracted the evaluation result that corresponds to the excerpt. In this way, we show that the frame
selection stage can give suitable representative frames for atarget frame from an entire video.
Figure 17 shows the evaluation results. The results for
BEPS with frame selection stage were almost identical tothose when  = 0 in the previous section. Without theframe selection stage, BEPS’s background estimation failedin VID2 and gave low RR values. This is because the excerpt did not contain the background pixels, while BEPS
with frame selection stage selected suitable frames from theother part of the original video. The slight diﬀerence in RR
and PR was caused by the diﬀerence in the frames used forthe background estimation. These results indicate that theframe selection stage enabled BEPS to handle long video sequences without significantly the RR and PR performances.
8.

Discussion

This section discusses (i) how BEPS fulfills the requirements in Sect. 1 and (ii) the limitations of BEPS.
(i) Fulfillment of requirements. A system for social

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1231

ICPs. However, our results also exposed two limitations.
First, in principle, background estimation fails if nonICPs are stationary, which leads to privacy disclosure. VID1
clearly showed this problem. From the third to sixth framesin Fig. 13, the person on the left side was a non-ICP, but thecurrent implementation of BEPS failed to remove him. To
address this problem, we need to use the output of a person detection algorithm so that BEPS can identify the regions that potentially contain non-ICPs. If the backgroundfor these regions is not estimated (i.e., ln = LT ), such a technique as image inpainting removes non-ICPs. However, thisapproach spoils one of the advantages of BEPS: insensitivity
to the low detection accuracy of non-ICPs. Another possible
approach is incorporating [32], which automatically detects
regions in which the background estimation failed and applies image inpainting to them.
The second problem is the inflexibility of the shapeprior. Currently, we use a shape prior of persons that does
not change regardless of their pose. In addition, since webuilt it as an averaged region of a body, our ICP extraction
fails to extract body parts that are invisible in many videos in
our dataset, such as feet. As mentioned above, human pose
estimation techniques (e.g., [28]) can solve this problem bymodifying the shape prior based on estimated poses.
9.

Fig. 17 Comparisons: RR (left) and PR (right) between BEPS with andwithout frame selection stage for VID1–VID4 (from top to left).

video privacy protection must fulfill the requirements mentioned in Sect. 1. Similar to our previous system [10], [11],BEPS successfully satisfied requirement 1) in most cases
using ICP detection. For requirement 2), BEPS can provide various appearance obscuration methods, as in Fig. 2,
based on the estimated background. Although we did notexperimentally demonstrate various appearance obscuration
methods except the complete removal of the appearance, implementation is trivial for most of them.
(ii) Limitations. From the experimental results, BEPS
worked well for video in which non-ICPs move. Comparedwith such conventional systems as [10], which first detect
individuals to be obscured and then apply a privacy obscuration method, BEPS was immune to the failure of person
detection. This immunity is preferable for privacy protection. In conventional systems, the failure of person detection directly resulted in privacy disclosure. In contrast, it
resulted in missing ICPs in BEPS but did not disclose non-Conclusion

We developed a system called BEPS to protect privacy insocial videos. Privacy protection for videos in SNSs is a
new class of problems that requires privacy obscuration andvideo content preservation. For this problem, BEPS firstestimates the background of the video frames and superimposes the detected ICPs on it. The advantage of BEPS isthat it does not require accurate non-ICP detection, which
is usually more diﬃcult than accurate ICP detection. We
experimentally showed that our ICP detection achieved an
AUC value of 0.91, and our ICP extraction and backgroundestimation worked well if non-ICPs move. Also, we demonstrated that BEPS was preferable to conventional systems
due to its immunity to detection failure. Our experimental
results also implied that background estimation and ICP extraction might fail, but BEPS’s advantage over conventional
systems makes it a beneficial alternative for social video privacy protection. Future work includes implementing a moresophisticated shape prior using, e.g., human pose estimation [28] and various privacy obscuration as shown in Fig. 2.
Acknowledgements
This work was partly supported by the Ministry of Internal Aﬀairs and Communications SCOPE No. 152107001,
JSPS KAKENHI Nos. 24240031, 25730115, 15H01686,
and NICT No. 178B0504.
References[1] S. Tansuriyavong and S. Hanaki, “Privacy protection by concealing

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.4 APRIL 20161232

persons in circumstantial video image,” Proc. 2001 Workshop on
Perceptive User Interfaces, 4 pages, 2001.
[2] D. Chen, Y. Chang, R. Yan, and J. Yang, “Tools for protecting theprivacy of specific individuals in video,” EURASIP Journal on Applied Signal Processing, vol.2007, no.1, 9 pages, 2007.
[3] F. Dufaux and T. Ebrahimi, “Scrambling for privacy protection invideo surveillance systems,” IEEE Trans. Circuits and Systems forVideo Technology, vol.18, no.8, pp.1168–1174, 2008.
[4] K. Chinomi, N. Nitta, Y. Ito, and N. Babaguchi, “PriSurv: privacy
protected video surveillance system using adaptive visual abstraction,” Advances in Multimedia Modeling, Lecture Notes in Computer Science, vol.4903, pp.144–154, 2008.
[5] A. Frome, G. Cheung, A. Abdulkader, M. Zennaro, B. Wu, A.
Bissacco, H. Adam, H. Neven, and L. Vincent, “Large-scale privacy
protection in google street view,” Proc. Int’l Conf. Computer Vision,
pp.2373–2380, 2009.
[6] A. Flores and S. Belongie, “Removing pedestrians from google
street view images,” Proc. Int’l Workshop on Mobile Vision,
pp.53–58, 2010.
[7] J. Chaudhari, S.S. Cheung, and M.V. Venkatesh, “Privacy protection
for life-log video,” Proc. Workshop on Signal Processing Applications for Public Security and Forensics, pp.1–5, 2007.
[8] J. Brassil, “Technical challenges in location-aware video surveillance privacy,” in Protecting Privacy in Video Surveillance,
pp.91–113, Springer Verlag, 2009.
[9] J.A. Halderman, B. Waters, and E.W. Felten, “Privacy management
for portable recording devices,” Proc. ACM Workshop on Privacy in
the Electronic Society, pp.16–24, 2004.
[10] Y. Nakashima, N. Babaguchi, and J. Fan, “Intended human object detection for automatically protecting privacy in mobile video
surveillance,” Multimedia Systems, vol.18, no.2, pp.157–173, 2012.
[11] Y. Nakashima, N. Babaguchi, and J. Fan, “Automatically protecting
privacy in consumer generated videos using intended human object
detector,” Proc. Int’l Conf. Multimedia, pp.1135–1138, 2010.
[12] Y. Nakashima, N. Babaguchi, and J. Fan, “Detecting intended human objects in human-captured videos,” Proc. Conf. Computer Vision and Pattern Recognition Workshop, 8 pages, 2010.
[13] Y. Nakashima, N. Babaguchi, and J. Fan, “Automatic generation ofprivacy-protected videos using background estimation,” Proc. Int’l
Conf. Multimedia and Expo, 6 pages, 2011.
[14] I. Mitsugami, M. Mukunoki, Y. Kawanishi, H. Hattori, and M.
Minoh, “Privacy-protected camera for the sensing web,” Proc. Information Processing and Management of Uncertainty in Knowledge-Based Systems. Applications, pp.622–631, 2010.
[15] I. Kitahara, K. Kogure, and N. Hagita, “Stealth vision for protecting
privacy,” Proc. Int’l Conf. Pattern Recognition, pp.404–407, 2004.
[16] “Face blurring: when footage requires anonymity,” July 2012.
[17] J. Fan, H. Luo, M.-S. Hacid, and E. Bertino, “A novel approach for
privacy-preserving video sharing,” Conf. Information and Knowledge Management, pp.609–616, 2005.
[18] J. Peng, N. Babaguchi, H. Luo, Y. Gao, and J. Fan, “Constructing
distributed hippocratic video databases for privacy-preserving online
patient training and counseling,” IEEE Trans. Information Technology in Biomedicine, vol.14, no.4, pp.1014–1026, 2010.
[19] N. Babaguchi, T. Koshimizu, I. Umata, and T. Toriyama, “Psychological study for designing privacy protected video surveillancesystem: PriSurv,” in Protecting Privacy in Video Surveillance,
pp.147–164, Springer Verlag, 2009.
[20] M.K. Saini, P.K. Atrey, S. Mehrotra, and M.S. Kankanhalli, “Privacy aware publication of surveillance video,” International Journal of Trust Management in Computing and Communications, vol.1,
no.1, pp.23–51, 2013.
[21] T. Koyama, Y. Nakashima, and N. Babaguchi, “Markov random
field-based real-time detection of intentionally-captured persons,”
Proc. Int’l Conf. Image Processing, pp.1377–1380, 2012.
[22] T. Koyama, Y. Nakashima, and N. Babaguchi, “Real-time privacy
protection system for social videos using intentionally-captured per-sons detection,” Proc. Int’l Conf. Multimedia and Expo, 6 pages,
2013.
[23] D.-W. Kim and K.-S. Hong, “Practical background estimation formosaic blending with patch-based Markov random fields,” Pattern
Recognition, vol.41, no.7, pp.2145–2155, 2008.
[24] Y. Boykov and V. Kolmogorov, “An experimental comparison ofmin-cut/max-flow algorithms for energy minimization in vision,”
IEEE Trans. Pattern Analysis and Machine Intelligence, vol.26,
no.9, pp.1124–1137, 2004.
[25] J.B. MacQueen, “Some methods for classification and analysis ofmultivariate observations,” Proc. 5th Berkeley Symposium on Mathematical Statistics and Probability, pp.281–297, 1967.
[26] B.J. Frey and D. Dueck, “Clustering by passing messages between
data points,” Science, vol.315, pp.972–976, 2007.
[27] N. Dalal and B. Triggs, “Histograms of oriented gradients for humandetection,” Proc. Conf. Computer Vision and Pattern Recognition,
pp.886–893, 2005.
[28] M. Andriluka, S. Roth, and B. Schiele, “Pictorial structures revisited: People detection and articulated pose estimation,” Proc. IEEE
Conf. Computer Vision and Pattern Recognition, pp.1014–1021,
2009.
[29] S. Walk, N. Majer, K. Schindler, and B. Schiele, “New features andinsights for pedestrian detection,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp.1030–1037, 2010.
[30] G.E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, pp.1771–1800, 2002.
[31] P. Pérez, M. Gangnet, and A. Blake, “Poisson image editing,”
ACM Trans. Graphics (Proc. ACM SIGGRAPH 2003), vol.22, no.3,
pp.313–318, 2003.
[32] X. Chen, Y. Shen, and Y.H. Yang, “Background estimation usinggraph cuts and inpainting,” Proc. Graphics Interface, pp.97–103,
2010.

Yuta Nakashimareceived the B.E. and
M.E. degrees in communication engineeringfrom Osaka University, Osaka, Japan in 2006
and 2008, respectively, and the Ph.D. degree
in engineering from Osaka University, Osaka,
Japan, in 2012. He is currently an assistant professor at Graduate School of Information Science, Nara Institute of Science and Technology.
He was a research fellow of the Japan Society
for the Promotion of Science (JSPS) from 2010
to 2012, and was a visiting scholar at the University of North Carolina at Charlotte in 2012. His research interests include
video content analysis using probabilistic and statistical approaches. He isa member of the IEEE, the ACM, the IEICE, and the IPSJ.

Noboru Babaguchi
is currently a Professorof the Department of Communication Engineering, Osaka University. He has published over
200 journal and conference papers and several
textbooks. He is a fellow of the IEICE, a senior member of the IEEE, and a member of theACM, the IPSJ, the ITE and the JSA.

NAKASHIMA et al.: PRIVACY PROTECTION FOR SOCIAL VIDEO VIA BACKGROUND ESTIMATION

1233

Jianping Fan
received the M.S. degreein theory physics from Northwest University,
Xian, China,in 1994, and the Ph.D. degree in optical storage and computer science from Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China,
in 1997. He was a Postdoctoral Researcher
with Fudan University, Shanghai, China, during
1998. From 1998 to 1999, he was a Researcher
with the Japan Society of Promotion of Science(JSPS), Department of Information System Engineering, Osaka University, Osaka, Japan. From September 1999 to 2001,
he was a Postdoctoral Researcher with the Department of Computer Science, Purdue University, West Lafayette, IN. In 2001, he joined the Department of Computer Science, University of North Carolina, Charlotte, as
an Assistant Professor and became Associate Professor and Professor. His
research interests include image/video analysis, semantic image/video classification, personalized image/video recommendation, surveillance videos,and statistical machine learning.