Evaluation of a TV Programs Recommendation
using the EPG and Viewer’s Log Data
Kazuki Ikawa

Tomohiro Fukuhara

Graduate School ofEngineering,
The University of Tokyo
7-3-1 Hongo, Bunkyo-ku
Tokyo, Japan

National Institute of Advanced
Industrial Science andTechnology
2-3-26 Aomi, Koto-ku
Tokyo, Japan

tomohiro.fukuhara@aist.go.jp
Hideaki Takeda

Hideki Fujii
Graduate School ofEngineering,
The University of Tokyo
7-3-1 Hongo, Bunkyo-ku
Tokyo, Japan

National Institute ofInformatics
2-1-2 Hitotsubashi,
Chiyoda-ku
Tokyo, Japanfujii@save.sys.t.utokyo.ac.jp

takeda@nii.ac.jp

ABSTRACT1.

Evaluation results of a TV program recommendation using EPG (Electronic Program Guide) and viewer’s log data
are described. Two experiments using the log data of aJapanese video service provider are conducted: (1) an experiment of prediction of TV programs that each viewer usually
watched, and (2) a survey of TV program recommendation
using the log data and a questionnaire. High precision values are obtained in the ﬁrst experiment using keyword-basedand celebrity-based recommendation methods for terrestrial
and satellite broadcasting programs. Meanwhile, the channel & time-based recommendation method for programs in
paid channels of the video service provider obtained the highprecision in the second experiment. An overview of experiments and results are described.

EXPERIMENTATION

Today, many TV programs are broadcasted on various
channels. Terrestrial digital broadcasting services and satellite broadcasting services brought us many channels andprograms. As the increase of channels and programs, recommendation services for TV programs and channels areneeded that allow us to choose suitable programs to watch
for each use[4]. For designing a good TV program recommendation service, fundamental data that compares performances among various recommendation methods is needed[6].
We had two experiments of a TV program recommendation using EPG (electronic program guide) and the viewer’s
log data: (1) an experiment of prediction of TV programs
that each viewer watches usually, and (2) a survey of TV
program recommendation using the log data and a questionnaire. We created a prototype system of a TV program
recommender system that recommends programs for eachuser using EPG and the viewer’s log data. An overview ofexperiments and results are described.
In the following, section 2 describes the related work. Section 3 describes an overview of our approach for TV program
recommendation. Section 4 describes an overview of our experiment and its results. Section 5 discusses the results ofthe experiments. We summarize arguments in section 6.

Keywords2.

TV program recommendation, evaluation

There are many work on TV program recommendation.
TiVo, which is a TV program recommendation system basedon the collaborative ﬁltering (CF) method, is a popular
product in U.S[1]. Although the CF method works wellwhere a large number of judgement data is available, we
consider that the CF method have diﬃculties when new programs that few people watched before.
Smyth and Cotter introduces an overview and performanceof a personalized TV guide system[8]. They also adopted the
CF method. In their experiment, 310 households attended
to evaluate the precision of recommendation, and found that

Categories and Subject Descriptors
H.4.2 [Information Systems Applications]: Types of
Systems

General Terms.

182

INTRODUCTION

RELATED WORKTable 1: An example of a viewer’s log (dummy data)
Household ID
Date Time Channel ID
1000 01/Nov/2009 07:18
235
1000 01/Nov/2009 07:19
235
1000 01/Nov/2009 07:20
234

3.3

Recommendation method

Based on programs that each household watched, we ﬁrst
calculate feature values of keywords and celebrities using
following formulas[2][3][7].
Pkey (k) =Pperson (p) =Table 2: An example of EPG data.
Channel
ID
235

Start
Time
01/Nov/2009
07:00:00

End
Time
01/Nov/2009
07:59:00

Title

Description

***

***

61% of households answered positively. Their result showedthat the CF method showed the best performance, comparedto the content-based (CB) recommendation and a naive random recommendation.
Although there exist recommender systems based on the
CF method, few data is reported with respect to the evaluaion of the CB recommendation. In this paper, we will
evaluate the performance of a CB recommendation method
using the real data of a video service provider.

|W atched(k)||P rograms(k)||W atched(p)||P rograms(p)|

(1)
(2)

Pkey (k) is a function that calculates a feature value for a keyword k. W atched(k) is a set of programs that a household
watched. P rograms(k) is a set of programs that contains
the keyword k in their EPG data. Pperson (p) is a functionthat calcuates feature value for a celebrity person p. Pkey (k)and Pperson (p) are calculated for each household.
For calculating recommendation values of a program title,
we use following formulas[9].
X
Rkey (title) =Pkey (k)(3)
k∈Keywords(title)

Rperson (title)=

X

Pperson (p)(4)

k∈P ersons(title)

Rmix (title)=+

Rkey (title)
maxt∈T IT LES Rkey (t)Rperson (title)
maxt∈T IT LES Rperson (t)(5)

3. APPROACH FOR THE TV PROGRAM RECRkey (title) is a function that calculates a recommendation
OMMENDATION
3.1 Log data
We used the viewer’s log data of a video service provider
in Japan. We call this provider TV-999 in this paper. Eachhousehold has a set top box (STB) that receives TV programs of TV-999. TV-999 broadcasts terrestrial programs,
satellite broadcasting (BS1 ) programs, and paid channels
such as movie, drama, sports, animation and so on2 . Theview history of a household is recorded at the server machineof the provider3 .
Table 1 shows an example of log data. The log data consits
of following parts: (1) household ID, (2) date and time when
the household watched a channel, and (3) Channel ID atwhich the household watched. The data is recorded in eachminute. In this case, household ID 1000 watched channel
ID 235 and 234 during 7:18 through 7:20 a.m. on November
1st, 2009. For zapping action, we removed the view historyof which duration is under ﬁve minutes.

3.2 EPG data

value for a program title containing a keyword k in its EPG
data. Keywords(title) is a function that returns a set of
keywords contained in the EPG data of a program title.
Formula (3) is used as a keyword-based recommendation in
the next section.
Rperson (title) is a function that calculates a recommendation value for a celebrity person appeared in the EPG ofprogram title. P ERSON S is a set of all of persons appearedin all of EPG data. Formula (4) is used as a celebrity-based
recommendation.
Rmix (title) is a function that calculates recommendation
value for a program title. T IT LES is a set of all titles appeared in all of EPG data. Formula (5) is used as a keyword& celebrity-based recommendation.

4.

EXPERIMENTS

4.1

Prediction of daily viewing programs

Approach

EPG data contains content ID, date and time when thecontent is broadcasted, duration of the content (minute),
title and summary of the content. Table 2 shows an example
of EPG data. We extract keywords and names of celebrities
appeared in title and description. We can identify whichhousehold watched which programs by combining EPG and
the log data[5].
1
BS is a common name of one of the direct broadcast satellites in Japan.
2
There are 30+ paid channels in TV-999.
3
The log data is recorded under the agreement of each household. If a household does not want, the data is not recorded.

183

We evaluated the accuracy of prediction of programs thateach household daily watches. We used the log data in
February 2008. The data is divided into twofold: (1) theﬁrst part (February 1st to 15th 2008) is the training data
which is used for calculating feature values of keywords andcelebrities. (2) the second part (February 16th to 29th 2008)is the test data that is used for evaluating prediction of programs.
The log data contains the view history of 923 households.
111, 343 programs are contained in this data. Among these
programs, 29, 887(26.8%) are terrestrial/BS programs, and81, 456(73.2%) are programs of paid channels.

P recision =

R/N

(6)

Recall

R/C

(7)

=

0.2
Keyword-based recommendation(for Terrestrial/BS Programs)

0.15

RecallWith respect to programs watched by households, 669
programs are watched by each household in the test stage.
Among these programs, 70.1% (469) are terrestrial / BS
programs, and 29.9% (200) are programs of paid channels.
We prepared two types of recommendations: (1) keywordbased recommendation, and (2) celebrity-based recommendation. For the ﬁrst type of recommendation, we use the formula (3) to calculate a recommendation value of a programfor a household. For the second type of recommendation, we
use the the formula (4) to calculate recommendation a valueof a program for a household. We randomly picked up 10
households, and evaluated 2*100 programs, which is createdby two types of recommendations, for each household. We
used terrestrial / BS programs because the large amount oflog data was available for these programs.
As evaluation measures, we used precision and recall measures described in the following.

0.1

0.05

0
0

Results

0.15

Based on the results of the ﬁrst experiment, we had a survey of TV program recommendation using keywords andcelebrities. We prepared a questionnaire containing recommended programs for households using the log data. In thisexperiment, we chose programs in paid channels.
Table 3 shows the periods of this experiment. As a training period, we used the log data during November 9th through22nd (14 days), 2009. As a test period in which TV programs are recommended in the questionnaire, we chose theperiod during November 28th to December 4th (7 days).
For avoiding the change of preferences of each household5 ,we sent the questionnaire after the test period, i.e., eachhousehold received the questionnaire after December 5th.
The questionnaire contains programs recommeded by keywordbased and celebrity-based recommendations for each household. We asked each household up to ﬁve programs foreach day, i.e., up to 35 programs broadcasted in the test
period are asked. We asked each household to judge programs according following four categories: (1) Watched &
4
5

0.8

1

RecallCelebrity-based recommendation(for Terrestrial/BS Programs)

0.1

0.05

0
0

Approach

0.6(a) Recall and precision of the keyword-based recommendation for terrestrial/BS programs.

0.2

4.2 Evaluation of recommendations using a questionnaire

0.4
Precision

R is the number of programs that the system suggested anda household watched, and N is the number of programs thatthe system suggested. C is the number of programs that ahousehold watched.

Figure 1(a) and Figure 1(b) show the results for keywordbased and celebrity-based recommendations respectively. In
these ﬁgures, the recall values are quite low because the number of C in the formula (7) is much larger than the numberof programs that we recommended in the experiment4 . So,
the recall values are low. With respect to the precision, bothof recommendation types showed high precision values. We
considered that predicting programs that a household usually watches based on keywords-based and celebrities-based
recommendation was possible.

0.2

0.2

0.4

0.6

0.8

1

Precision(b) Recall and precision of the celebrity-based recommendation for terrestrial/BS programs.

Good, (2) Watched & ¬Good, (3) ¬Watched & Expected, and(4) ¬Watched & ¬Expected. Each household checked one ofthese categories for each recommendation.
For comapring performances of recommendation types, weclassiﬁed households into four types. Table 4 shows the listof recommendation types. Type A&B is the mixture of type
A and B, which is calculated using the formula (5). Type
C is the baseline type where programs are chosen randomly
according to the joint probability of channels and time-slotsthat a household usually watches TV. This probability is
calculated from the log data of a household in the training
period. We used the log data of 907 households of the service. We sent a questionnaire containing recommendation ofprograms for each household. Questionnaires are sent to 875
households via mail on December 4th, 20096 . By December
25th, 2009, 605 answers (69.3%) are returned.
6
Households received the questionnaire after December 5th,2009.

We only evaluated 100 programs for each household
This was a request from TV-999.

184

Table 3: Periods of experiment.
DatasetTerm# of Days
Training Nov.9th - Nov.22nd, 200914 days
Test
Nov.28th - Dec.4th, 2009
7 days

Rec.
Type
A
B
A&B
C

Table 5: Results of the experiment (%).
Rec.
Type
A
B
A&B
C

Table 4: Types of recommendation.
# of Received/
Description
Recommended (Ratio)
Keyword-based
112/160 (70.0%)
Celebrity-based
141/200 (70.5%)
Keyword&Celebrity
132/200 (66.0%)
Channel&Time
220/315 (69.8%)

Results
Table 5 shows the summary of the result. Most of recommendations were judged as ¬Watched & ¬Expected. Except
for ¬Watched & ¬Expected category, type C obtained thebest score in Watched & Good category. This was an unexpected result.
Table 6 shows another summary of the result which is organized according to viewpoints of prediction and recommendation. The prediction viewpoint is the union of Watched& Good and Watched & ¬Good evaluations, and recommendation viewpoint is the union of Watched & Good and¬Watched & Expected evaluations. For both of viewpoints,
type C (channel & time) showed the best score. Whereas
type B showed the lowest score in both of prediction andrecommendation viewpoints. Althgouh type A at the recommendation viewpoint showed the second score, it did notexceed the type C.

5. DISCUSSION
One of our unexpected things is that the keyword-basedand celebrity-based recommendations did not work in thesecond experiment. Although we expected that both of theserecommendations would show good scores, the results wasa contrary. Especially, type celebrity-based recommendation
was the worst among four recommendations in Table 6.
For reasons of this failure, we consider that there are preferences for paid channels. Because paid channels in TV999 are highly specialized, we consider that each household
already has its own preference for speciﬁc channels. For
understanding this phenomenon, further investigation using
the qualitative data such as the age of a respondent, numberof people in his/her household, and channels and celebrities
that s/he likes is needed.

6. CONCLUSION
We described evaluation results of experiments on TV program recommendation using EPG and viewer’s log data. We
had two experiments: (1) a TV program prediction experiment using the log data, and (2) a survey of TV program
recommendations using the log data and a questionnaire. In
the latter experiment, we found that the proposed methods
did not work well, and the baseline method recommendation
showed the best score. We will continue to investigate thisphenomenon by analyzing the obtained questionnaire data.

185

Watched&
Good
9.9
3.1
6.3
13.9

Watched&
¬Good
0.5
0.4
0.4
1.1

¬Watched&
Expected
17.5
12.1
16.7
16.3

¬Watched&¬Expected
72.1
84.4
76.6
68.7

Table 6: Performances with respect to prediction
and recommendation.
Prediction
Recommendation
Rec.
(Watched&Good ∪(Watched&Good ∪Type
Watched&¬Good) ¬Watched&Expected)
A
10.4 (%)
27.4 (%)
B
3.5 (%)
15.2 (%)
A&B
6.7 (%)
22.9 (%)
C
15.0 (%)
30.2 (%)

7.

REFERENCES[1] K. Ali and W. van Stam. Tivo: making show
recommendations using a distributed collaborativeﬁltering architecture. In KDD ’04: Proceedings of thetenth ACM SIGKDD international conference on
Knowledge discovery and data mining, pages 394–401.
ACM, 2004.
[2] L. Ardissono, C. Gena, P. Torasso, F. Bellifemine,
A. Chiarotto, A. Diﬁno, and B. Negro. Personalizedrecommendation of TV programs. In AI*IA 2003:
Advances in Artificial Intelligence, Lecture Notes in
Computer Science, volume 2829, pages 474–486, 2003.
[3] A. Bär, A. Berger, S. Egger, and R. Schatz. A
lightweight mobile TV recommender. In EUROITV’08: Proceedings of the 6th European conference on
Changing Television Environments, pages 143–147.
Springer-Verlag, 2008.
[4] A. Basso, M. Milanesio, and G. Ruﬀo. Events discovery
for personal video recorders. In EuroITV ’09:
Proceedings of the seventh european conference on
European interactive television conference, pages
171–174. ACM, 2009.
[5] G. Graefe. Query evaluation techniques for large
databases. ACM Comput. Surv., 25(2):73–169, 1993.
[6] M. Gude, S. M. Grünvogel, and A. Pütz. Predicting
future user behaviour in interactive live TV. In
EUROITV ’08: Proceedings of the 6th European
conference on Changing Television Environments, pages
117–121. Springer-Verlag, 2008.
[7] R. J. Mooney and L. Roy. Content-based book
recommending using learning for text categorization. In
DL ’00: Proceedings of the fifth ACM conference on
Digital libraries, pages 195–204. ACM, 2000.
[8] B. Smyth and P. Cotter. A personalized television
listings service. Commun. ACM, 43(8):107–111, 2000.
[9] W. E. Spangler, M. Gal-Or, and J. H. May. Using data
mining to proﬁle TV viewers. Commun. ACM,
46(12):66–72, 2003.