Letters

Non-intrusive QoS Monitoring Methodfor Realtime Telecommunication Services
Takanori Hayashi†, Masataka Masuda,
Toshiko Tominaga, and Kazuhisa Yamagishi
Abstract
This article describes a non-intrusive QoS (quality of service) monitoring method for realtime
telecommunication services. The key point of this method is that the invalid packet ratio and the invalid
frame ratio are introduced to represent the factors affecting the speech and video qualities, respectively.
Moreover, we can estimate multimodal quality by considering the individual qualities and their interaction. The results of subjective tests showed that the estimation accuracy of these factors was sufficientfor practical use. This method enables us to manage QoS on a call-by-call basis for interconnections
among multiple network service providers including a variety of terminals and applications.

1. Introduction
With recent advances in IP (Internet protocol) andmultimedia technologies, various realtime telecommunication services such as IP telephony, videophones, and video-streaming services are being
developed and provided. It is important to properly
evaluate and manage the end-to-end quality of service (QoS) to provide comfortable services because
their performances are generally not guaranteed inpresent IP networks. In the next-generation IP network, moreover, commercial realtime telecommunication services must have an acceptable QoS. The
prime criterion for QoS is subjective quality, i.e., theusers’ perceptions of service quality. This can bemeasured through subjective quality assessment.
However, while subjective quality assessment is themost reliable method, it is time-consuming andexpensive. In particular, it is highly impracticable asa method for monitoring/managing quality in realtime. Therefore, we need an objective method that isbased solely on physical measurement but produces
results comparable with those of subjective testing.
For managing the end-to-end QoS on a call-by-call† NTT Service Integration Laboratories
Musashino-shi, 180-8585 Japan
E-mail: t.hayashi@lab.ntt.co.jp

Vol. 4 No. 4 Apr. 2006

basis, passive (non-intrusive) QoS monitoring ismore suitable than active (intrusive) QoS monitoring.
Non-intrusive methods use either speech/video signal
or network performance metrics. We focus on the latter approach because it does not require signal
recording equipment and can be implemented easily.
This article describes a non-intrusive QoS monitoring
method for realtime telecommunication services thatcan evaluate and manage the end-to-end QoS on acall-by-call basis by passive measurement of network
performance. First, we introduce the framework of
the non-intrusive QoS monitoring method andexplain the quality estimation factors for speech,video, and multimodal qualities. Then, we show theeffectiveness of this method through the results ofsubjective quality assessment tests.
2. Framework of non-intrusive QoS monitoring
The framework of non-intrusive monitoring to estimate and manage QoS for realtime telecommunication services is outlined in Fig. 1. Three key techniques are required: intermediate quality parameter
estimation, intermediate quality parameter reporting,and end-to-end QoS estimation.
(1) Intermediate quality parameter estimation
It is difficult to estimate end-to-end QoS of realtime
telecommunication services by measuring only net35

Letters(1) Intermediate quality
parameter estimation
Quality estimation factors(2) Intermediate quality
parameter reporting(3) End-to-end QoS estimation

RTCP XR

Multimodal quality estimation

Network performance reporting,application performance reporting, etc.

Speech quality
estimation

Network and terminal/application
performance measurement

Video quality
estimation

Packet loss, delay, jitter,
codec type, playout buffer size, etc.

QoS management system
Terminal/application

Quality information

Network domain 1

Terminal/application

Network domain 2

End-to-end QoS

Fig. 1. Framework of non-intrusive QoS monitoring and estimation.

work performance parameters such as IP packet loss
and transmission delay because it depends not only
on network performance but also on terminal/application implementation such as codec type and jitter
buffer size. For example, even if no IP packets arelost in the network, data can still be lost by jitter
buffer overflow/underflow due to variations in network delay. On the other hand, packet-loss concealment techniques used at a receiving terminal canensure that the user does not perceive the quality
degradation even if some IP packets are lost in thenetwork. Therefore, it is necessary to estimate intermediate parameters by measuring both network andterminal/application performances in order to estimate the end-to-end user’s perceptual QoS at thereceiving terminal.
(2) Intermediate quality parameters reporting
For integrated QoS management for realtime
telecommunication services, information about theintermediate quality parameters should be collected
in a QoS management system. The Internet Engineering Task Force (IETF) defined RTCP XR (realtime protocol control protocol extended report) in
RFC3611 [1]. It defines the extended report packet
type for RTCP and a container for the information
obtained by intermediate quality parameter estimation. This mechanism gives quality information about
interconnection calls in environments with multiple
network service providers or a variety of terminals/applications. It should be noted that RTCP XR itself
does not have functions for estimating quality. A
quality estimation metric is not recommended in this36

RFC but is for further study.
(3) End-to-end QoS estimation
Speech and video qualities are estimated individually using reported intermediate quality parameters.
Finally, the quality of realtime telecommunication
services should be managed by estimating multimodal quality considering the interaction among theindividual speech and video qualities. In Fig. 1, theQoS management system has functions for estimating individual and multimodal qualities, but thesefunctions could be implemented in terminals/applications.
3. Intermediate quality parameter estimation
We use an intermediate quality parameter metricfor estimating the end-to-end QoS from network performance metrics [2], [3]. There is a strong correlation between the end-to-end QoS and the total data
loss in the network and terminals/applications.
Therefore, the key point in properly evaluating users’perceptual QoS is how to measure the data loss at thejitter buffer in a terminal/application. Although thedata loss in a terminal/application generally cannot
be measured by network managers, our method canestimate it from network performance measurements.
We chose to use the invalid packet ratio*1 as the*1 Invalid packets: These are packets lost in the network or packets
that arrive late at a terminal/application. The number of invalid
packets divided by the total number of sent packets is defined as
the invalid packet ratio.

NTT Technical ReviewLetters• Number of packets lost in the network

Delay variation of each packet

Delay variation

Provisional
buffer size

Calculate coefficient ofdetermination between MOSs
and invalid packet ratio

Coefficient ofdetermination

MOSs for speech
or video• Number of late-arriving packets

Provisional buffer size

Invalid packet ratio

Equivalent buffer size
Packet numberFig. 2. Method of deriving an equivalent buffer size.

main quality estimation factor. In most cases, subjective quality is represented as a mean opinion score(MOS) calculated by averaging the subjects’ scores,
which can be estimated from the invalid packet ratioin our method. The end-to-end QoS can thus be estimated by determining the relationship between theMOS and the invalid packet ratio. We define theinvalid packet ratio as the ratio of sent packets to thesum of the lost and late-arriving packets*2, i.e., thosethat do not enter the terminal’s jitter buffer in therequired time and are treated as lost data. The invalid
packet ratio is calculated as a function of the jitter
buffer, packet loss, and delay variation*3 of packets.
The feature of this approach is that these network andterminal performances can be converted into a single
metric, which is calculated in the following way.
Note that this approach does not take into account theeffects of FEC (forward error correction), ARQ(automatic repeat request), etc.
When the size of the terminal’s jitter buffer is notknown and is not constant, i.e., the buffer is dynamic,
we can derive an equivalent jitter buffer size based on
the subjective quality evaluation, as illustrated in Fig.
2. Invalid packet ratios are calculated for various provisional buffer sizes, and a regression curve of theMOS versus invalid packet ratios is calculated. The
buffer size for which the coefficient of determination*2 Late-arriving packets: The number of these packets is calculatedusing the delay variation and the jitter buffer size. When the delay
variation for each packet is greater than an equivalent jitter buffer
size (described later), the packet is defined as a late-arriving packet.
*3 Delay variation: We can use the packet-receiving interval or thedelay variation metrics recommended by ITU-T Y.1540 [4] orIETF RFC3393 [5] as the delay variation metrics.

Vol. 4 No. 4 Apr. 2006

is maximum is defined as the equivalent buffer size.
The regression curve calculated with the equivalentbuffer size can then be used as the quality estimation
function. If the equivalent buffer size and this regression curve are identified beforehand, then the MOS
can be estimated simply by measuring the network
performance for the same terminals/applications.
Therefore, our method enables efficient network performance management considering the end-to-end
QoS because it is not necessary for network managers
to perform subjective quality evaluation trials in eachnetwork.
4. End-to-end QoS estimation
A subjective quality experiment was conductedusing a point-to-point audiovisual communication
service to evaluate the efficiency of our method. The
experimental parameters for network performance
were packet loss and delay variation.
4.1 Speech quality estimation
The relationship between speech quality and packet loss ratio is shown in Fig. 3(a). Figure 3(b) shows
the result of converting the packet loss ratio in Fig.
3(a) into an invalid packet ratio derived by the intermediate quality parameter metric. Speech quality wasevaluated using a five-grade quality rating scale (5:excellent, 4: good, 3: fair, 2: poor, 1: bad) and represented as a MOS calculated by averaging the subjects’ scores. The results in Fig. 3 show that theinvalid packet ratio enables the accuracy of subjective speech quality estimation to be dramaticallyimproved compared with using the packet loss ratio.

37

Letters

5

4

3

Quality estimation
is impossible.

2

1

Speech quality (MOS)

Speech quality (MOS)

5

Coefficient ofdetermination:0.99

43

2

1
00.001

0.002

0.003

0.004

00.025 0.050 0.075 0.100 0.125

Packet loss ratio

Invalid packet ratio(a) Quality estimation using
packet loss ratio(b) Quality estimation using
invalid packet ratio

Measured video quality (MOS)

Fig. 3. Speech quality estimation using two quality estimation factors.

5
Example of GOP structure

4Time

3P B B I B B P B B P B B P B B P B B I B B P

21Case 1)
12

3

4

5

Estimated video quality (MOS)

Fig. 4. Video quality estimation using
invalid packet ratio.

17 video frames are degraded by one invalid packet in an I-picture.
Case 2)
5 to 14 video frames (11 in this case) are degraded by one invalid packet ina P-picture. The number of affected frames depends on the position of theP-picture with the invalid packet.

Case 3)
4.2 Video quality estimation
The relationship between meaOne video frame is degraded by one invalid packet in a B-picture.
sured video quality and video qualPosition of a frame with an invalid packet
ity estimated using the invalid
packet ratio is shown in Fig. 4.
Duration of degraded video frames
Although the coefficient of determination is sufficient, it is lower
Fig. 5. Examples of how an invalid packet can affect video frames.
than that in Fig. 3(b) because thevideo coding method used in thesubjective test was MPEG2 based
the header information in invalid packets. Quality
on inter-frame coding. There are some cases in which
estimation using the invalid frame ratio is shown in
one invalid packet leads to quality degradation in
multiple video frames whose duration depends on thetype of picture*4 with the invalid packet and on the*4 An MPEG2 video sequence is constructed from three types of pictures. I-pictures (intra-coded pictures) are coded without reference
structure of a group of pictures (GOP), as shown into other pictures. P-pictures (predictive-coded pictures) are coded
Fig. 5. Therefore, we used a new quality estimation
with a past intra- or predictive-coded picture and are generally
factor, the invalid frame ratio. It is calculated as theused as a reference for further prediction. B-pictures (bidirectionnumber of degraded frames divided by the total numally predictive-coded pictures) are coded with both past and futurereference pictures and are not used as references for prediction.
ber of frames. Frames are judged to be degraded from38

NTT Technical Review

Letters

Fig. 6. The quality estimation accuracy in Fig. 6 ismuch better than that in Fig. 4.

5. Conclusion
We described a non-intrusive QoS monitoring
method for realtime telecommunication services. The
invalid packet ratio and the invalid frame ratio areeffective intermediate quality parameters for estimating speech and video qualities, respectively. We verified that the method’s quality evaluation accuracy
was sufficient not only for the individual speech andvideo qualities but also for multimodal quality.
In the next-generation IP network, proper management of the end-to-end QoS will be an importantfunction in the network. We believe that non-intrusive measurement based on passive network measurement is the most promising way of managing
quality on a call-by-call basis.

4.3 Multimodal quality estimation
Finally, we consider a model that can accurately
evaluate multimodal quality from the individual
media qualities and their interaction [6], [7]. The relationship between subjectively measured and objectively estimated multimodal quality is shown in Fig.
7. We found that the model’s evaluation error (rootmean square error) was less than the mean of the 95%
confidence interval for the subjective MOS, so thequality evaluation accuracy of this model is sufficientfor practical use.

References
Measured video quality (MOS)

[1]
5[2]
4[3]

3[4]
2[5]
1[6]

1
2
3
45
Estimated video quality (MOS)[7]Fig. 6. Video quality estimation using invalid frame ratio.

RFC3611, “RTP Control Protocol Extended Reports (RTCP XR),”
Nov. 2003.
M. Masuda and K. Ori, “Network performance metrics in estimating
the speech quality of VoIP,” APSITT 2001, pp. 333-337, Nov. 2001.
M. Masuda and T. Hayashi, “Non-intrusive Quality Monitoring
Method of VoIP Speech Based on Network Performance Metrics,”
IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 304-312, Feb. 2006.
ITU-T Rec. Y.1540, “Internet protocol data communication service—
IP packet transfer and availability performance parameters,” Dec.
2002.
RFC3393, “IP Packet Delay Variation Metric for IP PerformanceMetrics (IPPM),” Nov. 2002.
T. Hayashi and K. Yamagishi, “Multi-Modal Quality Estimation
Model based on Interaction of Audio and Video Quality,” Proc. The
2004 IEICE General Conference, B-11-12, p. 505, March 2004 (inJapanese).
K. Yamagishi and T. Hayashi, “Opinion Model using Psychological
Factors for Interactive Multimodal Services,” IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 281-288, Feb. 2006.

Measured multimodal quality(MOS)5
The multimodal quality can be
judged to be above or below a
threshold with a certain probability.
For example, about 95% for a
threshold of MOS = 3.5.

4

3Good quality region
2Bad quality region

112

3

4

5

Estimated multimodal quality(MOS)

Fig. 7. Estimation of multimodal quality from individual media qualities.

Vol. 4 No. 4 Apr. 2006

39

Letters

40

Takanori Hayashi

Toshiko Tominaga

Senior Research Engineer, Communication
Traffic & Service Quality Project, NTT Service
Integration Laboratories.
He received the B.E. and M.E. degrees in physical engineering from the University of Tsukuba,Ibaragi in 1988 and 1990, respectively. Sincejoining NTT Laboratories in 1990, he has been
engaged in the subjective quality assessment ofmultimedia telecommunications and network
performance measurement methods. He is currently working on a method of assessing multimodal quality on IP networks. He is a member of
the Institute of Electronics, Information andCommunication Engineers (IEICE) of Japan.

Research Engineer, Communication Traffic &Service Quality Project, NTT Service Integration
Laboratories.
She received the B.E. and M.E. degrees inindustrial engineering from the University ofElectro-Communications, Tokyo in 1987 and1989, respectively. Since joining NTT Laboratories in 1989, she has been studying quality management methods for telecommunication services. She is currently working on quality management methods for video communications
over IP networks. She is a member of IEICE.

Masataka Masuda

Kazuhisa Yamagishi

Research Engineer, Communication Traffic &Service Quality Project, NTT Service Integration
Laboratories.
He received the B.E. and M.E. degrees in electrical engineering from Shibaura Institute ofTechnology, Tokyo in 1997 and 1999, respectively. Since joining NTT Service Integration
Laboratories in 1999, he has been working onperformance evaluation and management in IPnetworks and speech quality evaluation of VoIP.
He is a member of IEICE.

Research Engineer, Communication Traffic &Service Quality Project, NTT Service Integration
Laboratories.
He received the B.E. degree in electrical engineering from Tokyo University of Science,
Chiba in 2001 and the M.E. degree in electronics,
information and communication engineeringfrom Waseda University, Tokyo in 2003. Sincejoining NTT Service Integration Laboratories in2003, he has been engaged in subjective quality
assessment of multimedia telecommunications
and image coding. Currently, he is working on
the quality assessment of interactive multimodal
services over IP networks. He is a member of
IEICE.

NTT Technical Review