Comparison of MOS Evaluation Characteristics
for Chinese, Japanese, and English in IP Telephony
Zhenyu CAI, Nobuhiko KITAWAKI, Takeshi YAMADA

Shoji MAKINO

Graduate School of Systems and Information Engineering
University of Tsukuba
Tsukuba, Japan

Center for Tsukuba Advanced Research Alliance
University of Tsukuba
Tsukuba, Japan

Abstract—Communication quality in IP telephony is rated inpointed out at the ITU-T SG12 convention 2005-2007 thatthe subjective evaluation of MOS may be affected bynationality, and therefore the dependencies of language,culture, and nationality should be investigated in relation tosound, image and multimedia QoE evaluation [2]terms of the Mean Opinion Score (MOS), which is an Absolute
Category Rating (ACR) scale. There is a problem whencomparing subjectively evaluated MOSs in that the evaluation
results are strongly affected by differences in language, theinstruction words used for the evaluation, and the nationality ofthe evaluator. To solve these problems, ITU-T SG12 has started
to investigate the cultural and language dependencies ofsubjective quality evaluations undertaken with the MOS method
for speech/video/multimedia. In this paper, we present the resultsof a comparison of the MOS evaluation characteristics forChinese, Japanese, and English.

This paper describes the effect of language, instructional
index, and nationality on the opinion assessment method bycomparing the MOS characteristics of Chinese, Japanese,and English.

II.

The essential factors affecting Internet Protocol (IP)telephony are loudness, coding distortion, and packet loss
distortion. The volume can be controlled in a digital
network, but digital signal distortions should be measured
using an opinion assessment method.

Keywords: Quality of Experience (QoE), Mean Opinion Score (MOS),Language Dependency, Nationality Dependency, ITU-T Study Group 12I.

INTRODUCTION

The opinion assessment method is specified by ITU-T
Recommendation P.800. Five absolute quality assessment
categories from excellent to bad are used in the opinion
rating test. Then the MOS can be obtained from the averagevalue of all the opinion scores. However, the opinion
assessment method may be influenced by assessment
conditions such as spoken language, the translation of the
instructional index, and the nationality of the evaluator.
Therefore, for an international comparison of absolute
MOSs, we must pay careful attention to the influence ofnationality.

Communication services and networks are designed andmanaged based on the customer’s Quality of Experience(QoE), which measures the level of satisfaction using
standard measurement methodologies recommended by theInternational Telecommunication Union-Telecommunication
Sector (ITU-T). QoE is expressed as an average value ofopinion scores, namely, the Mean Opinion Score (MOS),
obtained using an opinion assessment method with 5-stage
absolute qualities [1]. But, the absolute MOS value canvary from country to country because of differences in thenationality, language, and technical background of theevaluators. Therefore, to compare the absolute MOS valuesof different countries, the MOS characteristics for eachcountry should be investigated in terms of differences inlanguage, instruction words, and nationality. In fact, it was

978-1-4244-7820-0/10/$26.00 ©2010 IEEE

PROBLEM OF OPINION SCORE COMPARISON

III.

OPINION ASSESSMENT EXPERIMENTThis section compares the MOS assessment characteristicsof Chinese, Japanese, and English under the same experimental
conditions. Section III.A provides a detailed description of the111IUCS2010

English corresponding to the 5-stage absolute quality
assessment. However, Recommendation P.800 only includes
an English instructional index. The index must betranslated into other languages before use. The translation

conditions of the opinion assessment experiment. Section III.B
compares Chinese and Japanese MOSs. Section III.C compares
those of Japanese and English, and describes the relationshipbetween those absolute MOSs. Section III.D compares
subjective and objective MOSs for Chinese obtained for
wideband coding by using the objective quality assessment
method “Wideband Perceptual Evaluation of Speech Quality(PESQ) [3] [4],” and the relative correlations in the equation are
derived from the experimental results. Finally, section III.Eshows the difference and influence of absolute MOSs indifferent areas of China.

of the words may affect the absolute MOS. “非常によい”
in Japanese may be the same as or slightly different from“excellent” in English. It is also said that Japanese has atendency to rate “非常に良い” strictly. On the other hand,
the Japanese listening quality scale shown in Table III has
been used for a long time. The instructional index forChinese is not yet fully established.

A. Experimental conditions

TABLE III.

The opinion assessment experiment was carried out
according to ITU-T Recommendation P.800. Table I lists
the experimental conditions used for Chinese, and Table IIlists the same conditions used for Japanese.
TABLE I.

：EXPERIMENTAL CONDITIONS (CHINESE)

Speech sample

2 females and 2 males

Subjects

25 (Chinese)

Experimental environment

Listen with headphones in soundproof room

Test method5-point ACR method (ITU-T P.800)

Sampling

16 kHzCoding conditions(bit rate)

G.722(64 56 48)kbps, G.722.1(32 24)kbps,G.722.2(6.6 12.6 16 20 24)kbps

Packet loss

0, 1, 3, 5, 10%

TABLE II.

：LISTENING QUALITY SCALE

Score

ChineseJapanese

English

5

非常好

非常によい

Excellent

4

良好

よいGood

3

尚可

まあよい

Fair

2

差

悪い

Poor

1

非常差

非常に悪い

Bad

B. Comparison of Chinese and Japanese MOS
Figure 1 shows the relationship between the Chinese andJapanese MOSs under the same coding conditions [6]-[7].
Their correlation coefficient is 0.903, and the root meansquare error (RMSE) is 0.645. Figure 1 shows that theChinese MOS is higher than the Japanese MOS.
5.0

：EXPERIMENTAL CONDITIONS (JAPANESE)

G.722

Speech sample

2 females and 2 males

Subjects

32 (Japanese)

Experimental environment

Listen with headphones in soundproof room

Test method5-point ACR method (ITU-T P.800)

Sampling

16 kHzCoding conditions(bit rate)

G.722(64 56 48)kbps, G.722.1(32 24)kbps,G.722.2(16 24)kbps

Packet loss

0, 1, 5, 10%

Subjective MOS(Chinese)

G.722.1
G722.2

4.0

3.0

2.0

1.0

1.0

We have already reported the MOS results of the opinion
assessment experiment for Japanese in Ref. [5] for which
the experimental conditions are given in Table II. Table III
gives the instructional index in Chinese, Japanese, and2.0

3.0

4.0

5.0

Subjective MOS(Japanese)

Figure 1. Relationship between Chinese MOS and Japanese MOS.
(Correlation coefficient is 0.903 RMSE is 0.645)

112

the English MOS. Therefore, we propose the following
reflection Equation (3) for Chinese.

Further research is necessary to determine whether thedifference depends on the instructional index or nationality.
The relationship between the Chinese MOS and theJapanese MOS is derived in Equation (1).

MOS(subjective)＝1.3843×MOS(objective)－0.2819 (3)
5.0

G.722

MOS（Chinese）＝0.7527×MOS（Japanese）＋1.219 (1)G.722.1

Figure 2 shows the Chinese MOS converted byEquation (1). The RMSE after conversion is 0.435, whichis smaller than before.

Objective MOS

5.0
G722

Subjective MOS modified（Chinese）

G.722.2

4.0

G722.13.0

2.0

G722.2

4.0

1.0
1.0

2.0

3.0

3.0

Subjective MOS

4.0

5.0

Figure 3. Relationship betweetn subjective MOS and objective MOS.
(Correlation coefficient is 0.95 RMSE is 0.75)

2.0

Figure 4 shows the Chinese objective MOS reflected byproposed Equation (3).

1.0
1.0

2.0

3.0

4.0

5.0

5.0

Subjective MOS（Japanese）

G.722

Figure 2. Relationship between Chinese MOS converted byEquation(1)and Japanese MOS.
(Correlation coefficient is 0.903 RMSE is 0.435)

G.722.1

Objective MOS modified

C. Comparison of MOS between Japanese and English
The relationship between the English MOS and theJapanese MOS is described in TTC standard JJ.201.01
conforming to the Japanese Domestic Standard [8]. Under
the same coding conditions, most English MOSs are higher
than Japanese MOSs. Their relationship is expressed byEquation (2).
MOS(Japanese)＝ 0.8681×MOS(English)＋0.0271

G.722.2

4.0

3.0

2.0

1.0
1.0

2.0

3.0
Subjective MOS

4.0

5.0

Figure 4. Relationship between subjective MOS and objective MOS
converted by Equation(3)(Correlation coefficient is 0.95 RMSE is 0.24)(2)

D. Comparison of subjective and objective MOSs inChinese

The RMSE of the Chinese objective MOS andsubjective MOS after conversion is 0.24, which is smaller
than before.

Figure 3 shows the relationship between subjective andobjective MOSs in Chinese. The Chinese objective MOS iscalculated from Wideband PESQ. In Figure 3, thehorizontal axis shows the subjective MOS and the verticalaxis shows the objective MOS. The correlation coefficientis 0.95 and the RMSE is 0.75. It can be seen that the RMSEis large. This is because the wideband PESQ is based on

E. Difference between Chinese MOSs south and northof HuaiHe RiverThe Han ethnic group accounts for 90.56% of thepopulation of China, and Chinese is the first language.

113

Under the experimental conditions shown in Section III.A,the Chinese subjects consisted of 14 people from the northregion of the HuaiHe River (ex.Beijing), and 11 people
from the south region of the HuaiHe River (ex.Shanghai).
Figure 5 compares MOSs in terms of this south and north
HuaiHe River division. There was no significant difference
between them although the South MOS was very slightly
lower than the North MOS.

“excellent” is low for Japanese. The latter result has also
been noted when comparing Japanese with English. The
reason is believed to be based on national character(namely, nationality). The instructional index must be
properly translated taking national characteristics intoconsideration.

V.

This paper compared the MOS characteristics of Chinese,Japanese and English. It should be noted that Japanese isstricter regarding quality assessment than Chinese, andconversion equations were introduced comparison of theabsolute MOSs in Chinese, Japanese, and English.

5.0
G.722
G.722.1
G722.2

Objective MOS（South）

4.0

CONCLUSIONAt the same time, a Chinese mapping function fromWideband PESQ to objectively estimated MOS wasproposed. Finally, when comparing the absolute MOSs ofdifferent counties, it is necessary to take nationality intoaccount.

3.0

2.0

REFERENCES

1.0

1.0

2.0

3.0

4.0

5.0

[1] ITU-T Recommendation P.800, “Methods fordetermination of transmission quality,” Aug. 1996.

Subjective MOS(North)Figure 5. Relationship between North of China MOS and South ofChina MOS.
(Correlation coefficient is 0.95 RMSE is 0.75)[2] ITU-T, “Report of the Meeting of Study Group 12,” TD-R17, Oct.
2007.
[3] ITU-T Recommendation P.862, “Perceptual evaluation of speech
quality (PESQ): An objective method for end to end speech
quality assessment of narrow-band telephone networks and speech,”Feb. 2001.

IV. ANALYSIS OF NATIONALITY
In section III, the Chinese MOS tended to be higher thanthe Japanese MOS. To investigate the reason for this, thissection analyzes Chinese and Japanese voting underidentical experimental conditions. Table IV shows thefrequency with which subjects voted in each category.
TABLE IV.
Quality of speech (Score)[4] ITU-T Recommendation P.862.2, “Wideband extension toRecommendation P.862 for the assessment of Wideband telephone
networks and speech codec,” Nov. 2005.
[5] N. Kitawaki, K. Nagai, T. Yamada, “Objective quality assessmentof wideband speech coding,” IEICE Trans. Comm., Vol. E88-B, No.
3, pp. 1111-1118, Mar. 2005.

： FREQUENCY OF VOTING

Chinese[6] Zhenyu CAI, N. Kitawaki, T. Yamada, “Comparision of MOS
evaluation characteristics between Chinese and Japanese, “2009
IEICE General Conference., B-11-12, March 2009 [in Japanese].

Japanese

Excellent（5）

16%

7%

Good（4）26%

19%

Fair（3）29%28%

Poor（2）22%

35%

Bad（1）

7%

11%

subjective[7] Zhenyu CAI, N. Kitawaki, T. Yamada, “Comparision of MOS
evaluation characteristics by Chinese, Japanese and English inTelephony, “IEICE Technical Committee on Communication
Quality,IEICE Technical Report CQ2009-28, pp. 17-20,September
2009 [in Japanese].
[8] TTC JJ201.01, “A Method for Speech Quality Assessment of IP
Telephony,” 4th Version, March 2007.

It can be seen from Table IV that the frequency of voting“bad” is low for Chinese, while the frequency of voting

114