ETSI EG 202 396-3 V1.7.1 (2018-03)

ETSI GUIDE

Speech and multimedia Transmission Quality (STQ);
Speech Quality performancein the presence of background noise;Part 3: Background noise transmission Objective test methods

2

ETSI EG 202 396-3 V1.7.1 (2018-03)

Reference
REG/STQ-270

Keywords
noise, QoS, quality, speech

ETSI
650 Route des Lucioles
F-06921 Sophia Antipolis Cedex - FRANCE
Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16
Siret N° 348 623 562 00017 - NAF 742 C
Association à but non lucratif enregistrée à laSous-Préfecture de Grasse (06) N° 7803/88

Important notice
The present document can be downloaded from:http://www.etsi.org/standards-search
The present document may be made available in electronic versions and/or in print. The content of any electronic and/or
print versions of the present document shall not be modified without the prior written authorization of ETSI. In case of any
existing or perceived difference in contents between such versions and/or in print, the only prevailing document is theprint of the Portable Document Format (PDF) version kept on a specific network drive within ETSI Secretariat.
Users of the present document should be aware that the document may be subject to revision or change of status.
Information on the current status of this and other ETSI documents is available athttps://portal.etsi.org/TB/ETSIDeliverableStatus.aspx
If you find errors in the present document, please send your comment to one of the following services:
https://portal.etsi.org/People/CommiteeSupportStaff.aspx

Copyright Notification
No part may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying
and microfilm except as authorized by written permission of ETSI.
The content of the PDF version shall not be modified without the written authorization of ETSI.
The copyright and the foregoing restriction extend to reproduction in all media.
© ETSI 2018.
All rights reserved.
DECTTM, PLUGTESTSTM, UMTSTM and the ETSI logo are trademarks of ETSI registered for the benefit of its Members.
3GPPTM and LTETM are trademarks of ETSI registered for the benefit of its Members andof the 3GPP Organizational Partners.
oneM2M logo is protected for the benefit of its Members.
GSM® and the GSM logo are trademarks registered and owned by the GSM Association.

ETSI

3

ETSI EG 202 396-3 V1.7.1 (2018-03)

Contents
Intellectual Property Rights ................................................................................................................................5
Foreword.............................................................................................................................................................5
Modal verbs terminology....................................................................................................................................5
1

Scope ........................................................................................................................................................62

References ................................................................................................................................................6

2.1
2.2

3
3.1
3.2

Normative references ......................................................................................................................................... 6
Informative references ........................................................................................................................................ 6

Symbols and abbreviations.......................................................................................................................8
Symbols .............................................................................................................................................................. 8
Abbreviations ..................................................................................................................................................... 8

4

Speech signals to be used .........................................................................................................................95

Selection of the data within the scope of the wideband objective model: Experts evaluation ...............10

5.1
5.2
5.3

6
6.1
6.2
6.2.1
6.2.2
6.3
6.4
6.5
6.5.1
6.5.2
6.5.3
6.6
6.6.1
6.6.2
6.6.3
6.7
6.7.1
6.7.2

7
7.1
7.2
7.2.1
7.2.2
7.2.3
7.3
7.3.0
7.3.1
7.3.2
7.3.3

8
8.0
8.1
8.2
8.3

Selection process .............................................................................................................................................. 10
Results .............................................................................................................................................................. 10
French database ................................................................................................................................................ 11

Description of the wideband objective test method ...............................................................................11
Introduction ...................................................................................................................................................... 11
Speech sample preparation and nomenclature .................................................................................................. 12
Speech sample preparation ......................................................................................................................... 12
Nomenclature .............................................................................................................................................. 15
Additional Training data .................................................................................................................................. 16
Principles of Relative Approach and Δ Relative Approach .............................................................................. 16
Objective N-MOS............................................................................................................................................. 19
Introduction................................................................................................................................................. 19
Description of N-MOS algorithm ............................................................................................................... 20
Comparing subjective and objective N-MOS results .................................................................................. 23
Objective S-MOS ............................................................................................................................................. 24
Introduction................................................................................................................................................. 24
Description of S-MOS Algorithm ............................................................................................................... 25
Comparing Subjective and Objective S-MOS Results ................................................................................ 28
Objective G-MOS............................................................................................................................................. 29
Description of G-MOS Algorithm .............................................................................................................. 29
Comparing subjective and objective G-MOS results .................................................................................. 30

Validation of the Wideband Objective Test Method ..............................................................................31
Introduction ...................................................................................................................................................... 31
ETSI EG 202 396-2 Database Results Analysis ............................................................................................... 33
Comparing subjective and objective N-MOS results .................................................................................. 33
Comparing subjective and objective S-MOS results .................................................................................. 33
Comparing Subjective and Objective G-MOS Results ............................................................................... 34
Orange Validation Database results Analysed ................................................................................................. 35
Introduction................................................................................................................................................. 35
Comparing subjective and objective N-MOS results .................................................................................. 35
Comparing subjective and objective S-MOS results .................................................................................. 35
Comparing Subjective and Objective G-MOS Results ............................................................................... 36

Objective Model for Narrowband Applications .....................................................................................37
Introduction ...................................................................................................................................................... 37
File pre-processing ........................................................................................................................................... 37
Adaptation of the Calculations ......................................................................................................................... 38
Prediction results .............................................................................................................................................. 38

Annex A:

Detailed post evaluation of listening test results ................................................................40

ETSI

4

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex B:

Results of PESQ and TOSQA2001 - Analysis of ETSI EG 202 396-2 database ..............43

Annex C:

Comparison of objective MOS versus auditory MOS for the complete STF 294
database .................................................................................................................................50

Annex D:

Comparison of objective MOS versus auditory MOS for rejected conditions ................52

Annex E:

Void ........................................................................................................................................54

Annex F:

Detailed STF 294 subjective and objective validation test results ....................................55

Annex G:

Void ........................................................................................................................................58

Annex H:

Extension of the Speech Quality Test Method to Narrowband: Adaptation,
Training and Validation .......................................................................................................59

Annex I:

Void ........................................................................................................................................61

Annex J:

Summary of Czech samples not used for model training ..................................................62

J.0

Introduction ............................................................................................................................................62

J.1

Selection process - Czech database ........................................................................................................62

J.2

General differences between the databases ............................................................................................64

J.3

Comparison of the objective method results for Czech and French samples .........................................67

J.4

Czech conditions results analysis ...........................................................................................................72

J.4.1
J.4.2
J.4.3

Comparing subjective and objective N-MOS results ....................................................................................... 72
Comparing subjective and objective S-MOS results ........................................................................................ 72
Comparing Subjective and Objective G-MOS Results..................................................................................... 73

J.5

Language Dependent Robustness of G-MOS.........................................................................................74

J.6

Regression Coefficients for Czech data .................................................................................................75

J.7

Post selection ..........................................................................................................................................76

Annex K:

Relative Approach Non-Linear Transformation ...............................................................80

Annex L:

Bibliography ..........................................................................................................................81

History ..............................................................................................................................................................82

ETSI

5

ETSI EG 202 396-3 V1.7.1 (2018-03)

Intellectual Property Rights
Essential patents
IPRs essential or potentially essential to normative deliverables may have been declared to ETSI. The information
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be foundin ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web
server (https://ipr.etsi.org/).
Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web
server) which are, or may be, or may become, essential to the present document.
Trademarks
The present document may include trademarks and/or tradenames which are asserted and/or registered by their owners.
ETSI claims no ownership of these except for any which are indicated as being the property of ETSI, and conveys noright to use or reproduce any trademark and/or tradename. Mention of those trademarks in the present document does
not constitute an endorsement by ETSI of products, services or organizations associated with those trademarks.

Foreword
This ETSI Guide (EG) has been produced by ETSI Technical Committee Speech and multimedia Transmission Quality(STQ).
The present document is a deliverable of ETSI Specialized Task Force (STF) 294 entitled: "Improving the quality ofeEurope wideband speech applications by developing a performance testing and evaluation methodology forbackground noise transmission".
The present document is part 3 of a multi-part deliverable covering Speech and multimedia Transmission Quality(STQ); Speech Quality performance in the presence of background noise, as identified below:
Part 1:"Background noise simulation technique and background noise database";

Part 2:"Background noise transmission - Network simulation - Subjective test database and results";

Part 3:"Background noise transmission - Objective test methods".

Modal verbs terminology
In the present document "should", "should not", "may", "need not", "will", "will not", "can" and "cannot" are to beinterpreted as described in clause 3.2 of the ETSI Drafting Rules (Verbal forms for the expression of provisions).
"must" and "must not" are NOT allowed in ETSI deliverables except when used in direct citation.

ETSI

6

1

ETSI EG 202 396-3 V1.7.1 (2018-03)

Scope

The present document aims to identify and define testing methodologies which can be used to objectively evaluate theperformance of narrowband and wideband terminals and systems for speech communication in the presence of
background noise.
Background noise is a problem in mostly all situations and conditions and need to be taken into account in both,terminals and networks. The present document provides information about the testing methods applicable to objectively
evaluate the speech quality in the presence of background noise. The present document includes:•

The description of the experts post evaluation process chosen to select the subjective test data being within thescope of the objective methods.

•

The results of the performance evaluation of the currently existing methods described in Recommendations
ITU-T P.862 [i.16] and P.862.1 [i.17] and in TOSQA2001 [i.19] which is chosen for the evaluation ofterminals in the framework of ETSI VoIP speech quality test events [i.8], [i.9], [i.10] and [i.11].

•

The method which is applicable to objectively determine the different parameters influencing the speech
quality in the presence of background noise taking into account:•-the speech quality;-

the background noise transmission quality;-

the overall quality.

The present document is to be used in conjunction with:-ETSI ES 202 396-1 [i.1] which describes a recording and reproduction setup for realistic simulation of
background noise scenarios in lab-type environments for the performance evaluation of terminals andcommunication systems.

-

ETSI EG 202 396-2 [i.2] which describes the simulation of network impairments and how to simulate
realistic transmission network scenarios and which contains the methodology and results of thesubjective scoring for the data forming the basis of the present document.

-

French speech sentences as defined in Recommendation ITU-T P.501 [i.13] for wideband and English
speech sentences as defined in Recommendation ITU-T P.501 [i.13] for narrowband.

2

References

2.1

Normative references

Normative references are not applicable in the present document.

2.2

Informative references

References are either specific (identified by date of publication and/or edition number or version number) or
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of thereferenced document (including any amendments) applies.
NOTE:

While any hyperlinks included in this clause were valid at the time of publication, ETSI cannot guarantee
their long term validity.

The following referenced documents are not necessary for the application of the present document but they assist theuser with regard to a particular subject area.
[i.1]

ETSI ES 202 396-1: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise; Part 1: Background noise simulation techniqueand background noise database".

ETSI

7

ETSI EG 202 396-3 V1.7.1 (2018-03)[i.2]

ETSI EG 202 396-2: "Speech Processing, Transmission and Quality Aspects (STQ); Speech
Quality performance in the presence of background noise; Part 2: Background Noise Transmission- Network Simulation - Subjective Test Database and Results".

[i.3]

Recommendation ITU-T P.835: "Subjective test methodology for evaluating speechcommunication systems that include noise suppression algorithm".

[i.4]

Recommendation ITU-T P.800: "Methods for subjective determination of transmission quality".

[i.5]

Recommendation ITU-T P.831: "Subjective performance evaluation of network echo cancellers".

[i.6]

Genuit, K.: "Objective Evaluation of Acoustic Quality Based on a Relative Approach", InterNoise'96, Liverpool, UK.

[i.7]

Recommendation ITU-T SG 12 Contribution 34: "Evaluation of the quality of background noisetransmission using the "Relative Approach"".

[i.8]

ETSI 2nd Speech Quality Test Event: "Anonymized Test Report", ETSI Plugtests, HEAD
acoustics, T-Systems Nova.

NOTE:

Available at http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx. Also available as ETSITR 102 648-3.

[i.9]
NOTE:

ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Gateways".
Available at http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.

[i.10]ETSI 3rd Speech Quality Test Event: "Anonymized Test Report "IP Phones".

[i.11]ETSI 4th Speech Quality Test Event: "Anonymized Test Report "IP Gateways and IP Phones".

NOTE:

Available at http://www.etsi.org/WebSite/OurServices/Plugtests/History.aspx.

[i.12]

F. Kettler, H.W. Gierlich, F. Rosenberger: "Application of the Relative Approach to Optimize
Packet Loss Concealment Implementations", DAGA, March 2003, Aachen, Germany.

[i.13]

Recommendation ITU-T P.501: "Test Signals for Use in Telephonometry".

[i.14]

R. Sottek, K. Genuit: "Models of Signal Processing in human hearing", International Journal of
Electronics and Communications (AEÜ) volume 59, 2005, p. 157-165.

NOTE:

Available at http://www.elsevier.de/aeue.

[i.15]SAE International - Document 2005-01-2513: "Tools and Methods for Product Sound Design ofVehicles" R. Sottek, W. Krebber, G. Stanley.

[i.16]

Recommendation ITU-T P.862: "Perceptual evaluation of speech quality (PESQ): An objective
method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs".

[i.17]

Recommendation ITU-T P.862.1: "Mapping function for transforming P.862 raw result scores to
MOS-LQO".

[i.18]

Recommendation ITU-T P.862.2: "Wideband extension to Recommendation P.862 for theassessment of wideband telephone networks and speech codecs".

[i.19]

Recommendation ITU-T SG 12 Contribution 19: "Results of objective speech quality assessmentof wideband speech using the Advanced TOSQA2001".

[i.20]

Recommendation ITU-T G.722: "7 kHz audio-coding within 64 kbit/s".

[i.21]

Recommendation ITU-T G.722.2: "Wideband coding of speech at around 16 kbit/s using Adaptive
Multi-Rate Wideband (AMR-WB)".

[i.22]

Recommendation ITU-T P.56: "Objective measurement of active speech level".

[i.23]

Recommendation ITU-T P.57: "Artificial ears".

ETSI

8

ETSI EG 202 396-3 V1.7.1 (2018-03)[i.24]

M. Spiegel: "Theory and problems of statistics", McGraw Hill, 1998.

[i.25]

Void.

[i.26]

M. Kendall: "Rank correlation methods", Charles Griffin & Company Limited, 1948.

[i.27]

Sottek, R.: "Modelle zur Signalverarbeitung im menschlichen Gehör", PHD thesis RWTH Aachen,
1993.

[i.28]

Recommendation ITU-T P.830: "Subjective performance assessment of telephone-band andwideband digital codecs".

[i.29]

Void.

[i.30]ANSI S1.1-1986 (ASA 65-1986): "Specifications for Octave-Band and Fractional-Octave-Band
Analog and Digital Filters", 1993.

[i.31]

Recommendation ITU-T G.160 Appendix II, Amendment 2: "Voice enhancement devices:
Revised Appendix II - Objective measures for the characterization of the basic functioning ofnoise reduction algorithms".

[i.32]ETSI TS 103 106: "Speech and multimedia Transmission Quality (STQ); Speech quality
performance in the presence of background noise: Background noise transmission for mobile
terminals-objective test methods".

[i.33]

Hastie T.; Tibshirani R. and Friedman J.: "The Elements of Statistical Learning: Data Mining,Inference, and Prediction", New York: Springer-Verlag, 2001.

[i.34]ETSI EG 202 396-3 (V1.1.1 to V1.3.1): "Speech Processing, Transmission and Quality Aspects(STQ); Speech Quality performance in the presence of background noise; Part 3: Backgroundnoise transmission - Objective test methods".

3

Symbols and abbreviations

3.1

Symbols

For the purposes of the present document, the following symbols apply:
σ2

3.2

Variance

Abbreviations

For the purposes of the present document, the following abbreviations apply:
AMR
ASL

Adaptive MultiRate
Active Speech Level

NOTE:

According to Recommendation ITU-T P.56 [i.22].

BGN
CDF
dB SPL
DB
DUT
EFR
FR
G-MOS

BackGround Noise
Cumulative Density Function
Sound Pressure Level re 20 µPa in dB
Data Base
Device Under Test
Enhance Full Rate
Full Rate
Global MOS

NOTE:
GSM

MOS related to the overall sample.
Global System for Mobile Communication

ETSI

9

HATS
HiQ
IP
IRS
ITU
ITU-T
LQ
MMSE
MOS
MOS-LQSN
MRP
NB
NBGN
NI
NII
NIII
N-MOS
NOTE:
NR
NR (filter)
NSA
PESQ
PLC
RCV
RMS
RMSE
SG
S-MOS
NOTE:
SND
SNR
SQTE
SPL
STD
STF
TMOS
TOR
VAD
VoIP
WB

4

ETSI EG 202 396-3 V1.7.1 (2018-03)

Head And Torso Simulator
High Quality (codec mode)
Internet Protocol
Intermediate Reference System
International Telecommunication Union
Telecom Standardization Body of ITU
Low Quality (codec mode)
Minimum Mean Square Error
Mean Opinion Score
Mean Opinion Score - Listening Quality Subjective Noise
Mouth Reference Point
NarrowBand
Level of background noise
Network I conditions
Network II conditions
Network III conditions
Noise MOS

MOS related to the noise transmission only.
Noise Reduction
Noise Reduction (filter)
Noise Suppression Algorithm
Perceptual Evaluation of Speech Quality
Packet Loss Concealment
ReCeiVe
Root Mean Square
Random Mean Square Error
Study Group
Speech MOS
MOS related to the speech signal only.
Sending Direction
Signal to Noise Ratio
Speech Quality Test EventSound Pressure Level
STandard Deviation
Specialized Task ForceTOSQA Mean Opinion Score
Terms Of Reference
Voice Activity Detection
Voice over IP
WideBandSpeech signals to be used

As with any objective model, the prediction of speech quality depends on the conditions under which the model wastested and validated (see clauses 6.1 and 8). This dependency also applies to the speech material used in conjunction
with the objective model.
The wideband version of the model uses French speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can betaken from Recommendation ITU-T P.501 [i.13].
The narrowband version of the model uses English speech sentences. The near end speech signal (clean speech signal)
consists of 8 sentences of speech (2 male and 2 female talkers, 2 sentences each). Appropriate speech samples can betaken from Recommendation ITU-T P.501 [i.13].

ETSI

10

ETSI EG 202 396-3 V1.7.1 (2018-03)

5

Selection of the data within the scope of the
wideband objective model: Experts evaluation

5.1

Selection process

The aim of the selection process was to identify those data in the databases described in ETSI EG 202 396-2 [i.2] which
are consistent with the scope of the objective models to be studied within the present document.
The experts were selected on the based on the definition found in e.g. Recommendation ITU-T P.831 [i.5]: experts areexperienced in subjective testing. Experts are able to describe an auditory event in detail and are able to separate
different events based on specific impairments. They are able to describe their subjective impressions in detail. They
have a background in technical implementations of noise reduction systems and transmission impairments and do havedetailed knowledge of the influence of particular implementations on subjective quality.
Their task was to select the relevant conditions within the scope of the model to be developed. Therefore they had toverify the consistency of the data with respect to the following selection criteria:
1)

Artefacts others than the ones which should have been produced by the signal processing described in ETSIEG 202 396-2 [i.2] e.g. due to the additional amplification required in order to provide a listening level of
79 dB SPL.

2)

Inconsistencies within one condition due to the selection of the individual speech samples from the databasefor subjective evaluation.

3)

Inconsistencies within one condition due to statistical variation of the signal processing described in ETSIEG 202 396-2 [i.2] leading to non consistent judgements within this condition.

4)

Inconsistencies due to Recommendation ITU-T P.56 [i.22] level adjustment process chosen for the complete
files including the background noise.

As a result of the experts listening test a set of data was selected which is used for the development of the objectivemodel.
In the selection process five expert listeners (non-native French speakers) were involved. Their task was not to produce
new judgements, but to check all the samples in the database with respect to the possible artefacts described above.
A playback system with calibrated headphones was used for the test. The equalization provided by the headphone
manufacturer was used since this was the one used in the auditory French test setup.
NOTE:

These headphones and headphone amplifiers were used in the tests since they provide the performancerequired. Other products providing the equivalent performance could be used if such an experimentshould be repeated by others. This information is given for the convenience of users of the present
document and does not constitute an endorsement by ETSI of these products.

All samples could be heard by the experts as often as required in order to get final agreement about the applicability ofthe data within the terms of reference of the model. There was no limitation in comparing samples to the ones
previously heard.

5.2

Results

In general it could be observed that the 4 seconds sample size chosen in the experiment according to Recommendation
ITU-T P.835 [i.3] lead to a more difficult task even for expert listeners, especially in the case of non-stationary
background noises. It is more difficult to identify the nature of the noise itself and then identify in addition possible
impairments introduced by the signal processing or by the network impairments. It is very likely that somecomparatively high standard deviations seen in the data are caused by these effects.

ETSI

11

5.3

ETSI EG 202 396-3 V1.7.1 (2018-03)

French database

In general the French database is in line with the ToR except network condition NII. In network condition NII 1 %packet loss was chosen which is too low for the conditions to be evaluated. Due to the inhomogeneously distributed
packet losses there are conditions where no packet loss is audible up to conditions where 5 out of 6 samples show
packet loss. Furthermore the packet loss may occur during speech as well as during the noise periods. The impact of thedifferent packet losses is not controlled with respect to their occurrence due to the statistical nature of the packet loss
distribution, even within a set of 6 samples used for evaluating one condition. Since packet loss is clearly audible under
NIII conditions (3 % packet loss) and much better distributed amongst the different samples the NII conditions are notused within the scope of the objective method. They are either covered by the NI condition (0 % packet loss) or by the
NIII conditions. This results in 144 NII conditions which are not retained for the development of the model.
From the 288 NI and NIII conditions 28 conditions are not retained. The main reasons therefore are:•Not consistent signal levels due to the amplification process.

•Insufficient S/N, speech almost inaudible.

The individual reasons for the samples of these conditions being not retained can be found in table A.1.
In total 260 out of 432 conditions are used as the reference for the objective model. In other words, 60,2 % of the data
can be used for the model. The distribution of the ratings is between 1,2 and 4,96 MOS for S-/N-/G-MOS.

6

Description of the wideband objective test method

6.1

Introduction

The present objective test method is developed in order to calculate objective MOS for speech, noise and the overall
quality of a transmitted signal containing speech and background noise, designated N-MOS, S-MOS and G-MOS in thefollowing.
The new model is based on an aurally-adequate analysis in order to best cover the listener's perception based on thepreviously carried out listening test ETSI EG 202 396-2 [i.2].
The wideband objective model is applicable for:•wideband handset and wideband hands-free devices (in sending direction);•noisy environments (stationary or non-stationary noise);•different noise reduction algorithms;•

AMR Recommendation ITU-T G.722.2 [i.21] and Recommendation ITU-T G.722 [i.20] wideband coders;•

VoIP networks introducing packet loss.

NOTE 1: For the NIII conditions jitter was introduced. Finally jitter was observed for less than 2 % of the selectedconditions. The jitter consideration of the new objective method could therefore not be validated on an
appropriate amount of data. Quality impairments typically introduced by different strategies of packet
loss concealment and different adaptive jitter buffer control mechanisms were not considered in thelistening test database and therefore also not in the objective method.
NOTE 2: The method is not applicable for such background situations where speech intelligibility is the major
issue.
Due to the special sample generation process the new method is only applicable for electrically recorded signals. The
quality of terminals can therefore only be determined in sending direction.
The method was developed by attaching importance to a high reliability. The results of the listening test (selectedconditions, see clause 5) were best modelled. Furthermore mechanisms were implemented to provide high robustness
also for other than the present samples.

ETSI

12

ETSI EG 202 396-3 V1.7.1 (2018-03)

The sample preparation and nomenclatures for the new method are described in clause 6.2.
The calculation of N-MOS, S-MOS and G-MOS is described in detail in clauses 6.5 to 6.7.

6.2

Speech sample preparation and nomenclature

6.2.1

Speech sample preparation

Based on the data selected in clause 5 an objective model is developed in order to determine:•

the Noise-MOS (N-MOS);•

the Speech-MOS (S-MOS); and•the "Global"-MOS (G-MOS), the overall quality including speech and background noise.

Different input signals can be accessed during the recording process and subsequently can be used for the calculation of
N-MOS, S-MOS and G-MOS. Beside the signals used in the listening test ("processed signal"), two additional signals
are used as a priori knowledge for the calculation:
1)

The "clean speech" signal, which was played back via the artificial mouth at the beginning of the sample
generation process.

2)

The "unprocessed signal", which was recorded close to the microphone position of the simulated handset
device/hands-free telephone (see figure 6.1 and ETSI EG 202 396-2 [i.2]). Note that no real phone/hands-freedevice was used. Phones and handsfree devices were simulated by a free-field microphone and an offline
simulation for filtering, VAD, noise reduction, etc.

Both signals are used in order to determine the degradation of speech and background noise due to the signal processing
as the listeners did during the listening tests.
The sample generation process is shown in figure 6.1.

ETSI

13

ETSI EG 202 396-3 V1.7.1 (2018-03)

NOTE 1: Calibrated for each file with B&K HATS (3.3 ears) to 79 dB SPL ASL (Recommendation ITU-T P.56 [i.22]).
NOTE 2: Once calibrated: -26 dBoV resulting to 79 dB SPL measured with a type 3.2 ear (Recommendation ITU-T P.57 [i.23]), 5N application force.

Figure 6.1: Sample generation process, indicating "clean speech", "unprocessed speech" and "processed speech"

ETSI

14

ETSI EG 202 396-3 V1.7.1 (2018-03)

The processed signal consists of the unprocessed signal after being processed via noise reduction algorithms, voicecoder, network simulation, etc. This signal was subjectively rated in the previously carried out listening test (see ETSIEG 202 396-2 [i.2] and figure 6.1).
In order to calculate S-MOS, N-MOS and G-MOS, all three signals are required for each sample. The a priori signals(clean speech and unprocessed) were extracted for each processed signal used in the listening tests.
The following preparation steps are required to be carried out for all three files:
1)

The clean and unprocessed speech signals were shortened to 4 seconds in order to match the length of theprocessed signal in the listening tests.

2)

The signals were time-aligned. This was achieved after pre-processing followed by a cross-correlation
analysis.

NOTE 1: For samples with an instationary background noise or including packet loss and jitter it should be ensured
that the cross-correlation analyses lead to non-ambiguous results. E.g. by applying further processing
algorithms in order to better separate between speech and noise parts.
Due to time alignment, several parts in signals may be obtained, where no corresponding part exists in the other signals.
Thus these segments are discarded. Figure 6.1a illustrates the strategy of signal cropping after time alignment.

Figure 6.1a: Signal alignment
For some of the following calculations, the information about speech and noise-only parts is needed. After the timealignment between the three signals as described above, the clean speech signal is segmented into frames and classified
according to Recommendation ITU-T G.160 [i.31]. The method described in [i.31] performs a frame categorization on
the clean input signal (see section II.4.1 in Recommendation ITU-T G.160 [i.31]). It first transforms the signal into alevel-vs-time transformation based on 10 ms frames. Each frame is then categorized as either silence, pause, uncertain,
low/mid/high speech activity. The signal parts classified as silence are assumed as background noise/silence sections forunprocessed and processed signal. All other frames are considered as active speech.
For the recording procedure, the clean speech signals are expected to have an Active Speech Level (ASL, see
Recommendation ITU-T P.56 [i.22]) of -4,7 dB Pa at the mouth reference point (MRP). Additional level increments
may be added for compensating Lombard effect (typically +3 dB), i.e. obtaining a more realistic signal-to-noise ratio.
For the instrumental prediction method, all three input signals are scaled to an active speech level of either 73 dB SPL(narrowband mode) respectively 79 dB SPL (wideband mode). These levels correspond to the scaling used in theunderlying listening test databases.
NOTE 2: The unprocessed signal and also the processed signal as well may include too much noise for the proper
calculation of active speech level according to Recommendation ITU-T P.56 [i.22]. In this case, the levelof the noisy speech is calculated via the speech part detection previously described.
NOTE 3: Speech level calculations are carried out over speech including noise. The more noise is present in theprocessed or unprocessed signal, the less speech-only energy contributes to the overall level. In borderline
cases this may result in an unreasonable biased estimate of speech-only level only, but this method
corresponds to the level calibration used in the auditory experiments.

ETSI

15

6.2.2

ETSI EG 202 396-3 V1.7.1 (2018-03)

Nomenclature

In order to provide a consistent nomenclature within the present document, the relevant terms are briefly described
below.
The combination of speech sequences, a background noise, a phone type and simulation (filtering, NR level andaggressiveness), a speech codec and a network scenario leads to one condition in the terms of the present document andETSI EG 202 396-2 [i.2].
Each condition was generated by processing the clean speech file containing eight sentences per language via thecorresponding scenario, see figure 6.2.

Clean speech file of 8 sentences

Frenc hlistening
test

Unprocessed
speech file(phone simulation ,codec , network )Cz echlisteningtes t

1 test condition

4 listeners
per sentence
24 percondition
24 listeners
per sentence
and per
condition

Processed speech file of 8 sentences;
6 French and 1 Czech sentences are
selected for listening test

Figure 6.2: Nomenclature (file, condition, sentence)
For the listening tests different parts of the resulting processed files were used. Six of the French sentences percondition were chosen and assessed by 4 persons each. The resulting auditory S-/N-/G-MOS per sentence wereaveraged to the condition MOS.
The consecutively described algorithms calculate the S-/N-/G-MOS sentence-wise. For the French database the MOS
scores for one condition were calculated based on 6 sentences. Beside the processed signal p(k) also the a priori signals(clean speech c(k) and unprocessed u(k)) are necessary (see figure 6.1). The bundle of those three signals for one
sentence is called a sample in the following, see figure 6.3.

Figure 6.3: Nomenclature (sample)
All calculations in the following clauses 6.5 to 6.7 are always based on single sentences. The calculated objective MOS
values of one condition are averaged to one objective condition MOS value. Comparisons with subjective MOS values
are never conducted on a per-sample basis, only per-condition analyses are performed.

ETSI

16

ETSI EG 202 396-3 V1.7.1 (2018-03)

The present database contains 179 (French) conditions which were selected according to clause 4. Their S-/N-/G-MOS
values were known during the development phase of the model.

6.3

Additional Training data

In order to enlarge the training database regarding amount of conditions and real devices (the original work of ETSIEG 202 396-2 [i.2] only included simulated terminals), Orange kindly provided audio files and subjective results of a
new auditory test. This new database was used for the development of ETSI TS 103 106 [i.32]. The database consists of
90 conditions with 12 sentences of 6 different talkers (3 male/3 female), including the talkers presented in theexperiments in ETSI EG 202 396-2 [i.2].
The focus of this additional database concentrates on state-of-the-art mobile devices (year 2012) in handset mode. Since
the database in the original work ETSI EG 202 396-2 [i.2] also included many hands-free conditions, the bias between
both datasets are different. All S-/N-/G-MOS values were known during the development phase of the model.
The overall training dataset then includes 179 + 90 = 269 conditions.

6.4

Principles of Relative Approach and Δ Relative Approach

The Relative Approach [i.6] is an analysis method developed to model a major characteristic of human hearing. This
characteristic is the much stronger subjective response to distinct patterns (tones and/or relatively rapid time-varying
structure) than to slowly changing levels and loudnesses.

Figure 6.4: Block diagram of Relative Approach

ETSI

17

ETSI EG 202 396-3 V1.7.1 (2018-03)

The idea behind the Relative Approach analysis is based on the assumption that human hearing creates a continuous
reference sound (an "anchor signal") for its automatic recognition process against which it classifies tonal or temporal
pattern information moment-by-moment. It evaluates the difference between the instantaneous patterns in both time andfrequency. In evaluating the acoustic quality of a complex "patterned" signal, the absolute level or loudness is almost
without any significance. Temporal structures and spectral patterns are important factors in deciding whether a sound isjudged as annoying or disturbing (see also [i.12], [i.14], [i.15] and [i.27]).
Similar to human hearing and in contrast to other analysis methods the Relative Approach algorithm does not require
any reference signal for the calculation. Only the signal under test is analyzed. Comparable to the human experienceand expectation, the algorithm generates an "internal reference" which can be best described as a forward estimation.
The Relative Approach algorithm objectifies pattern(s) in accordance with human perception by resolving or extracting
them while largely rejecting pseudo-stationary energy. At the same time, it considers the context of the relative
difference of the "patterned" and "non-patterned" magnitudes.
Figure 6.4 shows a block diagram of the Relative Approach. The time-dependent spectral pre-processing can either bedone by a filter bank analysis according to ANSI S1.1-1986 [i.30] or a spectral analysis based on a hearing model [i.27].
Both of them result in a spectral representation versus time. All calculations described in the following are based on the
non-squared, but absolute magnitude of this spectrogram. All time-frequency bins are regarded in the physical unit [Pa](not [Pa]^2).
The Relative Approach takes the absolute signal level into account. Therefore, the input data is calibrated to a realistic
listening level and the physical unit of the input signals is pressure in Pascal (Pa). As input for either the filter bank or
the Hearing Model signals adjusted to 79 dB SPL can be used (e.g. according to the French listening test) or signals
with their original level after signal processing (e.g. according to the Czech listening test).
In the calculation, two regression/smoothing modes can be applied to the pre-processed signals (see figure 6.5), once
versus time or versus frequency bands.
The estimation of the current magnitude ௧ within a certain frequency band is calculated via a linear regression from atime window of the past 200 ms (see figure 6.5 a)).
The estimation of the current magnitude ௙ within a certain time slot is calculated via a linear regression fromneighbouring frequency bands (see figure 6.5 b)); 8 frequency bands above and 8 below are used for this calculation.
For each time slot, this principle results into a sliding window of 17 frequency bands, including the current one.

Figure 6.5: Regression modes of Relative Approach:a) versus time, b) versus frequency
Another calculation method is the non-linear transformation according to the hearing model of Sottek [i.27]. No further
hearing threshold or other spectral weighting is used. Due to the non-linear relationship between sound pressure andperceived loudness, the term "compressed pressure" in compressed Pascal (cPa) is used here as the physical unit of theoutput signal. Figure 6.6 compares this transformation against the transformation to a common sound pressure level(SPL).

ETSI18

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure 6.6: Non-linear transformation of sound pressure according to hearing model of Sottek
For the first branch of the Relative Approach, a regression versus time is applied for each frequency band (seefigure 6.5 a)). Afterwards, a smoothing versus frequency is performed for each time slot (see figure 6.5 b)), which gives
the intermediate result g1 (see figure 6.4).
Its output is subtracted from the source signal which is transformed in the same way (result f1, figure 6.4). Finally, allnegative and non-relevant components (for human hearing) are set to zero by a threshold (experientially determined to
0,53 cPa). This variant focuses on the detection of tonal components.
The second variant first smoothes versus frequency within a time slot and then applies the regression versus time. This
output signal is transformed non-linearly through the hearing model and leads to the intermediate result g2. It is
compared to the result f2, which is determined as the output of the smoothing versus frequency only. Again all negativeand non-relevant components are set to zero. Thus more transient structures are detected by this branch.
At this point, two representations are available. The final result of the Relative Approach is then a weighted sum of
both. The positive weights λ1 and λ2 for each representation are chosen so that λ1 + λ2 = 1. In general, the factors λ1 andλ2 are used to describe the weighting of the Relative Approach for tonal and transient signals.
The result of the Relative Approach analysis is typically a 3D spectrogram displaying the deviation from the "close to
the human expectation" between the estimated and the current signal. Because of the non-linear transformation, thephysical unit of the magnitude remains cPa.
In order to adopt the Relative Approach for speech analysis, the following simplifications were applied:•

For our new objective speech quality models, λ1 = 0 and λ2 = 1 was chosen. Thus, the model is tuned to detect
time-variant transient structures.

•

To reduce computational complexity, a 1/12th octave filter bank representation according to ANSI
S1.1-1986 [i.30] is used as the input of the algorithm. The filterbank may compensate for the delays introducedby the filters, but it is not required. Compensation, or non-compensation, will have no significant effect on the
Relative Approach results. (Filter delay is most significant at low frequencies which are not used for the
analysis.).

Currently the Relative Approach uses a time resolution of Δt = 6,66 ms (150 blocks per second). Depending on the levelcalculation of the preceding filter bank, the resulting representation may provide more samples per second. Thus theoutput block size is achieved by applying a maximum filter over all samples within one frame.
The frequency range from 15 Hz to 24 kHz is divided into 128 frequency bands Δfm which corresponds to a 1/12th
octave resolution. Due to the nonlinearity in the relationship between sound pressure and perceived loudness, the term"compressed pressure" in compressed Pascal (cPa) is used to describe the result of applying the nonlinear transform.
A detailed explanation of the non-linear transformation used by the Relative Approach is given in annex K.

ETSI

19

ETSI EG 202 396-3 V1.7.1 (2018-03)

The N-MOS (and also the S-MOS) calculation of the present objective model is based on the Relative Approach. Due tothe time variant characteristic of speech and most of the background noise signals, the 3D Relative Approach
spectrogram always shows a deviation between the expected and the current signal which is indicated by patterns in thetime-variant signal. A first attempt using Relative Approach for analysing time variant background noises was
submitted as a contribution in Recommendation ITU-T SG 12 [i.7]. For time variant signals this "estimation error" canbest be interpreted as the "attention" which is attracted by the patterns of the particular signal on human perception. The
3D spectrogram of a time variant signal therefore provides some information for the N-MOS (and also S-MOS)
determination. But it needs additionally be considered what humans expect if they think of a "good" sound quality fortime variant background noise and speech signals. The unprocessed signal and the clean speech signal respectively (seeclause 6.2) can be seen as such a "good quality reference". The knowledge about "good" or "poor" quality is not yet
covered by Relative Approach. Relative Approach can only determine how "close to the human expectation" a signal is,but not if this expectation is of a high or a low quality origin.
The 3D Relative Approach spectrogram is therefore calculated for the processed as well as for the unprocessed signal.
Both spectrograms are then subtracted from each other in order to determine what has changed due to the transmission.
This differential analysis, the Δ Relative Approach, between the transmitted processed signal and the undisturbed
unprocessed signal provides the information how "close to the human expectation" the processed signal still iscompared to the unprocessed signal. The calculation is carried out using equation 6.1.

ΔRA( Δti , Δf j ) = RAp ( Δti , Δf j ) − RAu ( Δti , Δf j )(6.1)∀ Δti , Δf j within Δfmin ≤ Δfj ≤ Δfmax ,Δti = 6,66 ms between tmin and tmax given by the beginning and the end of the sample.
An undisturbed transmission would lead to a homogeneous differential spectrogram indicating a "close to the original"transmission. A transmission leading to highly modulated background noises will result to an inhomogeneous
differential spectrogram showing distinct patterns (time and frequency wise). They are caused by the signal processing
during the transmission and raise compared to the original, unprocessed signal. They are aurally-adequate detected by
the Δ Relative Approach. Those kinds of transmissions typically lead to a low N-MOS.
The Δ Relative Approach analysis was already successfully applied during the 4th SQTE [i.11] for VoIP transmission
evaluating "transparency" of background noise transmission influenced, e.g. by VAD or comfort noise.

6.5

Objective N-MOS

6.5.1

Introduction

The N-MOS calculation is based on three principles:1)

Choice of a hearing-adequate analysis in order to reproduce human perception.

2)

Tuning to the database in order to provide in a high correlation between auditory and objective N-MOS.

3)

Ensure robustness for scenarios outside the database.

The objective N-MOS algorithm is based on the results of the subjective listening test and conclusions drawn from theconsecutive expert listening analysis. Expert analysis led the extraction of the main parameters leading to the subjective
N-MOS:•

Absolute background noise level.

•

Modulation of background noise, e.g. musical tones.

•"Naturalness" of the background noise.

•

Lost packets (minor influence).

ETSI

20

6.5.2

ETSI EG 202 396-3 V1.7.1 (2018-03)

Description of N-MOS algorithm

The aim of the N-MOS calculation is to reproduce the relevant parameters influencing subject's assessment by atechnically analysis. These parameters are the absolute level, disturbing "modulations" and the "naturalness" as derivedrdby the experts listening test. Simple analyses like A-weighted sound pressure level, 3 octave analyses and also evenmost of the known psychoacoustic analyses were not capable to fully describe human listening perception in suchcomplex listening situations. Besides level analyses, an analysis which is capable to adequately analyse the acoustic
quality as typically perceived by humans is the Relative Approach [i.8], an aurally-adequate analysis.
The N-MOS is calculated as shown in figure 6.5. Scalar signal paths are shown with thin solid lines, vector signals areshown with dashed lines and 3D spectrograms are given with thick solid lines. Note that in advance of the N-MOS
calculation the pre-processing steps described in clause 6.2 have to be carried out.
The N-MOS is calculated on basis of the Relative Approach and the absolute level of the processed background noise.
High background noise levels were typically judged with low N-MOS in the listening test. This background noise level
NBGN is calculated for those sections of the processed signal p(k) which contain only background noise and no speech.
The clean speech signal c(k) is used as a mask in order to determine the beginning and end of these sections.
The level NBGN is then calculated in dB Pa for the extracted background noise sections in the processed signal pBGN(k)by using equations 6.2 and 6.3. The French subjects listened to the signal p(k), which was adjusted to an acoustic levelof 79 dB SPL active speech level. The level NBGN is therefore also calculated as an acoustics level. 79 dB SPL
corresponds to -15 dB Pa. This is furthermore necessary since the Relative Approach analysis requires a dB Pa
calibrated signal.
'
N BGN
=

1
K

∑p

2
BGN

(k )

⎛ N'

⎞

(6.2)

k

N BGN = 10 ⋅ log⎜

BGN ⎟

(6.3)

⎜ 1Pa ⎟⎝⎠

Where:
k are the sample bins during the background noise sections of the processed signal p(k).
The 3D Relative Approach spectrogram is calculated for the unprocessed signal u(k) and the processed signal p(k)(RAu(t, f), RAp(t, f)). In these spectrograms the background noise sections are again extracted using the clean speech
signal as a mask resulting in RABGN,p(t, f) and RABGN,u(t, f). Note that the Relative Approach calculation is carried out
for the whole 4 s duration before the noise sections are extracted and in order to guarantee a fully adapted Relative
Approach, an adaptation time of 250 ms is considered.
In the next step the 3D spectrograms are subtracted from each other (RAp(t, f) - RAu(t, f)) in order to assess thesimilarity between the processed versus the unprocessed background noise for human perception. The resulting 3D
spectrogram is designated as ΔRABGN,p-u(t, f) in the following. In order classify these spectrograms with numericalvalues the variance σ2 for RAp(t, f), RAu(t, f) and ΔRABGN,p-u(t, f) and the mean µ for RAp(t, f) and ΔRABGN,p-u(t, f) arecalculated according to equations 6.4 and 6.5. Note that the calculation of σ2 and µ is again started after the adaptation
time of Relative Approach (250 ms).

μ=Δf maxtmax
1⋅ ∑ ∑ RABGN (ti , f m ) ⋅ dA(Δf m )Ages ti = tmin Δfm = Δfmin

(6.4)

and
⎛
⎜

σ2 = ⎜

1⎜ Ages⎝t max⋅

Δf max

∑ ∑

⎞
⎟

2
RABGN
(ti , f m ) ⋅ dA(Δf m ) ⎟ − μ 2

ti = t min Δf m = Δf min

ETSI

⎟
⎠

(6.5)

21

with:

ETSI EG 202 396-3 V1.7.1 (2018-03)

Ages =tmax -tmin fmax -fmin .

dA(Δf m ) = Δt ⋅ Δf m .
Δt = 6,66 ms (= 1/150 s).
Δfm ≠ constant (1/12th octave frequency band resolution).
Fmin = 50 Hz, lower frequency of band Δfmin.
Fmax = 8 kHz, upper frequency of band Δfmax.
Fm centre frequency of band Δfm.
Tmin + 250 ms and tmax given by the background noise section extracted before.
The mean values μ(RABGN,U) and μ(RABGN,P) as well as the variance σ²(ΔRABGN,P-U) are calculated for the spectrogramΔRABGN,p-u(t, f) in order to determine the similarity between unprocessed and processed signal ("close to original"). Fora high similarity both parameters should be low leading to a high N-MOS.
If the variance is high - independent of the mean - the processed signal is e.g. highly modulated compared to theunprocessed signal. A typical reason is musical tones. These modulations lead to patterns in the Relative Approach
spectrograms RABGN,p(t, f) and ΔRABGN,p-u(t, f). These indicate a high "attraction" on human perception, because thesecomponents are unexpected. They were not present in the unprocessed signal. These patterns appear typically only
temporarily in ΔRABGN,p-u(t, f) and also only for distinct frequencies. They indicate which parts of the signal havechanged compared to the unprocessed signal.
A high mean and variance of ΔRABGN,p-u(t, f) typically indicates a low "naturalness" of the processed signal comparedto the unprocessed signal. This might be caused by a high level difference between unprocessed and processed signal.
Consequently a low N-MOS can be expected independent of the variance.
Mean and variance of ΔRABGN,p-u(t, f) alone are still not sufficient to predict the N-MOS reliable, because they arederived from a differential spectrogram. "Anchors" to the unprocessed and the processed signal are needed in order tojudge this mean and variance for the N-MOS calculation correctly. For the processed signal therefore the mean value
μ(RABGN,P) is calculated in order to get references for the signal level, the potential SNR improvement (e.g. due to anoise reduction) and the degree of the "attention" attracted. The mean of the unprocessed signal is redundant due to thelinearity of the operations (Δ Relative Approach and mean).

ETSI

22

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure 6.7: Block diagram of N-MOS calculation algorithm;
u(k) unprocessed signal, p(k) processed signal, c(k) clean speech signal
Therefore the variances σ(RABGN,U) and σ(RABGN,P) are calculated for the unprocessed and the processed signal in orderto provide a measure for the "attention" attracted by each of the signals on human perception. In case of the unprocessedsignal this is mainly depending on the structure of the background noise. Stationary noises lead to low variance values,
whereas non-stationary noises lead to high variances corresponding to a high "attention" attracted. For the processedsignal the variance is not only influenced by the structure of the background noise, but also by the changes noise
reduction algorithms and other signal processing components introduce to the signal. Table 6.1 gives an overview about
the extracted parameters. Note that the variance parameters are square-rooted (σ = sqrt(σ²)) for better conditioning of
the linear regression.
Table 6.1: Extracted parameters for N-MOS
P0
P1
P2

NBGN,P

σ(RABGN,U)
σ(RABGN,P)

P3
P4
P5

ETSI

σ(∆RABGN,P-U)
μ(RABGN,U)
μ(RABGN,P)23

ETSI EG 202 396-3 V1.7.1 (2018-03)

Finally the N-MOS is the result of a linear, quadratic regression algorithm applied to all six parameters P0 to P5:
2

5

NMOS = c0 + cBGN ⋅ N BGN + ∑∑ c ji ⋅ Pi j

(6.6)

j = 1 i =1

where:
c0, cBGN and cji are the coefficients for the linear regression;
j is the regression order index;
Pi are the Relative Approach related parameters according to table 6.1.
NOTE:

The influence of packet loss is not considered separately, but indirectly by the Relative Approach. A lost
packet is typically a simple gap in the signal. The phase information is also completely lost. Gaps andphase errors sound very unpleasant and are detected by the Relative Approach as a highly disturbing
wideband pattern or, in other words, as a high "attention" attracted at human perception. In case of a lost
packet during the background noise sections the mean and the variance of the Δ Relative Approach andthe 3D Relative Approach spectrogram of the processed signal are effected and will increase. Thisdecreases the N-MOS accordingly. The influence of jitter is so far not considered. A maximum jitter of
20 ms was applied within the present data. But only for a very few conditions jitter could be observed.
Jitter could therefore not be covered reliable by the model. Higher amounts of jitter and adaptive jitter
buffers are not found in the present database and were therefore not yet investigated.

It should be noted that the expert study of the processed signals used in the listening tests (see ETSI EG 202 396-2 [i.2])
showed that packet loss during the background noise sections only slightly decreased the N-MOS. Furthermore "realpacket losses" occur only rarely in today's networks because VoIP devices like gateways and IP-phone are typicallyequipped with packet loss concealment (PLC) algorithms. Those PLC algorithms were not applied during the sample
generation process of the present database used in the listening tests. In principle the Relative Approach algorithm wasalready successfully applied in the past to scenarios using different PLC and jitter buffer implementations [i.8], [i.9],[i.10], [i.11] and [i.12]. The N-MOS algorithm is therefore expected to work properly also for PLC scenarios.
Training and validation of the model were carried out using the regression coefficients for the N-MOS calculation
summarized in table 6.1a.
Table 6.1a: Coefficients for linear, quadratic N-MOS regression algorithm

6.5.3

Comparing subjective and objective N-MOS results

The coefficients for the linear quadratic regression were determined during the training of the algorithm by averaging
the six contributing parameters (NBGN,P, σ(RABGN,U), σ(RABGN,P), σ(ΔRABGN,P-U), μ(RABGN,U) and μ(RABGN,P)) for the six
French sentences of one condition. In the second step these averaged parameters were mapped by the regression
formula to the auditory N-MOS derived in the listening test.

ETSI

24

ETSI EG 202 396-3 V1.7.1 (2018-03)

Merged training dataset

Error distribution

5
84.8%

→

rmse (mapped) = 0.327

CDF(e) [%]

80

3

60
47.6%

→

Objective N-MOS

4

100.0%→

Kendalls tau = 0.748

99.6%

→

97.0%

100

rank order = 0.918→

rmse = 0.362

40

2
20
N-MOS
Mapping Function [3] (r = 0.917)
1
1

2

3

4

Distribution of residual error
Error Steps005

Auditory N-MOS

0.25

0.5
0.75
N-MOS Error e

1

1.25

Figure 6.8: Left: Objectively calculated N-MOS versus auditory N-MOS;
Right: CDF of residual error versus N-MOS Error e
All selected (French) training conditions according to clause 4 (independent of the network condition) and the new
training database provided by Orange according to clause 6.3 were used for this mapping. All per-sample predictions
belonging to one condition are averaged to a per-condition N-MOS which is used for comparison with the subjective
per-condition N-MOS.
The left hand graph in figure 6.8 shows that the per sample deviation between the subjective and objective N-MOS isless than 0,5 MOS for nearly all (269) conditions. This results in an overall correlation of 91,7 %.
The right graph in figure 6.8 shows the cumulative density function CDF(e) versus the N-MOS Error e.
e = NMOS auditory − NMOS objective(6.7)

Based on the cumulated density function the right hand graph in figure 6.6 shows additionally an adaptive tolerance
scheme indicating the CDF(e) values for e = 0,25, e = 0,5, e = 0,75 and e = 1. For example is the N-MOS Error e lower
than 0,5 for 85 % of the conditions and lower than 0,75 for 97 % of all conditions.

6.6

Objective S-MOS

6.6.1

Introduction

The objective S-MOS is also aimed to reproduce the listening impression of the test persons in the listening test, toprovide a high correlation to the given database and also a high robustness for other databases. The experts groupverified the subjective S-MOS values and in combination with their listening impression they extracted the parameters
relevant for the S-MOS:•

Level and quality of processed background noise.

•

Signal to noise ratio (SNR) between speech and noise in the processed signal.

•

Improvement or impairment of SNR between unprocessed and processed signal.

•

Packet loss.

•

Modulation of speech/speech sound.

•"Naturalness".

ETSI

25

ETSI EG 202 396-3 V1.7.1 (2018-03)

At a first glance it seems surprisingly that one of the main influences on the S-MOS seems to be the background noise
quality. The experts found out that if the quality of the background noise at the beginning of the sample is good, thespeech quality is also expected to be good. And if the processed background noise sounds unpleasant - for whateverreason - also the speech quality is expected to be low. Between both extremes a sliding crossover area can be observed.
The Δ Relative Approach is again chosen to determine parameters like "modulation" or "naturalness" and also in order
to cover packet loss effects.

6.6.2

Description of S-MOS Algorithm

Similar to the N-MOS calculation also the S-MOS algorithm is also designed to reproduce the parameters which wereextracted by the experts' analysis.
The principle of the S-MOS calculation is shown in the block diagram in figure 6.9. Again it should be noted that theclean speech c(k), the unprocessed u(k) and the processed signal p(k) have to be pre-processing along the steps
described in clause 6.2. The input for the neural network leading to the objective S-MOS are ΔSNR and five Relative
Approach related parameters.
The difference between the SNR of the unprocessed and the processed signal (ΔSNR) is one of the extracted parametersby the experts. In order to determine the SNR in each signal, the clean speech signal is again used as a mask in order toseparate the speech sections (uSP(k) and pSP(k)) and the noise sections (uBGN(k) and pBGN(k)). The level is thencalculated along equation (6.3), which results in the speech and noise level for those sections without ((S+N)"SP,u and(S+N)"SP,p) and in the noise level during only background noise sections (N"BGN,u and N"BGN,p). For the unprocessed
and the processed signal SNRu and SNRp are then calculated in dB according to equation 6.8:'
⎛ ( S+N )'SP − N BGN⎞⎟⎟
SNR = 10 ⋅ log ⎜⎜
'
N BGN
⎝
⎠

(6.8)

The ΔSNR is the simple difference between SNRu and SNRp:
ΔSNR = SNR p − SNRu

(6.9)

In order to cover the influence signal processing on the sound of the transmitted signal, the modulation and"naturalness" (potentially impaired e.g. by noise reduction algorithms) the Relative Approach and the Δ Relative
Approach are used.
The 3D Relative Approach spectrograms are calculated for all three signals, the unprocessed, the processed and forthe clean speech signal (RAu(t ,f), RAp(t, f) and RAc(t, f)). With the clean speech as mask the speech sections of the 3D
spectrograms are extracted (RASP,u(t, f), RASP,p(t, f) and RASP,c(t, f)).
In the next step two Δ Relative Approach spectrograms are calculated between the processed and the unprocessedsignal (ΔRASP,p-u(t, f)) and between the processed and the clean speech signal (ΔRASP,p-c(t, f)).
The variance σ2 and the mean µ are calculated for both deltaq spectra using the equations (6.4) and (6.5) resulting in
σ²(ΔRASp, P-C), σ² (ΔRASp, P-U), μ(ΔRASp, P-C) and μ(ΔRASp, P-U). Additionally the mean μ(RASp, P) is calculated for
RASP,p(t, f).
The variance σ²(ΔRASp, P-C) is a measure for the amount of patterns in the differential spectrogram between processedand clean speech signal. Patterns may occur due to e.g. musical tones or modulations introduced by noise reductions orother signal processing components. Those patterns attract the listeners' attention. The variance σ²(ΔRASp, P-C) cantherefore also be seen as a measure for the amount of "attention" attracted.
A similar effect could be observed for those listening examples providing low N-MOS scores: if the quality of the
background noise is poor at the beginning of the sample, subjects expect a poor speech quality. They compare the actual
speech to a signal containing speech and background noise. Mean and variance are therefore calculated for the Δ
Relative Approach between the processed and the unprocessed signal (ΔRASP,p-u(t, f)).

ETSI

26

ETSI EG 202 396-3 V1.7.1 (2018-03)

The mean μ(RASp, P) is used in both cases in order to characterize the absolute "attention" attracted by the processedsignal. The comparison of μ(RASp, P) and μ(ΔRASp, P-C) covers the influence of added or removed patterns introduced byroom acoustics, background noise, the phone and the signal processing during the transmission. Similarly μ(RASp, P) andμ(ΔRASp, P-U) can be compared in order to assess only the influence of the terminal and the transmission. The
combination of these three parameters indicates whether the speech quality was impaired or improved.
Note that again the influence of packet loss is not covered separately but implicitly in the variance and the mean of the
Δ Relative Approach (see also end of clause 6.5.2).
The resulting values ΔSNR, μ(RASp, P), σ²(ΔRASp, P-U), σ²(ΔRASp, P-C), μ(ΔRASp, P-U) and μ(ΔRASp, P-C) are used as input
parameters Pi for a feed forward neural network as described e.g. in [i.33]. Table 6.2 shows an overview over the
extracted parameters. Note that again the square-rooted values of variances are used as the input of the neural network
for better conditioning of the calculation.

Figure 6.9: Block diagram of S-MOS calculation algorithm;
u(k) unprocessed signal, p(k) processed signal, c(k) clean speech signal

ETSI

27

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table 6.2: Extracted Parameters for S-MOS∆SNR
μ(RASp, P)
μ(∆RASp, P-C)

P1
P2
P3

μ(∆RASp, P-U)
σ(∆RASp, P-C)
σ(∆RASp, P-U)

P4
P5
P6

Figure 6.10: Structure of neural network for S-MOS
The setup of the neural network is shown in figure 6.10. It consists of 5 units (also known as "node" or "neuron") in one
hidden layer; each layer Nj includes a connection from each transformed input parameter Ii. The output Oj of each layer
is calculated as the weighted sum of each input Ii using the weights wij. The outputs Oj are then weighted by wj andsummed up to the output S-MOS. Both, wij and wj are the result of the training of the network.
The parameters according to table 6.2 are composed to a vector P according to equation 6.10 including a bias as the firstelement.

   1       

(6.10)The output calculation of the neural network shown in figure 6.10 can be described as concatenated matrix operations asshown in equation 6.11.
S-MOS



 ,   

೔೙
೔೙

⋅ ⋅

(6.11)First the parameter vector P is normalized to mean 0.0 and standard deviation 1.0. This is done by subtracting the
average of all training data for each parameter from each item of the input parameter vector. The averages for eachparameter Pi can be described as a vector, which is different for narrow- and wideband mode:(6.12)NOTE 1: The first element is set to zero to be compatible with the bias element in P.
A similar approach can be made for the input standard deviation Sin for each parameter Pi, also separated for wide andnarrowband in equation:(6.13)NOTE 2: The first element is set to one to be compatible with the bias element in P.
After normalizing the input data, the sigmoid function fsigmoid(x) is applied to the each normalized parameter Pi.

ETSI

28

ETSI EG 202 396-3 V1.7.1 (2018-03)

This ensures that each input of each neuron of the hidden layer is soft-limited to the range ± 1,0 and guarantees thatparameters out of the training range cannot produce an overflow which results in eventually unreasonable scores.
For the current model, the hyperbolic tangent was chosen to a sigmoid function:(6.14)
Thus the input of the hidden neuron layers can also be given as a transformed parameter vector P :̃!"#݅݊෦ ෦ ෦ ෦ ෦ ෦ ቀ $݅݊ ቁ   1 1 2 3 4 5 6 ෤ 


(6.15)NOTE 3: The sigmoid function is not applied to the bias component.
̃P and H. H describes all weights fromThe output of the hidden layer is calculated with a matrix multiplication of P each input parameter to each neuron in the hidden layer. These weights are the results of the training with the
back-propagation algorithm. In consequence, H is different for each bandwidth mode:(6.16)
The five transformed output values of the hidden layer are then passed to the output layer. Here the output of the neuralnetwork is calculated with another matrix multiplication with the matrix O, which weights the outputs of the hidden
layers to an output score SMOSobjective, raw. This output layer matrix O is also given for wide and narrowband mode
independently:(6.17)
Another part of the back-propagation algorithm is also to normalize the output data to mean 0,0 and standard deviation
1,0. To revise this step and transform the output of the neural network back to the MOS scale, the objective S-MOS iscalculated from the raw score:

S-MOS

 max1.0, min+ ⋅ S-MOS ,

!+ " , 5.0"

(6.18)The objective S-MOS is finally calculated with Mout = 3.0, Sout = 2.0 and a hard limiter [1.0; 5.0].

6.6.3

Comparing Subjective and Objective S-MOS Results

In contrast to the N-MOS calculation, the contributing parameters ΔSNR, μ(RASp, P), σ²(ΔRASp, P-U), σ²(ΔRASp, P-C),
μ(ΔRASp, P-U) and μ(ΔRASp, P-C) are evaluated per sample with the neural network described above. The six S-MOS
values for each samples are then averaged to the per-condition result.

ETSI

29

ETSI EG 202 396-3 V1.7.1 (2018-03)

Merged training dataset

Error distribution

5

97.4%

→

100

rank order = 0.927
rmse (mapped) = 0.304→

80
CDF(e) [%]

63.9%

3

→

Objective S-MOS

100.0%→

88.1%

Kendalls tau = 0.768

4

99.6%

→

rmse = 0.307

60
40

2
20
S-MOS
Mapping Function [3] (r = 0.919)
1
1

2

3

4

Distribution of residual error
Error Steps
0
0

5

0.25

Auditory S-MOS

0.5
0.75
S-MOS Error e

1

1.25

Figure 6.11: Left: Objectively calculated S-MOS versus auditory S-MOS;
Right: CDF of residual error versus S-MOS Error e
Similar to the N-MOS training all training samples -were used. All per-sample predictions belonging to one condition
are averaged to a per-condition S-MOS which is used for comparison with the subjective per-condition S-MOS.
The left hand graph in figure 6.11 shows that the per sample deviation between the subjective and objective S-MOS ishigher than 0,5 MOS only for about 10 % of all (269) conditions. This results in an overall correlation of 91,9 %.
The right hand graph in figure 6.11 indicates the cumulated density function CDF(e) versus the S-MOS Error e (seealso equation 6.7). It also give an adaptive tolerance scheme indicating the CDF(e) values for e = 0,25, e = 0,5,e = 0,75 and e = 1. The S-MOS Error e is e.g. lower than 0,5 for 88,1 % of all conditions.

6.7

Objective G-MOS

6.7.1

Description of G-MOS Algorithm

The subjectively derived global quality is expected to be a combination of speech quality and noise quality. The expert
analysis did not only extract those conditions of both languages which were somehow inconsistent. This test was also
carried out to extract the main influencing parameters during the subjective ratings of N- and S-MOS. These parameters
were then reproduced by the N-MOS and S-MOS calculation described in clauses 6.4 and 6.5 in order to model thehuman perception concerning speech and noise quality during the listening test.
Both, N-MOS and S-MOS calculation are optimized on the reproduction of the perceptual effects during the listening
test. They were not optimized for "artificial" conditions like a highly modulated background noise together with a clean
speech signal or vice versa. Those kinds of data were not considered in the listening test and were therefore also notconsidered by the objective model.
In accordance to the human perception, the new model first calculates the noise and speech quality. In a second step theoverall quality is modelled. The G-MOS is therefore calculated by applying a linear, quadratic regression algorithm to
N-MOS and S-MOS. The principle is shown in figure 6.9.
The corresponding G-MOS calculation equation is:2

GMOS = c0 +

∑
j =1

2

c Sj ⋅ SMOS j +c Nj ⋅ NMOS j∑
j =1

where:
c0, cSj and cNj are the coefficients for the linear quadratic regression;
j is the regression order index.

ETSI(6.19)30

S-MOS

ETSI EG 202 396-3 V1.7.1 (2018-03)

N-MOS

Linear, quadratic
regression

G-MOS
Figure 6.12: Block diagram of G-MOS calculation algorithm
Training and validation of the S-MOS regression were carried out using the regression coefficients in table 6.3.
Table 6.3: Coefficients for linear, quadratic G-MOS regression algorithm

6.7.2

Comparing subjective and objective G-MOS results

The coefficients for the G-MOS regression were derived by mapping the previously calculated objective N-MOS andS-MOS to the G-MOS results collected in the listening test using the linear, quadratic regression. The result comparedto the auditory G-MOS is shown in figure 6.13. All per-sample predictions belonging to one condition are averaged to a
per-condition G-MOS which is used for comparison with the subjective per-condition G-MOS.
The left hand graph in figure 6.13 shows that the per sample deviation between objective and auditory G-MOS is lessthan 0,5 MOS for most of the (269) conditions. The overall correlation is determined to 95,1 %.
The cumulated density function CDF (e) versus the G-MOS Error e (see also equation 6.7) is shown on the right infigure 6.13. The CDF indicates that for 68 % of all conditions the G-MOS Error e is less than 0,25 MOS and for nearly
all conditions e is less than 0,5 MOS.

ETSI

31

ETSI EG 202 396-3 V1.7.1 (2018-03)

Error distribution
99.6%

→

100

rank order = 0.950→

rmse = 0.241

95.9%

→

Merged training dataset

5100.0%

Kendalls tau = 0.811
rmse (mapped) = 0.235

80
CDF(e) [%]

68.4%

3

→

Objective G-MOS

4

60
40

2
20
G-MOS
Mapping Function [3] (r = 0.951)
1
1

2

3

4

Distribution of residual error
Error Steps
0
0

5

0.25

Auditory G-MOS

0.5
0.75
G-MOS Error e

1

1.25

Figure 6.13: Left: Objectively calculated G-MOS versus auditory G-MOS;
Right: CDF of residual error versus G-MOS Error e

7

Validation of the Wideband Objective Test Method

7.1

Introduction

In order to validate the Objective Test Method results, 130 out of the 432 initial conditions per language were reservedto the validation activity. Due to the consistent problems related in clauses 5.3 and J.1, the final validation conditions
retained were 81 considering the French Database. These condition results are shown in annex F. Additionally, anothersubjective database provided by Orange with 18 conditions was provided to validate the test method (see clause 7.3).
The process carried out to validate the objective test method had the following steps:
1)

Objective results obtaining: using the developed calculation algorithms, described in clauses 6.5, 6.6 and 6.7(N-MOS, S-MOS and G-MOS).

2)

Comparison between obtained objective and the subjective results (see ETSI EG 202 396-2 [i.2]) considering
all the validation condition samples and statistical evaluation. This evaluation will consist on the accuracy,
monotonicity and consistency Test Method characterization.

To carry out this characterization, the following statistical metrics will be used:
Root Mean Square Error (RMSE)
The root mean square error according to [i.24] measures the difference between values predicted by the algorithm andthe auditory values to evaluate its accuracy:
RMSE =

1N∑ Perror[i]2NPerror (i) = MOS (i ) − MOS p (i )where N is the number of samples, MOS(i) is the subjective MOS and MOSp is the predicted MOS.

ETSI

(7.1)
(7.2)

32

ETSI EG 202 396-3 V1.7.1 (2018-03)

Pearson Correlation
The Pearson Correlation coefficient according to [i.24] measures the linear relationship between the algorithm
performance and the subjective data. This coefficient varies from -1,0 to 1,0; a value of 1,0 shows that a linear equation
describes the relationship perfectly and positively, with all data points lying on the same line and having the same
behaviour; a score of -1,0 shows that all data points lie on a single line but having opposite behaviour; a value of 0,0
shows that a linear model is inappropriate and that there is no linear relationship between the variables.
NR=∑ ( Xi − X ) * (Yi − Y )i =1∑ ( Xi − X ) * ∑ (Yi − Y )
2

2

(7.3)

Where N is the number of samples, Xi denotes the subjective score MOS and Yi the objective one.
Spearman's Rank Correlation
The Spearman's Rank Correlation coefficient according to [i.24] is a non-parametric measure of correlation. It assesses
how well an arbitrary monotonic function could describe the relationship between two variables. This parameter varies
from -1,0 to +1,0, similar as the Pearson Correlation:
6⋅

ρ =1−

∑d

i

2

N

N ( N 2 − 1)

(7.4)

Where N is the number of samples and d the difference between each rank (position in an ordered table of conditions)of corresponding values of x and y.
Kendall Tau Rank Correlation
The Kendall Tau Rank Correlation Coefficient [i.26] is used to measure the degree of correspondence between two
rankings. If the agreement is perfect the coefficient value is 1,0, on the other hand if the disagreement is perfect thevalue is -1,0, if the rankings are completely independent , the coefficient has value 0,0:

∑q

4

τ=

i

N

N ( N − 1)

−1

(7.5)

Where N is the number of samples and qi the sum, over all samples, of samples ranked after the given sample by bothrankings.
Residual Error Distribution
The residual error distribution according to [i.24] evaluates the consistency of the model using the Cumulative Density
Function (CDF) applied to the Error e:
e = |MOSauditory - MOSobjective|(7.6)

The graphical representation of the CDF will show the number of conditions which yields a maximum residual error.
NOTE:

The prediction results for training and validation reported in previous versions of the present document(up to V1.3.1) [i.34] reported somewhat better performance of the model than in this present document.
This apparently lower prediction accuracy is caused by several reasons:
In previous versions, it was assumed that the extracted parameters for regression and neuralnetwork according to clauses 6.5.2 and 6.6.2 are averaged for one condition and then mapped
against subjective data. The average subjective condition MOS values are much more reliable than
the average sample MOS values.

ETSI

33

ETSI EG 202 396-3 V1.7.1 (2018-03)

In the present document, training is applied on a per-sample base. Due to the low amount of votes
per sample (only 4), the training may obtain lower precision. On the other hand, with the new
approach it is now possible to obtain also per-sample scores (with eventually lower significance),
which was not recommended before.
The new training set consists of two similar, but in general different databases. Even though themerged datasets seem to fit together, it may also decrease the prediction performance on the single
databases. But with this additional data, which introduces real device recordings, the neuralnetwork gets more robust against new, unknown data.
In almost all presented prediction results, it seems like that a mapping function could additionally
improve the performance. Especially for the training conditions, it is expected that due toleast-mean-square-mapping (regression and neural network use these algorithms), no offset or shift
is present. This assumption is only valid for the per-sample training. After averaging, this principle
is not necessarily observable.

7.2

ETSI EG 202 396-2 Database Results Analysis

7.2.1

Comparing subjective and objective N-MOS results

Only validation recordings from the original ETSI EG 202 396-2 [i.2] were used for the following results.
Figures 7.1 and 7.2 show that the per sample deviation between the subjective and the objective N-MOS is less than0,5 MOS for most conditions. This results in an overall Pearson correlation of 92,3 %. The Spearman Correlation
Coefficient is 0,931 and the Kendall Tau is 0,782, both of them are near to 1.
Error distribution

rmse = 0.361

96.3%
84.0%

rmse (mapped) = 0.297

CDF(e) [%]

80

3

60
50.6%

→

Objective N-MOS→

Kendalls tau = 0.782

4

100.0%→

100

rank order = 0.931→

STF294 database - validation set

5

40

2
20
N-MOS
Mapping Function [2] (r = 0.923)
1
1

2

3

4

Distribution of residual error
Error Steps
5

Auditory N-MOS

Figure 7.1: Left: Objectively calculated N-MOS vs.
auditory N-MOS for validation conditions

0
0

0.25

0.5
0.75
N-MOS Error e

1

1.25

Figure 7.2: Objectively CDF of residual error vs.
N-MOS Error e for validation conditions

For this situation, the RMSE value is 0,36 and the distribution of the residual error is shown in figure 7.2 where the
N-MOS Error e is lower than 0,25 for approximately 50 % of the conditions and lower than 0,75 for 96 % for all
conditions.

7.2.2

Comparing subjective and objective S-MOS results

Only validation recordings from the original ETSI EG 202 396-2 [i.2] were used for the following results.
Figures 7.3 and 7.4 show that the per sample deviation between the subjective and the objective S-MOS is less than0,5 MOS for a large amount of conditions. This results in an overall correlation of 89,3 %. The Spearman Correlation
Coefficient is 0,880 and the Kendall Tau is 0,716, both of them are near to 1.

ETSI

34

ETSI EG 202 396-3 V1.7.1 (2018-03)

STF294 database - validation set

Error distribution

5

92.6%
84.0%

→

rmse (mapped) = 0.379

CDF(e) [%]

80

3

60

60.5%

→

Objective S-MOS

4100.0%→

Kendalls tau = 0.716

97.5%

→

100

rank order = 0.880→

rmse = 0.379

40

2
20
S-MOS
Mapping Function [3] (r = 0.893)
1
1

2

3

4

Distribution of residual error
Error Steps
0
0

5

0.25

0.5
0.75
S-MOS Error e

Auditory S-MOS

Figure 7.3: Left: Objectively calculated S-MOS vs.
auditory S-MOS for validation conditions

1

1.25

Figure 7.4: Objectively CDF of residual error vs.
S-MOS Error e for validation conditions

For this situation, the RMSE value is 0,37 and the distribution of the residual error is shown in figure 7.4 where the
S-MOS Error e is lower than 0,25 for approximately 60 % of the conditions and lower than 0,75 for 93 % for all
conditions.

7.2.3

Comparing Subjective and Objective G-MOS Results

Only validation recordings from the original ETSI EG 202 396-2 [i.2] were used for the following results.
Figures 7.5 and 7.6 show that the per sample deviation between the subjective and the objective G-MOS is less than0,5 MOS for nearly all conditions. This results in an overall correlation of 91,2 %. The Spearman Correlation
Coefficient is 0,892 and the Kendall Tau is 0,735, both of them are near to 1.
STF294 database - validation set

Error distribution

5→

rmse = 0.315

100

rank order = 0.892→

Kendalls tau = 0.735
rmse (mapped) = 0.302

CDF(e) [%]

80

3

60

61.7%

→

Objective G-MOS

4100.0%

87.7%

40

2
20
G-MOS
Mapping Function [3] (r = 0.912)
1
1

2

3

4

Distribution of residual error
Error Steps
5

Auditory G-MOS

Figure 7.5: Left: Objectively calculated G-MOS vs.
auditory G-MOS for validation conditions

0
0

0.25

0.5
0.75
G-MOS Error e

1

Figure 7.6: Objectively CDF of residual error vs.
G-MOS Error e for validation conditions

For this situation, the RMSE value is 0,31 and the distribution of the residual error is shown in figure 7.6 where the
G-MOS Error e is lower than 0,25 for approximately 62 % of the conditions and lower than 0,75 for 100 % for all
conditions.

ETSI

1.25

35

ETSI EG 202 396-3 V1.7.1 (2018-03)

7.3

Orange Validation Database results Analysed

7.3.0

Introduction

In addition to the new training database described in clause 6.3, another validation database from Orange was provided.
This database was used for the validation process of ETSI TS 103 106 [i.32] and also included the same speech materialas the Orange training database. The database consists of 18 conditions with 8 sentences each (2 male/2 female talkers).
It is used for a full external validation in terms of validating conditions which were not part of the same listening test as
the training set.
Since this additional validation database is completely unknown, an eventual mapping function can be applied in orderto improve the prediction performance. A third order mapping function is always printed within the result plots, but is
not applied to the results, also for the error distribution plots. Only the RMSE after mapping is reported for
informational purposes.

7.3.1

Comparing subjective and objective N-MOS results

Figures 7.7 and 7.8 show that the per sample deviation between the subjective and the objective N-MOS is less than0,5 MOS for almost all conditions. This results in an overall Pearson correlation of 97,4 %. The Spearman Correlation
Coefficient is 0,937 and the Kendall Tau is 0,818, both of them are near to 1.
Orange validation database

Error distribution

5→

rmse = 0.162

100

rank order = 0.937

100.0%→

91.7%

Kendalls tau = 0.818
rmse (mapped) = 0.133

80
CDF(e) [%]Objective N-MOS

4

3

60
40

2
20
N-MOS
Mapping Function [3] (r = 0.974)
1
1

2

3

4

Distribution of residual error
Error Steps
5

Auditory N-MOS

Figure 7.7: Left: Objectively calculated N-MOS vs.
auditory N-MOS for validation conditions

0
0

0.25

0.5
0.75
N-MOS Error e

1

1.25

Figure 7.8: Objectively CDF of residual error vs.
N-MOS Error e for validation conditions

For this situation, the RMSE value is 0,16 and the distribution of the residual error is shown in figure 7.8 where the
N-MOS Error e is lower than 0,25 for approximately 92 % of the conditions and lower than 0,5 for 100 % for all
conditions. An additional mapping can be applied (RMSE after mapping is 0,13), but does not change the overall
prediction behaviour.

7.3.2

Comparing subjective and objective S-MOS results

Figures 7.9 and 7.10 show that the per sample deviation between the subjective and the objective S-MOS is less than0,5 MOS for a large amount of conditions. This results in an overall correlation of 70,7 %. The Spearman Correlation
Coefficient is 0,818 and the Kendall Tau is 0,606.

ETSI

36

ETSI EG 202 396-3 V1.7.1 (2018-03)

Orange validation database

Error distribution

5
rmse = 0.618

100

rank order = 0.818

83.3%

80
CDF(e) [%]Objective S-MOS

rmse (mapped) = 0.281

75.0%

3

→

4

→

Kendalls tau = 0.606

60
40
25.0%

→

2
20
S-MOS
Mapping Function [1] (r = 0.707)
1
1

2

3

4

Distribution of residual error
Error Steps
0
0

5

0.25

0.5
0.75
S-MOS Error e

Auditory S-MOS

Figure 7.9: Left: Objectively calculated S-MOS vs.
auditory S-MOS for validation conditions

1

1.25

Figure 7.10: Objectively CDF of residual error
vs. S-MOS Error e for validation conditions

For this situation, the RMSE value is 0,62 and the distribution of the residual error is shown in figure 7.10 where the
S-MOS Error e is lower than 0,25 for approximately 25 % of the conditions and lower than 0,75 for 83 % for all
conditions.
An additional mapping would dramatically increase the prediction performance; the RMSE would decrease to 0,28.

7.3.3

Comparing Subjective and Objective G-MOS Results

Figures 7.11 and 7.12 show that the per sample deviation between the subjective and the objective G-MOS is less than0,5 MOS for nearly all conditions. This results in an overall correlation of 81,8 %. The Spearman Correlation
Coefficient is 0,811 and the Kendall Tau is 0,667.
Orange validation database

Error distribution

5→

rmse = 0.608

100

rank order = 0.811
rmse (mapped) = 0.193

CDF(e) [%]

80

3

60
50.0%

→

Objective G-MOS→

Kendalls tau = 0.667

4

100.0%

83.3%

40

2
20
G-MOS
Mapping Function [3] (r = 0.818)
1
1

2

3

4

Distribution of residual error
Error Steps
5

Auditory G-MOS

Figure 7.11: Left: Objectively calculated G-MOS vs.
auditory G-MOS for validation conditions

0
0

0.25

0.5
0.75
G-MOS Error e

1

1.25

Figure 7.12: Objectively CDF of residual error
vs. G-MOS Error e for validation conditions

For this situation, the RMSE value is 0,61 and the distribution of the residual error is shown in figure 7.12 where the
G-MOS Error e is lower than 0,5 for approximately 50 % of the conditions and lower than 1 for 100 % for all
conditions.
Again, an additional mapping would dramatically increase the G-MOS prediction performance; the RMSE would
decrease to 0,19. This is mainly the influence of the also overrated S-MOS, since G-MOS is just a combination ofN- and S-MOS.

ETSI

37

ETSI EG 202 396-3 V1.7.1 (2018-03)

8Objective Model for Narrowband Applications

8.0

Introduction

The objective model described in the clauses before in general is also applicable for narrowband scenarios. However
some modifications have to be made in order to address the narrowband case. These modifications are described below.
The narrowband version of the model is based on an aurally-adequate analysis in order to best cover the listener's
perception based on the previously carried out listening tests.
The test method is applicable for:•narrowband handset and narrowband hands-free devices (in sending direction);•noisy environments (stationary or non-stationary noise);•different noise reduction algorithms;•

G.711, G.726, G.729A, iLBC, Speex HiQ/LQ and GSM FR, GSM EFR, and AMR narrowband coders;•

VoIP networks introducing packet loss.

Due to the special sample generation process the method is only applicable for electrically recorded signals. The qualityof terminals can therefore only be determined in sending direction.

8.1

File pre-processing

The processed signal p(k) is already calibrated to the active speech level (ASL) of -21 dB Pa / 73 dB SPL and filteredwith an modified intermediate reference system (IRS) according to Recommendation ITU-T P.830 [i.28] in receiving
direction for the presentation in the listening test. Exactly this signal is used in the objective model.
For the new narrowband mode, the clean speech and the unprocessed signal (c(k) and u(k)) are filtered with a modified
IRS filter according to Recommendation ITU-T P.830 [i.28] in sending and receiving direction. With thispre-processing step, all following analyses refer to a perfect transmission over a typical narrowband telephony network.
After filtering, both reference files are calibrated to the same active speech level like the processed signal. This refers to
the acoustical presentation of the listening test. The overall pre-processing steps result in figure 8.1.

u(k)

p(k)

c(k)

Filter IRS SND

Filter IRS SND

Filter IRS RCVFilter IRS RCVFilter IRS RCV

ASL = 73 dB SPL

ASL = 73 dB SPL

ASL = 73 dB SPL

Input Signals of Objective Model EG 202 396-3
Figure 8.1: Modified pre-processing for narrowband mode

ETSI

38

8.2

ETSI EG 202 396-3 V1.7.1 (2018-03)

Adaptation of the Calculations

The input parameters for the narrowband adapted model are the same as in the wideband mode. In the calculation ofmean and variance from (Delta-) Relative Approach spectrograms, the limits of the frequency range are also adapted to
the narrowband mode.
Table 8.1: Comparison of frequency ranges narrowband/wideband
fmin
fmax

WB Data
50 Hz
7 000 HzNB Data
200 Hz3 600 HzThe new coefficients for N- and G-MOS regression are given in the following tables.
Table 8.2: Coefficients for linear, quadratic N-MOS regression algorithm

Table 8.3: Coefficients for linear, quadratic G-MOS regression algorithm

The new values (vectors Min, Sin, O and matrix H) according to clause 6.6 for the neural network configuration to
calculate objective S-MOS are given in the following equations.
(8.1)

(8.2)

(8.3)

(8.4)

8.3

Prediction results

Overall, there are 263 conditions in the new narrowband database. The training of the model was done using
213 randomly chosen conditions; the remaining 50 conditions were used to test the model against unknown, retained
data (in terms of data which were not used to train the model). This process of training and validation data was also
used in the ETSI STF 294 project (see ETSI EG 202 396-2 [i.2]).

ETSI

39

ETSI EG 202 396-3 V1.7.1 (2018-03)

The correlation coefficients and root-mean-square error between the subjective data from the listening test and theprediction of the narrowband adapted model are shown in the following scatter plots below (see figure 8.2).
Training Results:

Validation Results:

Figure 8.2: HEAD acoustics NB Database - Comparison subjective versus Objective data

ETSI

40

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex A:
Detailed post evaluation of listening test results
Table A.1 contains the conditions and related auditory S-MOS, N-MOS and G-MOS for French. Also standard deviations for all MOS scores are given. The results forvalidation purposes are blinded.
Table A.1: Result of subjective experiment results -experts listening:
Samples not retained from the French database in addition to the NII condition (hs - handset, hf - hands-free, f - female, m - male speaker)
Extension
Condition
FrenchNoiseRecording Speaker

Network

NSA

Sharp/
smoothdB

MOS
Speech
4,08

19

19

Lux_Carhsf

AMR _NIyes

Smooth

18

145

145

Crossroadshff

AMR _NI

no

Sharp

9

151

151

Crossroadshff

AMR _NIyes

Smooth

9

157

157

Crossroadshff

AMR _NI

yes

Sharp

9

160

160

Crossroadshff

AMR _NI

yes

Sharp

18

1,88

162

162

Crossroadshff

AMR _NIII

yes

Sharp

18

1,38

168

168

Crossroadshsm

AMR _NIIIno

Smooth

9

2,96

169

169

Crossroadshsm

AMR _NIno

Smooth

18

3,08

175

175

Crossroadshsm

AMR _NI

no

Sharp

18

3,21

178

178

Crossroadshsm

AMR _NIyes

Smooth

9

3,96

180

180

Crossroadshsm

AMR _NIII

yes

Smooth

9

2,83

183

183

Crossroadshsm

AMR _NIII

yes

Smooth

18

3,25

ETSI

2,96

FRENCH
MOS MOS
STD
STD
STD
Comment
Noise Global Speech Noise Global
3,42
3,46
0,58
0,58
0,59 Wideband noise
Not consistent, Sample
4 loud Samples 3 and 6
too low speech level
Inconsistent Levels of
1,54
1,71
1,37
0,66
0,81
Samples
Not consistent, Sample
4 loud Samples 3 and 6
too low speech level
Inconsistent Levels of
1,63
1,54
1,03
0,71
0,78
samples
Inconsistent,
1,54
1,13
0,71
0,93
0,45 amplification 2 and 6 toohighInconsistent, noise 2
and 6 too high, not
2,42
2,29
1,27
0,88
0,91
visible in the gains but
audibleInconsistent Levels of
2,92
2,75
1,06
1,18
1,11
samples
Inconsistent Levels of
3,17
2,88
1,06
1,05
0,85
samples
Inconsistent Levels of
2,92
3,13
0,81
0,93
1,03
samples
Inconsistent, noise 2
2,63
2,5
1,17
0,97
0,98 and 6 too high, visible in
the gains (up to 5 dB)
Inconsistent, noise 2
3
2,79
1,15
1,29
1,22 and 6 too high, visible in
the gains (up to 5 dB)

41
Extension
Condition
FrenchNoiseRecording Speaker

ETSI EG 202 396-3 V1.7.1 (2018-03)
FRENCH
MOS MOS
STD
STD
STD
Noise Global Speech Noise Global

Network

NSA

Sharp/
smoothdB

MOS
Speech
3,25

3,46

2,67

1,15

0,93

0,87

189

189

Crossroadshsm

AMR _NIII

yes

Sharp

18

193

193

Crossroadshfm

AMR _NIno

Smooth

9

199

199

Crossroadshfm

AMR _NI

no

Sharp

9

208

208

Crossroadshfm

AMR _NIyes

Smooth

18

2,67

1,96

2,04

1,2

0,91

0,86

211

211

Crossroadshfm

AMR _NI

yes

Sharp

9

2,88

1,75

2,13

1,33

0,94

0,9

214

214

Crossroadshfm

AMR _NI

yes

Sharp

18

1,92

2,13

1,55

1,02

1,12

0,71

216
279

216
252

Crossroads
Road

hf
hs

m
m

AMR _NIIIAMR _NIII

yes
no

Sharp
Smooth

18
18

1,92
2,31

1,67
2,21

1,54
2,09

0,88
0,8

0,7
0,98

0,59
0,78

357

303

Officehf

f

G722_NIIIno

Smooth

9

373

319

Officehf

f

G722_NI

yes

Sharp

9

406

352

Officehfm

G722_NI

noNSA

no NSA

no
NSA

423

369

Officehfm

G722_NIII

yes

Smooth

9

4,25

2,53

2,79

0,99

0,77

0,88

447

393

Pub

hs

f

G722_NIII

no

Sharp

18

478

424

Pub

hs

m

G722_NIyes

Smooth

18

3,17

2,41

2,5

1,13

0,66

0,78

ETSI

Comment
Inconsistent, noise 2
and 6 too high, visible in
the gains (up to 5 dB)
Bad S/N sounds
unprocessed speech
low 3 and 6, notintelligible
Bad S/N sounds
unprocessed speech
low 3 and 6, notintelligible
Inconsistent Levels ofsamples
Inconsistent Levels ofsamples
Inconsistent Levels ofsamples
Example 2 too loud
Example 2 too loudPoor S/N, packet loss
determines speech
quality, processing
errors in sample 6
Processing noise,
processing errors insample 4
Fair S/N processing
errors in sample 6
6 examples with packet
loss, Result Speech andnoise influenced bypacket loss, processing
noise
Packet loss during
speech determines
speech quality, highly
modulated BGN,
processing errors insample 4
Strong amplificationdifference

42
Extension
Condition
FrenchNoiseRecording Speaker

ETSI EG 202 396-3 V1.7.1 (2018-03)

Network

NSA

Sharp/
smooth

dB

480

426

Pub

hs

m

G722_NIII

yes

Smooth18

MOS
Speech
2,58

484

430

Pub

hs

m

G722_NI

yes

Sharp

18

2,92

ETSI

FRENCH
MOS MOS
STD
STD
STD
Comment
Noise Global Speech Noise Global
2,33
2,08
1,02
0,87
0,88 Inconsistent levels
Strong amplification
2
1,96
1,06
0,83
0,81
difference

43

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex B:
Results of PESQ and TOSQA2001 - Analysis of
ETSI EG 202 396-2 database
Although it is known that neither PESQ (Recommendation ITU-T P.862.2 [i.18]) nor TOSQA2001 [i.19] are capable topredict MOS values for scenarios with speech being transmitted and processed together with background noise somedata were analysed in order to document these limitations. This data set consists of 32 conditions (out of 179 overall
selected conditions with known MOS values) with French speech, different types of packet loss, voice coders,background noise and noise reduction.
Table B.1: Test set chosen from ETSI EG 202 396-2 [i.2] database to be analysedwith PESQ and TOSQA2001
ExtensionFrench
3
7
28
54
55
57
82
87
109
120
138
151
166
174
205
207
231
232
291
295
328
339
361
369
382
393
414
418
445
456
466
483

Noise
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Road
Road
Road
Road
OfficeOfficeOfficeOfficeOfficeOfficeOfficeOffice
Pub
Pub
Pub
Pub

Recording Speaker
hs
hs
hf
hf
hs
hs
hf
hf
hs
hs
hf
hf
hs
hs
hf
hf
hs
hs
hs
hs
hs
hs
hf
hf
hs
hs
hf
hf
hs
hs
hs
hs

f
f
f
f
m
m
m
m
f
f
f
f
m
m
m
m
f
f
m
m
f
f
f
f
m
m
m
m
f
f
m
m

Network

NSA

AMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIIIAMR _NI
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NIII

no NSAnono NSAyes
no NSAno NSAno NSAnono NSAnono NSA
yes
no
no
yes
yes
no
yes
yes
yes
no
no
no
yes
no
no
no
no
no
yes
no
yes

Sharp/
smoothno NSA
Smoothno NSASharp
no NSAno NSAno NSA
Smoothno NSASharp
no NSA
SmoothSmoothSharp
SmoothSmoothSharp
SmoothSmoothSharp
SmoothSharp
Sharp
SmoothSmoothSharp
Smooth
Sharp
Sharp
Sharp
SmoothSharpdB
no NSA18
no NSA18
no NSAno NSAno NSA9no NSA9no NSA
9
9
9
9
9
18
9
18
18
9
18
9
9
9
18
18
18
18
9
18
9

MOS
MOS
Speech Noise
3,63
3,13
4,21
3,71
3,79
2,25
2
1,92
4,33
3,04
3,46
3
4
2,21
2,71
2
4,38
3,29
2,88
2,42
1,92
1,58
2,96
1,54
4,13
2,83
2,75
2,08
3
1,67
2,67
1,29
2,21
2,25
4
2,29
2,38
2,46
2,54
2,92
4,53
3,88
3,25
3,83
4,08
2,67
3,46
2,33
4,75
3,79
2,86
3,54
2,75
2,54
3,54
2,67
3
2,25
2,71
1,9
3,25
2,21
2,75
1,58

MOS
Global
3,08
3,63
2,54
1,63
3,21
2,79
2,54
2,21
3,42
2,25
1,29
1,71
3
2
1,71
1,5
1,92
2,88
2,08
2,38
4,08
2,96
3,21
2,46
4,13
3
2,25
2,88
2,25
2,25
2,71
1,96

As shown in table B.1, the data set combines the various conditions and is somehow representative for the full database
ETSI EG 202 396-2 [i.2].
Only French samples were chosen since these are the only ones which were judged with a listening level ofapproximately 79 dB SPL.
NOTE 1: The sample length is less than 3,6 seconds for all samples listed above. Both algorithms,
TOSQA2001 [i.19] and PESQ [i.18], require a sample length of 8 seconds to 32 seconds.
NOTE 2: None of the methods was originally designed to work on files recorded in presence of background noise.

ETSI

44

ETSI EG 202 396-3 V1.7.1 (2018-03)

Analysis Description
Each condition consists of six different sentences (French language). In the listening test, the resulting MOS values are
the mean over these sentences. Both PESQ [i.18] and TOSQA2001 [i.19] were therefore tested with all sentences; themean of these measurements is finally compared to the auditory S-MOS values.
Since both algorithms are known to be very sensitive to background noise, a modified version of each sample wasanalysed in addition. The sequences were cut in order to minimize the noisy parts. The original test samples have alength of exactly 4 seconds; the speech part is active between 0,750 seconds and 3,250 seconds for all conditions. Thus
only 2,5 seconds of speech with background noise were analysed by PESQ and TOSQA2001 in this test case.
PESQ and TOSQA2001 usually use a clean speech signal as the reference in order to estimate the degradation of aprocessed speech sample. For the present database in ETSI EG 202 396-2 [i.2] both, a clean speech as well as
unprocessed signal with (unprocessed) background noise are available as reference signals. Due to the fact, that thealgorithms were not tested with noisy speech signals yet, both types of references, clean speech and the unprocessedsignal, were analysed.
Altogether, the four test cases are summarized in table B.2.
Table B.2: Test cases
Number1
2
3
4

Cut/Full sample
Full
Full
Cut
Cut

Reference
Unprocessed
Clean Speech
Unprocessed
Clean Speech

After all, 4 different test cases were analysed for the 32 conditions with 6 sentences each. This results into
32 x 6 x 4 = 768 single values for PESQ and also for TOSQA2001, which can be considered as a reliable base to draw
conclusions. The PESQ and TOSQA2001 settings listed in table B.3 were used for testing.
Table B.3: Settings of PESQ/TOSQA2001PESQ
TOSQA2001

Sampling rate 16 kHz
Wideband extension (P862.2)
Electrical measurement, Compare to Headphone (Wideband)No fixed delay (all samples were exactly realigned in a prior step)
Variable delay up to 62 ms (due to packet loss and jitter)In order to provide a better overview of the results, the analysis was split into the two different network conditions NI
and NIII. The results are listed separately for both algorithms and network conditions in tables B.4 to B.7.
As expected, the results clearly indicate, that neither PESQ nor TOSQA2001 is able to estimate S-MOS values reliable.
As expected, almost all calculated MOS values are lower than the corresponding auditory S-, N- and G-MOS values.
There is no linear relationship between the S- or G-MOS values and the PESQ/TOSQA2001 results, as the Pearson
correlation coefficient shows. The correlation of the S-MOS data is always below 0,8, the G-MOS data correlate up to
0,89 with the calculated data (TOSQA2001 measurements for Network I + III, cut sample, clean speech as reference).
The assumption of a relationship between G-MOS and calculated data cannot be verified when analysing the scatter plotof this condition. It is obvious that too many TOSQA2001 MOS values are mapped to 1,0, a value close to a virtual, butmeaningless regression line.
The results of both algorithms show MOS values less than 1,5, often close or equal to 1,0 for a lot of conditions. It canbe assumed, that the algorithms completely fail and return a kind of a mapped minimum value for these samples.
The stochastic character of these measurements also arises, when comparing the auditory N-MOS values to thesecalculated by PESQ/TOSQA2001. The correlation between N-MOS and TOSQA2001 / PESQ MOS is often higher
than between TOSQA2001 / PESQ MOS and S- or G-MOS, which should originally be approximated with thesealgorithms.
In order to show that there is also no non-linear relationship between the PESQ/TOSQA2001 scores and auditory
S-MOS values, the scatter plots for all test cases are shown below in figures B.1 to B.4 (Network NI and NIII
conditions).

ETSI

45

ETSI EG 202 396-3 V1.7.1 (2018-03)

On the other hand, the calculated MOS value seemed to be close to the subjective results for a lot conditions. For these
the standard deviation (STD) of the calculated MOS averaged over the six sentences is high. This could not be expected
because the same voice, background noise and processing were used for the recording.
These itemized points and the scatter plots given below show that the MOS values calculated by PESQ andTOSQA2001 measurements do not correlate at all with the results of the listening test.
Table B.4: TOSQA2001 results for NI conditions (clean network)

Reference
Full/Cut
Condition
7
28
55
82
109
151
166
205
232
295
328
361
382
418
445
466
Correlation:
S-MOS
N-MOS
G-MOS

TOSQA2001, Network NI
MOS Var. MOS Var. MOS Var. MOS Var.
Clean
Clean
Unprocesse
unprocesseddSpeechSpeech
full
full
cut
cut
1,26
2,17
1,79
1,88
1,69
1,52
1,86
1,45
1,60
1,26
4,15
3,06
3,64
2,03
2,19
2,66

0,20
0,28
0,57
0,46
0,32
0,37
0,50
0,29
0,24
0,46
0,12
0,34
0,53
0,29
0,28
0,10

0,48
0,44
0,60

2,52
1,42
2,16
1,22
2,18
1,02
1,35
1,00
1,09
1,31
3,73
2,20
3,32
1,93
1,38
1,17

0,30
0,17
0,52
0,19
0,34
0,04
0,33
0,00
0,11
0,24
0,27
0,28
0,29
0,34
0,23
0,15

0,72
0,88
0,89

1,87
3,23
3,27
2,58
3,19
1,80
2,19
1,49
2,13
1,41
4,15
3,57
3,71
2,27
2,51
2,57

0,43
0,23
0,42
0,23
0,67
0,29
0,28
0,33
0,28
0,64
0,11
0,23
0,39
0,31
0,18
0,33

0,73
0,52
0,70

ETSI

2,35
1,50
2,19
1,32
2,18
1,02
1,25
1,00
1,08
1,28
3,71
2,21
3,32
1,89
1,32
1,17

0,34
0,20
0,55
0,22
0,35
0,03
0,27
0,00
0,10
0,29
0,29
0,27
0,27
0,32
0,26
0,16

0,73
0,87
0,89

Auditory MOS
S-MOS N-MOS G-MOS

4,21
3,79
4,33
4,00
4,38
2,96
4,13
3,00
4,00
2,54
4,53
4,08
4,75
3,54
3,00
3,25

3,71
2,25
3,04
2,21
3,29
1,54
2,83
1,67
2,29
2,92
3,88
2,67
3,79
2,67
2,25
2,21

3,63
2,54
3,21
2,54
3,42
1,71
3,00
1,71
2,88
2,38
4,08
3,21
4,13
2,88
2,25
2,71

46

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table B.5: TOSQA2001 results for NIII conditions(3 % packet loss, 20 ms jitter)

Reference
Full/Cut
Condition
3
54
57
87
120
138
174
207
231
291
339
369
393
414
456
483
Correlation:
S-MOS
N-MOS
G-MOS

TOSQA2001, Network NIII
MO
MOS
Var.
S Var. MOS
Var.
MOS Var.
Clean
Clean
Unprocessed
Unprocessed
SpeechSpeech
full
full
cut
cut
1,46
1,11
1,33
1,44
1,00
1,62
1,01
1,00
1,00
1,00
2,69
1,71
2,09
1,00
1,59
1,60

0,34
0,18
0,15
0,28
0,00
0,16
0,02
0,00
0,00
0,00
0,63
0,34
0,46
0,00
0,28
0,24

0,37
0,56
0,53

2,24
1,17
1,90
1,32
1,22
1,19
1,31
1,00
1,02
1,00
2,60
1,85
1,97
1,11
1,19
1,27

0,44
0,20
0,30
0,26
0,26
0,25
0,38
0,00
0,06
0,00
0,56
0,34
0,53
0,14
0,11
0,27

0,75
0,81
0,75

2,13
1,22
2,03
1,43
1,08
1,87
1,29
1,06
1,04
1,04
2,67
1,63
2,01
1,05
2,03
1,61

0,83
0,18
0,25
0,22
0,09
0,20
0,44
0,08
0,09
0,09
0,62
0,43
0,51
0,11
0,54
0,42
0,51
0,57
0,62

2,18
1,17
1,89
1,33
1,27
1,17
1,19
1,00
1,02
1,00
2,66
1,85
1,94
1,09
1,23
1,14

0,33
0,19
0,32
0,26
0,29
0,21
0,31
0,00
0,04
0,00
0,61
0,33
0,56
0,15
0,12
0,19

Auditory MOS
SMOS

N-MOS

G-MOS

3,63
2,00
3,46
2,71
2,88
1,92
2,75
2,67
2,21
2,38
3,25
3,46
2,86
2,75
2,71
2,75

3,13
1,92
3,00
2,00
2,42
1,58
2,08
1,29
2,25
2,46
3,83
2,33
3,54
2,54
1,90
1,58

3,08
1,63
2,79
2,21
2,25
1,29
2,00
1,50
1,92
2,08
2,96
2,46
3,00
2,25
2,25
1,96

0,74
0,83
0,83

Table B.6: PESQ results for NI conditions (clean network)
PESQ, Network NI
MOS Var. MOS Var. MOS Var.
Auditory MOS
Clean
Clean
Reference Unprocessed
Unprocessed
SpeechSpeechS-MOS N-MOS G-MOS
Full/Cut
full
full
cut
cut
Condition
7
1,91 0,05 1,65 0,24 2,30
0,11 1,05 0,01
4,21
3,71
3,63
28
1,14 0,03 1,03 0,00 1,25
0,06 1,02 0,00
3,79
2,25
2,54
55
1,40 0,16 1,31 0,12 1,86
0,50 1,12 0,05
4,33
3,04
3,21
82
1,12 0,05 1,06 0,02 1,22
0,10 1,02 0,01
4,00
2,21
2,54
109
1,81 0,13 1,30 0,08 2,61
0,37 1,08 0,02
4,38
3,29
3,42
151
1,23 0,12 1,04 0,02 1,32
0,16 1,02 0,00
2,96
1,54
1,71
166
2,19 0,27 1,41 0,23 2,60
0,44 1,10 0,07
4,13
2,83
3,00
205
1,27 0,09 1,12 0,06 1,28
0,06 1,03 0,01
3,00
1,67
1,71
232
2,69 0,37 1,15 0,07 2,86
0,46 1,06 0,02
4,00
2,29
2,88
295
1,23 0,12 1,25 0,19 1,47
0,22 1,09 0,09
2,54
2,92
2,38
328
3,32 0,20 2,64 0,20 3,80
0,18 2,53 0,12
4,53
3,88
4,08
361
2,85 0,31 1,38 0,13 3,41
0,26 1,21 0,05
4,08
2,67
3,21
382
3,11 0,24 2,15 0,27 3,39
0,25 2,46 0,24
4,75
3,79
4,13
418
2,16 0,19 1,37 0,11 2,38
0,27 1,41 0,09
3,54
2,67
2,88
445
1,99 0,11 1,22 0,10 2,27
0,16 1,41 0,09
3,00
2,25
2,25
466
2,00 0,30 1,15 0,04 2,43
0,16 1,18 0,07
3,25
2,21
2,71
Correlation:
S-MOS
0,56
0,59
0,61
0,45
N-MOS
0,57
0,81
0,65
0,62
G-MOS
0,73
0,80
0,79
0,65
MOS

Var.

ETSI

47

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table B.7: PESQ results for NIII conditions (3 % packet loss, 20 ms jitter)

Reference
Full/Cut
Condition
3
54
57
87
120
138
174
207
231
291
339
369
393
414
456
483
Correlation:
S-MOS
N-MOS
G-MOS

PESQ, Network NIIIMOS Var. MOS Var. MOS Var. MOS Var.
Auditory MOS
Clean
Unprocesse
Clean
UnprocessedSpeechd
Speech S-MOS N-MOS G-MOS
full
full
cut
cut
1,27
1,06
1,19
1,08
1,58
1,11
1,35
1,15
1,31
1,39
1,24
1,48
1,58
1,51
1,50
1,52

0,12
0,02
0,05
0,02
0,15
0,03
0,13
0,06
0,09
0,24
0,07
0,13
0,11
0,20
0,16
0,12

1,15
1,07
1,17
1,08
1,26
1,03
1,26
1,09
1,15
1,19
1,24
1,17
1,37
1,18
1,10
1,13

0,26
0,22
0,34

0,04
0,03
0,03
0,03
0,10
0,01
0,15
0,03
0,05
0,09
0,09
0,09
0,19
0,10
0,02
0,02

1,44
1,08
1,34
1,15
1,57
1,14
1,58
1,22
1,34
1,50
1,26
1,73
1,64
1,72
1,56
1,55

0,41
0,70
0,63

0,25
0,02
0,09
0,07
0,23
0,04
0,36
0,09
0,12
0,34
0,08
0,26
0,12
0,37
0,19
0,27

1,05
1,02
1,11
1,03
1,07
1,02
1,11
1,03
1,06
1,09
2,40
1,22
2,49
1,40
1,14
1,18

0,42
0,24
0,40

3,63
2,00
3,46
2,71
2,88
1,92
2,75
2,67
2,21
2,38
3,25
3,46
2,86
2,75
2,71
2,75

3,13
1,92
3,00
2,00
2,42
1,58
2,08
1,29
2,25
2,46
3,83
2,33
3,54
2,54
1,90
1,58

3,08
1,63
2,79
2,21
2,25
1,29
2,00
1,50
1,92
2,08
2,96
2,46
3,00
2,25
2,25
1,96

0,27
0,74
0,59

TMOS (TOSQA2001; Processed vs. Unprocessed) vs.
auditory S-MOS

TMOS (TOSQA2001; Processed vs. Unprocessed,
Speech Part (0.75 - 3.25s)) vs. auditory S-MOS

5.0

5.0

4.5

4.5

4.0

4.0

3.5

3.5

TMOS

TMOS

0,01
0,00
0,04
0,01
0,02
0,00
0,07
0,01
0,02
0,09
0,21
0,06
0,25
0,09
0,05
0,07

3.0

3.0

2.5

2.5

2.0

2.0
1.5

1.5

1.0

1.0
1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

1.0

5.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Auditory S-MOS

Auditory S-MOS

Figure B.1: TOSQA2001 results (TMOS) of processed data versus auditory S-MOS(unprocessed signal used as TOSQA2001 reference)

ETSI

5.0

48

ETSI EG 202 396-3 V1.7.1 (2018-03)

TMOS (TOSQA2001; Processed vs. Clean Speech) vs.
auditory S-MOS

TMOS (TOSQA2001; Processed vs.Clean Speech,
Speech Part (0.75 - 3.25s)) vs. auditory S-MOS

5.0
5.0

4.5
4.5

4.0

4.0
3.5

3.0

TMOS

TMOS

3.5

2.5

3.0
2.5

2.0

2.0

1.5

1.5

1.0

1.0

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

1.0

1.5

2.0

Auditory S-MOS

2.5

3.0

3.5

4.0

4.5

5.0

Auditory S-MOS

Figure B.2: TOSQA2001 results (TMOS) of processed data versus auditory S-MOS(clean speech signal used as TOSQA2001 reference)

MOS-LQO (PESQ / P.862.2; Processed vs.
Unprocessed, Speech Part (0.75 - 3.25s)) vs. auditory SMOS

5.0

5.0

4.5

4.5

4.0

4.0

3.5

3.5

MOS-LQO

MOS-LQO

MOS-LQO (PESQ / P.862.2; Processed vs.
Unprocessed) vs. Auditory S-MOS

3.0
2.5

3.0
2.5

2.0

2.0

1.5

1.5

1.0

1.0
1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

1.0

Auditory S-MOS

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Auditory S-MOS

Figure B.3: PESQ (MOS-LQO, P.862.2) results of processed data versus auditory S-MOS(unprocessed signal used as PESQ reference)

ETSI

49

ETSI EG 202 396-3 V1.7.1 (2018-03)

MOS-LQO (PESQ / P.862.2; Processed vs.Clean
Speech, Speech Part (0.75 - 3.25s)) vs. auditory S-MOS

5.0

5.0

4.5

4.5

4.0

4.0

3.5

3.5

MOS-LQO

MOS-LQO

MOS-LQO (PESQ / P.862.2; Processed vs. Clean
Speech) vs. auditory S-MOS

3.0

3.0

2.5

2.5

2.0

2.0

1.5

1.5

1.0

1.0

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

1.0

5.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Auditory S-MOS

Auditory S-MOS

Figure B.4: PESQ results (MOS-LQO, P.862.2) of processed data versus auditory S-MOS(clean speech signal used as PESQ reference)

ETSI

50

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex C:
Comparison of objective MOS versus auditory MOS for thecomplete STF 294 database
This annex shows the correlation plots between the objective and the auditory S-/N-/G-MOS for all French data usedduring the training of the new method. Note that the MOS scores for all conditions of the STF 294 database (see ETSIEG 202 396-2 [i.2]) are compared to the listening test results, including also rejected samples as described in clause 5.
Figures C.1, C.3 and C.5 show the results for the French training data and figures C.2, C.4 and C.6 or the French
validation data. In order to distinguish between the selected data and the ones which were not used for the model
development, the conditions not used (rej.) are indicated by a "+" (red) and the accepted (acc.) by a " " (blue). 123
training and 49 validation conditions were rejected for the development of the model.
For the training data, the correlation for the objective N-MOS decreases to 89,6 %. This lower performance could becompensated with a mapping function. The correlation of the objective N-MOS to the auditory N-MOS for thevalidation data even increases slightly to 93,0 %.
STF294 database - training set rejectected/selected

STF294 database - validation set rejectected/selected

55rmse = 0.401

rmse = 0.357

4
Objective N-MOS

Objective N-MOS

4

3

2

2

N-MOS selected
N-MOS rejected
Mapping Function [1] (r = 0.896)
1
1

2

3

4

3

N-MOS selected
N-MOS rejected
Mapping Function [1] (r = 0.930)

5

Auditory N-MOS

1
1

2

3

4

Auditory N-MOS

Figure C.1: Objective vs. auditory N-MOS for all
French training conditions

Figure C.2: Objective vs. auditory N-MOS for all
French validation conditions

The correlation of the objective to the auditory S-MOS data decreases to 89,6 for the training data and but increases to93,0 % for the validation data.

ETSI

5

51

ETSI EG 202 396-3 V1.7.1 (2018-03)
STF294 database - validation set rejectected/selected

STF294 database - training set rejectected/selected

55rmse = 0.409

rmse = 0.383

4
Objective S-MOS

Objective S-MOS

4

3

3

2

2

S-MOS selected
S-MOS rejected
Mapping Function [1] (r = 0.880)

S-MOS selected
S-MOS rejected
Mapping Function [1] (r = 0.886)
1
1

2

3
Auditory S-MOS

4

1
1

5

Figure C.3: Objective vs. auditory S-MOS for all
French training conditions

2

3

4

5

Auditory S-MOS

Figure C.4: Objective vs. auditory S-MOS for all
French validation conditions

In consequence, the correlation between the objective and auditory G-MOS decreases also only slightly decreases to92,6 % for the training data. Again for the validation data the correlation increases to 92,2 % as shown in figure C.6.
STF294 database - validation set rejectected/selected

STF294 database - training set rejectected/selected

55rmse = 0.298

rmse = 0.292

4
Objective G-MOS

Objective G-MOS

4

3

2

2

G-MOS selected
G-MOS rejected
Mapping Function [1] (r = 0.922)

G-MOS selected
G-MOS rejected
Mapping Function [1] (r = 0.926)
1
1

3

2

3

4

5

Auditory G-MOS

Figure C.5: Objective vs. auditory G-MOS for all
French training conditions

1
1

2

3

4

Auditory G-MOS

Figure C.6: Objective vs. auditory G-MOS for all
French validation conditions

Generally it can be concluded that the selection process on the French data made the training set more consistent.
Training prediction results on the accepted samples showed to be more accurate. On the other hand, including therejected validation data would have even increased the prediction performance.

ETSI

5

52

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex D:
Comparison of objective MOS versus auditory MOS forrejected conditions
For information purpose, figures D.1 to D.6 show the correlation plots for the objective and auditory S-/N-/G-MOS
only for the rejected conditions of both training and validation set (see also annex C). Again the data not used for themodel development (123 training, 49 validation conditions) are indicated by a "+" in the scatter plots.
STF294 database - training set rejectected only

STF294 database - validation set rejectected only

5

5

rmse = 0.419

rmse = 0.354

4
Objective N-MOS

Objective N-MOS

4

3

2

3

2
N-MOS rejected
Mapping Function [1] (r = 0.873)

1
1

2

3

4

N-MOS rejected
Mapping Function [1] (r = 0.945)
1
1

5

Auditory N-MOS

4

5

Figure D.2: Objective vs. auditory N-MOS only forvalidation data not used for the model
development

STF294 database - training set rejectected only

STF294 database - validation set rejectected only

5

5

rmse = 0.420

rmse = 0.457

4
Objective S-MOS

4

3

2

3

2
S-MOS rejected
Mapping Function [1] (r = 0.867)

1
1

3
Auditory N-MOS

Figure D.1: Objective vs. auditory N-MOS only fortraining data not used for the model development

Objective S-MOS

2

2

3

4

S-MOS rejected
Mapping Function [1] (r = 0.867)
5

Auditory S-MOS

1
1

2

3

4

Auditory S-MOS

Figure D.3: Objective vs. auditory S-MOS only fortraining data not used for the model development

ETSI

Figure D.4: Objective vs. auditory S-MOS only
for validation data not used for the model
development

5

53

ETSI EG 202 396-3 V1.7.1 (2018-03)

STF294 database - training set rejectected only

STF294 database - validation set rejectected only

5

5
rmse = 0.319

rmse = 0.269

4
Objective G-MOS

Objective G-MOS

4

3

2

3

2
G-MOS rejected
Mapping Function [1] (r = 0.924)

1
1

2

3

4

G-MOS rejected
Mapping Function [1] (r = 0.943)
5

Auditory G-MOS

1
1

2

3

4

5

Auditory G-MOS

Figure D.5: Objective vs. auditory G-MOS only fortraining data not used for the model development

ETSI

Figure D.6: Objective vs. auditory G-MOS only
for validation data not used for the model
development

54

Annex E:
Void

ETSI

ETSI EG 202 396-3 V1.7.1 (2018-03)

55

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex F:
Detailed STF 294 subjective and objective validation test results
Table F.1 contains the conditions and related auditory and objective S-MOS, N-MOS and G-MOS for French language. Also standard deviations for all MOS scores are given.
Table F.1: Subjective and objective experiment results - French validation part
(Recording: hs - handset, hf - hands-free. Speaker: f - female, m - male)
FRENCH
Conditions

Id
real

Noise

Recording Speaker

NetworkVAD Smooth dB
no
NSA
no
no
no
no
yes
yes
no
no
no
no
no
yes
yes
yes
yes
no
no
yes
yes
yes
no
no
yes
yes
yes
no
yes

Speech11

Lux_Car

4
6
9
10
22
24
31
34
37
39
42
43
48
49
52
63
67
78
79
81
90
94
99
102
103
114
132

4
6
9
10
22
24
31
34
37
39
42
43
48
49
52
63
67
78
79
81
90
94
99
102
103
114
132

Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Crossroads
Crossroadshsf

AMR _NI

hs
hs
hs
hs
hs
hs
hf
hf
hf
hf
hf
hf
hf
hf
hf
hs
hs
hs
hs
hs
hf
hf
hf
hf
hf
hs
hs

f
f
f
f
f
f
f
f
f
f
f
f
f
f
f
m
m
m
m
m
m
m
m
m
m
f
f

AMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIIIAMR _NIIIAMR _NIAMR _NIIIAMR _NIII

no NSAno
NSA
Smooth 9
Smooth 9
Smooth 18
Sharp
9
Sharp
9
Sharp
9
Smooth 9
Smooth 18
Sharp
9
Sharp
9
Sharp
18
Smooth 9
Smooth 18
Sharp
9
Sharp
18
Smooth 18
Sharp
18
Sharp
9
Sharp
18
Sharp
18
Smooth 18
Sharp
18
Smooth 9
Smooth 18
Sharp
9
Smooth 9
Sharp
9

ETSI

Subjective
MOS
Noise
GlobalStandard deviation
Speech Noise
GlobalObjective
MOS
Speech Noise

Global

4,42

3,67

3,96

0,65

0,64

0,36

4,29

2,85

3,46

4,88
3,50
3,46
4,54
4,42
3,79
3,79
3,04
3,21
2,71
1,92
3,96
2,33
3,71
2,67
3,21
3,71
2,96
3,96
3,04
2,38
2,88
3,04
2,54
3,63
3,04
3,46

3,63
3,21
3,31
3,46
3,50
3,29
2,21
2,29
2,04
1,71
1,92
2,00
1,83
2,17
2,04
3,29
3,60
2,79
3,58
3,29
2,13
2,42
2,46
2,21
2,46
2,53
2,96

3,92
3,08
3,08
3,63
3,63
3,21
2,75
2,42
2,29
1,96
1,71
2,54
1,79
2,54
1,96
2,67
3,44
2,50
3,46
2,63
1,88
2,38
2,25
2,08
2,58
2,31
3,00

0,45
1,07
1,18
0,59
0,58
0,83
1,02
1,00
0,98
1,04
0,88
0,91
0,76
0,86
0,92
1,02
1,08
1,00
0,75
1,00
1,17
0,90
0,86
1,18
0,71
1,04
0,83

0,71
0,72
0,62
0,66
0,72
0,69
0,88
0,75
0,69
0,69
0,72
0,83
0,70
0,76
0,62
0,81
0,71
0,66
0,78
0,81
1,03
0,88
0,72
0,93
0,88
0,88
0,75

0,65
0,88
0,93
0,71
0,71
0,78
0,74
0,78
0,55
0,62
0,62
0,78
0,66
0,66
0,69
0,81
0,65
0,78
0,59
0,88
0,90
0,88
0,74
0,97
0,78
0,80
0,72

4,63
3,34
3,08
4,64
4,42
3,38
2,97
2,44
2,99
2,46
1,92
2,90
2,31
2,88
2,71
3,08
4,08
3,17
3,39
3,09
2,35
2,93
3,00
2,88
3,51
2,83
2,99

2,77
2,69
2,96
2,80
2,93
2,77
2,09
2,28
2,09
1,94
2,05
2,13
2,06
2,12
2,56
2,84
3,00
2,52
3,36
3,07
2,05
2,31
2,09
2,15
2,19
2,42
2,79

3,69
2,68
2,63
3,72
3,61
2,76
2,09
1,82
2,09
1,65
1,34
2,06
1,61
2,04
2,16
2,57
3,39
2,47
3,06
2,70
1,64
2,18
2,11
2,05
2,53
2,17
2,48

56

ETSI EG 202 396-3 V1.7.1 (2018-03)
FRENCH

Conditions

Id
real

Noise

Recording Speaker

NetworkVAD Smooth dB
no
NSA
no
no
no
NSA
no
NSA
no
no
no
yes
yes
yes
yes
no
NSA
no
no
yes
no
no
no
yes
yes
no
NSA
no
NSA
no
no
no
no
no
yes
yes
yes
yes
no
NSA
no
no

SpeechCrossroadshff

AMR _NI

Crossroads
Crossroads
Crossroads

hf
hf
hs

f
f
m

AMR _NIIIAMR _NIIIAMR _NI

Crossroadshsm

AMR _NIII

Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads

hs
hs
hs
hs
hs
hs
hs
hf

m
m
m
m
m
m
m
m

AMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIII

hf
hf
hf
hs
hs
hs
hs
hs
hs

m
m
m
f
f
f
f
f
m

AMR _NIIIAMR _NIIIAMR _NIIIAMR _NIIIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIhsm

AMR _NIII

hs
hs
hs
hs
hs
hs
hs
hs
hs
hf

m
m
m
f
f
f
f
f
f
f

AMR _NIAMR _NIIIAMR _NI
G722_NIII
G722_NI
G722_NI
G722_NI
G722_NIII
G722_NIII

136
141
150

136
141
150

163

163

165
171
172
177
181
184
186
187

165
171
172
177
181
184
186
187

192
201
204
213
225
226
229
238
241

192
201
204
213
225
226
229
238
241

Crossroads
Crossroads
Crossroads
Road_Noise
Road_Noise
Road_Noise
Road_Noise
Road_Noise

244

271

Road_Noise

246
247
249
256
276
277
289
292
294
297

273
274
276
283
330
331
343
346
348
351

Road_Noise
Road_Noise
Road_Noise
Road_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise

298
301
309

352
355
363

Office_Noise
Office_Noise
Office_Noise

hf
hf

f
f

G722_NI
G722_NI
G722_NIII

no NSAno
NSA
Smooth 9
Sharp
18
no NSA no
NSA
no NSA no
NSA
Smooth 18
Sharp
9
Sharp
18
Smooth 18
Sharp
9
Sharp
9
Sharp
18
no NSA no
NSA
Sharp
9
Sharp
18
Sharp
9
Smooth 18
Sharp
9
Sharp
18
Sharp
9
Sharp
18
no NSA no
NSA
no NSA no
NSA
Smooth 9
Smooth 9
Sharp
18
Smooth 9
Smooth 18
Smooth 18
Sharp
9
Sharp
9
Sharp
18
no NSA no
NSA
Smooth 9
Sharp
9
ETSI

Subjective
MOS
Noise
GlobalStandard deviation
Speech Noise
GlobalObjective
MOS
Speech Noise

Global

2,66

1,67

1,92

1,27

0,96

0,93

3,14

2,07

2,19

2,79
1,92

1,29
1,67

1,54
1,42

1,28
0,78

0,46
0,82

0,72
0,58

2,01
1,99

1,65
2,06

1,20
1,40

3,92

2,46

2,67

1,14

0,93

0,87

3,97

2,41

3,00

2,83

2,38

2,17

1,27

0,98

1,01

2,83

2,34

2,12

2,42
3,88
2,48
3,04
3,96
3,75
2,96

3,00
2,42
3,00
3,08
2,57
2,58
3,50

2,21
2,83
2,46
2,83
3,29
2,67
2,88

1,02
0,85
0,88
1,08
1,08
1,22
1,27

1,18
0,93
1,25
1,14
1,06
1,10
1,10

0,93
0,92
0,88
0,92
0,75
1,13
1,15

2,55
4,07
2,67
3,01
4,06
3,31
3,28

3,01
2,34
2,87
3,00
2,57
2,45
3,33

2,28
3,03
2,28
2,60
3,15
2,53
2,96

1,77

1,42

1,46

0,98

0,72

0,59

2,36

1,95

1,58

2,29
1,63
2,13
2,29
3,67
3,17
3,71
2,96

1,38
1,88
1,42
2,46
2,54
2,50
2,29
3,13

1,71
1,38
1,46
2,17
2,88
2,58
2,70
2,46

1,23
0,97
0,95
0,62
0,96
1,09
0,95
1,30

0,58
1,15
0,58
0,78
0,72
0,88
0,62
0,85

0,69
0,65
0,59
0,64
0,80
0,72
0,75
0,83

2,05
1,77
2,13
2,46
3,34
2,70
3,52
3,49

1,63
1,98
1,80
2,49
2,19
2,64
2,56
3,42

1,21
1,20
1,34
1,94
2,41
2,19
2,74
3,16

3,75

2,00

2,49

0,85

0,93

0,84

3,40

2,08

2,39

2,46

1,54

1,67

1,28

0,88

0,87

2,56

2,01

1,75

3,33
1,38
2,75
3,29
4,75
4,54
4,83
3,54
2,67

1,54
1,21
2,17
3,71
3,83
3,63
4,13
3,38
3,88

2,29
1,00
2,08
3,09
4,04
3,67
4,33
3,17
2,79

1,09
0,97
0,99
0,95
0,53
0,59
0,48
1,14
0,96

0,66
0,83
0,82
0,69
0,82
0,65
0,45
0,71
0,61

0,75
0,00
0,88
0,72
0,81
0,76
0,64
0,76
0,72

3,44
2,46
2,86
3,13
4,69
4,71
4,70
3,48
3,18

1,88
1,37
2,20
3,28
3,38
3,51
3,63
3,21
3,38

2,31
1,34
2,07
2,82
4,05
4,13
4,17
3,04
2,91

4,21

2,96

3,21

0,72

0,69

0,78

4,04

2,58

3,13

4,21
2,54

3,00
2,50

3,08
2,33

0,93
0,78

0,78
0,51

0,65
0,48

4,24
2,82

2,66
2,43

3,33
2,17

57

ETSI EG 202 396-3 V1.7.1 (2018-03)
FRENCH

Conditions

Id
real

Noise

310
316

364
370

Office_Noise
Office_Noise

327
331
334
366
372
373
378

381
385
388
420
426
427
432

Office_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise
Office_Noise

379
384
390
396
415
420
423
427
432

433
438
444
450
469
474
477
481
486

Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise
Pub_Noise

Recording Speaker
hf
hf
hs
hs
hs
hf
hf
hf
hf
hs
hs
hs
hs
hs
hs
hs
hs
hs

f
f
m
m
m
m
m
m
m
f
f
f
f
m
m
m
m
m

NetworkVAD Smooth dB

G722_NI
G722_NI

no
yes
no
NSA
no
no
no
yes
yes
yes
no
NSA
no
no
yes
no
no
yes
yes
yes

G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NIII
G722_NIII
G722_NI
G722_NIII
G722_NIII
G722_NI
G722_NIII

Sharp
Smoothno NSA

18
18
no
NSA
Smooth 18
Sharp
9
Sharp
18
Smooth 18
Sharp
9
Sharp
18
no NSA no
NSA
Smooth 9
Sharp
9
Smooth 9
Sharp
9
Sharp
18
Smooth 9
Sharp
9
Sharp
18

ETSI

Speech3,383,75Subjective
MOS
Noise
Global
2,83
2,75
2,75
2,79

Standard deviation
Speech Noise
Global
1,17
0,87
0,79
1,03
0,79
0,98

Objective
MOS
Speech Noise
4,05
2,97
3,79
3,14

Global
3,35
3,24

3,50

3,71

3,25

1,10

0,62

0,94

3,15

2,80

2,60

4,83
4,71
2,79
3,00
4,67
2,88

4,08
3,88
2,33
2,52
3,17
2,67

4,08
4,08
2,17
2,30
3,38
2,38

0,48
0,69
1,25
1,14
0,48
0,95

0,65
0,54
0,76
0,77
0,56
0,96

0,65
0,88
0,76
0,80
0,65
0,82

4,59
4,58
3,12
3,00
4,28
3,09

3,38
3,17
2,66
2,88
2,73
2,71

3,96
3,86
2,50
2,52
3,40
2,50

4,04

2,08

2,58

0,91

0,88

0,65

3,60

2,18

2,59

3,00
2,42
2,38
3,96
1,88
2,46
3,79
1,96

1,63
1,79
1,79
1,67
1,21
1,75
1,99
1,88

1,92
1,79
1,67
2,08
1,38
1,89
2,25
1,54

1,02
1,02
0,88
1,08
0,90
0,98
1,06
0,95

0,65
1,06
0,66
0,96
0,41
0,68
0,92
0,68

0,58
0,66
0,87
0,88
0,49
0,75
0,79
0,66

2,82
2,71
2,74
4,15
2,81
2,89
3,74
2,70

1,74
1,84
1,96
1,80
1,68
1,97
2,34
2,50

1,78
1,76
1,85
2,79
1,73
1,96
2,79
2,12

58

Annex G:
Void

ETSI

ETSI EG 202 396-3 V1.7.1 (2018-03)

59

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex H:
Extension of the Speech Quality Test Method toNarrowband: Adaptation, Training and Validation
The first version of the present document was restricted to wideband application only. Due to the lack of freely
available databases containing narrowband speech and evaluated according to Recommendation ITU-T P.835 [i.3], a
new database including 263 conditions was created. This database includes a wide variety of different impairments
found in today's communication systems including mobile and stationary handset/hands-free terminals.
The annex describes the adaptation of the model to narrowband scenarios, as well as some adjustments in thecalculation and pre-processing which had to be done but without modifying the main principles of the algorithm.
Design of the new database
The base for each objective model is a database, containing speech samples (with references) and subjective
MOS-LQSN scores from listening tests. The output scores of the model are always related to these subjective ratings.
For an extension to narrowband mode, a new database had to be designed. The database from the ETSI STF 294 project(see ETSI EG 202 396-2 [i.2]) allowed the prediction of wideband speech based on French (and Czech) speech
sequences. Based on the good experience with the well-balanced distribution of background noises and handset/handsfree modes in this old database, the new database was designed in a similar way.
General design of Recommendation ITU-T P.835 [i.3] listening-only test.
Table H.1: Comparison of Databases
ETSI STF WB Database
Language
Speakers
Different speakers
Training
Validation

1 male or 1 female
per condition
2
179
81

HEAD acoustics NB Database
English2 male and 2 female
per condition
8
216
50

Table H.2: Distribution of conditions according background noiseand handset/hands-free mode

Overall
Background Noises:
Car
CrossroadRoad
OfficePub/CaféETSI STF WB Database
HandsetHands-free
116
63
23
18
25
27
23

HEAD acoustics NB Database
HandsetHands-free
200
66

22
18
0
23
0

40
36
43
39
42

25
8
9
13
10

In the ETSI STF 294 project (see ETSI EG 202 396-2 [i.2]), all conditions were simulated offline. In the newnarrowband database, 184 of 266 conditions were recorded from real devices in sending direction, 82 conditions werealso simulated offline in the same way like in the STF 294 project:•

Recording of "Unprocessed Signal" at position of DUT.

•

Background Noise Simulation according ETSI ES 202 396-1 [i.1].

ETSI

60

•

ETSI EG 202 396-3 V1.7.1 (2018-03)

Simulation Steps:-

IRS SND Filter.

-

Speech Enhancements/Noise Reduction.

-

Coder + Decoder.

To simulate typical communication systems, the following processing steps were used:••"Speech Enhancement":-

Different MMSE Algorithms.

-

Different Algorithms with spectral subtraction.

-

Without any processing.

Coder + Decoder:-G.726, G.729A.

-

iLBC.

-

Speex HiQ/LQ.

-

Without any coding/decoding.

Presentation of speech material in listening test
The listening test for the new database was performed according to Recommendation ITU-T P.835 [i.3], where naïve
listeners give three different votes (S-, N- and G-MOS for speech, noise and global quality) to a single sample.
Compared to the STF 294 project (see ETSI EG 202 396-2 [i.2]), some moderate changes based on the experience fromthe STF 294 project were introduced in the procedure. The design differences between the STF 294 database (see ETSIEG 202 396-2 [i.2]) and the new database are listed in table H.3.
Table H.3: Design differences between the STF 294 database
ETSI STF WB Database
HEAD acoustics NB Database
Sentences/Sample
1
2
Duration of Sample
4s
8-9s
Samples/Condition
6
4
Votes/Sample
4
6
Votes/Condition
24
24
Diotic ASL
79 dB SPL
73 dB SPL
Pre-Filtering
None/flat
IRS RCVNOTE:
See ETSI EG 202 396-2 [i.2] and the new database mode.

The main differences are:•

The amount of speech and noise parts in each sample was increased, so that the vote is more reliable.

•

The listening level of active speech was decreased from 79 dB SPL to 73 dB SPL - an expert test led to theconclusion that this diotic level is preferred by listeners over a large range of signal-to-noise ratios.

•

The narrowband speech material was prefiltered with an IRS RCV - simulates a reference listening system
according to Recommendation ITU-T P.800 [i.4].

ETSI

61

Annex I:
Void

ETSI

ETSI EG 202 396-3 V1.7.1 (2018-03)

62

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex J:
Summary of Czech samples not used for model training
J.0

Introduction

Almost all topics regarding the Czech database presented in ETSI EG 202 396-2 [i.2] were already marked as invalid orat least not recommended to use, mainly due to the insufficient audio material and the corresponding subjective
database. Thus it was decided to move all related content regarding this topic from the main body of a previous version
to this annex for informational purposes.
Due to the high diversity between the Czech and the French listening test the development of the objective model isbased only on the French database being within the ToR and such provides the higher amount of selected samples.

J.1

Selection process - Czech database

For every combination of background noise and speaker gender, a single Czech sentence was used (see table J.1). The
24 Czech listeners had to rate this single sentence, while the French ratings are a mean value of six different sentences(assessed by 4 listeners each).
Table J.1: Sentences from the test corpus
chosen for the different conditions
Condition
Lux Car 130 kmh Female2
Lux Car 130 kmh Male1
Crossroads Female2
Crossroads Male1
Road Noise Female2
Road Noise Male1
Office Noise Female2
Office Noise Male1
Pub Noise Female2
Pub Noise Male1

Sentence No.
S3
S2
S4
S3
S5
S4
S6
S5
S7
S6

This leads to a limited representation of the individual background noise conditions especially in the case of timevarying background noises. Furthermore the NII conditions were even more critical in judgement compared to the
French data since either there was no packet loss at all or, if there was packet loss, all listeners rated this particular
packet loss because they all listened to the same sentence for one condition. In the French listening test 6 sentenceswere listened for one condition which provided a higher variance of the distributed packet loss.
The listening level variation in the Czech database, preserved from previous database processing adds another degree ofcomplexity to the problem. The listening levels are generally lower as within the French database and as compared to
the general rules laid down in ITU-Recommendations P.800 [i.4] and P.835 [i.3]. The listening level variation within
the Czech database is up to 16 dB. In the experts tests the following conclusions were drawn:•

The conditions AMR NII and G.722 NII (1 % packet loss) were not selected, because in most cases, the soundfiles had too low packet loss. A distinction between and NI and NII conditions is hardly possible.

•

The effect of packet loss in the samples should be audible in AMR NIII and G.722 NIII conditions. Since
every single Czech condition consists just of one sentence, the packet loss may not be distributed uniformly inthe sample. Therefore, only samples with at least one packet loss in speech and background noise (before orafter speech) were selected.

•

Due to the fact that every Czech sound file has a different level (which depends on codec, noise reduction
algorithm, etc.), a minimum level of 69 dB SPL was set (10 dB below the recommended listening level of
79 dB SPL). All conditions below this limit were not retained.

ETSI

63

•

ETSI EG 202 396-3 V1.7.1 (2018-03)

Analysis of NI conditions:a)

AMR Codec:
70 conditions were not retained based on the following selection criteria:b)

1)

Too low level (54).

2)

Inconsistent BGN level (12).

3)

Too low S/N (2).

4)

Too low overall level/given listening level not correct (2).

G.722 Codec:
19 conditions were not retained based on the following selection criteria:c)1)

Too low level (15).

2)

MOS values irreproducible (4).

Selected conditions dependent of BGN: see table J.2.
Table J.2: Selected Czech NI conditions

BGN-Condition

Total notretained

Total
retained

Lux_Car
Crossroads
Road
Office
Pub

17
36
17
14
5

19
0
1
22
13

d)
•

Selected test
samples/MOS
available
10
0
1
16
10

Selected verification
samples/no MOS
available
9
0
0
6
3

Overall NI acceptance: 48 % of NI conditions are useful (22 % AMR, 65 % G.722).

Analysis of NIII conditions:a)

AMR Codec:
76 conditions were not retained based on the following selection criteria:b)

1)

Too low level (43).

2)

Inconsistent packet loss (33).

G.722 Codec:
35 conditions were not retained based on the following selection criteria:c)1)

Too low level (13).

2)

Inconsistent packet loss (22).

Selected samples dependent of BGN: see table J.3.

ETSI

64

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table J.3: Selected Czech NIII conditions
BGN-Condition

Total notretained

Total
retained

Lux_Car
Crossroads
Road
Office
Pub

30
30
16
24
11

6
6
2
12
7

d)

Selected test
samples/MOS
available
4
5
2
10
2

Selected verification
samples/no MOS
available
2
1
0
2
5

Overall NIII acceptance: 23 % of NIII conditions are useful (16 % AMR, 35 % G.722).

The list of the selected Czech conditions is found in table A.1.
In total 88 conditions out of 432 (20,4 %) are suited to be used in a further step for checking language dependencies.

J.2

General differences between the databases

The most important differences between the French and the Czech database can be summarized as follows:•

The French and Czech listening samples of one condition do not have the same levels. The French sound files
are louder than the Czech ones, in some random tests, the mean of these level differences is given in table A.2of ETSI EG 202 396-2 [i.2]. This may have lead to different ratings for the Czech samples compared to the
French samples. This has to be regarded especially for further processing of the sound files.

•

For every background noise condition, a single Czech sentence was used (see table J.1). To quantify the lastpoint, the correlation between French and Czech ratings (S-, N- and G-MOS) can be calculated. As shown
below, this correlation is very low. It seems that the differences mentioned above are reflected here.
Coefficients of correlation (Pearson's equation) are summarized in table J.4.

r=

with:

x
x
y
y

∑ (x − x )( y − y )∑ (x − x ) ∑ ( y − y )
2

2

(J.1)

MOS Data (Czech)
Mean of MOS Data (Czech)
MOS Data (French)
Mean of MOS Data (French)

Table J.4: Comparison of correlation

Over all available ratings(French and Czech, 302 condition each)

Only selected French MOS
Data (NI and NIII conditions, ratings
reviewed by experts)(179 selected French conditions)

S-MOS: 0,703
N-MOS: 0,816
G-MOS: 0,668

S-MOS: 0,736
N-MOS: 0,822
G-MOS: 0,776

Only Czech and French selected MOS
Data(NI and NIII conditions, ratings
reviewed by experts)(59 conditions selected for French and Czech)

S-MOS: 0,830
N-MOS: 0,897
G-MOS: 0,871

As shown in the scatter plots below, a slight correlation for the French-optimized data can be noticed, but for a usable
correlation, the measurement points are distributed too far away from a (virtual) regression line of best fit (seefigures J.1, J.3 and J.5).

ETSI

65

ETSI EG 202 396-3 V1.7.1 (2018-03)

If the calculation of the correlation is limited only to the selected data (86 conditions are selected for French and Czech
speech), the correlation increases for all values, especially for the G-MOS data (see figures J.2, J.4 and J.6).

Figure J.1: Scatter plot of the French data versus the Czech data for the different conditions,
S-MOS, before experts' selection

Figure J.2: Scatter plot of the French data versus the Czech data, S-MOS, after experts' selection(only data selected for both languages)

ETSI

66

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.3: Scatter plot of the French data versus the Czech data for the different conditions,
N-MOS, before experts' selection

Figure J.4: Scatter plot of the French data versus the Czech data, N-MOS, after experts' selection(only data selected for both languages)

ETSI

67

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.5: Scatter plot of the French data versus the Czech data for the different conditions,
G-MOS, before experts' selection

Figure J.6: Scatter plot of the French data versus the Czech data, G-MOS, after experts' selection(only data selected for both languages)

J.3

Comparison of the objective method results for
Czech and French samples

Due to the differences between the Czech and the French listening tests already described in clause B.2 of [i.2] thedatasets for the model generation and validation were completely different in terms of level. While the level of theprocessed French signals was adjusted to 79 dB SPL, the level of the processed Czech signals was left unmodified.
Therefore also the characteristic of the listening tests is different. The processed French signals are much louder (up to
16 dB) than the Czech ones - but all French samples are equal in terms of level: French listeners probably have nottaken into account the absolute overall active speech level of the processed signal. It is very likely that in contrary
Czech listeners took into account the different absolute overall active speech levels.

ETSI

68

ETSI EG 202 396-3 V1.7.1 (2018-03)

This also affects the results of the objectively calculated N-MOS, S-MOS and G-MOS values. As shown in figure 6.5
the level of the processed background noise is one influencing factor for the N-MOS calculation. This level is relatively
high for all French samples. If the N-MOS is now calculated for the Czech samples using the regression coefficientsacquired for the French sentences the resulting objective N-MOS scores are higher than the auditory scores. This is dueto the lower background noise level of the Czech sentences. This could be expected: if a French listener would havelistened to the Czech sentences among the French ones, he would have probably rated them with a higher N-MOS - dueto the lower background noise level.
Figures J.7 and J.8 show the scatter plots for the objectively calculated N-MOS (for the selected French and Czech
samples) versus the auditory N-MOS derived in the corresponding listening tests. The regression coefficients were
optimized for the French dataset in both plots.
As already analysed in clause 6.5.3 the objective N-MOS correlates with 94,8 % to the results of the French listening
test. Figure J.8 shows that the objective N-MOS calculated for the Czech data using the French coefficients do not
sufficiently correlate to the auditory results (correlation of 88,4 %). The results tend to be too good, which is mainly
caused by the lower background noise level of the Czech samples. They would be assessed better by French listeners
than the French samples with the higher level.
For another "cross check" the N-MOS regression algorithm is tuned on the Czech data, and the N-MOS scores are again
calculated for the French and the Czech samples.
Note that for this training of the Czech data not only the selected (60) conditions were used, but also the selectedconditions of network condition 1 (clean network). The disadvantage of this approach is that also conditions with very
low signal levels and irreproducible ratings were considered. The big advantage is that the number of conditions
increases from 60 to 120. This allows a higher numerical stability, especially for the S-MOS calculation, where theamount of conditions is separated in three groups according to the N-MOS. Using only a total of 60 Czech conditions
would lead to a non-stable regression for the S-MOS due to the splitting in three groups. Only 20 conditions per groupare too few to reliably calculate the 11 S-MOS regression coefficients.
The scatter plots are given in figures J.9 and J.10. They show that the objective results for the French data (figure J.9)
tend to be about 1 MOS lower than the auditory results (correlation of 82,1 %) whereas the objective N-MOS scores for
the Czech samples correlate with 98 % to the auditory results (figure J.9). Figure J.9 indicates that a Czech listener
would assess all French samples with a lower N-MOS - probably caused by the higher background noise level.
The conclusion of the scatter plot analysis is that:•

The new objective model is in principle applicable for both databases.

•

Different regression coefficient sets are needed in order to reproduce the different level strategies used in thetwo datasets and listening tests.

Comparable analyses are carried out for S-MOS and G-MOS. The analyses results for the objective S-MOS are given in
figures J.11 to J.14. Figures J.11 and J.14 show that if the regression coefficient set matching to the input data is used,the correlation is high (92,9 % for French data and 96,4 % for Czech data).

ETSI

69

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.7: Objective vs. Auditory N-MOS for
French samples calculated with regression
coefficients optimized for French data

Figure J.8: Objective vs. Auditory N-MOS for
Czech samples calculated with regression
coefficients optimized for French data

Figure J.9: Objective vs. Auditory N-MOS for
French samples calculated with regression
coefficients optimized for Czech data

Figure J.10: Objective vs. Auditory N-MOS for
Czech samples calculated with regression
coefficients optimized for Czech data

ETSI

70

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.11: Objective vs. Auditory S-MOS for
French samples calculated with regression
coefficients optimized for French data

Figure J.12: Objective vs. Auditory S-MOS for
Czech samples calculated with regression
coefficients optimized for French data

Figure J.13: Objective vs. Auditory S-MOS for
French samples calculated with regression
coefficients optimized for Czech data

Figure J.14: Objective vs. Auditory S-MOS for
Czech samples calculated with regression
coefficients optimized for Czech data

If vice versa the coefficients of the other language are used, the correlation for the S-MOS decreases down to 46 %.
Note that the objective S-MOS shown in figures J.9 and J.10 are based on the objective N-MOS which are also
calculated using the "wrong" coefficient set of the other language. This "wrong" N-MOS may be the reason for
ambiguous distribution of the objective S-MOS calculated for the Czech samples using the French coefficient comparedto the auditory S-MOS. The objective S-MOS calculated for the French data using the Czech coefficients tend to belower for auditory S-MOS lower than 3,5. For auditory S-MOS higher than 3,5 the objective S-MOS leads again to
ambiguous results. One reason may again be the higher level of the French data.

ETSI

71

Figure J.15: Objective vs. Auditory G-MOS for
French samples calculated with regression
coefficients optimized for French data

Figure J.17: Objective vs. Auditory G-MOS for
French samples calculated with regression
coefficients optimized for Czech data
(N-MOS and S-MOS optimized for Czech data)ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.16: Objective vs. Auditory G-MOS for
Czech samples calculated with regression
coefficients optimized for French data (N-MOS
and S-MOS optimized for French data)

Figure J.18: Objective vs. Auditory G-MOS for
Czech samples calculated with regression
coefficients optimized for Czech data

The analysis for the objective G-MOS are shown with the same principle in figures J.15 to J.18. For both datasets using
their optimized coefficient set the correlation is higher than 95 %. Note that the objective G-MOS calculation using the"wrong" coefficients was based on also the wrong N-MOS and S-MOS coefficients. This cumulated error leads tocorrelations of only 79 % and 81 % respectively.

ETSI

72

ETSI EG 202 396-3 V1.7.1 (2018-03)

J.4

Czech conditions results analysis

J.4.1

Comparing subjective and objective N-MOS results

All selected Czech conditions were used for this mapping - independent of the language and the network condition.
Figures J.19 and J.20 show that the per sample deviation between the subjective and the objective N-MOS is less than0,5 MOS for nearly all (27 out of 28) conditions. This results in an overall correlation of 95,9 % (R=0,959 very near to
1 with a confidence interval [0,912, 0,981]). The Spearman Correlation Coefficient is 0,961 and the Kendall Tau is0,856, both of them are near to 1.

Figure J.19: Objectively calculated N-MOS vs.
auditory N-MOS for Czech validation conditions

Figure J.20: Objectively CDF of residual error vs.
N-MOS Error e for Czech validation conditions

For this situation, the RMSE value is 0,293 and the distribution of the residual error is shown in figure J.20 where the
N-MOS Error e is lower than 0,25 for approximately 47 % of the conditions and lower than 0,55 for 99 % for all
conditions.

J.4.2

Comparing subjective and objective S-MOS results

All selected Czech conditions were used for this mapping - independent of the language and the network condition.
Figures J.21 and J.22 show that the per sample deviation between the subjective and the objective S-MOS is less than0,5 MOS for nearly all (25 out of 28) conditions. This results in an overall correlation of 94,3 % (R=0,943 near to 1
with a confidence interval [0,879, 0,974]). The Spearman Correlation Coefficient is 0,930 and the Kendall Tau is 0,808,
both of them are near to 1.

ETSI

73

Figure J.21: Objectively calculated S-MOS vs.
auditory S-MOS for Czech validation conditions

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.22: Objectively CDF of residual error vs.
S-MOS Error e for Czech validation conditions

For this situation, the RMSE value is 0,22 and the distribution of the residual error is shown in figure J.22 where the
N-MOS Error e is lower than 0,25 for approximately 58 % of the conditions and lower than 0,77 for 99 % for all
conditions.

J.4.3

Comparing Subjective and Objective G-MOS Results

All selected Czech conditions were used for this mapping - independently of the language and the network condition.
Figures J.23 and J.24 show that the per sample deviation between the subjective and the objective G-MOS is less than0,5 MOS for nearly all (25 out of 28) conditions. This results in an overall correlation of 94,9 % (R=0,949 near to 1
with a confidence interval [0,892, 0,976]). The Spearman Correlation Coefficient is 0,935 and the Kendall Tau is 0,793,
both of them are near to 1.

Figure J.23: Objectively calculated S-MOS vs.
auditory G-MOS for Czech validation conditions

Figure J.24: Objectively CDF of residual error vs.
G-MOS Error e for Czech validation conditions

For this situation, the RMSE value is 0,21 and the distribution of the residual error is shown in figure J.24 where the
G-MOS Error e is lower than 0,25 for approximately 50 % of the conditions and lower than 0,65 for 99 % for all
conditions.

ETSI

74

J.5

ETSI EG 202 396-3 V1.7.1 (2018-03)

Language Dependent Robustness of G-MOS

The listening tests carried out with French and Czech subjects used in principle the same database, but different levelstrategies. The French listening examples were all played back with the same active speech level of 79 dB SPL (seeRecommendation ITU-T P.56 [i.22]), whereas the Czech listening examples had different play back levels reflecting thelevel and level differences after the processing (see also clause B.2 of [i.2]).
The listening tests in two different languages were originally carried out in order to verify language dependencies forthe new objective method. Due to the different level strategies it is not possible to use the same regression coefficientsof the new model for calculating N-MOS and S-MOS for both languages (see clause B.2 of [i.2]). However the G-MOS
regressions for both, Czech and French data, can be used in order to verify whether Czech and French listeners
combined speech and noise quality to a "global" quality in the same way or if there are significant differences.
The G-MOS is therefore again calculated for Czech and French data. As input parameters N-MOS and S-MOS are usedbased on the individual ("correct") coefficient set. In other words, S-MOS and N-MOS for the French data arecalculated using the corresponding French coefficients and vice versa. The G-MOS is then finally calculated using thecoefficients of the other language each.
The results are given in figures J.25 and J.26.They show that the correlation between objective and auditory G-MOS isstill higher than 94 % in both cases. This means, the final calculation of the G-MOS is very similar for both datasets andlevel strategies - if N-MOS and S-MOS consider all listening perception influences including levels. This indicates that- independent of the listening level strategy - Czech and French listeners combined speech and noise quality in a similarmanner to the global quality.

Figure J.25: Objective vs. Auditory G-MOS for
French samples calculated with regression
coefficients optimized for Czech data (N-MOS andS-MOS optimized for French data)

Figure J.26: Objective vs. Auditory G-MOS for Czech
samples calculated with regression coefficients
optimized for French data (N-MOS and S-MOS
optimized for Czech data)

This effect can also be proved by comparing the G-MOS regression planes for the Czech and French coefficients asgiven in figures J.27 and J.28. The G-MOS regression planes for French and Czech coefficients are very similar. This
indicates that the G-MOS dependency of S-MOS and N-MOS is similar for both languages.

ETSI

75

ETSI EG 202 396-3 V1.7.1 (2018-03)

Figure J.27: Comparison of French (left, blue) and Czech (right, green) regression plane

Figure J.28: Comparison of French (blue) and Czech (green) regression plane

J.6

Regression Coefficients for Czech data

This clause summarizes the regression coefficients for the S-, N- and G-MOS calculation of the Czech data.
Table J.5: Coefficients for linear, quadratic N-MOS regression algorithm (Czech)
Order

c0

cBGN
(NBGN)

cj1(vRABGN, u)

cj2(vRABGN, p)

1
2

0,6733--0,0908-

3,0159-0,5760

0,2811-0,0334

ETSI

cj3

cj4

(vΔRABGN,p-u)-0,3802
0,0578(mΔRABGN,p-u)-6,2485
2,0176

cj5(mRABGN,p)0,2150-0,1686

76

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table J.6: Coefficients for linear, quadratic S-MOS regression algorithm,
N-MOS ≤ N-MOSlow = 2,25 (Czech)
Order

1cj0

1
2

9,7860
-

1cj1

1cj2

(ΔSNR)-0,0211-(mRASP,p)0,0835
0,0936

1cj3

1cj4

1cj5

1cj6

(mΔRASP,p-c)
1,70080,0779(mΔRASP,p-u)-1,1706-0,4926(vΔRASP,p-c)-0,02890,0008(vΔRASP,p-u)-0,2701
0,0022

Table J.7: Coefficients for linear, quadratic S-MOS regression algorithm,
N-MOSlow < N-MOS < N-MOShigh (Czech)
Order

2cj0

1
2

2,2623
-

2cj1

2cj2

(ΔSNR)-0,0283-(mRASP,p)1,7981-0,2587

2cj3

2cj4

2cj5

2cj6

(mΔRASP,p-c)-1,1318-0,1753(mΔRASP,p-u)-1,2940-0,9927(vΔRASP,p-c)-0,1389
0,0022(vΔRASP,p-u)-0,2207
0,0051

Table J.8: Coefficients for linear, quadratic S-MOS regression algorithm,
N-MOS ≥ N-MOShigh = 3,0 (Czech)
Order

3cj0

1
2

4,2104
-

3cj1

3cj2

(ΔSNR)-0,0371-(mRASP,p)1,9003-0,2983

3cj3(mΔRASP,p-c)-0,2506-0,0167

3cj4(mΔRASP,p-u)-0,5132-0,3223

3cj5(vΔRASP,p-c)-0,2349
0,0031

3cj6(vΔRASP,p-u)
0,0428-0,0043

Table J.9: Coefficients for linear, quadratic G-MOS regression algorithm (Czech)
Order
1
2

J.7

c0
-0,9326-cNj (N-MOS)
0,8097-0,0696cSj (S-MOS)
0,5074
0,0443

Post selection

Table J.10 contains the conditions and related auditory S-MOS, N-MOS and G-MOS for the Czech language. Also
standard deviations for all MOS scores are given. The results for validation purposes are blinded.

ETSI

77

ETSI EG 202 396-3 V1.7.1 (2018-03)

Table J.10: Result of subjective experiment results -experts listening:
Samples selected from the Czech database (hs - handset, hf - hands-free, f - female, m - male speaker)
Condition

Noise

1
10
18
22
24
25
28
31
37
40
43
49
51
55
61
73
76
79
81
82
84
85
87
97
103
111
120
138
174
195
201
217
219
243
271
274
276
277
280

Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Lux_Car
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Crossroads
Road
Road
Road
OfficeOfficeOfficeOfficeOfficeRecording Speaker
hs
hs
hs
hs
hs
hs
hf
hf
hf
hf
hf
hf
hf
hs
hs
hs
hs
hs
hs
hf
hf
hf
hf
hf
hf
hs
hs
hf
hs
hf
hf
hs
hs
hs
hs
hs
hs
hs
hs

f
f
f
f
f
f
f
f
f
f
f
f
f
m
m
m
m
m
m
m
m
m
m
m
m
f
f
f
m
m
m
f
f
f
f
f
f
f
f

Network

NSA

Sharp/
smoothdB

AMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIIIAMR _NIAMR _NIAMR _NIIIAMR _NIIIAMR _NIIIAMR _NIIIAMR _NIIIAMR _NIIIAMR _NIAMR _NIIIAMR _NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NIno NSA
no
yes
yes
yes
yes
no NSA
no
no
no
yes
yes
yes
no NSA
no
yes
yes
yes
yes
no NSAno NSA
no
no
yes
yes
no NSAnono NSAnono
nono NSAno NSAyes
no NSAnono
no
nono NSA
Sharp
Smooth
Sharp
Sharp
Sharp
no NSA
SmoothSharp
Sharp
SmoothSharp
Sharp
no NSA
SmoothSmooth
Sharp
Sharp
Sharp
no NSAno NSA
SmoothSmoothSmoothSharp
no NSASharp
no NSA
Sharp
SmoothSharp
no NSAno NSASharp
no NSA
SmoothSmoothSmoothSharp

no NSA999918
no NSA9918
999
no NSA
18
18
9
18
18
no NSAno NSA99
9
9no NSA9no NSA99
9
no NSAno NSA18
no NSA
9
9
18
9

ETSI

MOS
Speech

MOS
Noise

CZECH
MOS
STD
Global Speech

STD
Noise

STD
Global

2,42

3,25

2,58

0,72

0,53

0,65

3,29
3,54

3,92
1,5

3,33
2,17

0,86
0,88

0,58
0,66

0,82
0,87

2,83

2,42

2,38

0,64

0,72

0,49

2,25
3,75
3,79
4,17
4,42

1,75
2,88
4,17
4,08
3,25

1,88
3,29
3,88
4,17
3,88

0,61
0,61
0,78
0,76
0,5

0,61
0,9
0,48
0,41
0,61

0,54
0,55
0,54
0,38
0,61

3,58
2,29
3,96
2,13
3,88

1,42
1,5
2,54
2,13
2,29

2,17
1,67
2,92
1,96
3,08

1,14
0,86
0,62
0,74
0,8

0,58
0,59
0,66
0,74
0,69

0,82
0,56
0,65
0,62
0,72

2,21
2
1,79
2,42
1,38

1,88
1,96
1,29
2,38
1,42

1,88
1,92
1,33
2
1,21

0,78
0,72
0,88
0,93
0,65

0,61
0,55
0,46
0,58
0,58

0,61
0,41
0,56
0,66
0,41

2,5
1,67
1,54
4,54
4,58

1,67
1,5
2,58
4
4,17

1,92
1,5
1,54
4,25
4,42

0,83
0,64
0,66
0,59
0,58

0,64
0,51
0,88
0
0,38

0,5
0,59
0,59
0,44
0,5

4,58

3,71

4,21

0,58

0,46

0,66

Listeninglevel dB SPL
72,8
69,33
69,02
70,18
71,41
71,85
78,06
70,3
71,44
71,5
69,85
70,79
70,74
74,86
72,34
70,59
69,24
73,81
71,64
78,13
77,71
69,77
70,16
69,08
69,71
71,23
69,34
73,3
72,27
69,57
70,94
72
72,26
70,91
74,23
72,49
73,68
73,06
75,22

78

Condition

Noise

282
283
285
286
289
291
292
295
297
325
328
331
334
336
337
339
340
342
343
346
348
349
351
354
361
367
373
375
376
379
384
385
387
388
390
391
393
394
396
400
403
406

OfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOfficeOffice
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub
Pub

Recording Speaker
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hf
hf
hf
hf
hf
hf
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs
hs

f
f
f
f
f
f
f
f
f
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
m
f
f
f
f
f
f
f
f
f
f
f
f
m

ETSI EG 202 396-3 V1.7.1 (2018-03)

Network

NSA

Sharp/
smoothdB

G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NIII
G722_NI
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NIII
G722_NI
G722_NI
G722_NI

no
no
no
yes
yes
yes
yes
yes
yes
no NSA
no
no
no
no
no
no
yes
yes
yes
yes
yes
yes
yes
no NSA
no
yes
yes
yes
yes
no NSA
no
no
no
no
no
no
no
yes
yes
yes
yes
no NSA

Sharp
Sharp
Sharp
SmoothSmoothSmooth
Sharp
Sharp
Sharp
no NSA
SmoothSmooth
Sharp
Sharp
Sharp
Sharp
SmoothSmoothSmooth
Sharp
Sharp
Sharp
Sharp
no NSA
Sharp
Smooth
Sharp
Sharp
Sharp
no NSA
SmoothSmoothSmooth
Sharp
Sharp
Sharp
Sharp
SmoothSmoothSharp
Sharp
no NSA

9
18
18
9
18
18
9
18
18
no NSA
9
18
9
9
18
18
9
9
18
9
9
18
18
no NSA999918
no NSA
9
18
18
9
9
18
18
9
9
9
18
no NSAETSI

MOS
Speech
3,83
4,33
2,71
4,38

MOS
Noise
3,92
4,04
3,71
4,08

CZECH
MOS
STD
Global Speech
3,79
0,87
4,17
0,48
2,75
1,12
4,42
0,58

STD
Noise
0,5
0,36
0,46
0,28

STD
Global
0,78
0,56
1,03
0,58

2,42

4,29

2,67

1,25

0,55

0,92

4,38

4,04

4,17

0,71

0,46

0,56

4,54
4,54

4,04
4,58

4,46
4,63

0,72
0,72

0,55
0,5

0,72
0,49

3,75
4,67
4,13
4,75
4
4,25
4,83
3,17
4,46
4,67
4,17
4,71
4,88

4,38
4,21
4,08
4,13
4,29
4,46
4,21
4,17
4,71
4,58
3,25
3,67
3,92

4,08
4,63
4,17
4,67
4,21
4,25
4,63
3,33
4,58
4,63
3,63
4,25
4,5

0,94
0,64
0,8
0,44
0,88
0,68
0,48
1,05
0,59
0,48
0,64
0,46
0,34

0,49
0,41
0,41
0,45
0,46
0,72
0,51
0,38
0,46
0,5
0,68
0,56
0,5

0,83
0,49
0,64
0,48
0,51
0,94
0,58
0,92
0,5
0,49
0,71
0,53
0,51

2,88
4,67

3,67
4,25

3
4,58

0,85
0,56

0,7
0,61

0,83
0,58

2,75
2,88
3,29

2,5
2,08
1,42

2,5
2,33
2,13

0,68
0,8
0,95

0,59
0,58
0,58

0,51
0,7
0,61

2,83

2,04

2,21

0,82

0,62

0,72

3,46

1,67

2,42

0,83

0,56

0,58

3,04
2,08
3,5

1,63
2,54
1,63

2,42
2,17
2,5

0,69
0,83
0,66

0,58
0,93
0,58

0,72
0,64
0,72

Listeninglevel dB SPL
73,6
72,64
74,77
74,81
73,77
74,05
75,57
75,24
72,38
75,74
74,1
72
75,41
74,73
71,98
73,17
75,37
74,51
74,52
75,38
74,36
74,55
75,26
69,13
70,54
69,88
70,68
70,53
69,67
69,94
70,95
70,71
69,22
74,31
72,13
70,61
72,13
72,84
69,49
73,24
75,43
70,97

79

Condition

Noise

408
409
415
421
424
427
432

Pub
Pub
Pub
Pub
Pub
Pub
Pub

Recording Speaker
hs
hs
hs
hs
hs
hs
hs

m
m
m
m
m
m
m

ETSI EG 202 396-3 V1.7.1 (2018-03)

Network

NSA

Sharp/
smoothdB

G722_NIII
G722_NI
G722_NI
G722_NI
G722_NI
G722_NI
G722_NIII

no NSA
no
no
yes
yes
yes
yes

no NSA
SmoothSharp
SmoothSmoothSharp
Sharp

no NSA
9
9
9
18
9
18

ETSI

MOS
Speech
1,88
3,46

MOS
Noise
1,5
2

CZECH
MOS
STD
Global Speech
1,54
0,74
2,67
0,66

STD
Noise
0,51
0,72

STD
Global
0,59
0,48

3,96
2,83

1,83
2,67

2,75
2,58

0,48
0,7

0,68
0,58

0,62
0,82

Listeninglevel dB SPL
70,62
69,39
72
70,45
69,35
70,89
69,19

80

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex K:
Relative Approach Non-Linear Transformation
The non-linear transformation can be described by a function which is defined in two intervals:(K.1)
Where x is Peff / P0. Peff is the RMS value of the sound pressure level and P0 is the reference sound pressure level
equivalent to 20 uPa. Then y1 represents the normalized envelope value of the non-linear transformation.
When assuming a linear characteristic and applying a sinusoidal signal with a sound pressure level of 20 dB the value
derived by (K.1) will be 10.
The constants a, b and c are predetermined for the variable v (exponent is approximately 0,20 to 0,25) and xs (thethreshold of nonlinearity is approximately 5,65) in such a manner that the function is twice continuously differentiable
at the transition xs.

(K.2); (K.3); (K.4)
The inverse function of (K.1) is:(K.5)
In which ys = xs - a × xs2 has been used.
Alternatively an additional function was used. However the inverse function cannot be represented as a closed form
expression:(K.6)
The asymptotes of the expression in (K.6) are y2 = x when x << xs and y2 = xs × (x/xs)v when x>>xs>> 1. When plottedin a double-logarithmic graph two straight lines with a slope of 1 respective v and intersection at (x = xs, y2 = xs).
The parameter alfa allows to adjust the size of the transition region between the asymptotic approximations.
The following values are used: v = 0,2; xs = 5,66.

ETSI

81

ETSI EG 202 396-3 V1.7.1 (2018-03)

Annex L:
Bibliography
R.A. Fisher: "Statistical methods and scientific inference", Oliver and Boyd, 1956.
ITU-T contribution COM 12-117, Study Period 1997-2000: "Report of the question 13/12 rapporteur's meeting(Solothurn, Germany, 6-10 March 2000)".
3GPP S4-120542: "Common subjective testing framework for training of P.835 test predictors".

ETSI

82

ETSI EG 202 396-3 V1.7.1 (2018-03)

History
Document history
V1.1.1

August 2007

Publication

V1.2.1January 2009

Publication

V1.3.1February 2011

Publication

V1.4.1

June 2014

Publication

V1.5.1

October 2015

Publication

V1.6.1January 2017

Publication

V1.7.1March 2018

Membership Approval Procedure

V1.7.1

May 2018

Publication

ETSI

MV 20180513: 2018-03-14 to 2018-05-14