Recent Mission Critical Voice QoE

Awarded Projects

Don Bradshaw, PSCR

Brad Fain and Alessio Medda, Georgia Tech Research Institute

Henning Sculzrinne, Columbia University

1

#PSCR2019
2

DISCLAIMER

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an 

experimental procedure or concept adequately. 

Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended 

to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

*Please note, unless mentioned in reference to a NIST 

Publication, all information and data presented is 

preliminary/in-progress and subject to change
Summary

• Mission Critical Voice (MCV) Quality of Experience (QoE) Background

• PSCR’s MCV QoE Measurement Development

• What Are Good Numbers?

• Federal Funding Opportunity Goals

• Georgia Tech Research Institute

• Columbia University

3
QoE KPIs for MCV - MCV Roundtable 2017

• Mouth-to-Ear (M2E) Latency

• Time it Takes Audio to Get from Transmitting User to Receiving User

• End-to-End Access Time

• Time Between Button Press and Receiving User Hearing Voice

• M2E Latency + Access Delay 

• Audio Quality/Intelligibility

• Public Safety Cares Most About Intelligibility

• Access/Retention Probability

• Ability to Establish Call

• Ability to Retain Call

4
QoE KPIs for MCV

5
QoE KPIs for MCV

6
QoE KPIs for MCV

7
QoE KPIs for MCV

8
QoE KPIs for MCV

9
What Are Good Numbers?

10
End-to-end Access Time Results 

for 85% Intelligibility

11

PTT Technology* M2E Latency 

(ms)

Access Delay 

(ms)

End-to-End Access Time 

(ms)

Analog Direct 76.5 ± 0.3 136.5 ± 3.3 213.1 ± 3.3

Analog Conventional 78.5 ± 0.3 286.1 ± 2.5 364.7 ± 2.5

P25 Direct 220.9 ± 0.3 71.6 ± 4.1 292.4 ± 4.1

P25 Trunked (Phase 1 –

FDMA)

356.6 ± 3.8 640.1 ± 5.1 996.7 ± 6.3

P25 Trunked (Phase 2 –

TDMA)

575.9 ± 8.1 692.2 ± 7.1 1268.1 ± 10.7

*Analog Conventional operates in VHF band.

All P25 technologies operating in 700 MHz band. 
Federal Funding Opportunity Goals

• LMR Simulation Tools

• Equipment Like What First Responders Use

• Measured Same as PSCR MCV QoE Measurements

• Development of Test Facilities, Scenarios and Protocols

• Mimic Real-Life Operational Environments

• Development of Public Safety Testing Cadre

• Test and Measurement of Public Safety User Performance

• Analysis and Modeling of Public Safety User QoE
Recent Mission Critical Voice QoE

Awarded Projects

Don Bradshaw, PSCR

Brad Fain and Alessio Medda, Georgia Tech Research Institute

Henning Sculzrinne, Columbia University

13
14

Project Overview

Brad Fain and Alessio Medda

July 8-12, 2019

PSCR Stakeholder Meeting, Chicago, IL
15

QUARC (Quality Under Adjustable Realistic Conditions)

To develop a framework for the evaluation of mission critical voice (MCV) 

quality of experience (QoE) for first responders operating in real field 

scenarios

q GOAL 1 - Simulated Land Mobile Radio (LMR) Equipment Development

q GOAL 2 - Development of Test Facilities, Test Scenarios, and Test Protocols for MCV QoE Tests

q GOAL 3 - Development of Public Safety Testing Cadre

q GOAL 4 - Test and Measurement of Public Safety Users’ Performance

q GOAL 5 - Analysis and Modeling of Public Safety User QoE
16

Project Schedule
17

Tunable KPIs Implementation

Audio In 

(wav file or 

microphone)

+

TX Background 

noise

MtE Latency 

and Access Time

Tunable time –

Queueing 

Audio 

Encoder

Link 

Retention

Audio 

Decoder 

Save CD Quality 

(16 bit, 44 kHz)

Speech 

Intelligibility Score 

(PESQ/POLQA)

Network 

Impairments

Loss, Pause and 

Jump

+

RX Background 

noise

P25, PCM MLT,... Prob. of dropped 

channel

Audio OUT Tunable KPIs

• KPI-1 – Mouth to Ear Latency

• KPI-2 – Access Time

• KPI-3 – Link Establishment and retention

• KPI-4 – Speech Intelligibility and 

background audio quality

• KPI-5 – Network Impairments

KPI-4

KPI-4

Link

KPI-1 KPI-2 KPI-5 KPI-4
18

Architecture for Laboratory Testing

• Possibility to test in VR environment or 

in normal conditions

• VR background noise guaranteed by 3D 

sound controlled over UNITY and 

reproduced using high quality flat 

response headsets

• Normal conditions uses a Dolby Atmos

5.1.2 system for reproducing 

background noise in 3D

• Units at the receiver end will be 

implemented by a push-to-talk 

speaker/microphone

• Possibility to input live voice (speaker) 

or to chose a pre-recorded utterance
19

Architecture for Field Testing

Centralized Node 

Controls all 

communications 

and KPIs

...

User 1 User 2 User N

Dedicated LTE/WiFi

Channels

• Architecture based on Mangoh Development Kit

• Allows scalability with small footprint

• Integrated LTE, WiFi, BTLE
20

Testing Facility – The Guardian Center in Perry, GA
21

Thank you!

Brad Fain brad.fain@cacp.gatech.edu

Alessio Medda alessio.medda@gtri.gatech.edu
Experimentally-Driven Mapping 

of QoS-to-QoE for MissionCritical Voice

Henning Schulzrinne (PI), Dan Rubenstein Charles Jennings, Norm Groner

v1.3
Project objectives

• How does the quality of the communication 

channel affect first responder communications?

• Four phase approach:

• Build communication testbed with tunable 

parameters that emulates realistic (poor) LMR 

communication channel conditions

• Experiment using trained first responders to 

communicate across communication 

infrastructure

• Measure communication performance (delay, accuracy)

• Build mathematical models: channel conditions 

→ performance measures
QoE measures

• Comprehension errors

• e.g., repeat transmitted messages

• Task errors

• e.g., wrong information recorded

• Usage errors

• e.g., pressing talk to speak button too early or too late

• Length and latency of responses

• e.g., pauses between requests and start of transmission

• Subjective ratings of user experience

• e.g., rated frustration with ratios
Year 1: Testbed development

• Intel NUC will manage all services

• End-user communication gear 

connected to Android or RPi

Devices

• Centralized control to adjust 

communication quality parameters

• mouth-to-ear and PTT delay

• noise level

• packet loss (outage bursts)
First (early) prototype
More detailed system architecture

UE
Year 2: Testing

• Generate scenarios that first 

responders will describe

• First responder officer volunteers will 

help design appropriate scenarios

• First responders “in-the-field” will test 

in the scenario (in a lab)

• QoE: measure communication time 

and accuracy
Year 3: Modeling

• Produce mappings from input parameters (delay, noise, loss) to 

output parameters (communication time and accuracy)

Input 

Parameters

Experimental 

Results

MultiDimensional 

Curve Fitting

Input parameters

Performance
Accomplishments to date

• Testing platform development

• Initial design of network-based general voice quality platform

• portable, replicable, building on standard software

• Demo: initial prototype using Raspberry Pi (Linux) + VoIP clients

• Engagement with local first responder organizations

• Actively engaged with Columbia University’s Office of Public Safety

• James McShane, VP Public Safety

• Jeannine Jennette, Executive Director, Public Safety

• Empress EMS (major private EMS provider)

• Teaneck, NJ fire department

• your name here - inviting additional scenario input & test participants
Come back for the 

Next 

Session BACK AT

1:50 PM

31

#PSCR2019