Evaluation of an immersive audio experience using questionnaire andinteraction data
Jon FRANCOMBE(1) , Kristian HENTSCHEL(2) .
(1) BBC Research & Development, United Kingdom, jon.francombe@bbc.co.uk(2) BBC Research & Development, United Kingdom, kristian.hentschel@bbc.co.uk

Abstract
Device orchestration is the concept of using an orchestrated set of connected devices to reproduce or augment
a media experience. To evaluate the concept applied to immersive audio experiences, an orchestrated audio
drama was created and released on BBC R&D’s public-facing trial platform. The experience allowed users
to easily connect personal devices, which reproduced sounds including speech and spot effects, ambient background sounds, and additional storyline content. The trial was evaluated through a simple questionnaire anduser interaction logs. The 210 questionnaire responses suggested a positive reaction (mean rating of 4 out of 5stars, 72% loved or liked using phones as speakers, and 80% would use the technology again). The interaction
logs (for 2174 sessions) showed reasonable engagement, with 18% of sessions nearing the end of the content.
For sessions lasting over 1 minute, over 60% had at least one extra device connected. The results suggest usefulstrategies for optimising engagement with orchestrated experiences. There are limitations and challenges todrawing strong conclusions from large-scale trials in uncontrolled conditions (including unrepresentative sampling of the population and lack of context for logged interactions). However, benefits include larger sample
sizes and greater ecological validity than in controlled laboratory testing.
Keywords: Sound, Orchestration, Immersive, Evaluation, Perception

1

INTRODUCTION

There are many ways of creating immersive audio experiences—channel-, scene-, and object-based audio representations can be rendered over loudspeaker arrays or headphones. However, there are considerable barriers
to uptake for existing technology; for example, Cieciura et al. [1] found that few listeners have access to asurround sound system. One approach to addressing the challenge of creating immersive experiences in realistic
home environments is to use devices that are present in the space, have loudspeakers, and can be connected
wirelessly; for example, mobile phones, tablets, smart speakers, and Bluetooth speakers. This is termed device
orchestration. Orchestrated devices have been used for second screen visual experiences [2], music compositionand performance [3], reproduction of existing audio content [4], and multi-modal interactive experiences [5].
However, the recent trend towards object-based audio has unlocked greater potential for optimal use of such a
distributed array, particularly for sound reproduction in the home. Recent evaluations suggest potential benefitsof this approach, and that quality of experience can match that of five-channel reproduction (or exceed it inoff sweet spot listening positions) [6, 7]. However, there are many technical and creative challenges, including
discovery, pairing, and device synchronisation, and authoring of content that can adapt to the available devices.
1.1 Trial production: The Vostok-K Incident
Based on the positive perceptual experiment results, a trial production for orchestrated devices was created. The
Vostok-K Incident is a 13-minute science-fiction audio drama that was designed to demonstrate the concept ofdevice orchestration and enable evaluation outside of controlled ‘laboratory’ conditions. A writer was commissioned to develop a concept that demonstrated the capabilities of the orchestration technology. The resultant
piece is based around a stereo bed, but features various types of sound that can be reproduced from connected
devices (in this case, any device running a modern web browser, such as mobile phones, tablets, and laptops),

6137

adapting to the available devices using detailed metadata and an orchestration rule set. A bespoke production
environment [8] and web framework [9] were developed to create and deliver The Vostok-K Incident. The piece
was released on 13th September 2018 on BBC Taster, BBC R&D’s public-facing trial platform.
1.1.1 The Vostok-K Incident terminology
A session refers to a single viewing of the BBC Taster pilot. Each session is allocated a unique identifier anda shorter pairing code. All sessions include a main device—the device used to start the session. Some sessions
might also include one or more connected, auxiliary, or extra devices (terms used interchangeably), on which
the user enters the pairing code to identify the session to which they wish to connect. The Vostok-K Incident
has three sections: setup, introductory text describing the experience and a start button; loop, a 31-second
content loop that plays whilst listeners can connect devices before pressing continue to proceed to the main
piece; and main, the main drama (800 seconds long). The interfaces for each section are shown in Figures 1a
to 1c. Audio objects are sent to devices based on the number of connected devices and their positions, which
are identified by the user with a simple map (two concentric circles with divisions; see Figure 1d). Six settings
were available: near or far for front, back, or side.

(a) Setup(b) Loop(c) Main(d) Device position setting(e) Connected device

Figure 1. The user interfaces for the three sections in The Vostok-K Incident on the main and connected devices
1.2 Research questions
The aims of the public release of The Vostok-K Incident were to: (i) provide motivation for development of
production and delivery tools; (ii) produce a high quality demonstrator of device orchestration; and (iii) evaluate the concept of immersive audio with orchestrated devices in a real-world setting. In this paper, aim (iii)
is addressed by considering the following research questions. (1) Did listeners enjoy The Vostok-K Incident?
(2) Were listeners able to understand and use the technology? (3) Would listeners be interested in using thistechnology again? (4) Did listeners engage with the content? (5) How were additional connected devices used?
Two data collection approaches were taken. (1) A short, simple questionnaire was designed and participants
were asked to complete this after listening to the piece. (2) Where users consented, the Comscore Digital
Analytix (DAX) service was used to log user interaction with the experience. The analysis is based on 3121
visits to the trial between 13/09/18 and 11/12/18.
The detailed methodologies and results of these two approaches are presented in Sections 2 and 3 respectively.
In Section 4, the results are used to answer the research questions outlined above. These approaches to evaluation of an audio experience are a significant departure from common methodologies (for example, controlled
listening experiments conducted in laboratory conditions, perhaps based on standard methodologies such as ITUR BS.1116 [10]). In Section 4.3, the advantages and disadvantages of evaluating a listening experience using
an uncontrolled public trial are discussed.

6138

2

RATINGS AND QUESTIONNAIRE

At the end of The Vostok-K Incident, a questionnaire was presented, including a star ratings (one to five stars)
followed by a series of multiple choice questions where the user could select one of the available options foreach question. In total, 210 star ratings were made. Figure 2 shows the distribution of ratings. The mean star
rating was 4.0; the median and mode were 5.0. Only 11% of ratings were one star, with 55% five star ratings.

Percentage of ratings(% from 210)

60
50
40
30
20
10

Mean

Median

4

5

0
1

2

3
Star rating

Figure 2. Distribution of star ratings

The multiple choice questions and response frequencies are detailed in Table 1. Each question could also be
skipped. The questionnaire responses were generally positive about the experience, with the vast majority of
respondents expressing that they loved or liked the concept, would use the technology again, and found using
devices as speakers the most positive aspect. The results are discussed in more detail in Section 4.1.
Table 1. Questions and responses in The Vostok-K Incident questionnaire. The number and percentage of responses are given in parentheses, and the modal response is in bold.
Question

Option 1

Option 2

Option 3

Option 4

Option 5

Option 6

How was using phones
as speakers? (166)

Loved it(81, 49%)

Liked it(50, 30%)

Unsure(12, 7%)Not great(16, 10%)Hated it(7, 4%)-

Would you use thisYes
No
technology again? (166) (141, 85%) (16, 10%)

Not sure(9, 5%)---

What would you listen
to/watch most? (156)Sport(9, 6%)

Drama(72, 46%)

Entertainment(28, 18%)

Music(26, 17%)

Other(2, 1%)

What was best about the Storyline
experience? (152)(18, 12%)

Sound effects(53, 35%)

Devices as
Not sure
speakers (72, 47%) (9, 6%)--

And what was worst?
(131)

Hard to follow Poor sound quality
story (32, 24%) (6, 5%)

Unsure(30, 23%)-

3

Factual(19, 12%)The set up(26, 20%)

Unbalanced
volumes (37, 28%)

INTERACTION LOGGINGInteraction logs were collected from a total of 2740 sessions for which users opted in to data logging. Preprocessing was used to remove sessions with errors in the logging or with outlying parameters (e.g. excessively
long sessions or those with an unusually large number of connected devices). After this preprocessing, a totalof 2174 sessions were analysed (i.e. 20.7% of sessions were removed). The results from the analytics data
are shown for three main categories: time spent engaging with the content (Section 3.1); number of devices
connected (Section 3.2); and user interaction (Section 3.3).

6139

Content section
reached

3.1 Engagement time
Figure 3 shows the proportion of sessions that reached each section of the content. It can be seen that the
majority of listeners progressed from the setup to the loading loop, but that only around 60% of the sessions
that got to the loading loop continued to the main piece. However, 1068 sessions (49% of the total) did reachthe main content.
Setup (2174)
Loop (1854)
Main (1068)
0

20

40
60
Percentage of sessions (% from 2174 total)

80

100

Figure 3. Percentage of sessions that reached each section of the content
Figure 4a shows the cumulative percentage of time spent in the loading loop for sessions with and without any
connected devices, for all sessions that reached the loading loop. Unsurprisingly, when devices were connected,
the time spent in the loading loop was considerably longer (median at approximately four loops, or 2 minutes,
compared to under one loop where no devices were connected). For the sessions that reached the main content,
Figure 4b shows the cumulative percentage of maximum content time reached (i.e. the furthest point reachedin the timeline, not necessarily the time spent listening to the content). The figure shows a relatively steep
drop-off of sessions in the first two minutes of the content, followed by a narrower slope towards the length ofthe piece. The median time reached is 2.8 minutes, but 195 sessions (18%) reached further than 13 minutes.
100

No connected devices (1315)
At least one device connected (539)80Percentage of sessions(% from 1068 total)

Percentage of sessions (%)

100

60
40
20
MedianMedian

0

80
60
40
Mean

20

Median00246Time (mins)(a) Loading loop

8

10

0

2

4

6
8
Time (mins)

10

12

14

(b) Main piece

Figure 4. Cumulative percentage of sessions for: (a) time spent in the loop for all sessions that reached theloop, and (b) max. time reached in the main content. Dashed vertical lines in (a) are spaced by loop duration.
3.2 Connected devices
Figure 5 shows a breakdown of the number of devices connected for all sessions that started the main piece,
those that reached 1 minute, and those that reached the end of the content. For all sessions, the majority had
no connected devices, with a monotonic reduction in the number of sessions with more devices connected.
However, the figure also shows that more devices were connected in sessions that got further into the piece.
Whilst approximately 50% of all sessions had one or more connected devices, over 70% of sessions that reached
the end of the content had one or more connected devices, and approximately 20% had three or more.

6140

Percentage of sessions (%)

100

Sessions that reached at least 0.0 minutes (1068 sessions)
Sessions that reached at least 1.0 minutes (728 sessions)
Sessions that reached at least 13.3 minutes (152 sessions)

80

60

40

20

0
0

1

2

3
4
At least n devices connected

5

6

7

Figure 5. Cumulative percentage of number of devices connected broken down by listening length

80

30
25

Total number of devices connected
Mean number of ‘move’ actions per device

60

20
Number

Percentage of overall device
connection time (% from 161 hours)

3.3 User interaction
As discussed in Section 1.1.1, listeners could select position zones for the connected devices. Figure 6a shows
the percentage of the total time that extra devices were connected for during which each zone was selected.
The far zones were used more than the near zones (57% of time in the far zones, 39% in the near zones).
The side zones were most frequently selected; however, it should be noted that users could select either theleft or right side of the position wheel and the choice would be recorded as side. With this in mind, thedistribution of position selection is approximately uniform. These figures do not give an indication of why
certain zones were selected. Anecdotal evidence suggests that some listeners may not have realised that there
was a distinction between near and far. As the far zones account for more area of the positioning interface,
the more frequent selection of far zones could be because positions were selected indiscriminately. There isalso no way of knowing where the devices were physically positioned.

15
10

40

20

5
0
Far Far Far Near Near Near Nonefront rear side front rear side
Zone(a) Percentage connection time in each zone0
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Minute(b) Minute-by-minute breakdown of connected devices and move actions

Figure 6. User interaction with The Vostok-K Incident

6141

For sessions that reached the main content and had at least one connected device, Figure 6b shows the total
number of connected devices for each minute of the content. It also shows the mean number of times thatdevices were moved during that minute (i.e. the total number of move actions divided by the total number ofconnected devices). There is a steady drop off in the number of connected devices as the piece progresses (inline with the reduced number of sessions progressing through the content as seen in Figure 4b). However, theaverage number of move actions per connected device stays reasonably consistent at approximately five moves.
This suggests that participants remain reasonably engaged throughout the content, and that the ability to interact
with the experience is valuable.

4

DISCUSSION

The analysis reported above was used to address the research questions outlined in Section 1.2 about orchestrated experiences and specifically The Vostok-K Incident. The results were also used to draw conclusions about
future development of orchestrated experiences (Section 4.2) and the evaluation methodology used (Section 4.3).
4.1 Evaluation of The Vostok-K Incident
Did listeners enjoy The Vostok-K Incident? The questionnaire results presented in Section 2 suggest that users
enjoyed the experience, and that the orchestration technology was one of the primary drivers of this. The
experience received a high average rating (mean 4 and median 5 from 5 stars). Furthermore, 79% of respondents
either loved or liked using phones as speakers, and 85% would use the technology again. Using devices as
speakers was rated as the best thing about the experience by 47% of respondents. The options for the worst
aspect of the experience were more evenly distributed, although poor sound quality was only selected by six
respondents. The most frequently selected option was unbalanced volumes (28%).
Were listeners able to understand and use the technology? It it difficult to ascertain this from statistical dataalone (as discussed further in Section 4.3). However, the positive questionnaire feedback, amount of time spentengaged with the experience (1068 sessions reaching the main content, with a mean time of over 5 minutes
spent in the main content), and number of connected devices all suggest that listeners were indeed able tounderstand and use the technology.
Would listeners be interested in using this technology again? When explicitly asked this question, over 80%
of respondents said that they would use the technology again. It is less clear from the available data exactly
what they would want to use the technology for. When asked what they would listen to or watch most, drama
was the most common response with 46% of respondents (which might potentially be biased by the nature of
the content used in this trial). Responses for other content genres were roughly evenly distributed (with the
exception of sport, which was slightly less common).
Did listeners engage with the content? Figure 3 shows a steep drop in the number of sessions that progressedfrom the loading loop to the main content. This could potentially be attributed to users testing out the technology without listening to the whole piece; however, only 9% of sessions that got to the loading loop but did notcontinue to the main content included one or more connected devices. Once listening to the main content, there
was a quick drop-off in the number of sessions lasting over 1 minute (Figure 4b). The median listening timewas 2.8 minutes. However, the distribution shows a long tail, with approximately 40% of listeners reaching 5
minutes and around 20% reaching 13 minutes. Figure 6b shows that even though the overall number of devices
connected dropped off as the content progressed, the number of interactions with the controls presented on theuser interface (i.e. the position setting controls) remained relatively consistent.
How were additional connected devices used? The results in Figure 5 show that it was most common for no
extra devices to be connected, and increasingly uncommon for each additional device. However, for sessions
that progressed further through the content (at least 1 minute), the modal session included at least one connected
device, and for sessions that reached the end of the content, 70% had one or more devices connected. It wasvery uncommon for 4 or more devices to be connected. There was no clear indication that any of the position
zones were particularly favoured (Figure 6a).

6142

4.2 How could these findings influence future development of orchestrated experiences?
The results presented above show that device orchestration can be valuable when applied to audio drama. It
appears likely that in many listening situations, it will not be possible to connect a large number of devices,
so future development or content creation should strive to generate as much additional benefit as possible withone (or few) connected devices. The drop-off in listeners after the first minute suggests that demonstration
content should aim to achieve as much impact as possible within the opening minute. The results show thatparticipants are prepared to interact with the experience throughout its length; different interaction functions
could be investigated in future trials.
4.3 Methodology
Evaluating a novel audio experience through a large scale public facing trial is considerably different to thecommon approach of evaluation in controlled laboratory conditions according to standardised test methodologies. There are some benefits of this approach, but also some signifiant challenges and weaknesses—these are
discussed in the following sections. Given the pros and cons, it is advisable to use multiple evaluation strategies (including formal tests, public trials, and user experience testing) when developing new technologies orexperiences, as they each lead to different information and can be used to build up an aggregate picture of thetechnology or product under test.
4.3.1 Benefits
A public facing trial enables collection of thousands of ratings (as opposed to tens of ratings as suggested bystandard audio test methodologies [10]). It can also reach a wider range of participant groups (e.g. different
demographics), whereas it is not uncommon for listening experiments to be performed by members of a researchgroup (who are likely to have a different understanding and experience of the technology under test than thegeneral public, and might also be relatively homogeneous in terms of educational background etc.). It mightalso be beneficial to evaluate an experience in an environment that is close to how it would be consumed inreal life, i.e., in the home rather than in an audio laboratory. (Of course, there are many situations wheresuch controlled testing is essential.) Collecting analytic data that records how participants acts is also a more
direct way of getting feedback compared with self-reported values or ratings, which may be more susceptible tobiases. Finally, the task required of participants (i.e. to listen to some audio and answer a few short questions)is relatively simple and enjoyable to complete compared with some complex tasks that require participants tolisten to a large number of (sometimes unpleasant) audio clips and make ratings requiring considerable cognitiveeffort.
4.3.2 Weaknesses and challenges
Whilst public facing tests allow a wide range of participants to be reached, there is still risk of sampling or
selection bias if recruitment is not controlled. In this case, The Vostok-K Incident was released on the BBC
Taster platform and advertised using various social media accounts. This is likely to attract an audience with aparticular interest in new technology. The scenario in which content is consumed on BBC Taster also might notexactly match how similar content would be consumed in reality (for example, how many people are present
and how long the content might be engaged with). Furthermore, the participants are presented with branding
and copy designed to engender a positive experience, whereas in controlled listening experiments, participants
are given as little supplementary information as possible to avoid sources of bias.
The questionnaire data is useful for getting direct feedback and easy to analyse, but offers limited choices andlow resolution responses (only single choices are enabled in the BBC Taster questionnaire format). Free text
data allows for richer and more detailed responses, but is much more challenging to analyse.
Observing interaction logs is interesting and can give a useful overview; however, it can be difficult to draw
meaningful conclusions. The interaction logs do not facilitate interpretation as to why certain actions weretaken, or where participants particularly enjoyed certain aspects or found aspects challenging or hard to under-

6143

stand. There are other user testing approaches that can be used to address these questions, often with smaller
sample sizes but utilising more direct observations or interview questions. This type of user experience testing
methodology will be applied to The Vostok-K Incident in the future.

5

SUMMARY

It is possible to use orchestrated personal devices to create an immersive audio experience. A trial production,
The Vostok-K Incident, was created to demonstrate and evaluate this concept. Evaluation was conducted using asimple questionnaire and user interacting logging. The results suggested that participants enjoyed, understood,and engaged with the concept, but that future similar experiences should endeavour to generate maximum impact
within the first minute and gain benefit from few connected devices. There are considerable advantages anddrawbacks to using a public facing trial for relatively uncontrolled evaluation of an experience; such experiments
are a valuable tool to use alongside more formal evaluation techniques.

ACKNOWLEDGEMENTS
The authors would like to acknowledge the work of members of The Vostok-K Incident production team: James
Woodcock and Richard Hughes (University of Salford); Eloise Whitmore and Tony Churnside (Naked Productions); and Ed Sellek. The production was supported by the EPSRC Programme Grant S3A: Future Spatial
Audio for an Immersive Listener Experience at Home (EP/L000539/1).

REFERENCES[1] Cieciura C, Mason R, Coleman P, Paradis M. Survey of media device ownership, media service usage, andgroup media consumption in UK households. In: Audio Eng. Soc. 145th Conv. (eBrief no. 456); 2018.
[2] Bober M, Feldmann I, García S, Messina A, Paschalakis S. BRIDGET: An approach at sustainable andefficient production of second screen media applications. In: IBC 2015, Amsterdam, Netherlands; 2015.
https://doi.org/10.1049/ibc.2015.0001.
[3] Taylor B. A history of the audience as a speaker array. 17th Int Conf on New Interfaces for MusicalExpression. 2017; 481–486.
[4] Kim H, Lee S, Choi JW, Bae H, Lee J, Song J, et al. Mobile Maestro: enabling immersive multi-speakeraudio applications on commodity mobile devices. In: ACM International Joint Conference on Pervasive
and Ubiquitous Computing; 2014. https://doi.org/10.1145/2632048.2636077.
[5] BBC R&D. Living room of the future; 2019. Available from: https://www.bbc.co.uk/rd/projects/
living-room-of-the-future, accessed May 2019.
[6] Francombe J, Woodcock J, Hughes RJ, Mason R, et al. Qualitative evaluation of media device orchestration
for immersive spatial audio reproduction. J Audio Eng. Soc. 2018;66(6):414–429. https://doi.org/10.
17743/jaes.2018.0027.
[7] Woodcock J, Francombe J, Hughes RJ, Mason R, Davies WJ, Cox TJ. A quantitative evaluation of media device orchestration for immersive spatial audio reproduction. In: Audio Eng. Soc. Conf. on Spatial
Reproduction, Tokyo, Japan; 2018.
[8] Francombe J, Woodcock J, Hughes RJ, Hentschel K, Whitmore E, Churnside A. Producing audio drama
content for an array of orchestrated personal devices. In: Audio Eng. Soc. 145th Conv. (eBrief no. 461);
2018.
[9] Hentschel K, Francombe J. Framework for web delivery of immersive audio experiences using device
orchestration. In: ACM TVX, Salford, UK; 2019.
[10] ITU-R rec. BS 1116-3. Methods for the subjective assessment of small impairments in audio systems.
ITU-R Broadcasting Service (Sound) Series; 2015.

6144