# Enhance Engine Room Diagnostics Through Audio-Focused VR Simulation

Tychonas Michailidis, Christopher Barlow, Gordon Meadow, John Gouch, Eshan Rajabally

► To cite this version:

Tychonas Michailidis, Christopher Barlow, Gordon Meadow, John Gouch, Eshan Rajabally. Enhance Engine Room Diagnostics Through Audio-Focused VR Simulation. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.386-394, 10.1007/978-3-030-29390-1_21. hal-02877655

HAL Id: hal-02877655

https://inria.hal.science/hal-02877655

Submitted on 22 Jun 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Enhance Engine Room Diagnostics Through
## Audio-Focused VR Simulation

Tychonas Michailidis<fnref n="1" />[0000-0002-7196-3811], Christopher Barlow<fnref n="2" />, Gordon
Meadow<fnref n="3" />, John Gouch<fnref n="3" />, and Eshan Rajabally<fnref n="4" />

**Abstract.** A marine engineer's task is to maintain all systems in an operational state, to diagnose and rectify problems arising, and to understand what maintenance will be required to keep the vessel appropriately operational and safe. This capability is built upon the training and experience of the engineering crew, the information that can be gained by reading and interpreting engine room instrumentation, and the familiarity with the vessel and an in-situ *intuitive feel* for normal operation. In this paper, we examine how audio can enhance remote interaction and feedback information. We gathered real-world data from an engine room that allow us to create a realistic virtual engine room for testing. We carried out usability test on simulated failure scenarios where we look at how VR technology might enable engineers to experience immersive information from a remote location and allow them to diagnose and give feedback on the system. Our findings suggest that sound plays a vital role in identifying failures and could potentially be used in the operation of unmanned and autonomous vessels.

**Keywords:** audio failures · engine room · diagnostics · VR · feedback

## 1 Introduction

Marine engineers on board ship use a wide variety of multi-sensory feedback in addition to the system data [3]. This feedback includes sound, vibration, heat and smell. The sense of sound and vibration is an invaluable tool for initial detection of faults by engineers. Pumps, filters, generators, main engine and other moving parts in the engine room (ER) have a unique audio signature which changes over time when potential failure is imminent. Sounds may be caused by increased friction, insufficient lubrication or general component failure which result in changes of frequency, knocking sounds, whining, squeaking or arrhythmic sounds all of which have a distinctive sound quality [4]. Engineers are intuitively trained to accumulate this rich source of audio information, and it subconsciously becomes part of their diagnostics ritual and inspection. While

<footnote n="1">¹ Research, Innovation and Enterprise, Solent University, Southampton, UK
[tychonas.michailidis@solent.ac.uk](mailto:tychonas.michailidis@solent.ac.uk)</footnote>

<footnote n="2">² School of Media Arts and Technology, Solent University, Southampton, UK</footnote>

<footnote n="3">³ Warsash Maritime Academy, Solent University, Southampton, UK</footnote>

<footnote n="4">⁴ Central Technology, Rolls-Royce plc</footnote>
it may not always allow an engineer to diagnose the cause of failure only by listening, it is particularly useful in identifying and narrowing down the source of a problem as well as confirming the validity of known data. Engineers commonly report not fully trusting the control data [3] due to regular false alarms from systems or 'wrong readings' from the sensors. Despite advances in monitoring and diagnostics technology, onboard engineers still rely on their intuitive feel and senses to assess system health or confirm control data before system failure is imminent. Engineers may use audio to identify if a piece of machinery failed to run; to identify the state of machinery/pipes; to access the harshness of a component through variations in sound level and pitch.

Due to excessive loudness within the ER (above 100db), engineers are required to wear ear defenders. As a result, potential details of audio features are lost. Audio can play a significant role in providing a diagnostics tool otherwise unavailable in-situ for example, in the potential crewless vessel scenario. It is possible to isolate the audio of a component from background noise, and therefore focus more closely into audible failures. Within a VR environment, engineers can enhance audio information and their ability and have a clear sound source of information to support their diagnostics.

## 1.1 Real World Challenges

The primary challenge of this project was to understand and measure how the engineering crew uses sensory feedback to characterise the state of the vessel while onboard, for both normal and abnormal conditions, and to assess the mechanisms by which this tacit information may be communicated and interpreted effectively to a remote location. Experimentation with artificial data, including a non-realistic visual representation of engine room, data from machinery (e.g. temperature and pressure), and audio, would have resulted in false readings and results. The ferry company Red Funnel<fnref n="1" /> granted access to the team to collect data from the engine room of a Raptor Class ro-pax ferry that operates daily services between Southampton and Isle of White with a round-trip duration around two hours. We were able to get the schematics of machinery layout, photographs and audio recording of the ER ambience sound and the machinery as well as real data values from pressure and temperature of relevant components for developing the VR system. The simulated engine room is 1:1 ratio with the real engine room.

Computer simulations have the potential to immerse remote operators concerning the system in question. Maritime, military and aerospace industries have been using computer-simulated systems for operator training, including navigation and bridge systems [2]. Current VR systems have now become readily available to implement them in a wide range of industries apart from gaming, offering a low cost large-scale simulation. Their potential for the presentation of real-time data is already being explored [6]. Engineers on board ship use a wide variety of sensory feedback information in addition to the system data. However,

<footnote n="1">¹ https://www.redfunnel.co.uk</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Engine room of the Red Falcon, Red Funnel fleet</figcaption></figure>

sound, vibration, heat and smell, these are not commonly replicated effectively
in simulations [5].

## 2 Methodology

Initial user requirements were assessed through a combination of interviews with stakeholders, engineers and literature review through a successive approximation method [1] enabling us to develop a basic VR simulation in Unreal Engine 4 using the HTC VIVE Pro HMD system. Feedback by a dedicated of five subject matter experts (SMEs) from Warsash Maritime Academy were employed to refine the simulation at each iteration of the development cycle as well as accessing how realistic were the simulated failure sounds created for each scenario. SMEs were invited to provide feedback on multiple iterations throughout the development stage.

### 2.1 Representation of Sound and Vibration

Our research was not only aiming to replicate the real-world experience into virtual space but also to offer some potential enhancement of audio feedback. In particular, we examined how audio features, that are being lost due to the high sound pressure levels within the ER (measured value of 106dBLAeq,15 using the NTi XL2 acoustic analyzer), can enable engineers to diagnose current and potential issues and interact with.

The general background sound of the engine room was captured with a Soundfield SPS200 placed in key positions. To capture airborne sound with maximum rejection of spill from other components we use the Røde NTG2 shotgun microphone placed around 20cm from the source. We also use the Cold GoldTM electret contact microphones to capture sound directly from different locations of each machinery. These microphones are ‘electronic stethoscopes’ in which the electret transducer is embedded in a silicone cup, allowing direct connection to a
device while ensuring very high levels of rejection of spill. With the three different recording data, an operator from a remote location would be able to isolate and listen to specific components, as well as to the overall space, which would add to the capability of the system in terms of identification of mechanical problems. During interviews and discussions with engineers, they all point how experiencing vibrations as a whole or from individual components play a significant role in identifying failures. Sound and vibration are intrinsically linked to the performance and state of the machinery. To increase the level of impressiveness as part of the diagnostic feedback, it was important to include a representation of this vibration into the simulation. Several different options were considered; however, the majority of vibration based systems require the user to be seated. The *Subpak M2* is a body-worn bass transducer, which is designed to apply vibration directly from the audio signal to the back of the user. Therefore the user is mobile, which is an integral part of the VR experience.

## 2.2 System Failure Simulation

Mechanical failures such as cavitation of a pump, misfiring of the main engine and generator, single phasing of the compressor, air bleeds from the compressor, and damaged bearings or insufficient lubrication on several systems have particular sound characteristics. During our recording sessions, it was not feasible to disable active systems on the vessel or wait for a machinery failure to appear, and as a result, mechanical failure had to be simulated.

Recordings of similar system failures on isolated components were sourced through sound libraries and recordings from the engineering workshop. Sound files were combined ensuring that the simulated failure blended effectively with the original audio. The new sound was recognisably the same component, but with a characteristic of the sound associated with a particular mechanical failure. The SMEs who took part were asked to identify which sounds from engine room components were most accurately identified and which are the most accurate representations of particular mechanical failures to be used for the final simulator.

## 2.3 Server and User Interface

In order to test the ability of engineers to identify failures from a remote location, we had to be able to introduce failure scenarios into the VR system. A web interface hosted on a remote server was developed. The server enables the reproduction of real data and audio as if they are streamed from a real vessel into the remote location control centre. The server feeds data and audio control to the simulation in real time. Data for each component in the simulation including audio, and thus the felt vibrations, can be controlled to provide realistic representations of various levels of failure.

All critical components were modelled and receive simulated control data from the remote server. This enables the system administrator to set up varying scenarios, in which a combination of appropriate data values as well as audio
and vibration signals are used across several components to represent a par-
ticular mechanical failure. As a result, a user can explore and identify virtual
failures using familiar techniques as he/she might do normally in-situ. Hand-

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Selected component outlined, virtual handheld display shown towards the bottom of the figure</figcaption></figure>

held controllers allow the user to navigate around the ER, through a transport
feature, and select components to examine within the engine room. When se-
lecting a component, a virtual handheld display (Figure 2), attached to one of
the controllers, shows the available data from the conventional sensors. These
are mainly pressure and temperature, ability to isolate the sound of the selected
machinery, mute background sounds to listen to the component directly as well
as the ability to scroll through a ten-hour time history of system performance,
including audio, vibration and data for snapshots at hourly intervals.

3 User Testing

Seven marine engineers went through different simulate failure scenarios user with an average of 27 years of experience ranking between Chief Engineer and Second engineer. The usability test was undertaken using a blend of quantitative and qualitative methods in order to triangulate results. A set of performing tasks was aimed to determine the usability of the VR system for engineers with different levels and types of experience. Mainly to access how marine engineers were able to diagnose system failures using the simulation; examine to what extend audio feedback plays a role in enabling marine engineers to diagnose and predict early failures of components in remote monitoring situations.

The signal was spatialised using a binaural encoder and played simultane-
ously on a 4 channel (4.0) arrangement as well on the headset built-in head-
phones. This enabled users to choose to listen to either the loudspeaker based
audio, or the headphone-based audio, and get spatial cues from each. Engineers
were first familiarised with the system, without any simulated faults, in order
to ensure that they were able to navigate around the virtual space and access
the data and controls. After a period of familiarisation, each participant went
through one of four simulated scenarios. Two scenarios involve simulated faults,
and two had no fault. Simulations were presented in a randomised order. Par-
ticipants were not told whether or not there would be faults present but were
asked to undertake a system health check of the simulated ER as if they were
undertaking a normal ‘rounds check’. They were asked to assess whether or not
they felt there was a potential system failure, or if the system was operating
normally. If they felt failure was present, they were asked to state what was the
likely problem and cause.

Participants were asked to ‘think aloud’ throughout the familiarisation as
well as during the test so that a verbal record existed of their interaction with
the system. The facilitator observed noted the user’s behaviour, comments, and
actions.

A post-test questionnaire took place providing feedback regarding Attraction
fulness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty of the
system. A semi-structured interview followed aiming to gather further information and comments about their experience and functionality of the system.
Interviews were audio recorded, transcribed and analysed using a Qualitative data analysis (QDA) method.

## 4 Results

The focus of the usability testing was primarily qualitative, as the primary aim
was to ascertain whether engineers felt that the system had the potential for di-
agnostic use, whether the virtual environment and user interface were effective,
their perception of the immersive elements, and what improvements or alterna-
tive uses they felt the system could have. All seven engineers were able to identify
whether or not there was a system failure accurately, and also the location of
the failure, even if the simulated failure was a low risk. Three participants ac-
curately diagnosed the exact failure type, while the others identified a problem,
but stated that they would need further investigation in order to identify the
exact cause.

The overall response strongly focused on the immersive features, with ‘sound’
appearing in 22.8% of coded phrases, ‘vibration’ in 8.7% and ‘immersion’ in
5.8%. Within the features, ‘sound’ was by far the strongest theme. Participants
focused on how sound and vibration enable them to be immersed and made the
space familiar. Overall participants consider stimulation as ‘Excellent’ (Figure 3)
against the benchmark provided by the *User Experience Questionnaire (UEQ)*
<fnref n="2" />.

The use of vibrations was also a key theme as it was specifically identified in
terms of improving the ability to diagnose failures in the simulation along with

<footnote n="2">² https://www.ueq-online.org</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3. UEQ results showing the mean values of different aspects of the questionnaire</figcaption></figure>

the audio. One participant said that even though the vibration was limited it did give an edge. Another participant said that he couldn't hear changes of sound but could feel the vibration.

# 5 Discussion

The project focused on exploring how sound can be enhanced to compensate and even exceed the missing sensory information from a remote location. The simulation demonstrated that immersive audio and vibration feedback could aid towards remote operators in diagnosis of engine room failures. Engineers mentioned:

> If the pressures are fine, the temperatures are fine, but the noise has changed you know something's up (P5). Even though you see the data...you are more convinced that something has gone wrong through sound rather than the data(P1).

There were some negative statements regarding the fidelity of the system as the data values used for simulated machinery readings could be improved. These statements generally focused on technical aspects of implementation. For example, a machine which has been switched off in the simulation should retain data due to the heat at the sensors. In terms of user experience, the overall focus was on positive aspects. Users found the system to be generally easy to navigate and stated that they very quickly adapted to the simulation. Some concerns were raised regarding the use of the system when unfamiliar with the interface and controls, and the comfort of the system. From a usability perspective overall, the system was considered highly satisfactory. When analysing statements which expressed opinion, 74.9% of coded statements were 'positive' in focus. When questioned about the system potential for remote diagnosis through audio, all respondents gave overall positive feedback regarding the potential use. The ability to identify failures and the possible cause of the problem under time constrained circumstances on an unfamiliar vessel, suggests that the virtual environment and the integration of real-time immersive features offer significant potential for enhancing remote diagnostics. A key statement from several participants was that they did not listen for specific characteristics, but for change
of audio. Scrolling of historical data of any system was listed as a particularly positive feature. It allowed engineers to quickly identify changes in both audio/vibration and the data. Often the audio change would occur before a significant change in system's temperature or pressure. Engineers commented they were easily immersed in the space which resulted in becoming alert as if it was a real engine room. One engineer, when putting on the HMD to enter the simulation started talking louder even though the background sound levels remained the same. Due to high sound pressure levels, engineers need to talk loudly in order to communicate. The participant unconsciously spoke louder, feeling like he was in a real ER. When asked, after the session, why he spoke louder, he was unaware that he had done so.

# 6 Conclusion

This paper looks at how marine engineers can interact and diagnose failures from a remote location through audio. The potential of using VR systems with immersive audio and vibration enhances the diagnostic capabilities, particularly for remote operation of vessels at sea. Engine rooms are complex and active systems that change over time which are hard to replicate. Information like smell, heat and whole-body vibration proved to be impossible to reproduce precisely within the scope of this research. We recognise that due to excessive sounds in the ER vital information is often missed. Even though data from machinery can provide a good overview of the system they are not always reliable. As a result, the knowledge and experience of an engineer play an essential role in identifying failures including anticipating future failures.

We focused on exploring how sound qualities can be enhanced to compensate and even exceed the missing sensory information. The sound can be analysed and provide focused information to the operator in a remote location. The perception of sound experience in a virtual environment can be enhanced and refined to provide the best possible experience and information to the user. Our research showed that being able to control the volume, ability to filter out any unwanted surrounding noise, focus on a particular component, and the ten-hour audio history, engineers were able to understand failures and issues associated with the system failure. With the advances of sound techniques, live audio feed can be analysed and transmitted to a remote location providing a holistic view of the situation.

From the results presented above, we are confident that real-time feeds of sound as a diagnostic feature would be a significant benefit to remote operation and engine room diagnostics in the autonomous shipping industry, and this would benefit from further exploration and investigation. Engineers showed great interest in the ‘time history’ (from data and sound) as they could easily and quickly identify not only changes in the ER but also identify possible causes of failure. VR technology is a promising way forward in an immersive experience and possible development in monitoring and interaction in future autonomous shipping.
References

1. Allen, M.: Leaving ADDIE for SAM, an Agile Model for designing the best learning experiences. ASTD Press, IL (2012).

2. Fachot, M.: Virtual reality is central to aviation and maritime training. IEC e-tech, https://iecetech.org/issue/2016-04/Virtual-reality-is-central-to-aviation-and-maritime-training. Last accessed 21 Mar 2019 (2016)

3. Meadow, G., Barlow, C., Michalidis, T., Bracagna, S., Arrigoni, L., Broadhurst, E., Rajabally, E.: IMAGINE Work Package 1. Tech. Report (unpublished). Rolls-Royce/Solent University (2017)

4. Kuschel J., Ljungberg F.: Decentralized Remote Diagnostics: A Study of Diagnostics in the Marine Industry. In: Fincher S., Markopoulos P., Moore D., Ruddle R. (eds) People and Computers XVIII Design for Life. Springer, London (2005).

5. Kilteni, K., Groten, R., Slater, M.: The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments, 21(4), 373–387 (2012).

6. Resch, B., Wichmann, A., and Gll, N.: Usability in 4D AR: Visualising Multi-temporal Real-time Geo-data in Augmented Reality Environments. iJIM, 9(4), 23–33 (2015).