Towards concurrent real-time audio-aware agents with deep reinforcement learning

Venue: NLDL_org_2025_Conference
Published: 2024-09-07 11:54:01

Abstract: Audio holds significant amount of information about our surroundings. It can be used to navigate, assess threats, communicate, as a source of curiosity, and to separate the sources of different sounds. Still, these rich properties of audio are not fully utilized by current video game agents. 

We use spatial audio libraries in combination with deep reinforcement learning to allow agents to observe their surroundings and to navigate in their environment using audio cues. In general, game engines support rendering audio for one agent only. Using a hide-and-seek scenario in our experimentation we show how support for multiple concurrent listeners can be used to parallelize the runtime operation and to enable using multiple agents. Further, we analyze the effects of audio environment complexity to demonstrate the scalability of our approach.

Main paper: 
 introduction: 
 Modern video games have rich and high-quality audio scenes. Such audio holds significant amount of information on the virtual environment. Deep learning is becoming increasingly popular as an approach to extract information from virtual environments, but is not fully utilized to its potential especially when audio is considered.For video game non-player characters (NPCs), the utilization of audio could support intuitive, humanlike behavior. For example, an audio-aware agent could have a fear of loud noises as a self preservation mechanic. Audio-aware agents could be curious about odd noises, gather information about the surrounding environment and become susceptible to audio-based distractions deliberately made by human players or even other NPCs.While using deep learning for audio sensing in video games is not yet an extensively researched topic, there exists a wide range of applications that utilize it. For example, audio has been used to classify gunshot noises in video games [1] , to enhance exploration during training [2] and as a generic addition to visual observations [3] . Games engines can integrate multi-disciplinary functionalities and have wide applicability beyond entertainment. As such, game engines can be seen as generic platforms [4] for research on deep learning methods.However, game engines are often created with the 047 assumption that audio is rendered for the human 048 player only. As such, the environments can have 049 only one audio listener at a time. This listener can 050 either be assigned to the player or a single audio-051 aware agent. In order to support multiple concurrent 052 audio-aware agents, advanced methods are needed. 053 In this paper, we describe our experimentation 054 utilizing deep reinforcement learning (DRL). Our 055 experimentation focuses on sound source localiza-056 tion and is based on a hide-and-seek scenario, where 057 agents must locate targets based on spatial audio 058 cues. Our main contribution is to show that game 059 engines can be enhanced by using multi-listener au-060 dio rendering to enable parallelization of the runtime 061 operation and use multiple agents. We demonstrate 062 how spatial audio libraries can be used in a popular 063 game engine with machine learning support (the 064 Unity engine) to construct a multi-listener based 065 system that can be run in real-time on modern 066 workstations and yield good performance even if the 067 audio environments are complex. Our methodology 068 is not limited to games. It can be used also in other 069 domains that use similar virtual reality tools and 070 have similar requirements. Our code is available in 071 GitHub1 .The structure of this paper is the following. We 073 begin our presentation by reviewing related work in 074 Section 2. In Section 3, we describe our methodology 075 and our selection of tools. We continue by describing 076 our experimental setup (Section 4) and the results 077 from our experimentation (Section 5). We end our 078 paper with a short discussion (Section 6) and our 079 conclusions (Section 7). There exists a vast amount of machine learning 090 literature on audio. Considering our application 091 need to be perfect. Unlike with real-world embodied agents, mistakes can even be beneficial for the gameplay experience. Additionally, the agents need to be able to work with soft real-time simulations. Missing a deadline is not critical, but can harm the experienced quality of service. The agents also do not need to make decisions on every simulation tick, but still often enough to be able to fluently navigate the simulated environment.Considering the use of deep reinforcement learning for the audio-aware agents there are two challenges: 1) The computational requirements of audio-aware agents and the related spatial audio rendering by the runtime system should be relatively low. Running in real-time also during training avoids audio distortions.2) The training of the agents should be efficient enough in order to make their use realistic in the context of video game development. Especially the latter aspect is dependent on the operating environment of the agents as complex audio scenes make the training challenging. 
platform: 
 We use Unity [4] as our platform. We combine Unity with its deep reinforcement learning framework ML-Agents [4] . ML-Agents includes a training pipeline for simple DRL architectures with popular methods, such as PPO [13] . It also exposes the Unity API for external tools, such as Ray RLlib [14] enabling the use of complex DRL models.Virtual agents observe their surroundings through abstractions called sensors. In the context on ML-Agents, the simplest sensors are fixed length vector sensors, that can be filled with arbitrary values.To be able to use audio data as the input for our policy network, we need to have an audio sensor, but ML-Agents does not include any audiorelated sensors. We opted to use a third-party sensor from GitHub user mbaske [15] , which is available on GitHub with the MIT license. The sensor supports observing stereo audio in spectral domain using Short-Time Fourier Transform (STFT) [16] with different windowing functions [17] , such as Hanning and Rectangular window. The windowing function affects the amplitude and frequency accuracy of the sensor. The sensor constructs an observation batch by sampling the audio into a buffer over several simulation ticks.By default, popular game engines such as Unity and Unreal Engine support simplified spatialization without accounting for reflections or occlusions. To enable more complex audio observations, we use  
experiment setup: 
 In this section, we describe the setup for our experiment. Our goal is to create and evaluate the effect of using multiple listeners as proposed in Figure 1 (leftmost), while also training and evaluating a DRL-based audio-aware agent using Unity and Steam Audio. The multi-listener measurements are done on hardware configuration A as seen in Table 1 . The agents are trained using both configurations A and B.We consider a hide-and-seek scenario, where an audio-aware agent attempts to find the target (e.g., the player) in an indoors area. The target is constantly making noise by randomly playing back one of eight footstep audio clips. In order to make simplify the evaluation of the agents, the target is represented as a static object that does not move around in the environment. The motivation for this setup comes from a typical scenario, where the player attempts to evade detection.The agent can observe its surroundings with sensors as described in Figure 2 . The audio data is fed through STFT to a convolutional neural network (CNN) and the integration of other sensor data is done with a fully connected neural network (FCNN). In our experiment, the other sensors are excluded and the audio sensor is the only source of data.We measure the impact of running multiple simulation instances in parallel as proposed in Figure 1 (leftmost). Each Unity instance is running the exact same scene with Steam Audio and one audio agent with a CNN-based audio model. The number of audio sources is varied between 1, 10 and 30 audio sources to evaluate the impact of having several audio sources.The agent performance is compared in multiple environments with increasing complexity. The easiest environment ("Simple") is an empty hall without any obstructions. The second environment ("Medium") adds some obstructions in the form of rooms and walls, making it necessary to utilize indirect audio reflections. The third environment ("Complex") increases the difficulty by including more walls. The environments are described in more detail in Appendix B.The agents are trained using PPO [13] The reward R at physics update n can be expressed as:EQUATIONwhere a is the action vector, (ŝ) is a unit vector pointing to the shortest path to target and ∆t is the time between two physics updates (called fixed delta time in Unity). Maximum achievable reward during an episode is 2.To prevent agents from getting stuck between two locations, they need information on locations already visited. Instead of using more complex neural networks like RNNs [19] or LSTMs, we used a simple weighted navigation grid (Appendix A.3). The grid node weights indicate the likelihood of audio coming from that direction. We adjust these weights based on neural network outputs and navigate towards the highest-weighted neighboring node. If there are no occlusions between the target and the agent, the navigation grid is ignored and the agent will travel directly towards the output direction.We evaluate the viability of our multi-listener approach by measuring the CPU and memory (MEM) utilization with different configurations.We measure the agent performance using sample the agent with a fixed set of 100 randomized episodes 327 in each environment using the SPL metric [20] .SPL is defined as329 SPL = 1 N N i=1 S i l i max(p i , l i ) (2) 330where N is the number of episodes, S i is 1 if the 331 episode i succeeded and 0 otherwise. The l i is the 332 shortest path to target and p i is the length of the 333 path taken by the agent.As such, SPL accounts for both failure to reach 335 the goal and the optimality of the path compared to 336 the shortest path. It is better than simply measuring 337 number of steps or time to reach the target, since the 338 distance to the target varies by episode. However, 339 as SPL relies on knowing the optimal path length, 340 it is not suitable for moving targets as such. In this section, we describe our measurement results 343 on multi-listener performance, training performance, 344 and agent performance with some analysis on the 345 probable causes of the observed behavior. and STFT-H for Hanning window). The training was repeated three times. During training, we collect a sample on every simulation step. This increases the sampling throughput, but the samples will have overlap with each other as the audio buffer holds data over 10 steps.The Figure 5 shows the training reward curves. The curves are averaged over all three runs and then smoothed. The reward curves converge fast and there is little improvement after 5 million steps.Figure 6 shows the total training time for 10 million steps with four different configurations. In addition to the total time, it shows how much time is spent in collecting samples from the agents and how long is spent updating the model with the Stochastic Gradient Descent (SGD) [13] . With these configurations, increasing the amount of parallel Unity instances up to 10 significantly increases the sample throughput. Additionally, using GPU can nearly halve the total training time by drastically reducing time spent in computing SGDs.Despite having a higher-end CPU, the configuration A (CPU) is slower than B (CPU). This could be due to the different operating system or some other differences in hardware details, such as the memory speed or motherboard configuration.In conclusion, the sample throughput is a clear bottleneck in a GPU-accelerated system. It might be possible to further increase the sample throughput by having multiple agents in one Unity instance and by having faster-than-realtime audio rendering. Both of these could reduce the overhead from running multiple Unity instances in parallel. 
audio-aware agent performance: 
 After training the STFT-R and STFT-H agents, we compared them to a random agent that traverses randomly across the environment. As we repeated the training three times, the results are also averaged over three evaluations.The Figure 7 shows a comparison of SPL values achieved by different agents. The best possible SPL value is 1. As expected, the audio-aware agents perform best in the simple scene with no obstacles with an approximate SPL value of 0.85. The STFT-H However, it seems that our approach of using mul-464 tiple clients can ease these limitations workaround in Unity nearly a dozen of audio-aware agents can be supported. However, resource intensive games likely need more game-specific optimization than what we have done in our experimentation.Our results also highlight that minor details, such as the STFT windowing function, can noticeably affect agent behavior. Therefore, the audio properties of the environment and the audio sensor must be chosen carefully.As future work, we suggest looking at the other proposed approaches for creating multi-listener support to reduce overhead. Having only one game instance would remove the need for synchronization between the clients and reduce memory usage. Similar to Cowan et al. [10] it is also likely that some of the audio propagation computation could be reused.Additionally, modifying the audio engine to support faster-than-realtime playback would be helpful. For example, in our experiment, the sample throughput was around 250 samples per second. This is rather slow compared to the 120 000 samples per second reached with the faster-than-realtime solution by Hegde et al. [3] for VizDoom. Similar to the solution by Hedge et al., the audio backend would likely need to be modified to enable full software rendering of the audio at any sampling rate 
conclusion: 
 In this paper, we presented our experimentation on using multi-listener rendering in game engines for enabling concurrent audio-aware agents with deep reinforcement learning. Our work demonstrates how current audio libraries can be utilized to create audioaware agents with reasonable training results.While our results show the viability of using deep learning based audio-aware agents in video games, our research also highlights the need for further research on the methodology and development of applications and tools. The ability of faster-thanrealtime audio operations could significantly speed up training of audio-aware agents. Likewise, native support for multiple audio listeners would make training and runtime more efficient while enabling the use of multiple agents.  
 
 Appendix: The random agent does not have any NN-based model. Instead, it chooses random coordinates in the environment and navigates towards them using the Unity NavMesh. Upon reaching these coordinates, it selects new coordinates. This process is repeated until the agent gets within 15 units of the target.When the agent arrives within 15 units of the target, we assume that it can see the target and the agent navigates directly towards the target. Without having this kind of an detection range, it would be unlikely for the agent to ever encounter the target.The environments are 100x100 units large.Without any assistance, the DRL-based agents can easily get stuck between two points. Typically, this happens near walls: an agent infers that the audio is coming from the direction of the wall. The agent gets stuck in a loop of retreating and approaching.The behavior can be corrected by keeping a record of historical actions and observations, which we implement with a navigation grid (Navgrid).Navgrid is a grid of evenly spaced navigation nodes placed across the environment. Adjacent of the action will get an increase in weight, while 672 nodes in the opposite direction will get a reduction 673 in weight. The change in weight is propagated to 674 further neighbors with a discount of λ d-1 (where 675 d is the depth and λ = 0.5) up to the maximum 676 depth d max , as the uncertainty of the audio direction 677 increases the deeper we progress into the graph.The weight of the closest node to the agent is 679 set to 0, as we can be certain that the target is 680 not nearby. This is because the agent does not use 681 the Navgrid, if it has line-of-sight to the target. In 682 our experiment, the agent is given the line-of-sight 683 information, but it could also be predicted by the 684 agent from sensor data. In the experiment we used three separate environments. They can be seen in https://github.com/anonamee-333/anon-nldl-aaaa 


Decision: The paper addresses the (sometimes) overlooked area of audio based RL agents. The authors propose a solution where they focus on the parallelization of multi-listener systems, with a complete and well-motivated set of experiments. The work is both relevant and timely and can generate interesting discussions within the community. 

All the reviewers agree on the value of the contribution. The authors have also carefully considered their suggestions and improved the paper accordingly.

Reviews: review: title: title: Official Review of Submission32
paper_summary: paper_summary: This paper explores the application of reinforcement learning to train audio-aware agents that utilize both visual and auditory cues in real-time environments, particularly in a hide-and-seek scenario. The paper aims to overcome the limitations of game engines that typically support only a single audio listener by constructing a multi-listener based system. Through several experiments, the authors show the results on multi-listener performance and agent efficiency, while also measuring CPU and memory utilization to evaluate the feasibility of running multiple audio-aware agents simultaneously.
main_review: main_review: The authors have addressed the main concerns, particularly the motivation of this paper. The primary focus of this paper is on the parallelization of the multi-listener system, rather than advancing methodologies for audio-aware agents. The revision provides several new experiments with detailed explanations. Figure 6 also demonstrates the benefits of running multiple Unity instances concurrently. In conclusion, although the novelty of this paper is limited, the authors have effectively addressed most of my concerns, and I am inclined to accept this paper.
strength_weakness: strength_weakness: [Methods]
* The novelty of this paper is not enough. For example, compared to [1], both works present similar contributions in the domain of reinforcement learning with audio inputs. Both papers use similar techniques to integrate audio data into the reinforcement learning framework.

* It is unclear how the value loss is calculated, regarding whether the agent is rewarded for decreasing the length of the shortest path to the target.

* The procedure for updating the Navigrid is not described. For example, what is the formula for updating the weights in the Navigrid? What is the range for the weights of nodes in Navigrid? If a node propagates the increase in weights to its neighbors with a decay factor, why the weights around the agent are lower than the weights near the target in Figure B.1?

* In Appendix A (line 602), the paper mentions that "the target could be moving," and the GitHub README states, "Target-related scripts (Target of the audio agent, can move around randomly)." However, in line 254, it is mentioned that "the player is represented as a static object that does not move around in the environment." It is unclear how many targets are used in the environments and whether the targets move randomly during evaluation.

* The paper mentions the faster-than-realtime method in the methodology section, but it appears that it has not been implemented. It would be better to move it to the future work section.

[Experiments]
*  The paper does not have comparisons with other established baselines in the field of reinforcement learning with audio inputs.

* The paper does not include any ablation studies that examine the performance of the agent when trained solely on positional data and ray tracking, without incorporating audio data. As a result, it remains unclear whether adding audio information is necessary for this task.

* The README file on GitHub does not provide sufficient detail regarding the setup and execution of the code, which impedes the ability of other researchers to reproduce the experimental results reported in the paper.

Questions for authors:
1. The approach involves reinforcement learning training, which typically needs GPU resources. What are the specifications for GPU? If GPU resources are required, would training multiple agents simultaneously cost a lot of computation resources? Could you provide details on how resource usage is managed during training?
2. In section 3.4, it mentions head related transfer function (HRTF), which models the interaural level differences (ILD) and the interaural time difference (ITD). Is this function also used in the framework?
3. In Section 3.5, does this paper employ the faster-than-realtime technique as described in [1]?
4. Section 4.1 mentions that each Unity instance is running the exact same scene with Steam Audio and one audio agent with a CNN-based audio model. Do multiple agents train by the shared training data collected by all agents, or does each agent only trained by its own training data?
5. Can the agent successfully locate the target if it is trained solely using audio data?
6. Figure 5 shows the training curves obtained with 10 concurrent Unity instances. Do the results in Figure 6 use the same agents that were trained in Figure 5?
7. In Section A.2, do the random agents also use the Navigrid? If they do not use the Navigrid, would it be unfair to compare their performance with that of the trained agents?

[1] Hegde, A. Kanervisto, and A. Petrenko. “Agents that listen: High-throughput reinforcement learning with multiple sensory systems”. In: 2021 IEEE Conference on Games (CoG). IEEE, Aug. 2021. doi: 10.1109/cog52621. 2021.9619096.
title: title: Official Review
paper_summary: paper_summary: This paper explores the use of deep reinforcement learning (DRL) to train audio-aware agents in video games, addressing the challenge of limited multi-listener support in game engines. The authors propose a distributed architecture where each client runs its own audio engine instance (Figure 1). They train and evaluate their approach using a hide-and-seek task, measuring performance in environments of varying complexity. Results suggest that their method enables the training of multiple concurrent audio-aware agents with good results, when compared to random baselines.
main_review: main_review: Thank you for the response and additional clarification.
strength_weakness: strength_weakness: - While the STFT-H agent appears to achieve slightly higher performance than the STFT-R agent (Figures 5, 6), there is a lack of discussion regarding possible reasons and implications of such results. Why does the STFT-H agent perform slightly better? Is it the simplicity bias from the STFT-R technique? Or is there something fundamentally beneficial in the Hanning window approach used to perform the STFT that collaborates with the results? 

- The authors compare two variations of their proposed approaches with a RANDOM agent as a baseline. While the RANDOM agent serves as a reasonably good lower-bound performance, the comparisons lack an upper-bound performance which would demonstrate how well their approach compares with existing methods.  

- Lack of Deep Reinforcement Learning Insights: Given the paper's title and focus on audio-aware agents, the analysis of the deep reinforcement learning is somewhat superficial. I missed a bit deeper exploration of the agents' learning dynamics, the impact of hyperparameter choices on the agent's behavior, or a justification for the PPO algorithm.