Patent Description:
Augmented Reality (AR) is the merging of real and virtual worlds to produce new environments and visualizations where actual or real physical objects and digital or virtual objects co-exist and may interact in real time. AR brings a virtual world into a real-world environment of a user with true-to-life visuals and audio. AR mixes virtual sounds from virtual sound objects with real sounds in a real acoustic environment. Virtual sound from a virtual sound object should match equivalent real-world sound as played through headphones to a user to ensure a pleasing AR experience. Otherwise, the user experiences a degradation of the AR experience. Conventional techniques use complex multistep processes to match the virtual sound to the equivalent real-world sound. Such complexity introduces noticeable aural delays into an AR simulation, which may degrade the user experience. Moreover, the complexity disadvantageously increases processing requirements for, and thus the cost of, AR devices.

Document <CIT> discloses a method for acoustic classification and optimization for rendering of real-world scenes. The method generates a three-dimensional (3D) virtual model of a real-world scene from images that a camera captures of the scene. The method further determines acoustic material properties of surfaces in the 3D virtual model by using a visual material classification algorithm to identify materials in the real-world scene that make up the surfaces and known acoustic material properties of the materials. A visual material segmentation on camera images is performed, thereby producing a material classification for each triangle in a triangle mesh. The visual material classification algorithm may be based on a convolutional neural network classifier.

The invention provides for a method, an apparatus and a non-transitory computer readable medium with the features of the independent claims. Embodiments of the invention are identified in the dependent claims.

Extended reality (XR) generally encompass virtual reality (VR) and augmented reality (AR), sometimes referred to as mixed reality (MR). Audio signal reproduction systems have evolved to deliver three-dimensional (3D) audio to a listener. In 3D audio, sounds are produced by headphones or earphones (for simplicity, collectively referred to herein as "headphones") and can involve or include virtual placement of a sound source in a real or theoretical 3D space or environment auditorily perceived by the listener. For example, virtualized sounds can be provided above, below, or even behind a listener who hears 3D audio-processed sounds. Conventional audio reproduction via headphones tends to provide sounds that are perceived as originating or emanating from inside the head of the listener. In an example, audio signals delivered by headphones, including using a conventional stereo pair of headphones, can be specially processed to achieve 3D audio effects, such as to provide the listener with a perceived spatial sound environment.

A 3D audio headphone system can be used for VR applications, such as to provide the listener with a perception of a sound source at a particular position in a local or virtual environment where no real sound source exists. Similarly, a 3D audio headphone system can be used for AR applications, to provide the listener with the perception of the sound source at the position where no real sound source exists, and yet in a manner that the listener remains at least partially aware of one or more real sounds in the local environment. Computer-generated audio rendering for VR or AR can leverage signal processing technology developments in gaming and virtual reality audio rendering systems and application programming interfaces, such as building upon and extending from prior developments in the fields of computer music and architectural acoustics. Various binaural techniques, artificial reverberation, physical room acoustic modeling, and auralization techniques can be applied to provide users with enhanced listening experiences. A VR or AR signal processing system can be configured to reproduce some sounds such that they are perceived by a listener to be emanating from an external source in a local environment rather than from headphones or from a location inside the head of the listener.

Compared to VR 3D audio, AR audio involves the additional challenge of encouraging suspension of a participant's disbelief, such as by providing simulated environment acoustics and source-environment interactions that are substantially consistent with acoustics of a local listening environment. This presents a challenge of providing audio signal processing for virtual or added signals in such a manner that the signals include or represent the environment of the user, and such that the signals are not readily discriminable from other sounds naturally occurring or reproduced over headphones in the environment. Such audio signal processing provides accurate sound sources in a virtual sound field by matching and applying reverberation properties, including decay times, reverberation loudness characteristics, and/or reverberation equalization characteristics (e.g., spectral content of the reverberation) for a given listening environment. In audio-visual AR applications, computer-generated sound objects (referred to as "virtual sound objects") can be rendered via acoustically transparent headphones to blend with a physical environment heard naturally by the viewer/listener. Such blending can include or use binaural artificial reverberation processing to match or approximate local environment acoustics.

Embodiments presented herein provide a practical and efficient approach to extend 3D audio rendering algorithms or simulations to faithfully match, or approximate, physical local environment acoustics. The embodiments provide solutions to the above-mentioned problems and/or challenges, and also provide advantages that will become apparent from the ensuing description. The embodiments may be used in 3D audio applications, such as VR and AR, for example. The embodiments use machine learning (ML) techniques to predict acoustic properties of the local environment, such as reverberation characteristics, directly from images of the local environment captured by an image sensor. The embodiments may then use the predicted acoustic properties in an acoustic simulation of the environment that matches or approximates actual acoustics of the local environment. Based on the predicted acoustic properties, the acoustic environment simulation seamlessly blends virtual sound with the local environment, when perceived by a listener via headphones.

More specifically, embodiments presented herein use ML techniques to train one or more neural networks of an ML classifier to predict the acoustic properties of an unknown environment accurately using an image sensor. The predicted acoustic properties are then used to create an acoustic context for virtual sound objects in the form of an acoustic environment simulation created within that environment in real-time. The embodiments advantageously: make use of camera sensors that are generally available on an XR device; allow the use of typical audio plugins used in machine learning engines, such as Unity and Unreal engines; reduce complexity, processing requirements, and delay associated with matching virtual sound to an equivalent real-world sound in real-time AR environments compared to conventional techniques; provide scalable implementations depending on image sensor availability; and may be implemented as a deep learning inference engine.

At a high-level, embodiments presented herein employ ML techniques to classify images of a real-world (i.e., an actual) environment directly to an acoustic preset that represents a set of acoustic parameters for an acoustic environment simulation (AES). The set of acoustic parameters represent a set of properties sufficient to perform the AES. The AES simulates or models a sound response of the real-world environment based on the set of acoustic parameters of the acoustic preset. The acoustic preset is a parametric representation of the sound response. The AES applies the sound response to sound from virtual sound objects placed (virtually) in the real-world environment, to convert the sound to realistic sound that appears to originate, realistically, from the virtual sound objects when played to a user through headphones. The aforementioned real-world environment includes any real-world environment or space with reverberant qualities, such as, but not limited to, a room, auditorium, concert hall, outdoor theatre, and so on. The rooms may also include rooms in a home, such a kitchen, a living room, a dining room, a bathroom, and so on. The rooms may also include office spaces, and the like.

With reference to <FIG>, there is a high-level block diagram of an example XR system <NUM> configured to provide an AR experience to a user according to embodiments presented herein. Although the embodiments are described primarily in the context of AR applications, the embodiments apply equally to VR applications. In <FIG>, dashed-lines generally represent parametric flow, e.g., flows of acoustic parameters, while solid-lines generally represent image and sound signal flow.

XR system <NUM> includes an image sensor <NUM> to capture a sequence of images or video (collectively, "images") <NUM>, an AR display <NUM>, a headset <NUM> including left and right headphones, an optional position sensor <NUM>, and an XR processor or processor <NUM> coupled to, and that communicates with, the image sensor, the AR display, the headset, and the position sensor. XR processor <NUM> includes (i) an ML-based acoustic environment classifier <NUM> (referred to simply as an "ML classifier" <NUM>) that includes one or more neural networks to classify images <NUM> into acoustic presets <NUM> according to embodiments presented herein, and an interactive audio engine (IAE) <NUM>. IAE <NUM> may be implemented as part of XR processor <NUM> as shown in <FIG>, or may be separate from the XR processor. In an example, ML classifier <NUM> may include one or more convolutional neural networks (CNNs), such as AlexNet, GoogLeNet, and ResNet50. In other examples, ML classifier <NUM> includes non-CNN neural networks suitable for classifying images as descried herein. IAE <NUM> generates or performs an AES <NUM> based on acoustic presets <NUM> from ML classifier <NUM>, and also generates one or more virtual sound objects <NUM> for virtual placement into scenes of a real-world environment.

Image sensor <NUM> may include a video camera to capture a sequence of images <NUM> of the real-world environment. Image sensor <NUM> may be positioned at different positions and orientations (collectively, "vantage points") in the real-world environment to capture images <NUM> of different scenes of the real-world environment from the different vantage points. For example, image sensor <NUM> may include a video camera that is worn by a user who is a target of an AR experience, such that video camera operates to capture different scenes of the real-world environment as the user moves around in the real-world environment. Position sensor <NUM> senses or determines a position and an orientation of one or more objects, including the user, in the environment, and provides position information <NUM> indicative of the position and the orientation of the objects to XR processor <NUM>.

At a high-level, in operation, XR processor <NUM> processes (i) images <NUM> of the real-world environment, (ii) sound (i.e., sound signals) from virtual sound objects <NUM>, and (iii) position information <NUM>, when available, to produce a video signal <NUM> and a sound signal <NUM> representative scenes of the real-world environment augmented with the virtual sound objects and other virtual information. AR display <NUM> converts video signal <NUM> to video and plays the video to the user. The headphones of headset <NUM> convert sound signal <NUM> to sound and play the sound to the user. More specifically, ML classifier <NUM> of XR processor <NUM> employs deep learning neural network techniques to classify images <NUM> into acoustic presets <NUM>. Each of acoustic presets <NUM> represents a respective set of acoustic parameters, such as reverberation ("reverb") parameters, that represent sound properties of the real-world environment. IAE <NUM> performs AES <NUM> based on acoustic presets <NUM>, to simulate or model an acoustic response, including reverberation, for the real-world environment. IAE <NUM> also generates one or more virtual sound objects <NUM> placed at various virtual locations into scenes of the real-world environment. AES <NUM> applies the sound response to sound signals generated by virtual sound objects <NUM>, to convert the sound signals from the virtual sound objects to sound signals <NUM> that convey realistic sound for the virtual sound objects. That is, AES <NUM> models at least sound reverberation, for example, for the virtual sound objects.

With reference to <FIG>, there is a perspective view of an AR device <NUM> that may be worn by a user and used to convey an AR experience to the user. Device <NUM> includes a wide angle tracking camera <NUM>, a red, green, blue (RGB) camera <NUM>, a microphone array <NUM>, and stereo headphones <NUM> all coupled to a built-in XR processor, not shown in <FIG>. AR device <NUM> may also include a time of flight depth sensor and additional stereoscopic cameras coupled to the XR processor.

With reference to <FIG> there is an illustration of a generic sound response <NUM> for a real-world environment that may be simulated by the AES based on acoustic presets <NUM>. Sound response <NUM> occurs in response to a sound impulse that originates from a sound source in the real-world environment. Sound response <NUM> may be recorded at a listener position in the real-world environment that is spaced-apart from the sound source. Sound response <NUM> includes direct path (DP) sound, reflections (REF) including early reflections that follow the direct path sound, and reverberant energy or reverberations (REV) that follow the reflections. Reflections REF begin after a reflection delay from when the direct path sound DP occurs, and reverberations REV begin after a reverberation delay from when the reflections REF begin. The amplitude of reverberations REV decay according to a decay time of the reverberation. In the embodiments presented herein, AES <NUM> employs the acoustic parameters of acoustic presets <NUM> in addition to other acoustic parameters to simulate/represent direct path sound, early reflections, and reverberation with respect to sound from virtual sound objects <NUM>.

<FIG> is an illustration of an example method <NUM> of ML-based classifying of images <NUM> directly to acoustic presets <NUM>, thus not falling within the scope of the claims, and rendering of sound from virtual sound objects <NUM>, which may be performed by ML classifier <NUM> and IAE <NUM>. Method <NUM> assumes that ML classifier <NUM> has already been trained in an a priori training operation to classify images to acoustic presets, directly. Briefly, the training includes training ML classifier <NUM> on different images labeled with different ones of the acoustic presets. In an example, the acoustic presets may include M, e.g., <NUM>, <NUM>, and so on, acoustic presets P1-PM associated with labels L1-LM, respectively. Each acoustic preset Pi represents a set of acoustic parameters AP1-APN used for an acoustic environment simulation, e.g., AES <NUM>. Labels L1-LM and acoustic presets P1-PM (and their sets of associated acoustic parameters AP1-APN) may be stored in memory of XR processor <NUM>. Acoustic parameters AP1-APN have respective values for a given acoustic preset Pi, and the values vary across acoustic presets P1-PM. Acoustic parameters AP1-APN may include at least acoustic reverberation parameters. Generally, reverberation parameters may include, reverberation decay times, reverberation loudness characteristics, and/or reverberation equalization characteristics (e.g., spectral content of the reverberation), for example. More specifically, the reverberation parameters may include, high frequency attenuation and decay time, low frequency attenuation and decay time, damping, diffusion, density, room size, and so on. The reverberation parameters may include parameters as defined in in any known or hereafter developed acoustic-related standard, such as the Interactive 3D Audio Rendering Guidelines Level <NUM> (I3DL2).

Acoustic parameters AP1-APN may include additional acoustic parameters, such as one or more sound reflection parameters/coefficients, one or more sound absorption parameters/coefficients, and so on.

At <NUM>, XR processor <NUM> selects or establishes one of acoustic presets P1-PM as a default or initial acoustic preset for the AES. Acoustic parameters AP1-APN of the default acoustic preset represent initial acoustic parameters.

At <NUM>, ML classifier <NUM> receives an image among the sequence images <NUM> captured by image sensor <NUM>. In steady state operation, the image may be a current image among previous and future images among the sequence of images <NUM> to be processed sequentially through method <NUM>.

At <NUM>, referred to as "inference," (pre-trained) ML classifier <NUM> directly classifies the image into a set of multiple (current) classifications corresponding to acoustic presets P1-PM. The set of classifications may simply include labels L1-LM indicative of acoustic presets P1-PM with confidence levels C1-CN associated with respective ones of the labels. Labels L1-LM may be used to access respective ones of (known) acoustic presets P1-PM, and thus (known) acoustic parameters AP1-APN of the acoustic presets. For example, acoustic presets P1-PM may be stored so as to be indexed and thus retried based on labels L1-LM. Confidence level Ci represents a probability that the associated label Li/acoustic preset Pi is correct for the image, i.e., that the image was classified correctly to label Li/acoustic preset Pi. In this way, the classifications may be considered soft decisions, rather than hard decisions.

At <NUM>, XR processor <NUM> selects the label/acoustic preset associated with the greatest confidence level among confidence levels C1-CN among the classifications, to produce a (current) selected label/acoustic preset. The selected acoustic preset replaces the default acoustic preset from operation <NUM>. The selected acoustic preset is retrieved from memory (i.e., acoustic parameters AP1-APN of the selected preset are retrieved from memory).

At <NUM>, XR processor <NUM> updates IAE <NUM> with the selected acoustic preset, i.e., with parameters AP1-APN of the selected acoustic preset.

Method <NUM> repeats sequentially as next images among the sequence of images <NUM> arrive for classification, to produce a sequence of classification results corresponding to the sequence of images, and that are sequentially passed to IAE <NUM> for AES <NUM>.

A variation of method <NUM> conditions acoustic preset updates to IAE <NUM> on a predetermined confidence level threshold, which may introduce hysteresis into the updates provided to the IAE as the method repeats to classify successive images. More specifically, the variation only updates IAE <NUM> when one or more (current) classifications have confidence levels that exceed the confidence level threshold, in which case operations <NUM> and <NUM> proceed as described above. Otherwise, the variation does not update IAE <NUM>, i.e., the variation simply maintains a last, previous update to the IAE that exceeded the confidence level threshold. Assuming the classifications include softmax values (i.e., soft decisions) that represent or are associated with confidence levels as probabilities, the confidence level threshold may be set equal to a probability of <NUM>, for example. In that case, an update occurs only when the corresponding probability exceeds > <NUM>. To add hysteresis, the update may occur only when an average confidence level over a predetermined number > <NUM> of consecutive classifications (through operation <NUM>) exceeds <NUM>.

Various methods of classifying images using ML techniques are now described in connection with flowcharts of <FIG>. The methods may be performed to classify the images to classifications indicative of acoustic presets P1-PM described above. The methods may be performed by XR system <NUM>.

<FIG> is a flowchart of an example method <NUM> of using ML classification to classify an image of a real-world environment directly to a "best match" acoustic preset of acoustic parameters for an AES (e.g., AES <NUM>), and thus not falling within the scope of the claims. Method <NUM> summarizes operations described above in connection with method <NUM>. Method <NUM> assumes an ML classifier (e.g., ML classifier <NUM>) that was trained on many images of different real-world environments and that were labeled with various ones of acoustic presets P1-PM, so that the ML classifier is configured, as a result of the training, to classify an image directly to the acoustic presets (i.e., to the acoustic parameters of the acoustic presets), without intervening classifications or operations.

At <NUM>, an initial acoustic preset among acoustic presets P1-PM is established.

At <NUM>, an image of a scene of a real-world environment is captured.

At <NUM>, using a deep learning neural network inference, the image (received from <NUM>) is classified directly to M classifications indicative of acoustic presets P1-PM and their respective confidence levels C1-CN. The acoustic preset among acoustic presets P1-PM associated with the highest confidence level among confidence levels C1-CN is considered a "best match" acoustic preset to the real-world environment depicted in the image. That is, the simulated sound response generated by AES <NUM> based on the best match acoustic preset is closer to an actual sound response of the real-world environment than would be generated based on any of the other acoustic presets. At <NUM>, the best match acoustic preset may be identified/selected based on the confidence levels associated with the classifications/acoustic presets.

At <NUM>, it is determined whether to update AES <NUM> with the best match acoustic preset, as described above in connection with <FIG>, for example. If it is determined to update AES <NUM>, the best match acoustic preset is provided to the AES, and thus replaces the previous acoustic preset. Otherwise, AES <NUM> is not updated with the best match acoustic preset, and the AES uses a previous best match acoustic preset (i.e., the previous acoustic preset is not replaced).

From <NUM>, flow control returns to <NUM> and the process repeats for a next image.

<FIG> is a flowchart of an example method of using ML classification to classify an image of a real-world environment to a room type, from which an acoustic preset for an acoustic environment simulation may then be derived. Method <NUM> assumes an ML classifier that was trained on images of different real-world environments that were labeled with room types (e.g., kitchen, bathroom, living room, and so on), so that the ML classifier is configured, as a result of the training, to classify an image to a room type. Method <NUM> also assumes that respective ones of acoustic presets P1-PM may be assigned to, or derived from the room types resulting from the aforementioned classification.

At <NUM>, using a deep learning neural network inference, the image (received from operation <NUM>) is classified to a room type, e.g., kitchen.

At <NUM>, an acoustic preset among acoustic presets P1-PM associated with/assigned to the room type is retrieved.

At <NUM>, the acoustic preset from <NUM> may be used to update the AES.

In method <NUM>, inference operation <NUM> does not classify directly to an acoustic preset. Therefore, an extra operation, <NUM>, is used to identify the acoustic preset after the classification is performed. That is, the room type is translated to the acoustic preset.

<FIG> is a flowchart of an example method of using ML classification to classify an image of a real-world environment directly to an acoustic preset of acoustic parameters for AES <NUM> based on a cache of scenes of real-world environments and their associated acoustic presets. Method <NUM> is similar to method <NUM>, except that method <NUM> includes additional operations <NUM> and <NUM>, described below. Method <NUM> assumes that XR processor <NUM> determines which real-world environments, e.g., rooms, a user has been in, and records in a cache a (best) acoustic prefix for each of the rooms.

Flow proceeds from <NUM> and <NUM> to <NUM>. At <NUM>, it is determined whether the user has previously been in the room in which the user is currently positioned. If the user has been in the room previously, flow proceeds to <NUM>, where the acoustic prefix for the room is retrieved from the cache. Flow proceeds from <NUM> to <NUM>, which uses the acoustic prefix retrieved from the cache. If the user has not been in the room previously, flow proceeds to <NUM>, and operation continues as described above. An example of an XR processor configured to perform method <NUM> is described below in connection with <FIG>.

<FIG> is a flowchart of an example method of using ML classification to classify an image of a real-world environment directly to a general/primary acoustic preset and secondary acoustic modifiers for AES <NUM>, and thus not falling within the scope of the claims. More specifically, method <NUM> uses a first neural network of an ML classifier (e.g., ML classifier <NUM>) trained to classify an image of a real-world environment directly to general acoustic presets (also referred to as "primary acoustic presets"), as in method <NUM>. Each of the general acoustic presets includes a respective set of general acoustic parameters. For example, the general acoustic parameters may be reverberation parameters. Method <NUM> also uses a second neural network of the ML classifier trained to further classify the image to additional or secondary acoustic parameters, such as absorption and/or reflection parameters or coefficients, room volume, and so on, that may be used to modify the general acoustic presets.

At <NUM>, At <NUM>, an initial acoustic preset among acoustic presets P1-PM is established.

At <NUM>, using the first neural network, the image is directly classified to the general acoustic presets, from which the best general acoustic preset is selected, i.e., the acoustic preset associated with the highest confidence level is selected as the best acoustic preset.

At <NUM>, using the second neural network, the image is directly classified to the secondary acoustic parameters.

At <NUM>, one or more of the general acoustic parameters of the general acoustic preset selected at <NUM> are modified/adjusted based on one or more of the secondary acoustic parameters, to produce a modified general acoustic preset. For example, values of the general acoustic parameters of the general acoustic preset may be increased or decreased based on values of the secondary acoustic parameters. Alternatively, one or more of the general acoustic parameters may be replaced by one or more of the secondary acoustic parameters.

In a simple example, an absorption coefficient α in a fractional range <NUM><α<<NUM> may be used as a secondary acoustic parameter, in which case operation <NUM> may multiply one or more of the general acoustic parameters by the absorption coefficient α, to produce one or more modified general acoustic parameters. In practice, such a modification based on absorption may be more complex for the following reason. Since each material has its own absorption coefficient, early reflections from the material are usually directly influenced by the absorption coefficient of the material. Thus, reverberation in an acoustic environment comprising many different materials can be influenced by an aggregate of the materials in the environment, which collectively produce an aggregate absorption. The aggregate absorption may affect the delay rate of the reverberation differently in different frequency bands, which can be taken into account at operation <NUM>.

At <NUM>, the modified general acoustic preset may be used to update the AES.

From <NUM>, flow returns to <NUM>, and the process repeats.

With reference to <FIG>, there is a flowchart of an example method <NUM> that is similar to method <NUM>, except that method <NUM> includes 3D mesh processing operations <NUM> linked with operation <NUM> of method <NUM>. Only 3D mesh processing operations <NUM> are described. Generally, 3D mesh processing operations <NUM> map one or more of the secondary acoustic parameters produced at operation <NUM> to components of 3D mesh generated using a depth camera, for example.

At <NUM>, a depth camera captures a depth map (image) of the same real-world environment for which the image was captured at operation <NUM>.

At <NUM>, the 3D mesh is created from the depth map.

At <NUM>, a secondary acoustic parameter (e.g., material sound absorption) produced at operation <NUM> is mapped to the 3D mesh.

At <NUM>, the 3D mesh and the secondary acoustic parameter are exported.

Training and real-time operations of ML classifier <NUM> are now described in further detail in connection with <FIG>.

<FIG> is an illustration of an example method of training ML classifier <NUM> based on training images according to a first training scenario, and using the ML classifier, once trained, to classify images. Once trained, ML classifier <NUM> classifies images, typically in real-time, in what is referred to as an "inference" stage or operation. In the example of <FIG>, ML classifier <NUM> is configured with a CNN. For example, ML classifier <NUM> includes a convolutional layer <NUM> coupled to a fully connected layer <NUM>. In practice, ML classifier <NUM> may include many convolutional layers leading to the fully connected layer.

For training and for the inference stage, post training, ML classifier <NUM> receives an image <NUM> and produces classifications <NUM> in the form of labels representative of acoustic presets. In the inference stage, at <NUM>, an acoustic preset with a highest confidence is selected based on the labels and their confidence levels, as described above. During training, image <NUM> represents a training image on which ML classifier <NUM> trains.

In the first training scenario, training of ML classifier <NUM> may include the following operations:.

Operations (a)-(c) may be performed based on subjective sound design, i.e., substantially manually by a sound designer. The sound designer uses his/her experience with room acoustics to design respective acoustic presets with respective sets of the most likely sounding acoustic parameters for corresponding ones of scenes depicted in training pictures among many training pictures in a training database. That is, the sound designer designs each respective set of acoustic parameters to best represent or match the acoustic properties of a corresponding scene depicted in one of the training pictures based on subject design experience of the designer. For example, the designer selects a first set of reverberation parameters of a first acoustic preset for a "live" room (e.g., a live kitchen), selects a second set of reverberation parameters for a "dead" room (e.g., a heavily carpeted bedroom including fabric covered furniture), selects a third set of reverberation parameters of a third acoustic preset for a room having intermediate reverberation characteristics between those of the "live" room and the "dead" room, and so on. Then, the designer labels the training pictures with their most likely acoustic presets (which each represents a respective set of the acoustic parameters). For example, the designer labels training pictures of similar live-looking rooms with the first acoustic preset, labels training pictures of similar dead-looking rooms with the second acoustic preset, and labels training pictures of similar rooms that appear to have intermediate reverberation with the third acoustic preset, and so on.

An alternative to relying primarily on the experience of the sound designer to establish the acoustic presets for training uses actual acoustic measurements of rooms with different reverberant properties, and then algorithmically derives the acoustic presets from the acoustic measurements. For example, an acoustic impulse response for each room may be measured using any known or hereafter developed technique for measuring the acoustic impulse response of a real-world environment. Then, a set of acoustic parameters of an acoustic preset is algorithmically derived from the measured acoustic impulse response using any known or hereafter developed technique to derive reverberation parameters, for example, from the acoustic impulse response.

In one simplified example, the absolute value of the impulse response can be normalized and converted to a dB magnitude. The time from the initial pulse (normalized to 0dB) at which the dB magnitude falls below 60dB is taken as an RT60 decay time (i.e., how long it would take for a sound to decay 60dB in a room). With added frequency domain analysis, such methods can be extended to multiband analysis of RT60 times. Similarly, values for initial spectral energies, onset times, early reflection timing, and density, etc., can be directly observed in the impulse response or windowed sections thereof. It is understood that this particular technique is provided by way of example, only, and any additional or alternative methods of impulse analysis may be used.

Once trained, ML classifier <NUM> may be validated by determining that an arbitrary room model "sounds like" one would expect.

For the inference stage, ML classifier <NUM> (or logic external to the ML classifier) may be configured to apply a smoothing function on the softmax (output) classification produce by the ML classifier, such that the classification only transitions from its previous state (i.e., previous acoustic preset provided to AES <NUM>) if the softmax classification exceeds a softmax threshold, with some built in hysteresis to avoid spurious classification, similar to the thresholding described above in connection with method <NUM> of <FIG>. For example, the acoustic presets may transition smoothly using appropriate delay line interpolation and gain crossfading.

Training may also leverage transfer learning that takes advantage of a pre-trained neural network that already performs traditional room type classification. This approach freezes the convolutional layer of the pre-trained neural network (at feature extraction) continues to adapt the fully connected layer (classification) using the labels described above.

<FIG> is a diagram of an example operational flow for ML classifier <NUM> that shows training of ML classifier <NUM> according to a second training scenario and a third training scenario, and using the ML classifier, once trained, to classify images. The example of <FIG> is similar to the example of <FIG>, except that, in the inference stage, at <NUM>, acoustic parameters are updated from the labels output by classification.

In the second training scenario, the labels may be based on lower level acoustic parameters, such as reverberation parameters. The reverberation parameters may include I3DL2 acoustic parameters, for example. Initially, a sound designer uses his/her experience with room acoustics to design respective acoustic presets with sets of the most likely sounding acoustic parameters for corresponding ones of scenes depicted in training pictures among many training pictures in a training database. That is, each respective set of acoustic parameters is designed to best represent or match the acoustic properties of a corresponding scene depicted in one of the training pictures. Then, during inference, acoustic parameters are updated based on the labels, as shown at <NUM>.

In the third training scenario, the labels are based on lower level acoustic parameters that are derived from acoustic measurements of real acoustic properties taken in the same room as depicted in a training image. The acoustic measurement may include a measurement of a room (sound) impulse response, for example. Then, pre-training data preparation includes analyzing the room impulse response to automatically tune the appropriate acoustic parameters, i.e., perform automated tuning. The automated tuning, itself, may be based on an ML neural network.

Both the second and third training scenarios may take advantage of ML neural networks.

<FIG> is a diagram of an example operational flow for ML classifier <NUM> that shows training of the ML classifier according to a fourth training scenario, and using the ML classifier, once trained, to classify images. The example of <FIG> is similar to the examples of <FIG> and <FIG>, except for the following differences. In the example of <FIG>, ML classifier <NUM> includes a long short-term memory (LSTM) <NUM> following convolutional layer <NUM>. The neural network based on LSTM <NUM> is suitable for image descriptions. In the flow of <FIG>, LSTM <NUM> classifies to acoustic descriptors <NUM>. An operation <NUM> translates acoustic descriptors <NUM> to reverberation parameters, which are used in update operation <NUM>.

In the fourth training scenario, ML classifier <NUM> is trained on descriptive features of pictures that have acoustic relevance. Data preparation for pre-training includes labeling pictures of scenes of rooms with the given acoustics vocabulary. Although the example of <FIG> includes LSTM <NUM>, the LSTM may be replaced by a fully connected layer (e.g., fully connected layer <NUM>) if the focus is the most likely combination of labels (e.g. big, live, tile, and so on). Training includes a large number of training pictures labeled with the acoustic-specific descriptors.

With reference to <FIG>, there is an illustration of an example of image sensor <NUM> (e.g. a camera) that includes back-to-back <NUM>° fish-eye lenses, which together capture a <NUM>° image of a room in two camera perspectives (images) I180-<NUM> and I180-<NUM>, i.e., which capture a scene of an entire room. In this example, XR processor <NUM> may stitch together the different camera perspectives into a single rectangular image frame using any known or hereafter developed equi-rectangular projection technique. XR processor <NUM> provides the single rectangular image frame to ML classifier <NUM> for processing, during training or during the inference/real-time stage. In another example, traditional rectangular images may be used to train ML classifier <NUM>. Alternatively, the traditional rectangular images may be stitched together into a larger image, e.g., a composite image, based on tracing a room as a user/viewer points a camera at different views of the room. The resulting stitched image may be applied to ML classifier <NUM> even when the ML classifier has been trained on <NUM> images. In another example, image sensor <NUM> captures the rectangular image of the room, and XR processor <NUM> maps the rectangular image to an area on an equi-rectangular space, to produce a mapped image, and ML classifier <NUM> classifies the mapped image. Any known or hereafter developed technique may be used to map the rectangular image to the equi-rectangular space.

<FIG> is a block diagram of XR processor <NUM> according to an embodiment that includes additional functionality compared to that provided by the embodiment of the XR processor of <FIG>. In the example of <FIG>, XR processor <NUM> includes an image classification path <NUM>, a material estimation path <NUM>, acoustic parameter consolidation (APC) logic <NUM>, and IAE <NUM>. Image classification path <NUM> processes images <NUM> from image sensor <NUM> to produce a general/primary acoustic preset <NUM> and a secondary acoustic modifier <NUM>, and provides the general acoustic preset and the secondary acoustic modifier to APC logic <NUM>. Material estimation path <NUM> process images <NUM> from image sensor <NUM> in parallel with image classification path <NUM> to produce early reflection model data (ERE) (also referred to as "early reflection parameters") <NUM>, and provides the early reflection model data to APC logic <NUM>. APC logic <NUM> processes general acoustic preset <NUM>, secondary acoustic modifier <NUM>, and early reflection model data <NUM> together to produce final acoustic tuning parameters <NUM>, and provides them to AES <NUM> of IAE <NUM>.

Image classification path <NUM> includes an image preprocessor <NUM> (for acoustic analysis) followed by ML classifier <NUM>. Image preprocessor <NUM> processes images <NUM>, i.e., raw image data, to produce images in a format suitable for consumption by ML classifier <NUM>. Image preprocessor <NUM> formats the raw image data, and/or selects, recalls, or aggregates the raw image data to match training assumption for ML classifier <NUM>. For example, image preprocessor <NUM> may stitch together successive ones of images <NUM> to produce stitched images for classification, as described above.

Assuming ML classifier <NUM> has been trained to classify images to both general acoustic presets (with their confidence levels) and secondary acoustic modifiers, directly, the ML classifier classifies each of the images from image preprocessor <NUM> to general acoustic preset <NUM> and acoustic modifier <NUM>, directly. In an example, general acoustic preset <NUM> incudes initial reverberation parameters, and secondary acoustic modifier <NUM> may include one or more of an acoustic absorption parameter, an acoustic reflection parameter, an acoustic diffusion parameter, and specific environment (e.g., room) dimensions.

ML classifier <NUM> may produce general acoustic preset <NUM> and secondary acoustic modifier <NUM>, concurrently, provided there is sufficient image information, and sufficient ML classifier (e.g., neural network) processing power, for both types of classification to proceed concurrently. Alternatively, ML classifier <NUM> may (i) initially produce only general acoustic preset <NUM> based on initially received images and/or initially limited processing power, and, (ii) when further images arrive and/or further processing power is available, concurrently produce both the general acoustic preset <NUM> and secondary acoustic modifier <NUM>.

APC logic <NUM> modifies the (initial) reverberation parameters of general acoustic preset <NUM> based on acoustic modifier <NUM>, to produce a modified general acoustic preset including modified reverberation parameters, and provides the modified general acoustic preset to AES <NUM> in final acoustic tuning parameters <NUM>.

Material estimation path <NUM> includes an image preprocessor <NUM> (for geometric analysis) followed by an architectural mesh and material estimator (referred to simply as a "material estimator") <NUM>. Image preprocessor <NUM> processes the raw image data in images <NUM>, to produce images for consumption by material estimator <NUM>. Material estimator <NUM> constructs a (digital) architectural 3D mesh for the scenes depicted in the images, estimates types of materials depicted in the scenes based on the architectural 3D mesh, and estimates acoustic properties of the materials, to produce early reflection model data (e.g., parameters) <NUM> that includes the acoustic properties. Image preprocessor <NUM> and material estimator <NUM> may perform geometrical image analysis, generate an architectural mesh, and estimate material properties from the mesh using any known or hereafter developed techniques.

APC logic <NUM> combines early reflection model data <NUM> with the modified general acoustic preset into final acoustic tuning parameters <NUM>. Alternatively and/or additionally, APC logic <NUM> may further modify the modified general acoustic preset using various parameters in early reflection model data <NUM>.

In an embodiment that omits material estimation path <NUM>, early reflection model data <NUM> may still be used, but set to default values, for example.

<FIG> there is a block diagram of a portion of IAE <NUM> used to perform AES <NUM> based on acoustic parameters of final acoustic tuning parameters <NUM>. IAE <NUM> includes sound channels <NUM>(<NUM>)-<NUM>(<NUM>) that receive respective sound signals S1-SO from respective ones of virtual sound objects <NUM>. Each channel <NUM>(i) provides a respective direct path for sound signal Si to multi-channel output bus <NUM>, through a series of tunable gain (G), delay, and panoramic potentiometer (pan) stages. The resulting per-channel direct path sound signals are mixed into multi-channel output bus <NUM>. Each channel <NUM>(i) also provides a respective reflection path for sound signal Si to multi-channel output bus, through a tunable reflection stage (refl) that controls the reflection responsive to a reflection control signal <NUM>. Reflection control signal <NUM> may include one or more acoustic parameters of final acoustic tuning <NUM>, described above. The resulting per-channel reflections are also mixed into multi-channel output bus <NUM>. IAE <NUM> also includes a reverberation generator (rev) <NUM> fed by the per-channel reflections, and configured to reverberate the combined direct path sound signals and reflections combined on multi-channel output bus <NUM> responsive to a reverberation control signal <NUM>. Reverberation control signal <NUM> may include acoustic parameters (e.g., reverberation parameters) of final acoustic tuning parameters <NUM>.

<FIG> is a diagram of an example acoustic parameter refinement process <NUM> that may be performed by XR processor <NUM> of <FIG>, for example. At <NUM>, initial image data flows into ML classifier <NUM>, and the ML classifier classifies the initial image data directly, to produce soft decision labels for general acoustic presets P1-PM (e.g., general acoustic presets <NUM>). Each acoustic preset Pi includes a respective set of N acoustic parameters AP1-APN (Param <NUM> - Param N). One of general acoustic presets P1-PM is selected based on confidence levels, to produce a selected general acoustic preset. Early reflection model data (ERE) default parameters (e.g., default values for early reflection model data <NUM>) may be added to the selected general acoustic preset.

At <NUM>, further image data flows into ML classifier <NUM> and, based on the further image data, the ML classifier produces secondary acoustic modifiers (e.g., secondary acoustic modifiers) <NUM> in addition to general acoustic presets P1-PM.

At <NUM>, acoustic parameter safety check logic performs acoustic parameter safety checks on the selected general acoustic preset and the secondary acoustic modifiers to ensure the aforementioned acoustic parameters are within reasonable bounds given the (current) selected general acoustic preset, and additional information useful for performing the safety check. Following the safety checks, APC logic <NUM> modifies the selected general acoustic preset based on the secondary acoustic modifiers, to produce a modified/consolidated acoustic preset, including the N acoustic parameters, as modified. The ERE default parameters are retained with the modified/consolidated acoustic preset.

At <NUM>, material estimation path <NUM> generates early reflection model data <NUM> based on the initial image data and the further image data.

At <NUM>, the acoustic parameter safety check logic performs acoustic parameter safety checks on the modified/consolidated acoustic preset and early reflection model data <NUM>. APC logic <NUM> further modifies the modified/consolidated acoustic preset based on early reflection model data <NUM>, or simply adds the early reflection data to the modified preset, to produce final acoustic tuning parameters <NUM>.

<FIG> is a table of example acoustic parameters generated by XR processor <NUM> of <FIG> for images of a room. The table maps general acoustic parameters of general acoustic presets of a first row in the table, secondary acoustic parameters of a second row of the table, and early reflection data parameters in a third row of the table to various properties of the aforementioned parameters depicted in columns of the table. The first column of the table identifies the aforementioned three types of acoustic parameters. The next or middle two columns include information associated with (descriptive) labels of tags produced by ML classifier <NUM> and material estimation path <NUM>, as indicated in the columns. The last column provides examples of reverberation parameters and ERE parameters.

<FIG> is a block diagram of XR processor <NUM> in an embodiment that uses cached acoustic presets associated with known real-world environments, as described above in connection with method <NUM> of <FIG>. During a calibration operation, image preprocessor <NUM> and ML classifier <NUM> operate together to store in a calibrated preset memory or cache <NUM> acoustic parameters corresponding to previous "scenes. " After calibration, in real-time, image preprocessor <NUM> (which performs image matching) along with logic <NUM> (which may include ML classifier <NUM>) identify acoustic parameters from known real-time images, and recall the identified acoustic parameters without a full image analysis, as described above. In the embodiment of <FIG>, the machine learning is focused on an easier task of matching the room to one of the pre-calibrated scenes.

In the embodiment of <FIG>, APC logic <NUM> may be omitted, when calibrated preset memory or cache <NUM> stores general acoustic presets, secondary acoustic parameters, and early reflection model data, and when such data has already been subjected to acoustic parameter safety checks. IAE <NUM> finalizes reflections based on position information for virtual sound objects <NUM> and position information <NUM>.

<FIG> are directed to methods of transitioning between acoustic presets based on confidence levels of softmax classifications, adding usable image safety checks to the transitioning, and performing the transitioning between acoustic presets in a calibrated scene embodiment, respectively. Generally, the methods of <FIG> prevent classifiers of general/primary acoustic presets and secondary acoustic modifiers from being forced to guess on images or real-world scenes that do not have meaningful/discernible features.

<FIG> is a flowchart of an example method <NUM> of transitioning between acoustic presets, e.g., reverberation presets. At a loop including <NUM> and <NUM>, method <NUM> establishes a default acoustic preset for IAE <NUM> and checks for a confident acoustic environment change. If there is confident acoustic environment change, flow proceeds to <NUM> to update the acoustic preset, from which flow proceeds to <NUM>. Otherwise, flow returns to the loop. At a loop including <NUM>, <NUM>, and <NUM>, method <NUM> receives an environment-based acoustic preset at <NUM>, and checks for a confident acoustic environment change at <NUM>. If there is a confident acoustic environment change, flow proceeds to <NUM> to update the acoustic preset. Otherwise, flow proceeds back to <NUM>.

<FIG> is a flowchart of an example method <NUM> of transitioning between acoustic presets and performing usable image safety checks. Method <NUM> is similar to method <NUM>, except that method <NUM> further includes operations <NUM> and <NUM> that validate that current sensor data (i.e., images) as usable images. For example, operation <NUM>, inserted between operations <NUM> and <NUM>, validates the current sensor data as usable image(s). If yes, flow proceeds from to <NUM> to <NUM>, while, if no, flow returns to <NUM>. Similarly, operation <NUM> is inserted between operations <NUM> and <NUM>, and conditions flows between those operations similarly to the way operation <NUM> conditions flow between operations <NUM> and <NUM>.

<FIG> is a flowchart of an example method <NUM> of transitioning between acoustic presets in a calibrated scene embodiment, such as the embodiment depicted in <FIG>. At a loop including <NUM> and <NUM>, method <NUM> establishes a default acoustic preset and checks the current scene against scenes associated with calibrated scenes. If a calibrated scene is not found, flow returns to <NUM>. Otherwise, when a calibrated scene is found, flow proceeds to <NUM>, where method <NUM> updates the default acoustic preset with a calibrated acoustic preset. Flow proceeds from <NUM> to <NUM>, <NUM>, and <NUM>, which repeat the operations performed at <NUM>, <NUM>, and <NUM>, but starting with the updated calibrated acoustic preset from <NUM> instead of the default acoustic preset.

<FIG> is a block diagram that shows an example training process <NUM> used for deep neural network (DNN) training of ML classifier <NUM> as initially untrained, representative of training methods discussed above in connection with <FIG>, <FIG>, and <FIG>. At <NUM>, image preprocessor <NUM> formats training images of real-world environments from image sensor <NUM> for consumption by untrained ML classifier <NUM>, and provides the training images (as formatted) to a training input of the ML classifier. At <NUM>, an image preprocessor function presents the training images to a user/human operator on a display in a user friendly, easily viewable format. At <NUM>, the user applies to each of the training images labels/acoustic tuning parameters corresponding to acoustic presets, secondary parameters, and so on, such that the acoustic parameters match the real-world environments/scenes in the training images. The user associates the labels/acoustic tuning parameters with respective ones of the training images input to untrained ML classifier <NUM>, so that the training images correlate with their labels. At <NUM>, the one or more neural networks of ML classifier <NUM> train on the labeled training images. The training at <NUM> updates filter coefficients of the neural networks of ML classifier <NUM> based on the features in the training images, which results in a trained version of ML classifier <NUM>. Thus, the training process configures the one or more neural networks of ML classifier <NUM> to classify images directly to their corresponding acoustic presets.

With reference to <FIG>, there is a block diagram of an example computer device <NUM> in which XR processor <NUM> and IAE <NUM> may be implemented. There are numerous possible configurations for device <NUM> and <FIG> is meant to be an example. Examples of device <NUM> include a tablet computer, a personal computer, a laptop computer, a mobile phone, such as a smartphone, and so on. Device <NUM> may include outputs <NUM> to drive a display and headphones. Device <NUM> may also include one or more network interface units (NIUs) <NUM>, and memory <NUM> each coupled to a processor <NUM>. The one or more NIUs <NUM> may include wired and/or wireless connection capability that allows processor <NUM> to communicate over a communication network. For example, NIUs <NUM> may include an Ethernet card to communicate over an Ethernet connection, a wireless RF transceiver to communicate wirelessly with cellular networks in the communication network, optical transceivers, and the like, as would be appreciated by one of ordinary skill in the relevant arts.

Processor <NUM> may include a collection of microcontrollers and/or microprocessors, for example, each configured to execute respective software instructions stored in the memory <NUM>. Processor <NUM> may be implemented in one or more programmable application specific integrated circuits (ASICs), firmware, or a combination thereof. Portions of memory <NUM> (and the instructions therein) may be integrated with processor <NUM>. As used herein, the terms "acoustic," "audio," and "sound" are synonymous and interchangeable.

The memory <NUM> may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible (e.g., non-transitory) memory storage devices. Thus, in general, the memory <NUM> may comprise one or more computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions and when the software is executed (by the processor <NUM>) it is operable to perform the operations described herein. For example, the memory <NUM> stores or is encoded with instructions for control logic <NUM> to perform operations described herein related to ML classifier <NUM>, IAE <NUM>, image preprocessors <NUM> and <NUM>, APC logic <NUM>, material estimation path <NUM>, and the methods described above.

In addition, memory <NUM> stores data/information <NUM> used and generated by logic <NUM>, such as images, acoustic parameters, neural networks, and so on.

With reference to <FIG>, there is a flowchart of an example method <NUM> of classifying an image that includes various operations described above.

At <NUM>, the method receives an image of a real-world environment. To do this the method may capture the image using an image sensor, or access the image from a file of prestored images.

At <NUM>, the method uses an ML classifier, already or previously trained as described herein, to receive the image captured at operation <NUM> and to directly classify the image to classifications associated with, and indicative of, (known) acoustic presets for an AES. The classifications include respective confidence levels. The acoustic presets each includes (known) acoustic parameters that represent sound reverberation for the AES.

At the time of the classifying in operation <NUM>, the acoustic presets and their respective parameters are already known from the a priori training of the ML classifier. Thus, the ML classifier classifies the image "directly" to the classifications associated with and indicative of the acoustic presets without classifying to a room type first (and thus not falling within the scope of the claims), which would then require further operations to derive acoustic parameters from the room type, for example. The directly classifying of operation <NUM> is essentially a single classifying operation flowing from the image to the classifications that provides direct access to known/predetermined acoustic parameters associated with the classifications, without intervening parameter translations. Moreover, the AES uses the acoustic presets directly, i.e., as is. In an embodiment, the ML classifier was trained on (labeled) training images of real-world environments divided into different groups of the training images. The training images of the different groups of the training images are labeled with respective ones of the acoustic presets that are the same within each of the different groups, but that differ across the different groups. The training images may also be further labeled with additional (secondary) acoustic parameters, exploited in further operations <NUM>-<NUM>, described below.

At <NUM>, the method selects an acoustic preset among the acoustic presets (i.e., a particular one of the acoustic presets) based on the confidence levels of the classifications. The method accesses/retrieves the acoustic preset.

At <NUM>, the method performs the AES based on the acoustic parameters of the acoustic preset. The AES models sound reverberation for one or more virtual sound objects placed virtually in the real-world environment based on the acoustic parameters of the acoustic preset.

At <NUM>, the method use the machine learning classifier to further classify the image, or to classify one or more further images, directly, to produce one or more acoustic parameter modifiers. The further classifying may be concurrent with the classifying of operation <NUM>. Alternatively, the further classifying may result from receiving and classifying additional or subsequent images.

At <NUM>, the method modifies the acoustic parameters of the acoustic preset from <NUM> based on the one or more acoustic parameter modifiers from <NUM>, to produce a modified acoustic preset including modified acoustic parameters for the AES.

At <NUM>, the method performs the AES using the modified acoustic parameters.

Different combinations of operations <NUM>-<NUM> of method <NUM> may represent separate and independent embodiments. For example, operations <NUM>-<NUM> collectively represent an independent embodiment.

With reference to <FIG>, there is a flowchart of an example method <NUM> of classifying a subsequent or second image relative to the image classified in method <NUM>. Method <NUM> includes various operations described above.

At <NUM>, the method captures/receives a second image of the real-world environment.

At <NUM>, using the machine learning classifier, the method directly classify the second image to produce second classifications that have respective second confidence levels.

At <NUM>, the method determines whether one or more of the second classifications have respective second confidence levels that exceed a confidence level threshold.

At <NUM>, if one or more of the second classifications have respective second confidence levels that exceed the confidence level threshold, the method selects a second acoustic preset among the acoustic presets (a second particular one of the acoustic presets) based on the second confidence levels of the second classifications, and updates/replaces the acoustic preset with the second acoustic preset for the acoustic environment simulation.

At <NUM>, if one or more of the second classifications do not have corresponding second confidence levels that exceed the confidence level threshold, the method does not select a second acoustic preset, and does not update/replace the acoustic preset for the acoustic environment simulation.

In methods <NUM> and <NUM>, individual classifications may be based on one image or more than one image. For example, considering the context of classifying a sequence of images (or a sequence of image frames), the methods may classify one image at a time, to produce a separate classification for each image (or image frame); however, the classification preset (i.e., the acoustic preset presented to the AES) changes or updates when there is a significant/substantial difference in a "running average" of confidence levels for classifications from several such images (or image frames). Also, an image under classification may be augmented using multiple images from the image sensor, e.g., by stitching multiple perspectives to generate a less cropped perspective of the environment.

In summary, in one unclaimed embodiment, a method is provided comprising: receiving an image of a real-world environment; using a machine learning classifier, classifying the image to produce classifications associated with acoustic presets for an acoustic environment simulation, the acoustic presets each including acoustic parameters that represent sound reverberation; and selecting an acoustic preset among the acoustic presets based on the classifications.

In another unclaimed embodiment, an apparatus is provided comprising: a processor configured to: receive an image of a real-world environment; use a trained machine learning classifier including one or more neural networks to classify the image directly to classifications associated with acoustic presets for an acoustic environment simulation, the acoustic presets each including acoustic parameters that represent sound reverberation; select an acoustic preset among the acoustic presets based on the classifications; and perform the acoustic environment simulation based on the acoustic parameters of the acoustic preset.

In a further unclaimed embodiment, a non-transitory computer readable medium is provided. The computer readable medium is encoded with instructions that, when executed by a processor, cause the processor to perform the methods presented herein, including to: receive an image of a real-world environment; use a machine learning classifier, previously trained on training images of real-world environments labeled with respective ones of acoustic presets, the acoustic presets each including acoustic parameters that represent sound reverberation, to classify the image directly to classifications associated with the acoustic presets for an acoustic environment simulation; select an acoustic preset among the acoustic presets based on the classifications; and perform the acoustic environment simulation based on the acoustic parameters of the acoustic preset.

In another unclaimed embodiment, a system is provided comprising: an image sensor to capture an image of a real-world scene; a processor coupled to the image sensor and configured to: implement and use a previously trained machine learning classifier to classify the image directly to classifications associated with acoustic presets for an acoustic environment simulation, the acoustic presets each including acoustic parameters that represent sound reverberation; select an acoustic preset among the acoustic presets based on the classifications; and perform the acoustic environment simulation based on the acoustic parameters of the acoustic preset, to produce a sound signal representative of the acoustic environment simulation; and one or more headphones coupled to the processor and configured to convert the sound signal to sound.

Claim 1:
A method comprising:
receiving an image of a real-world environment;
using a trained machine learning classifier, classifying the image to produce classifications associated with acoustic presets (P1-PM) for an acoustic environment simulation, the acoustic presets (P1-PM) each comprising a set of acoustic parameters (AP1-APN), the acoustic parameters (AP1-APN) having respective values for a given acoustic preset (P1-PM) with the values varying across the acoustic presets (P1-PM), and the acoustic parameters (AP1-APN) including at least acoustic reverberation parameters, wherein the classifications include room types; and
selecting an acoustic preset among the acoustic presets (P1-PM) based on the classifications;
wherein the classifying includes classifying the image to produce the classifications such that the classifications have respective confidence levels (C1-CM);
wherein the selecting includes selecting the acoustic preset (P1-PM) such that the acoustic preset is associated with a classification among the classifications that has a highest one of the respective confidence levels (C1-CM).