<figure><img src="image_1.png" /></figure>

HAL
open science

# A method to convert neural signals into sound sequences

Alain Destexhe, Luc Foubert

► To cite this version:

Alain Destexhe, Luc Foubert. A method to convert neural signals into sound sequences. Journal of the Acoustical Society of America, 2022, 151 (6), pp.3685-3689. 10.1121/10.0011549. hal-03814504

HAL Id: hal-03814504

https://hal.science/hal-03814504

Submitted on 23 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A method to convert neural signals into sound sequences

Alain Destexhe •

<figure><img src="image_1.png" /></figure>

Luc Foubert

<figure><img src="image_2.png" /></figure>

Check for updates

J. Acoust. Soc. Am. 151, 3685–3689 (2022)

https://doi.org/10.1121/10.0011549

<figure><img src="image_3.png" /></figure>

View
Online

<figure><img src="image_4.png" /></figure>

Export
Citation

<figure><img src="image_5.png" /></figure>

ASA

LEARN MORE

Advance your science and career as a member of the
**Acoustical Society of America**
# A method to convert neural signals into sound sequences

Alain Destexhe ; Luc Foubert

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

Check for updates

J. Acoust. Soc. Am. 151, 3685–3689 (2022)

https://doi.org/10.1121/10.0011549

<figure><img src="image_5.png" /></figure>

View
Online

<figure><img src="image_6.png" /></figure>

Export
Citation

<figure><img src="image_7.png" /></figure>

ASA

LEARN MORE

Advance your science and career as a member of the
**Acoustical Society of America**
# A method to convert neural signals into sound sequences

Alain Destexhe<sup>a)</sup> and Luc Foubert

Paris-Saclay University, CNRS, Institute of Neuroscience, 91400 Saclay, France

**ABSTRACT:**

We present a method to convert neural signals into sound sequences, with the constraint that the sound sequences precisely reflect the sequences of events in the neural signal. The method consists in quantifying the wave motifs in the signal and using these parameters to generate sound envelopes. We illustrate the procedure for sleep delta waves in the human electro-encephalogram (EEG), which are converted into sound sequences that encode the time structure of the original EEG waves. This procedure can be applied to synthesize personalized sound sequences specific to the EEG of a given subject. © 2022 Acoustical Society of America. https://doi.org/10.1121/10.0011549

(Received 22 January 2022; revised 11 May 2022; accepted 14 May 2022; published online 2 June 2022)

[Editor: James F. Lynch]

Pages: 3685–3689

## I. INTRODUCTION

The sonification of brain activity has a long history, dating back to Adrian and Matthews (1934), who listened to the electro-encephalogram (EEG) translated directly to an (analog) audio signal. Music can also be made from discrete brain signals, such as the spiking activity of single neurons. In electrophysiology, it is very common to send the activity of a single neuron recorded by an electrode into an audio amplifier, which is well-known to help the identification and discrimination of neuronal sources by ear, much more than from visual inspection on an oscilloscope. One of the first "sonifications" of direct activity from neurons was due to Rodolpho Llinas and consisted of different "beeps" triggered by the activity of different simultaneously recorded neurons in the cerebellum. A similar idea was simultaneously implemented by the group of Ad Aertsen based on monkey unit recordings in the cerebral cortex (Aertsen and Erb, 1987), an approach called the "neurophone." There are also various other old and more recent examples of sonification of brain activity that can be found, such as the Spikiss Project (Destexhe and Foubert, 2018), which took advantage of the simultaneously recorded excitatory and inhibitory neurons to create musical sequences with at least two instruments.

More recently, the process known as neurofeedback (Papo, 2019; Sitaram et al., 2017) consists in a sonification of EEG, mostly associated with the alpha-rhythm frequency band (typically 10–12 Hz). The EEG is recorded and filtered in the alpha band, and the resulting signal is convolved with sound kernels to yield audible sound sequences, which are replayed in real-time to the subject. The subject, by modulating its own alpha-rhythm, is, therefore, able to modulate the sound and produce relaxation. This procedure is today at the basis of several commercial systems to help subjects to relax.

In the present paper, we introduce a method that does not depend on Fourier spectral filtering but relies uniquely on quantifying features in the time domain. This method consists of quantifying brain-wave motifs in the EEG and producing sounds based on the parameters of this quantification. We explain in detail how this process is realized by using slow-wave sleep (SWS) as an example.

## II. METHODS

Although the method described here is general (any physiological activity, any sensory modality), we will illustrate it by specific examples where the brain activity is used to generate acoustic stimuli. The implementation of this method proceeds according to three steps: (1) recording of brain activity, (2) analysis of brain signals, and (3) genesis of sound sequences. We consider each one of these steps successively.

### A. Recording of brain activity

The recording of brain activity is not part of the method itself but is a required step. We will consider here only one specific brain signal, the EEG, as an example all through. There exist many commercial systems to record the EEG, which can be used in the laboratory, in the clinical environment, or at home (portable devices). In particular, most EEG recording systems now proceed digitally, and one can thus obtain digital EEG recordings. They constitute the starting point of our study.

It must be kept in mind that the approach described here applies to other brain signals, such as neuronal (unit) recordings, local field potentials, magnetoencephalogram, brain imaging, etc. In all cases, the brain activity is obtained digitally, and the following procedures can be applied and adapted to each particular signal and each particular scale.

All signals shown here were obtained using a portable recording device [Dreem headband, version 2; see Arnal et al. (2020)], which is made specifically to record the EEG

<footnote><sup>a)</sup>Electronic mail: alain destexhe@cnrs.fr</footnote>
# A method to convert neural signals into sound sequences

Alain Destexhe<sup>a)</sup> and Luc Foubert

Paris-Saclay University, CNRS, Institute of Neuroscience, 91400 Saclay, France

## ABSTRACT:

We present a method to convert neural signals into sound sequences, with the constraint that the sound sequences precisely reflect the sequences of events in the neural signal. The method consists in quantifying the wave motifs in the signal and using these parameters to generate sound envelopes. We illustrate the procedure for sleep delta waves in the human electro-encephalogram (EEG), which are converted into sound sequences that encode the time structure of the original EEG waves. This procedure can be applied to synthesize personalized sound sequences specific to the EEG of a given subject. © 2022 Acoustical Society of America. https://doi.org/10.1121/10.0011549

(Received 22 January 2022; revised 11 May 2022; accepted 14 May 2022; published online 2 June 2022)

[Editor: James F. Lynch]

Pages: 3685–3689

## I. INTRODUCTION

The sonification of brain activity has a long history, dating back to Adrian and Matthews (1934), who listened to the electro-encephalogram (EEG) translated directly to an (analog) audio signal. Music can also be made from discrete brain signals, such as the spiking activity of single neurons. In electrophysiology, it is very common to send the activity of a single neuron recorded by an electrode into an audio amplifier, which is well-known to help the identification and discrimination of neuronal sources by ear, much more than from visual inspection on an oscilloscope. One of the first "sonifications" of direct activity from neurons was due to Rodolpho Llinas and consisted of different "beeps" triggered by the activity of different simultaneously recorded neurons in the cerebellum. A similar idea was simultaneously implemented by the group of Ad Aertsen based on monkey unit recordings in the cerebral cortex (Aertsen and Erb, 1987), an approach called the "neurophone." There are also various other old and more recent examples of sonification of brain activity that can be found, such as the Spikiss Project (Destexhe and Foubert, 2018), which took advantage of the simultaneously recorded excitatory and inhibitory neurons to create musical sequences with at least two instruments.

More recently, the process known as neurofeedback (Papo, 2019; Sitaram et al., 2017) consists in a sonification of EEG, mostly associated with the alpha-rhythm frequency band (typically 10–12 Hz). The EEG is recorded and filtered in the alpha band, and the resulting signal is convolved with sound kernels to yield audible sound sequences, which are replayed in real-time to the subject. The subject, by modulating its own alpha-rhythm, is, therefore, able to modulate the sound and produce relaxation. This procedure is today at the basis of several commercial systems to help subjects to relax.

In the present paper, we introduce a method that does not depend on Fourier spectral filtering but relies uniquely on quantifying features in the time domain. This method consists of quantifying brain-wave motifs in the EEG and producing sounds based on the parameters of this quantification. We explain in detail how this process is realized by using slow-wave sleep (SWS) as an example.

## II. METHODS

Although the method described here is general (any physiological activity, any sensory modality), we will illustrate it by specific examples where the brain activity is used to generate acoustic stimuli. The implementation of this method proceeds according to three steps: (1) recording of brain activity, (2) analysis of brain signals, and (3) genesis of sound sequences. We consider each one of these steps successively.

### A. Recording of brain activity

The recording of brain activity is not part of the method itself but is a required step. We will consider here only one specific brain signal, the EEG, as an example all through. There exist many commercial systems to record the EEG, which can be used in the laboratory, in the clinical environment, or at home (portable devices). In particular, most EEG recording systems now proceed digitally, and one can thus obtain digital EEG recordings. They constitute the starting point of our study.

It must be kept in mind that the approach described here applies to other brain signals, such as neuronal (unit) recordings, local field potentials, magnetoencephalogram, brain imaging, etc. In all cases, the brain activity is obtained digitally, and the following procedures can be applied and adapted to each particular signal and each particular scale.

All signals shown here were obtained using a portable recording device [Dreem headband, version 2; see Arnal et al. (2020)], which is made specifically to record the EEG

<footnote><sup>a)</sup>Electronic mail: alain destexhe@cnrs.fr</footnote>
(and possibly stimulate) during sleep. The signals were high-pass filtered (0.1 Hz) for movement artifacts, although very few such artifacts occur during sleep, and low-pass filtered at 128 Hz (acquisition rate was 250 Hz). Signals were scored manually to detect the SWS periods, where most delta waves occur. In this study, we used frontal leads (FP1, FP2) because they showed stable and high-amplitude slow-wave activity with little contamination from muscular activity.

## B. Analysis of brain signals

Signals were first converted to European Data Format (EDF) and visually inspected by using EDFbrowser, an open source software for data representation and analysis. Slow-wave sequences were detected based on the continuous presence of delta waves (which corresponds to N3 stage of sleep) and were cut out and exported for further analysis on a dataset of reduced size. EDFbrowser also contains low-pass, high-pass, or bandpass filtering capacities, which were used to obtain stable signals, and possible baseline drift was suppressed by high-pass filtering at 0.1 Hz. EDFbrowser also enables inspection of the signals by calculating power spectra. No noise removal was necessary.

The selected periods of slow waves were subsequently analyzed using custom-written PYTHON programs to scan the EEG, detect slow waves, and quantify them as indicated in Sec. III.

## C. Sound synthesis

Once the sequence of slow waves has been quantified and characterized, they are converted into sound sequences. The assumption here is that one EEG wave motif will correspond to one sound. This sound is triggered at the onset time of the motif, and its characteristics must reflect the size, rise, decay, and other parameters of the motif detected in the EEG. The natural parameters of synthesizer oscillators, attack, decay, sustain, release (ADSR), can be naturally obtained from the parameters of the motifs (see Sec. III).

The first parameter that must be precisely determined is the onset time of the detected delta wave. The onset time will be mapped to the onset time of the corresponding sound. The second parameter is the amplitude of the delta wave. Here, the amplitude can be mapped to different parameters of the sound wave. In Sec. III, we illustrate a mapping of the amplitude of the EEG wave onto the amplitude of the sound wave (which reflects its volume). It can also be mapped to another parameter, such as the pitch of the sound (for example, using a preset ensemble of notes), but this option was not considered here.

The third important parameter is the duration of the EEG wave. A natural conversion is the duration of the sound wave. Note that, in the case of ADSR parameters, the attack and decay can be mapped to the rise and decay times of the EEG wave, which will result in a similar duration between EEG and sound waves. In addition, the tail of the sound wave (with sustain parameter) could be used to let the sound last until the next sound comes in. There exist many possibilities of realizing this conversion.

More generally, we can assume that any parameter detected from the EEG wave can be routed, with transformation, to any parameter of the sound wave. These include parameters of the envelope, such as ADSR for example, but also parameters of a low-frequency modulation of the sound wave, which could be taken from another measurement, such as heart rate or respiration for example. Such a slow modulation was not attempted here.

The parameters extracted from the EEG signals were routed to a digital synthesizer (Triton, Korg, Tokyo, Japan), where the parameters were used to generate ADSR envelopes. The sound sequences generated by the synthesizer were exported in WAV format, which was subsequently compressed in MP3 format (see Mm. 1 and Mm. 2).

Mm. 1. Example of sound sequence generated from EEG delta waves of a human subject (see Fig. 4), using a single sound set repeated with different ADSR envelopes corresponding to the different delta waves. This is a file of type "mp3" (1733 KB).

Mm. 2. Example of sound sequence generated from the same set of EEG delta waves but using multiple sound sets. This is a file of type "mp3" (1819 KB).

## III. RESULTS

Our method generally addresses the genesis of sensory stimuli based on physiological signals. In this paper, we illustrate the particular case where the physiological signal is an EEG recording of brain activity, and the sensory stimulus is a sound sequence. We start by showing the procedure to convert EEG signals to sound sequences and then show concrete examples.

### A. A parametric analysis of brain waves

Figure 1 (top) shows an example EEG recording during SWS, which typically displays delta waves. The method consists of scanning the signal and detecting the delta waves using a two-threshold procedure (Fig. 1, bottom trace). The first threshold is used to detect an upward stroke of the signal at the beginning of the delta wave. The second threshold is used to detect the downward stroke, marking the end of the delta wave. Note that the use of two different thresholds makes the method adaptable to any particular waveform, including asymmetric waves. A minimal time (typically a few hundred ms) is imposed between the two threshold crossings to ensure that there is a sufficiently long excursion. When the delta wave is detected, the peak amplitude and peak time are measured, which automatically gives access to the rise time and decay time of the delta wave (Fig. 1, bottom).

The different delta waves are thus parametrized similarly, which results in a list of parameters for each wave, such as onset time, rise time, amplitude, peak time, decay time, etc. This sequence uniquely characterizes the sequence of EEG waves, encoding the main features of their dynamics.
(and possibly stimulate) during sleep. The signals were high-pass filtered (0.1 Hz) for movement artifacts, although very few such artifacts occur during sleep, and low-pass filtered at 128 Hz (acquisition rate was 250 Hz). Signals were scored manually to detect the SWS periods, where most delta waves occur. In this study, we used frontal leads (FP1, FP2) because they showed stable and high-amplitude slow-wave activity with little contamination from muscular activity.

## B. Analysis of brain signals

Signals were first converted to European Data Format (EDF) and visually inspected by using EDFbrowser, an open source software for data representation and analysis. Slow-wave sequences were detected based on the continuous presence of delta waves (which corresponds to N3 stage of sleep) and were cut out and exported for further analysis on a dataset of reduced size. EDFbrowser also contains low-pass, high-pass, or bandpass filtering capacities, which were used to obtain stable signals, and possible baseline drift was suppressed by high-pass filtering at 0.1 Hz. EDFbrowser also enables inspection of the signals by calculating power spectra. No noise removal was necessary.

The selected periods of slow waves were subsequently analyzed using custom-written PYTHON programs to scan the EEG, detect slow waves, and quantify them as indicated in Sec. III.

## C. Sound synthesis

Once the sequence of slow waves has been quantified and characterized, they are converted into sound sequences. The assumption here is that one EEG wave motif will correspond to one sound. This sound is triggered at the onset time of the motif, and its characteristics must reflect the size, rise, decay, and other parameters of the motif detected in the EEG. The natural parameters of synthesizer oscillators, attack, decay, sustain, release (ADSR), can be naturally obtained from the parameters of the motifs (see Sec. III).

The first parameter that must be precisely determined is the onset time of the detected delta wave. The onset time will be mapped to the onset time of the corresponding sound. The second parameter is the amplitude of the delta wave. Here, the amplitude can be mapped to different parameters of the sound wave. In Sec. III, we illustrate a mapping of the amplitude of the EEG wave onto the amplitude of the sound wave (which reflects its volume). It can also be mapped to another parameter, such as the pitch of the sound (for example, using a preset ensemble of notes), but this option was not considered here.

The third important parameter is the duration of the EEG wave. A natural conversion is the duration of the sound wave. Note that, in the case of ADSR parameters, the attack and decay can be mapped to the rise and decay times of the EEG wave, which will result in a similar duration between EEG and sound waves. In addition, the tail of the sound wave (with sustain parameter) could be used to let the sound last until the next sound comes in. There exist many possibilities of realizing this conversion.

More generally, we can assume that any parameter detected from the EEG wave can be routed, with transformation, to any parameter of the sound wave. These include parameters of the envelope, such as ADSR for example, but also parameters of a low-frequency modulation of the sound wave, which could be taken from another measurement, such as heart rate or respiration for example. Such a slow modulation was not attempted here.

The parameters extracted from the EEG signals were routed to a digital synthesizer (Triton, Korg, Tokyo, Japan), where the parameters were used to generate ADSR envelopes. The sound sequences generated by the synthesizer were exported in WAV format, which was subsequently compressed in MP3 format (see Mm. 1 and Mm. 2).

Mm. 1. Example of sound sequence generated from EEG delta waves of a human subject (see Fig. 4), using a single sound set repeated with different ADSR envelopes corresponding to the different delta waves. This is a file of type "mp3" (1733 KB).

Mm. 2. Example of sound sequence generated from the same set of EEG delta waves but using multiple sound sets. This is a file of type "mp3" (1819 KB).

## III. RESULTS

Our method generally addresses the genesis of sensory stimuli based on physiological signals. In this paper, we illustrate the particular case where the physiological signal is an EEG recording of brain activity, and the sensory stimulus is a sound sequence. We start by showing the procedure to convert EEG signals to sound sequences and then show concrete examples.

### A. A parametric analysis of brain waves

Figure 1 (top) shows an example EEG recording during SWS, which typically displays delta waves. The method consists of scanning the signal and detecting the delta waves using a two-threshold procedure (Fig. 1, bottom trace). The first threshold is used to detect an upward stroke of the signal at the beginning of the delta wave. The second threshold is used to detect the downward stroke, marking the end of the delta wave. Note that the use of two different thresholds makes the method adaptable to any particular waveform, including asymmetric waves. A minimal time (typically a few hundred ms) is imposed between the two threshold crossings to ensure that there is a sufficiently long excursion. When the delta wave is detected, the peak amplitude and peak time are measured, which automatically gives access to the rise time and decay time of the delta wave (Fig. 1, bottom).

The different delta waves are thus parametrized similarly, which results in a list of parameters for each wave, such as onset time, rise time, amplitude, peak time, decay time, etc. This sequence uniquely characterizes the sequence of EEG waves, encoding the main features of their dynamics.
Delta waves

<figure><img src="image_6.png" /></figure>

FIG. 1. Detection and parametrization of delta-wave motifs in the EEG. The top trace shows a typical EEG recording during the deepest phase of SWS in a normal human subject. A magnification of one of the detected delta waves is shown in the bottom trace. A procedure with two independent thresholds is used to detect the upstroke of the wave (threshold 1) and the downstroke (threshold 2). Examples of the different parameters extracted are indicated. Recording was during sleep stage N3, FP1 derivation.

## B. Sonification of EEG waves

The basis for the sonification of the EEG waves is that the sequence of parameters may be very easily converted to the classic parametrization of sound envelopes in digital synthesizers: ADSR parameters (Fig. 2). These ADSR parameters determine the shape of the envelope, and the basis of the method here is to create sound envelopes that will precisely reflect the parameters of the EEG waves.

Thus, for each wave, the ADSR parameters are extracted from the EEG wave parameters determined above, and a sound sequence is generated. An example of such a procedure is shown in Fig. 3. In this illustration, four delta waves are characterized and translated in four sounds. For each sound, the duration, amplitude, and dynamics correspond to the parameters of the delta waves. One can see very well that, in this scheme, the sequence of wave motifs in the EEG is translated into a corresponding sequence of

sounds, respecting their timing and form. The obtained sound waves thus strictly respect the sequence of events in the EEG, which is at the basis of the method presented here. An example of such translation is shown in Fig. 4 for a set of delta waves recorded in human SWS.

It is important to note that this method is not bound to the delta waves produced in SWS but could be applied as well to other types of EEG activity that display numerous rhythms (Buzsaki, 2006). Figure 5 shows an example of EEG recording in an awake subject during focused attention (reading), where frontal leads can express 15-25 Hz oscillations called beta waves. As for delta waves, the beta waves can be characterized by detecting their onset time, rise, decay, amplitude, etc. These parameters could also form the basis of sound sequences that would be specific to beta waves.

Finally, we provide a few sound samples as Mm. 1 and Mm. 2. The two sound samples correspond to the same sequences of delta waves that were analyzed and quantified. In the first sound sequence (Mm. 1), the same sound was used and was played with different amplitudes and durations. This example corresponds to that shown in Fig. 4. In the second sound sequence (Mm. 2), different sounds were randomly used, but the envelopes are the same as in sequence 1. These two sequences illustrate that the method can be used to generate sounds that encode the dynamics of EEG waves.

<figure><img src="image_7.png" /></figure>

FIG. 2. Example of an envelope generator. This example shows the scheme of a sound envelope with four parameters. A, attack: rise of the envelope; D, decay: decay time of the envelope after reaching its maximum amplitude; S, sustain, which quantifies the sustained part of the sound (usually expressed as a percentage of its maximal amplitude); R, release: the decay of the sound after the note has been released. These ADSR parameters constitute the most common parameters to drive oscillators to generate synthetic sounds.

## IV. DISCUSSION

We have shown here a new paradigm to convert physiological activity into sensory stimuli, which we illustrated for the case of brain activity translated into acoustic stimuli. The principal originality and innovation of this method is that the acoustic stimulus consists of a sequence of sounds that strictly preserves the temporal sequence of events in brain activity. It is thus necessary to first identify these events (or “wave motifs”) in brain activity, which was illustrated here with the delta waves that form the activity during
<figure><img src="image_8.png" /></figure>

FIG. 1. Detection and parametrization of delta-wave motifs in the EEG. The top trace shows a typical EEG recording during the deepest phase of SWS in a normal human subject. A magnification of one of the detected delta waves is shown in the bottom trace. A procedure with two independent thresholds is used to detect the upstroke of the wave (threshold 1) and the downstroke (threshold 2). Examples of the different parameters extracted are indicated. Recording was during sleep stage N3, FP1 derivation.

## B. Sonification of EEG waves

The basis for the sonification of the EEG waves is that the sequence of parameters may be very easily converted to the classic parametrization of sound envelopes in digital synthesizers: ADSR parameters (Fig. 2). These ADSR parameters determine the shape of the envelope, and the basis of the method here is to create sound envelopes that will precisely reflect the parameters of the EEG waves.

Thus, for each wave, the ADSR parameters are extracted from the EEG wave parameters determined above, and a sound sequence is generated. An example of such a procedure is shown in Fig. 3. In this illustration, four delta waves are characterized and translated in four sounds. For each sound, the duration, amplitude, and dynamics correspond to the parameters of the delta waves. One can see very well that, in this scheme, the sequence of wave motifs in the EEG is translated into a corresponding sequence of

sounds, respecting their timing and form. The obtained sound waves thus strictly respect the sequence of events in the EEG, which is at the basis of the method presented here. An example of such translation is shown in Fig. 4 for a set of delta waves recorded in human SWS.

It is important to note that this method is not bound to the delta waves produced in SWS but could be applied as well to other types of EEG activity that display numerous rhythms (Buzsaki, 2006). Figure 5 shows an example of EEG recording in an awake subject during focused attention (reading), where frontal leads can express 15-25 Hz oscillations called beta waves. As for delta waves, the beta waves can be characterized by detecting their onset time, rise, decay, amplitude, etc. These parameters could also form the basis of sound sequences that would be specific to beta waves.

Finally, we provide a few sound samples as Mm. 1 and Mm. 2. The two sound samples correspond to the same sequences of delta waves that were analyzed and quantified. In the first sound sequence (Mm. 1), the same sound was used and was played with different amplitudes and durations. This example corresponds to that shown in Fig. 4. In the second sound sequence (Mm. 2), different sounds were randomly used, but the envelopes are the same as in sequence 1. These two sequences illustrate that the method can be used to generate sounds that encode the dynamics of EEG waves.

<figure><img src="image_9.png" /></figure>

FIG. 2. Example of an envelope generator. This example shows the scheme of a sound envelope with four parameters. A, attack: rise of the envelope; D, decay: decay time of the envelope after reaching its maximum amplitude; S, sustain, which quantifies the sustained part of the sound (usually expressed as a percentage of its maximal amplitude); R, release: the decay of the sound after the note has been released. These ADSR parameters constitute the most common parameters to drive oscillators to generate synthetic sounds.

## IV. DISCUSSION

We have shown here a new paradigm to convert physiological activity into sensory stimuli, which we illustrated for the case of brain activity translated into acoustic stimuli. The principal originality and innovation of this method is that the acoustic stimulus consists of a sequence of sounds that strictly preserves the temporal sequence of events in brain activity. It is thus necessary to first identify these events (or “wave motifs”) in brain activity, which was illustrated here with the delta waves that form the activity during
<figure><img src="image_8.png" /></figure>

FIG. 3. (Color online) Illustration of the synthesis of a sound sequence reflecting the shape of EEG delta waves. (A) Scheme of four successive delta waves (marked by green dots) from the EEG signal. Their onset and offset times are indicated by dotted lines. (B) Each delta wave is parametrized, resulting in a list of parameters for each wave. (C) These parameters are passed to a sound synthesis unit, where the parameters of each delta wave will determine the parameters of each sound envelope. The sound sequence obtained reflects the sequence of EEG waves, both in amplitude and duration in this particular case.

the deepest phases of SWS. The next step is to parametrize these motifs by an algorithm that scans the signal(s). Once this parametrization is done, the resulting list of parameters (including the most important, the onset times) is easily converted into sound sequences by mapping these parameters into the parameters that drive audio synthesizers. The obtained sound sequence can be replayed to the subject who has produced the brain activity, and we expect this personalized replay will be particularly efficient in affecting the

subject's brain state, such as relaxation or falling asleep for example. A preliminary study shows that a majority of subjects (about 90%) report relaxing effects, while some (about 66%) could fall asleep—this study is presently ongoing and will be reported later, including an evaluation of the personalized effect.

In a more general perspective, in addition to brain activity, one could also use more complex sets of physiological signals, such as the heart, blood pressure, oxygenation level, respiration, etc. The same method would lead to a larger number of simultaneous parameters, which could be translated either to multiple simultaneous sound sequences or to several sensory modalities. For example, the brain activity during attentiveness (beta waves) and the respiration could be used together to design visual stimuli that could be tested as a means to help maintain the attention of the subject. This could be through variations of the background light or

<figure><img src="image_9.png" /></figure>

FIG. 4. (Color online) Example of translation of EEG waves to sound waves. (A) 1-min recording of sleep EEG from a normal human subject during deep SWS (N3 stage). Green dots indicate the delta waves detected. (B) Sequence of sound waves produced by this set of delta waves. The sound corresponds to Mm. 1.

<figure><img src="image_10.png" /></figure>

FIG. 5. (Color online) Detection and quantification of beta-wave motifs in an attentive subject. The EEG signal during attentive wakefulness (top, FP1 derivation) consists of oscillations in the 15-25 Hz frequency range, called beta oscillations. The bottom trace shows a magnification of the signal and the detection of individual motifs (green dots) within the oscillatory sequence (same quantification procedure as in Fig. 1). The procedure yields a list of parameters characterizing the sequence of beta waves.
<figure><img src="image_10.png" /></figure>

FIG. 3. (Color online) Illustration of the synthesis of a sound sequence reflecting the shape of EEG delta waves. (A) Scheme of four successive delta waves (marked by green dots) from the EEG signal. Their onset and offset times are indicated by dotted lines. (B) Each delta wave is parametrized, resulting in a list of parameters for each wave. (C) These parameters are passed to a sound synthesis unit, where the parameters of each delta wave will determine the parameters of each sound envelope. The sound sequence obtained reflects the sequence of EEG waves, both in amplitude and duration in this particular case.

the deepest phases of SWS. The next step is to parametrize these motifs by an algorithm that scans the signal(s). Once this parametrization is done, the resulting list of parameters (including the most important, the onset times) is easily converted into sound sequences by mapping these parameters into the parameters that drive audio synthesizers. The obtained sound sequence can be replayed to the subject who has produced the brain activity, and we expect this personalized replay will be particularly efficient in affecting the

subject's brain state, such as relaxation or falling asleep for example. A preliminary study shows that a majority of subjects (about 90%) report relaxing effects, while some (about 66%) could fall asleep—this study is presently ongoing and will be reported later, including an evaluation of the personalized effect.

In a more general perspective, in addition to brain activity, one could also use more complex sets of physiological signals, such as the heart, blood pressure, oxygenation level, respiration, etc. The same method would lead to a larger number of simultaneous parameters, which could be translated either to multiple simultaneous sound sequences or to several sensory modalities. For example, the brain activity during attentiveness (beta waves) and the respiration could be used together to design visual stimuli that could be tested as a means to help maintain the attention of the subject. This could be through variations of the background light or

<figure><img src="image_11.png" /></figure>

FIG. 4. (Color online) Example of translation of EEG waves to sound waves. (A) 1-min recording of sleep EEG from a normal human subject during deep SWS (N3 stage). Green dots indicate the delta waves detected. (B) Sequence of sound waves produced by this set of delta waves. The sound corresponds to Mm. 1.

<figure><img src="image_12.png" /></figure>

FIG. 5. (Color online) Detection and quantification of beta-wave motifs in an attentive subject. The EEG signal during attentive wakefulness (top, FP1 derivation) consists of oscillations in the 15-25 Hz frequency range, called beta oscillations. The bottom trace shows a magnification of the signal and the detection of individual motifs (green dots) within the oscillatory sequence (same quantification procedure as in Fig. 1). The procedure yields a list of parameters characterizing the sequence of beta waves.
screen background in computers, which may be helpful for maintaining the attention of subjects. Future studies should examine such possible applications of the method.

## ACKNOWLEDGMENTS

This research was supported by CNRS, the European Commission (Grant Nos. H2020-785907 and H2020-945539), and ANR (projects PARADOX and SHOOTINGSTAR). The authors wish to declare that the present method is the object of a patent application (Destexhe and Foubert, 2017, 2020).

Adrian, E. D., and Matthews, B. H. (1934). "The interpretation of potential waves in the cortex," *J. Physiol.* **81**(4), 440–471.

Aertsen, A., and Erb, M. (1987). "Scherzo cerebri, ma non troppo: Das neurophon" ("Brain fantasies, but not so much: the Neurophon"), in *Wahrnehmungen Des Gehirns (Perceptions of the Brain)* (Tuebingen Press, Germany), pp. 67–86.

Arnal, P. J., Thorey, V., Debellemaniere, E., Ballard, M. E., Bou Hernandez, A., Guillot, A., Jourde, H., Harris, M., Guillard, M., Van Beers, P., Chennaoui, M., and Sauvet, F. (2020). "The Dreem Headband compared to polysomnography for electroencephalographic signal acquisition and sleep staging," *Sleep* **43**(11), zsaa097.

Buzsaki, G. (2006). *Rhythms of the Brain* (Oxford University, New York).

Destexhe, A., and Foubert, L. (2017). French patent FR-3065366, "Physio-sensory transduction method and device," WO2018197155 (April 25, 2017), owner CNRS.

Destexhe, A., and Foubert, L. (2018). "Composing music from neuronal activity—The Spikiss Project," in *Exploring Transdisciplinarity in Art and Sciences* (Springer, New York), pp. 237–253.

Destexhe, A., and Foubert, L. (2020). U.S. patent 2020-0138322-A, "Physio-sensory transduction method and device" (May 7, 2020), owner CNRS.

Papo, D. (2019). "Neurofeedback: Principles, appraisal, and outstanding issues," *Eur. J. Neurosci.* **49**(11), 1454–1469.

Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis-Peacock, J., Weiskopf, N., Blefari, M. L., Rana, M., Oblak, E., Birbaumer, N., and Sulzer, J. (2017). "Closed-loop brain training: The science of neurofeedback," *Nat. Rev. Neurosci.* **18**(2), 86–100.
screen background in computers, which may be helpful for maintaining the attention of subjects. Future studies should examine such possible applications of the method.

## ACKNOWLEDGMENTS

This research was supported by CNRS, the European Commission (Grant Nos. H2020-785907 and H2020-945539), and ANR (projects PARADOX and SHOOTINGSTAR). The authors wish to declare that the present method is the object of a patent application (Destexhe and Foubert, 2017, 2020).

Adrian, E. D., and Matthews, B. H. (1934). "The interpretation of potential waves in the cortex," *J. Physiol.* **81**(4), 440–471.

Aertsen, A., and Erb, M. (1987). "Scherzo cerebri, ma non troppo: Das neurophon" ("Brain fantasies, but not so much: the Neurophon"), in *Wahrnehmungen Des Gehirns (Perceptions of the Brain)* (Tuebingen Press, Germany), pp. 67–86.

Arnal, P. J., Thorey, V., Debellemaniere, E., Ballard, M. E., Bou Hernandez, A., Guillot, A., Jourde, H., Harris, M., Guillard, M., Van Beers, P., Chennaoui, M., and Sauvet, F. (2020). "The Dreem Headband compared to polysomnography for electroencephalographic signal acquisition and sleep staging," *Sleep* **43**(11), zsaa097.

Buzsaki, G. (2006). *Rhythms of the Brain* (Oxford University, New York).

Destexhe, A., and Foubert, L. (2017). French patent FR-3065366, "Physio-sensory transduction method and device," WO2018197155 (April 25, 2017), owner CNRS.

Destexhe, A., and Foubert, L. (2018). "Composing music from neuronal activity—The Spikiss Project," in *Exploring Transdisciplinarity in Art and Sciences* (Springer, New York), pp. 237–253.

Destexhe, A., and Foubert, L. (2020). U.S. patent 2020-0138322-A, "Physio-sensory transduction method and device" (May 7, 2020), owner CNRS.

Papo, D. (2019). "Neurofeedback: Principles, appraisal, and outstanding issues," *Eur. J. Neurosci.* **49**(11), 1454–1469.

Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis-Peacock, J., Weiskopf, N., Blefari, M. L., Rana, M., Oblak, E., Birbaumer, N., and Sulzer, J. (2017). "Closed-loop brain training: The science of neurofeedback," *Nat. Rev. Neurosci.* **18**(2), 86–100.