# Using reverse correlation to study individual perception: Including an auditory model in the experimental design loop

Alejandro Osses, Léo Varnet

► To cite this version:

Alejandro Osses, Léo Varnet. Using reverse correlation to study individual perception: Including an auditory model in the experimental design loop. ARCHES meeting, Nov 2022, Oldenburg, Germany. , 2022. hal-03866646

HAL Id: hal-03866646

https://hal.science/hal-03866646

Submitted on 22 Nov 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Using reverse correlation to study individual perception:
Including an auditory model in the experimental design loop

Alejandro Osses & Léo Varnet

Laboratoire des systèmes perceptifs, ENS Paris, PSL University, France

ARCHES Oldenburg, 21-22 November 2022

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

1/19

## Outline

<figure><img src="image_16.png" /></figure>

### 1 Introduction

<figure><img src="image_17.png" /></figure>

### 2 Algorithm for noise generation

<figure><img src="image_18.png" /></figure>

### 3 Results

<figure><img src="image_19.png" /></figure>

### 4 Discussion & conclusions

<figure><img src="image_20.png" /></figure>

### 5 (Supplementary)

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

1/19
# Phoneme discrimination using the ACI method

* N trials needed

* N is too high (N > 4000)

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

# Phoneme discrimination using the ACI method: Optimisation?

<figure><img src="image_36.png" /></figure>

* Other speech samples

* Other noises

* Another T-F transformation

* Another statistical fitting scheme

* Reviewed and unified for all our previous studies

[Osses et al. (2022, ICA)]

<figure><img src="image_37.png" /></figure>
# Phoneme discrimination using the ACI method: Optimisation?

<figure><img src="image_38.png" /></figure>

* Other speech samples

* Other noises

* Another T-F transformation

* Another statistical fitting scheme

* Previous studies: White noise

* We wanted other noises

<figure><img src="image_39.png" /></figure>

## Outline

1. Introduction

2. Algorithm for noise generation

3. Results

4. Discussion & conclusions

5. (Supplementary)

<figure><img src="image_40.png" /></figure>
# Rationale: speeding up the ACI assessment

<figure><img src="image_41.png" /><figcaption>Fig. 2</figcaption></figure>

<figure><img src="image_42.png" /></figure>

* In the end: 30 bumps for each noise

* Random locations along time and frequency

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

# Rationale: Hypotheses

Using bump noises:

* H1: We will be able to obtain clean ACIs: pilots with one participant

* H2: Different sets of noises will lead to the same ACIs: to be checked by the model

Experimental design:

* Baseline: White noises

* Data collection from 12 participants (4000 trials ≈ 4h sessions x noise type x participant)

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>
# Outline

1. Introduction

2. Algorithm for noise generation

3. Results

4. Discussion & conclusions

5. (Supplementary)

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

# H1 – Experiments

* H1: Yes, we were able to obtain clean ACIs, but...

<figure><img src="image_50.png" /><figcaption>Fig. 3</figcaption></figure>

Maybe: Individual differences related to the specific set of bump noises (H2)?

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>
# H2 – ACIs from simulations: The auditory model

<figure><img src="image_53.png" /></figure>

* Perceptual model based on the modulation-filter-bank model by Dau *et al.* (1997)

* Updated parameters, updated modulation filter bank

* Two template-approach as detailed by *Osses & Kohlrausch (2021)*, here:

* Template 1: /aba/; Template 2: /ada/. Obtained from 1000 noisy samples

* Supra-threshold level: SNR= -6 dB

<figure><img src="image_54.png" /></figure>

# H2 – ACIs from simulations

* H2: Yes, the ACIs were independent from the set of bump noises

<figure><img src="image_55.png" /><figcaption>Fig. 4</figcaption></figure>

<figure><img src="image_56.png" /></figure>
Comparing ACIs from experiments/simulations (1 of 2)

<figure><img src="image_57.png" /><figcaption>Fig. 4a</figcaption></figure>

<figure><img src="image_58.png" /></figure>

Comparing ACIs from experiments/simulations (1 of 2)

<figure><img src="image_59.png" /><figcaption>Fig. 4a</figcaption></figure>

<figure><img src="image_60.png" /></figure>
Comparing ACIs from experiments/simulations (2 of 2)

<figure><img src="image_61.png" /><figcaption>Fig. 4b</figcaption></figure>

This means that all the cues in the experimental ACIs are a subgroup of the cues from ACIsim

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

13/19

Comparing ACIs from experiments/simulations (2 of 2)

<figure><img src="image_78.png" /><figcaption>Fig. 4b</figcaption></figure>

This means that all the cues in the experimental ACIs are a subgroup of the cues from ACIsim

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

13/19
## Outline

1. Introduction

2. Algorithm for noise generation

3. Results

4. Discussion & conclusions

5. (Supplementary)

<figure><img src="image_94.png" /></figure>

## Using the auditory model: possibilities and limitations

* Template matching: based on the idea of “optimal detection” (Green & Swets, 1966)

* A very simple top-down decision back-end

* The auditory model as shown here: simulates a normal-hearing listener

<figure><img src="image_95.png" /></figure>
Using the auditory model: possibilities and limitations

* Template matching: based on the idea of “optimal detection” (Green & Swets, 1966)

* A very simple top-down decision back-end

* The auditory model as shown here: simulates a normal-hearing listener

* Are these results related to the specific auditory model (front end)?

<figure><img src="image_96.png" /></figure>

AMT models using the two-template matching (Figure taken from *Osses et al. 2022* (AABBA))

Using the auditory model: possibilities and limitations

* Template matching: based on the idea of “optimal detection” (Green & Swets, 1966)

* A very simple top-down decision back-end

* The auditory model as shown here: simulates a normal-hearing listener

* Are these results related to the specific decision (back end)?

<figure><img src="image_97.png" /></figure>

**Figure 3:** (Colour online) ACIs for the artificial listeners, i.e., the auditory model using the decision device 1 with $K=0.39$ MU (left panel) and using the decision device 2 (right panel). The colour codes and axis legends are as in Fig. 1.

osses2021.m model using cross correlations (left) or
two-dim Pearson correlations (right)
[Figure taken *Osses & Varnet (2021, DAGA)*]
# Using the auditory model: possibilities and limitations

* Individualisation of the model?

* This model was developed to simulate psychoacoustic tasks

* Hearing impairment can be introduced, but it is not straightforward:

<figure><img src="image_98.png" /></figure>

* We cannot introduce an audiogram as in other models (e.g., from AMT [Majdak et al. (2022)]: zilany2014, bruce2018)

<figure><img src="image_99.png" /></figure>

* Impairment can be introduced by this requires a careful understanding of the model stages and their expected effect (Holube & Kollmeier (1996))

<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

# Using the auditory model: possibilities and limitations

* Individualisation of the model?

* **HI model:** Model with wider auditory filters and modified time constants in the adaptation loops (inspired by Holube1996)

<figure><img src="image_104.png" /><figcaption>ACI using HI simulations (noises from S04)</figcaption></figure>

<figure><img src="image_105.png" /><figcaption>But, big advantage: Int. representations at fc=1250 Hz</figcaption></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>
# Conclusions

## ACI method

* Assessment of individual listening strategies

* Interesting tool to obtain T-F cues for both models and participants

## Use of auditory models (my main focus today)

* “Optimal integration of cues”

* Can be used to support/reject hypotheses

* Careful: T-F cues may not reflect the actual (but candidate) cues

* Careful: There might be results that are model dependent

> I still believe that much can be learned from auditory models and the estimated internal representations

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

> Thank you for your attention

* Questions or comments?

<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>

<figure><img src="image_133.png" /></figure>

<figure><img src="image_134.png" /></figure>

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>
References

Dau, T., Kollmeier, B., & Kohlrausch, A. (1997). Modeling auditory processing of amplitude modulation. I. Detection and masking with narrow-band carriers. *J. Acoust. Soc. Am.*, **102**, 2892–2905. https://doi.org/10.1121/1.420344

Holube, I., & Kollmeier, B. (1996). Speech intelligibility prediction in hearing-impaired listeners based on a psychoacoustically motivated perception model. *J. Acoust. Soc. Am.*, **100**, 1703–1716. https://doi.org/10.1121/1.417354

Majdak, P., Hollomey, C., & Baumgartner, R. (2022). AMT 1.x: A toolbox for reproducible research in auditory modeling. *Acta Acustica*, **6**, 19. https://doi.org/10.1051/aacus/2022011

Osses, A., & Varnet, L. (2021). Consonant-in-noise discrimination using an auditory model with different speech-based decision devices. *DAGA*, 298–301. https://hal.archives-ouvertes.fr/hal-03345050

Osses, A., & Kohlrausch, A. (2021). Perceptual similarity between piano notes: Simulations with a template-based perception model. *J. Acoust. Soc. Am.*, **149**, 3534–3552. https://doi.org/10.1121/10.0004818

Osses, A., Lorenzi, C., & Varnet, L. (2022). Assessment of individual listening strategies in amplitude-modulation detection and phoneme categorisation tasks. *International Congress on Acoustics*. Gyeongju, South Korea (Presentation: [click here](#))

Osses, A., & Varnet, L. (2022). *fastACI toolbox: the MATLAB toolbox for investigating auditory perception using reverse correlation (v1.2)*. https://doi.org/10.5281/zenodo.7314014

Osses, A., Varnet, L., & Lorenzi, C. (2022). Simulating the perception of soundscapes, speech-, AM- and FM- sounds. *AABBA*. Vienna, Austria. https://hal.archives-ouvertes.fr/hal-03554232