Modulation masking patterns for complex-shaped envelopes
Arne OETJEN(1) , Julian SCHILLER(1) , Steven VAN DE PAR(1)(1) Carl-von-Ossietzky University Oldenburg, Acoustics Group, Germany, arne.oetjen@uol.de

Abstract
The concept of modulation ﬁlters is widely applied in in models of auditory perception. This concept was derived from modulation masking experiments using a narrowband masker and a sinusoidal target modulation.
Using other periodic waveforms as a target modulation implies that "overtones" are added to the fundamental
modulation frequency. This target modulation containing overtones will result in changes across the output ofmultiple modulation ﬁlters. In order to investigate this phenomenon listening tests measuring modulation masking for different target modulation waveforms, varying in both the modulation magnitude and phase distributionof the overtones, were carried out. These data can not be reproduced by a model which is simply adding up theoutput from multiple modulation ﬁlters or just considering the modulation ﬁlter with the highest output. Instead,a new model approach using additional onset information for processing the output from multiple modulation
ﬁlters is developed. This approach can reproduce the experimental data qualitatively and quantitively.
Keywords: Auditory system, Modulation ﬁlters, Basic psychoacoustics

1

INTRODUCTION

Acoustic features related to amplitude-modulation perception are used in many ﬁelds such as for example speech
recognition or roughness models. In this context the paradigm of modulation ﬁlters like the ones purposed by
Ewert and Dau [1] is widely used. These ﬁlters were derived from experiments observing the masked detection
thresholds for sinusoidally-shaped modulators. The supra threshold percept of roughness also depends on thepresence of a speciﬁc modulation pattern. It has been shown to strongly depend on the wavefrom of the modulator [2].
These ﬁndings are very important for amplitude-modulation related perceptual attributes that can occur in practical applications since modulation spectra from environmental sounds usually show more than one component.
Often harmonic complexes can be found for example in the modulation spectra for combustion engine sounds.
This behavior can not be modeled with the output from a single modulation ﬁlter since most of the shape information caused by the overtones of the modulator waveform will be lost after processing through a bandpass
ﬁlter. So either the output from multiple modulation ﬁlters tuned to different frequencies has to be consideredor possibly a different approach using periodicity information has to be considered.
Since there is no information of how these complex-shaped modulations are perceived in the auditory systeman experiment using the same modulation-masking paradigm as a previous study with sinusoidal target modulations [1] was carried out using a set of seven different waveforms as the target modulation. These waveforms
have been chosen such that information about the general principles of human auditory processing of amplitude
modulation could possibly be derived from the modulation masking data.

2

PROCEDURE

Thresholds were measured using a three interval adaptive forced choice (3AFC) procedure using a dedicatedMatlab framework [3]. For the measurement of the detection thresholds one interval contained the target modulation and the two other intervals contained only the carrier signal. In the modulation-masking experiment one
interval contained the masker modulation and the target modulation while the two other intervals only contained
the masker modulation.
An one-up two-down rule was used to determine the modulation depth at which 70.7 percent correct responses

5848

can be expected [4]. The starting step size of the modulation depth variation was 4 dB, it was halved after
each upper reversal. During the measurement phase, the step size was 1 dB, 8 reversals were obtained and amean value was calculated from these last 8 reversals. The subjects were seated in a sound-attenuating booth.
They were notiﬁed whether they identiﬁed the target modulation correctly after each trial by displaying ’Richtig’(right) or ’Falsch’ (wrong). The stimuli were presented via Sennheiser HD 650 headphones at a level of 65dB
SPL.
The listener collective consisted of eight, 3 female and 5 male, normal-hearing (self-reported) subjects fromwhich two were authors of this study. The other participants were paid for taking part in the study. The ageof the participants ranged between 22 and 36 years (Ø 27 years). First, the detection thresholds for the sevenmodulation waveforms were measured in a randomized order. Afterwards, the masked detection thresholds were
measured with the adaptive runs for each combination of waveform and masking modulator frequency in a randomized order. This overall procedure was carried out twice from each listener, the average measuring timewas 12 hours for each participant (96 hours in total). The single measurement sessions lasted between one andtwo hours.

3

STIMULI

Each stimulus had a duration of 500ms, including 50ms cosinus windows at the on- and offset, separated bygaps of 500ms. There were seven conditions that consisted of different combinations of overtone spectrum andphase settings which are also illustrated together with their spectra in Figure 1.
S

T

T-2

TJ

Sch

Sch-2

SchJ

1

1

1

1

1

1

0.5

0.5

0.5

0.5

0.5

0.5

0.5

0

0

0

0

0

0

0

-0.5

-0.5

-0.5

-0.5

-0.5

-0.5

-0.5

Amplitude

1

-1

-1
0

0.05

-1
0

0.05

-1
0

0.05

-1
0

0.05

-1
0

0.05

-1
0

0.05

0

0.05

0

500

L (dB)

Time (s)
80

80

80

80

80

80

80

60

60

60

60

60

60

60

40

40

40

40

40

40

40

20

20

20

20

20

20

20

0

0

500

0

0

500

0

0

500

0

0

500

0

0

500

0

0

500

0

Frequency (Hz)Figure 1. Modulation waveforms (upper panels) used in the experiment and the corresponding spectra (lower
panels). For a detailed description of the stimuli see text below.

5849

• S: A 32-Hz sinusoidal modulator• T: A harmonic complex modulator with a fundamental frequency of 32 Hz and 14 harmonics and a 1/f
scaling factor with 0◦ phase building a sawtooth moduluation shape• T-2: Like T, but with the fundamental frequency and the ﬁrst harmonic removed• TJ: Like T, but with a jitter of a factor of 1/4 of the fundamental frequency for each frequency component at maximum. The jitter factors for each component were determined from a random distribution
once and then stored, so the waveform was the same for each stimulus presentation.
• Sch: A harmonic complex modulator with a fundamental frequency of 32 Hz and 14 harmonics with nodecay and a positive Schroeder phase π · n · (n − 1)/N [5], with n being the number of each harmonic
including the fundamental and N = 15
• Sch-2: Like Sch, but with the fundamental frequency and the ﬁrst harmonic removed, N = 13
• SchJ: Like Sch, but with the harmonics jittered similar to TJ, with a maximum displacement of 1/32
The carrier signals were Gaussian noises newly generated for each stimulus interval. In case of masked measurements, the noises were amplitude modulated by half octave wide noises with center frequencies between 4
and 256 Hz. The modulation depth was 0.35 plus a randomized component between 0 and 0.05. Both target
and masker modulation were added together before applying the modulation to the carrier.
The starting modulation depth of the target was set to -10 dB. To prevent overmodulation, the maximal modulation depth was set to -4.5 dB. If the subject failed to detect the target modulation three times at the highest
modulation depth, the track was skipped.

4

RESULTS

4.1 Modulation detection thresholds

RMS corrected modulation depth (dB)

Modulation depth (dB)

-10

-15

-20

-25

S

T

T-2

TJ

Sch Sch-2 SchJ

Modulation waveform

-10

-15

-20

-25

S

T

T-2

TJ

Sch Sch-2 SchJ

Modulation waveform

Figure 2. Left panel: modulation detection thresholds (y-axis) for the seven different modulation waveforms
described in Chapter 3. Right panel: Same as left panel but with compensation for the different root meansquare values with reference to the sinusoidal modulation.

5850

The modulation detection thresholds for the different waveforms are shown in the left panel of Figure 2. One
may argue that the differences in the detection threshold arise from the different root mean square (RMS)
values since the modulation detection thresholds are related to the peak values of the of the target modulator
waveforms. For a comparison the difference in the RMS values with respect to the sinusoidal modulation
waveform was added to these values in the right panel of Figure 2.
This compensation shows that the difference in detection threshold observed for the sinusoidal modulator (S) andthe sawtooth-shaped modulator (T) may be mainly explained by their difference in RMS. The small increasein detection threshold for the sawtooth without the ﬁrst two frequency components (T-2) could possibly be
explained by the modulation ﬁlters limited to about 100 - 120Hz [6]. There seems to be no inﬂuence on
the detection threshold by jittering the overtone frequencies (TJ), also for the Schroeder-phase complexes thejittering inﬂuence could be completely compensated by the RMS compensation (Sch and SchJ). This could be
an argument against a model for modulation perception based on temporal periodicity information. The detection
thresholds for all Schroeder-phase complexes are signiﬁcantly higher than for the other waveforms. This could
possibly also be explained by the higher amount of modulation energy in high modulation frequencies.
4.2 Masked modulation detection thresholds

Modulation depth (dB)

-5
-10

-10

-15

-15

-20

-20

-25

-25
4

-5

Modulation depth (dB)-5S

8 10.716 20.2

64

96 128 256

4

-5

S
Sch
Sch-2

-10

32 50.8

1

1

1

-15

-20

-20

-25

-25
8 10.716 20.2

32 50.8

64

Masker frequency (Hz)96 128 256

8 10.716 20.2

S
TJ
SchJ

-10

-15

4

S
T
T-2

4

8 10.716 20.2

32 50.8642

3

32 50.8

64

96 128 256

96 128 256

Masker frequency (Hz)Figure 3. Masked modulation thresholds (y-axes) for different modulation masker frequencies (x-axes). In theupper left panel the masking pattern for the sinusoidal target modulation (S) is shown, the upper right panel
shows the patterns for the sawtooth-shapes modulator (T) and its pendant without the ﬁrst two frequency components (T-2). The lower left panel shows the pattern for the corresponding Schroeder-phase complexes (Sch andSch-2) and the lower right panel for the waveforms with jittered harmonics (TJ and SchJ). The numbers next
to some datapoints for the Sch-2 and SchJ waveforms show the number of skipped tracks for these conditions.

5851

Although some interesting observations were already made for the detection threshold, masked thresholds may
give a larger insight to processing mechanisms. These data are shown in Figure 3.
The masked thresholds for the sinusoidal target modulation (S) show similar results to those shown by Ewert
and Dau (2000) [1]. This was somehow expected and shows the validity of the experimental approach. Fora further interpretation of these data the difference between the masked threshold and the detection thresholds
without a masker will give a better insight since in these patterns only the effect of the masker modulation on
the detectability of the target modulation waveform can be observed. These patterns are shown in Figure 4.

12

12
S

Masking (dB)

10
8

8

6

6

4

4

2

2

0

0

-2

-2
4

8 10.716 20.2

32 50.8

64

96 128 256

124

8 10.716 20.2

32 50.8

64

96 128 256

12
S
Sch
Sch-2

10

Masking (dB)

S
T
T-2

10

8

S
TJ
SchJ

10
8

6

6

4
1

2

1

1

0

0
-2
8 10.716 20.2

32 50.8

3

2

-2
4

2

4

64

Masker frequency (Hz)96 128 256

4

8 10.716 20.2

32 50.8

64

96 128 256

Masker frequency (Hz)Figure 4. Masking patterns for the different waveforms. The representation is the same as in Figure 3.
For the sawtooth-shaped modulator (T) the pattern of the masked thresholds is similar to the one for the sinusoidal target which surprises a bit because the additional overtones of the modulator do not seem to inﬂuence
the overall pattern. For the sawtooth without the ﬁrst two frequency components (T-2) a different pattern canbe observed. Here, the maximum masking is lower than for the previous conditions, also the maximum occurs
for a masker centered around 64Hz. For the Schroeder-phase complexes Sch and Sch-2, masking is even a
bit lower. Again, the frequency for maximum masking is shifted slightly towards higher frequencies for thetone complex without the ﬁrst two components. Like already observed for the detection thresholds in Figure
2, jittering the harmonics of the tone complexes had nearly no inﬂuence on the masking patterns. Again thisbehavior would be hard to reproduce with an approach using periodicity information like a detector based on
autocorrelations would do.

5852

5

MODEL PREDICTIONS

The data presented in the previous chapter form a suitable basis for testing and developing models for modulation perception. For modulation perception models, in most cases the auditory periphery is not modeled.
This can lead to wrong assumptions about processing mechanisms since stages like auditory ﬁltering, nonlinear
compression or envelope extraction have an inﬂuence on the signal arriving at the modulation ﬁlters.
A model being able to detect different target modulations has to make a "decision" based on the output froma signal containing the target modulation and the model output from a signal without the target. In this casethis was realized by comparing the model output of these two signals. So for the detection thresholds both thecarrier and the modulated carrier were fed into the model. For the masked thresholds a signal containing masker
and target modulation and a signal containing the masker alone were processed. Depending on the difference
between these outputs the model can "detect" a signal. This difference value was the same for all signals andwas set such that if all signals are processed at the exerimentally determined detection thresholds, 50% of thesignals would be detected. The modulation depths of the target modulation were then varied around this point.
An optimal model would create a very steep psychometric function detecting 0% of the modulations for values
slightly below the detection thresholds and 100% of the signals for values slightly above the detection threshold.
The auditory frontend was the same for all approaches to process modulations. First, a 65 dB-SPL iso-loudness
based pre-ﬁltering was applied to the signal. Further processing included a bark-spaced gammatone ﬁlterbank[7], half-wave rectiﬁcation, lowpass ﬁltering and the "venelope" an envelope extraction based on the hilbert envelopes [8].
A white noise low-pass ﬁltered at 1kHz was then added to the envelope signal to simulate internal noise. Thissignal and the internal noise alone were then fed into the modulation ﬁlter stage. Afterwards the power of these
two signal is subtracted to form the output of this stage. The output values across all modulation ﬁlters foreach auditory ﬁlter are then summed up to give the ﬁnal model output.
Results from the detection stage for several different realizations of implementing the modulation ﬁlter stage areshown in Figure 5.

% detected target modulations

100
80

32Hz
32Hz or 96Hz
max
sum
sum and diff

60
40
20
0
-10

-5

0

5

10

Modulation depth relative to threshold (dB)Figure 5. Percentage of detected target modulations from the whole set of 84 stimuli (y-axis) as a function of
the change in modulation depth relative to the detection thresholds shown in Figure 2 and 3. The orange curveshows the detection behavior of a model only using the 32Hz modulation ﬁlter. The violet line shows a modelusing the 32Hz modulation ﬁlter or, for the T-2 and Sch-2 conditions, the 96Hz modulation ﬁlter. The blue line
shows a model using only the modulation ﬁlter with the maximum energy, the green line was obtained froma model using energetic summation over the RMS outputs from each modulation ﬁlter and the red line usedadditional information calculated from the signal derivative after passing through the modulation ﬁlters.

5853

All these realizations were based on the second order bandpass ﬁlters with Q = 1 suggested in a previous study[1]. One alternative way of processing the modulations would be a single modulation ﬁlter tuned to the target’s
modulation frequency. In this study it was not quite clear to which frequency this ﬁlter should be tuned for theconditions without the ﬁrst two fundamentals (T-2 and Sch-2) since the periodicity was at 32 Hz while the ﬁrst
frequency component was at 96Hz. So two approaches were tested, one being tuned to 32Hz for all conditions
and one tuned to 96Hz only for the T-2 and Sch-2 conditions. Also a model only using the modulation ﬁlter
with the highest output was realized. All three models show a rather poor performance for the complete setof stimuli since not all conditions were detected correctly even in a very large range of +-10dB around themeasured detection thresholds.
These observations show that a single modulation ﬁlter would cut off too much information relating to the overtones of the target stimulus. One idea was to overcome this problem by summing up the outputs from multiple
modulation ﬁlters. Instead of a single ﬁlter a bank of 20 modulation ﬁlters logarithmicly spaced between ﬁve
and 150Hz was used for an improved model. The outputs from the ﬁlters were then energeticly summed up.
Using this implementation a much better performance was achieved, 71% of all stimuli were correctly detected
within a range of +-3dB around the detection thresholds.
Another slight improvement was made with an approach to amplify the impact of steep on- and offsets. Thiswas done by taking the absolute value of the differences between samples, adding them with a norm using thepower of four and then multiplying this value with the power of the ﬁlter output. This method mainly improved
the performance for the T-2 condition since here very steep onsets are found and this condition was underestimated by the other methods. With this modiﬁcation now 82% of all stimuli were detected correctly within arange of +-3dB. The model predictions for this approach are shown in Figure 6.

Detection thresholds

Modulation depth (dB)

-10

-15

-20

-25

Modulation depth (dB)

S

S

T

T

T-2

-5

-5

-5

-10

-10

-10

-15

-15

-15

-20

-20

-20

-25

-25

-25

T-2 TJ Sch Sch-2SchJ

4 8

16

32

64

256

4 8

16

32

64

256

4 8

32

64

Masker frequency (Hz)Masker frequency (Hz)Masker frequency (Hz)

TJ

Sch

Sch-2

SchJ

-5

-5

-5

-5

-10

-10

-10

-10

-15

-15

-15

-15

-20

-20

-20

-20

-25

-25

-25

-25

4 8

16

Modulation waveform

16

32

64

Masker frequency (Hz)

256

4 8

16

32

64

Masker frequency (Hz)

256

4 8

16

32

64

Masker frequency (Hz)

256

4 8

16

32

64

256

256

Masker frequency (Hz)Figure 6. Unmasked and masked detection thresholds (open symbols) and model predictions (ﬁlled symbols)for the data shown in Figure 2 (upper left panel) and Figure 3 (all other panels)

5854

6

SUMMARY

Modulation masking patterns were measured for target modulations with various different modulation waveforms. These patterns differ a lot depending on the waveform. From these data and trying different model
approaches one may conclude that the outputs from different modulation ﬁlters have to be combined to predictthe detectability for all of these conditions. Furthermore from the observation that jittering the overtones of amodulation waveform does not signiﬁcantly change the masking patterns it can be concluded that modulation
processing is not based on autocorrelation-like temporal features.
A model being able to detect a large variety of stimuli correctly within a small range of modulation-depth
variations was implemented based on these observations. This model is based on combining the outputs from a
modulation ﬁlterbank and using additional on- and offset features from the absolute value of the signal derivative. However, the model over- or underestimates the masker inﬂuence for some stimuli and therefore further
improvements to the model could be made. Despite these small shortcomes it may deliver a hint towards
mechanisms in the auditory system involved in the processing of amplitude modulations.

REFERENCES[1] Ewert, S.D. and Dau, T. Characterizing frequency selectivity for envelope ﬂuctuations, J. Acoust. Soc. Am.
108, 1181–1196 (2000).
[2] Yasui, N. and Miura, M. Perception of roughness on sounds amplitude-modulated with triangular wave,
Forum Acusticum (2011).
[3] Ewert, S.D. AFC - A modular framework for running psychoacoustic experiments and computational perception models, Proceedings of the International Conference on Acoustics AIA-DAGA 2013, Merano, Italy,
pp. 1326-1329. (2013)[4] Levitt, H. Transformed up–down procedures in psychoacoustics, J. Acoust. Soc. Am. 49, 467–477. (1971)[5] Schroeder, M.R. Synthesis of low-peak-factor signals and binary sequences with low autocorrelation (Corresp.). IEEE Trans. Information Theory 16(1): 85-89 (1970)[6] Moore, B.C.J., Füllgrabe, C., and Sek, A. Estimation of the center frequency of the highest modulation
ﬁlter, J. Acoust. Soc. Am. 125, 1075–1081. (2008)[7] Hohmann, V. Frequency analysis and synthesis using a Gammatone ﬁlterbank. Acustica / acta acustica,
88(3), p. 433-442. (2002)[8] Verhey, J.L., Ewert, S.D., and Dau, T. Modulation masking produced by complex tone modulators, J.
Acoust. Soc. Am. 114, 2135–2146. (2003)

5855