IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999

323

Improved Phase Vocoder
Time-Scale Modification of Audio
Jean Laroche, Member, IEEE, and Mark Dolson, Member, IEEE

Abstract—The phase vocoder is a well-established tool for timescaling and pitch shifting speech and audio signals via modification of their short-time Fourier transforms (STFT’s). In contrast
to time-domain time-scaling and pitch-shifting techniques, thephase vocoder is generally considered to yield high quality results,
especially for large modification factors and/or polyphonic signals. However, the phase vocoder is also known for introducing acharacteristic perceptual artifact, often described as “phasiness,”“reverberation,” or “loss of presence.” This paper examines the
problem of phasiness in the context of time-scale modificationand provides new insights into its causes. Two extensions tothe standard phase vocoder algorithm are introduced, and theresulting sound quality is shown to be significantly improved.
Moreover, the modified phase vocoder is shown to provide afactor-of-two decrease in computational cost.
Index Terms—Phase coherence, phase vocoder, pitch shifting,short time Fourier transform, time scaling.

I. INTRODUCTION

T

IME-SCALE and pitch-scale modification of signals haslong been a subject of interest in the audio and speechprocessing community. In recent years, though, there has beena dramatic increase in commercial application of these techniques. Time-scaling and/or pitch-shifting algorithms are now
being used in a widening array of devices such as telephoneanswering systems, musical effect processors, professional CD
players, hard-disk recorders, PC-based sound editors, and soon. As the computational resources of these devices increase,so too do expectations for their audio fidelity.
Most commercial implementations of time scaling (or pitch
shifting)1 use time-domain-based techniques which rely uponsome form of synchronized overlap-add of signal excerpts.
These methods are attractive for their relatively low computational cost and because they yield good results in somespecial cases of interest (e.g., modification factors close to
one or monophonic sounds). However, these techniques tend
to perform poorly when applied to complex, polyphonic, or
nonpitched signals, or when large modification factors must
be used (e.g., factors greater than 20% to 30%). In thesecases, typical artifacts include warbling (a type of periodic frequency modulation observed in processed polyphonic signals),transient doubling or skipping (especially troublesome for
Manuscript received June 9, 1997; revised May 14, 1998. The associate
editor coordinating the review of this manuscript and approving it for
publication was Dr. Juergen H. Herre.
The authors are with Joint E-mu/Creative Technology Center, Scotts Valley,
CA 95067 USA (e-mail: jeanl@emu.com markd@emu.com).
Publisher Item Identifier S 1063-6676(99)02745-5.
1 Since pitch-scale modification can be performed by combining time scaling
and sample-rate conversion, we focus in this paper exclusively on time scaling.

percussive signals), and tempo modulation. A full discussionof time-domain time-scaling techniques and their shortcomings
can be found in [7] or [5].
In contrast, frequency-domain-based time-scaling techniques such as the phase vocoder [1] employ a fixed
overlap-add approach; synchronization between overlapping
frames is obtained by modifying phases in the signal’s shorttime Fourier transform. Phase vocoder time-scaling is not
limited to near-unity modification factors nor to monophonicsignals, and it is free of many of the typical time-domain
artifacts.2 This makes it a potentially attractive approach.
However, the computational cost of the phase vocoder is muchhigher than that of time-domain techniques and the algorithm
introduces distinctive artifacts of its own. Ultimately, it isthese artifacts that pose the major barrier to more widespreaduse of the phase vocoder.
The two most prominent phase vocoder time-scaling artifacts are “transient smearing” and “phasiness.” Transient
smearing occurs even with modification factors that are close
to one, and is heard as a slight loss of percussiveness in thesignal; piano attacks, for example, may be perceived as having
less “bite.” Phasiness (or reverberation or “loss of presence”)
also occurs even with near-unity modification factors, and isheard as a characteristic coloration of the signal; in particular,
time-expanded speech often sounds as if the speaker is muchfurther from the microphone than in the original recording.
In general, neither time-domain nor frequency-domain timescaling artifacts have received much attention in the technicalliterature, probably because assessments of fidelity have varied
according to local standards (and over time as well). The
problem of transient smearing in subband-based time-scale
modifications is addressed in [11] and an improved techniquebased on phase-locking at transient times is proposed. The
phenomenon of phasiness has been noted by several authors[10], [14], and the root of the problem is known to lie inthe modification of phases in the short-time Fourier transform(STFT). To date, however, no thorough explanation for thisphenomenon has yet been given, and proposed solutions have
proven to be either cumbersome or only marginally effective.
This paper proposes an explanation for the presence ofphasiness in time-scaled signals, and offers new phase calculation techniques that are shown to significantly reduce theproblem. In addition, these new techniques make it possible toreduce the computational cost of the phase vocoder by morethan a factor of two.
2 For example, tempo modulation is virtually nonexistent, and warbling canbe eliminated by an appropriate choice of parameters.

1063–6676/99$10.00  1999 IEEE

324

IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999

The remainder of this paper is divided into two sections. In
the first part, the emphasis is on understanding the problem:the standard phase-vocoder technique for time scaling is
described, and a detailed investigation of potential phase errors
is presented. In the second part, the focus is on solutions:two previously-proposed solutions are briefly reviewed, andtwo new phase-modification techniques are introduced andevaluated.
II. THE BASIC PHASE VOCODER TIME-SCALING ALGORITHM
The essence of time scaling is the modification of a signal’s
temporal evolution while its local spectral characteristics are
kept unchanged. Phase-vocoder-based time-scaling techniques
accomplish this via an explicit sequence of analysis, modification, and resynthesis.
A. Phase Vocoder Analysis/SynthesisforDuring the analysis stage, analysis time-instants
successive values of integer are set along the original signal,
where
is the so-called
possibly uniformly:
analysis hop factor. At each of these analysis time-instants,
a Fourier transform is calculated over a windowed portionof the original signal, centered around . The result is thenonheterodyned STFT representation of the signal, denoted:(1)
is the original signal,is the analysis window,
is the center frequency of the th vocoder “chanis the size of the discrete Fourier transform. In
nel,” andhas a limited time span (typicallysamples)practice,isand the sum above has a finite number of terms.
both a function of time (via variable ) and frequency (via ).
The resynthesis stage involves setting synthesis time, where
instants , usually uniformly, so thatis the synthesis hop factor. At each of these synthesis
is obtained by inversetime-instants, a short-time signal. EachFourier-transforming the synthesis STFT
short-time signal is then multiplied by an optional synthesis, and the windowed short-time signals are allwindow:summed together, yielding the output signal

strong consistency conditions because the Fourier transforms
correspond to overlapping short-time signals. The formula
above merely yields a signal whose STFT is close to
in a sense that depends on the choice of the synthesis window. Further elaboration of this point can be found in [4], [7].
B. Time-Scale Modifications
In phase-vocoder-based time scaling, the STFT is modified
is different fromin two ways: 1) the analysis hop factor, and 2) the phase values of thethe synthesis hop factor
are calculated explicitly according
synthesis STFT
to a formula given below. These modifications are based on anunderlying sinusoidal signal model, but no explicit parametricsinusoidal estimation is performed.
According to the underlying model, the input signal isof sinusoids with time-varying
the sum of a number
and instantaneous frequencies
amplitudes
with
(3)

and
are called the instantaneous phasein whichand frequency of the th sinusoid.
Based on (3), for a constant modification factor such that, the ideal synthesis phaseof the time-scaled
sinusoid would be

where

with
(2)

and
In the absence of modifications (i.e.,), this output signal is identical tothe original signal , under mild conditions on the analysis
and synthesis windows [4]. In general, however, a modified
is not the STFT of any actual signal. In particular,
obtained via the above reconstruction
the output signal
as its STFT.
formula does not necessarily have
The sequence of STFT frames for a given signal must satisfy(4)
is an arbitrary initial synthesis phase.
where
Phase-vocoder-based time scaling modifies the STFT ofthe sinusoidal input signal components so as to produce theabove time-scaled sinusoids. The time-evolution of the sinewave amplitudes is modified simply by setting
where. However, modification of thesine-wave phases is more challenging.
, the standard phaseTo calculate the phase of
vocoder technique requires phase unwrapping, a process
whereby the phase increment between two consecutive
frames is used to estimate the instantaneous frequency of anearby sinusoid in each channel. The instantaneous frequency
is estimated by first calculating the heterodyned phaseincrement

then taking its principal determination (between
and deriving the instantaneous frequency
closest sinusoid using) denotedof the(5)

LAROCHE AND DOLSON: TIME-SCALE MODIFICATION OF AUDIO

325

This procedure is called phase unwrapping, because the actual(nonwrapped) value of the phase increment is calculated fromits principal (wrapped) determination. The heterodyned phase
is simply the small phase shift resulting fromincrement
being close but not necessarily equal to.
is estimated,
Once the instantaneous frequency at timeis set according
the phase of the time-scaled STFT at timeto the following phase-propagation formula(6)
Equation (6) guarantees what can be called “horizontal phasecoherence”: for a constant-frequency sinusoid, successive
short-time signals will overlap coherently. Another way ofsaying this is that there is coherence within each frequency
channel over time (i.e., along the horizontal dimension of astandard sonagram).
For constant-frequency sinusoids, the phase unwrapping(5) yields a good estimate of the instantaneous frequency
if channel is influenced by only one sinusoid, and if theis such that.
analysis window’s cutoff frequency
In practice, for standard analysis windows (such as Hanningor Hamming windows), this constrains the analysis windows
to overlap by at least 75%. The phase unwrapping equation
involves the calculation of a four-quadrant arc tangent, and thephase propagation equation (6) requires the use of trigonometric functions in order to calculate the real and imaginary part
. An important point is that (6) does not indicateof
how the short-time Fourier phases should be set at the first
synthesis time-instant . As will be shown later, the choiceof initial phases can significantly influence the quality of theoutput signal.
The technique outlined above is the standard frequencydomain time-scaling method based on the phase vocoder.
Variants of this approach have also been proposed. Portnoffin [9] describes a technique applicable to speech signals,
where the phases of the underlying sinusoidal components
are decomposed into a system phase (resulting from thevocal tract filtering) and a source phase corresponding toglottal pulses. Interested readers can also refer to [10] for analternative phase-updating method that does not involve phaseunwrapping nor any trigonometric calculation, but instead uses
an additional analysis Fourier transform.
C. Phase Problems in Phase-Vocoder Time-Scaling
1) Phase Coherence: Because phase propagation errors
are at the heart of many of the sound quality issues in thephase vocoder, it is important to understand how sinusoidal
phases are altered by vocoder-based time-scale modifications.
The phase-vocoder time-scaling algorithm ensures phase consistency within each frequency channel over time, but it is also
important to have phase consistency across the channels in agiven synthesis frame. We call this latter requirement verticalphase coherence.
Both horizontal and vertical phase coherence must be preserved upon resynthesis for the STFT to be a “valid” one.
If phase coherence is not preserved in the synthesis STFT
, the synthesis equation (2) will yield a signal whose

STFT is not close to. This new signal will likely
exhibit beating in its individual harmonics, and this will beheard as phasiness or reverberation.
How can we recognize vertical phase coherence in a STFT?
For a constant-amplitude, constant-frequency sinusoid, there
is a simple phase relation between adjacent channels locatedaround the sinusoidal frequency. If a sinusoid with a constantfalls in channel , and if the analysis window
frequency
is symmetric around zero, it is easy to show thatthe channels around channel which are influenced by thiswhereis thesinusoid (channels such that
cutoff frequency of the analysis window) have an analysis
phase equal to that of channel . In practice, the analysis
is more usually nonzero forandwindow
is symmetric around its middle point, but this changes things
only slightly: If the size of the discrete Fourier transform isequal to the analysis window length , then adjacent channels.
exhibit phase differences ofFor more complicated signals, unfortunately, no comparably simple phase relationship exists. For a sinusoid with aslowly varying frequency, the phases in channels around theinstantaneous frequency are still nearly equal, but an analyticalformula is difficult to develop. Consequently, there is no
simple way to check for vertical phase coherence.
An a posteriori way to check the consistency of a STFT
, andconsists of reconstructing the synthesis signals
is indeed very close tochecking that the STFT ofin both amplitude and phase. We propose a measureof consistency derived from that proposed in [4]:(7)
is obtained by performing a STFT on thewhere.
is the total number of short-time
modified signal
first andlast
frames, and the summation excludes the
few frames to avoid taking into account errors due to missing
overlapped segments in the resynthesis formula. The smaller, the better the consistency. If total consistency is achieved,
for all times
and all channels
and. Vertical and horizontal phase coherence willplay an important role in the following sections, and we will
use the measure above to estimate the degree of consistencyof our algorithms.
2) Output Phase versus Input Phase: In this section, we
seek to relate the phase of the modified STFT in channel
to the phase of the corresponding analysis STFT in the same
, andchannel. Assuming a constant modification factor, wegiven an initial synthesis Fourier transform phasecan use the phase propagation equation (6) to express the phaseof the output STFT at any given synthesis time-instant . By
, we
iterating (6) for successive values of , starting at
obtain326

IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999

whereis the estimated instantaneous frequency at timein channel . Now, using (5), we get(8), we can express the synthesis phase of the peak channel
at timeaswithand using the definition of

where
:

we getis the unwrapping factor at the analysis time-instant. This yields(8)Equation (8) gives the expression of the phase of the synthesis
as a function of the synthesis initial phase
STFT at time
, the phase of the analysis STFT at time , the initial
analysis phase, the modification factor , and the series of.
phase-unwrapping integers
Several conclusions can be drawn from this equation.
• Equation (8) indicates that the synthesized phase depends
on the analysis phases only at the current analysis timeinstant and at the origin. This means that if an analysis
phase is estimated incorrectly at any given time-instant,this error will not generate phase drift in subsequent
frames, provided that the phase-unwrapping factor
remains correct.
• On the other hand, the series of phase-unwrapping factors
do have a cumulative effect: if an erroneous phaseunwrapping factor is calculated at a given frame, allsubsequent frames will show a phase bias.
• Potential phase-unwrapping errors manifest themselves
being added to the synthesis phase.
by multiples ofis an integer, then phase-unwrapping errors are
If. Astransparent since they always are multiples ofa result, integer-factor time-scaling operations can be
performed without phase unwrapping by use of (8) whereis dropped. Skipping the phasethe factor
unwrapping stage significantly reduces the computation
cost of such modifications.
Equation (8) also provides a solid analytical foundation forunderstanding the lack of vertical phase coherence in standard
phase vocoder implementations. This issue is examined indetail in the next two sections.
3) Loss of Vertical Phase Coherence: According to (8),vertical phase coherence depends upon two factors: 1) initial
phase values, and 2) accumulated phase-unwrapping errors.
To see this, suppose at first that the modification factor
is a constant integer, so phase-unwrapping errors do not
influence the modified signal. Now, consider a sinusoid whose
instantaneous frequency varies across time so that it will
migrate from channel to channel. Rearranging the terms in(9)where the sum of unwrapping factors has been dropped,
being a multiple of . This expression differs from the ideal
is not necessarily asynthesis phase equation (4) in that
constant, but varies with the channel index . The fact that
may not be constant has two adverse and related consequences:
1) The synthesis phases in adjacent channels may be very
vary significantly fromdifferent, if the values ofchannel to channel. As mentioned in II-B1, this should
not be the case.
2) When the sinusoid’s instantaneous frequency migrates
at timeto channel
at timefrom channel, the synthesis phase undergoes a jump equal to, which is also very undesirable.
for successive channels
As shown by (9) the values ofdepend only on the analysis and the synthesis phases at time
zero. If for example an area of the spectrum was dominated byand consequently
noise at that time, the values offor the corresponding channels will be random, andofare likely to exhibit large variations from channel to channel,
such thatunless we set the initial synthesis phase(10)
is a channel-independent constant. If the above
where
equation is satisfied, then none of the two problems mentioned
above occur, and the synthesis phase becomes identical to theideal synthesis phase in (4).
Now, consider the more general case in which the modification factor is not an integer, and again suppose that there is asinusoidal component in the vicinity of channel . Equation(8) indicates that, even if (10) is satisfied, phase coherenceis guaranteed only if the sums of the unwrapping factors
are equal (modulo ) in nearby channels. Thereis no danger of phase-unwrapping errors in channels near thesinusoid’s instantaneous frequency so long as the analysis hop
is small enough. The deeper problem, though, isfactorthat no single sinusoidal component of an audio signal islikely to persist without interruption across the entire durationof the signal. Thus, there will inevitably be times during
which channel and its neighbors are influenced by unrelated
sinusoids or even noise. It is during these times that phaseunwrapping differences will necessarily accumulate, making
different in adjacent channels; as a
the terms
result, vertical phase coherence will quickly be lost forever.
Considering the universality of the above scenario, it seems
truly amazing that the phase vocoder should work at all!
For noninteger modification factors, vertical phase coherenceis almost guaranteed to be lacking unless each sinusoidal
component remains in the same phase vocoder channel for
all time. Clearly, this assumption is violated by most signalsof interest, including speech and music.

LAROCHE AND DOLSON: TIME-SCALE MODIFICATION OF AUDIO

Fig. 1. Factor-two time-scaling of a constant amplitude chirp. Time-domain
amplitude-envelope of the modified signal.

4) Examples: In this section, we present a few timescale modification examples that demonstrate the problems
described above. We begin by noting that constant-frequency,
constant-amplitude sinusoids pose no problem at all to thephase vocoder; the results are usually excellent, with consis60 dB, provided the initial synthesis
tency measuresphases are all set according to (10). Variable-amplitude sinusoids pose some problems, but the most dramatic illustrations
are obtained with chirp signals.
time-expansion
The first example is a factor-of-twoof a sinusoid of constant amplitude whose frequency sweeps
linearly from the center frequency of channel 30
to the center frequency of channel 40 in 80 analysis STFT
frames. This signal was analyzed with the standard phasevocoder technique described above. The fast Fourier transform(FFT) size was set to 1024, the analysis hop size was 128, thesynthesis hop size was 256, and both analysis and synthesis
used Hanning windows of size 1024. Instead of conformingto (10), however, the initial synthesis phases were set equal
to the initial analysis phases:

This is a standard initialization choice because it makes it
to apossible to switch from a nonmodified signal
modified signal without introducing any phase discontinuity.
Fig. 1 shows the amplitude envelope of the resulting signal
in the time-domain (obtained by plotting the maximum andminimum values of the sinusoid for each period). The clearlyvisible amplitude modulation is due to (10) not being satisfied.
Fig. 2 shows the analysis and synthesis phases for successive
short-time Fourier transform frames. The figure was obtainedby measuring the phases at the maximum of the analysis
or synthesis
Fourier transforms in eachframe, then unwrapping them along successive frames. The
analysis phase shows the characteristic parabolic shape due tothe linearly varying frequency. The synthesis phase roughly
follows this parabolic shape, but exhibits “discontinuities” atframes 38, 53, and (to a lesser extent) 22. These phase jumps

327

Fig. 2. Analysis phase (solid line) and synthesis phase (dotted line) in
numbers of 2 , showing phase jumps at frames 22, 38, and 53.

Fig. 3. Factor-two time-scaling of a constant amplitude chirp, “correct”initial
phases. Time-domain amplitude-envelope of the modified signal.

result fromnot being a constant in (10); it can be easily
verified that they occur when the instantaneous frequency ofthe chirp jumps from one channel to the next. The consistency
10 dB.
measure for this resynthesis was
are set toIf the initial synthesis phases(11), then the time-scaling operation yields the signal
withshown in Fig. 3. The resulting sinusoid shows only marginal
amplitude modulation, and the synthesis phase is free offor integer
jumps. This confirms the fact that setting
modification factors provides a significant improvement inphase coherence. The consistency in this case was measured
25 dB, far better than the preceding result.
to be
When the modification factor is no longer an integer, theresulting signal exhibits phase coherence problems even if theinitial phases are set according to (11). This can be seen in
Fig. 4. The same chirp signal as above is time-scaled by a

328IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999

Algorithms for STFT magnitude-only reconstruction were
presented in [4] and [8]. Unfortunately, these algorithms require numerous iterations of the STFT analysis-synthesis cycle
in order to arrive at an internally consistent STFT. This makes
them far too computationally demanding for contemporary
real-time applications. In addition, while convergence has been
proven for some of these methods, there is no guarantee thata global minimum will be reached. More recently, various
authors have noted that the iterative process can be greatly
accelerated by calculating good sets of initial STFT phasevalues [13]. However, this procedure is still considerably morecomputation-intensive than the phase vocoder.
The other frequency-domain alternative to the phase
vocoder is so-called sinusoidal modeling. This approach is also
more computationally demanding than the phase vocoder, andit introduces its own perceptual artifacts which are beyond
the scope of this paper.
Fig. 4. Factor-1.4 time-scaling of a constant amplitude chirp, initial phases
set according to (11). Time-domain amplitude-envelope of the modified signal.

factor, with the phase initialization of (11). The
resulting signal is severely modulated in amplitude, a resultof phase-unwrapping errors in channels distant from channel
30. These errors caused erroneous unwrapping factors to be
added to the synthesis phase when the sinusoid’s instantaneous
frequency reached those channels. The consistency here was
6.5 dB.
measured as
Taken together, these very simple test signals demonstrate
that, for integer modification factors, initial phases should be
set according to (11) to avoid phase jumps. Moreover (andregardless of how the synthesis phases are initialized), timescale operations with noninteger modification factors are likely
to introduce phase discontinuities, leading to significant phaseincoherence in the synthesis STFT.
Informal listening tests on speech and music signals confirm
these findings. For factor-of-two modifications, high-quality
results are obtained when the synthesis phases are initializedaccording to (11), while phasiness occurs for any other typeof initialization. Noninteger modifications always sound phasy,no matter what phase initialization is used.
In addition, time-scaling operations with integer modification factors greater than or equal to three exhibit an increasinglevel of phasiness even with proper phase initialization. The
explanation for this seems to lie in the fact that, for such large
modification factors, satisfying (10) still does not guarantee
consistency for the synthesis STFT as a whole. Equation (10)ensures vertical phase coherence, but the modified sequenceof STFT magnitudes may be inconsistent.

B. Loose Phase Locking
A suboptimal but less computation-intensive solution to thephase vocoder’s phase-unwrapping errors is simply to apply
a-posteriori constraints to the synthesis phases. An efficientmechanism for accomplishing this was introduced by Puckette [10]. Recognizing that a constant-amplitude, constantfrequency sinusoid in channel should have identical analysis
phases in all nearby channels, Puckette proposed a simple way
to loosely constrain synthesis phases to obey the same rule.
For each channel in the synthesis STFT, the synthesis phase iscalculated by use of the standard phase-propagation formula,(6). However, the final synthesis phase attributed to channel
is that of the complex number

As a result, if channel
is the maximum of the Fouriertransform magnitude, its phase is basically unchanged, because
andare of much lower amplitude.
But if channel is left of the maximum, its phase will bewhose magnitude is larger thanroughly that of: the channels around a peak in the Fourier transformare “phase locked” to the peak.
This technique is attractive, especially because it requires
only a few additional multiplications per channel. It produces a visible improvement in the phase vocoder output
for simple test signals and a measurable improvement in
STFT consistency. However, informal listening tests showthat the reduction in phasiness is very signal-dependent and,
unfortunately, never dramatic.
C. Rigid Phase Locking

III. OLD AND NEW STRATEGIES FOR REDUCING PHASINESS
A. Magnitude-Only Reconstruction
The phase vocoder is not the only frequency-domain technique that can be applied to the problem of time scaling. Phase
unwrapping issues can be avoided entirely by 1) reconstructing
purely from the STFT magnitude, or 2) fitting the STFT to an
explicit sum-of-sinusoids model.

The fundamental limitation (and also the attraction) of theloose phase locking scheme is that it avoids any explicit
determination of the signal structure: the same calculation is
performed in every channel, independently of its content. The
result is that synthesis phases in the channels around a given
sinusoid only gradually and approximately develop verticalphase coherence. If we really want to restore vertical phasecoherence to the synthesis STFT, we need to take a step closer

LAROCHE AND DOLSON: TIME-SCALE MODIFICATION OF AUDIO

329

to an actual sum-of-sinusoids model. We now present two versions of a new phase-locking technique, inspired by the loose
phase locking above, but based on the explicit identificationof peaks (and thus, presumably, sinusoids) in the spectrum.
The new phase-updating technique begins with a coarse
peak-picking stage where vocoder channels are searched for
local maxima. In the simplest implementation, a channel
whose amplitude is larger than its four nearest neighbors is said
to be a peak; this criterion is both simple and cost-effective(although admittedly primitive). The series of peaks subdivides
the frequency axis into “regions of influence” located aroundeach peak. The basic idea is to update the phases for the peakchannels only according to the standard phase-propagation (6);the phases of the remaining channels within each region arethen “locked” in some way to the phase of the peak channel.
In our experiments, the upper limit of the region around peak
was set to the middle frequency between that peak and the. Another reasonable choice would
next onebe the channel of lowest amplitude between the two peaks.
1) Identity Phase Locking Rather than imposing a strong
phase-equality constraint (based on the assumption that thepeak is the trace of a constant-amplitude, constant-frequency
sinusoid), one can constrain the synthesis phases around thepeak to be related in the same way as the analysis phases: thephase differences between successive channels around a peak
are made identical in the synthesis STFT to the correspondingisphase differences in the analysis Fourier transform. If
the center frequency of the dominant peak, we set

Fig. 5. Factor-1.4 time-scaling of a constant amplitude chirp, identity
peak-phase-locking. Time-domain amplitude-envelope of the modified signal.

Second, this new technique requires trigonometric calculations only for peak channels: once the synthesis phase of thepeak channel has been determined, one can calculate the angle
required to rotate
into
as follows:(13)
and obtain the neighboring
then calculate the phasor
channels by use of simple complex algebra
(14)

(12)
for all channel in the peak’s region of influence, as defined
above.
Note that the idea of preserving the phase-relations between
nearby bins has been independently proposed in [2] in the context of integer-factor time-scale modifications, and previouslyin [11] as a means to reduce transient smearing.
This phase-locking mechanism significantly improves theconsistency of the resulting series of STFT and greatly reduces
the phasiness of the modified signal. Applying this techniqueto the chirp signal of our previous example (again with atime-scale factor of 1.4) produces the signal shown in Fig. 5.
The resulting signal shows no sign of amplitude modulation,
37
and the consistency distance was measured to bedB, a very large improvement over the mere 6.5 dB of thenon phase-locked modification.
Identity phase locking has two major computational advantages. First, since phase unwrapping is performed only on
peak channels, the instantaneous frequency of the underlying
sinusoid is sure to be near the center frequency of the channelin question. This means that the phase-unwrapping constraint
can be relaxed, and larger values ofcan beused. In practice, an input overlap of 50% is possible without
generating phase-unwrapping errors. Since the usual requirement for phase-vocoder-based time-scale modification is aminimum of 75% overlap (for standard analysis windows, suchas Hanning or Hamming), identity phase locking essentiallycuts the computational cost in half!

which can be easily shown to satisfy the phase-locking equation (12): neighboring channels only require one complex
multiply!
The identity phase-locking scheme can be summarized in
the following steps.
1) For the new STFT frame, locate prominent peaks.
2) For each peak, calculate the instantaneous frequency
using horizontal phase unwrapping, and calculate theupdated synthesis phase, according to (6).
3) Calculate rotation angle according to (13) and phasor.
4) Apply rotation to all channels around and including peakchannel, according to (14).
5) Repeat the above steps for the next peak, until all peaks
have been processed.
6) Proceed to the next synthesis frame.
2) Scaled Phase Locking: An improvement over the preceding technique comes from recognizing that if a peakat frameto channel
switches from channel
at framethe unwrapping equation (5) should be basedinstead ofon. Likewise, the phase-propagation equation (6)should be(15)
where the phase increment
rather thanis accumulated to. It is easy to show

330

IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999

that in that case, the synthesis phase at the peak channel
is indeed,corresponding to sinusoid at time
which is not necessarily the case in the preceding technique.
The problem is then to determine what peak in frame
corresponds to the peakin frame . A very simple way of
doing this is to pick the peak of the region to which channel
belonged in frame. Accordingly, to calculate theat frame , one can simply look
synthesis phase of channel
belonged toup the dominant peak in the region channel, and use its analysis and synthesis phases,in framewhen applying (5) and (6). This being done, the neighboring
channels can be synchronized to the peak, and the identity
phase-locking equation can be generalized as

TABLE ICONSISTENCY MEASURE FOR VARIOUS
SYNCHRONIZATION TECHNIQUES, CHIRP SIGNALTABLE IICONSISTENCY MEASURE FOR VARIOUS
SYNCHRONIZATION TECHNIQUES, SPEECH SIGNAL(16)
where is a phase scaling factor. Identity phase locking is. Exactly how the phases should be modified
simplyupon synthesis to ensure proper vertical phase coherence is
not easy to assess, as explained in Section II-B1. However, itappears that identity phase locking can be further improved
to a value between one and . There is little
by setting
theoretical ground to justify such a choice, since the phasesin the modified signal should probably not be related to thosein the original signal in a linear way. However, when integer
modification factors are used in the standard implementationof the phase-vocoder, and when the initialization of (11) isused, it is easy to verify that phase-differences are also scaled. Moreover, informal listening tests have shownbyhelps further reduce phasiness.
that setting
must be unwrapped acrossNote that the phases
channels around the peak channel before applying (16), inchannel jumps in the synthesis phases.
order to avoid
In contrast to identity phase locking, scaled phase locking
does not permit the STFT values in neighboring channels to
be calculated simply via a complex multiplication; therefore
its implementation requires somewhat more computation. On
the other hand, the quality of the time-scaled signals has beenfound to be consistently higher with scaled phase-locking thanwith identity phase locking.
The scaled-phase-locking scheme can be summarized in thefollowing steps.
1) For the new STFT frame, locate prominent peaks.
2) For each peak channel , locate corresponding peak inpreceding frame, calculate instantaneous frequency using horizontal phase unwrapping, and calculate updated
synthesis phase according to (15).
3) Unwrap analysis phases across all channels in the regionof influence.
4) For each channel around the peak channel, calculate
analysis phase difference between peak and current
channel, and calculate current synthesis phase using(16).
5) Repeat the above steps for the next peak, until all peaks
have been processed.
6) Proceed to the next synthesis frame.

D. Comparison of Phase-Locking Techniques
1) Consistency Measure: Although the consistency measure of (7) is not a very accurate measure of phasiness, it still
gives an indication of how well various synchronizing schemes
perform. Two signals were used to compare the synchronizing
techniques: a chirp sinusoid identical to that used in Section IIC4, and a segment of speech signal. The speech signal was amale speaker uttering the word “before,” sampled at 16 kHz.
Four synchronization techniques were compared, along withthe standard, nonsynchronized phase-vocoder technique. The
synchronization techniques used were Puckette’s loose phaselocking [10], iterative magnitude-only reconstruction [4] (measured after five and 50 iterations), identity phase locking, and. The modification factor
scaled phase locking with, and the initial synthesis phases were set to thewas. The FFT size
initial analysis phases
was 1024, with an output hop factor of 256.
Table I presents the consistency measure for the chirp
signal. The results indicate that some type of synchronizationis required to insure some degree of consistency. As expected,
loose phase locking is significantly less effective than either
identity or scaled phase-locking. Iterative magnitude-only reconstruction, even after 50 iterations, still does not match our
synchronization techniques as the iterative procedure seems
to be caught in a local minimum. Most of the consistency
improvement is achieved within the first few iterations.
Table II presents the consistency measure for the speech
signal. Again, our phase-synchronization techniques outperform loose phase locking, but some marginal improvement inconsistency can be obtained by the iterative procedure, witha large number of iterations. Also included is the consistency
measure for pitch-synchronous overlap add (PSOLA), a highquality time-domain technique based on overlap-adding small
segments of waveform [7].
2) Informal Listening Tests Listening tests on speech andpolyphonic musical signals partially confirm the results of theLAROCHE AND DOLSON: TIME-SCALE MODIFICATION OF AUDIO

consistency measure. The standard, nonsynchronized phasevocoder technique has the poorest consistency measure, and it
always sounds worse than the other techniques.
However, the consistency measure does not always accurately reflect the perceived phasiness of the modified signal.
For example, identity phase locking and scaled phase locking
have similar consistency scores, but the latter consistently
sounds better than the former. Conversely, the higher consistency measure obtained by the iterative reconstruction does
not correspond to any perceptible quality improvement. In fact,
the modified signal is plagued with an undesirable roughness
that other techniques do not exhibit (an artifact mentioned in
the original paper [4]). Nevertheless, it appears that low valuesof the consistency measure (above 5 dB) always indicate thepresence of phasiness in the signal.
The influence of the parameter in scaled phase locking ismore dramatic when 75% overlap is used. In that case, and for), identity phase locking canlarger modification factors (in (16) significantly
still be somewhat phasy. Using
reduces phasiness. When 50% overlap is used, however,must be kept closer to one to avoid undesirable roughness inthe output signal. On speech signals, the modified voice has
more presence, and nonvoiced segments sound slightly morenatural with scaled phase locking than with identity phaselocking.
For integer modification factors, the standard algorithm
without phase unwrapping but with proper initial conditions) yields high-quality results which only
(such thatscaled phase locking can match for noninteger modification
factors. Some residual phasiness can still be heard in themodified signals, especially for larger modification factors.
Finally, when speech signals are processed, all the abovephase-locked techniques still exhibit more reverberation orphasiness than time-domain techniques such as the PSOLA
technique [7]. Surprisingly, the consistency measure for the
speech signal modified via the PSOLA technique is worse
than with other techniques; this confirms that the consistency
measure is not a clear indicator of phasiness.

E. Conclusion
We have presented a detailed explanation for the presence ofphasiness or reverberation in signals that have been time scaledby phase-vocoder techniques. We have shown that sinusoidal
components crossing channels generate both phase incoherence between adjacent channels and phase “discontinuities”
between successive frames. For integer modification factors,a suitable choice of initial analysis and synthesis phases
eliminates this problem entirely, yielding high-quality results.
For noninteger modification factors, phase incoherence can be
avoided by using one of the two phase-locking techniquespresented above. These techniques dramatically reduce thephasiness of the output signal and also enable the use of 50%
overlap between frames; this decreases the computational costby at least a factor of two.
Although the quality of the modified signals obtained via
phase locking is far better than that obtained using the standard

331

phase-vocoder technique, it nevertheless remains inferior to thequality attainable via time-domain techniques for monophonic,
pitched signals such as speech. One reason for this is that,
ideally, not only phases but also amplitudes should be modified
when performing time-scale modification. In the frequency
domain, a chirp sinusoid has a wider center lobe than aconstant-frequency sinusoid. Time-stretching it by a large
factor should turn it into a near-constant frequency with anarrower center lobe. This fact is not taken into account in
the phase-vocoder techniques presented in this paper. Incorporating such refinements could increase the quality of themodification, but the resulting complication might rival thatof sinusoidal-modeling methods [6]. Refer to [11] for thedescription of a technique which attempts to modify both
STFT phases and amplitudes.
Another reason might be linked to the lack of phasesynchronization between harmonics of the same fundamental,
or lack of “shape-invariance” as it has been called in theliterature. Achieving shape-invariance would require synchronizing peak-phases with one another, an issue which is notaddressed by the techniques presented here. Also, note thatthe concept of shape-invariance is only valid for sounds madeup of quasiharmonic signals. Refer to [3], [12], or [14] forthe description of techniques that attempt to achieve shapeinvariance.
ACKNOWLEDGMENT
The authors would like to thank O. Cappé for his help in
reviewing early versions of this contribution, and the reviewers
for their insightful suggestions to improve the clarity of thispaper.
REFERENCES[1] M. Dolson, “The phase vocoder: A tutorial,” Comput. Music J., vol. 10,
pp. 14–27, 1986.
[2] H. J. S. Ferreira, “An odd-DFT based approach to time-scale expansionof audio signals,” to be published.
[3] E. B. George and M. J. T. Smith, “Analysis-by-synthesis/Overlap-add
sinusoidal modeling applied to the analysis and synthesis of musicaltones,” J. Audio Eng. Soc., vol. 40, pp. 497–516, 1992.
[4] D. W. Griffin and J. S. Lim, “Signal estimation from modified short-time
fourier transform,” IEEE Trans. Acoust., Speech, Signal Processing, vol.
ASSP-32, pp. 236–243, Apr. 1984.
[5] J. Laroche, “Time and pitch scale modification of audio signals,” inApplications of Digital Signal Processing to Audio and Acoustics, M.
Kahrs and K. Brandenburg, Eds. Boston, MA: Kluwer, 1998.
[6] R. J. McAulay and T. F. Quatieri, “Speech analysis/synthesis basedon a sinusoidal representation,” IEEE Trans. Acoust., Speech, Signal
Processing, vol. ASSP-34, pp. 744–754, Aug. 1986.
[7] E. Moulines and J. Laroche, “Non parametric techniques for pitch-scale
and time-scale modification of speech,” Speech Commun., vol. 16, pp.
175–205, Feb. 1995.
[8] S. H. Nawab, T. Quatieri, and J. S. Lim, “Signal reconstruction fromshort-time fourier transform magnitude,” IEEE Trans. Acoust., Speech,
Signal Processing, vol. ASSP-31, pp. 986–998, Aug. 1983.
[9] R. Portnoff, “Time-scale modifications of speech based on short-time
Fourier analysis,” IEEE Trans. Acoust., Speech, Signal Processing, vol.
29, pp. 374–390, 1981.
[10] M. S. Puckette, “Phase-locked vocoder,” IEEE ASSP Workshop on
Applications of Signal Processing to Audio and Acoustics, 1995.
[11] T. F. Quatieri, R. B. Dunn, and T. E. Hanna, “A subband approach totime-scale expansion of complex acoustic signals,” IEEE Trans. Speech
Audio Processing, vol. 3, pp. 515–519, Nov. 1995.
[12] T. F. Quatieri and J. McAulay, “Shape invariant time-scale and pitch
modification of speech,” IEEE Trans. Signal Processing., vol. 40, pp.
497–510, Mar. 1992.

332

IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 7, NO. 3, MAY 1999[13] S. Roucos and A. M. Wilgus, “High quality time-scale modification of
speech,” in Proc. IEEE ICASSP’85, Tampa, FL, pp. 493–496.
[14] B. Sylvestre and P. Kabal, “Time-scale modification of speech using an incremental time-frequency approach with waveform structurecompensation,” in Proc. IEEE ICASSP’92, pp. 81–84.

Jean Laroche (M’92) was born in Bordeaux,France, on July 20, 1963. He received the M.S.
degree from the Ecole Polytechnique in 1986
and the Ph.D. degree from the Ecole Nationale
Supérieure des Télécommunications, Paris, France,
in 1989, both in signal processing.
In 1990 he was a Fellow of the ITT international
grant at the Center for Music Experiment,
University of California, San Diego. In 1991, hebecame an Assistant Professor at Telecom Paris in
the Signal Department, teaching audio and speechprocessing and acoustics. Since 1996, he has been a researcher at the Joint Emu/Creative Technology Center, Scotts Valley, CA, helping design techniques
for advanced music and audio processing.

Mark Dolson (S’81–M’82) was born in Pittsburgh,
PA, in 1955. He received the B.S. degree in electrical engineering from Carnegie Mellon University,
Pittsburgh, PA, in 1976, and the Ph.D. degree inelectrical engineering from California Institute ofTechnology, Pasadena, in 1983.
From 1978 through 1982, he was a Scientist at
Hughes Research Laboratory, Malibu, CA. He spent
the next 11 years as a Research Scientist at the
University of California, San Diego, focusing on theapplication of digital signal processing to problems
in music, speech, and perception. In 1994, he joined the Joint E-mu/Creative
Technology Center, Scotts Valley, CA, where he is now Manager of Digital
Signal Processing Research.