language for web browsers) as a bridge between the user and
libPd.

We implemented knobs (a.k.a. dials), added support for text and number boxes, including Pd-L2Ork's copy/paste and input exclusivity support. We added support for vu-meters to show audio peaks, and support for links, both to other patches and to external Web pages. We implemented drop-downs to select between multiple objects, and we added support for images, as well as raw mouse and keyboard event objects. Finally, we implemented arrays, which are effectively interactive point, line, or bar charts containing many values inside one object. L2Ork Tweeter uses arrays to allow the user to define the amplitude and modulation envelopes of their instruments, so we also support the modification of arrays via click and drag.

Given the increasingly mobile nature of the Web, through-
out the process of implementing these GUI objects we kept mo-
bile browsers in mind. All text entry fields support the virtual
keyboards used by mobile browsers, and other fields support
touch interactions in addition to mouse interactions. Because
of this, Pd-L2Ork patches (including L2Ork Tweeter) can now
be opened on mobile devices without installing any software, by
using WebPdL2Ork in a browser, though the UI of many patches
(including L2Ork Tweeter) is not very conducive to touch inputs
because of small click targets.

3.5. Files

Many patches rely on reading files from the local file system. L2Ork Tweeter, for instance, allows users to load instrument and loop presets from preset files bundled alongside it. Emscripten provides a virtual file system for Pd-L2Ork to use, but that file system starts out empty with only Pd-L2Ork's executable code and the shared library code of any externals which were built alongside it. When a patch is fetched from an arbitrary relative or absolute URL and loaded into WebPdL2Ork, it is injected into Emscripten's file system, but there is no way to know at load-time what other files the patch will attempt to execute at run-time. These can be other patches, or just about any other file format supported by Pd-L2Ork, such as images, text files, sound files, etc.

To allow patches to find files stored in the same location as
themselves, we replaced Emscripten's implementation of open-
ing a file in the virtual file system with our own. When Pd-
L2Ork attempts to open a file, we first check if it already ex-
ists in the virtual file system. If it does, then we open the file
and give the reference to Pd-L2Ork in exactly the same man-
ner as Emscripten's default function does. Otherwise, we try
to find the file in the same location as the patch was loaded
from. For instance, if the open patch's source is http://
example.org/some/patch.pd and it tries to load "path/
data.txt", we will ask the server to fetch data.txt from http:
//example.org/some/path/data.txt. If the file ex-
ists at that location, we inject it into Emscripten's file system
and then return a reference to Pd-L2Ork. All of this is transpar-
ent to the core, since opening a file blocks until we either find or
fail to find the file. From the core's perspective, it is interacting
with a normal (albiet somewhat slow) file system.

3.6. Abstractions

Another core feature of Pd-L2Ork (and all other variants of Pure-
Data) is the ability to embed patches (a.k.a. abstractions) inside
other patches. In this way, the same patch can be re-used many
times, similar to an object-oriented class in more traditional lan-
guages. PdWebParty did not support this, so we added support
for it in WebPdL2Ork. While we could use the same method as
mentioned above for allowing the core to read files, there are sev-
eral major problems with that method because of how Pd-L2Ork
loads patches.

The first issue is that Pd-L2Ork's core loads abstractions se-
quentially when parsing a patch. As mentioned in the previous
section, the process of opening a file is entirely synchronous.
The thread that attempts to open an abstraction is blocked until
the network request which attempts to fetch that abstraction ei-
ther succeeds or fails. Repeated accesses to the same abstraction
can complete quickly since after the first request the file will be
saved in the virtual file system, but as the number of unique ab-
stractions adds up, the latency increases drastically. What takes
a matter of milliseconds on a real file system may take seconds
in WebPdL2Ork, since every new abstraction requires a syn-
chronous network request. Another major issue is that Pd-L2Ork
uses a list of search paths to find abstractions. For any given ab-
straction it will attempt to open it in each search path that it has
registered, one after the other, until it finds the abstraction or ul-
timately fails, outputting an error. In WebPdL2Ork, each attempt
at guessing the file path would require its own network request
until the correct one is found, further increasing the latency.

For these reasons, handling abstractions at run-time is not
feasible or scalable. Since all references to abstractions are eas-
ily found in the patch's code, unlike other types of file accesses,
we decided to perform static analysis of the patch at load-time to
make all the necessary network requests in a more efficient man-
ner. To do this, we added a fetch stage to the patch processor
before the main processing stage which handles GUI objects.
When the patch is first received from WebPdL2Ork's server,
the processor scans the entire patch to compile a list of possi-
ble references to additional patches. Once it has this list, it re-
quests all of the patches from WebPdL2Ork's server in parallel.
WebPdL2Ork's server then responds with all of the patches that
it found, and the processor performs the same analysis on those
new patches recursively until no more patches are found. Then,
during the main processing, it inlines any references to the ab-
stractions with the abstractions themselves. This has a side effect
of increasing the total in-memory size of the patch, but the per-
formance gains are well worth it. Using L2Ork Tweeter as an
example, we perform the 281 searches necessary to resolve all
abstractions in only 3 network cycles. Since processing time is
negligible compared to network latency, this reduces the latency
to just 1% of what it would have been if all requests were per-
formed in sequence.

Another side effect of fetching all abstractions at load time
is that we are able to handle graph-on-parent patches (patches
which have GUI objects inside of abstractions). After the ab-
stractions are inlined into the main patch, the patch processor
finds their GUI objects in the exact same manner as it finds GUI
objects that were in the main patch and creates DOM elements
for the user to interact with.

3.7. Externals

We have added support for 20 of the 23 external libraries that Pd-L2Ork supports and PdWebParty does not. These include all externals used by L2Ork Tweeter, as well as others. Most notably, WebPdL2Ork now supports pdlua, pdjs, neuraln, flite, cyclone, and OSCx. Plua allows patches in WebPdL2Ork to run Lua [16] scripts in the browser for logic and audio processing, and pdjs does the same for JavaScript code. Neuraln provides support for patches to train and run neural networks and use them in audio processing. Flite provides text-to-speech support, clone is widely used in many patches for logic processing, and OSCx provides Open Sound Control [17] interfaces for patches to use. For most externals, the only work was to adjust their build scripts to support using Emscripten's compiler rather than gcc. For some patches this was straightforward, and for others it was very tedious. However, some externals required major changes in order to run in the browser. We had to make modifications to the flite external in order to get it to run in a Web-based environment, and we re-implemented pdjs from scratch. On the desktop version, pdjs bundles its own pre-compiled V8 [18, 19] engine.
There are no pre-compiled V8 binaries for browsers because the browser itself already necessarily contains a JavaScript engine. Because of this, we re-implemented pdjs in the client JavaScript rather than in Pd-L2Ork's C core, so that it could make use of the browser's JavaScript engine. We used dedicated Web workers to run scripts and set up a custom namespace inside them to emulate the namespace that desktop pdjs exposes to V8.

## 3.8. Wired Connections

Adding support for the missing GUI objects, files, abstractions, and externals goes a long way towards running L2Ork Tweeter, but libPd is still very limited in how it allows the core and the client to interact. Pd-L2Ork supports both wired and wireless connections, but libPd only allows the client to monitor wireless connections. Yet, wired connections are much more common in patches than wireless connections (in great part because wired connections make tracing execution order considerably easier and as such are considered a better coding practice, where possible), and L2Ork Tweeter is no exception. Because of this, we created a compatibility layer inside WebPdL2Ork to deal with this libPd's limitation. When the patch processor scans the patch for GUI objects, it maintains a list of all the GUI objects it has found. Then, when the it comes across a wired connection it checks its list of GUI objects to determine if either end of the connection connects to a GUI object. If so, it knows that the data over that connection won't be received through libPd. Thankfully, the Pure-Data ecosystem has two objects which we use to solve this problem: "send" (a.k.a. "s") and "receive" (a.k.a. "r"). "s" takes in a wired connection and passes through any data to a wireless connection, and "r" does the inverse. When a wired connection is found with a GUI object, the processor injects either an "r" or "s", connects it to the wired end of the connection, and gives it a unique wireless connection name to wirelessly connect to the GUI object. If the connection is between two GUI objects, the processor skips the step of adding an "s" or "r" and makes a direct wireless connection between the two objects. There are additional edge cases which add more complexity for full compatibility with wired patches, but they follow the same logic as these simple cases. As a result of our wired to wireless conversion, WebPdL2Ork is now able to load any patch regardless of the connection scheme it uses.

## 3.9. Networking

Another major compatibility issue when running patches inside a Web browser is networking. Many patches use TCP [20, 21] sockets to send and receive data over the network. L2Ork Tweeter, for instance, uses TCP sockets to connect to a Tweeter server which synchronizes the musical experience between multiple users performing or composing simultaneously. Browsers do not support TCP sockets, instead opting for HTTP-based WebSockets [22]. Emscripten has a built-in support for converting the TCP sockets created by C's standard library into WebSockets, but this assumes that whatever service the C code is trying to connect to is also capable of communicating using WebSockets. Since WebSockets use a completely different protocol than TCP sockets, this is rarely the case.

To solve this problem, we created a WebSocket bridge which sits in between Pd-L2Ork's core and the endpoints that it tries to connect to. Similarly to how we intercept when a patch tries to open a file, we intercept when the browser tries to open a WebSocket (as a result of Emscripten converting a TCP socket). When this happens, the interceptor redirects the WebSocket to WebPdL2Ork's server and immediately sends a packet containing the original IP that the socket was meant to connect to. WebPdL2Ork's server then opens a TCP socket on the client's behalf to that IP and forwards all data sent on either socket to the other.

Although Emscripten provides TCP to WebSocket conversion as a part of its compilation process, it does not provide DNS [23, 24] resolution. Because of this, any patches that connect to a server using its domain name rather than its IP address fail to resolve the IP address for that domain. To solve this, we replace C's low-level "gethostbyname" function (which all higher level DNS resolution functions use internally) with our own custom implementation. Instead of using the system's DNS resolver, which doesn't exist in WebAssembly, we open a socket to WebPdL2Ork's server, which resolves the domain on behalf of the client and returns the result.

By implementing transparent DNS resolution and Web-Socket bridging, we have been able to implement networking without making any changes to existing Pd-L2Ork objects. This was a major goal of ours since there isn't one standard Pd-L2Ork object for networking, but rather a large selection of objects, each of which does things slightly differently. If we were to bake WebPdL2Ork's networking support into specific objects, we would need to maintain the functionality as objects evolve and add it to new objects as they are created. Instead, we have managed to avoid this by making all existing C networking code work without specific changes for WebPdL2Ork.

## 3.10. Multiple Open Patches

The last major area where we addressed PdWebParty's limitations is having multiple patches open simultaneously. Pd-L2Ork allows the user to open multiple patches simultaneously, as well as to communicate with each other using wireless connections. We have added similar functionality to WebPdL2Ork using a browser service worker. Service workers exist at the site level rather than the page level, so we used them to allow multiple tabs of WebPdL2Ork to communicate with each other and manage their relationships. When a WebPdL2Ork tab is opened, we perform dependency and GUI processing on the patch as detailed in previous sections, but there is one final step which we have not yet mentioned. Before installing the patch to the virtual file system and starting Pd-L2Ork, the page communicates with WebPdL2Ork's service worker to check if it is a parent page or a child page. Browser tabs that were opened manually by the user are defined as parents, and they run their own patches. When the user clicks on a link in a patch that goes to another patch, we create a new child tab on the user's behalf. Instead of running its own patch, it sends its patch to the service worker, who in turn sends it to the parent who created that child. The parent then opens the child's patch using the same libPd instance used to run the parent's patch. From the core's perspective this is the same as when multiple patches are opened on the desktop (all windows are shared between the single Pd-L2Ork application). Finally, instead of subscribing to events from libPd, child pages will subscribe to events from their parent via the service worker. In this way, multiple tabs of WebPdL2Ork can share the same libPd instance and function similarly to how multiple windows of Pd-L2Ork behave on the desktop.

# 4. OUTCOME

## 4.1. L2Ork Tweeter in the Browser

As a result of all these enhancements to PdWebParty, WebPdL2Ork is now able to run L2Ork Tweeter with near-parity to desktop Pd-L2Ork. All of the GUI elements in L2Ork Tweeter are supported by WebPdL2Ork, as well as all the externals that Tweeter uses. All of L2Ork Tweeter's presets are available because of our file interceptor, and its networking features work thanks to our networking compatibility layer. Even L2Ork Tweeter's multi-window features, like the loop and preset widgets (independent patches that supplement L2Ork Tweeter with additional functionality), are able to open in popups and properly communicate with the
<figure><img src="image_111.png" /><figcaption>Figure 2: L2Ork Tweeter running inside WebPdL2Ork (4.1).</figcaption></figure>

main window. We have deployed a WebPdL2Ork instance
to https://l2ork.music.vt.edu:3000/. That
instance includes L2Ork Tweeter, which can be accessed
by navigating to https://l2ork.music.vt.edu:
3000/?url=L2Ork-Tweeter/L2Ork-Tweeter.pd.

To experiment with other patches hosted at external lo-
cations, place the URL to that patch in the url query
parameter: https://l2ork.music.vt.edu:
3000/?url=https://example.org/patch.pd.

The following figure showcases L2Ork Tweeter running using
WebPdL2Ork. 2

## 4.2. Limitations

Although we have achieved near-parity to desktop Pd-L2Ork,
there are several areas where WebPdL2Ork still falls short com-
pared to its desktop counterpart. First, while we are able to load
external files into the virtual file system on demand, we currently
do not support saving files from the virtual file system to an ex-
ternal source. In the case of L2Ork Tweeter, this means that
any presets provided alongside the hosted L2Ork Tweeter patch
can be loaded and modified, but they cannot be saved, as the
virtual file system is lost when the browser tab is closed. Ad-
ditionally, unlike desktop Pd-L2Ork, which allows the user to
select an audio API (such as JACK or ALSA), WebPdL2Ork is
limited to the audio interface which the browser exposes. Fi-
nally, we do not support editing patches via WebPdL2Ork. So,
while L2Ork Tweeter is now available in the browser, we will
continue to do all development on Tweeter via the desktop ver-
sion of Pd-L2Ork. Finally, when opening multiple windows in
WebPdL2Ork, once a window is closed it cannot be reopened.
We have not yet determined why, but libPd does not function
properly when reopening a previously closed patch. We hope to
resolve this limitation in the near future.

## 4.3. Future Work

We plan to continue the development of WebPdL2Ork to address
the limitations detailed above. In particular, we look forward to

designing a mechanism to save and load data in the virtual file
system. Once this functionality is implemented, both composers
and performers will have the flexibility to use L2Ork Tweeter
whenever and wherever they want, both on the desktop and on
the Web. At some point in the future, we look forward to adding
the ability to edit patches using a WebPdL2Ork editor. This ef-
fort will commence once WebPdL2Ork has addressed all run-
time incompatibilities and limitations.

We are also interested in exploring the possibility of a central
repository for Pd-L2Ork patches. Since WebPdL2Ork can load
patches from arbitrary URLs, it lends itself to a repository where
users could create and share patches for anybody on the internet
to view and run.

# 5. CONCLUSIONS

Through our work on WebPdL2Ork, L2Ork Tweeter is now ac-
cessible directly inside a Web browser, eliminating the need for
users to install Pd-L2Ork or configure their system. This sig-
nificantly reduces the barrier to entry, allowing more people,
students, musicians, and researchers alike, to experiment with
L2Ork Tweeter using any device and any operating system. By
broadening access to this tool, we also extend the reach of the
Linux Laptop Orchestra, making collaborative live music pro-
duction and sound design more widely available than ever be-
fore.

More broadly, WebPdL2Ork represents a step forward in the
evolution of Pure-Data based environments. By filling the gaps
in PdWebParty's prototype, we have demonstrated how power-
ful interactive audio tools can be adapted for greater compati-
bility while maintaining their core functionality and the features
that set them apart. Though challenges and shortcomings re-
main, WebPdL2Ork's ability to run complex patches like L2Ork
Tweeter in a browser shows the potential for further innovation
in this field. We look forward to the future of WebPdL2Ork,
Pd-L2Ork, and L2Ork Tweeter.
## 6. ACKNOWLEDGMENTS

This project is sponsored by the Department of the Navy, Office of Naval Research under ONR award number N00014-22-1-2164. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.

We acknowledge Zach Lee for creating PdWebParty, which served as the foundation for our work on WebPdL2Ork.

## 7. REFERENCES

[1] Ico Bukvic, "L2Ork » Linux Laptop Orchestra," Jan. 2009.

[2] Ivica Ico Bukvic, "pd-l2ork," Mar. 2025, original-date: 2020-10-14T13:55:01Z.

[3] Ivica Bukvic, Liesl Baum, Bennett Layman, and Kendall Woodard, "Granular learning objects for instrument design and collaborative performance in k-12 education," in *Proceedings of the International Conference on New Inter-faces for Musical Expression*, 2012.

[4] Peter Brinkmann, Peter Kirn, Richard Lawler, Chris McCormick, Martin Roth, and Hans-Christoph Steiner, "Embedding pure data with libpd," in *Proceedings of the Pure Data Convention*. 2011, vol. 291, Citeseer.

[5] Peter Brinkmann, Dan Wilcox, Tal Kirshboim, Richard Eakin, and Ryan Alexander, "libpd: Past, Present, and Future of Embedding Pure Data," *PdCon 2016*.

[6] David Kushner, "Engineering Spore," *IEEE Spectrum*, vol. 45, no. 9, pp. 36–40, Sept. 2008, Conference Name: IEEE Spectrum.

[7] Zack Lee, "PdWebParty," Mar. 2025, original-date: 2020-12-01T15:08:18Z.

[8] Andreas Haas, Andreas Rossberg, Derek L. Schuff, Ben L. Titzer, Michael Holman, Dan Gohman, Luke Wagner, Alon Zakai, and Jf Bastien, "Bringing the web up to speed with WebAssembly," in *Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation*, Barcelona Spain, June 2017, pp. 185–200, ACM.

[9] Alon Zakai, "Emscripten: an LLVM-to-JavaScript compiler," in *Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion*, Portland Oregon USA, Oct. 2011, pp. 301–312, ACM.

[10] Gerard Gallant, *WebAssembly in Action: With examples using C++ and Emscripten*, Simon and Schuster, 2019.

[11] Azat Mardan, "Using Express.js to Create Node.js Web Apps," in *Practical Node.js*, pp. 51–87. Apress, Berkeley, CA, 2018.

[12] Evan Hahn, *Express in Action: Writing, building, and testing Node.js applications*, Simon and Schuster, 2016.

[13] Alessandro Benoit, *NW. js Essentials*, Packt Publishing Ltd, 2015.

[14] Will Reese, "Nginx: the high-performance web server and reverse proxy," *Linux Journal*, vol. 2008, no. 173, pp. 2, 2008, Publisher: Belltown Media Houston, TX.

[15] Dirk Merkel, "Docker: lightweight linux containers for consistent development and deployment," *Linux j*, vol. 239, no. 2, pp. 2, 2014.

[16] Roberto Ierusalimschy, *Programming in lua*, Roberto Ierusalimschy, 2006.

[17] Matthew Wright and Adrian Freed, "Open soundcontrol: A new protocol for communicating with sound synthesizers," in *ICMC*, 1997.

[18] Georgiy Krylov, Maria Patrou, Gerhard W. Dueck, and Joran Siu, "The evolution of garbage collection in v8: google's javascript engine," in *2020 9th Mediterranean Conference on Embedded Computing (MECO)*. 2020, pp. 1–6, IEEE.

[19] Jan Kasper Martinsen, Håkan Grahn, and Anders Isberg, "Preliminary Results of Combining Thread-Level Specification and Just-in-Time Compilation in Google's V8," in *Sixth Swedish Workshop on Multicore Computing (MCC-13)*. 2013, Halmstad University.

[20] W. Eddy, "Transmission Control Protocol (TCP)," Tech. Rep. RFC9293, RFC Editor, Aug. 2022.

[21] J. Postel, "Transmission Control Protocol," Tech. Rep. RFC0793, RFC Editor, Sept. 1981.

[22] I. Fette and A. Melnikov, "The WSocket Protocol," Tech. Rep. RFC6455, RFC Editor, Dec. 2011.

[23] Paul Mockapetris, "Domain names: Implementation specification," Tech. Rep., 1983.

[24] P.V. Mockapetris, "Domain names - implementation and specification," Tech. Rep. RFC1035, RFC Editor, Nov. 1987.
# ZITA-AT2 AUTOTUNE

Fons Adriaensen

Zita Audio

fons@linuxaudio.org

## ABSTRACT

The *zita-at1* autotuner (by the author) was first released in 2011 as a Jack application, and it became more popular as one of the x42 LV2 plugins created by Robin Gareus. While it works reasonably well, its main drawback has been that it doesn't preserve the formant spectrum when changing the pitch of a voice signal. The new *zita-at2* to be launched at LAC 2025, should fix this issue. It does this by using the PSOLA algorithm instead of just resampling. This paper documents the main design of and the algorithms used in this application.

## 1. INTRODUCTION

Autotuning has become one of the standard processes used in the production of popular music ever since it was introduced by Antares Audio Technologies in 1997. It can be used to subtly correct a singer's small pitch errors, or as an effect of its own, creating a sort of robotic voice that has such perfect pitch that it no longer appears natural. In particular the latter use has become a matter of controversy and some people <fnref n="1" /> really hate it.

Developing a usable autotune application is of course a challenge to anyone interested in audio DSP, and that's why the author wrote *zita-at1* many years ago, and decided to have a fresh look at it more recently.

## 2. THE NATURE OF VOICE SIGNALS

The singing or speaking human voice waveform can be of two types: *voiced* or *unvoiced*.

A *voiced* signal is a periodic impulsive waveform generated by the vocal cords and filtered by the upper respiratory tract, the larynx, the pharynx and the oral and nasal cavities. Voiced signals include all vowels and also sounds such as a sustained "mmm".

The *pitch* is the fundamental frequency of the impulsive waveform, controlled by the tension of the vocal cords. The pitch frequency range for humans depends on the type of voice: approx. 80 to 330 Hz (E2 to E4) for a male bass singer and 250 to 1050 Hz (C4 to C6) for a soprano.

The waveform generated by the vibrating vocal cords is rich in higher harmonics, and the filtering by the vocal tract produces a spectrum that has a number of distinct peaks (at least two) called *formants*. The actual formant frequencies are typical for each vowel, and while they are slightly different between male and female voices they do not depend on the voice pitch. The lower formant is usually in the range from 200 to 900 Hz, the upper ones from 500 to 3500 Hz. Trained singers tend to have more pronounced higher formants.

An *unvoiced* signal is one that is produced without vibrating vocal cords. It doesn't have any defined pitch and consists of filtered noise, mostly produced by air flowing around the tongue and teeth and through the lips, or for whispered vowels, deeper down the vocal tract. Although they can be sustained, in normal speech and singing unvoiced sounds tend to be mostly transient.

## 3. THE ALGORITHMS

Three steps are required to modify the pitch of a voice in a controlled way. These will be the subject of the following sections.

### 3.1. Pitch Detection

The first task is to separate the voice signal into voiced and unvoiced parts, and to determine the exact pitch of the voiced ones.

Many methods have been developed to do this, see e.g. [1], [2] or [3]. Among the most practical ones is to compute the *autocorrelation* of a short fragment of the input signal, and this is what is actually used in *zita-at2*.

The autocorrelation function indicates how similar a signal is to a time-shifted version of itself. For a periodic waveform it will have a maximum value for shifts equal to an integer multiple the waveform period. For unvoiced input, which will mainly be filtered wideband noise, its value will drop down rapidly even for small time shifts.

The autocorrelation of a real valued fragment $X[i]$ is defined as $R[k] = \sum_{i} X[i]X[i-k]$. Dividing this by $R[0]$, which is just the total energy of the fragment, makes the result independent of signal amplitude.

The most efficient way to compute $R[k]$ is not by using the time domain definition above, but as the inverse FFT of the power spectrum of $X$, itself computed by a forward FFT. Some care is required when using this method as the FFT operations interpret time in a cyclic way – for an N-point FFT a shift of $k$ samples is equivalent to one of $N - k$ samples. So for values of $k$ approaching $N/2$ two smooth peaks will overlap which will offset the apparent peak position. To avoid this a time domain window with a length of twice the longest expected period should be applied before the FFT, and the FFT size should be somewhat larger than this window.

A bandpass filter (80 Hz to 5 kHz) is applied before computing the autocorrelation, anything outside this range doesn't contribute to a correct result. Given $R[k]$, the basic algorithm is to find the highest peak which also exceeds a minimal value. The exact position and magnitude of the correlation peaks is obtained by fitting a parabola to a small region around the maximum value.

There are some corner cases, mainly for signals having a single very strong harmonic. Figure 1 shows such a 'near miss'. This is for a synthetic signal with a frequency of 120 Hz (400 samples at 48 kHz sample rate), and a third harmonic 25 dB stronger than the fundamental. The first two peaks of the autocorrelation function are due to this strong third harmonic. The third one, which is only marginally stronger, corresponds to the actual signal period.

This does happen also for real-life signals, but usually in those cases there will be more than one strong harmonic. This results in a much less ambiguous correlation, see Fig. 2 for an example.

Most errors occur at the onset of a voiced fragment before it fills the entire window. The *zita-at2* autotuner computes the

<footnote n="1">¹including the author</footnote>
<figure><img src="image_112.png" /><figcaption>Figure 1: Ambiguous autocorrelation of a waveform with a single strong harmonic. Top left: waveform and window, top right: spectrum, bottom: autocorrelation of windowed waveform, X-axis in samples. The blue dot at 400 samples indicates the estimated pitch period</figcaption></figure>

autocorrelation every 5 milliseconds. This is more than strictly necessary, but allows the inclusion of some logic that helps to reject wrong estimates. Also allowing the user to select the expected pitch range and using a smaller window size for a high (female) voice helps to reduce false estimates.

<figure><img src="image_113.png" /><figcaption>Figure 2: A more complex spectrum removes the ambiguity. Top left: waveform and window, top right: spectrum, bottom: autocorrelation of windowed waveform, X-axis in samples.</figcaption></figure>

## 3.2. Computing the Corrected Pitch

The correct retuned pitch will usually be one of the notes of a 12-note per octave scale.<fnref n="2" /> The default scale is the equal temperature one, but others can be used by importing a Scala<fnref n="3" /> file. Also the user can set the frequency (normally 440 Hz) corresponding to A4 or MIDI note 69.

In *Auto Mode* the set of allowed notes is selected using the one octave virtual keyboard in the GUI. The enabled note nearest to the detected pitch is selected regardless of in which octave it is, so the pitch change will never be more than half an octave up or down. The *Bias* control allows to add some preference for

the current note, allowing the pitch to go off-tune more before a different one is selected.

<figure><img src="image_114.png" /><figcaption>Figure 3: Real life example. X-axis: time (s). Top Y: signal level (dB). Bottom Y: MIDI note number, blue: estimated pitch, red: correct pitch.</figcaption></figure>

Auto mode can work well for a trained singer who has good pitch control to start with, but it has its limitations. Figure 3 shows an example of what can be seen in real life. Here the correct melody is [Bb, F, D], [Bb, C, Bb, A, F#, D], as shown by the red lines. The blue line is the measured pitch. Note how the second Bb starts almost an octave below the correct pitch, and how this singer has a general tendency to slide up towards a note as it starts and to drop the pitch at the end. Strange enough all this doesn't sound as bad as it looks, but it will if the sliding pitch is quantised. Auto mode will not handle such cases very well.

In *MIDI mode* the allowed note selection is taken from MIDI input. This can be used 'live' by playing along, but it is most useful when the data is coming from a MIDI track in a DAW such as Ardour.<fnref n="4" /> MIDI mode can be used in two ways. If enabled together with Auto mode, any MIDI input takes precedence over the GUI, and the nearest active note, modulo octaves, will be selected. When used on its own, the *last* played MIDI note will determine the output pitch. In this case the absolute pitch (not ignoring octaves) is used. Also any pitch bend input will be taken into account. When there is no active MIDI note, the input will be considered *unvoiced* and just copied to the output.

A few more controls will affect the final corrected pitch. These mainly allow a choice between the very obvious autotune effect and a more natural sounding result The *Speed* control sets how fast the quantised pitch will slide from one note to the next, this is similar to portamento in a synth. The *Corr* control determines how much of the pitch error is corrected. This also has a *Var* option. When *Var* is on, only the variation of the error is used while any constant part is removed. The result is that the average pitch will be correct while at the same time any expression (e.g. vibrato) is passed through.

Finally there are two controls to add a constant pitch offset (± one octave), and a constant formant shift (± a third).

## 3.3. Generating the Retuned Signal

### 3.3.1. The PSOLA Algorithm

In *zita-at1* the retuning was done by resampling the original signal, skipping forward or back by an integer number of periods to

<footnote n="2">²Currently *zita-at2* only supports 12-note scales, but others could be added in later releases.</footnote>

<footnote n="3">³Scala is a program to define and analyse musical scales. See https://www.huygens-fokker.org/scala</footnote>

<footnote n="4">⁴A nice YouTube video demonstrates this using the LV2 version of *zita-at1*. See https://www.youtube.com/watch?v=ttXCsIb_2L8</footnote>
preserve the length. This does not only shift the pitch, but the entire spectrum – and hence the formant frequencies – up or down. The consequence is that only relatively small pitch corrections will sound natural.

The method used in *zita-at2* instead is known as **PSOLA** – Pitch Synchronous OverLap and Add. [4, 5] This is in fact a form of granular synthesis: the output signal is formed by adding overlapping *grains*, each of them representing one period of the waveform.

Given the estimated pitch period $P_E$, we can take a fragment of the input signal of length $2P_E$, and apply a raised cosine window. When this grain is added to the output every $P_E$ samples the result is a perfect reconstruction of the input. This is shown in the top part of Fig.4. In this example $P_E$ is 240 samples (200 Hz at a sample rate of 48 kHz).

<figure><img src="image_115.png" /><figcaption>Figure 4: Top: reconstruction with original pitch, bottom: reconstruction at 1.5 times original frequency</figcaption></figure>

Changing just the distance between the grains still produces a periodic signal but with a different pitch. The bottom part of Fig.4 shows the result when the original grain is output every 160 samples. This corresponds to a pitch change with ratio 3/2 – a musical fifth. In practice the amplitude of the grains will need to be adjusted in order to preserve the original loudness, but this is easy enough.

For best results the position of a grain in the input stream should be selected such that most of its energy is near the center of the window. Of course this can be done only for the first grain of a voiced fragment. The position of the following grains is then determined by the measured pitch period.

The main advantage of PSOLA over resampling is that it preserves the formants of the original signal. Figure 5 shows the spectra of the waveforms in Fig. 4. The retuned signal still has the same spectral peaks at 1 kHz and 3 kHz as the original, even if the fundamental frequency has changed by much more than would normally be required for autotuning.

The reason why PSOLA tends to preserve the formants is that the spectral envelope is a property of the basic grain. Resampling the grain will shift the formant frequencies, and this can be an interesting artistic effect. Moving the formants up in frequency produces a sort of 'squeaky' quality, while lowering them results in a 'hollow' voice.

On the downside, PSOLA may distort waveforms with a very narrow spectrum. Take the extreme case of a pure sine wave of frequency *F*. It can be said to have a strong 'formant' at *F*. When this is tuned down PSOLA will preserve this 'formant' and add harmonics around *F* even if these were not present at

<figure><img src="image_116.png" /><figcaption>Figure 5: Spectra of the waveforms shown in Fig.4</figcaption></figure>

all in the input. This is in practice not a problem with voice signals as these tend to be wideband and rich in harmonics.

### 3.3.2. *Implementation of PSOLA*

Although the basic PSOLA algorithm is not really complicated, any real-life implementation will require some nontrivial code.

In general neither the input nor the output pitch frequency will correspond to a period equal to an integer number of samples. This means we will need to shift the grain waveform by fractions of a sample. In practice this doesn't need to be exact, as long as the error does not accumulate over time. The current implementation uses eight-point interpolation. The delays are quantised to 1/32 of a sample, so all the necessary coefficients can be precomputed. If resampling is required in order to shift the formants, this can be done at the same time, it just means that the delay changes as the algorithm iterates over the samples of a grain. In *zita-at2* a single *Grain* class combines the fractional sample shift, the resampling and the windowing in a single highly optimised method.

In addition to that, there is a lot of rather tricky bookkeeping required in order to correctly select and position grains in the input and output streams, in particular if minimal latency is required.

## 3.4. Latency Issues

The current *zita-at2* implementation has a latency of around 20 ms, reducing to around 15 ms if configured for a female voice.

While latency is more or less irrelevant when using autotuning on a recorded track (as long as it is reported correctly so the DAW can compensate for it), it seems that some people want to use an autotuner for live performance.<fnref n="5" />

The minimal latency that can be achieved consistently depends strongly on the lowest pitch frequency that can be handled. Autocorrelation requires at least two periods to detect the pitch with any level of confidence. When configured for a minimum pitch frequency of 80 Hz (for a 'bass' male voice) the window size used is 32 ms, so the estimate is already 16 ms behind 'real time'. Then selecting the grain such that most of its energy is near the center of its window may add additional delay. Also at the same lower limit of 80 Hz, a single output grain can be as long as 25 ms. Only configuring the process for a higher minimum pitch frequency will allow lowering these figures.

<footnote n="5">⁵What good can come from that remains a mystery to the author.</footnote>
What can be done is to ensure that latency is as low as possible for the *unvoiced* fragments of the input signal. Since these tend to be transient, they probably contribute more to the perceived latency than the voiced parts.

Anyway, if someone claims that a derived version of *zita-at2* provides just a few milliseconds of latency, there are only two possibilities: either this is pure snake oil, or other aspects of the original performance will be seriously affected.

## 4. CONCLUSIONS

The preceding sections provide an overview of the algorithms used in *zita-at2*. It should be emphasised that this paper only scratches the surface, but the author hopes it will provide a useful starting point for anyone interested in developing the code, or creating a similar application.

Without doubt someone will turn the original Jack application into an LV2 plugin.<fnref n="6" /> This will certainly make *zita-at2* more accessible to most users, as well as provide new functionality, e.g. by allowing automation of the parameters.

## 5. ACKNOWLEDGMENTS

Many thanks to the anonymous reviewers for their comments and suggestions for improving this paper.

## 6. REFERENCES

[1] D. Gerhard, “Pitch extraction and fundamental frequency: History and current techniques,” Tech. Rep., Dept. of Computer Science, University of Regina, 2003, https://www2.cs.uregina.ca/~gerhard/publications/TRdbg-Pitch.pdf.

[2] Philip McLeod, “Fast, accurate pitch detection tools for music analysis,” 2008, https://www.cs.otago.ac.nz/research/publications/oucs-2008-03.pdf.

[3] Adriano Mitre, Marcelo Queiroz, and Regis R. A. Faria, “Accurate and efficient fundamental frequency determination from precise partial estimates,” 2006, http://www.ime.usp.br/~mqz/Mitre_AESBR2006.pdf.

[4] Eric Moulines and Francis Charpentier, “Pitch-synchronous waveform processing techniques for text-to-speech synthesis using diphones,” *Speech Communication*, vol. 9, pp. 453–467, 1990, https://courses.grainger.illinois.edu/ece420/sp2017/PSOLA.pdf.

[5] Sami Lemmetti, “Review of speech synthesis technology,” M.S. thesis, Helsinki University of Technology, Helsinki, Finland, 1999, http://research.spa.aalto.fi/publications/theses/lemmetty_mst/thesis.pdf.

<footnote n="6">⁶There are some 'usual suspects' expected to do this.</footnote>
# Presentation of Workshops

## Ardour Lua Scripting

Robin Gareus

The free/libre software Digital Audio Workstation (DAW) Ardour has extensive integrated support for the Lua scripting language. This allows one to extend functionality, streamline custom workflows, implement batch processing, or extend Ardour in ways useful for a given production. A basic introduction to Lua will be given, followed by a quick overview of Ardour's internal structure. This workshop will explain existing scripts and participants are encouraged to create custom scripts or audio-effects using Lua.

Target Audience:

* Audio engineers, producers, and creators using Ardour

* Intermediate to advanced users looking to enhance their workflow with scripting.

* Curious people, interested in free/libre software

## C++ Audio Development on Android with LDSP

Victor Zappi, Carla Tapparo

In this hands-on three-hour workshop, we will introduce attendees to LDSP, an open-source software environment that enables users to hack Android phones and program them as embedded Linux audio boards. We will bring phones already rooted and configured for LDSP for the attendees to use. During the first hour of the workshop, we will introduce the project and set up the necessary dependencies for LDSP on the participants' computers. The next hour and a half we will go through the different examples already made, covering oscillators, sensors and filters, amongst other capabilities. Throughout this process, the participants will be able to modify said code to familiarize themselves with LDSP. The remaining half hour of the workshop can be destined for the participants' free exploration, a more in-depth talk regarding the LDSP environment or how to root and configure phones, depending on the participants' interests.

## Yoshimi Insights and Developments

Will Godfrey

A wide ranging talk on how Yoshimi has developed over the last 16 years, and the lessons learned in the process. Also, how the team has grown, and the shifting roles of the developers. There will be demonstrations of various features that were developed, and an explanation of how some of the earlier ones created opportunites for the later ones.
# Presentation of Demos

## AudioReach: An End-to-End Audio Solution

Lai Patrick

The AudioReach solution developed by Qualcomm Technologies, Inc. addresses significant obstacles in the audio development landscape. This paper explores the comprehensive infrastructure provided by AudioReach, including its capabilities for off-target development, graphical design, testing, and deployment. The framework supports heterogenous distributed processing and offers a unified environment for developing audio use cases across various processors and SW & HW platforms. By open-sourcing AudioReach, Qualcomm aims to foster a community-driven project that enhances audio development efficiency and lowers costs for vendors, algorithm designer, tuning engineers, and system integrators.

## Free & open tools for Dante protocol, made through reverse engineering

Teodor Woźniak

Dante protocol makes it possible to transfer low-latency (sub-millisecond) audio through regular IP networking equipment. The data transfer and clocking is very similar to AES67, but Dante has many more APIs for management of audio flows, discovery of devices etc. Unfortunately the protocol is binary and not documented at all, but people all around the Internets have been reverse engineering it, thus we have:

* network-audio-controller, command line alternative to Dante Controller

* Inferno-AoIP, alternative to Dante Virtual Soundcard.

## Native Wayland Audio Plug-In User Interfaces

Julian Wolff

Wayland is a display server protocol that is aimed to replace the existing X11 window system protocol. The Wayland project started in 2008. Major Linux distributions have used Wayland by default since 2016. Still, there is currently no official Wayland support in any audio plug-in format. We present new interfaces that allow plug-ins to connect to a host application using standard Wayland protocols and showcase implementations in Studio One using in-house and third-party VST3 plug-ins.

## The HaptiCoupler: An Open Source System for Bidirectional Interaction with Hybrid Resonant Instruments

Matthew Davison

Latest breaking news about the HaptiCoupler!
<figure><img src="image_117.png" /></figure>
<figure><img src="image_118.png" /></figure>

9 782959 791123

19th Linux Audio Conference

Proceedings

<figure><img src="image_119.png" /></figure>