# Helmholtz Drum Machine: A Real-time Human-Computer Interactive Music System

Jingwei Liu*

University of California San Diego
La Jolla, CA 92092
jil121@ucsd.edu

## Abstract

This paper introduces an application that performs real-time machine learning-based music generation, synthesis, streaming, and training in Python. The real-time training based on user preference is a typical HCI (Human-Computer Interaction) system, which can be interpreted under various models such as TTOM (thinking through other minds) [8], active inference of communication [4], emotional inference devices [3], agent-environment system [1], and niche construction [2]. Here, we use the Helmholtz machine as the generative model to generate percussive drum musical patterns. The real-time training is instructed by user preference based on-line data construction, and the multi-processing real-time system is realized by threading in Python ¹.

## 1 The Helmholtz Machine

<figure><img src="image_1.png" /><figcaption>Figure 1: The Helmholtz Machine.</figcaption></figure>

<footnote>*Ph.D. candidate in Computer Music, UC San Diego
¹Please see the code at GitHub: https://github.com/Jovie-Liu/Helmholtz-Drum-Machine</footnote>
The Helmholtz Machine is a fully connected feedback neural network consisting of stochastic binary neurons with a hierarchical architecture. The model encodes the inputs into some internal representations via the bottom-up recognition model Q, and generates samples via the top-down generative model P (see Figure 1). Different from variational autoencoders [6], the Helmholtz Machine has separated recognition and generation processes, which are parameterized by φ and θ respectively, and are updated alternatively by the wake-sleep algorithm. The system is optimized under the minimization of free energy F, a variational term that measures the KL-divergence between the recognition and generative models.

$$F_{\phi, \theta} = D_{KL}[Q_{\phi}(\alpha|d) || P_{\theta}(\alpha, d)] \qquad (1)$$

The wake-sleep algorithm[5] disentangles the recognition parameters φ and the generative parameters θ by separating the training into two phases. In the wake phase, the bottom-up recognition process is performed using the current weights φ to get an instance of complete neuron assignments α by sampling activities from the binary value based on the probability of each neuron $q_n^m$. The generative parameters θ are updated based on the neuron activities encoded in α, which makes the target values available for training the hidden units locally, thus disentangling them from the multi-layer coupling. In the sleep phase, the recognition weights φ are turned off and a random instance is generated based on the current top-down weights θ. Alternatively, we update φ according to the neuron assignments of this generated instance while keeping the generative weights fixed. By iterating the wake and sleep phases, updating φ and θ alternatively, the objective function $F_{\phi,\theta}$ is minimized in an EM manner.

By adopting the wake-sleep algorithm and calculating the derivatives, we get the updating rule, which is called the local delta rule,

$$\frac{\partial F}{\partial \theta_{k,n}^{m+1,m}} = -s_k^{m+1}(s_n^m - p_n^m) \qquad (2)$$

$$\frac{\partial F}{\partial \phi_{k,n}^{m-1,m}} = -s_k^{m-1}(s_n^m - q_n^m) \qquad (3)$$

The parameters φ and θ are trained under the gradient descent as other neural networks. For a detailed analysis of the Helmholtz Machine, please refer to [?].

## 2 Helmholtz Drum Machine

We try to build a simple drum machine out of the Helmholtz machine. Since we have binary outputs of fixed-length patterns, it's natural to use them as the onset/offset (1/0) for drum hits.

In this implementation, we use the same structure as Fig.1. Since a drum machine usually presents a combined soundscape of kick, snare, hi-hat, etc., one generated pattern is insufficient to realize this multi-layered sound structure. There are several ways to cope with this. The first is to divide the 12-length pattern into groups of 3 or 4, and each sub-pattern corresponds to one instrument. The drawback is that the time for playing each sample is drastically shortened, which results in less coherence in generating through time, and hinders user selection in the subsequent applications due to short response time.

The easiest way, without compromise or modification, is to use independent samples as beat patterns for each instrument, respectively. The generation of the Helmholtz machine gives a distribution over all pattern combinations, so 6 independent samples (we specified 6 instruments for the drum machine) will give distinct patterns for each instrument if the distribution is not concentrated.

In this way, however, we lost the interrelation of the layered instruments. Besides the horizontal time propagation, music also has a vertical layered dimension where the instruments interact with one another. In principle, there should be some underlying regulation that arranges the interaction of these layers so that they don't "beat" with each other. Intuitively, we'd love to keep the latent distribution consistent at some level that regulates the global arrangement of the beat, and then have separate controls over each instrument to make variations.

One way of doing this is to impose an instrument-specific distribution at the output layer that either multiplies with the generated probability or the pattern sampled from this probability. For example, for a regular beat with the kick at each downbeat, the kick-specific distribution would have a probability
of 1 at the 1st, 5th, and 9th positions. One thing detrimental about this approach is, for example, the probability computed by the neural network at the 5th position is 0.1, then whatever probability the instruments have, it's highly possible that all instruments will be muted at this position. In other words, the patterns of all layers tend to overlap with one another, causing an imbalance among inter-layer distributions.

However, this thought is already close to the approach we designed in the experiment. Instead of imposing a pre-defined or statistically computed instrumental distribution, we could learn it. As Fig.2 shows, we decompose the bottom layer to visualize the pre-activation $a_0$, which is the linear combination of the penultimate layer. When one sample is generated in the sleep phase, to derive different patterns for layered instruments under common latent regularities, we add a bias term to $a_0$ before the sigmoid, where the bias is instrument-specific (Fig.3).

<figure><img src="image_2.png" /><figcaption>Figure 2: The Helmholtz Machine. The same model as Fig.1. The linear combinations of the penultimate layer are explicitly depicted as $a_0$, which goes through the sigmoid activation and then is sampled at layer $d_0$.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 3: Generation of Instrument Patterns from Fixed Pre-activation. Given the pre-activation $a_0$ of the output layer from one single generation, separate instrument patterns are specified by different biases that give rise to distinct outputs subject to shared latent control.</figcaption></figure>
As Fig.3 shows, a bias vector $\mathbf{b}^i$ is added to the pre-activation, which arrives at the output $\mathbf{h}^i$ with distinct probabilities after sigmoid. The upper script $i$ is the instrument index; if we have 6 instruments, there are 6 copies of bias vectors and sampled patterns. The biases are learned as local instrument statistics.

The derivative for $\mathbf{b}^i$ is simply the local delta rule (2) with the prior layer assignment as 1, namely

$$ \frac{\partial L}{\partial \mathbf{b}^i} = \mathbf{p}^i - \mathbf{s}^i \qquad (4) $$

where $\mathbf{p}^i$ is the probability at output $h_i$, and $\mathbf{s}^i$ is the sampled binary pattern at $h_i$.

Besides bias, we also learn another instrument-specific parameter, dynamics $\mathbf{f}^i$. As a drum pattern, there are naturally strong and weak beats when playing, which roughly correspond to the dynamics. We specify a 12-note vector $\mathbf{g}^i$ as the dynamics for playing every note in the pattern by instrument $i$.

The dynamics vector $\mathbf{g}^i$ is sampled from a Gaussian distribution $\mathcal{N}(\mathbf{f}^i, \frac{1}{4})$ with mean vector $\mathbf{f}^i$ and specified deviation $\frac{1}{4}$. $\mathbf{f}^i$ is a statistic we learn from given instrumental patterns. Given a musical note with dynamic $g$, to update its corresponding Gaussian mean $f$, intuitively, we should move $f$ one step toward $g$ to narrow their distance.

To derive this, we treat $g$ as the mean for another Gaussian $\mathcal{N}(g, \sigma_2)$, where $\sigma_2$ is a constant. The minimization objective is the KL divergence between two Gaussians,

$$ J = D_{KL}[\mathcal{N}(f, \sigma_1) || \mathcal{N}(g, \sigma_2)] \qquad (5) $$

This expression can be computed analytically, where

$$ J = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (f-g)^2}{2\sigma_2^2} - \frac{1}{2} \qquad (6) $$

thus

$$ \frac{dJ}{df} = \frac{1}{\sigma_2^2}(f-g) \qquad (7) $$

The update of dynamics mean $f$ is negatively proportional to the difference $f-g$ by gradient descent, $f = f - \delta(f-g)$.

Sampled dynamics $g$ is a value between 0 and 1, $g \in [0, 1]$. The standard deviation of $\mathcal{N}(\mathbf{f}^i, \frac{1}{4})$ is chosen as $\frac{1}{4}$ so that the Gaussian covers $2\sigma$ confidence interval with mean value 0.5. When the sampled $g$ lies beyond $[0, 1]$, we hard clip it to 0 or 1, whichever is closer.

Now that we have constructed a simple mapping from the Helmholtz machine generation to a drum machine, it's time to consider synthesis. In our implementation, we use 6 instrument samples - kick, snare, closed hi-hat, open hi-hat, clap, and bass, from an audio library. The synthesis is a sequencer that places these sound samples at locations specified by the Helmholtz machine generated patterns. To avoid the mechanical, rigid sounds caused by grid quantization, we randomize the onset time to make it a little bit ahead or behind the real beat to add some musical "groove". The dynamics are multiplied on the audio samples to amplify or decrease the volume.

The fixed-length 12-note pattern corresponds to 12 beats, where each beat has a fixed length specified by a user-defined BPM, beats per minute. The BPM we chose is 160; therefore, the duration (in seconds) for each beat is

$$ 60/BPM = 0.375 \text{ s} \qquad (8) $$

Thus, the duration of a 12-beat audio is 4.5 seconds, which specifies the buffer length for streaming.

Simply speaking, real-time audio play is to read something into a fixed-length buffer and then write the samples to the stream at equal time intervals (Fig.4). The pointer usually sweeps from left to right, then jumps from the end to the beginning to continue the loop, where every sample is output at a steady pace.

The content in the buffer could be fixed (the audio writes a looped pattern), updated after each sweep (reading a sound file), or updated at every sample (delay line of a filter, e.g. Karplus-Strong algorithm). The buffer is looped until a stop is sent to the pointer. In our case, because the reading material is synthesized on the fly, this process needs to be altered a little bit.

One feature of audio synthesis is that, the sound doesn't cut off at the edge. At every beat, depending on the lengths of the instrument samples, the sound usually ends at different places. Sometimes the synthesized audio will cross the bar line and span until the next phrase, as Fig.5 shows.
<figure><img src="image_4.png" /><figcaption>Figure 4: Reading Audio Buffer. The pointer writes the samples to stream at equal time intervals based on the sampling rate.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 5: Generation Spans to the Next Pattern. The buffer affords a 12-beat audio while the synthesized generation spans over the fixed length until the beginning of the next pattern.</figcaption></figure>

## 3 Thread-based Multi-Processing

As mentioned above, we aim to design the system to stream and synthesize concurrently in real time. Inherently, the difficulty comes from writing audio samples at extremely short and equal time intervals (22050 samples per second in our case), which requires high precision. If this precise time scheduling could be carried out, the idle time among the output of samples could be used to perform other tasks like synthesis, control, and even training. These tasks are actually carried out in a linear fashion, one after another, but since they are so well scheduled and fully utilize the idle time in between, these tasks look like happening concurrently or parallelly.

Unfortunately, the author is unable to program the aforementioned delicate time scheduling from scratch. As a programming amateur, I can only solve the problem circuitously. For audio streaming, we use a library in Python called PyAudio, which provides basic audio I/O functions. Since streaming is pre-programmed, we cannot access and modify it internally, which means there is no direct way to schedule different tasks linearly.

This is where we introduce multi-processing. Multi-processing is a way to perform multiple tasks "simultaneously". On the basic level, all instructions are executed one by one, but viewed from the high-level abstraction, these tasks are implemented concurrently. In other words, in multi-processing, or more precisely, *threading* in our case, the tasks are auto-scheduled by the machine due to its internal flow (CPU usage and idle time), which interleaves instructions from different tasks so that these tasks run concurrently.

In real-time generation, synthesis and streaming take place at the same time. Because the pattern is generated 4.5s at once, the buffer should be kept invariant while streaming and be updated after each sweep. The problem is, if we finish writing out the buffer and then synthesize a new sample, update the buffer, and then stream it, there will be a delay between two patterns, which will cause incoherence in playing.

To make a smooth transition between 2 patterns, it's imperative to synthesize ahead of time, and start writing the next pattern immediately afterwards with the least operations in between. In our implementation, we used 2 buffers to stream alternatively, as shown in Fig.6. Synthesis is performed while the last pattern is streaming. The synthesized audio is added on top of the prior residual as described in Fig.5, which is filled into the other buffer (which is not played). When the current buffer finishes streaming, the pointer moves immediately to the other buffer and starts streaming seamlessly, without extra operations in between that delay the audio play.
<figure><img src="image_6.png" /><figcaption>Figure 6: Multi-processing for Real-time Synthesis. Streaming thread alternates between two buffers. The synthesis thread operates sparsely, which prepares the other buffer while the current one is playing. Control thread takes in user input that terminates the process.</figcaption></figure>

The real-time process is terminated by an external user input, which operates in another thread called control. When the stop is sent by the user, synthesis terminates immediately, while the streaming finishes when the synthesized materials are fully output with both buffers emptied.

The real-time synthesis could be viewed as generations with undetermined lengths. The generated materials make no difference whether it's generated in real-time or ahead of time. The uniqueness of every performance lies in its unreplicable reflection of the moment, including the environment, the feelings, and the audience. The adapt and fit happen in real-time, which brings much spontaneity and naturalness via a dynamic interaction evolved through time.

In this work, it's far too early to study enactivism or embodiment. In our case, to reflect the virtues of the real-time process, instead of a pre-trained model, we add another thread to train the model in real-time (Fig.7).

<figure><img src="image_7.png" /><figcaption>Figure 7: Multi-processing for Real-time Training. Added training to the real-time threading, which imposed temporal sensitivity to synthesis (synthesize as late as possible) and added user preference selection to the control thread.</figcaption></figure>

The process is modified as such: with real-time training, the model parameters are adjusted in real-time in the process. One consequence is that now the synthesis timing matters. In Fig.6, the synthesis can happen at any time within the block, even pre-generated before play, because the model parameters are fixed. However, in Fig.7, the synthesis is supposed to be performed as late as possible (right before streaming the next buffer) to utilize the latest updated parameters.

If there is no change during the process – fixed dataset trained over many epochs, the real-time training is suboptimal to the pre-trained model in that the generations at early stages of training present large discrepancies from the dataset. This brought about a new problem of real-time training. In machine learning, a generative model is always fully trained before it's used for generative purposes, so that its generation could most accurately reflect the ground truth.

In the Helmholtz machine, the generation is explicitly sampled and used as a valid instance to train the recognition weights. The generative model is sampled in real-time, which lends the model to
real-time applications. Briefly speaking, sampling is an integrated part of training for the Helmholtz
machine, which lends it to the real-time process naturally. That being said, if we train on fixed data,
the equilibrium will be reached quickly, and the temporality of the real-time process in adapting and
shaping with the environment will be buried. Therefore, we add a simple mechanism to simulate this
interaction – user preference data construction.

In the control thread, besides the stop criteria, we add the user preference to like or dislike a generated pattern. The preference selection works as active data construction in a way that increases or decreases the proportion of the selected sample, thus modifying the data distribution accordingly. One design trick is that, as the audio is streaming seamlessly, there is no boundary between two patterns that separates them, also no space for the user to respond. Usually, a user can only judge a sample near the end of it or after it finishes. The tricky thing for a real-time system is that, once a buffer finishes playing, its memory is cleaned immediately and the index switches to the next pattern immediately. In other words, without special design, if a user reacts to a sample after it's played, the preference will go to the new sample that is playing now, causing an error in constructing the desired data.

This is an immediate logic that doesn't imply any intellectual complexity. However, it requires a level of circumspection to spot it and arrange the events in real-time. In our implementation, we shift the control thread half buffer behind the real time, so that the window for marking a sample covers the latter half of it and the first half of the next buffer while the new sample is playing. We spare enough time for the user to respond after the sample is played but we also impose a limit so that the attention is shifted to the next sample immediately.

Fig.8 records the flow of threading in Fig.7. One glitch is that we named the synthesis thread "control" when programming, so the control thread here actually corresponds to the synthesis. The decimal numbers in the flow indicate the time elapsed since the last operation. The time interval between "write to buffer" and "play buffer" is minimized so that the latest updated weights can be used for synthesis. One thing to point out is the user-selected index. The first sample, index [875], is written to buffer 2. The user disliked this sample in the latter half of its playing, and the message shows that sample [875] is disliked before buffer 1 starts playing. Sample [374] is written to buffer 1 and being played. However, the user didn't react until the next sample [3845] started playing. The user liked it and we see that the liked index is still [374] instead of the new one [3845]. Getting the time precisely arranged is crucial in real-time programming, especially under multi-thread processing.

<figure><img src="image_8.png" /><figcaption>Figure 8: Flow of Threading. Program Designed as Fig.7.</figcaption></figure>

In summary, Fig.7 presents a multi-processing real-time system with the following structure:

**Thread 1: Streaming.** Real-time audio streaming under PyAudio's audio I/O system. Two alternative buffers to write synthesized audio.
**Thread 2: Synthesis.** Sample from the Helmholtz machine to synthesize the next drum audio. A bywork is to update the progress bar.

**Thread 3: User control.** The user has 3 types of control: like (1 or l), dislike (2 or d), and stop (0 or s). The timing of liking/disliking a sample is shifted backwards by half of a buffer length. Stop is to terminate the real-time streaming.

**Thread 4: Training.** Training of the neural network with real-time user-constructed datasets by active sampling. A bywork is to plot the data and the generative distribution after several epochs.

## 4 Action-Perception Loop on Data Construction

The real-time training based on user preference is a typical HCI (Human-Computer Interaction) system. Although this model looks like a user feedback-based recommendation system, the original design idea wasn't from these applications.

Based on the structure of the Helmholtz machine, it produces samples in real-time that can be synthesized as drum patterns as described above. However, it's difficult to collect or construct a dataset of valid drum patterns for the model to learn, so we have to resort to other methods.

One of the original ideas is to build a self-organizing system that, by sampling from its own generation, the model converges to a self-constructed dataset where its equilibrium stands. This is a feasible way to find the dataset that the model learns best, but it doesn't encode any information to make the generation musically satisfying. The idea is that, if we can have some external force guiding the model toward better auditory practices, such as the discriminators in GAN, or hard-coded rules in reinforcement learning, the model can be shaped from the outside based on the feedback instead of constructing data by its internal regularities.

This idea leads to the user-feedback real-time training in Fig.7, where the generative system is the Helmholtz machine, whose data samples are regulated by user preferences. Interestingly, in [7] we simulated the agent's brain using the Helmholtz machine. If we treat the user's decision-making process as an implicit Helmholtz machine-like brain activity, we derive a communication diagram in Fig.9.

<figure><img src="image_9.png" /><figcaption>Figure 9: Communication between two Helmholtz Machines via Action-Perception Loop. The Helmholtz machines can be artificial (simulated by computer) or neuronal (hierarchical cortical responses).</figcaption></figure>

Fig.9 depicts the communicative flow between the agent and the Helmholtz generator via an action-perception loop. This is another way to describe the dynamics in Fig.7. The generated drum audio is perceived by the agent as auditory input. After processing this signal in the brain, a valence decision
<figure><img src="image_10.png" /><figcaption>Figure 10: Active Data Construction via Real-time Training</figcaption></figure>

is made on the sample – positive, neutral, or negative. Then the agent acts on the external system by labeling the auditory sample as like or dislike (no action if neutral), which exerts influence on the music generator by modifying its ground truth, +1 for liked samples and -1 for disliked samples. By actively constructing the data, the Helmholtz generator is driven toward generations conforming to the agent's expectations.

This interpretation naturally biases toward the generating system, the model under training, and the subject under investigation. The opposite situation happens when the agent or the brain model is the focus of the study (eg, TTOM in [8]). As widely discussed in [2, 4, 8], the key move is to create a symmetry between internal and external states under a reciprocal exchange between the agent and its environment. Namely, for two bodies sharing the same Markov blanket to communicate, they must be *inferring each other*. The reciprocal loop of communicative exchanges depicted in Fig.9 can be decomposed as below:

* User's action (explicit): active niche construction. Modified data distribution by user preference selection.
* Machine's perception (explicit): training of the Helmholtz machine. Optimize model representations to approximate the data distribution.

* Machine's action (implicit): generated samples of the Helmholtz machine. The outputs screened by the user potentially act on the user's internal models.

* User's perception (implicit): the user's perception or expectation is subject to machine generations. User's ears might get used to the surprising patterns after long enough time.

As the two processes described in FEP, the agent not only samples from the environment based on its current representations, but its representations are also under dynamic construction in approximating the external states. Like Wave Three described in [3], the user's generative model will also begin to predict the artifact's behavior. Although the brain dynamics of a real agent remain opaque, we can imagine that after a long time running the reciprocal process in Fig.9, the agent's ear will also adapt to the machine-generated patterns, no matter how unused it was to these sounds. The valence judgments are also subjective and are subject to even unconscious changes under the mutual adapting processes.

Different from the user-preference recommendation systems, the real-time learning and generating under the Helmholtz machine happen in a variational manner. The user preference is recorded in the dataset, which is used to train the model, but the result may not be obvious. There could be a discrepancy between the data distribution and the generative distribution, and this discrepancy could be large when the training is conducted dynamically in real-time, or the model couldn't fit the dataset well due to systematic errors. According to the law of large numbers, the generative distribution is well approximated only when there's a large corpus of samples. Therefore, the interaction depicted in Fig.9 is supposed to take place over a long and slow time scale, and the run-in period between two variational models could take a substantial amount of time before any regular pattern starts to emerge.

We visualized the beginning stage of active data construction and real-time training of our model in Fig.10. The data distribution (in blue) is modified by the user preferences, and the generative distribution (in orange) reflects its real-time adaptation to the dynamic data. The user preferences are partially decoded in the generations, but the generative sampling is not a faithful reflector of the current ground truth.

## References

[1] J. Bruineberg, E. Rietveld, T. Parr, L. van Maanen, and K. J. Friston. Free-energy minimization in joint agent-environment systems: A niche construction perspective. *Journal of theoretical biology*, 455:161-178, 2018.

[2] A. Constant, M. J. Ramstead, S. P. Veissiere, J. O. Campbell, and K. J. Friston. A variational approach to niche construction. *Journal of the Royal Society Interface*, 15(141):20170685, 2018.

[3] D. Demekas, T. Parr, and K. J. Friston. An investigation of the free energy principle for emotion recognition. *Frontiers in Computational Neuroscience*, 14:30, 2020.

[4] K. J. Friston, T. Parr, Y. Yufik, N. Sajid, C. J. Price, and E. Holmes. Generative models, linguistic communication and active inference. *Neuroscience & Biobehavioral Reviews*, 118:42-64, 2020.

[5] G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The "wake-sleep" algorithm for unsupervised neural networks. *Science*, 268(5214):1158-1161, 1995.

[6] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. *arXiv e-prints*, page arXiv:1312.6114, Dec. 2013.

[7] J. Liu. A new theory of data processing: Applying artificial intelligence to cognition and humanity. In *Proceedings of the 25th International Conference on Multimodal Interaction*, pages 679-683, 2023.

[8] S. P. L. Veissière, A. Constant, M. J. D. Ramstead, K. J. Friston, and L. J. Kirmayer. Thinking through other minds: A variational approach to cognition and culture. *Behavioral and Brain Sciences*, 43:e90, 2020.