# Haptic Carillon: A Computationally Enhanced Mechanical Performing Instrument

Mark Havryliv, Fazel Naghdy, Greg Schiemer, Timothy Hurd

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mark Havryliv, Fazel Naghdy, Greg Schiemer, Timothy Hurd. Haptic Carillon: A Computationally Enhanced Mechanical Performing Instrument. Second IFIP TC 14 Entertainment Computing Symposium (ECS) / Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.75-86, 10.1007/978-3-642-15214-6_8. hal-01056330

HAL Id: hal-01056330

https://inria.hal.science/hal-01056330

Submitted on 18 Aug 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Haptic Carillon: a Computationally Enhanced Mechanical Performing Instrument

Mark Havryliv<fnref n="1" />, Fazel Naghdy<fnref n="1" />, Greg Schiemer<fnref n="2" /> and Timothy Hurd<fnref n="3" />

**Abstract.** This paper describes the development of a haptic device for emulating the sonic and haptic dynamics of a carillon, specifically the National Carillon in Canberra, Australia. The carillon is one of only a few instruments that elicit a sophisticated haptic response from the amateur and professional player alike. Force-feedback varies widely across the range of the instrument and developing an intuition for the heaviness of different bells is a critical part of carillon pedagogy. Unfortunately, rehearsal time available to individual carillonneurs is limited by competition from other carillonneurs and environmental factors like civic noise limits and carillon maintenance schedules. Rehearsal instruments do exist but they do not accurately display the haptic dynamics of the real carillon. Our device couples the notions of *entertainment* and *cultural* computing; while musical instruments are now regularly digitised for purposes of entertainment the haptic carillon is motivated by an awareness of the musicianship of carillonneurs and the public cultural space they inhabit with their instrument.

## 1 Introduction

The carillon is perhaps the most public of musical instruments. The development of its repertoire and sense of musicianship is entwined with the culture and politics of statehood and religion in Western Europe for a significant part of the past five hundred years. Even today individual carillons are dependant on civic, ecumenical or private patronage — or combinations thereof — and rehearsal and musical opportunities for their carillonneurs are constrained by a combination of competition from other performers and the resources of host bodies. Although practice instruments exist they share a critical shortcoming: an inability to replicate the *haptic* sensation of carillon performance, force-feedback felt by a carillonneur that varies across the range of an individual instrument and with further idiosyncratic variation from one instrument to another.

These variations range from the number of bells in a carillon, the size of the bells and their corresponding clappers, and the dimensions and positioning of the keyboard and pedals. The mechanism, or action, for each bell in a single carillon is subtly and not-so-subtly different, and the general design and construction of the carillon varies from one to the next. The magnitude of these variations

<footnote n="1">¹ Faculty of Informatics, and</footnote>

<footnote n="2">² Sonic Arts Research Network, University of Wollongong, Australia</footnote>

<footnote n="3">³ Olympic Carillon, Seattle.</footnote>
is compounded by factors like the regularity with which a carillon is serviced and the extent to which it is exposed to the elements. This is entirely natural for an instrument that has been in development for over five-hundred years and the specifications for which have only very recently approached any level of uniformity [12].<fnref n="1" />

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** (a) The National Carillon, Canberra, Australia is located on Aspen Island in Lake Burley Griffin (b) Simplified illustration of the carillon mechanism (c) Bell 54 (d) Bell 4.</figcaption></figure>

The application of synthesised force-feedback based on an analysis of forces operating in a typical carillon mechanism offers a blueprint for the design of an electronic practice clavier and with it the solution to a problem that has vexed carillonneurs for centuries, namely the inability to rehearse repertoire in private. The need for carillonneurs to develop musicianship and extend the instrument's repertoire offers a compelling musical reason to build a haptic practice instrument. Unlike other traditional instruments, the carillon always has an audience,

<footnote n="1">¹ And even that uniformity concerns only the dimensions of the keyboard, saying nothing of forces and/or dynamics.</footnote>
willing or unwilling, even if the carillonneur is only trying to practice. The professional practise of concert carillonneurs provides another motivation: that one might prepare for a performance at a foreign carillon well ahead of actually having the chance to play it.

Measurements taken at the National Carillon in Canberra, shown in Figure 1(a), lay the groundwork and verification methodology for the design and construction of our haptic carillon prototype baton. As a prelude to a discussion on the role of haptics in musical instruments observe from Figure 1(b) that the carillonneur interacts with the instrument through a wooden baton which is linked to a crank that rotates and pulls a clapper against the inside of the bell wall. The mass of this clapper can be as great as 2 tonne and small as 600 gram, however the influence of this mass on the force felt at the baton is usually mediated by forward or return springs that respectively assist or resist the carillonneur's stroke. Figure 2(a) shows the range of forces felt at the baton tip across the National Carillon, along with the changing clapper stem lengths.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** (a) Force required to stabilise the baton over the carillon range (b) Measured and interpolated clapper lengths (c) Torque versus clapper lengths.</figcaption></figure>

This paper will focus primarily on the haptic aspects of simulating the carillon; the acoustics of carillons/bells are generally well-understood and regularly modelled. Fletcher and Rossing [5] is an excellent resource for understanding the physics of bell sounds while Karjalainen et al. [11] and Hibbert [10] discuss modern synthesis and modelling techniques.
# 2 Haptics and Musical Instruments

This project builds on advances in haptic engineering and design that permits the accurate display of the sensation of force-feedback. Haptically rendered instruments are designed to remove a major flaw in otherwise useful practise instruments; namely, the absence of an authentic sense of touch, or *feel*, which accompanies almost any instrumental interaction [16]. Feel can be simply defined as force felt by a player at the point of contact with an instrument; a brass player feels the vibration of his lips [19], a violinist experiences vibration and resistance at the point where the bow is held and where the string is depressed [1], a pianist feels different levels of resistance at the key [7] [8] and so on.

## 2.1 Musical Skill Acquisition

Together, the acoustic and haptic feedback from a musical instrument play a crucial part in the early stages of familiarisation with an instrument. O'Modhrain extends motor skill acquisition theory to develop a model of musical skill acquisition [16] in which she argues that at the higher levels of motor skill acquisition and musical performance proficiency the musician ceases to be actively (in real-time) concerned with feedback from individual gestures at individual keys; instead, the musician relies on an intuition that anticipates the instrument's state at a particular time.

This intuitive anticipation is developed over the course of much repetitive practise and is key to expert performance because it allows the execution of sequences of gestures, or motor patterns, at a frequency that exceeds the reaction time (RT) of the human motor system. This RT, generally determined to be between 120–180 mSec, or less than 10 Hz [17], is the upper limit for human motor system to work in a closed loop control; that is, in order for a gesture scheduled for time $t_{later}$ to adapt or respond to some acoustical or mechanical feedback, that feedback must occur at a time ($t_{later} - 180$)mSec or earlier.

A corollary to the intuitive anticipation described above — greatly relevant to the instrument builder — is that an expert performance becomes unstable when learned interaction constraints are violated; this is the case whether a musician is performing on a traditional instrument, an electro-mechanical replica of a traditional instrument or a novel instrument, electronic or otherwise.

The principle of lawfully relating a performer's gesture to a consequent instrument state that is received in some form of sensory feedback, whether it be acoustic, visual, haptic or otherwise takes a second important, if abstracted, form when considering the haptic carillon. One of the principle aims of the project is to prepare a carillonneur for performance on a foreign instrument; in order to do so, a carillonneur must have confidence in the haptic carillon's ability to replicate the dynamics of a foreign carillon and this confidence will be eroded through experience over time if a set of intuitions developed in practice do not match foreign realities. This would undermine not only a single performance but the haptic carillon's artistic *raison d'être*. Carillonneurs also have historical
reasons to be sceptical of technological ‘advancements’, to wit: “Finally there is
the computer, but it will also be defeated” [13, p. 305].

It is possible to predict and recreate (at least, theoretically) forces felt by a
player at the point of contact with an instrument by analysing its mechanical
properties. This requires an understanding of how the mechanical components
of a particular instrument interact, or a kinematic analysis, prior to considering
the effects of user input.

## 2.2 Physical Modelling

This type of analysis is already being used to develop physically modelled syn-
thesis algorithms; the interactions between physical components that contribute
to sound production are expressed as equations that model an instrument's re-
sponse to an excitation from a player. A physical model for the synthesis of a
violin, for example, will consider the interaction of the bow against the strings,
the width of the bow, the damping and resonance of the string, the transfer of
energy through the bridge, and the resonance of the soundboard [6] [18].

A kinematic analysis with a view to haptically rendering an instrument,
however, looks at the interactions between physical components that contribute
to the force felt by the user. The kinematic constraints of a brass instrument
include, for example, the width and depth of the mouthpiece, the length of
the tube, and the type of metal used. In haptically rendered instruments such
as the Nichol's vBow [14], Gillespie's Touchback Keyboard [8], Oboe's Multi-
Instrument Keyboard [15] and Berdahl's Haptic Drumstick [3] physical models
in the form of differential equations compute the behaviour of their respective
traditional instrument's mechanical systems under different excitations, or ges-
tures, performed by a player.

Using sensors (force, displacement sensors etc.) to monitor a player's ges-
tures, the computational models are able to determine what a player might
expect to feel in response, and then actuate this response using a motor. The
means of generating haptic response need not be based on a physical model —
Beamish's Haptic Turntable [2] uses a combination of simple linear analysis and
user-programmable look-up tables for the display of assistive haptic feedback for
DJs.

A thorough taxonomy of new instruments which are either originally con-
ceived or emulate traditional instruments can be found in the author's previous
work [9], and O'Modhrain [16] and Berdahl [4] present useful discussions and a
framework for evaluating the effectiveness of haptic interaction in novel musical
instruments.

# 3 The Carillon

Existing practise carillons make little effort to replicate the force-feedback of
a real carillon. A typical practice instrument employs extension springs to em-
ulate the resistance felt by the carillonneur as the baton is displaced from its
détente position downward. As well as providing an inaccurate force-feedback this arrangement is unable to simulate the dynamic behaviour of a baton in free motion. These limitations speak to the real purpose behind practice carillons: to familiarise a carillonnuer with the *geometry* of a particular carillon keyboard. Practice carillons are almost always associated with a particular carillon and help a performer rehearse for the dimensions of that carillon; to this end a new practice carillon has been built by Timothy Hurd that permits the manual adjustment of the batons and pedals in the x/y/z axes (Figure 3). This instrument, though, is still limited to the linear force-feedback of extension springs. Upon

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Chamber carillon with adjustable keyboard and pedal dimensions.</figcaption></figure>

analysis of the carillon mechanism it is clear why a simple extension spring is unable to accurately simulate the force-feedback of the carillon. An immediate limitation of the spring is its lack of memory over time; Hooke's law for the restoration force of a spring

$$F(x,t) = -k(x(t) - x_0) \qquad (1)$$

is simply the linear relationship between the spring's force constant $k$ and displacement from its equilibrium at any time, $(x(t) - x_0)$. It has no capacity to 'remember' previous values of $x$ as a function of time $t$.<fnref n="2" /> However, not only is force-feedback in the carillon non-linear with respect to position but it is also dependent on the baton's velocity.

## 3.1 Non-linear Force Response

The haptic sensation felt by the carillonneur at the point of interaction, the *baton*, is the result of static and dynamic forces acting on mass components of

<footnote n="2">² Oboe [15] showed this to be sufficient for modelling the force-feedback of a Hammond organ and many interesting haptic/musical applications use virtual springs exclusively or in concert.</footnote>
the carillon mechanism. Along with haptic cues, the carillonnuer observes the
baton's motion as an indicator of the mechanism's kinetic energy and adapts the
force of the next stroke accordingly. For example, a baton for a low bell stays in
periodic motion for as long as 3 seconds (see §3.3) and a carillonneur can reduce
the amount of force they need to apply for a repeated note if they strike the
baton while it is already in compliant motion.

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Force, position and velocity relationships for bell 9, (a) Baton displacement as a function of player force at baton tip, (b) Regression lines for force as a function of position for downward (blue; R² = 0.88) and upward (red; R² = 0.9) motion.</figcaption></figure>

The typical force profile for a low bell (here, bell 9) is shown in Figure 4,
where user-applied force is measured at the tip of the baton with a finger pressure
sensor and the resultant vertical position of the baton over time is recorded using
a motion sensor<fnref n="3" /> with a sampling rate of 50 Hz. After the method for capturing
the force profile of switches outlined by Weir et al. [20], a player was instructed
to depress the baton to its maximum displacement and then allow it to return
to détente while staying in contact with the baton at all times; the procedure
was repeated several times with increasing velocity.

Figure 4(a) shows that greater force is felt during a downward stroke than the return stroke; observing the rainbow spectrum as a visualisation of velocity one notes that ‘hot’ colours which indicate negative velocity (downward motion) are grouped to the right of ‘cool’ colours representing positive velocity. The adjacent Figure 4(b) combines two scatter plots grouped by the direction of baton motion (blue and red for downward and upward, respectively) showing the force felt by a player as a function of baton displacement. The regression lines show that the linear change $\beta \approx 0.8$ is very similar in both directions and there is a force

<footnote n="3">³ Xsens MTi inertial motion sensor, http://xsens.com</footnote>
offset of approximately +1 N for downward motion. This velocity- and position- dependant change in force is encapsulated in the following function:

$$F_{\text{off}}(\dot{x}, t) = \begin{cases} 1.2 & \dot{x}(t) < 0 \\ 0.2 & \dot{x}(t) > 0 \end{cases} \\ F(x, \dot{x}, t) = \beta x(t) + F_{\text{off}}(\dot{x}, t). \qquad (2)$$

Simple velocity-dependant linear regression, however, is clearly far from an ideal representation of the relationship between user input and carillon force-output. Further, the carillonneur's experience is not constituted of dynamic force interactions alone: the carillon is unique amongst musical instruments in that its dynamic state is strongly time-dependent and the connection between performer gesture and acoustic result is mechanically and temporally decoupled.

## 3.2 Carillon Mechanics & Dynamics

The plots in §3.3 demonstrate the persistent influence of the carillon's internal dynamics on the baton's state at any time; those particular bells also display the extent of dynamical variation even between nearby bells, like 4 and 7.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Three rotational systems linked by cables (a) Carillon system with lengths and angles labelled (b) Centres of mass and tension forces.</figcaption></figure>

For the purposes of dynamic modelling the carillon mechanism is considered as three rotating sub-systems: the clapper, the crank and the baton systems. The three systems are linked by thin wound cables and forces exchanged between the systems are encapsulated in the tensions of the two cables. From Figure 5 we see the systems are linked in series, with a cable from the mid-length of the baton joined to the tip of the straight crank rod, and a cable from the tip of the L-shaped crank rod joined to the tip of the lower clapper rod.
The linkage cables impose velocity constraints on the motion of the three sub-systems: the two points at either end of a single cable exhibit uniform velocity, and therefore acceleration. The rotational accelerations of the three sub-systems are therefore constrained:

$$ \ddot{\theta}_1 l_1 = \ddot{\theta}_2 l_{2C} \qquad (3a) $$

$$ \ddot{\theta}_3 l_3 = \ddot{\theta}_2 l_{2B} \qquad (3b) $$

If we include the tension forces from the linkage cables along with the effective torques $\tau_i$ and moments of inertia $I_i$ about the pivot point for each of the rotational systems, we derive equations for the respective rotational accelerations:

$$ \ddot{\theta}_1 = \frac{\tau_1 + T_C l_1}{I_1} \qquad (4a) $$

$$ \ddot{\theta}_2 = \frac{\tau_2 + T_C l_{2C} - T_B l_{2B}}{I_2} \qquad (4b) $$

$$ \ddot{\theta}_3 = \frac{\tau_3 + T_B l_3}{I_3} \qquad (4c) $$

where the torques are

$$ \tau_1(t) = gl_{C1}m_1 \cos \theta_1(t) + f_{\text{ext}}(\theta_1, \dot{\theta}_1, t) \qquad (5a) $$

$$ \tau_2(t) = gl_{C2}m_2 \cos \theta_2(t) \qquad (5b) $$

$$ \tau_3(t) = gl_{C3}m_3 \cos \theta_3(t). \qquad (5c) $$

The function $f_{\text{ext}}(\theta_1, \dot{\theta}_1, t)$ in (5a) represents the external forces acting on the clapper system: impact with the bell, any forward or return springs, and impact with a rubber damper that prevents the clapper from swinging away from the bell past a certain threshold.

**Impact Constraints** The carillon mechanism is constrained to rotate over only a small range; at the baton this is between an upper and lower felt-lined row of wooden beams, and at the clapper this is between the inside of the bell shoulder and approximately 3 cm away from the bell. The latter constraint is implemented by means of a fixed rubber damper mounted below the forward spring which is linked to the lower end of the clapper (the top cable in Figure 1(d)). The baton and clapper constraints are related such that the baton's lower beam stops the baton bringing the clapper into contact with the bell — like in a piano, the clapper is in free-flight, or 'let-off' just prior to impact with the resonating surface. Ideally, the upper beam is positioned to coincide with the impact of the rubber damper.

These relations are determined by the length of the cable linking the baton and crank; a turnbuckle attached to this cable allows non-invasive length adjustments and over the course of a single performance it is not unlikely for the
carillonneur to do so. In our simulation, we assume that the impact constraints for the baton and clapper coincide perfectly and all position constraints are en-capsulated as impact forces applied directly to the clapper in the $f_{ext}$ function in (5a).

## 3.3 Simulation

<figure><img src="image_8.png" /><figcaption>**Fig. 6.** Bell simulation responses (a) Bell 28 free motion (b) Bell 4 free motion (c) Bell 7 free motion (d) Bell 48 free motion.</figcaption></figure>

We now reduce the three equations (4a) – (4c) into just one by finding a solution for either $T_C$ or $T_B$; if we solve for $T_B$ and eliminate $T_C$, for instance, we can then solve (4c) and apply the relationships in (3) to determine accelerations for $\theta_2$ and $\theta_1$.

In the model presented above, the number of parameters that change over the range of the carillon is rather low; indeed, using principally the measured and estimated values for clapper length shown in Figure 2(c) we are also able to estimate a changing clapper mass and its rotational range. The baton mechanism is the same across the entire carillon, while the crank mechanism is slightly smaller for the upper twenty bells.

The remaining variables to be estimated are damping and springiness coefficients for the clapper impacts against the wall and rubber damper; variation in these values is responsible for the difference in ‘bounciness’ shown in the figures above as the baton returns to détente position.
## 3.4 Prototype

<figure><img src="image_9.png" /><figcaption>Fig. 7. Haptic prototype of single baton.</figcaption></figure>

The mathematical model presented here is arranged such that it can be solved in real time using forward dynamics, the system's motion in response to forces. It is programmed in Simulink and compiled to run on a standalone target PC which connects to an electromagnetic linear actuator through a dedicated analogue I/O board. This actuator (Figure 7) controls the position of the baton, and back-EMF at the actuator windings is measured in order to close the feedback loop by determining the force applied by the player. The problem of generating appropriate sound synthesis is somewhat mitigated in this environment. Typically, a carillonneur only hears their instrument through loudspeakers amplifying the signal picked up by strategically-positioned microphones in the bell tower. The National Carillon, for example, provides only this type of aural feedback to the performer.

## 4 Summary and Future Work

This paper has demonstrated analytical techniques that permit the accurate simulation of batons of varying force-feedback and dynamic behaviour across the range of the National Carillon. Future user-testing will build on current haptic research to assess the nature of a performers perception of a traditional instrument against this haptically rendered one.

The construction of an electro-mechanical version of the carillon has ramifications beyond simply helping carillonneurs rehearse. A potential source of musical innovation lies in the construction of a 'remote' carillon that would allow a carillonneur to perform from outside the bell tower, and hear themselves as they are heard by their audience. The introduction of haptics into an otherwise normally functioning instrument also opens the door to novel ensemble performance opportunities for carillonneurs; haptic detents could be built into the performance of a musical work that helps the carillonneur stay in time, or receive cues from performers without them necessarily being in the same room as is required now.
References

1. Askenfelt, A., Jansson, E.V.: On vibration sensation and finger touch in stringed instrument playing. Music Perception 9(3), 311–350 (1992)

2. Beamish, T., Maclean, K., Fels, S.: Designing the haptic turntable for musical control. In: Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. pp. 327–336. Vienna, Austria (2003)

3. Berdahl, E., Verplank, B., Smith, J.O., Niemeyer, G.: A physically-intuitive haptic drumstick. In: Proceedings of the International Computer Music Conference (August 2007)

4. Berdahl, E., Niemeyer, G., Smith, J.O.: Using haptics to assist performers in making gestures to a musical instrument. In: Proceedings of New Interfaces for Musical Expression (2009)

5. Fletcher, N., Rossing, T.: The Physics of Musical Instruments. New York: Springer (1998)

6. Florens, J., Henry, C.: Bowed string synthesis with force feedback gesture interaction. In: Proceedings of the International Computer Music Conference (2001)

7. Furuya, S., Kinoshita, H.: Organization of the upper limb movement for piano key-depression differs between expert pianists and novice players. Journal of Experimental Brain Research 185, 581–593 (2008)

8. Gillespie, B.: Haptic Display of Systems with Changing Kinematic Constraints: The Virtual Piano Action. Ph.D. thesis, Stanford University (1996)

9. Havryliv, M., Schiemer, G., Naghdy, F.: Haptic carillon: Sensing and control in musical instruments. In: Proceedings of the Australasian Computer Music Conference (2006)

10. Hibbert, W.A.: The Quantification of Strike Pitch and Pitch Shifts in Church Bells. Ph.D. thesis, The Open University (2008)

11. Karjalainen, M., Esquef, P.A.A., Vlimki, V.: Making of a computer carillon. In: Proceedings of the Stockholm Music Acoustics Conference (SMAC'03) (2003)

12. Keyboard Committee, Word Carillon Federation: Consensus on Technical Norms for a World Standard Carillon Keyboard (2006)

13. Lehr, A.: The Art of the Carillon in the Low Countries. Tielt (Belgium): Lannoo (1991)

14. Nichols, C.: The vbow: A virtual violin bow controller for mapping gesture to synthesis with haptic feedback. Organised Sound Journal 7, 215–220 (2002)

15. Oboe, R.: A multi-instrument, force-feedback keyboard. Computer Music Journal 30(3), 38–52 (2006)

16. O'Modhrain, S.: Playing by Feel: Incorporating Haptic Feedback into Computer-Based Musical Instruments. Ph.D. thesis, Stanford University (2000)

17. Schmidt, R.A., Lee, T.D.: Motor Control and Learning: A Behavioral Emphasis - 4th Edition. Human Kinetics Publishers (2005)

18. Serafin, S., Vergez, C., Rodet, X.: Friciton and application to real-time physical modeling of a violin. In: Proceedings of International Computer Music Conference (1999)

19. Vergez, C., Rodet, X.: Model of the trumpet functioning: Real-time simulation and experiments with an artificial mouth model. In: Proceedings of the International Symposium on Musical Acoustics (1997)

20. Weir, D.W., Peshkin, M., Colgate, J.E., Buttolo, P., Rankin, J., Johnston, M.: The haptic profile: Capturing the feel of switches. In: IEEE Processings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. pp. 186–193 (March 2004)