Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1698059535401.pdf (Liberia)
AGENTIAL SCORES: EXPLORING EMERGENT, SELF-ORGANISING AND ENTANGLED MUSIC NOTATION Jack Armitage Intelligent Instruments Lab Reykjavik, Iceland jack@lhi.is ABSTRACT Dynamic scores have gained popularity as an innovative intervention in musical performance, providing novelty for both performers and audiences.

In this paper, we discuss agential scores and the implications of their emergent, self- organising, and entanglement affordances for musical per- formance ecologies.
We approach this through practice- based research, introducing T¨olvera, an artificial life soft- ware library for agential scores.

We propose a typology of interaction scenarios for agential scores and investigate a subset of these, presenting the outcomes of early artis- tic encounters with T¨olvera between two improvising gui- tarists.
Reflecting on our experiences, we emphasise the unique challenges that emerge from engaging with scores as real-time agents, suggesting that agential scores pro- mote fluidity of form in notation, which consequently prompts performers to identify with, mirror, and attune to them.
Although scores have always possessed agency, we argue that a more explicit and practical focus on agency raises new questions and offers new possibilities for the intra- actions between human and non-human agents within mu- sical ecologies.

1.

INTRODUCTION Musical scores serve as pivotal components within the mu- sical cultures in which they are utilised, influencing the thoughts and actions of composers, performers, concert or- ganisers, audiences, and music as a whole.
This impact can be characterised in terms of material agency.

Histor- ically, the vigour of this agency has been limited by dis- play and reproduction technologies, such as print media.

However, computational multimedia scores are now truly coming to life through innovative mediums encompassing biology, artificial life simulation, and artificial intelligence.
In the mediaeval era, music notation evolved into standards that represented principles in instrument making, musical training, and performance contexts.

However, notation ex- perienced significant diversification beginning in the twen- tieth century.

Under prescriptive notation, new developments and sym- bols have been added to the traditional Western notational Copyright: © 2023 Author Name.
This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 Unported License, which per- mits unrestricted use, distribution, and reproduction in any medium, provided the Thor Magnusson Intelligent Instruments Lab Reykjavik, Iceland thor.magnusson@lhi.is system.
These include graphic scores, enhanced tablatures, verbal notation [1], event scores [2], generative software and live coding [3], instruction pieces [4], realisation scores [5], soundpainting [6], S-notation for DJs [7], and ani- mated notation [8].

Additionally, tangible [9] and hap- tic [10] scores, artificial life scores [11], and more have emerged.
Generative and dynamic scores have been extensively in- vestigated, and the role of computational technologies is crucial in their implementation: computers enable us to achieve truly “dynamic media” [12] that were impossible with acoustic or purely analogue electronic technologies.

These scores exhibit such diversity that their own becomes fluid and interchangeable with that of the composer, in- strument or audience.
Such systems span from being open to interpretation, where performers interpret as they wish, to more rigid systems where performers precisely execute what the system generates, for instance, by playing gener- ated notes displayed in staff notation.

In our work involving dynamic and agential scores, we endeavour to utilise the emergent and self-organising prop- erties of artificial life as a notational system for compo- sition and performance.
This is part of our wider effort to develop a library of technical components for compos- ing musical systems, encompassing real-time MIDI mod- els [13], feedback systems [14], sensors, actuators, and more, which we refer to as the Organolib 1 .

The Organolib does not target a specific combination of composer, inter- preter, instrument, and audience; instead, it aims to pro- mote an ecological mindset regarding the distribution of agencies among technical elements and musical roles.
Ul- timately, we strive to facilitate the emergence of epiphe- nomenal structures that could be characterised as musical agency.

This paper presents our preliminary investigations into In contrast to the agential scores through artificial life.

recent trends in artificial intelligence that favour machine learning approaches, both in industry and the arts, artifi- cial life remains a relatively marginalised field, albeit one that may be experiencing a resurgence in interest.
For us, artificial life serves as an almost literal Petri dish for prac- tical exploration into musical agency [15], providing a di- verse array of systems and species that inevitably engage with topics such as biology, evolution, ecology, philoso- phy, computation, and more.

original author and source are credited.

1 https://iil.is/research/organolib 2.

BACKGROUND 2.1 Perspectives on Agency The concept of agency frequently arises in the realm of new musical instrument design.
These instruments are per- ceived to possess a certain degree of agency, influencing how we think, play, and behave within a specific perfor- mance ecology [16].

However, there is no general consen- sus on the meaning of agency, and various theoretical fields offer different perspectives.
In this context, we can contrast analytical philosophy, which views agency as a property of an intelligent being, typically the intentional action of an ethical human being [17], with theoretical biologists who regard agency as a characteristic of living organisms, as exemplified in the theory of autopoiesis [18].

Additionally, sociologists and philosophers of technology consider agency as a property of matter and objects [19], or of their intra-action [20].
Frauenberger refers to the convergence of these diverse viewpoints as “entanglement HCI” [21], asserting that “it is through non-human agency that we can create nuanced links between design intent, context of use, and people intra-acting with technology.” In response to this notion, Nordmoen and McPherson describe a “decentring from hu- man to more-than-human” approach in their practical ex- plorations of interactive systems, acknowledging the “ecolo- gies of different types of knowledge and raw materials” in- volved [22].
The autopoietic theory by Maturana and Varela [18] char- acterises living organisms not as physical entities, but rather as networks of processes [23].

Some have advocated for a “more embodied reformulation of autopoiesis” to address the perceived overly disembodied nature of the autopoi- etic theory’s original formulation [24].

This argument for embodiment resonates with Sarah Kember’s cyberfeminist critique of ALife, which challenges earlier conceptions of life as information [25].
2.2 Exploring Agency through Boundary Objects In the realm of musical instrument theory, agency is char- acterised as a relational concept [26].

Although sociologist Latour finds it reasonable to attribute agency to a specific door-locking mechanism [27] (thing-agency), such a no- tion may not be appealing to a biologist (bio-agency).
As we engage with intelligent instruments, we are not only confident in our recognition of agency within the technolo- gies we create but also actively investigating the bound- aries surrounding the term.

This is not a straightforward task, particularly as the concept of agency is rapidly evolv- ing within the context of contemporary AI advancements.
In this regard, we perceive our agential scoring system, delineated in the subsequent section, as a boundary ob- ject [28] for examining agency from a multitude of diverse perspectives.

Stapleton and Davis [29] discuss agency as distributed and relational, occurring in the interaction be- tween the performer and the instrument.
We concur with this ontology; however, in our work with artificial life and machine learning, it is also intriguing to investigate how we and the users of our instruments engage with these terms when projecting our ideas onto the dynamic behaviour of our systems [14].

Our research programme does not aim to rigidly define these terms from the outset; instead, we endeavour to observe how they are applied in the context of use by composers, performers, and audience members.

3.
3.

AGENTIAL SCORES Taking into account the perspectives reviewed in Section 2.1, we propose an approach to musical notation that we Initially, we have been considering call agential scores.

an agential score as one that responds to environmental in- puts, possesses goals or directions, seeks equilibrium or development based on these objectives, and maintains a self-identical unity that endures over time.
In our work in- volving artificial life as components of dynamic scores and musical instruments, we explore emergent behaviours that arise only when individual parts interact within a larger system.

Although attributing goal-seeking characteristics may appear anthropomorphic, recent biological theories have challenged the notion of restricting such terms exclu- sively to humans [31].

In this spirit, we prefer to maintain openness regarding defining this space.
In this section we start from the bottom-up, by describing the agency of raw notational inscriptions, and then move to the top-down, by describing the entanglements [21] be- tween scores, instruments, composers and performers.

Then we turn to describing what emerges out of these entangle- ments in terms of assemblages [32] and intra-action [20].
3.1 Agency of Points and Lines The historical significance of points and lines can be traced back to the earliest known human writing systems [33], and primitive geometries have been found to possess con- sistent interpretations across cultures and geographies [34].

Similarly, points and lines are also deemed to be prevalent elements in numerous forms of music notation throughout history [35].
In static, dynamic, and computational musical scores, points and lines exhibit distinct agencies, akin to their differences in paintings, animations, and real-time computer graph- ics.

The painter Kandinsky ascribed concealed human at- tributes to points, characterising them as “the highest de- gree of restraint which, nevertheless, speaks.” [36] An- other artist, Paul Klee, illustrated in his pedagogical sketch- book, “An active line on a walk, moving freely, without goal.

A walk for a walk’s sake.


The mobility agent is a point, shifting its position forward.” [37] Although paint- ings remain static, our eyes move in relation to them, and both Kandinsky and Klee imbue their stationary media with a sense of motion.

Conversely, animated points and lines exhibit actual movement.

Pioneer Norman McLaren’s con- cept of “pure” cinema emphasised “little or no reliance on factors other than motion”, favouring the expressive poten- tial of basic forms over cinematography [38].
Nonetheless, once drawn or rendered, the motion of points and lines in animation remains fixed in time.

In this con- text, we can differentiate between a temporally fixed dy- namic score and a procedural dynamic score, where the (a) FerroNeural (2023) featuring gestural control of a T¨olvera scene via handheld magnetic disc controllers [30].

Authors: Nicola Privato and Jack Armitage.

(b) Motherbird (2023) featuring interactive T¨olvera murmuration simulation.
Authors: Jessica Shand, Manuel Cherep and Jack Armitage.

Photo: James Day, MIT Media Lab.

Figure 1: Photos from the development of the first two collaborative musical works that incorporate T¨olvera.

Figure 2: Three examples of entanglements [21] with agential scores, along a continuum of complexity.

We investigate the phenomenology of self-organised scores (a, left) in Section 5.
We are exploring instrument-score entanglements (b, middle) in the piece FerroNeural (Figure 1a), and multiple entanglements (c, right) in the piece Motherbird (Figure 1b).

latter is rendered in context, such as during live drawing or when utilising software.

Specifically, with software, the preconception of motion is abstracted into the code.

Self- organising swarms represent one artificial life approach for delegating notational dynamism in a manner that leads to the emergence of intriguing behaviour.
Within this emer- gence, there exists the potential for points and lines with a fluid materiality, which reside “in the grey area of the con- tinuum and have both the affordance of things and stuff” [39].

Self-organising systems advance the tradition of scores as points and lines, offering novel ways of experiencing fluid points and lines as notation.

way of decomposing the space of agential scores.

There are many more possibilities that could potentially be de- scribed in this typological manner.
We have found distin- guishing types of entanglements can aid in the early stages of the agential score composition process, and when in- troducing agential scores to collaborators.

There is also potential for such a typology to be developed further and used as an analysis tool, since it can also be used to de- scribe performance ensembles and ecologies that do not necessarily make use of dynamic or computational media.
We envision this might be useful as one of many “ad-hoc taxonomies” in a heterarchical organological interpretation [40].

3.2 A Typology of Entanglements with Agential Scores 3.3 Assemblages and Intra-action Owing to the potential combinations of mappings between scores, instruments, and musicians, there is plenty to ex- plore.
Figure 2 depicts three examples of what we describe as entanglements with agential scores, following Frauen- berger’s description of entanglement human-computer in- teraction (HCI) [21].
In this way we suggest a typology of entanglements with agential scores based on identify- ing the network of relationships between “agents”, as one We must also address the limitations of a purely typolog- ical approach to agential scores, which can only describe the elements of an entanglement and their relationships in simple terms, and not the emergent properties that arise from the interactions between them.
Th´eberge proposes to consider musical instruments as assemblages, “compo- nents within a network” where the instrument maker’s role is to “design relationships” rather than objects [32].
Barad performer(s)instrument(s)agential score(a) Self-organised scoresimple / unidirectional / feedforwardcomplex / networked / feedbackperformer(s)instrument(s)agential score(b) Instrument-score entanglement(c) Multiple entanglementsinstrument(s)agential scorecomposer/conductorperformer(s) Figure 3: Examples of T¨olvera combining Boids particles and Physarum pheromone trails, where the Boids particles also deposit pheromones to attract the Physarum.
The variations arise from each species’ behavioural parameters and the weighting of the mapping between them.

would describe working with musical scores from this per- spective as a form of intra-action, “the mutual constitution of entangled agencies”, which is contrasted with “the usual ’interaction’, which assumes that there are separate indi- vidual agencies that precede their interaction”.
For Barad, “agencies are only distinct in relation to their mutual en- tanglement; they don’t exist as individual elements”.

[20] Though emergent properties can be described in the reduc- tive context of computer simulations, these are potentially much less determinable in the context of live musical per- formance.
Detailed interpretations and reflections of these do exist in the context of highly entangled musical instru- ments [26, 29], but less appears to have been said about the role and agency of (particularly dynamic) scores in such entanglements.

It is here that we perceive a research need that the framing of agential scores can potentially help to address, if it can be put into practice.
3.4 Agential Scores in Practice via Artificial Life This work represents our first attempt to practically inves- tigate agential scores, using artificial life, as many have done before although not with our specific aims and moti- vations in mind.

Artificial life has been employed in both the sciences and the arts for various purposes.

Christo- pher Langton, a founding advocate for the field, defines it as “life-as-it-could-be” [41].
Researchers within the field have elucidated and investigated how it can assist biolo- gists and cognitive scientists in exploring the intricate is- sues of mind, agency, intelligence, and autopoiesis [18].

Although machine learning, particularly deep learning, has recently been the most publicised subdomain of artificial intelligence, there is now a growing interest in incorporat- ing self-organising systems within these fields [42].
Ad- ditionally, machine learning is being utilised to train high- level goals in self-organising systems [43, 44].

In the arts, artificial life has been applied in the realm of computa- tional creativity as systems capable of producing creative outcomes, such as in music or computer graphics [45].

In this context, artificial life is not perceived as a tool for solv- ing problems but rather as an instrument that offers inno- vative, generative patterns and creative pathways [46].

1 4.
4.

T ¨OLVERA: A LIBRARY OF NUMBER BEINGS T¨olvera 2 is an open-source Python library for designing musical instruments and musical notations using artificial life 3 .

Compared with existing tools like NetLogo and Golly, T¨olvera is designed to use a modern and portable imple- mentation, be more accessible and extensible to creative coders, and to be more easily integrated into musical in- struments and notation systems.
The name is an example of a kenning, a metaphorical compound expression found in Old Norse and Old English poetry.

T¨olvera combines the Icelandic words for computer (t¨olvu, derived from tala - number - and v¨olva - prophetess or oracle) and being (vera).

Our lab is situated in Iceland, and we have discov- ered that immersing ourselves in the local culture enhances our work and encourages public participation and collab- oration [14].
We are also intentionally engaging with the agency of the Icelandic language itself, juxtaposing arti- ficial life and musical scores with Icelandic folklore and mythology.

Combining number with being also alludes to the discourse surrounding embodiment in artificial life, as discussed in Section 2.

Listing 1 shows a simple T¨olvera program that renders particles to a window.
import taichi as ti import tolvera as tol def main(x=1920, y=1080, n=512, species=3): ti.init() particles = tol.Particles(x,y,n,species) pixels = tol.Pixels(x,y) def render(): pixels.clear() particles(pixels) pixels.show(render) Listing 1: Example of a T¨olvera program that renders par- ticles to a window.
2 https://github.com/Intelligent-Instruments-Lab/ iil-python-tools/tree/master/tolvera 3 Please see the T¨olvera YouTube playlist for demos and work-in- progress experiments https://www.youtube.com/playlist?

list=PL8bdQleKUA1vNez5gw-pfQB21Q1-vHn3x.

4.1 Number Beings T¨olvera consists of a small number of simulations (vera, or beings) that are variously inspired by physics, biology, computation, or some mixture of these.

Familiar exam- ples include Boids, the classic flocking algorithm by Craig Reynolds.
[47], and Physarum, a slime mould simulation popularised by artists such as Sage Jensen 4 .

[48].

We are experimenting with adding many more vera such as Lenia, an artificial life continuous cellular automata (CCA) sys- tem discovered by Bert Chan, as a smooth generalisation of Conway’s Game of Life.

[49].

However, so far we have focused on the design of the library, so that it can be easily used by creative coders, and extended by other developers.
Listing 2 shows a T¨olvera program that renders multiple species of particles to a window.

1 def render(): pixels.diffuse() boids(particles) particles(pixels) physarum(particles) pixels.set(physarum.trail.px) Listing 2: Example of a T¨olvera render function that creates compound motion of particles by combining boids and physarum algorithms.

In future this will be syntactically simplified via method chaining to e.g.

particles.boids(args).physarum(args).


Any real-time Taichi program can potentially be added or contributed to the T¨olvera library, and though we do not prescribe a coding style, so far we follow Taichi’s objec- tive data-oriented programming paradigm by implement- ing each species or simulation as a data-oriented class.

5 As in other real-time creative computing contexts, each class aims to provide simple “process” functions that step the simulation forward and update behavioural parameters.
4.2 Mappings and Visualisations One of the expressed aims of T¨olvera is to commingle in- terfaces and instruments with artificial life.

In this way, we envision that instruments provide input and output data streams to T¨olvera, becoming its connection to physical reality.

At the same time, T¨olvera becomes integrated into the instrument for its performer.

Listing 3 shows how to send and receive OSC messages from a T¨olvera program.
T¨olvera can by default run as a window on a regular com- puter screen.

However, we have observed that projecting T¨olvera onto horizontal surfaces (either from above or be- low) encourages intimate engagements, through tactile in- terventions such as hand-drawing notation, and improvised use of objects as notation (see Figures 1 and 4).
Further, T¨olvera could in future integrate computer vision to recog- nise these as inputs to the simulations, resulting in a com- mingling of physical and computational processes.
4 https://cargocollective.com/sagejenson/ physarum 5 https://docs.taichi-lang.org/docs/odop p = tol.Particles(x,y,n,species) # ... osc_update = OSCUpdaters(osc, client="simple", receives={ "/particles/set/pos": p.osc_set_pos, "/particles/set/vel": p.osc_set_vel }, receive_count=10, sends={ "/particles/get/pos/all": p.osc_get_pos_all }, send_count=60 ) # ... def render(): osc_update() pixels.clear() particles(pixels) Listing 3: Example of OSC messages being sent and re- ceived by the OSCUpdaters class in T¨olvera.
Counters are used to rate-limit how frequently received messages are processed and how frequently messages are sent.

This al- lows OSC messages to be sent and received at a rate that is independent of the frame rate.

4.3 Implementation T¨olvera programs can be run in real-time on the CPU or GPU of a modern laptop, 6 and combined with other Python libraries enabling additional features such as machine learn- ing.
T¨olvera is implemented in Taichi 7 , an imperative, parallel, domain specific language embedded in Python, for high-performance numerical computation.

Taichi uses just-in-time (JIT) compiler frameworks such as LLVM, to offload compute-intensive code to native GPU or CPU in- structions.
We chose Taichi because it enables optimised graphics programming through a familiar high-level lan- guage, and because, being embedded in Python, it can in- teroperate with the rich tapestry of the machine learning ecosystem.

Taichi programs can also be run on the GPU without any visual rendering, allowing the programs to be run “headless” in more resource-constrained contexts.
They can be compiled to C/C++, and then to JavaScript via Em- scripten, and also run in Unity, 8 enabling a wide variety of target contexts to be explored in future (see Section 6.4).

5.

MUSICAL ENCOUNTERS WITH T ¨OLVERA T¨olvera is designed for instrumental and notational design, and we’re researching how musicians and audiences in- teract with the concepts of intelligence, emergence, and agency.
We’ve been conducting informal musical sessions, called encounters, with our research prototypes to inform design iterations and inspire future encounters.

We present an ethnographic account of these encounters in this sec- tion, and offer an interpretation of the emerging themes in Section 6.

Two professional guitarists participated in an 6 Modern laptop with a GPU supporting backends such as OpenGL, CUDA, Vulkan and Metal: https://docs.taichi-lang.org/ docs/hello_world.
7 https://github.com/taichi-dev/taichi 8 https://github.com/taichi-dev/ Taichi-UnityExample (a) Boids clustered into spheres, with players ’identifying’ as metal bowls on the floor.

(b) Players improvising in response to Physarum pheromone trails.

(c) Boids and Physarum combination, with Player A ’conducting’ Player B’s attention.

(d) Boids and Physarum combination, with Player B ’conducting’ Player A’s attention.

Figure 4: Video stills of four setups from the T¨olvera improvisation session.
T¨olvera scenes were projected from above onto the floor.

The two guitarists sat in chairs nearby.

informal session with T¨olvera in an arts university studio lab space.

The primary aim was to determine the extent to which basic configurations of Boids and Physarum par- ticle systems could maintain engagement in an informal improvisation setting.

The guitarists played their own in- struments, and the T¨olvera designer edited and randomised the T¨olvera scenes in response to the musicians.
The en- counters lasted between three and seven minutes each.

5.1 Encounters Summaries 5.1.1 Encounter 1: Boids & Two Guitars Figure 4a shows a video still from the first encounter be- tween two guitarists and Boids.

The Boids were divided into two species, referred to as Species 1 and Species 2.

Player A focused on Species 1, which formed lengthy, sin- uous streams, while Player B engaged with Species 2, which developed into small, swiftly agitated clusters.
Player A alternated between playing phrases akin to those from ear- lier when Species 1 crossed their bowl and maintaining a steady rhythm of atonal chords for Species 2.

Similarly, Player B played as before when Species 2 traversed their bowl and executed loud, heavy chord riffs when Species 1 emerged.

The Boids species colours and parameters were randomised several times to generate variations.
As more variations emerged, Player B asked, ”how do I play this?” The music grew increasingly disjointed and chaotic as they attempted to adapt to new scenarios.

The music ultimately ceased following a final randomisation, with both musi- cians pausing before Player B stated plainly, ”I don’t know,” and they both shared a laugh.

5.1.2 Encounter 2: Physarum & Two Guitars In the second encounter (Figure 4b), the players responded to the Physarum simulation this time.
Player A played sim- ilar spacious slide guitar notes as before, while Player B utilised a pen to produce slide guitar sounds as well.

Their sound was considerably quieter than in Encounter 1, pos- sibly due to the increased amount of empty space in the scene and the reduced motion of the Physarum.

Their flow- ing, sliding gestures appeared to correspond with the twist- ing, continuous pheromone trails.

After a relatively calm encounter, the players concurred at the end that “this was fun”.
5.1.3 Encounter 3: Boids, Physarum, Guitar & Conductor Encounter 3 (Figure 4c) involved Boids and Physarum com- bined in a T¨olvera scene.

The Boids left an additional pheromone trail for the Physarum to follow, resulting in some interaction between them.

Player A acted as a con- ductor by pointing a baton at the projection on the floor to direct Player B, who played guitar.

The encounter be- gan with randomisation of T¨olvera parameters to find a preferred setting.
Player A tracked a Physarum ring of pheromone trail that was slowly closing in on itself while Player B gradually increased their tempo, playing two semi- tones.

As the ring exploded back out into the space, Player A swept around with their baton before settling on trac- ing another more stable pheromone trail.

Player B adapted their playing accordingly.

Player A then followed vari- ous Boids clusters from different species, some of which were calmly floating and others that were more animated.
Player B responded to each Boid cluster differently, adapt- ing their previous musical ideas.

The conducting and play- ing matched the progression of the T¨olvera scene, becom- ing more dynamic than the previous encounters.

5.1.4 Encounter 4: Reversing Roles from Encounter 3 In the final encounter (Figure 4d), Player A and B switched roles from Encounter 3, with Player B becoming the ‘con- ductor’ and Player A returning to playing the guitar.
They began by randomising T¨olvera parameters to find an ap- pealing scenario, with comments such as ”Mmm hmm, this I like!”, ”It’s good.”, and ”It’s beautiful, no?”.

Player B started by tapping specific points and lifting their ba- ton from the floor, causing Player A to inquire about when to play silence.

Both players laughed and shrugged in re- sponse.


Player B followed the trails of Physarum and col- lapsing rings, responding with spacious slide guitar phrases that mapped both ends of a trail to low and high pitches.

During the performance, Player A accidentally knocked the projector mounting, causing both players to react with surprised laughter as the entire scene shook.
Player B then traced a specific Boids cluster around the scene, which suddenly changed direction and darted off the edge of the screen.

Both players laughed as Player B exclaimed and stepped back, saying, ”I had started to sympathise [with that cluster]... Ah, it comes back this way, right?” upon realizing that the particles wrap around the edges of the scene.
5.2 Post-Encounters Discussion In a discussion, the researcher and author asked open-ended questions about encounters, leading to a conversation about potential T¨olvera development.

The players noticed the Boids clusters would occasionally suggest a musical rep- resentation, but were sometimes random.
Player A talked about their film scoring practices and their awareness of “Mickey Mousing.” They found this approach safe and comforting because it allowed for an easy association be- tween the particles and musical gestures.

However, they mentioned feeling lost when tracked clusters vanished and would sometimes intentionally try to lose their identity with a cluster.

Player A also described experiencing various emotional states, such as feeling safe, lost, bored, and com- forted.
Player B commented that musicians are not pre- dictable but follow certain expectations, while artificial life is less predictable.

The players compared their relationships to musical scores in classical and jazz contexts, respectively.

Player A be- lieved they adhered more strictly to rules in response to classical scores, while Player B felt jazz players had more freedom in interpretation.
They both acknowledged that their experiences influenced their reactions to T¨olvera scenes and appreciated the contrast in their musical responses.

They discussed the limitations of guitar as an instrument, such as difficult fingerings and challenges with microtonal- ity.

Player A felt constrained by their instrument and thought a symphony orchestra was necessary to express diverse activity.

Having two guitarists alleviated some pressure, but did not influence their reactions to specific encounters.
Both players enjoyed playing and conducting, with the lat- ter providing more freedom and the ability to anticipate the conductor’s intentions.

This suggests a shared attraction to similar types of activity and interpretations.

During a conduct by Player B, Player A wondered if a more formal, rule-based approach was anticipated, akin to their classical training.
Player A suggested dynamically morphing or navigating between parameter states for fu- ture T¨olvera development, instead of instantaneously ran- domizing all parameters.

They often interpreted the dis- tance between objects as pitch intervals and discussed the possibility of a visual demarcation of pitch space through a stave-like overlay.

This stave could be microtonal or con- tinuously graded, but it was uncertain whether it should indicate temporality.
Player B proposed dividing the space into a grid of cells, each with different scoring instructions or mappings, and assigning players to specific cells.

They also discussed explicitly representing players as particles or clusters and providing control over their range of mo- tion and responses through mapping.

6.
6.

DISCUSSION Despite the nascent state of T¨olvera, we discovered that simple emergent patterns of Boids and Physarum particles facilitated diverse musical interpretations, evoked visceral emotional responses, and stimulated reflective discussions about the agency of musical scores and their influence on human and non-human interpreters.
As discussed in Sec- tion 2, we are interested in agency from multiple perspec- tives, viewing our research systems as boundary objects [28] that can ignite pluralistic dialogue.

In that spirit, we explore various themes of agency throughout the encoun- ters to enrich our overall theme of agential scores.

6.1 Fluid Material Agency The emergent patterns of the Boids and Physarum particles accentuate points and lines in the T¨olvera scores.
In accor- dance with the theories of material agency discussed ear- lier in Section 2, we suggest that real-time software-based dynamic scores alter the agency of points and lines, en- abling them to convey the fluidity of these primitive forms through procedurally adaptive motion.

This fluidity, ex- perienced as emergent and self-organising music notation, subsequently restructures the musician’s perceptive eye and interpretive mind, leading to a more spontaneous and dia- logic negotiation of attention.
In the T¨olvera scores, Boids are represented as points and Physarum as curved lines by default.

As an open nota- tional score, these fundamental forms raise questions about their musical interpretation.

The guitarists improvised re- sponses, with point sizes corresponding to volumes and line lengths to pitch ranges, among other mappings.
How- ever, certain species of Boids seemed to exhibit fluid-like behaviour, while Physarum trails could adopt more dis- crete appearances, such as rings, points, and grids.

Indi- vidual Boids could merge into a cluster, which could then form larger clusters, containing internal oscillations of in- dividual members.

Similarly, Physarum rings could con- verge inwards to a single point before expanding outwards into tree-like structures or dissipating entirely.
These mor- phological transformations between forms increased the demands on musicians’ interpretation, requiring them to seamlessly combine rapid chains of diverse playing tech- niques and musical phrases, while adjusting their mappings in real-time.

The fluid materiality provided by artificial life consequently alters our perception of the agency of a point or line, ac- knowledging their inherent fluid potential.
Barad contends that physical nature itself exhibits greater fluidity and queer- ness than ontological discreteness and causality, urging us to adopt a perspective of fluidity when observing the world [50].

How does our sense of self evolve when we asso- ciate with such fluid material in a symbolic musical rela- tionship?
6.2 Mapping of Self Onto Agential Materials The typology we described in Section 3.2 led us to de- scribe this encounter as an entanglement between an agen- tial score, two performers, and their instruments, with a simple unidirectional flow between them (Figure 2).

How- ever, we observed another kind of entanglement that was not explicitly described in the typology.
It became appar- ent that the life-mimicking behaviour of the T¨olvera scores prompted musicians to associate with specific aspects of the scores, even eliciting a subtle sympathetic response in Encounter 4.

This projection of oneself onto the dynamic behaviour of the score was incidental and transient, as the life forms within the score continuously evolved, formed new configurations, and subsequently disintegrated.
Extrapolating from this, we envisage that only a slight perception of agency in a score is necessary to trigger hu- man tendencies to identify, empathise, mirror, and attune.

This process establishes an entanglement or mapping at a higher level than between notations and gestures and sounds, connecting human selves with agential materials.

When unexpected behaviour from an agential material disrupts this mapping, visceral responses can be elicited ranging from surprise and excitement to disgust.
Attunement to an agential score and its subversion presents a novel compo- sitional device for musicians to explore and researchers to examine.

In future work we seek to elaborate on these psy- chological and emotional aspects of experience with agen- tial materials through an investigation into agentology [51].
6.3 Perceiving the Intra-Actants Following Barad’s agential realism and the concept of intra- action [20], we propose that each of our encounters gen- erated a transient intra-actant, emerging from the assem- blage of musicians, media, and materials, and possessing its own distinct agency.

The qualities of each intra-actant were inextricably connected to each T¨olvera score and the moment it took place in, and resist typological decompo- sition.
What language and methods are appropriate for de- scribing and comparing intra-actants, that evade the trap of a decompositional approach?

Perhaps a new approach is needed that builds on astute methods like micro-phenomenonology [52, 53] to compare multiple experiences of the same mo- ment, leading to a macro-phenomenonological account.

For inspiration, composers encountering agential scores for the first time might find it advantageous to examine the experiences of feedback instrumentalists [54].
These mu- sicians have dedicated significant time to initially unruly complex musical systems and have developed aesthetics that are compatible with extended and augmented instru- mental agencies.

Agential instruments also present a natu- ral pairing for agential scores.

6.4 Future Considerations In this paper, we established a context for discourse on agential scores and recount early experimental artistic ap- plications.
We intentionally avoided providing a rigid def- inition or framework to allow for alternative perspectives and interpretations.

Here we outline what will be our fo- cus as we seek to develop the themes of this work further.

Our aim is to develop the T¨olvera library through artis- tic and empirical encounters, generating insights surround- ing agential scores and their design.
The T¨olvera library will expand in multiple directions, including incorporat- ing more number beings, additional inputs, and a broader range of outputs.

We also anticipate a vertical develop- ment towards goal-oriented and trainable behaviours and instrument mappings through integrating online machine learning processes.
We have yet to explore the roles of the composer and au- dience within the context of agential scores, and we envis- age composers will eventually engage with agential scores through more accessible interfaces such as graphic design, drawing, and tangible or gestural interactions.
In this sce- nario, a composer could create artificial life patterns and parameters using an assortment of familiar media, and sub- sequently instruct conductors and performers on interpret- ing these elements as notation (or even conduct and per- form themselves).

With regard to audiences, their behaviour in large groups has been extensively modelled in the field of crowd sim- ulation, providing a compelling entry point for integrating them with agential scores.
Additionally, there is a growing research interest in enabling audience participation in mu- sic performances through networked, mobile, and gestural interfaces, which is highly relevant to this subject [55].

We consider T¨olvera as an element in our Organolib system (see earlier footnote), and we anticipate that our commu- nity will discover increasingly diverse applications for it in combination with other elements.
As the system matures, we envisage designing encounters where more detailed accounts of intra-action can be cap- tured, not only during performance but also during design and composition [56].

At the micro scale [57], this could involve comparing eye or gesture tracking data with parti- cle data, and analysing their convergence and divergence across various score states.
Based on our other studies, we are also interested to know how perception of symmetry and algorithmic pattern [58] affects projection or attribu- tion of agency.

And qualitatively, micro-phenomenology can be used to capture the experience of the musicians and audience mem- bers.

Such encounters would enable us to further develop linguistic and conceptual tools for the analysis and inter- pretation of the macro scale [59] agency of intra-actants.

7.
7.

CONCLUSION In this paper, we introduced the concept of agential scores.

An agential score is characterised by a self-identical unity that endures over time, responding to environmental in- puts, possessing goals or directions, and striving for equi- librium or development in line with these objectives.

We explored this idea by examining how points and lines gain fluid material agency through animation via dynamic com- putational materials and simulations.
We compared a ty- pological and decompositional approach to agential scores with a more holistic, macro-phenomenonological approach that situates agential scores as entangled with their mate- rial and social contexts.

This broader context we described as an intra-actant, inspired by Barad’s idea of intra-action, to emphasise the study and comparison of these high-level emergent entities.
We introduced T¨olvera, an open-source software library written in Python and inspired by Icelandic kennings and mythology, for exploring agential scores in real-time through artificial life simulations.

An initial musical encounter with T¨olvera involving two improvising guitarists was conducted to ground the development process in empirical observa- tion.
Even at an early stage, T¨olvera demonstrated that simple emergent patterns of Boids and Physarum particles facilitated diverse musical interpretations, elicited emotional reactions, and sparked thoughtful conversations about the agency of musical scores and their human and non-human interpreters.

Reflecting on our observations, we proposed that agen- tial scores foreground a particular compositional mapping layer that mediates between musical selves and agential materials.
We perceived this as having implications for ecological music perspectives.

Lastly, we posed several challenging issues and questions that we anticipate will require addressing as more advanced agential scoring ap- proaches emerge.

Acknowledgments Thanks to Nicola Privato and Hafd´ıs Bjarnad´ottir for their participation in the encounter.

Thanks to Motherbird col- laborators Jessica Shand and Manuel Cherep.
Thanks to Victor Shepardson for software development support, and Marco Donnarumma for testing and feedback.

Thanks to the anonymous reviewers for their helpful comments.

8.

REFERENCES [1] J. Lely and J. Saunders, Word Events: Perspectives on Verbal Notation.

Continuum New York, NY, 2012.

[2] L. Kotz, Words to Be Looked at: Language in 1960s Art.

Mit Press Cambridge, MA, 2007.

[3] G. Cox and A. McLean, Speaking Code: Coding as Aesthetic and Political Expression.

MIT Press, 2012.
MIT Press, 2012.

[4] Y. Ono, Grapefruit: A Book of Instructions and Draw- ings by Yoko Ono.

Tokyo: Wunternaum Press, 1964.

[5] J. Dack, “The ’open’ form – literature and music,” in Scambi Symposium’, Goldsmiths College, vol.

334, 2005, p. 336.

[6] W. Thompson, Soundpainting: The Art of Live Com- position.

Workbook I. Walter Thompson, 2006.


[7] A. Sonnenfeld and K. F. Hansen, “S-notation: A com- plete musical notation system for scratching and sam- ple music derived from” Theory of Motions”,” in In- ternational Conference on Technologies for Music No- tation and Representation.

Anglia Ruskin University, 2016, pp.

50–57.

[8] C. Hope, “Electronic scores for music: The possibil- ities of animated notation,” Computer Music Journal, vol.

41, no.

3, pp.

21–35, 2017.


[9] E. Tom´as and M. Kaltenbrunner, “Tangible Scores: Shaping the Inherent Instrument Score,” in Proceed- ings of the International Conference on New Interfaces for Musical Expression.

London, United Kingdom: Goldsmiths, University of London, Jun.

2014, pp.

609– 614.

[10] S. Bhagwati, M. Giordano, J. Berzowska, A. Bach- mayr, J. Stein, J. Browne, F. D. Tredici, D. Egloff, J. Sullivan, M. Wanderley, and I. Cossette, “Musicking the Body Electric.
The ”body:suit:score” as a polyva- lent score interface for situational scores.” in Proceed- ings of the International Conference on Technologies for Music Notation and Representation - TENOR2016, R. Hoadley, C. Nash, and D. Fober, Eds.

Cambridge, UK: Anglia Ruskin University, 2016, pp.

121–126.
121–126.

[11] W. V. Rodrigues and W. de Macedo Rodrigues, “Tex- tural Composition in 3D Environment Through Swarm Algorithm,” in Proceedings of the International Con- ference on Technologies for Music Notation and Rep- resentation - TENOR2022, 2022.

[12] A. Kay and A. Goldberg, “Personal dynamic media,” Computer, vol.

10, no.

3, pp.

31–41, 1977.


This research is supported by the European Research Coun- cil (ERC) as part of the Intelligent Instruments project (IN- TENT), under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No.

101001848).

[13] V. Shepardson, J. Armitage, and T. Magnusson, “No- tochord: A Flexible Probabilistic Model for Embodied MIDI Performance,” in Proceedings of the 3rd Confer- ence on AI Music Creativity.

AIMC, Sep. 2022.
AIMC, Sep. 2022.

[14] J. Armitage, T. Magnusson, V. Shepardson, and H. Ul- farsson, “The Proto-Langspil: Launching an Icelandic NIME Research Lab with the Help of a Marginalised Instrument,” in Proc.

New Interfaces for Musical Ex- pression.

[15] D. Borgo, Sync or Swarm: Improvising Music in a Complex Age.

A&C Black, 2005.

[16] S. Waters, “Performance Ecosystems: Ecological ap- proaches to musical interaction,” EMS: Electroacous- tic Music Studies Network, pp.

1–20, 2007.
1–20, 2007.

[17] G. E. M. Anscombe, Intention.

Harvard University Press, Oct. 2000.

[18] H. R. Maturana and F. J. Varela, Autopoiesis and Cog- Springer Sci- nition: The Realization of the Living.

ence & Business Media, 1991, vol.

42.

[19] B. Latour, Reassembling the Social: An Introduction to Actor-Network-Theory.

Oup Oxford, 2007.

[20] K. Barad, Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning.

Duke University Press, 2007.


[21] C. Frauenberger, “Entanglement HCI the next wave?” ACM Transactions on Computer-Human Interaction (TOCHI), vol.

27, no.

1, pp.

1–27, 2019.

[22] C. Nordmoen and A. P. McPherson, “Making space for material entanglements: A diffractive analysis of woodwork and the practice of making an interactive system,” in Designing Interactive Systems Conference, ser.

DIS ’22.

New York, NY, USA: Association for Computing Machinery, Jun.

2022, pp.

415–423.
2022, pp.

415–423.

[23] N. Virgo, “The necessity of extended autopoiesis,” Adaptive Behavior, vol.

28, no.

1, pp.

23–26, Feb. 2020.

[24] M. Villalobos and P. Razeto-Barry, “Are living beings extended autopoietic systems?

An embodied reply,” Adaptive Behavior, vol.

28, no.

1, pp.

3–13, Feb. 2020.

[25] S. Kember, Cyberfeminism and Artificial Life.

Rout- ledge, 2003.

[26] A. P. Melbye, “Resistance, Mastery, Agency: Impro- vising with the feedback-actuated augmented bass,” Organised Sound, vol.
26, no.

1, pp.

19–30, 2021.

[27] B. Latour, “The Berlin key or how to do words with things,” Matter, materiality and modern culture, vol.

10, p. 21, 2000.

[28] S. L. Star and J. R. Griesemer, “Institutional ecology, ’translations’ and boundary objects: Amateurs and pro- fessionals in Berkeley’s Museum of Vertebrate Zool- ogy, 1907-39,” Social studies of science, vol.

19, no.

3, pp.

387–420, 1989.


[29] P. Stapleton and T. Davis, “Ambiguous Devices: Improvisation, agency, touch and feedthrough in distributed music performance,” Organised Sound, vol.

26, no.

1, pp.

52–64, Apr.

2021.

[30] N. Privato, E. Torfi, and T. Magnusson, “The Mag- netic Score: Somatosensory Inscriptions and relational design in the instrument-score,” in Proceedings of the 8th International Conference on Technologies for Mu- sic Notation and Representation.
[31] M. Levin, “Technological Approach to Mind Every- where: An Experimentally-Grounded Framework for Understanding Diverse Bodies and Minds,” Frontiers in Systems Neuroscience, vol.

16, 2022.

[32] P. Th´eberge, “Musical Instruments as Assem- blage,” in Musical Instruments in the 21st Century, T. Bovermann, p. u. family=Campo, given=Alberto, H. Egermann, S.-I.

Hardjowirogo, and S. Weinzierl, [Online].

Eds.

Available: http://link.springer.com/chapter/10.1007/ 978-981-10-2951-6 5 Springer Singapore, pp.
59–66.

[33] B. Bacon, A. Khatiri, J. Palmer, T. Freeth, P. Pettitt, and R. Kentridge, “An Upper Palaeolithic Proto-writing System and Phenological Calendar,” Cambridge Ar- chaeological Journal, pp.

1–19, Jan. 2023.

[34] A.

´Cwiek, S. Fuchs, C. Draxler, E. L. Asu, D. Dediu, K. Hiovain, S. Kawahara, S. Koutalidis, M. Krifka, and P. Lippus, “The bouba/kiki effect is robust across cul- tures and writing systems,” Philosophical Transactions of the Royal Society B, vol.

377, no.

1841, p. 20200390, 2022.


[35] T. Ingold, Lines: A Brief History.

Routledge, 2016.

[36] W. Kandinsky and H. Rebay, Point and Line to Plane.

Courier Corporation, 1979.

[37] P. Klee, Pedagogical Sketchbook.

Introduction and translation by Sibyl Moholy-Nagy.

Praeger: New York, 1953.

[38] D. McWilliams and S. Huycke, “Creative Process: Norman McLaren,” 1991.
[39] M. Le Goc, L. H. Kim, A. Parsaei, J.-D. Fekete, P. Dragicevic, and S. Follmer, “Zooids: Building Blocks for Swarm User Interfaces,” in Proceedings of the 29th Annual Symposium on User Interface Soft- ware and Technology, ser.

UIST ’16.

New York, NY, USA: ACM, 2016, pp.

97–109.

[40] T. Magnusson, “Musical Organics: A Het- to Digital Organology,” [Online].

Avail- https://www.tandfonline.com/doi/full/10.1080/ erarchical Approach vol.

46, no.

3, pp.

286–303.


able: 09298215.2017.1353636 [41] C. G. Langton, “Studying artificial life with cellular automata,” Physica D: Nonlinear Phenomena, vol.

22, no.

1-3, pp.

120–149, 1986.

[42] D. Ha and Y. Tang, “Collective intelligence for deep learning: A survey of recent developments,” Collec- tive Intelligence, vol.

1, no.

1, p. 26339137221114874, Aug. 2022.


[43] S. Sudhakaran, E. Najarro, and S. Risi, “Goal-Guided Neural Cellular Automata: Learning to Control Self- Organising Systems,” Apr.

2022.

[44] G. Hamon, M. Etcheverry, B. W.-C. Chan, C. Moulin- Frier, and P.-Y.

Oudeyer, “Learning Sensorimotor Agency in Cellular Automata,” Jan. 2022.

[45] M. Whitelaw, Metacreation: Art and Artificial Life.

Mit Press, 2004.
Mit Press, 2004.

[55] A. Xamb´o and G. Roma, “Performing audiences: Composition strategies for network music using mo- bile phones,” in Proceedings of the International Con- ference on New Interfaces for Musical Expression, R. Michon and F. Schroeder, Eds.

Birmingham, UK: Birmingham City University, 2020, pp.

55–60.

[56] J. Armitage, T. Magnusson, and A. McPherson, “De- sign Process in Visual Programming: Methods for Vi- sual and Temporal Analysis,” in Proc.

Sound and Mu- sic Computing Conference.


[57] ——, “Studying Subtle and Detailed Digital Lutherie: Motivational Contexts and Technical Needs,” in Proc.

New Interfaces for Musical Expression.

[46] M. Sato and J. McKinney, “The Enactive and Inter- active Dimensions of AI: Ingenuity and Imagination Through the Lens of Art and Music,” Artificial Life, vol.

28, no.

3, pp.

310–321, Aug. 2022.


[58] ——, “Sculpting Algorithmic Pattern: Informal and Visuospatial Interaction in Musical Instrument De- sign,” in Proc.

Sound and Music Computing Confer- ence.

[59] ——, “A Scale-Based Ontology of Digital Musical In- strument Design,” in Proc.

New Interfaces for Musical Expression.


[47] C. W. Reynolds, “Flocks, herds and schools: A dis- tributed behavioral model,” in Proceedings of the 14th Annual Conference on Computer Graphics and Inter- active Techniques, 1987, pp.

25–34.

[48] J. Jones, “Characteristics of Pattern Formation and Evolution in Approximations of Physarum Transport Networks,” Artificial Life, vol.

16, no.

2, pp.

127–153, Apr.

2010.


[49] B. W.-C. Chan, “Lenia and Expanded Universe,” in The 2020 Conference on Artificial Life, 2020, pp.

221–229.

[50] K. Barad, “Nature’s queer performativity,” Qui Parle: Critical Humanities and Social Sciences, vol.

19, no.

2, pp.

121–158, 2011.

[51] J. Armitage, “Agentology for Organology: Cyber-, Bio-, and Ecosemiotic Perspectives on Instrumental Agency,” in Instruments, Interfaces, Infrastructures: An Interdisciplinary Conference on Musical Me- dia.
https://sites.harvard.edu/ instruments-interfaces-infrastructures/ [Online].

Available: [52] C. N. Reed, C. Nordmoen, A. Martelloni, G. Lepri, N. Robson, E. Zayas-Garin, K. Cotton, L. Mice, Experiences and A. McPherson, through Micro- with New Musical phenomenology,” jun 16 2022, https://nime.pubpub.org/pub/o7sza3sq.
Instruments in NIME 2022, “Exploring [53] M. Prpa, S. Fdili-Alaoui, T. Schiphorst, and P. Pasquier, “Articulating Experience: Reflections from Experts Applying Micro-Phenomenology to the Design Research in HCI,” in Proceedings of 2020 CHI Conference on Human Factors in Com- puting Systems, ’20.

Association for Computing Machinery, pp.

1–14.

[Online].

Available: https://doi.org/10.1145/3313831.3376664 ser.
CHI [54] T. Magnusson, C. Kiefer, and H. Ulfarrson, “Reflex- ions upon Feedback,” in NIME 2022, Jun.

2022.