# Combining Three Graphs to Represent Music for Composition, Performance and Analysis

Myriam Desainte-Catherine, Joséphine Calandra, Jean-Michaël Celerier, Jean-Marc Chouvel

► To cite this version:

Myriam Desainte-Catherine, Joséphine Calandra, Jean-Michaël Celerier, Jean-Marc Chouvel. Combining Three Graphs to Represent Music for Composition, Performance and Analysis. International Journal of Music Science, Technology and Art, 2025, 7 (2), pp.24-40. 10.48293/IJMSTA-133. hal-05370233

HAL Id: hal-05370233

https://hal.science/hal-05370233v1

Submitted on 18 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Combining Three Graphs to Represent Music for Composition, Performance and Analysis

Myriam Desainte-Catherine<fnref n="1" />✉, Joséphine Calandra<fnref n="2" />✉, Jean-Michaël Celerier<fnref n="3" />✉, Jean-Marc Chouvel<fnref n="4" />✉

## ARTICLE INFO

Received: June 30, 2025

Accepted: July 10, 2025

Published: September 8, 2025

DOI: https://doi.org/10.48293/IJMSTA-133

Keywords:

Musical pieces representation

Musical composition

Musical analysis

Performance

Combination of graphs

## ABSTRACT

This article focuses on musical representation for composition, performance and analysis. It highlights three graphs that are widely used for structuring musical pieces in the contexts of composition and performance. These graphs represent relationships linking the elements that make up a piece of music, whether temporal, functional or hierarchical. This article shows how they can be combined in a single graph. This approach is then illustrated in the context of composition and performance with the OSSIA score software and in the context of analysis with the MORFOS software. The ways these graphs are displayed and edited in the interfaces of those two software are compared and discussed.

Copyright © 2025 Author *et al.*, licensed to IJMSTA. This is an open access article distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/3.0/), which permits unlimited use, distribution and reproduction in any medium so long as the original work is properly cited.

## 1 Introduction

This article focuses on representations of musical pieces in the fields of composition, performance and analysis, by studying specifically how musical elements are articulated together. We present a quick overview of existing tools which provides three ways of articulating those elements according to musical activity. These three types of articulations represent different relationships between musical elements that can be modelled by three types of graphs. For performance, languages such as MAX rely on real-time calculations to compute a piece of music. The relationships between objects are thus functional. For composition, it is rather the timeline that allows a piece of music to be defined, particularly on tracks in the case of sequencers. In this case, this approach is based on chronological, sequencing or simultaneous relationships between elements on the timeline, so that relationships are rather temporal. In music analysis, the paradigmatic approach provides a way to segment a piece of music at different time scales, from the note to the overall form. Thus, three types of graphs can be identified, each used to structure data in these three activities: the functional graph for performance, the temporal graph for composition and the hierarchical graph for analysis.

This article first provides definitions of these three types of graphs and proposes a method for combining them into a single graph. Then, in a second part, these concepts are illustrated using two software programs. In the field of composition and performance, we present the interface of the OSSIA score software [6], which has the particularity of using all three types of graphs. In the field of music analysis, we present the musical representations of the MORFOS software [5], which offers the particularity of combining them in a 3D space.

This study highlights the similarities and differences between different representations. Trying to get closer those representations for composition, performance and analysis may lead to get advantages of each of them. From a theoretical point of view, comparing the meaning of those representations for composition, analysis and interpretation may lead to a better

<footnote n="1">¹Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, F-33400 Talence, France</footnote>

<footnote n="2">²STMS-IRCAM, Collegium Musicae, Sorbonne Université, Paris, France</footnote>

<footnote n="3">³Société des Arts Technologiques, Montréal, Canada</footnote>

<footnote n="4">⁴IReMus, Sorbonne Université, Paris, France</footnote>

<footnote n="1">¹✉ Corresponding author: myriam.desainte-catherine@labri.fr</footnote>

<footnote n="2">²✉ Corresponding author: calandra@ircam.fr</footnote>

<footnote n="3">³✉ Corresponding author: jmcelerier@sat.qc.ca</footnote>

<footnote n="4">⁴✉ Corresponding author: jeanmarc.chouvel@free.fr</footnote>
understanding of those musical representations. From a software development point of view, graphic representations and features could be shared by different activities.

# 2 Three Graphs for Structuring Musical Pieces

This section introduces three graphs to represent musical pieces, the first one of them being a temporal graph stating temporal relationships between musical elements. The second is a graph binding musical contents and the third is the hierarchical organisation according to different time scales. Those three graphs are useful to understand the organisation of a piece of music. In the end, a way to combine them into a single graph is proposed.

## 2.1 Time Graph

Temporal relations are a good tool for structuring data over time. They are of two kinds:

<table><caption>Fig. 1. Allen's Relations and their meaning in terms of musical articulations. The inverse relationships (except for the equals relationship) should be added to this list, bringing the total number to 13. The figure also shows the musical meaning of the relations at the note level as well as upper levels of the form.</caption><thead><tr><th>Scheme</th><th>Name</th><th>Note level</th><th>Upper level of the form</th></tr></thead><tbody><tr><td>A B</td><td>A precedes B</td><td>Staccato</td><td>Detached succession</td></tr><tr><td>A B</td><td>meets</td><td>Esatto Legato</td><td>Exact succession</td></tr><tr><td>A B</td><td>overlaps</td><td>Legato</td><td>Tiling</td></tr><tr><td>A B</td><td>equals</td><td>Chord</td><td>Superposition</td></tr><tr><td>A B</td><td>starts</td><td>Synchronising attacks</td><td>Emergence</td></tr><tr><td>A B</td><td>finishes</td><td>Synchronising ends</td><td>Accumulation</td></tr><tr><td>A B</td><td>during</td><td>Conterpoint inclusion</td><td>Intervention</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Fig. 2. Example of a linear time graph of a sequencer.</figcaption></figure>

* Interval relations that are expressed by means of duration variables and Allen's relations [1].

* Points relations that are expressed by means of start and end date variables, that represent the boundaries of musical objects, and equality and inequality relations. These relations can express all Allen's relations as well as their disjunctions. For example, for two musical objects a and b with respective start dates $a_s$ and $b_s$ and end dates $a_e$ and $b_e$, the meets relation is expressed by: $a_e = b_s$ and $a_s < b_s$.

Those relationships can be represented as a graph whose nodes are the variables and whose edges are the relationships. Both models have equal expressiveness.

In the Allen's time graph, nodes are the intervals making up the musical piece and edges are the temporal relations. The Allen's relations are very useful in music and many of them can be read in musical scores at different levels of the form, as it is shown on figure 1. Allen's relations are qualitative, meaning the durations of the intervals are not considered.

In the case of a sequencer, each track corresponds to a time interval. Tracks can be used to represent Allen's relations between musical objects, but they can also be used in any other way, that is without any musical coherence. However, we will focus here on structures that have musical meaning and provide coherence between tracks and the musical objects that constitute them. From a representation point of view, all tracks are globally in an "equals" relation. That is, they all begin and end at the same time explicitly. Even if a track is composed of several musical objects that are carefully organized in time by the composer, the sequencer does not know about the inner temporal relations between the contents of the tracks. It only knows that to play the musical piece, it has to start all the tracks at the same time and stop them in the same way. Thus, the time graph of a sequencer is linear, that is a chain, as shown on the example in figure 2, where there are four tracks named A, B, C and D. In
this graph, as well as in all subsequent graphs in this article, transitive relationships are not shown. They can be inferred from
the relationships actually shown on the graph.

If we consider the temporal relationships between the musical objects making up the tracks, we obtain a more complex time graph. Figure 3 shows an example of two tracks, named A and B, composed each one of three musical objects, named s₁, s₂ and s₃ for the track A, and s₄, s₅ and s₆ for the track B. In this example, the object s₂ is in relation to s₁ because s₂ meets s₁, and with s₃, and in relation to s₄ and s₅.

Thus, we have more than two relations between the elements that are inside the tracks. Figure 3 shows the non-linear graph time associated to this example. Here again, other relations can be deduced from these by computing the transitive closure of this graph.

<figure><img src="image_3.png" /><figcaption>Fig. 3. Example of tracks in a sequencer (left) and its associated time graph (right).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 4. Example of a PureData patch.</figcaption></figure>

## 2.2 Functional Graph

Many languages, such as Max/MSP, PureData, CSound, allow the calculation of sounds and musical pieces by means of functional graphs known as patches, or OpenMusic<fnref n="1" /> at the musical level. The nodes of a functional graph are functions and objects such as notes, sound files and so on. Functions apply to data provided by the incoming edges and send their result(s) through outgoing edges.

Functions at the sound level process at the sample, like delays, filters and so on. Thus, the sound calculating graphs are run in a loop according to the sampling rate of the input data. Musical time is embedded within run-time. Such a graph describes what happens at execution time.

<footnote n="1">¹ https://support.ircam.fr/docs/om/om6-manual</footnote>
Thus, content relations can be inferred from the code of a patch. But the temporal relations between them are less readable. The example displayed on figure 4 is the famous gong of Jean-Claude Risset in PureData [9]. It is computed as the mix of the results of six oscillators, governed by Allen “equals” relation.

## 2.3 Hierarchical Graph

According in particular to Schenker [10], “The musical organization is made according to a musical hierarchy that depends on the importance of the musical events”. Schenker suggests that within a piece of music, certain events or notes hold more significance than others. This hierarchy is often determined by factors such as harmonic function, melodic importance, and structural placement. In Schenkerian analysis, the underlying structure of a piece is often represented through a graph that illustrates the relationships between these musical events. This approach allows for a deeper understanding of how music is constructed and how various elements interact.

In this article, we define a musical hierarchy as a multi-scale structure with the following characteristics:

* It is made of nested elements of different time scales, like notes, phrases, etc.

* It admits an unlimited number of levels.

* It has a tree-like structure of musical elements.

Thus, a hierarchical graph is often represented as a tree whose nodes are musical elements and edges are the nesting relations between them. Nested elements are often called children of the embedding element. The levels are numbered with integers, starting with the leaves of the tree, which are level 0.

This example of figure 3 is organised according to a three-level hierarchy.

* The level 0 is composed of the sounds s1, s2, s3, s4, s5, s6.

* The level 1 is composed of the tracks A and B.

* And the level 2 is the global mix.

The hierarchy of this example is represented in figure 5. It should be noted here that only the nesting relationship is considered. The temporal relationships between the elements are not specified.

## 2.4 Combining the three Graphs

We have three different graphs that are complementary for structuring the musical data.

* Time graph showing temporal relations between musical elements, but without explicit content relations between elements. Thus, there is no way to identify repetitions or transformations of elements.

* Functional graph showing how the result is computed, thus showing contents relations, but without explicit temporal relationships between musical elements.

* Hierarchical graph showing how the musical elements are organised at different time scales but without time information. To combine those three graphs, we propose to proceed to a kind of graph union. The result of a graph union of two graphs is a graph containing the union of the nodes and the edges of the two graphs.

<figure><img src="image_5.png" /><figcaption>Fig. 5. Hierarchy of example 3.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Fig. 6. Example of tracks with transformations of sounds.</figcaption></figure>

## Combination of the time and functional graphs

As an example, let us consider again the two tracks made up of three sounds, and suppose that the third element of the track A is transformed from $s_2$ by the function $g$, and the third element of the track B is transformed from $s_4$ by the function $h$, as it is shown on figure 6.

The time graph of this example is still the one displayed in figure 3.

Now, in order to express the meet relation using a functional graph, we have to define the concatenation function. This operation can be expressed using a mix function, which notated $+$, a delay function notated delay and a duration function notated $d$. The delay function has two parameters, the first one being the elements to delay and the second duration of the delay. Thus, let $C$ be the concatenation of elements A and B: $C = A + delay(B, d(A))$.

An example of a functional graph is given figure 7. The functional expression represented by this patch is the following:

$$s_1 + delay(s_2, d(s_1)) + delay(g(s_2), d(s_1) + d(s_2)) + s_4 + delay(s_5, d(s_4)) + delay(h(s_4), d(s_4) + d(s_5))$$

Let us extract the temporal relations between the objects of the patch. As the addition operation is associative, we proceed by decomposition of the expression from left to right.

* The addition of the sounds $s_1$ and $s_2$ delayed by the duration of $s_1$ gives as a result the concatenation $s_1s_2$.

* Then the addition with $g(s_2)$ delayed by the duration of $s_1$ plus the duration of $s_2$ gives the sequence $s_1s_2g(s_2)$.

* In the same way, the addition of the sounds $s_4$ and $s_5$ delayed by the duration of $s_4$ gives the concatenation $s_4s_5$.

* The addition with $h(s_4)$ delayed by the duration of $s_4$ plus the duration of $s_5$ gives the sequence $s_4s_5h(s_4)$.

* The addition of $s_1$ and $s_4$ implies that the resulting sounds start at the same time.

* If we assume that the durations of the sounds are such that: $d(s_1) + 2*d(s_2) = 2*d(s_4) + d(s_5)$, then the resulting sounds finish at the same time.

Let us point out that a delayed sound is given the same name than the original sound in the temporal expressions. This way, in the union graph, each node corresponding to a delayed sound in the functional graph can be merged with the node representing the original sound in the time graph and the delay operation is replaced by the meets temporal relation of the time graph. The addition operation is also replaced by temporal relations starts and finishes of the time graph. Thus, the union graph is composed of the time graph added with the functions $g$ and $h$ as shown in figure 8.

<figure><img src="image_7.png" /><figcaption>Fig. 7. Functional graph to produce example 6.</figcaption></figure>

## Combination with the hierarchical graph

At last, the hierarchical graph is also combined with the time and functional graph thanks to a graph union. We obtain the final graph which is represented in figure 8. It is obtained from a union of the two preceding graphs, keeping all the nodes and edges of the initial graphs.
Thus, this graph represents the nesting relations, the time relations and the content relations. This representation is complete but may become complicated as soon as the graph has many objects. Let us examine in the following sections how those graphs are displayed in the interfaces of two software, one for composition and the other for analysis.

<figure><img src="image_8.png" /><figcaption>Fig. 8. Combination of time and functional graphs (left) and of the three graphs (right).</figcaption></figure>

# 3 Structuring Musical Pieces for Composition in OSSIA score

This section focuses on the graphic representations of the time graph, the functional graph and the hierarchical organisation in the interface of OSSIA score.

The OSSIA score software [2] was designed to meet the needs of the performing arts community and has been developed thanks to a ANR project<fnref n="2" /> called Virage. This tool was intended to synchronize the different media with each other and with the time on stage. A limitation of the initial model was its focus solely on linear scripts, which led to the development of a second project, OSSIA [7], aimed at extending the model to non-linear scenarios.

In this section, we first introduce the elements needed to construct OSSIA score scenarios, and then we look at the graphs representing the example of the two tracks.

## 3.1 Time Graph

An OSSIA score scenario is made of the following objects.

* Processes are objects which perform a computation over time. Sounds are also represented by processes. They are placed on time intervals and they have inputs and outputs.

* Time intervals allow to define the span of time over which a set of processes will run. Those intervals are placed on the timeline and they have a fixed duration. The boundaries of the intervals are represented by two time points, that are the dates of the start and the end of the interval.

* Time syncs allow to synchronize the start and end of multiple intervals, either at a fixed time or by waiting for an external event. Time syncs are placed on the timeline.

Thus, the time graph in OSSIA score is the interconnection of time intervals and syncs and is called a scenario. Figure 9 shows how this graph is displayed in the interface of OSSIA score. Time intervals are represented by horizontal lines while syncs are represented by vertical lines.

Let us show here that this representation allows to specify all the Allen's relations. The argument is that OSSIA score is based on a logic of point relations and that the user edits relations as well as points. Let *a* and *b* be two time points corresponding to the boundaries of two musical objects:

* The relation *a* < *b* is obtained by placing a time interval between *a* and *b*;

* The relation *a* = *b* is obtained by placing a sync between *a* and *b*.

As it is stated in the next paragraph, there are two kinds of time intervals in order to allow to write real-time behaviour of the scenario. Thus, it should be noted that all temporal relationships can be stated between any type of intervals, whether fixed or flexible.

<footnote n="2">² Agence Nationale de la Recherche, French National research Agency</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 9. Standard graphical representation of a time graph in OSSIA score.</figcaption></figure>

## Time Modelling in OSSIA score

Two existing different visions of music correspond to two different activities: composition and performance. This section recalls the time model that combines in the OSSIA score software the two visions in the two activities. This model allows to manipulate live material while composing and provides ways to play with the time of the written material in real time. It combines the static and dynamic aspects of music in a single model and a single tool allowing both performing and composing. A good reference to explain the time model is to refer to Mc Taggart [12]. In his article “The Unreality of Time”, Mc Taggart introduces two time series based on linguistics representing the way we talk about time.

* A-series corresponds to the three states of events the time flow. Events are future, then become present and then past. Those properties change over time.

* B-series represents the temporal relations before, during and after between events. Those properties do not change over time.

* C-series [11] groups events into a permanent serial order without the directional relations found in the B-series and without the properties of past, present and future as found in the A-series.

Those three series are combined in the following way in the time model of OSSIA score:

* A-series corresponds to dynamic events in the time flow. A dynamic event is an event happening during execution, that is, at performance time. Indeed, during execution and depending on the moment we consider it, a dynamic event is in one of the following states: past (has already arrived), present (has just arrived in the time interval we are looking at) or future (expected).

* B-series corresponds to static temporal relations on the timeline. Those relations are written at composition time. Indeed, events are bound by time relations, that is, in sequence, or simultaneous. Those relations hold during performance.

* C-series is used to combine the time flow and the timeline. We define a hybrid timeline allowing dynamic time intervals to hold dynamic events, and static time intervals for static events.

In this model, static events are defined on the timeline at composition time like in a score editor or a sequencer and they are played by the computer during execution. This is how a basic sequencer works.

The difference with a basic sequencer lies in the fact that there are also dynamic events in this model. They are placed on dynamic intervals at composition time and they are played by the musician at performance time. In consequence the musical piece is not complete at composition time, because dynamic events are defined at performance time. The piece is thus completed by the musician<fnref n="3" />.

The hybrid timeline consists of static intervals and dynamic intervals. Static and dynamic intervals are placed on the hybrid timeline at composition time. Static intervals are used to predict static events at performance time, while dynamic intervals are used to track dynamic events.

In the end, static events are predictable but dynamic events are unpredictable so that all events are organized according to a partial order on the timeline. A total order is obtained on the time flow during execution.

## Flexible Time

In OSSIA score, dynamic intervals are called flexible intervals and static intervals are called fixed intervals. Dynamic events are represented by triggers and are displayed as down-arrow buttons in the interval in which they are expected (see figure 10). So, behind every trigger, there is a dynamic event that should happen in real time. Static events are called points. Triggers and points can be synchronized together by syncs.

While both types of intervals can start with a fixed point or a trigger, static intervals must end by a fixed point while flexible intervals must end by a trigger.

<footnote n="3">³ This corresponds to what Brian Eno calls “Unfinished pieces” and Philippe Manoury “Virtual pieces”.</footnote>
In fact, a flexible interval has two durations: a static duration, which is defined at the time of composition, and a dynamic duration, which is defined at the time of execution. Its static duration is defined by the interval between its beginning and its end on the timeline. As the flexible interval is used to wait for a dynamic event at execution time, its dynamic duration depends on the moment of arrival of this event. It is defined by the interval between its beginning and its end at the moment of execution, i.e. the moment of appearance of the expected dynamic event. In addition, a flexible interval may have a maximum duration that will force its end even if the dynamic event does not happen.

By contrast, the duration of a static interval is the same at composition and execution time.

On figure 10, flexible intervals are displayed in dashed lines, static interval are represented as solid lines, and maximum durations as right parenthesis.

<figure><img src="image_10.png" /><figcaption>Fig. 10. Flexible intervals in OSSIA score.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 11. Conditions in OSSIA score.</figcaption></figure>

## Other temporal features

The first version of the software could only manage linear scripts. Indeed, all the elements of the script had to be executed and only them. To switch to non-linear scripts, it implied to handle a tree-like time by introducing conditions.

A good example is the tree-like script of the movie by Alain Resnais (1993) "Smoking No smoking" based on the play Intimate Exchanges by Alan Ayckbourn (1982)<fnref n="4" />. It consists of several stories all originating from a single opening scene. As the play progresses the characters make choices, each of which causes the story to go in one of two directions, leading several possible endings. The model of multiple scenarios is an interactive version where you can yourself make the choices in real time and watch only one of the possible stories in a continuous way.

This model was inspired from the block universe in quantum physics, especially the theory of relative states. In this theory, the Neutral universe comprises all the possibilities predicted by quantum theory (superposition of physical configurations of different amplitudes) and the states are affected with an amplitude of existence given by its probability. In the same way, in the

<footnote n="4">⁴ For example: http://nezumi.dumousseau.free.fr/film/smoking.htm</footnote>
model of OSSIA score, all the scenarios exist but during execution only those which have a sufficient amplitude of existence
are played.

Thus, in OSSIA score, each interval can be assigned a condition. They are displayed as it is shown in figure 11. On this figure,
there are two intervals associated with sound processes synchronized and conditioned. The two conditions are evaluated at the
same time.

At last, OSSIA score provides also temporal loops. In the example shown on figure 12, there is one process with a loop coming
from its end to its beginning. It is embedded into a process with a trigger at its end creating a flexible interval named
interval.keys75. Moreover, the blue line of the flexible interval finished with a right parenthesis gives a maximum duration for
the flexible interval. This will force the end even if it has not been triggered. So the loop takes place during the interval.keys75
and stops at the same time as it, either if the end is triggered or forced.

<figure><img src="image_12.png" /><figcaption>Fig. 12. Loops in OSSIA score.</figcaption></figure>

## 3.2 Functional Graph

OSSIA score processes have inputs and outputs that can be interconnected, thus forming a functional graph. The functional graph can be displayed combined with the time graph or on its own.

<figure><img src="image_13.png" /><figcaption>Fig. 13. Time and functional graph in OSSIA score.</figcaption></figure>

### Representation of the time graph combined with the functional graph

Figure 13 shows the example of the two tracks A and B, each consisting of a sequence of three sounds. In the graphical user interface (GUI) of OSSIA score, each track is represented as a scenario including the three sounds. Time flows from left to right.
The track A is made up of the sounds $s_1, s_2$ and $g(s_2)$, where g is a low-pass filter applied to $s_2$. There is a red jack linking the output of $s_2$ with the input of $g(s_2)$. So in the third interval, the sound of $s_2$ is played with this filter. Note that the application of the filter is placed in time, that means that it is activated only during this time interval, unlike Max/MSP or PureData patches, for example, which are executed from the beginning to the end of the program. In OSSIA score, delays are automatically set up according to the temporal relations between arguments and functions.

In the same manner, the third interval of the track B, is a high-pass filter applied to the first sound $s_4$.

## Representation of the functional graph

In addition to the representation melting time and functions, OSSIA score provides a pure functional graph view that is more suitable when the scenario has a lot of effects.

The example shown on figure 14 represents a scenario made of two intervals. The first one is embedding a sound, an automaton curve and sound effects. The second interval is played after the first one and it contains a delay which takes as input the output of the low-pass filter (LPF) embedded into the first interval.

When clicking on the graph icon on the left at the bottom of the first process, the functional graph is displayed out of time. In this example, it is made of the sound, the automaton curve that are used in input of the distortion and the low-pass-filter, as it is shown on figure 15.

## 3.3 Hierarchical Organisation

In OSSIA score, the musical elements are nested in a hierarchical way. The musical hierarchy is not displayed as such, but the user can navigate within it by following the hierarchical links. To do this, she or he simply has to click on the names of the elements that appear on the time interval banner.

Let us take again the example shown in figure 14. When clicking on the name of an interval, a foreground view of its contents is displayed, like it is shown on figure 16. This operation allows to navigate within the hierarchy of scenarios top down or bottom up thanks to the path displayed on the time interval banner.

Thus, this representation only provides a way to navigate in the hierarchy of processes, but not to modify its structure. An explicit graphical representation of the hierarchy such as the one shown on figure 5 would allow editing operations to be performed on its structure, such as moving, copying, and deleting its components.

<figure><img src="image_14.png" /><figcaption>Fig. 14. A hierarchical scenario in OSSIA score.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Fig. 15. Pure functional view of a scenario in OSSIA score.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 16. Hierarchical navigation in OSSIA score.</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Fig. 17. Two formal diagrams (paradigmatic) at two different structural levels of Igor Stravinsky's Symphonies of Wind Instruments (1920), beginning (0 - 50'), in a Digital Audio WorkStation (Digital Performer).</figcaption></figure>

# 4 Structuring Musical Pieces for Analysis in MORFOS

This section presents a 3D representation that allows to combine several musical aspects in the context of musical analysis. Jean-Marc Chouvel [8] has proposed the Cognitive algorithm which is a model for performing a paradigmatic analysis of musical pieces over time. The formal diagram is the result of the cognitive algorithm and it represents musical paradigms as a function of time on different time scales.

The MORFOS software<fnref n="5" /> (Multi-scale Oracle Representations for Organized Sounds), developed by Joséphine Calandra [3], is an implementation of the cognitive algorithm on several hierarchical levels.

## 4.1 Multi-scale Formal Diagram

Figure 17 displays two formal diagrams at two different levels of the beginning of the Igor Stravinsky's Symphonies of Wind Instruments (1920). On the vertical axis, paradigms (or materials) are differentiated on a distinctive line of the sequencer. They represent a class of musical elements sharing a sufficient degree of similarity. The horizontal axis represents time.

The algorithm consists of examining the elements one after the other. When a new element arrives, if it constitutes a new material, an additional line is created in the diagram. On the contrary, the element has already appeared and it is placed on the line to which it corresponds.

To obtain the diagram according to several time scales, the same algorithm is applied at each level to build its upper level. The algorithm executes the following steps:

* Acquisition of the elements one after the other;

* Pattern segmentation test;

* Pattern classification test.

<footnote n="5">⁵ https://github.com/jcalandra/morfos</footnote>
The tests are performed according to the rules chosen by the musicologist.

## 4.2 3D Model

If we superimpose the different levels of the multi-scale diagram, we obtain three axis that are: the time axis, the material axis and the level axis. Therefore, a single 3D object is obtained. Those three axes define three planes.

* The plane of the musical form is obtained from the time axis and the materials axis. This is the formal diagram at different hierarchical levels and it shows the evolution of musical language over time and allows to study the effects produced when listening from the use of the materials.

* The plane of the musical structure is obtained from the time axis and the level axis. It shows the temporal organisation between the different time scales for different materials, and therefore in proportion to the durations of the elements considered.

<figure><img src="image_18.png" /><figcaption>**Fig. 18.** Two multi-scale formal diagrams of the two tracks example assuming on the left that *g* and *h* do not change the paradigms of the musical objects, and assuming the contrary on the right for *g*' and *h*'.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>**Fig. 19.** Musical form projection of the two tracks example.</figcaption></figure>

* The plane obtained from the material axis and the level axis is called the organization plane. It describes the dictionary and the grammar of the language used in the musical piece and the way it is articulated between the different levels of the structure at each moment. It can be interpreted as an imprint of the musical piece outside of time, as its *system*.
This 3D model provides some clarification on the difference between form and structure. In fact, in this model they are two different projections of the same 3D object. The form focuses on temporal relations between materials at different levels, while the structure focuses on hierarchical and temporal relations for different materials [4].

It is therefore interesting to examine projections or restrictions of this 3D representation to study the 3D object in detail, especially the temporal, structural and the organization planes.

## 4.3 Temporal Plane

Let us assume that the musical objects in tracks *A* and *B* that are simultaneous are differentiable according to criteria that do not depend on time, which allows them to be separated into two different paradigms. Figure 18 represents the multi-scale formal diagram of this example where we suppose, on the left that the applications of the functions *h* and *g* do not change the paradigm of the objects, and on the right, we assume the opposite for the functions *g'* and *h'*. It would also be possible to consider establishing a level of similarity with objects already heard.

* $g(s_2)$ is in the same paradigm than $s_2$, while $g'(s_2)$ is not of the same paradigm than $s_2$.

* $h(s_4)$ is in the same paradigm than $s_4$, while $g'(s_4)$ is not in the same paradigm than $s_4$.

In this case, the multi-scale formal diagram has two levels, the one of the musical objects and the one of the tracks that are superimposed.

On such a representation, it is possible to draw the transformations *h* and *g* in the same way than it is done in the interface of OSSIA score, even though information on transformations is not currently available in MORFOS. In the same way as in OSSIA score, Allen's temporal relationships are represented explicitly on this temporal plane.

In the following, we focus on the leftmost formal diagram of figure 18. Let us examine on figure 19 the projection on time and materials axes of this diagram giving the musical form. The hierarchy information is projected on the plane. This representation is quite similar to that of the OSSIA score software shown on figures 14 and 16 that enables navigation in the hierarchy, by clicking on the banner giving the hierarchical path.

<figure><img src="image_20.png" /><figcaption>Fig. 20. Structural plane of the two tracks example (left) and its hierarchical representation (right).</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Fig. 21, Restriction of the structural plane to the first track.</figcaption></figure>

## 4.4 Structural Plane

The structural plane is represented on figure 20. We obtain a complete segmentation at the first level. The membership relationships from elements of one level to the upper level are figured by vertical down-arrows.

If we restrict the projection to the first track, that is to the first material $s_1$ and the third material $s_3$ and $g(s_2)$, we can observe on the figure 21 the links between the levels 0 and 1. Moreover, the link of the transformation *g* can be represented between $s_2$ and $s_3$. It would even be possible to represent links of transformations between contents of different levels. Thus, it would be particularly relevant for the analysis to represent the similarity and transformation information on this level.
## 4.5 Organization Plane

The organization plane of the example is represented on figure 22. This representation is without time, so that there is only one occurrence of each material. This represents the nesting between fragments and their hierarchical organization.

This graph is a hierarchical graph to compare with the one of figure 5. This projection has more information about the chronological order of appearance because materials are listed according to their chronological order of appearance. But all the occurrences of each material are melted into a single material representing all of them.

# 5 Discussion

Let us try here to compare in more details the representations presented in the previous sections. This exercise has its limitations because we are considering musical pieces from the perspective of listening on the one hand and from the perspective of composing on the other, that are two points of view very different. However, composition often relies on listening, and analyzing also raises questions about production and they share a common object which is the musical piece.

First of all, the concept of paradigm and thus of class of material is completely absent of the representations of OSSIA score. Indeed, objects in OSSIA score are either sounds or functions. However, the notion of paradigm could seem relevant if it is limited to sounds. Thus, this concept could also apply to the results of processes, thereby providing a resulting structure defined especially on the data and organizing it as a multiscale formal diagram. So, this would bring the 3D space and the three planes with all their new perspectives of features and representations.

This situation would give rise to a dual vision of the musical piece, on the one hand functional and on the other hand data-driven. In the context of a composition editor, the question arises as to whether data or functions should take priority in editing. The structural organization appears to be quite similar in both models. It shows the hierarchical organization of materials over time.

The hierarchical organization out of time in MORFOS is not present at all in OSSIA score. Probably it would be useful as soon as there is a multi-scale formal diagram. However, this representation would provide a way to modify its structure thanks to editing operations such as moving, copying, and deleting its components.

<figure><img src="image_22.png" /><figcaption>Fig. 22. Organization plane of the two tracks example.</figcaption></figure>

Moreover, consideration should be given to the use of hierarchy to define levels of interaction. Currently, the hierarchy of scenarios is used as a means of modularizing a musical piece. However, hierarchical organization could be exploited in conjunction with OSSIA score's hybrid time model to implement new modes of interaction. The idea would be to assign intervals, and therefore the processes they carry, levels of importance depending on their hierarchical level, as in musicology. Thus, triggers would take on the level of importance of the intervals they trigger and would therefore also be hierarchical. This hierarchy of triggers could become an interactively navigable object. This would allow a musician to switch in real time from an expert level, where all low-level triggers must be executed, to a conductor level, where only triggers associated with the longest segments would be activated by the musician.

The functional graph is completely absent of the representations of MORFOS. Nevertheless, as we have shown previously, certain projections or restrictions of the 3D representation would make it possible to show transformations of materials, which could be very useful in the context of analysis. However, one may question the relevance of representing this functional graph
on its own and outside of time, as it is done in OSSIA score. Might such a graph showing functional relations between materials outside time form a fourth dimension? In this case, materials should be considered as differentiated paradigms to represent their functional links, for example by means of a self-similarity matrix.

At last, the time graph in MORFOS is merged on the timeline because all events are static, and therefore they are completely ordered.

Taking into account a hybrid timeline such as that of OSSIA score with flexible intervals and dynamic events, a formal diagram could represent interactive pieces.

# 6 Conclusion

This article focused on various representations of music. The three graphs that are presented are not new, but until now, they were understood as different points of view of the same object but they were not linked together. Indeed, those graphs are complementary and seem to correspond to different dimensions of music. The 3D model obtained in music analysis provides evidence in favour of this hypothesis.

These three graphs are used to represent music in the context of composition, performance and analysis, each providing essential structures in the relevant context. Highlighting them and linking them in a single model should bring a better understanding and contribute to the development of more powerful tools in those three contexts.

Analyzing a piece of music in a compositional context could enable the musicologist to make manual changes to the representation, and additionally to study variations and spaces interpretation. He could also study the process of creating the musical work.

Composing a piece in an analytical context could provide the composer with a better understanding of the structure and form of the piece being created, and features to edit his piece at a higher musical level. It could also provide and perhaps even assist the creative process by suggesting automatic completions.

Furthermore, by using a hybrid time model in musical analysis, it would become possible to consider dynamic events and analyze interactive pieces. In particular, taking into account the hierarchical level of interactions would bring new developments in both composition and analysis.

These formal models could also serve as a basis for defining properties that would make it possible to compare and classify musical pieces according to various criteria.

# Acknowledgements

This research was carried out within the framework of the SCRIME of the University of Bordeaux (Studio de Création et de Recherche en Informatique et en Musique Expérimentale) and funded by the French Ministry of Culture.

# References

[1] J. F. Allen. "Maintaining knowledge about temporal intervals". In: Communications of the ACM 26.11 (1983), pp. 832–843.

[2] A. Allombert et al. "Virage : designing an interactive intermedia sequencer from users requirements and theoretical background". In: Proc. of the International Computer Music Conference 2010, ICMC2010, New-York, USA. 2010.

[3] J. Calandra. "L'algorithmie cognitive et ses applications musicales". Theses. Sorbonne Université, Dec. 2023. https://theses.hal.science/tel-04581505.

[4] J. Calandra, J-M. Chouvel, and M. Desainte-Catherine. "MORFOS : Viewing Form, Structure and Organisation of the Cognitive Behaviour while Listening to Music". In: submitted to Journal of New Music Research (JNMR).

[5] Joséphine Calandra, Jean-Marc Chouvel, and Myriam Desainte-Catherine. "Hierarchisation algorithm for MORFOS : a music analysis software". In: Proceedings of the International Computer Music Conference. Boston, USA, 2025.

[6] Jean-Michaël Celerier, Pascal Baltazar, and Myriam Desainte-Catherine. "ossia score 3". In: Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2022. Ed. by Jonathan Bell Vincent Tiffon and Charles de Paiva Santana. Marseille, France: PRISM Laboratory, 2015, pp. 81–90. ISBN: 979-10-97498-03-0.
[7] Jean-Michaël Celerier et al. "OSSIA: towards a unified interface for scoring time and interaction". In: Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. ISBN: 978-2-9552905-0-7.

[8] J-M. Chouvel. Analyse Musicale, sémiologie et cognition des formes temporelles. L'Harmattan, Paris, 2006.

[9] Miller Puckette. "Pure Data: Another Integrated Computer Music Environment". In: Proceedings of the 1997 International Computer Music Conference. 1997, pp. 37–40.

[10] Heinrich Schenker. "The Nature of Musical Experience". In: Schenker Studies. Ed. by Hedi Siegel. Cambridge University Press, 1990, pp. 1–20.

[11] Mc Taggart. The Nature of Existence. Cambridge at the University Press, 1927.

[12] Mc Taggart. "The Unreality of Time". In: Mind 17 (1908), pp. 457–474.