# From the Ontology of Video Games to the Epistemology of Digital Movements. Towards a Semiotics of Virtual Practices

Enzo D'armenio

► To cite this version:

Enzo D'armenio. From the Ontology of Video Games to the Epistemology of Digital Movements. Towards a Semiotics of Virtual Practices. Games and Culture, 2024, 10.1177/15554120241263630. hal-05310103

HAL Id: hal-05310103

https://hal.science/hal-05310103v1

Submitted on 11 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# From the Ontology of Video Games to the Epistemology of Digital Movements. Towards a Semiotics of Virtual Practices

Enzo D'Armenio, FNRS Postdoctoral Researcher, University of Liège

enzo.darmenio@uliege.be

In this paper, we intend to construct an original integration between the ontology and hermeneutics of video games proposed in the field of game studies and the post-structuralist approach of contemporary semiotics. The aim is to propose a new epistemology of video game and virtual reality experiences grounded in the concept of movement. These media will be reconceptualized as image-movements: expressive systems that rely on the interrelation of two sets of qualities, visual qualities and kinetic qualities. First, we will perform an integration between the concept of ergodic literature developed in the field of game studies and recent developments as regards a semiotics of practices. Starting from this general convergence, we will rely on the meta-ontology of video games proposed by Espen Aarseth and Paweł Grabarczyk to build a theory capable of integrating as many dimensions as possible around a kinetic explanatory core.

Keywords: video games, semiotics, ontology, hermeneutics, movement

## Introduction

This paper proposes a new semiotic epistemology for the study of virtual experiences of video games, based on the analysis of physical and virtual movements made by players.

Semiotics applied to video games has a history of about 20 years now. Its debut can be located in correspondence with the publication of *Semiotica dei videogiochi* (2004) by Massimo Maietti, largely inspired by Umberto Eco's theory of textual cooperation (1979) and in particular by its declination of the properties of possible worlds. Since then, semiotic studies have multiplied: among general contributions, some have proposed the construction of a
ludosemiotics (Thibault, 2020) for the study of game interactions or have attempted to define broad theoretical frameworks for the analysis of interactive narratives (Ferri, 2015). More specific studies have focused on the analysis of particular aspects: videogame genres such as *god games* (Meneghelli, 2007), peculiar points of view such as in the study of the architecture of virtual spaces and their contribution to the game experience (Aroni, 2022) or videogame experiences that particularly stimulate the player's body and senses (Meneghelli, 2011). Some of these approaches are situated in continuity with respect to the semiotics of Umberto Eco and to the more recent cognitivist readings (Giuliana, 2018), while others choose to privilege the paradigm of the Paris school, represented by the works of Algirdas J. Greimas (Post, 2005), or develop in accordance with the proposals of Roland Barthes (Compagno, 2013).

Starting from this framework, our paper intends to present a specific perspective: that of a semiotics of virtual practices. Our aim is not to propose a new definition of video games or of virtual experiences, but to propose a new theory that can reposition these notions, and related debates, around a new explanatory core: that of virtual movement. Through a close dialogue between the contemporary semiotics of practices and a specific current of game studies, the hermeneutics (Aarseth & Calleja, 2015; Aarseth & Möring, 2020) and ontology (Aarseth, 2014) of games pursued by Espen Aarseth and his collaborators, we will propose to understand virtual experiences as kinetic images, or as image-movements: expressive systems based on the complex interrelation of two sets of qualities, the visual qualities and the qualities of movement.

This theoretical hypothesis is based on two basic questions concerning virtual experiences and the construction of their meaning. First of all: how do we, as users, access these virtual experiences from our position within ordinary, everyday experience? In the case of virtual experiences, access to meaning is certainly visual: on the screen, there is the opening of another scene lying beyond the space of ordinary experience. And yet, the user, in order to access the virtual experiences, to make them work, must make movements on one or more physical
interfaces (using joypads or virtual helmets) that will be transformed into movements within
the virtual world<fnref n="1" />. The second question, which is more complex to untangle, asks: how is this
dual visual and kinetic access then articulated in local solutions? In this article, we will try to
answer both questions in more detail.

First, we will perform an integration between the concept of ergodic literature (Aarseth, 1997) and recent developments as regards a semiotics of practices (Fontanille, 2008). Both of these epistemological frameworks have moved away from the traditional conception of the text in order to account for the open, partially unpredictable and processual features of game practices.

Starting from this general convergence, we will rely on the ontology of video games
proposed by Espen Aarseth and Paweł Grabarczyk (2018) to develop a new theory of virtual
practices, based on the interrelation between two syntaxes: visual syntax and kinetic syntax.

Finally, thanks to the discussion of a case study, *The Legend of Zelda: Breath of the Wild*,
we will begin to develop a methodology suitable for describing video game practices, and in
particular for analyzing the interpretative, cognitive and embodied processes resulting from the
exploitation of visual and kinetic syntaxes.

1. Ergodic literature within the framework of a semiotics of practices

With the concept of interaction, the scientific literature intends to distinguish media such
as video games and virtual reality from traditional media such as literature and film. The
interaction of video games would allow the viewer to participate in the experience, to make

<footnote n="1">¹ Movement has already been thematized within game studies, notably by Torben Grodal (2000), Gordon Calleja (2011), and Graeme Kirkpatrick (2011). Our perspective does not take a cognitivist approach, nor does it aim to explore the issue of immersion effects, although we are convinced that these effects are a consequence of movement. We will return to Kirkpatrick’s hypothesis later.</footnote>
active choices, as opposed to a movie or book in which the viewer adopts a passive role<fnref n="2" />.

However, the scientific literature has not fully clarified what distinguishes this meaning of
interaction from interpretative interaction. The now classical concepts of ergodic literature
proposed by Espen Aarseth (1997) allow us to understand this difference. The conceptual core
of the proposal sees two opposing conceptions of the text: a perspective that begins with literary
narratology and then a perspective that denounces the latter's shortcomings in the light of the
inherently reconfigurable nature of video games.

The performance of their reader takes place all in his head, while the user of cybertext also performs
in an extranoematic sense. During the cybertextual process, the user will have effectuated a semiotic
sequence, and this selective movement is a work of physical construction that the various concepts
of “reading” do not account for. This phenomenon I call *ergodic*, using a term appropriated from
physics that derives from the Greek words *ergon* and *hodos*, meaning “work” and “path”. In ergodic
literature, non-trivial effort is required to allow the reader to traverse the text. (Aarseth, 1997, p. 1)

On the one hand, ergodic literature differs from standard literature in the mechanics it requires
from its “reader”. It is no longer a matter of a reading that is resolved in the mind of the
individual against a negligible effort at the physical level, which concerns “eye movement and
the periodic or arbitrary turning of pages” (Aarseth, 1997, p. 2). The cybertexts require a non-
trivial effort to be traversed, i.e. an active exploration through their medial materiality, and
which we could simplistically refer to as a series of inputs and outputs, feeding interactive

<footnote n="2">² Whitin semiotics, Agata Meneghelli (2007, p. 97), following in the footsteps of Giovanna Cosenza (2004, p. 51), has proposed to consider face-to-face verbal dialogue as a benchmark for establishing the degree of interactivity in video game experiences. In general, however, this concept risks configuring itself as an umbrella term. For the purposes of this paper, we consider as decisive the critiques of interaction set forth by Espen Aarseth (1997, pp. 47-51).</footnote>
mechanics between player and software. Aarseth insists that the difference between traditional
texts and cybertexts lies in the fact that in the latter, “each decision will make some parts of the
text more, and others less, accessible, and you may never know the exact results of your choices;
that is, exactly what you missed” (Aarseth, 1997, p. 3). This difference has been further
explored by Makku Eskelinen:

[...] the dominant user function in literature, theatre and film is interpretative, but in games it is the configurative one. To generalize: in art we might have to configure in order to be able to interpret whereas in games we have to interpret in order to be able to configure, and proceed from the beginning to the winning or some other situation. (Eskelinen, 2001, § 2)

What Aarseth and Eskelinen want to emphasize is the specific nature of the reading
process of cybermedia compared to traditional media. With respect to this framework, we are
interested in emphasizing two aspects of the relationship between “dominantly interpretive
practices” (Eskelinen, 2001, §2) and cyber practices. While it is true that the latter require an
effort of configuration that selects certain possibilities of physical-material traversal to the
exclusion of others, it must be recognized that the semantic disambiguation typical of literary
textualities also characterizes virtual experiences. In other words, the first feature that
distinguishes cybermedia is the need to activate mechanisms of material traversal, and yet even
in the course of their “reading”, mechanisms of interpretative disambiguation must be activated.
The stakes of our reflection are twofold: on the one hand, to characterize more precisely the
“non-trivial effort” of traversal that is specific to virtual experiences, and on the other hand, to
understand the overall effects on their interpretation.

In an attempt to answer these two challenges, we intend to mobilize contemporary
semiotics, and in particular the current epistemological paradigm founded around the study of
practices. The definitive affirmation of this new approach took place in the 2000s, and was
systematized in the 2008 book *Pratiques sémiotiques* by Jacques Fontanille. The crucial aim of this proposal is to analyze experiences of meaning-making well beyond literary or artistic texts, and more generally beyond meaning understood in accordance with an object paradigm. By means of the principle of immanence postulated by Louis Hjelmslev (1961) in structural linguistics, the different types of experience are to be traced back to planes of formal descriptiveness: the figurative experience of sign recognition, the interpretative experience that governs textual meaning, the corporeity of objects, the practical scenes within which objects, individuals, and the interpreter as such are arranged, and finally, the strategies and forms of life.

"Overall, it is the conversion of an experience (and of a phenomenology) into a semiotically relevant device of expression, i.e. one that can be associated with a content plane" (Fontanille, 2008, p. 35, our translation). One of the key premises of this approach is related to the depowering of textual epistemology. According to Fontanille, Hjelmslev's principle of immanence was mistakenly identified with textualism. This theoretical slip produced a paradoxical situation: from the 1980s onwards, semiotic analyses have been carried out regarding a variety of phenomena including underground passenger routes (Floch, 1990), museum spaces (Pezzini & Cervelli, Eds., 2006), and supermarket shopping (Pozzato, 2012). And yet, these experiences of meaning-making that are intrinsically linked to a processual dynamic were treated as texts, albeit in accordance with an unorthodox conception of textuality, that is, understood not as a specific object of meaning (an artistic text or a literary text), but as the procedure of its description. Fontanille's proposal stems precisely from the observation that semiotic analysis has already been applied to objects quite different from the text-utterance and that there is a need to update the theory accordingly: "If it is true, as Hjelmslev says, that the linguist's data are presented as 'text', this is no longer true for the semiotician, who is also dealing with 'objects', 'practices' or 'forms of life' that structure entire sections of culture" (Fontanille, 2008, p. 15, our translation).
Fontanille's proposal is to consider different experiences of meaning organized according to an ascending order of relevance, and then to transform them into formal planes on the basis of Hjelmslev's principle of immanence (see Table 1).

<table><caption>Table 1. Levels of relevance in semiotic analysis. Excerpt from Fontanille (2008, p. 34).</caption><thead><tr><td>Type of experience</td><td>Formal instances</td><td>Interfaces</td></tr></thead><tbody><tr><td>Figurativity</td><td>Signs</td><td>Recurring formants</td></tr><tr><td rowspan="2">Interpretational coherence and cohesion</td><td rowspan="2">Utterance-texts</td><td>Figurative isotopies of expression</td></tr><tr><td>Enunciative/inscriptional device</td></tr><tr><td rowspan="2">Corporeality</td><td rowspan="2">Objects</td><td>Formal substrate of inscription</td></tr><tr><td>Praxis morphology</td></tr><tr><td rowspan="2">Practice</td><td rowspan="2">Practical scenes</td><td>Predicative scene</td></tr><tr><td>Processes of accommodation</td></tr><tr><td rowspan="2">Conjuncture</td><td rowspan="2">Strategies</td><td>Strategic management of practices</td></tr><tr><td>Iconization of strategic behavior</td></tr><tr><td>Ethos and behavior</td><td>Forms of life</td><td>Strategic styles</td></tr></tbody></table>

English transl. in Dondero (2020, p. 134).

To the sign-level experiences of the recognition of figurative formants (which can be analyzed by means of the commutation test), one must add the textual experiences of interpretation (which can be analyzed by finding semantic recurrences and by considering the plastic plane), and then the level of the objects and of the supports upon which the text is imprinted (which can be one-, two- or three-dimensional), until one reaches the level of practice. Practice is the level that accounts for the open and unpredictable interactive relationship between human operators and texts or situations.

However, the level of relevance of practices occupies a very special place within this epistemological proposal: “only the level of practices includes in its definition this principle of being ‘through action’, which precisely allows it to exploit the living schematizations and
emerging models identified on the other planes of immanence” (Fontanille, 2008, p. 118, our translation). It is also the only level that contains within it an interpretative instance that regulates its unfolding, even when the practice is not communicative-dominant. It follows that any kind of interpretation, including semiotic analysis, is a practice that seeks adaptation with respect to its horizon of reference, be it a text to be analyzed or an experience of some other kind. If practice is the only level capable of accounting for experience and interpretation “through action”, it is the starting point of any phenomenon of meaning-making, and not just one level among others: whether it is the level of sign, of textual or of strategic semiosis, it is always situated in an ongoing practice or experience. In other words, according to our point of view, practice corresponds to semiosis itself: an open and constitutively interpretative event, which requires a revision of the methodology of analysis. To this end, Fontanille proposes a process divided into four very general stages, which represent the syntagmatic structure of any practice: the meaning gap, the schematization, the regulation and the adjustment. The meaning gap is “the phase of confrontation between the practice and its alterity, which thus involves an experience of resistance (or non-resistance), strangeness (or familiarity), congruence (or incongruence), etc.” (Fontanille, 2008, p. 133, our translation). Fontanille gives the example of finding oneself in a lift with other people, a situation that typically demands meaning. Instead, schematization is the moment when the situation “is analyzed (or only felt) in its resistances and its most salient zones of otherness” in search of “an ‘organizing scheme’” (Fontanille, 2008, p. 133, our translation). This is followed by the adjustment phase, which consists in appraising the situation and seeking a schematization mainly in modal terms (“must do” vs. “can do”, for instance). Finally, the accommodation is the moment when, by applying schematization and adjustment to the practical situation, it is given (or not) a meaningful final form. Among these four phases, that of schematization is, in our opinion, particularly important, because it presents elements capable of globally describing its inflection. In fact, two modes of schematization are
distinguished: when it is developed through summoning an already known situation, the result of which is also already known, Fontanille speaks of heterodirected schematization, whereas in the case of an innovative conduct, there will be a self-adaptive schematization.

The consequences of this approach cannot be underestimated. If we take the example of a literary text, the minimal core of analysis will no longer be the structure of the text itself, the lexical choices used, or the narrative progressions that are definitely fixed. The minimal core will consist of the practical scene that includes a human interpreter (the reader), a media object (the book), and the textual structures (in this case verbo-literary). The act of reading will consist of a process of accommodation, in this case heterodirected through the filter of the verbal language code and the highly normed practice of reading. And yet, this approach opens up the field to dissonant readings of a text, to readings that emphasize, for example, the search for certain information among others, to public readings involving an accommodation with other actors and the modulation of the tone and rhythm of the voice, and so on.

At this stage of generality, it seems to us that the semiotic epistemology of practices lends itself particularly well to the study of ergodic literature. Both approaches reject the conferral of theoretical centrality to a traditional conception of textuality, which considers it sufficient to account for its narrative and semantic structures. Both point to the analysis of the materiality of media objects, of the processuality of interactions between interpreters and objects, and of the open and partially unpredictable nature that characterizes interaction.

## 2. From Ontology to Theory: Virtual Practices as Image-Movements

However, the dismissal of the traditional concept of textuality in favor of a practical epistemology is not sufficient to explain the construction of meaning in virtual experiences. In order to build our theory, we will rely on the ontological reflection carried out by Espen Aarseth and his collaborators. According to Aarseth, there are two different types of game ontologies:
(1) Formal or descriptive ontologies, asking what are the functional characteristics and components of game objects, and the relations between them; and (2) existential ontologies asking what are games and what *kind* of existence does a game have. (Aarseth, 2014, p. 484)

The first type is that which is best suited to support the purposes of this article, especially
due to the recent proposal of a meta-model capable of summarizing the fundamental dimensions
that characterize the existence of video games (see Table 2). Espen Aarseth and Pawel
Grabarczyk, in a recent study entitled "An Ontological Meta-Model for Game Research"
(2018), explicitly designed an overall investigation of the characteristics of video games that
can give "researchers a useful map of shortcuts connecting their specific research with the
research of their peers and provokes interesting research questions – for example: how does a
regularity I discovered on level x manifest itself on level y?" (Aarseth & Grabarczyk, 2018, p.
5).

The theoretical assumption behind this typology is to understand video games, and games
in general, as mechanisms: "We argue that the best way to achieve these goals is to treat the
object of game studies in terms of mechanism. On the surface, this idea may seem trivial, but
in practice it has very useful methodological consequences. Most mechanisms contain multiple
levels of description" (Aarseth & Grabarczyk, 2018, p. 4).

The meta-model consists of four main layers, each articulated in turn into four sublayers.
The physical layer concerns the material objects necessary for the existence of the games and
is divided into platforms ("a console, a computer, gaming board" (Aarseth & Grabarczyk, 2018,
p. 6)), the physical interface ("a gamepad, a joystick or a baseball bat" (Aarseth & Grabarczyk,
2018, p. 6)), and a behavioral substrate, that is, "the set of physical actions needed to play the
game (for example pushing the buttons, moving a piece on a board or kicking a ball)" (Aarseth
& Grabarczyk, 2018, p. 6).
The structural layer encompasses different aspects: first, the computational dimension, which can be summarized by the notion of digital code; the mechanical dimension pertains to what is commonly referred to as game mechanics; and finally, the economic dimension, which concerns, for example, “the coin the player has to put into an arcade machine to start a game” (Aarseth & Grabarczyk, 2018, p. 6) or free-to-play games.

The third layer is the communicational layer and consists of: the presentational sublayer, “which refers to the aesthetic aspects of the game” (Aarseth & Grabarczyk, 2018, p. 7), the semantic sublayer, pertaining to “any communicated semantic information, from a simple command to a whole narrative” (Aarseth & Grabarczyk, 2018, p. 7), and the interface sublayer, “which refers to non-diegetic information communicated to the player” (Aarseth & Grabarczyk, 2018, p. 7).

The mental layer represents the fourth and final layer of the meta-ontological grid and is also organized into three sublayers: the phenomenal sublayer concerns “the way the game is experienced by the player” (Aarseth & Grabarczyk, 2018, p. 7); the conceptual sublayer “which refers to the way the player understands (conceptualizes) the game” (Aarseth & Grabarczyk, 2018, p. 7); and finally, the social layer “which refers to the way players interact and perceive each other in the game” (Aarseth & Grabarczyk, 2018, p. 7).
<table><caption>Table 2. The meta-ontological model proposed by Aarseth and Grabarczyk (2018, p. 7).</caption><thead><tr><th>Main layers</th><th>Sub-layers</th></tr></thead><tbody><tr><td rowspan="3">Physical</td><td>Platform</td></tr><tr><td>Hardware Interface</td></tr><tr><td>Behavioral</td></tr><tr><td rowspan="3">Structural</td><td>Computational</td></tr><tr><td>Mechanical</td></tr><tr><td>Economical</td></tr><tr><td rowspan="3">Communicational</td><td>Presentational</td></tr><tr><td>Semantic</td></tr><tr><td>Interface</td></tr><tr><td rowspan="3">Mental</td><td>Phenomenal</td></tr><tr><td>Conceptual</td></tr><tr><td>Social</td></tr></tbody></table>

It is clear that this model aims to isolate as many dimensions as possible with respect to various approaches, and that different disciplines will carve out particular subdivisions within their theories, or merge multiple layers in accordance with their explanatory assumptions. If we stand from the perspective of a semiotics of practices, and more specifically with respect to a human operator, we can carve out these layers horizontally, and ask how his or her overall experience of meaning is activated by a video game or virtual experience. The minimal practical scene will consist of the presence of a human, a physical interface, a console or PC platform, and a specific video game or software. To understand the dynamics of a practice, according to Fontanille, one must first outline a predicative scene: an action internal to the practice that can be treated as a verb. It will be the theme of action that will distribute the “actantal positions”. In other words, two basic questions about meaning in virtual experiences arise: how do we, as users, access these virtual experiences from the position of our everyday experience? In the case of virtual experiences, certainly the access to meaning is visual: on the screen, there is an
opening onto another scene beyond that of ordinary, worldly experience. And yet, the user, in
order to access the virtual experiences, in order to make them work, must make movements on
one or more physical interfaces (using joypads or virtual helmets, for instance) that will be
transformed into movements within the virtual world (D'Armenio, 2014). In other words,
access to the meaning of virtual experiences is based on two closely interrelated macro-acts:
that of perceiving a scene delivered by devices and usually articulated in more or less figurative
virtual spaces, and a kinetic performance, through one's movements on the interface, which
will be transformed into virtual movements. For these reasons we propose to understand video
games and virtual experiences as image-movements, that is, expressive systems articulated
along two syntaxes: a visual syntax and a kinetic syntax<fnref n="3" />.

The preliminary result of this hypothesis is that interactive media develop the dual syntax
of the image-movement into a complex form: on the one hand, they build traditional forms of
meaning-making, relying on the conventions of literary, visual or cinematic genres. This is the
case with cutscenes, dialogues, and narrative junctures that also occur with the help of verbal
language. On the other hand, another kind of meaning emerges from the practice of movement,
adding another layer of meaning, that is, the movements performable by the spectator. For
instance, in the *Assassin's Creed* saga, and in particular in the second chapter (Compagno,
2013), a conspiracy plot in a Renaissance setting is set up against a system of commands which
allows one to traverse digital cities acrobatically.

<footnote n="3">³ We use the word “syntax” very broadly and generically. We do not mean to reduce the complexity of video game experiences to facts of language or, in particular, to trace them back to the structures of verbal language. We only mean to say that the organization of virtual spaces and figures, as well as the organization of the movements brought into play by each experience, are constituted as forms of grammaticalization: structures fixed upstream that regulate the access to and the unfolding of interaction, but that do not in any way undermine the processual and open-ended nature of these experiences.</footnote>
In this case, the conspiracy thriller provides the background for an acrobatic exploration of cities, with a kinetic obligation to climb monuments and dominate architectures from above, producing vertigo effects in a historical space (Aroni, 2022). It is a very specific and playful way of inhabiting virtual spaces, which configures a mixed generic arrangement between a classical detective and espionage type of narration, and kinetic meaning-making that is rather close to *parkour*. The overlap between visual and kinetic syntaxes builds the peculiarity of this experience. On the one hand, the visual syntax follows the canons of traditional narration, respecting the twists and turns, the investigations, and the motif of the storytelling. On the other hand, the kinetic syntax sets up specific movements, those of *parkour*, installing a precise rhythm of exploration, articulated with the historical scenarios. The overlap between these two syntaxes is particularly evident in the entity that gives access to the exploration of the virtual world: the digital prosthesis is in this case a character with an identity and a narrative background, but it is at the same time a body of movement that expresses itself kinetically through a precise set of thematic movements (run, climb, kill, etc.).

It is also possible for the visual and kinetic syntaxes to work in a kind of dynamic opposition in order to build particular effects of meaning. In *Shadow of the Colossus* (Sony Computer Entertainment, 2005-2018), for example, they build a conflict of modalities through the kinetic obligations: players are required, by means of movement and actions performed through the character-prosthesis, to do something they do not necessarily want to — kill sacred beings called colossi. This conflict between the obligation imposed by the kinetic system through thematic forms of movement (hitting with arrows, with swords, hurting and killing) and the beauty and sacredness of the victims contributes to create a dramatic effect of *pathos*.

## 2.1. *Visual and kinetic syntax beyond the split between gameplay and narration*
It is important to emphasize the fact that the two syntaxes work in interrelation and do not correspond to the traditional division between gameplay and narration. Firstly, this theory allows us to hold together multiple layers of the meta-ontological model proposed by Aarseth and Grabarczyk around a compact explanatory hypothesis. Indeed, kinetic syntax allows us to coalesce the physical layer, the structural layer and the communicational layer. The practical core of kinetic syntax concerns the transformation of the player's movements (behavioral aspect) made on the interface (physical interface), into the virtual movements (presentational and mechanical aspect) based on the operation of the specific software (computational aspect). More generally, according to our hypothesis, the intentionally broad term of “movement” — which includes actions performed on the physical and virtual interface, the movement of cursors, objects and virtual characters, as well as complex actions and reactions concerning them — allows video games and virtual simulations to be understood as kinetic systems. In other words, “the non-trivial effort” that Aarseth spoke of concerns not a generic interaction, nor the eventual choices among possibilities of traversal, but first and foremost the need to move on the physical interface and consequently within virtual worlds. It is through movement that nontrivial traversal enables the selection of some possibilities rather than others, and it is through physical and virtual movement that video game practices construct their meaning.

Visual syntax, on the other hand, concerns the organization of virtual spaces, the characterization of the avatar, the point of view, and the more or less dense figurativeness that characterizes them. For example, a video game can present an abstract visual syntax, which does not present recognizable figures (this is the case with *Tetris*), or it can set up worlds that aim at photorealism, or be organized in an intermediate or mixed way. This dimension exploits the languages of the image, and has long been studied in semiotics and art history<fnref n="4" />.

<footnote n="4">⁴ A recent and comprehensive reference on visual semiotics is Dondero's (2020) volume. Regarding the matter of point of view, and how images construct perceptual and cognitive pathways, see Fontanille (1989).</footnote>
One might think that visual syntax solely pertains to the presentational sublayer: the forms of visual storytelling and its genres used within virtual experiences, cutscenes, character depictions, and so on. And yet, we are convinced that visual syntax is also implicated by the mechanical, phenomenal, conceptual and social sublayers. In a movie, for example, we see actions organized in a narrative succession, a phenomenon that would allow us to classify these media as articulated solely in accordance with an (audio)visual syntax. But in a video game such as *Assassin's Creed II*, when we press the jump button, the avatar performs nearly at the same time the animation that allows us to climb a roof: we are thus perceiving, interpreting, and acting at the same time. The visual realization of movement is not a simple correlate of movement, but partakes to the complex superposition of our physical movement with our exploratory and visual movement in the virtual world. It is for this reason that the two syntaxes do not mirror the division between gameplay and narration. The game mechanics cannot be produced without a visual movement in the game world that embodies, transforms, and fully realizes the player's movements on the interface. We cannot say that jumping, in its visual dimensions, does not belong to gameplay. As gamers, we also and especially play through the visual representation of our motor performance. In other words, there are certainly elements that separately concern the two syntaxes: cutscenes are only visual/narrative, whereas navigation through menus will be predominantly kinetic because the images have a low figurative density. But the action underlying the game mechanics, which is the exploration of virtual spaces through movement, interdependently concerns the visual and kinetic syntaxes.

A final set of aspects of our proposal concern the semantic sublayer and the mental layer. Since our proposal develops from a precise semiotic epistemology, and since its main interest is to describe the meaning of virtual practices, we cannot but place the semantic layer at a more important hierarchical level: being interested in the meaning of virtual experiences, we take the semantic layer to be no less than the most important and global layer.
## 2.2 Aesthetics, Hermeneutics, and Ontology: Movement as mode of existence in virtual practices

Having positioned our proposal within the framework of an epistemology of practices, it is necessary to ask what is the strictly methodological correlate necessary for the analysis phase. In other words, if it is true that access to the meaning of virtual experiences is interdependently visual and kinetic, how can we instead analyze their actual dynamics? On the one hand, movements upon the interface are usually abstract and plastic, because they are not expressed through recognizable figures or themes, but through abstract commands such as up, down, left, right, or using other activation keys (A, B, X, Y) and progressive triggers. Movements within the virtual worlds, on the other hand, often translate the abstract movement upon the interface into a system of figurative and thematic movements (running, jumping, shooting, climbing, etc.). Between these two types of movements, a kinetic diagram<fnref n="5" /> associates the abstract commands upon the interface with more or less figurative and thematic movements within the virtual world. This diagram of kinetic relations establishes the rhythmic and aspectual resonance between the two systems of movement: for example, punctual movements such as jumping within the virtual world may be matched with equally punctual movements upon the interface, while durative movements such as running require prolonged activation of the

<footnote n="5">⁵ We refer to the definition of diagram proposed by Charles Sanders Peirce (CP 2.277): sensible representations that in their internal relations replicate the internal relations of another phenomenon. This is the case, for example, of a map that selects some characteristics of the physical space, such as distance and spatial extension, and expresses them visually by replicating some relations of the real object. But it is also the case of geometrical demonstrations, by means of which it is possible to manipulate graphical traits in order to build new knowledge through visual icons instead of categorical concepts. According to our hypothesis, the relations between the plastic movements made on the interface are translated into the system of kinetic relations expressed by the movements in the virtual worlds through a diagram of kinetic relations.</footnote>
commands. How to analyze this dynamic that directly concerns the semantic qualities of movement? Our hypothesis postulates that it is possible to analyze the kinetic syntax on the basis of two general parameters: the body and the form of movement. The body of the movement designates the substrate of the movement – a body which can be either absent from the frame of the image or abstract, present in the form of a human, animal or object, or constitute itself as an indirectly signified body, inferable from the type of movement exhibited. The form of movement concerns the qualities of the movement itself and involves various degrees of abstraction, ranging from purely plastic movements of trajectories, rhythms, and accelerations, to recognizable motifs of figurative movements (i.e. “falling”), up to thematic movements that can be described through verbal lexicalization (walking, shooting, climbing).

The study of kinetic diagrams makes it possible to clarify some theoretical proposals that have already thematized movement as a relevant dimension of video game analysis, in particular, the decisive remarks proposed by Graeme Kirkpatrick in *Aesthetic Theory and the Video Game* (2011). Kirkpatrick identifies the relationship between the player's body, particularly the hands, and the game controller as the fundamental core for studying video game forms: “It is in the silencing of the controller that we construct the boundary between ordinary experience and the illusion we enter when we relate to screen imagery and other game feedback ‘as if’ they constituted an environment, or immersive world for play” (Kirkpatrick, 2011, p. 105). The author also suggests in a specific passage the existence of a syntax related to the game interface: “Different game programs require us to do different things with the controller, to use its syntactic elements [...] in different ways and a lot of work goes into matching game programs” (Kirkpatrick, 2011, 96). Kirkpatrick also articulates the relationship between on-screen action and action on the controller through an artistic metaphor, that of dance:

> There is a formal continuity between the configuration of digits and the structured, dynamic action sequences in the program and on the screen that is best understood in terms of an idea from effect
shape analysis in dance theory, namely, the *kineme*, which grasps the role of expressive hand positions in relation to other parts of a dance. (Kirkpatrick, 2011, 103)

And yet, Kirkpatrick does not identify a structural link between movements on the interface and virtual movements, nor does he describe their articulation. He speaks of a formal continuity between actions on the interface and software responses, but emphasizes how such continuities are not mimetic, without elaborating further: “Kinemes are rarely mimetic, but are associated with other movements of the dance” (Kirkpatrick, 2011, 103). As far as we are concerned, movement is not a syntax limited to gestures to be made on the controller, but is a unified diagrammatic system that associates movements on the interface with movements in the game world. The player’s movements are intrinsically linked not only to the virtual movements, but also to the digital visualization that acts as its fundamental correlate. Between these two movements, there are no mimetic correlations, it is true, but often, their correlation is linked to *pars pro toto* mechanisms — the movement on the interface is a part that is completed by the virtual movement — or to analogy tensions that rhythmically associate, as we have already observed, punctual and durative actions. In other words, the kinetic syntax is not that of the controller, but it is the syntax that is generated by the articulation of the movements to be made on the controller with respect to the movements that can be realized in a specific video game.

Moreover, the analysis of kinetic diagrams and the interrelation between visual and kinetic syntax allows us to propose a specific answer to the question of the relation between the ontology of games and their hermeneutics. Aarseth and Möring (2020) have rightly noted that it is certainly possible to develop an analytical framework for the study of video games based on the hermeneutic circle. However, this tells us little about the practice of video gaming itself:
[...] the Gadamerian framing of the hermeneutics of artworks as a hermeneutic circle seems to tie in very well with cybernetics. However, conceptualizing computer game-play merely as a feedback loop is not sufficient in order to understand how the hermeneutic to-and-fro works in *the case of specific computer games* [emphasis added]. (Aarseth and Möring, 2020, p. 5)

According to the authors, a general hermeneutics of video games cannot be set up because it would be too general, especially in light of the fact that “There is no universal gameplay that pertains to every kind of game, and there is no hermeneutic method of game analysis as such; only game-specific or genre-specific” (Aarseth and Möring, 2020, p.7). The approaches that are referred to as ludo-hermeneutics actually constitute “the player’s hermeneutics: the process of becoming-a-player. This is also the uncovering of the game’s ontology” (Aarseth and Möring, 2020, p. 6). For this reason, it is first necessary to identify “the game’s ontological nature. This is the first step of ludo-hermeneutics, and the only step that implicitly or explicitly is necessary. The rest depends on game type and player motivation”. (Aarseth and Möring, 2020, p.7).

Our proposals aim to rethink the relation between ontology and the dynamic construction of meaning in video games around the semiotics of practices. Indeed, the need to make movements on the interface that are transformed into movements within virtual worlds describes, at a general level, the way in which the player is installed in videogame practice, the way in which he or she is made to exist within it and discovers, by experiencing it, the ontology of a specific videogame. Rather than simply stopping at the observation that these are interactive experiences, it is possible to immediately analyze the kinetic system that articulates each videogame experience. An *FPS* such as *Call of Duty*, will use a kinetic diagram that associates pressing commands on the interface with offensive and defensive actions such as shooting, reloading the weapon, running, crouching, or throwing grenades. In contrast, in *Super Mario Odyssey*, the diagram will allow jumping, walking, running, throwing Cappy, as well as articulating these movements in accordance with a precise acrobatic combination. In other
words, with kinetic diagrams, it is possible to analyze the articulation of movements that punctuate the semantic system of a video game or genre. These are specific micro-languages that structure the way video games work but without determining the experience. In other words, it seems that this proposal can combine an approach interested in analyzing video games as objects with an approach interested in understanding them as experiences. Kinetic syntax makes it possible to analyze the semantics expressed by a specific video game's system of movements, but it is immediately open to the study of how specific users appropriate this syntax to develop personal styles, purposes, and experiences, particularly through the integration of empirical methods inspired, for example, by anthropology (Servais, 2020).

### 3. *The Legend of Zelda: Breath of the Wild and the clever lies concerning materials*

One case that allows us to articulate the relationships between visual and kinetic syntaxes is *The Legend of Zelda: Breath of the Wild* (Nintendo, 2017). In Nintendo's masterpiece, the two syntaxes are in fact closely interrelated with the visual and interactive rendering of materials on the basis of which the game mechanics are built. We can't go into detail about the great variety of the interactions afforded by this video game, but we can give a timely example showing the articulation of the kinetic system with the elemental qualities of the materials.

One of the buttons on the interface allows one to “slash” using the right or left arm of Link, the playable character. This is precisely one of the movements imposed upon the player by the kinetic system. And yet, the effect resulting from this movement varies depending on which object is equipped and depending on its materiality. If we wield a sword, the pressure applied to the button will allow us to strike sword slashes and damage enemies, cut grass, cut ropes, and so on. However, a whole series of objects can be wielded: if we equip an axe, we can strike any tree, cut it down, and then use it as a bridge to cross streams, or we may continue hitting it with the axe until it turns into a bundle of branches with which we could, for example,
start a fire. If instead of the axe, we wield a palm tree, the blows will not injure the enemies but
will produce an air blast capable of throwing them off to a distance and potentially over a cliff.
Or if we use the palm tree to deliver a slash in direction of the sail of a boat on which we are
placed, the blast of air will inflate the sail and move the boat in the corresponding direction,
and so on. The result is a system of elemental interactions that gives great freedom to the player,
prompting him or her to improvise ingenious “thinking” solutions through actions that
manipulate objects and materials.

These game mechanics thus take advantage of a kinetic system that is organized into at
least two levels: 1. Basic commands concerning the general actions that the player can perform
with as many commands applied to the interface: striking slashes, throwing an object, shooting
arrows, and so on. 2. The variation of these actions based on the particular objects being
equipped, for example a palm tree that can manipulate the wind rather than an axe that can chop
down trees. However, these two levels are also closely related to the visual and auditory
recognition of the materials: we use an axe to chop down a tree because in our lived experience,
we know that a tree can be cut down, that grass can be set on fire, etc. Both kinetic manipulation
and its results functionally merge the visual and practical aspects into an interactive whole.

3.1. *The playful semiosis of digital materiality*

During the Game Developers Conference of 2017, the Nintendo designers explained the
process behind the game's production.<fnref n="6" /> The creative director, Hidemaro Fujibayashi, explained
how the idea of multiplicative gameplay was implemented. To expose the idea to the rest of the
team and for working on the actual development of the three-dimensional version of the world,
the director asked the chief programmer, Takuhiro Dohta, to build a prototype using stylized
graphics that could show the variety of elemental and physical interactions. The solution was

<footnote n="6">⁶ The video of the conference can be viewed at this address: https://www.youtube.com/watch?v=QyMsF31NdNc</footnote>
the following: the designers would take the graphics engine from the first episode of the saga, *The Legend of Zelda*, which was released in 1987, but implement it in a version that would allow for experimentation with materials.

What Nintendo designers and programmers seem to be doing is exploiting what we commonly know about the interaction between substances and elements to build playful mechanisms. The prototype allows us to enter into the process of semiotizing matter and substances and to identify some fundamental operations, which include:

1. A partial selection of objects, materials and elements that are manipulable within the game world: water, fire, wood, metals, rocks, electricity, and magnetism. This selection builds a finite set compared to the infinite complexity of the real world.

2. We also have an iconization or hypoICONIZATION of material qualities and their reactions (Eco, 1999, pp. 337-393). This is not aimed at an effect of extreme realism, but rather at achieving a balance in the figurative density of objects and material interactions in order to construct playful interactions. One trivial example: with a wooden tree in reality, we could do innumerous things, such as building statues, carving our name, building weapons, etc. In *Zelda*, we can burn it, cut it down, make wood from it, climb it, or push it onto a stream of water where it will float.

3. Finally, we have a global stylization of material manipulations. In reality, to get wood from a tree or to burn it, we would need to work hard and long. In *Zelda*, two blows of the axe allow us to cut it down, the canopy of foliage automatically detaches itself. As far as fire is concerned, its propagation is greatly diminished compared to how it behaves in reality. If we burn grass, the flames stop shortly afterwards and do not spread as they would in real life. Overly realistic fire would not be controllable and therefore not fun.

One concept is of particular interest to us: that of “clever lies” with which the programmers themselves describe the semiotic reproduction of materials and interactions. The
word “lies” immediately brings us to the heart of the problem<sup>7</sup>: this system is not meant to build
an effect of realism, nor is it meant to be purely playful. In other words, the clever lies about
the materials and elements build an element-oriented semiotic system: the iconic remodeling
enabled by digitality allows matter to “say” something more specific, to say not with words,
nor only with the visual language, but with transformative actions. The morphology of matter
becomes a playful semiotic function that prompts experimentation, discovery, trial and error,
on the basis of a thin but strongly combinatorial system. This is precisely the process underlying
the existence of the player as such inside the ludic experience, while discovering the ontology
of this specific video game. The elemental interactions included in the final product are
generators of play-like practices, whose balance between openness and closure, between
imagination, interpretation and action, is their experimental *raison d’être*.

Narratology approaches in general are somewhat at a loss in the face of this openness,
because it is not relevant to analyze a semiotic action as a narrative, logical function. The
experimentation with semiotic materials is simply lost. Zelda can be defined as a generator of
virtual narrative programs ready to be actualized and realized: the core meaning of this
experience is exactly the ponderation among these virtual narrative programs, the prefiguration
of possible acts of manipulation. In other words, the elemental system needs a twofold moment
of analysis: one about the openness of the elemental system, which is based on playful actions,
and the second for analyzing the actual narrative programs that can develop it or actualize it<sup>8</sup>.

<footnote n="7">⁷ In one of the most famous and provocative definitions of semiotics, Umberto Eco stated that a semiotic system is anything that can be used to lie (Eco, 1975, p. 17).</footnote>

<footnote n="8">⁸ Our reading of *Zelda* has more than one point in common with the proposals made in Ferri (2015). Ferri identifies video games as virtual interactive matrices that are actualized into practical narratives during each play. However, two elements differentiate our proposal, first, the fact that Ferri describes the general functioning of any video game, situating himself at a very broad level of generality. With respect to this hypothesis, we believe it is necessary to distinguish video games that particularly push for player experimentation. In addition, we believe that</footnote>
In short, this case is a striking example of ergodic literature and processual meaning. In addition, not only does *Zelda*, as does any video game, require nontrivial physical effort to be traversed, but the immanent elemental system, based on the recognizability of materials through their visual syntax, and based on their stylized manipulability through their kinetic syntax, exponentially increases the range of possibilities that can be carried out.

This seems to be a good example of the strategic position assumed by the proposition of the kinetic syntax. It allows video games to be described both as formal objects, identifying the sets of movements set up by each virtual production, and as processual experiences, which can be analyzed with respect to the specific conducts of different players using particular styles and values. In this regard, I would like to mention the paper by Michelle Westerlaken (2017), who made a “vegan run” of Zelda, appropriating the kinetic system in accordance with a distinct ideological project.

## Conclusions

In this paper, we proposed a semiotic theory and methodology for analyzing video games by understanding them as virtual practices.

We exploited the ontological meta-model proposed by Aarseth and Grabarczyk in order to construct a specific theory for video game practices. We proposed to conceive of the latter as image-movements, that is, expressive systems based on two sets of interrelated qualities, and thus on two syntaxes: a visual syntax and a kinetic syntax. The practical core of kinetic syntax concerns the transformation of the player's movements (behavioral aspect) made on the interface (physical interface), into the virtual movements (presentational and mechanical

<footnote>the study of kinetic diagrams allows us to explain how this actualization occurs between virtual matrices and actual practice, that is, through the player's appropriation of the kinetic syntax specific to each video game. It is therefore a kinetic actualization that is articulated through specific diagrams.</footnote>
aspects) based on the operation of the specific software (computational aspect). According to
our hypothesis, visual syntax is not only involved in the presentational layer, but also in the
mechanical, phenomenal, conceptual and social sublayers. Nevertheless, these two syntaxes
should not be separated, because during the meaning-making experience of video games, the
player plays not only through his or her movements, but also through the rhythmic and visual
execution of these movements in real time, thereby generating effects of immersion,
satisfaction, as well as cognitive and perceptual distancing.

Finally, we presented the case of *The Legend of Zelda: Breath of the Wild* for the way it integrates visual syntax and kinetic syntax around a complex physical and chemical engine. Indeed, the multiplicative gameplay that characterizes the Nintendo title is based on both the recognizability of materials through their visual syntax and on their stylized manipulability through their kinetic syntax.

**Bibliography**

Aarseth, E. (1997). *Cybertext: Perspectives on Ergodic Literature*. Johns Hopkins University Press.

Aarseth (2014). *Ontology*. In M. Wolf & B. Perron (Eds.), *The Routledge Companion to Video Game Studies* (pp. 484-492). Routledge.

Aarseth, E., & Calleja, G. (2015, June 22-25). *The Word Game: The ontology of an undefinable object* [Conference Proceedings]. Proceedings of the 10th International Conference on the Foundations of Digital Games (FDG), Pacific Grove, CA, USA.

http://www.fdg2015.org/papers/fdg2015_paper_51.pdf

Aarseth, E., and Grabarczyk, P. (2018, July 25-28). *An Ontological Meta-Model for Game Research* [Conference Proceedings]. Proceedings of the Digital Games Research Association
Conference (DiGRA), Turin, Italy. http://www.digra.org/wp-content/uploads/digital-library/DIGRA_2018_paper_247_rev.pdf

Aarseth, E., & Möring, S. (2020, September 15 - 18). *The game itself? Towards a Hermeneutics of Computer Games* [Conference Proceedings]. Proceedings of the 15th International Conference on the Foundations of Digital Games (FDG), Bugibba, Malta. http://fdg2020.org/schedule.php

Aroni, G. (2022). *The Semiotics of Architecture in Video Games*. Bloomsbury.

Calleja, G. (2011). *In-Game. From Immersion to Incorporation*. MIT Press.

Compagno, D. (2013). *Dezmond. Una lettura di Assassin's Creed 2*. Unicopli.

D'Armenio, E. (2014). *Mondi paralleli. Ripensare l'interattività nei videogiochi*. Unicopli.

Dondero, M.G. (2020). *The Language of Images. The Forms and the Forces*. Springer.

Eco, U. (1975). *Trattato di semiotica generale*. Bompiani.

Eco, U. (1979). *Lector in fabula*. Bompiani.

Eco, U. (1999). *Kant and the Platypus: Essays on Language and Cognition*. Translated by Alastair McEwen. Harcourt.

Eskelinen, M. (2001). *The Gaming Situation. Game Studies. The International Journal of Computer Game Research* 1 (1). http://www gamestudies org/0101/eskelinen/.

Ferri, G. (2015). Narrative structures in IDN Authoring and Analysis. In H. Koenitz, G. Ferri, M. Haahr, D. Sezen & T.İ. Sezen (Eds.), *Interactive Digital Narrative. History, Theory and Practice* (pp. 77-90). Routledge.

Ferri, G. (2015). Narrative structures in IDN Authoring and Analysis. In H. Koenitz, G. Ferri, M. Haahr, D. Sezen & T.İ. Sezen (Eds.), *Interactive Digital Narrative. History, Theory and Practice* (pp. 77-90). Routledge.

Floch, J.M. (1990). *Sémiotique, marketing et communication. Sous les signes, les stratégies*. PUF.
Fontanille, J. (1989). *Les espaces subjectifs: Introduction à la Sémiotique de l'observateur.*
Hachette.

Fontanille, J. (2008). *Pratiques sémiotiques*. PUF.

Giuliana, G. T., (2018, July 25-28). "Quilting the meaning: gameplay as catalyst of signification and why to co-op in game studies" [Conference Proceedings]. Proceedings of the Digital Games Research Association Conference (DiGRA), Turin, Italy. www.digra.org/wp-content/uploads/digital-library/DIGRA_2018_paper_147.pdf

Grodal, T. (2000). Video Games and the Pleasure of Control. In D. Zillmann & P. Vorderer, (Eds.), *Media entertainment: The psychology of its appeal* (pp 197-213). Lawrence Erlbaum Associates.

Hjelmslev, L. (1961). *Prolegomena to a Theory of Language*. University of Wisconsin Press.

Kirkpatrick, G. (2011). *Aesthetic theory and the video game*. Manchester University Press.

Maietti, M. (2004). *Semiotica dei videogiochi*. Unicopli.

Meneghelli, A. (2007). *Dentro lo schermo. Immersione e interattività nei god games*. Unicopli.

Meneghelli, A. (2011). *Il risveglio dei sensi. Verso un’esperienza di gioco corporeo*. Unicopli.

Peirce, C. S. (1931-1935). *Collected Papers of Charles Sanders Peirce 1-6*. Harvard University Press.

Pezzini, I. & Cervelli, P. (Eds.) (2006). *Scene del consumo: Dallo shopping al museo*. Meltemi.

Post, J. (2009). Bridging the Narratology-Ludology Divide. *E|C Serie Speciale*”, Year III, no. 5, pp. 31-36, http://www.ec-aiss.it/monografici/5_computergames/3_post.pdf

Pozzato, M.P. (2012). *Foto di matrimonio e altri saggi*. Bompiani.

Servais, O. (2020). *Dans la peau des gamers. Anthropologie d’une guilde de World of Warcraft*, Karthala.

Thibault, M. (2020). *Ludosemiotica. Il gioco tra segni, testi, pratiche e discorsi*. Aracne.
Westerlaken, M. (2017). *Self-Fashioning in Action: Zelda’s Breath of the Wild* *Vegan Run*.

*Game Philosophy Network*. p. 1-14

https://gamephilosophy2017.files.wordpress.com/2017/11/westerlaken_pocg17.pdf