# Agency in Co-Creativity: Towards a Structured Analysis of a Concept

Janin Koch, Prashanth Thattai Ravikumar, Filipe Calegario

► To cite this version:

Janin Koch, Prashanth Thattai Ravikumar, Filipe Calegario. Agency in Co-Creativity: Towards a Structured Analysis of a Concept. ICCC 2021 - 12th International Conference on Computational Creativity, Sep 2021, Mexico City (online), Mexico. pp.449 - 452. hal-03533245

HAL Id: hal-03533245

https://inria.hal.science/hal-03533245

Submitted on 18 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Agency in Co-Creativity: Towards a Structured Analysis of a Concept

**Janin Koch**

ExSitu Lab

Université Paris-Saclay, CNRS, Inria, LISN

Paris, France

Janin.Koch@inria.fr

**Prashanth Thattai Ravikumar**

Communications and New Media

National University of Singapore

Singapore

prashanth.thattai@gmail.com

**Filipe Calegario**

Mistic, Centro de Informática

Universidade Federal de Pernambuco

Brazil

fcac@cin.ufpe.br

## Abstract

Computational co-creativity is an emerging area of research in co-creativity, that connects to user-centered research in design, human-computer interaction and artificial intelligence. A common vocabulary among researchers is critical for advancing research in such interdisciplinary work, but it is often difficult to achieve. The concept of agency is one of the topics of interest for the development of computational co-creative systems. While agency can be broadly defined as a sense of control over the creative outcomes of a co-creative process, there is no shared understanding of how to design systems for and with agency. In this paper, we reflect on the works we have developed over the last five years as a starting point for discussing the definition of agency. Further research will allow us to investigate how agency manifests itself in human-computer creative collaboration, also known as co-creativity. We believe that having a common ground allows for a more efficient process of analyzing and designing human-computer collaborative systems.

## Introduction

Interdisciplinary domains, such as computational creativity (CC), frequently present difficulties for researchers in finding common ground between observed phenomena and formalizing concepts in a shared manner. Co-creative research faces an even greater challenge with computational co-creativity, as it has a greater degree of overlap with domains of user-centered research in design and HCI.

One of the challenges in this emerging area of co-creativity research is developing a common vocabulary that will allow researchers to better communicate their accomplishments in various domains of co-creativity. To that end, we propose developing a common vocabulary centered on the concept of agency in order to analyze and synthesize various strands of co-creativity research.

Our research group's necessity led to a better definition of the concept of agency. The group comprises of three researchers from three continents (South America, Europe, and Asia), each with our own set of experiences and research interests. Our concepts, practices, and processes can diverge significantly when working in multidisciplinary environments. However, to collaborate successfully in such an environment, a shared vocabulary is required to find common ground. To overcome such differences, we saw the

need to take a step back and better understand the concepts before applying them to creating a collaborative body of knowledge in human-computer co-creativity.

We further began a search for uses of agency in ICCC proceedings in the literature. Based on this work-in-progress analysis, we identify a few factors in current definitions of agency that we believe are influential to the agency of co-creative systems and would like to discuss with workshop participants. Identifying and designing systems with these factors in mind will assist researchers in developing co-creative applications and communicating their work to others.

## Developing Co-creative Systems with Agency

Our individual research interests range from human-to-human interaction to human-machine collaboration in a variety of domains. While not exhaustive, this review does provide insights into the role agency may play in the design, evaluation, and reflection of co-creative systems.

## Co-creative systems in human-human co-creative process

The third author discusses the human-human co-creation process of prototyping digital musical instruments (Calegario 2019). Musicians and designers used a modular prototyping toolkit that encapsulates technical details and accelerates experimentation.

The Probatio<fnref n="1" /> system comprises bases and blocks, which aims to bridge the gap between abstract concepts and working prototypes by making it simple to experiment with poses and gestures that generate sound in real-time.

Besides the individual interaction with the system, designers and musicians can cooperate with Probatio, physically communicating intentions, presenting and experimenting with blocks that can be used as sensors for particular intentions and circumstances. The artifact becomes a mediator between two human creative agents, making it straightforward for the group to impact each individual in the final creative result. In that sense, the concept of agency relates to feeling in control of the instrument's design choices and the means to produce music.

<footnote n="1">¹https://probat.io</footnote>
The agency concept in Probatio relates to the term *material agency* discussed by Brown (Brown 2016), who building upon the work of Malafouris, combines the human intention with the material affordance of the artifact (Malafouris 2008).

In addition to the perceptible material possibilities, the responsive nature of the system allows for the agency in the creation process of a digital musical instrument. The system's encapsulation of technical details opens possibilities for people with little technical experience to see themselves in constructing digital artifacts, clearly perceiving the effect of their actions in creating digital musical instruments.

## Computational co-creative systems for creative collaboration

Through experiments with a system developed for real-time rhythmic duets, the second author highlights the effect of agency on user experiences with a music co-creation system.

The MASSE system was developed for real-time rhythmic duets and employs a combination of high-level constraints and stochastic rhythm generation to select its rhythmic behaviors (Ravikumar and Wyse 2019). MASSE was evaluated by creating different configurations of the system that played duets with the musicians. The different configurations produced musical responses with different degrees of complementary in its rhythmic support (Ravikumar and Patel 2020). Musicians reported a satisfactory sense of co-creative engagement different from controlling a tool when the system produced suggestive rhythms and played rhythms that opposed their musical ideas. In configurations that the system behaved predictably and mirrored their contributions, musicians felt that the system was a good follower. Musicians perceived the system as least co-creative when it generated rhythms that were not correlated with their contributions.

The development and experimentation of the MASSE system augments the current work on agency in music co-creation systems. In prior work, humans have perceived agency in music co-creation systems that exhibit autonomy (Bown 2018), musical personality (Albert 2013), initiative through unpredictable behaviors (Lewis 2000), and supportive yet interactive behaviors (Brown, Gifford, and Voltz 2016). Experiments with the MASSE system suggest that complementary behaviors can have predictable effects of user experiences of co-creativity (Ravikumar and Patel 2020). This is, however, a preliminary result and needs to be examined in greater detail with subsequent experiments.

## Human-centered approaches on human-AI collaboration

Designing collaborative systems with a human-centric perspective requires developers taking into consideration the impact of systems on both the creative process and the practitioners themselves. Author one investigated how different design aspects of collaborative systems result in diverse perceptions of agency within a creative process through various studies.

The *MayAI?* system was designed to create a one-to-one situation between a designer and a system in order to find

and develop ideas visually in the form of an image collage (Koch et al. 2019). The system was able to explore and exploit visual ideas by suggesting images that were either similar to or diverged from the current collage in order to inspire new ideas or even provoke the designer. The system then inquired about the designer's reasoning for selecting specific images that differed from the current collage. As a result, designers attributed agency to the system in the form of authorship ("I feel like I don't work alone, I feel like there is another person" (P15)) or having its own agenda ("I think it was a 'she,' and she maybe heard me but she had her own opinions as well" (P14)). Such remarks were primarily attributed to the system's autonomy to follow and explore specific ideas. This is consistent with Bown's understanding that autonomy encourages users to perceive agency (Bown 2018). Interestingly, the system's ability to evaluate and request reasoning from participants was viewed as beneficial for reflection on current practice, but also sometimes as a criticism of their choices. This demonstrates the potential of interpretive agency in creative practice, as proposed by Fischer et al. In (Fisher and Shin 2019), they argue that in order to fully understand and model creative agency, we have to extend our view from generative agency, the act of creating objects, to computationally-modelled interpretive agency, the evaluation of the creative value. This would enable systems to contribute narrative critique, criticism, and commentary, in order to have more agency about their creations and contributions. However, our results also highlight the risk computational interpretation can pose to the creator's self-perception, potentially leading to negative experiences or rejection of collaboration.

Furthermore, the *ImageSense* system was created to investigate the role and perception of intelligent systems in the creative process (Koch et al. 2020). It incorporates five different intelligent tools with varying degrees of agency into a visual ideation system in which multiple designers can create and discuss ideas using a visual collage. In this context, agency can be defined in terms of two dimensions: pro-activity and adaptability. The former refers to how much it interacts with and makes its own suggestions, similar to (Lewis 2000; Brown, Gifford, and Voltz 2016), and the latter to how much the system's contributions adapt to the current creative process. In studies with eight design teams, the perceived level of agency increased with both dimensions. The role of the included tools in the process was defined by this perception. Systems that primarily adapted to the designer's actions were perceived as a design function, whereas systems that made their own suggestions were recognized as assisting the creative process. When both dimensions were present, the system was perceived as a potential partner who could take the lead if the designer became stuck on an idea. This preliminary finding will need to be investigated further in future studies that include more and different dimensions.

## Discussion

A common thread for furthering agency research in co-creative systems emerged from the interactions between the authors. The observations from co-creative systems for music and creative ideation suggest that pro-activity and adapt-
ability are desirable characteristics for an interactive system to be perceived as a co-creator. Furthermore, observations of human-human co-creation with the Probatio system point in the direction of future research on agency in co-creative systems via the design of co-creative spaces.

## Agency between pro-activity and adaptability

While some of the approaches described above differ from one another and are adapted to the type and domain of application, we can see recurring themes that influence system design and agency perception. All of our work tries to investigate the user's perception of agency, how it emerges, and to begin to understand how certain behavioral characteristics influence it. One topic that emerged in our discussions was the trade-off we see between pro-activity/autonomy and adaptability. According to the results of our individual experiments, both factors contribute to a user's perception of agency. However, the individual dimensions do not appear to be sufficient to elicit a strong sense of agency.

In *ImageSense*, systems that actively suggested images that were not directly related to the ideation process were frequently regarded as beneficial to the creative process, but did not convey a sense of agency that made them appear co-creative despite their active contributions. Similar observations were reported by musicians who played with the configuration of *MASSE* that produced uncorrelated rhythms. Multiple musicians acknowledged the system's ability to generate independent rhythms. However, they did not feel as co-creative as compared to the conditions in which the system generated related material.

On the other hand, adaptability, or the ability to adjust to new conditions, enabled the system in staying relevant to the current task. In *ImageSense*, however, systems that adapted closely to the designer's actions were perceived as design tools that helped to reflect and clarify current activities rather than active co-creators. Reports from *MASSE* were also consistent with this observation. In configurations that system behaved predictably and mirrored the musician's contribution, musicians felt that they were more in control of the performance and considered the system as a good follower. However, they did report that their interaction with the system was one of performing with a co-creator.

The above mentioned observations lend strength to the idea that humans may perceive interactive tools as co-creators when they exhibit a balance between pro-activity and adaptability. Such systems exhibit an ability to generate relevant contributions that lie on a range between exploiting (following) the current task closely and exploring (diverging) from it to some extent. Among our different works, the complementary behaviors developed in the *MASSE* system is one such example. While complementary rhythms oppose the musician, they also maintains relatedness to the inputs. Preliminary experiments with the system suggest that complementary responses have a positive on co-creative experiences. The relation between complementary behavior and agency is a connection that needs to be examined in greater detail with subsequent experiments.

## Agency and design space

Co-creative systems, such as the aforementioned, are frequently designed with a predefined design and decision space in which the system can navigate and make autonomous choices.

The Probatio system, on the other hand, explores a scenario in which one or more of the creative agents co-creates the design space itself. The concept of agency in this context refers to a sense of control over design choices (e.g., the instrument's design choices). However, extending and modifying the design space generally requires a more holistic understanding of the task and context, which is a challenge for current systems. We anticipate that computational co-creative systems that co-create the design space will become a viable approach to increase agency in the future.

## Beyond Individual Experiences with Agency

Our work aims to build a shared vocabulary for long-term design of the agency of co-creative systems. This paper focuses on a small brick that provides some hands-on examples of approaches to develop systems with agency. We believe that exercises like this can help us get closer to a more precise definition of the term agency and the various concepts that surround it. We have already begun to broaden our investigation in this context, and we believe that this collaborative research on agency will benefit the larger community in two ways: a better understanding of the impact of agency in practice, and a more general model of agency for developing more advanced co-creative systems.

## Understand agency in practice

The first goal of this work is to better understand from researchers what agency means in their work and how they use it in system development. Going forward, this entails generating discussions and developing questions that allow authors to articulate their systems from the standpoint of agency. We would like to pose the following questions to the researcher community in order for them to critically think about their system design and practice from the standpoint of agency:

* What do we mean by agency in computational creativity?

* What are the dimensions of agency?

* What are alternative words/concepts that people use in relation to agency?

* What effect does agency have on human-computer creative collaboration, i.e. decision making or leadership?

* How does agency manifest itself in human-computer creative collaboration, for example, in terms of control, steerability, or authorship?

* How does the definition of agency support us in developing co-creative projects?

* How could you convey agency in the system, for example, through goals or autonomy?

Based on our findings, there seems to be a disconnect between theoretical work on agency and practical implementation of agency in systems, which commonly refer to other
existing concepts. Collecting more examples of co-creative
systems designed with agency would not only broaden our
understanding of agency, but would also contribute to clos-
ing the gap, resulting in a more holistic definition of agency.

Develop a framework of agency

The second goal of this work is to create a framework for
using agency in system design and analysis.

To that end, we have begun to examine the definitions of agency in eleven editions of the proceedings of the International Conference on Computational Creativity (2010 to 2020). There are 513 papers in total, including full articles, short articles, demo papers, and position papers. First, we used Atlas.TI software to search the ICCC corpus for the string agency. Then we eliminated the papers in which the word differed from our researched theme, such as agency as institution or organization. Then, based on word frequency, we chose papers that contained the phrase agency at least five times. Finally, we came up with a shortlist of twelve papers that we divided among the research team for further examination.

Throughout these papers, one common observation was
that agency was either attributed to the system itself, refer-
ring to specific functionalities, or to the perceived agency of
the system by the users. Examples of the former include sys-
tem intentionality (Guckelsberger, Salge, and Colton 2017)
and decision making process (Gemeinboeck and Saunders
2013). The latter frequently referred to aspects such as per-
ceived agency in movements (Saunders and Gemeinboeck
2018) or the system's ability to provide alternatives that the
user can evaluate (Mateas, Mawhorter, and Wardrip-Fruin
2015). We intend to further investigate this divide in our
future work. As the number of papers that directly refer to
agency is rather limited we will also extend our scope to pa-
pers describing agency using related wordings and concepts.

Conclusion

Developing a common vocabulary around the notion of
agency for designing co-creative systems is a crucial matter.
It will benefit the collaboration of inter-disciplinary research
teams to advance co-creative systems. In order to achieve
this we need to explore how agency is understood in prac-
tice by the community as well as in theory. Finally, more
research is needed to identify the relevant dimensions con-
cepts that will enable a clearer definition of agency – how to
define it, measure it, and communicate it to others.

References

Albert, J. 2013. Interactive musical partner: A demonstra-
tion of musical personality settings for influencing the be-
havior of an interactive musical generation system. In *In
Ninth Artificial Intelligence and Interactive Digital Enter-
tainment Conference.*

Bown, O. 2018. Performer interaction and expectation with
live algorithms: Experiences with zamyatin. *Digital Cre-
ativity* 29(1):37–50.

Brown, A. R.; Gifford, T.; and Voltz, B. 2016. Stimulat-
ing creative partnerships in human-agent musical interac-
tion. *Computers in Entertainment (CIE)* 14(2).

Brown, A. R. 2016. Understanding musical practices as agency networks. In *ICCC*.

Calegario, F. 2019. *Designing Digital Musical Instruments Using Probatio: A Physical Prototyping Toolkit*. Computational Synthesis and Creative Systems. Cham: Springer International Publishing.

Fisher, D. H., and Shin, H. 2019. Critique as creativity: Towards developing computational commentators on creative works. In *ICCC*, 172–179.

Gemeinboeck, P., and Saunders, R. 2013. Creative machine performance: Computational creativity and robotic art. In *ICCC*, 215–219. Citeseer.

Guckelsberger, C.; Salge, C.; and Colton, S. 2017. Addressing the” why?” in computational creativity: A non-anthropocentric, minimal model of intentional creative agency.

Koch, J.; Lucero, A.; Hegemann, L.; and Oulasvirta, A. 2019. May ai? design ideation with cooperative contextual bandits. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems*, 1–12.

Koch, J.; Taffin, N.; Beaudouin-Lafon, M.; Laine, M.; Lucero, A.; and Mackay, W. E. 2020. Imagesense: An intelligent collaborative ideation tool to support diverse human-computer partnerships. *Proceedings of the ACM on Human-Computer Interaction* 4(CSCW1).

Lewis, G. E. 2000. Too many notes: Computers, complexity and culture in voyager. *Leonardo Music Journal* 10(nil):33–39.

Malafouris, L. 2008. At the potter's wheel: An argument for material agency. In *Material agency*. Springer. 19–36.

Mateas, M.; Mawhorter, P. A.; and Wardrip-Fruin, N. 2015. Intentionally generating choices in interactive narratives. In *ICCC*, 292–299. Citeseer.

Ravikumar, P. T., and Patel, D. 2020. Engendering co-
creative experiences through agent parametric control. In
ICCC, 49–56.

Ravikumar, P. T., and Wyse, L. 2019. Masse: A system for music action selection through state evaluation. In *in Proceedings of the Tenth International Conference on Computational Creativity, Charlotte, North Carolina, June 17-21, 2019*, 132–139.

Saunders, R., and Gemeinboeck, P. 2018. Performative body mapping for designing expressive robots. In *ICCC*, 280–287.