# A Cross-Temporal Robotic Dance Performance: Dancing with a Humanoid Robot and Artificial Life

Huiting Hong, Isadora Teles de Castro E Costa, Arnaud Tanguy, Abderrahmane Kheddar, Chu-Yin Chen

<figure><img src="image_1.png" /></figure>

## To cite this version:

Huiting Hong, Isadora Teles de Castro E Costa, Arnaud Tanguy, Abderrahmane Kheddar, Chu-Yin Chen. A Cross-Temporal Robotic Dance Performance: Dancing with a Humanoid Robot and Artificial Life. ISEA 2024 - 29th International Symposium on Electronic Art, Jun 2024, Brisbane, Australia. hal-04755684

HAL Id: hal-04755684

https://hal.science/hal-04755684v1

Submitted on 30 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Cross-Temporal Robotic Dance Performance: Dancing with a Humanoid Robot and Artificial Life

Hui-Ting Hong<fnref n="1" />,<fnref n="2" />, Isadora Teles de Castro e Costa<fnref n="1" />, Arnaud Tanguy<fnref n="2" />,

Abderrahmane Kheddar <fnref n="3" />,<fnref n="2" /> and Chu-Yin Chen <fnref n="1" />

## Abstract

We present a case study exploring the integration of artificial, autonomous, and interactive artifacts into the realm of choreography. Specifically, we assessed our experimentation on the HRP-4 humanoid robot and an evolving virtual ecosystem, utilized as characters and dynamic scenography in a dance performance. Our research sheds light on the mechanisms enabling dynamic interplay between human beings and artificial entities, with a specific emphasis on the significance of cross-temporal dialogues. Additionally, we demonstrate the potential of autonomous interactive systems in fostering improvisational co-creation in the process of stage development.

## Keywords

Human-Machine Co-Creation, Robotic Dance, Autonomy, Artificial Life, Interactive Art, Performance, Transdisciplinary Art.

## Introduction

Inspired by the context of the coronavirus pandemic and the subsequent implications of social isolation, we create this performance that traverses the temporal boundaries between the present (2023) and an envisioned future (2035). Our narrative delves into themes of existential solitude in the context of looming external organic entity threats, such as novel viruses, juxtaposed against the solace sought in the presence of artificial entities. The premise of our script is conceived against the backdrop of an imagined future where technological advancements have made it possible for individuals to access high-tech tools, such as humanoid social robots, with the same ease as acquiring a telephone today. Through an exploration of temporality, our research inquires into the process of improvised co-creation via live interactions with autonomous systems. How do these temporal conversations contribute to the dynamic interplay between the dancer and the humanoid robot and a virtual ecosystem? This dialogue extends beyond disciplinary confines, resonating with the notion like First Nations Australians once described "*before then, then, now and the future then existing in the constant presence of place*" and we imagine that this place creates a space that timely connects the thinking, the process and inspiration all in the same place: the co-creative live performance.

Over the past decades, robotics performances have garnered significant scholarly attention, showcasing a wide

spectrum ranging from rudimentary mechanical prototypes to intricately coded systems. While small-sized humanoid robots [16], and quadrupeds [5] are commonly observed in various artistic performances [27, 21, 29], the artistic presence with 34 degrees of freedom (DoF) humanoid robots like HRP-4 remains relatively rare, often confined to industrial robotic applications. For example, dynamic walking control, e.g., [13], stabilization, e.g., [6, 23], and synchronized motion, e.g., [12], yet the implementation of such insights in artistic endeavors has remained largely unexplored. Moreover, the quest for autonomous systems in robotics, especially in humanoid robots, occupies a central position, representing the primary objective of creating entities capable of self-directed creative expression. This pursuit, while promising, is entangled with the intricacies of artificial intelligence, machine learning, and the interface between human operators and machine functions. While instances of autonomy in human-robot interaction have been conducted in prior works [4, 24, 25, 3, 17], its integration into stage settings poses unique challenges due to the tension between the desirable reproducibility and unpredictability of live performance. One of the main challenges also arises from the inherently improvised nature of real-time interactions with highly autonomous artifacts. Navigating the interplay between scientific research precision and creative spontaneity, our research creation introduced a novel Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF), which enables the HRP-4 humanoid robot to dynamically respond to the dancer's movements in a staging context through a gesture responsive framework. By live-showcasing autonomous interactions between the dancer and the humanoid robot HRP-4 on the stage, this research presents a tailored control system designed for artistic performances and sheds light on the dynamics underlying the persistent quest for autonomous robotic live performances.

On the other hand, the concept of autonomy in digital arts, particularly within the realm of artificial life, also holds considerable significance. It pertains to the internal state space of computational artifacts, enabling them to manifest their internal conditions and exhibit self-directed behavior. Artists often draw inspiration from natural processes to shift from the creative process to autonomous, self-generating processes, where artworks or scenarios are created by self-sustaining mechanisms. This autonomy extends to the capacity for self-

<footnote n="1">¹AIAC, INREV, University of Paris 8 Vincennes - Saint-Denis, France</footnote>

<footnote n="2">²CNRS-University of Montpellier LIRMM, UMR5506, France</footnote>

<footnote n="3">³CNRS-AIST Joint Robotics Laboratory (JRL), IRL3218, Japan</footnote>

<footnote>w102060018w@gmail.com</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: The complete cross-temporal actions conducted in this research creation, which demonstrates a co-creation dialogue between a HRP-4 humanoid robot, a dancer, and an artificial life system, fostering an evolving interdisciplinary dialogue at the nexus of art and engineering.</figcaption></figure>

organization and interaction within various environments, cultivating diverse dialogues and adaptive behaviors. In this research we created an Evolving Virtual Ecosystem (E.V.E.) building upon our last year's experimentation [11] specifically adapted for this performance. E.V.E. is a virtual system that creates different behaviors and patterns through an agent-based model<fnref n="1" /> planning methodology. It manifests emergent and surprising behaviors and patterns by using algorithms inspired by complex biological systems, being part of the field of Artificial Life.

In this research, we initiate a cross-temporal dialogue between the humanoid robot, a dancer, and virtual ecosystem, focusing on the nature of the HRP-4's artificial body and the transformative capacity of organic entities embedded within artificial life, aiming to create the improvised interaction of movements and possible dancing gestures through the autonomous system.

## Cross-Temporal Actions and Research Statement

The development of two autonomous entities: Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF) and Evolving Virtual Ecosystem (E.V.E.), despite their fundamental differences, collaboratively promotes a dialogue on the stage that transcends temporal dimensions, emphasizing the potentially harmonious fusion of autonomous entities and human beings. The cross-temporal actions manifest in various dimensions:

* *From the present to the future:* the performance script weaves a fictional narrative that orchestrates a temporal voyage within the auditorium, transporting the audience from 2023 to the future 2035. At the same time, its narrative is entwined with the portrayal of the Evolving Virtual Ecosystem (E.V.E.), which serves as a dynamic canvas

to speculate on the challenges and responses to potential biological threats in our imagined future.

* *From the past to the present:* the movements of the robot, partially generated from the recorded movements of the dancer in previous instances by motion capture technology, which will be detailed in the Methodology section. This process engenders a dynamic interplay between the robot's artificial body, the dancer's archived movements, and the dancer's current gestures.

* *In the present moment:* the Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF) enables the robot to autonomously respond to the dancer's gestures in real-time, fostering a dynamic human-machine interaction that occurs in the present moment on the stage.

The complete cross-temporal actions performed in this research are shown in Figure 1 and to further elucidate the conducted performance within this practice-based research creation, the following performance script of our fictional narrative is provided.

> "In the unfolding narrative, the protagonist assumes the role of an isolated inhabitant confined within the confines of her apartment, grappling with a cacophony of emotions ranging from fear, hate, and joy to bouts of inexplicable ecstasy and despondency, triggered by the looming threat of external organic entities. Amidst her emotional turmoil, a temporal solace emerges in the form of the humanoid robot, mirroring her physical dimensions and serving as a source of companionship. A delicate interplay unfolds as she shares intimate moments with the robot, teaching it different movements and engaging in heartfelt conversations during the prolonged period of social seclusion. The interaction, at times, reveals the robot's mechanical and detached responses, while occasionally exuding an autonomous warmth, punctuated by unexpected surprises. Despite the initial challenges in communication, the robot gradually comprehends her intentions, drawing from its memory bank intertwined with her

<footnote n="1">¹Complexity Explorer - Introduction to Agent-based Modeling - Summer 2022, www.complexityexplorer.org</footnote>
own experiences. This convergence of memories forms a
poignant link, fostering a semblance of comfort, and catalyz-
ing the evolution of her vulnerabilities in this temporal voy-
age. Moreover, the lurking presence of the external organic
entities evolves through time, and the fear and joy turn to trig-
ger her to construct a new relationship with the robot also
with artificial life."

This fictional narrative illuminates our research exploration of temporality and the co-creation process in the staging context of interactions. Our research questions are i) What are the central mechanisms that underpin the interplay between the human performer and artificial entities within these temporal conversations? and ii) How do autonomous interactive systems establish a dynamic bridge for facilitating improvised co-creation between the humanoid robot, the dancer, and the virtual ecosystems?

In the following sections, we first go through the techni-
cal details of how to control the HRP-4 humanoid robot with
different controllers to accomplish various robot movements
in our robotic performance. Secondly, we introduce the Ges-
ture Dancing Humanoid Robot Interactive Framework (GD-
HRIF) we created which enables the HRP-4 humanoid robot
to autonomously respond to the dancer's gestures and facil-
itates human-robot interaction in the context of artistic per-
formance without the necessity of any wearable devices. At
the end of the section, we demonstrate how we created the
Evolving Virtual Ecosystem (E.V.E.).

Methodology

Robot Control System

The control of the HRP-4 humanoid robot is using the task-
space *mc RTC* framework<fnref n="2" />, an open-source interface de-
signed for the simulation and control of robotic systems.
Within this interface, we implemented several specialized
robot controllers using C++ [14], enabling the execution of
humanoid robot movements during our performance. Within
this interface, each controller is derived from the same gen-
eral class, thereby enabling the customization of functionality
based on specific requirements. In the context of this per-
formance, we integrated three distinct types of controllers,
each contributing to the orchestration of the humanoid robot's
movements during the performance:

• posture tasks controller (Section A)

• stabilization controller (Section B)

* motion capture retargeting controller: motion synchrony between human and robot (Section C)

**A. Posture tasks controller** The posture tasks controller is instrumental in organizing the precise configurations of the HRP-4 humanoid robot's postures, which entails precise control over its eight body segments encompassing 30 adjustable parameters, shown in Table 1. Leveraging the flexibility of this controller, we conducted comprehensive simulations in mujoco<fnref n="3" /> to visualize and choreograph the postures required for our performance. To streamline the creative process and

<table><caption>Table 1: HRP-4's adjustable segments in the posture tasks controller, where Y indicates Yaw, the rotation around the vertical axis; R indicates Roll, the rotation around the longitudinal axis; P indicates Pitch, the rotation around the transverse axis.</caption>
   <thead>
    <tr>
     <td>
      Body Segments
     </td>
     <td>
      Axes of Rotation
     </td>
     <td>
      Left (L)/Right (R)
     </td>
    </tr>
   </thead>
   <tbody>
    <tr>
     <td>
      Hip
     </td>
     <td>
      Y, R and P
     </td>
     <td>
      L and R
     </td>
    </tr>
    <tr>
     <td>
      Knee
     </td>
     <td>
      P
     </td>
     <td>
      L and R
     </td>
    </tr>
    <tr>
     <td>
      Ankle
     </td>
     <td>
      R and P
     </td>
     <td>
      L and R
     </td>
    </tr>
    <tr>
     <td>
      Chest
     </td>
     <td>
      P and Y
     </td>
     <td>
      -
     </td>
    </tr>
    <tr>
     <td>
      Neck
     </td>
     <td>
      P and Y
     </td>
     <td>
      -
     </td>
    </tr>
    <tr>
     <td>
      Shoulder
     </td>
     <td>
      Y, R and P
     </td>
     <td>
      L and R
     </td>
    </tr>
    <tr>
     <td>
      Elbow
     </td>
     <td>
      P
     </td>
     <td>
      L and R
     </td>
    </tr>
    <tr>
     <td>
      Wrist
     </td>
     <td>
      Y, R and P
     </td>
     <td>
      L and R
     </td>
    </tr>
   </tbody>
  </table>

<figure><img src="image_4.png" /><figcaption>Figure 2: Stabilization controller in walking along the pre-defined paths on the stage.</figcaption></figure>

ensure the synchronization of aesthetic choreography, we in-
tegrated a motion capture retargeting controller, enabling the
direct creation of robot posture sequences derived from the
dancer's movements recorded via a wearable device. We dis-
cuss it later in Section C.

**B. Stabilization controller** Originating from the ground-
breaking work of [6], this stabilization controller introduced
a transformative breakthrough to dynamic stair climbing and
walking, specifically designed for a manufacturing use-case
demonstrator. This methodology delved into a whole-body
admittance controller where both end-effector and Center of
Mass (CoM) strategies are applied simultaneously, effectively
bolstering the robot's stability and overall performance dur-
ing the climbing task. Open-sourced<fnref n="4" /> under the *mc RTC* in-
terface, we found a unique application of this innovative con-
troller within our artistic performance context, tailored to the

<footnote n="2">²https://github.com/jrl-umi3218/mc\_rtc</footnote>

<footnote n="3">³github.com/rohanpsingh/mc\_mujoco.</footnote>

<footnote n="4">⁴https://github.com/stephane-caron/lipm\_walking\_controller/.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3: The remapping results between the dancer's movements and the HRP-4 humanoid robot's movements.</figcaption></figure>

anticipation of multiple future steps for the robot, shown in Figure 2. This adaptation facilitated the seamless navigation of the robot along the predefined paths on the stage, ensuring continuous equilibrium and mitigating any potential operational instability.

**C. Motion capture retargeting controller: motion synchronization between human and robot** As elucidated in Section A, the inefficient process of manually configuring each movement using the posture tasks controller needs a solution to enhance the creation process of the robot's movements. The complexities of translating abstract artistic concepts into tangible robotic movements posed considerable challenges, since the execution of parameter modifications to achieve the envisioned robotic postures is a time-consuming task. Furthermore, realizing complex movement sequences, such as waist twisting, high-reaching leg kicks, or plank-like postures, remains a significant challenge for the HRP-4 given its primary design for industrial applications rather than artistic performances. Consequently, the better way to design HRP-4's movements is standing in 'the robot' place to create its movement sequence. The introduction of a motion synchronization method emerged as an efficient solution, leveraging low-lag human tracking technology<fnref n="5" /> to create the translation of dancer movements into corresponding robotic movements. The data is transmitted from tracking sensors to the robot control interface and retargeted onto the robot body. This synchronization process is executed through two steps:

1. Live-mode recording and data remapping: we employed Xsens MVN [20] motion capture system to record the dancer's movements. The integration of 17 Inertial Measurement Units (IMUs) [1] within the system enabled the accurate estimation of human pose by capturing essential sensor data such as position, velocity, and acceleration. To achieve this remapping, first involved string matching between the names of the motion capture sensors and the corresponding body parts of the HRP-4 humanoid robot. Furthermore, inherent differences in body dimensions between the dancer and the robot introduced certain offsets that required additional transformation procedures. Leveraging the quaternion data collected from the sensors, we

applied transformations, using quaternion data, on the extracted position data to correct the offset before assigning it to the corresponding body part of the HRP-4 humanoid robot. The transformation offset of position data is carried out by the SpaceVecAlg<fnref n="6" /> library.

2. Offline mode replay and movement filtering: post-execution of the live-mode recording and data remapping, the logged data within `mc_rtc` underwent a thorough analysis during the offline mode replay. This process involved re-reading the logged data to reconstruct the sequence of movements already constructed. Such an approach enabled the systematic selection of feasible HRP-4's movements on the stage. This offline phase provided crucial insights into the quality and feasibility of the recorded dancer movements, prompting the dancer to iteratively modify and adapt her gestures to ensure a harmonious synergy between her body language and the capabilities of the humanoid.

These two steps enable precise remapping of human movements by the HRP-4's body and ensure synchronicity between the dancer's movements and the robot's movements, the remapping results are demonstrated in Figure 3. The integration of motion capture technology in performance art is not uncommon; however, orchestrating a seamless synchronization between a high DoF humanoid robot and the dancer's movements represents a significant solution in translating abstract artistic concepts into tangible robotic movements within the creation process of the robotic performance. This endeavor marks the first instance, to the best of our knowledge, where HRP-4 performs on stage using movements recorded by the dancer in the past time, thereby fostering a dynamic cross-temporal dialogue between the robot's artificial body, the dancer's archived movements, and the dancer's current gestures.

Besides the utilization of distinct controllers to create movements of the robot in this performance, in order to investigate the question proposed in the first place: how do autonomous interactive systems establish a dynamic bridge for facilitating improvised co-creation between the humanoid robot, the dancer, and the virtual ecosystems? We further developed a Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF) integrated into the `mc_rtc` interface

<footnote n="5">⁵www.xsens.com/motion-capture.</footnote>

<footnote n="6">⁶github.com/jrl-umi3218/SpaceVecAlg.</footnote>
that enables the HRP-4 humanoid robot to dynamically respond to the dancer's gestures during the performance.

## Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF)

Expanding on the notion of a cross-temporal dialogue between the robot's movements and the previously recorded dancer's movements, our exploration delves deeper into the real-time interaction dynamics. Traditionally, human-robot interactions have heavily relied on the integration of wearable devices [7, 9]. However, the incorporation of such devices during a live performance might decrease the audience's watching experience and to circumvent this challenge, we created, specifically for this performance, a non-wearable approach for the gesture-based interactive system, named Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF). It's an autonomous interaction system that enables the robot to respond to the dancer's gestures in real-time, without requiring the use of any wearable equipment. That allows the dancer to freely express her movements during the performance, while the robot reacts to her gestures based on its own decision-making process. The holistic architecture of the system is illustrated in Figure 4, showcasing the integration of the communication channels and the components, which include 3D human pose estimation, gesture classification, stabilizer, and a gesture responsive framework. The system architecture is the combination of two components: i) a visual recognition system, implemented in Python, and ii) a specialized robot controller, implemented in C++. Leveraging the ROS [18] framework, we integrate ROS as a communicator between the two. Where a ROS publisher inside the visual component, transmits classified data while integrating a ROS subscriber inside the specialized robot controller components, receiving and processing the transmitted data.

We implemented our visual recognition system in conjunction with MediaPipe Pose Landmarker library [15], to deliver a deep learning solution of human pose estimation through the camera installed on HRP-4. Subsequently, this human pose estimation data is subjected to a comprehensive nine-class gesture classification system that operates on a distance-based multilayer perceptron (MLP) models. The nine-class classification is carried out and output integer from 0 to 8, shown in Figure 5(a), which serves as input for the subsequent stages of the specialized robot controller's gesture responsive process.

The specialized robot controller, on the other hand, integrated a standing stabilization controller, mentioned in the robot control system Section B, operates in tandem with the ROS subscriber. In the meanwhile, an autonomous gesture responsive controller operates as in the concurrency structure with stabilization. This dual-state configuration enables the robot controller component to execute the responsive gestures, based on the gesture classification result of the dancer, in real-time, exhibiting minimal latency while preserving the robot's standing equilibrium when interacting with the dancer. Despite an average latency of 0.4 seconds between the visual components and the specialized robot controller components, the imperceptibility of delay during performances can be attributed to the distractions introduced by

<figure><img src="image_6.png" /><figcaption>Figure 4: The complete architecture of our proposed Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF).</figcaption></figure>

other media elements such as lighting, music, and scenography.

The pivotal role of the autonomous gesture responsive framework lies in furnishing the robot with its own decision-making capabilities, thereby enabling the performer to engage with the robot through spontaneous and intuitive body gestures. Anchored in the principle of intuitive mimicry, the system integrates a set of nine pre-designed robot movements established by using the posture task controller mentioned in the robot control system Section A, shown in Figure 5(b). Each pre-designed robot movement is then dynamically paired with classified dancer's gesture using a customized policy function, in considering the time sequence information. Consequently, eliminating the need for any wearable devices or external equipment, the system classifies the dancer's gestures into 9 distinct categories. The robot then responds accordingly, based on the decision formulated by the autonomous gesture responsive framework.

Observations during the system's deployment have highlighted instances where the robot mimics the performer's actions. Intriguingly, the system has also yielded instances where the robot responds with opposing or distinctly different movements, imparting an element of unpredictability and artistic intrigue. Beyond its application in robotic performances, this proposed Gesture Dancing Humanoid Robot Interactive Framework (GD-HRIF) serves as a testimony to the heightened engagement potential with the audience, fostering interactive experiences with the humanoid robot in an installation setting. By showcasing the integration of GD-HRIF with HRP-4 humanoid robot in the context of a robotic dance performance, we present a potential form of participa-
<figure><img src="image_7.png" /><figcaption>Figure 5: (a) The results of our gesture classification system, using the camera installed on HRP-4. Take the results of classes 2 and 3 as examples. (b) Nine pre-designed robot movements are established using the posture tasks controller.</figcaption></figure>

tory interaction in live performance creation. The demonstra-
tion integrating GD-HRIF during our performance is shown
in Figure 6(c).

Evolving Virtual Ecosystem (E.V.E.)

In our fictional narrative, besides the exploration of tempo-
rality and the co-creation process through live interactions
with GD-HRIF, we also developed a virtual autonomous sys-
tem called E.V.E. (evolving virtual ecosystem). This virtual
ecosystem plays an important role in our investigation of how
artificial autonomy could transform interactions and the cre-
ative process in the staging context. We introduce a virtual
system designed to generate emergent patterns and behav-
iors, consequently provoking surprising emotions. The sys-
tem simulates threats from organic entities, such as swarm
behavior, suggesting virus and cellular aspects within our per-
formance setting.

The design of E.V.E. employs a multi-agent model and ge-
netic algorithms, enhancing our prior experiments in [11].
This upgraded version showcases distinct visual differences
among the three species. It incorporates a primordial par-
ticle simulation, as in [22], enriching pattern diversity and
system behavior. We’ve implemented predator-prey dynam-
ics using a version of evolutionary steering behavior [19, 26]
and a self-reproduction mechanism to overcome mating chal-
lenges caused by spatial perception limits, ensuring continued
growth. Additionally, we improved the texture buffer reso-
lutions to support more agents and refined the agents’ DNA
linkage to behavior weights, enhancing agent heterogeneity.
E.V.E is the acronym conceptualized for this purpose, rep-
resenting the culmination of our efforts to simulate intricate
self-organizing systems. This simulation system utilizes the
compute shader feature in Touchdesigner, enabling parallel
computation on the graphics card. Agent-related information
is stored in texture buffers, allowing rapid parallel computa-
tions without heavy reliance on the computer’s memory and
CPU. Furthermore, to interact with this evolving system, we
developed a specialized user interface to offer additional pa-
rameters for balancing control and autonomy. This interface
aided the artistic team in aligning the system’s development
with our aesthetic vision, fostering poetic and dramaturgical

coherence.

Experiments and Discussions

In the realm of our practice-based research creation, we pro-
duced a 10-minute performance at the Art Center of Enghien-
les-Bains’s auditorium during the 23/24 season launch event.
The experimentation process starts with the preparation and
execution of the performance and is followed by post-
performance analysis.

The preparation and execution phase involved the construc-
tion of a fictional narrative, designs of the robot's move-
ments, choreography of the dancer, and the refinement of
the E.V.E. and GD-HRIF. This phase also included the in-
tegration and adjustment of essential robot controllers, light-
ing, scenography, performer interactions, virtual artificial life
imagery, and meticulously rehearsing until the final live per-
formance. Post-performance analysis, on the other hand, in-
volved a comprehensive review of recorded video materials
for an objective evaluation, the interviews with the dancer
which provided firsthand insights into the observed interac-
tive and co-creative dynamics. Lastly, the gathering of audi-
ence feedback offered insights into the distinctive nature of
our research creation, guiding the trajectory of future explo-
rations.

Experimental Challenges

Unlike the adaptable and observable setting in the simulation
environment, the live performance on the stage encountered
several critical challenges concerning the transition from sim-
ulation to live execution:

* Fragility concerns with the robot, including instances of unstable mechanical motor function leading to occasional falls during walking stabilization rehearsals. Moreover, the extended power runtime required for rehearsals, typically lasting at least 40 minutes (10 minutes for core performance plus at least a total of 30 minutes for calibration and corresponding adaptation before and after), has posed unprecedented challenges, as sustained motor activity is not customary in standard scientific robotic testing scenarios.

* The dynamic and unpredictable nature of interactions be-
  between the performer, robot, and artificial life demanded
  continual on-stage adaptations, a contrast to the controlled
  and predictable environment of the laboratory setting.

Dancer on Stage

The dancer’s artistic approach follows an all-encompassing
principle that represents the natural order of the universe
and emphasizes the importance of living in harmony with
it, her dance practices are the cultivation of inner peace that
originated from the sage Laozi [2]. Following the previous
year’s experiments detailed in [11], the dancer integrates the
Tanztheater approach [10] this year into her bodily expres-
sions in this performance, communicating emotions and dra-
matic elements through interconnected movements and ges-
tures, woven into the narrative of the performance. Her dance
unfolds across four distinct stages, each contributing to the
narrative of her interaction with the evolving artificial enti-
ties.
<figure><img src="image_8.png" /><figcaption>Figure 6: Four distinct stages contributed to the narrative of the cross-temporal interaction with the two evolving artificial entities: HRP-4 humanoid robot and artificial life.</figcaption></figure>

Initially, her movements convey a palpable sense of fear and vulnerability, embodying the internal turmoil spurred by the external threat of the organic entities created by E.V.E. This portrayal is characterized by tense, contracted gestures and a visibly curled physical demeanor, effectively evoking a sense of imminent danger and emotional unrest, shown in Figure 6(a). The subsequent encounter with the HRP-4 humanoid robot elicits surprise and tentative curiosity, reflected in her movements and gestures transitioning from small and curled to more expansive. As she familiarizes herself with the robot, her attempts at interaction and communication with the robot, her gestures soften, revealing a poignant contrast between the delicacy of human expression and the mechanical precision of the robot, shown in Figure 6(b). Transitioning to the stage of the integration of the GD-HRIF system introduces a dynamic interplay between the dancer and the autonomous capabilities of the robot. She experiences a fluid exchange of energy, occasionally guided by the robot's pre-designed responses and, at times, by unexpected autonomous reactions. This stage amplifies the sense of improvisation and creative freedom for the dancer, she dances and celebrates with the robot, shown in Figure 6(c). The concluding stage highlights the convergence of shared memories between the dancer, the robot, and the virtual ecosystem. Symbolizing a collective moment of emotional catharsis, the dancer's movements convey a revitalized determination and emotional resurgence, where the dancer stretches out her hand again to try to restart the conversation with the robot, as shown in Figure 6(d).

Despite the predefined performance narration, the dancer often improvised to explore unexpected interactions with the artificial entities. A gradual realization emerged that the key to the interaction between the dancer, the humanoid robot, and the virtual ecosystem lay in the ongoing pursuit of a shared rhythm and continual adjustments of expressions, roles, and responses. The mutual adaptation and real-time co-creation remained essential during this pursuit and the interplay between the human and artificial entities relies on the ability to improvise and co-create in real-time.

## Audience Feedback

Following the performance, the feedback from the audience offered external perspective into our research creation. Audience members described a profound sense of shared silence that transcended the boundaries of the auditorium. They also noted a synchronous connection with the rhythmic essence of the performance, even describing a shared synchronization of breath. This shared experience highlighted the specific *enaction* moment transformed from the cross-temporal dialogue between the human, robot, and artificial life. As discussed in [28] and [8], being in(*en*) *action* is a looping redefinition of what our body fully lives and experiences with the environment. In this context, our perception is not static neither passive but actively shaped by our ongoing actions, perception thus being guided by action. As the audience watched the performance, characterized by a pursuit of shared rhythm through mutual adaptation and improvised co-creation, the cross-temporal conversations transform the environment of the auditorium and create a new shared, close to freezing moment, *enaction* time. It is a temporally extended moment where boundaries between the self and the environment blur, and audience become immersed in a shared, co-created experience. It suggests that the performance constituted more than a mere observational event. Instead, it served as a catalyst for a transformative experience, including the rhythmic flow, spatial dynamics, interactive components, and the dancer's gestures. This phenomenon enriched the temporal conversations and dynamic interplay between human and artificial entities, breathing life into the improvised co-creation and the performance narrative on stage.

## Conclusion

This practice-based research creation marks the first instance of a cross-temporal dialogue between the HRP-4 humanoid
robot, virtual artificial life, and the dancer. This co-creation
process combines expertise from the domains of robotics,
real-time computer graphics, dance, and scenography, foster-
ing an evolving interdisciplinary discourse at the nexus of art,
science, and technology. We demonstrate the successful inte-
gration of real-time autonomous interactions between human
and artificial entities under a staging creative process context
in a dance performance. We also pointed out that the key to
the improvised co-creation lay in the behavioral co-evolution
and ongoing pursuit of a shared rhythm. Also, the transfor-
mation of the cross-temporal dialogue in this performance
created a new *enaction* moment with the audience and this
phenomenon opens new possibilities and future discussions
in the robotic dance performance creation.

Acknowledgements

This work benefited from the support of EUR ArTeC and
from the assistance of the French State generated by the Na-
tional Research Agency under the future investment program
bearing the reference ANR-17-EURE-0008.

References

[1] Ahmad, N.; Ghazilla, R. A. R.; Khairi, N. M., and Kasi, V. 2013. Reviews on various inertial measurement unit (imu) sensor applications. *International Journal of Signal Processing Systems* 1(2):256–262.

[2] Ames, R., and Hall, D. 2003. *Dao de jing: A philosophical translation*. Ballantine Books.

[3] Aucouturier, J.-J.; Ikeuchi, K.; Hirukawa, H.; Nakaoka, S.; Shiratori, T.; Kudoh, S.; Kanehiro, F.; Ogata, T.; Kozima, H.; Okuno, H. G.; et al. 2008. Cheek to chip: Dancing robots and ai's future. *IEEE Intelligent Systems* 23(2):74-84.

[4] Beer, J. M.; Fisk, A. D.; and Rogers, W. A. 2014. Toward a framework for levels of robot autonomy in human-robot interaction. *Journal of human-robot interaction* 3(2):74.

[5] Biswal, P., and Mohanty, P. K. 2021. Development of quadruped walking robots: A review. *Ain Shams Engineering Journal* 12(2):2017-2031.

[6] Caron, S.; Kheddar, A.; and Tempier, O. 2019. Stair climbing stabilization of the hrp-4 humanoid robot using whole-body admittance control. In *2019 International conference on robotics and automation (ICRA)*, 277-283. IEEE.

[7] Chen, F.; Lv, H.; Pang, Z.; Zhang, J.; Hou, Y.; Gu, Y.; Yang, H.; and Yang, G. 2018. Wristcam: A wearable sensor for hand trajectory gesture recognition and intelligent human-robot interaction. *IEEE Sensors Journal* 19(19):8441-8451.

[8] Chen, C.-Y. 2017. Un monde pleinement énacté. *Action, énaction: l'émergence de l'œuvre d'art* 53-76.

[9] Cifuentes, C. A.; Frizera, A.; Carelli, R.; and Bastos, T. 2014. Human-robot interaction based on wearable imu sensor and laser range finder. *Robotics and Autonomous Systems* 62(10):1425-1439.

[10] Climenhaga, R. 2012. *The Pina Bausch Sourcebook: The Making of Tanztheater*. Routledge.

[11] Costa, I. T. d. C. E.; Hong, H.-T.; and Chen, C.-Y. 2023. A performance co-created with an autonomous virtual system: A symbiotic approach. In *International Symposium on Electronic Arts* [ISEA2023].

[12] Dallard, A.; Benallegue, M.; Kanehiro, F.; and Kheddar, A. 2023a. Synchronized human-humanoid motion imitation. *IEEE Robotics and Automation Letters*.

[13] Dallard, A.; Benallegue, M.; Scianca, N.; Kanehiro, F.; and Kheddar, A. 2023b. Robust bipedal walking with closed-loop mpc: Adios stabilizers.

[14] Josuttis, N. M. 2012. The c++ standard library: a tuto-
rial and reference.

[15] Lugaresi, C.; Tang, J.; Nash, H.; McClanahan, C.; Uboweja, E.; Hays, M.; Zhang, F.; Chang, C.-L.; Yong, M. G.; Lee, J.; et al. 2019. Mediapipe: A framework for building perception pipelines. *arXiv preprint arXiv:1906.08172*.

[16] Pandey, A. K., and Gelin, R. 2018. A mass-produced sociable humanoid robot: Pepper: The first machine of its kind. *IEEE Robotics & Automation Magazine* 25(3):40-48.

[17] Peng, H.; Zhou, C.; Hu, H.; Chao, F.; and Li, J. 2015. Robotic dance in social robotics—a taxonomy. *IEEE Transactions on Human-Machine Systems* 45(3):281-293.

[18] Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.; Foote, T.; Leibs, J.; Wheeler, R.; Ng, A. Y.; et al. 2009. Ros: an open-source robot operating system. In *ICRA workshop on open source software*, volume 3, 5. Kobe, Japan.

[19] Reynolds, C. W. Steering behaviors for autonomous characters. In *proceedings of Game Developers Conference*, 763-782. Miller Freeman Game Group, San Francisco, California.

[20] Roetenberg, D.; Luinge, H.; Slycke, P.; et al. 2009. Xsens mvn: Full 6dof human motion tracking using miniature inertial sensors. *Xsens Motion Technologies BV, Tech. Rep* 1:1-7.

[21] Ros, R.; Baroni, I.; and Demiris, Y. 2014. Adaptive human-robot interaction in sensorimotor task instruction: From human to robot dance tutors. *Robotics and Autonomous Systems* 62(6):707-720.

[22] Schmickl, T.; Stefanec, M.; and Crailsheim, K. How a life-like system emerges from a simplistic particle motion law. 6(1):37969. Number: 1 Publisher: Nature Publishing Group.

[23] Scianca, N.; De Simone, D.; Lanari, L.; and Oriolo, G. 2020. Mpc for humanoid gait generation: Stability and feasibility. *IEEE Transactions on Robotics* 36(4):1171-1188.

[24] Selvaggio, M.; Cognetti, M.; Nikolaidis, S.; Ivaldi, S.; and Siciliano, B. 2021. Autonomy in physical human-robot interaction: A brief survey. *IEEE Robotics and Automation Letters* 6(4):7989-7996.
[25] Stubbs, K.; Hinds, P. J.; and Wettergreen, D. 2007. Autonomy and common ground in human-robot interaction: A field study. *IEEE Intelligent Systems* 22(2):42–50.

[26] The Coding Train, and Shiffman, D. Coding challenge #69: Evolutionary steering behaviors - part 1.

[27] Thörn, O.; Knudsen, P.; and Saffiotti, A. 2020. Human-robot artistic co-creation: a study in improvised robot dance. In *2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)*, 845–850. IEEE.

[28] Varela, F. J., and Regnot, F. 1996. Quel savoir pour l'éthique?: action, sagesse et cognition. (*No Title*).

[29] Xia, G.; Tay, J.; Dannenberg, R.; and Veloso, M. 2012. Autonomous robot dancing driven by beats and emotions of music. In *Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1*, 205–212.

## Authors Biographies

**Hui-Ting Hong** is a Taiwanese artist based in Paris. She obtained her bachelor's degree in EECS and her master's degree in Electrical Engineering, both from National Tsing Hua University. Currently, Hui-Ting is a Ph.D. candidate at Paris 8 University in the INREV research team, focusing on the interaction and transformation between body, space and mind. She has participated in the *Créons au Musée* project and performed virtually at the Grand Palais and the *Co-évolution, Co-création et improvisation Homme-Machine (CECCI-H2M)* research project under the framework of EUR ArTec and performed at the season opening of Centre des Arts Enghien-les-Bains; she also has cooperated projects and solo projects that were presented at the Huashan Umay Theater, the Taipei Fringe Festival, the Hsinchu 241 Art Space and the Arts Center Tsing Hua. Her aim is to explore the improvisation and co-creation process with autonomous system by developing both dancing gestures and Artificial Intelligence integrated tools. Her artistic research practice focuses on the perception-action of body movements, discussing in particular the shared rhythm happened in human-machine interaction.

**Isadora Teles** is a Brazilian artist based in Paris. She is currently a Ph.D. candidate at the INREV research team, within the laboratory Art des Images et Art Contemporain at the University of Paris 8. Her artistic practice is in the fields of digital generative art, artistic modeling of self-organizing systems, and the design of interactive interfaces for live performances. She participated in practice-based research projects presented at the Grand Palais (Projet “Créons au Musée”), the Bains Numériques festival (“La Fabrique Numerique”), and the Ars Electronica festival (with the Interface Cultures Master students, exhibition “Made in Linz”). She also took part in the design of interactive installations and scenographies for events at the Gaîté Lyrique (“ClekclekBoom label anniversary”), the Nuit Blanche in Paris (with Romain Cieu tat, at La Rotonde), the K-Live Festival, and the Peacock Society festival (both with Superbien studio). As part of her practice-based research, she is currently experimenting with

the design of sensitive, actuating, and connected electronic costumes as well as the computer development of interactive evolving visuals for improvisational theatrical scenography. Other than digital arts, Isadora also experiments with video, 2D traditional animation, illustration, electronics, and, most recently, web data visualization.

**Arnaud Tanguy** has studied computer engineering at the Polytech’Nice-Sophia engineering school, specializing in computer vision. During his studies, he spent an ERASMUS year at Trinity College Dublin, where he specialized in technologies linked to video game development (physical simulation, graphics rendering), followed by a research internship at Technische Universität München, where he worked on a neural network artificial intelligence system for location recognition in a 3D map (e.g. where’s the robot?). He then went on to write a thesis combining real-time mapping and localization (SLAM) technologies with the control of humanoid robots, in order to give them greater autonomy in their interaction with their environment. He then pursued his career as a research engineer in the Joint Robotics Laboratory at the Advanced Institute of Science and Technology (AIST, Tsukuba, Japan) and in the Interactive Digital Human (IDH) team at the Laboratoire d’Informatique et de Microélectronique de l’Université de Montpellier (LIRMM), where he currently works.

**Abderrahmane Kheddar** received the BS in Computer Science degree from the Institut National d’Informatique (ESI), Algiers, the MSc and Ph.D. degree in robotics, both from Pierre et Marie Curie University, Sorbonne University, Paris. He is presently First Class Directeur de Recherche at CNRS. His research interests include haptics, humanoids and recently Bionics. He is a founding member of the IEEE/RAS chapter on haptics, the co-chair and founding member of the IEEE/RAS Technical committee on model-based optimization, he is a member of the steering committee of the IEEE Brain Initiative, Editor of the IEEE Robotics and Automation Letters, Founding member and Deputy Editor-in-Chief of Cyborg and Bionics System. He was Editor of the IEEE Transactions on Robotics (2013-2018) and within the editorial board of other robotics journals; he is a founding member of the IEEE Transactions on Haptics and served in its editorial board during three years (2007-2010). He is an IEEE fellow, AAIA fellow, titular full member of the National Academy of Technology of France and knight of the national order of merits of France

**Chu-Yin Chen** is an Artist and Professor in Digital Art. INREV research team at Paris 8 University. Her creations, based on Artificial Life and complex systems, develop interaction modes between audience and virtual creatures showing autonomous and evolving behaviors. Her digital artworks have been shown in numerous international exhibitions. Her research articulates two overlapping areas: 1) Digital Creation using algorithms of complexity and emergence, and 2) Metacognition and Elicitation of the processes of creation, enaction and aesthetic reception, via psychophenomenology and mindfulness.