<figure><img src="image_1.png" /></figure>

HAL
open science

# Supporting the Experience of Stakeholders of Multimedia Art – Towards an Ontology

Danzhu Li, Gerrit van Der Veer

► To cite this version:

Danzhu Li, Gerrit van Der Veer. Supporting the Experience of Stakeholders of Multimedia Art – Towards an Ontology. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.3-15, 10.1007/978-3-030-46540-7_1 . hal-03188821

HAL Id: hal-03188821

https://inria.hal.science/hal-03188821

Submitted on 2 Apr 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Supporting the Experience of Stakeholders of Multimedia Art – Towards an Ontology

Danzhu Li<fnref n="1" />⁻<fnref n="2" /> and Gerrit C. van der Veer<fnref n="2" />⁻<fnref n="3" />

**Abstract.** We introduce the rapid change of the visual art ecosystem, triggered by current science and technology development. ICT enables new multimedia based an interactive art forms, with an increasing variety of stakeholders. We provide examples of audience involvement, of immersion, and of brain-computer interaction as a new paradigm for participation. We point to the use of new material dimensions, as well as to expanding shared creation and cognition. We also point to opportunities to apply this development to accommodate special needs. In order to support the dissemination of these possibilities, we advocate the development of a task-modeling based ontology to describe, analyse, and support the evolving art ecosystem.

**Keywords:** Contemporary Visual Art Ecosystem, Stakeholders, Ontology.

## 1 Visual Art on the Move

Our profession includes to support modern art, i.e., to teach, and to publish, relevant knowledge and skills to participate in the current and future art ecosystem. For the various stakeholders, the relevant view will be different, but stakeholders need to collaborate and communicate so a common language is needed that supports an intuitive cultural base. We are aiming on that and we intend to discuss this with the workshop / panel.

From ancient times, innovation of tools and techniques did push art to a new level. But today, with ICT, dramatic changes occur everywhere, including in the world of visual arts. Rich media, as well as information visualization, became a common way of modern visual communication, and these interact with each other.

We have been exploring how contemporary artists are modifying the relationship between human beings and new realities. This will lead to an expansion of the multidimensional concept of experience, including the notion of values of visual art for its different stakeholders.

This paper analyses the current situation and developing trends of contemporary visual art from the perspective of enriching all stakeholders’ experience, which suggests that contemporary visual artists should be aware. Through observation, practice, interviews and other research methods, we analyze and study the development and changes

<footnote n="1">¹ Human Media Interaction, University Twente, Enschede, the Netherlands</footnote>

<footnote n="2">² Multimedia and Animation, Luxun Academy of Fine Arts, Liaoning, China</footnote>

<footnote n="3">³ Multimedia and Culture, Computer Science, Vrije Universiteit, Amsterdam, the Netherlands
lidanzhu@me.com, gerrit@acm.org</footnote>
of the work and survival of stakeholders in contemporary visual (mainly interactive) art. Contemporary visual arts move to a cross disciplinary or interdisciplinary context, related to the development of science and technology and the change of human aesthetic ability. A new type of cross-border artists is coming out, and the evolution of society will make the space of art broader, evolving to a new visual art ecology.

## 2 Different Experiences Brought by Science and Technology

Under the influence of science and technology, new forms of art have emerged. These new terms may be controversial, but their names do reflect a future trend of contemporary art: digital art; interactive art; technique art; generative art; bio art; and singularity art [1].

We observed that visual art in due time approaches motion art. In relation to this, the operations of the artists and performers changed, including a gradual transfer from mainly perceptual motor skill activities to cognitive activities, gradually including application of ICT. It urges us to predict the short-term future of art, based on our collection and collation of long-term historic technology-related information.

Contemporary dynamic visual art is oriented towards integrated media, the involvement of science and technology, and interactive, cross-border, multi-disciplinary cooperation. We predict that in the short term, art will go through a period where the practical value is greater than the aesthetic value. We analyze the impact and reflect on the changes and demands of experience and on the different levels of values from a cognitive perspective.

## 3 Interactive Media are Multi-dimensional

Modern visual art has developed to the interactive stage, resembling the development of HCI (human-computer interaction) technology. Artists explore the possibilities of interaction between human beings and machines, as well as between machines. Due to the rapid development of mobiles and wearable devices, interactive art features in a multi-dimensional context, serving a multi-sensory experience.

The change of experience elicits artists and audiences to co-create. The continuous improvement of technology has a direct relationship with the impact of artistic creation. Consequently, artists need to understand and learn to use new technologies.

Interactive media art based on virtual reality and augmented reality is becoming to a major form of contemporary visual art innovation. From the commercialization of tools, to the development of models, to the sharing of various open-source information platforms, artists are becoming more familiar with this performance form and are as devoted as their audience. Mixed Reality technology enables participants (artists and spectators) to extend their experience in the dimension of time and space, breaking the old way of appreciation. We will provide some examples:
## 3.1 Chinese Artists Play with their Audiences

XuanPin, “The Field”, is a comprehensive media art work. The work is intended to celebrate the tenth anniversary of the birth of a game DNF, created by LAFA [2] teachers and students, the Tencent Company, and Chinese folk artists.

Based on the ancient shadow play, this work combines shadow play with animation and laser printing, MR immersion interaction, and other comprehensive media. It triggers people to think about tradition and modern art and technology.

<figure><img src="image_3.png" /><figcaption>Fig. 1 a. Character for the game DNF (2018). Picture by the authors, Dalian, China.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>**Fig. 2 b.** The game DNF created with shadow play techniques (2018). Picture by the authors, Dalian, China.</figcaption></figure>

In order to let young people know about traditional culture, the team applied the latest Halo lens hybrid reality technology to interpret the scenes and images in DNF games and show them in the form of shadow play. In this arena, audiences can watch the performances of non-hereditary artists, and experience the performances made with new materials by players of cos-play roles, and, in addition, take HoloLens glasses to watch and try the performance of the shadows in virtual scenes.

This multi-dimensional interaction is an innovation and exploration of traditional visual art. Figure 1 shows the character designed in the game DNF, created with shadow play techniques. Figure 2 shows Cosplay actor performance in the game by using shadow play, where the audience manipulates the shadow puppet in the virtual scene through HoloLens. One of the authors participated in the creation.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Cosplay actors’ performance of the DNF game by using shadow play and the audience manipulating the shadow puppet in the virtual scene through HoloLens. Picture provided by Media and Animation college, LAFA.</figcaption></figure>
## 3.2 Lie down and enjoy arts - Immersion experience

Both authors participated in the Art exhibition of SIGCHI2018, lying down on the floor to experience the art and enjoying it.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Art exhibition, SIGCHI 2018, Montreal. Picture by the authors.</figcaption></figure>
Figure 3 shows one of the works. In the exhibition hall, artists built a dome theatre with dozens of cushions on the floor. Under the half dome, audiences had to lie down to see the dynamic visual art projected on the inside of the dome. In figure 3 we can see a Korean artist playing with instruments and electric fans moving under the dome. At the same time, they were "projected" into the dome.

This combination became a rich comprehensive performance. Obviously, artists have broken the traditional form of experience, like under the domes of ancient churches or palaces in Europe. The artists challenge the audience's experience to complete their co-creation and to become part of this dynamic artistic performance.

## 3.3 Life Information Visualization - Brain-Computer Interface Art

The history of BCI as an artistic means is still short. Current BCI for artistic creation is mainly the application of non-invasive systems (EEG) to ensure a safe and noninvasive experience for artists and viewers. The brain signals picked up by electrodes are sent to the computer, which uses sophisticated software to translate them into computer commands.

Portability and relatively low prices make the technology easy to promote. In this way, the audience can participate in the dynamics of the art without physical actions on the piece of art, and can even co-create, either individually or (when appropriately designed by the artist) as a group of spectators. This is the embodiment and the charm of BCI in an aesthetic application, which changes the form of traditional art and of the appreciation (no longer restricted to an objective perspective).

<figure><img src="image_7.png" /><figcaption>Fig. 4. Exquisite Corpse – Visual Arts, http://uhbmi.ee.uh.edu/portfolio/ec-m-2/</figcaption></figure>
Obviously, contemporary artists need to understand the changing perspective on the role of the audience, as well as the technical aspects of designing the interaction, to apply this in their creation. Fig.4 shows scientists using Mobile Brain-body Imaging (MoBI) technology to study the human improvisational creative process in the spirit of the “Exquisite Corpse” (an improvisational creative game created by surrealists in the 1920s, where three artists create a three-part art piece). The performance-study seeks to uncover clues to what happens in the brain as people create and contemplate art [3].

## 4 Opportunities and Challenges Faced by Artists in their Exploration of Materials

The development of art is accompanied by the use of different materials and tools. In modern society, with highly developed science and Internet, artists' exploration of new materials closely follows the pace of science and technology, though limited by the development of materials and technology. In this process, we can see that the symbiotic fields of art and technology affect each other, which catalyzes the innovation of art ecology.

According to the authors’ long-term observation, a new group is gradually growing in the contemporary visual art ecology. They have subverted the public's traditional understanding of art. They generally have multiple identities or skills. Their backgrounds are computer scientists, material scientists, engineers and even biologists. We have noticed this in our early research on the art stakeholders, and we have continuously tracked the development and change of this group. We have analyzed that the development of modern education supports contemporary artists to have rich knowledge, combined with a scientific development concept and methodology.

Because of the growing number of artists with comprehensive qualities, the art market is increasingly dynamic, making the exhibition more diversified, and the audience's experience more and more multidimensional. With the improvement of the public's aesthetic level, this promotes the artists' thinking and change. Furthermore, as the interaction between the audience and art becomes more and more active, it promotes the development of cross-border artists.

We understand that they (artists and art works) blur the boundary between art and technology. But there is a real controversy. We predict that, in the near future, this group will continue to expand which will inevitably affect the development of the next generation of artists, and will update human cognition of art. We must consider both sides of the development of things.

### 4.1 A Plea for the Role of Material Science Development

An increasing number of artists focus on interaction mechanisms with wearable and implantable devices as well as integrating Internet-of-Things technology with new interactive art paradigms. In fact, both artists and scientists are aiming at a substance between visible and invisible. With the development of material science, new artistic
forms such as nano-art, bio-art and integrated material art have brought challenges to those avant-garde artists in exploring future art. Though many mainstream artists and stakeholders turn a blind eye, or lack foresight, to the rapid development of contemporary science and technology, some artists are exploring the humanization of technology. New materials like nanomaterials are expected to be widely used in future artistic creation. Today we may already witness contemporary artistic practices in this direction. For example, nano-printing art, nano-sculpture, and nano-animation:

Jonty Hurwitz's work "The FRAGILE GIANT" (Fig 5) on animal protection is the smallest nano-sculpture in history. In this microcosmic world, the artist explores the relationship between man and nature. This elephant sculpture is just over on tenth of a millimeter high. It is walking along the stark and perilous landscape of a human fingerprint. It can be destroyed by a human breath. According to Hurwitz the sculptures are so tiny that they are invisible to the human eye, and able to be placed on the forehead of an ant. Details of the works are at 300-nanometer scale, similar to the wavelengths of visible light and are therefore nearly impossible, according to the laws of physics, to see in the visible spectrum. The only way to observe these works is through a non-optical method of magnification like a scanning electron microscope[4].

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Jonty Hurwitz : *"THE FRAGILE GIANT"*. The smallest man-made object ever to be filmed. (2015) https://jontyhurstwitz.com/fragile-giant.</figcaption></figure>

IBM Research claimed to make the “World’s Smallest Movie Using Atoms” (Fig.6). IBM took the challenge of moving 5,000 atoms around in order to create a short stop motion video, capturing the images using a scanning tunneling microscope. “A Boy and His Atom” depicts a character named Atom who befriends a single atom and goes on a playful journey that includes dancing, playing catch and bouncing on a trampoline.
Set to a playful musical track, the movie represents a unique way to convey science outside the research community [5].

<figure><img src="image_9.png" /><figcaption>**Fig. 6.** IBM: "*A Boy and His Atom*". The world's smallest movie (2013). https://www-03.ibm.com/press/us/en/photo/40982.wss</figcaption></figure>

Technology turns inspiration and creativity into reality, challenging traditional thinking and bringing about an art revolution, and even triggering to build a new world view. This innovation requires collaboration between many different interdisciplinary experts. The cooperation and co-creation will generate a new cognitive system (a symbiotic relationship between human beings' wearables, and the context). Wearable devices may have powerful effect on our experience of the context, of interactive art, and of life. Smart fabric in wearable devices is a representative case. Sensors are becoming smaller (to nanoscale units), and smart fabric applications become more flexible and comfortable [6]. Artists' exploration of science and art has stimulated people's recognition of the reality of contemporary art.

## 4.2 Appealing to contemporary visual art ecology to keep pace with the times

The workshop theme" Information Retrieved from (large) WHAT Networks" (INTERACT 2019, Cyprus) has inspired us to think about Privacy issued have to be considered (copyright, personal data of audience...), and information security, human rights and ethical disputes. It is essential for building a sound and beautiful future art ecology. This chapter is therefore extended. We are concerned about two aspects:

a. based on the era of the Internet of things, digital art copyright issues and works related to the collection and use of biological information security issues. and

b. with the application of life science and material science to artists' creation,
some controversies have touched on the rights and moral disputes of human beings, animals and other livings, such as "The Vacanti Mouse": Twenty years ago, Harvard surgeons Joseph and his brother Charles Vacanti, along with MIT engineer Bob Langer, experimented with techniques to create human body parts in the lab. They implanted the shape of a human ear in the back of a mouse [7]. Some people regard it as an artistic act, which has aroused controversy.

Again, the contemporary artwork named Sugababe (Exhibited in ZKM in 2014) is a Bio-Sculpture by Diemut Strebe (Fig 7). A living bioengineered replica of Vincent van Gogh's ear grown from tissue engineered cartilage cells procured from a direct male descendant and containing natural genetic information about Vincent as well as genetically engineered components amongst using genome editing CRISPRCas9 technique, and recent bioprinting technology. One can speak to the ear through a microphone system. The input sound is connected to a computer processor, using a software program to generate simulated nerve impulses from the sound signal in real time. This mimics sound recorded from an electrode inserted into the auditory nerve, when firing [8].

<figure><img src="image_10.png" /><figcaption>**Fig. 7.** Diemut Strebe: "Sugababe". the 3D print ear. (2014). http://diemutstrebe.altervista.org</figcaption></figure>

This is considered to be a representative contemporary art exploration of life science. At present, artists who can use advanced science and technology such as life science, gene technology and so on are few and far between. We explore this topic to arouse
stakeholders' thoughts on future ecological development. There is no doubt that science enables artists to enter a new era in the use of materials. We not only look forward to the development of art diversity, but also consider moral constraints and legal protection.

We try and practice scientific methodology and models to build a vision of future art ecology. Cognitive Science (Chapter 5) and task modeling (Chapter 6) are the methods we are studying and practicing. We hope more and more stakeholders will pay attention to this research.

# 5 Exploration of Cognition

For all stakeholders of visual art, the improvement of knowledge is accompanied by the development of technology, by interdisciplinary cooperation, lifelong learning, and the application advanced technology and machine learning to assist artistic creation. Candy and Edmonds [9], [10] mention three categories of activities in the creative person's thinking and working practice were identified: knowledge, visualization, and collaboration. The quality of the type of collaboration can be assessed in terms of its durability and stimulus to creative thinking as well as the outcomes achieved.

From brush to electronic pen; from clay and stone to 3D printing: artists need to master the accessibility of technology; need to learn to use new technology for creation. Tools should meet artists' needs, and be easy to learn and use, as well as be timely updated and upgraded to meet new needs. With the development of personal computing and the coming of the era of Internet of Everything, we foresee that customized tools will serve more and more artists and stakeholders of art.

We call for the exploration and development of cognitive ergonomics in a broader scope and the application of cooperation in the field of art [11]. This includes the special community of artists with special needs. In our practice, one graduate student from the art school (LAFA) in China is a hearing-impaired artist. He is very distressed by his limitation. He worries that if he always makes silent films, he will lose audiences and his works will be excluded from the art market. At present, he can only rely on the production of silent films to solve the problem of production. This case is representative of a considerable group of artists. In combination with our participation in the Artistic BCI workshop at CHI in Montreal in 2018, we analyzed the possibility of future support for hearing-impaired artists to edit music through EEG or other bioinformatics technologies [12]. Similarly, visually impaired artists and artists with language barriers can be assisted in their creation. We currently consider Ear Touch, a one-handed interaction technique that allows visually impaired people to interact with a smartphone using the ear to tap or draw gestures on the touchscreen, facilitating one-handed use as an alternative to headphones and addressing privacy and social concerns [13]. A smart glove can already work out what the wearer is manipulating from its weight and shape [14].
# 6 An Ontology of Modern Visual Art

When applying ICT, artists revise the way they work. We observe that contemporary interactive art is an artistic act co-created by artists and participants. We briefly review its production process: from the manufacturing stage, artists need to cooperate with participants in many disciplines (using brainstorming, sketching, technology and tools, exhibition forms, interactive models, etc.). After completion of the artwork, it is expected to be co-creative with audience, e.g., through recording the behavior of the experiencer, visualizing the emotional information and the interactive behaviors needed. The stronger the participation, the higher the experiential value.

This is precisely the purpose of some artists' 'creation: the value of such works of art. We envision an ontology to analyse, describe, and support the future art ecosystem: with new roles, new objects, and new activities.

Our conceptual framework is based on GTA [15], and we mainly consider to focus on development of the concepts Roles (with mandating and delegation), Objects (including tangible and intangible artifacts, and the context as an object), Tasks (as goal-triggered activities of (co-)creation, performance, and experiencing); and the multidisciplinary concept of values and forces that trigger action.

* **Objects** of art, both intangible (scripts, programs, video and sound streams) and tangible. Each may well have an electronic identity, with possible tags regarding ownership, location, history of use and movement

* Stakeholders are the various **Agents** in the art ecosystem: artist; supporters of techniques and tools; stage keepers, museums and gallery owners; brokers and auction houses; performers and actors; and the audience.

* Roles. In the art ecosystem, collaboration between people (and other agents) changes: new roles develop (co-creating members of the audience) and roles get exchanged more easily between actors, activities get more easily delegated to systems, and mandating of roles and delegation of activities occurs at a more detailed level than before.

* Each agent will have one or more different **roles** (defined by **goals**, and related sets of **activities** regarding art objects), and each role will relate to different types of experience (including: understanding; emotions; tendencies to act; values toward the piece of art).

* **Activities** with their **goals** will be related to creation; reproduction; performance; exhibition; ownership and maintenance; documentation and communication about; etc.

Our analysis will allow us to set design goals for supporting technology. A new addition contribution to GTA is exploration and research of experience and values. We decide to add values aspect for the case of art, based on observing the impact of values in the current art ecosystem on artists, art markets, audiences, buyers, and other stakeholders.

With the rise and development of industry, art is gradually industrialized, which is representative of film and television animation. The film industry is a complete industrial chain. The embodiment of aesthetic value is only one of the links, though it is the most basic. After the production and distribution of films, value is still fermenting. If
the cultural value and aesthetic value of a film have a broad and lasting influence, then
its collection value, commercial value and other values will change with time. This
phenomenon is not only controlled by the art market. Artists and agents should think
about how to create valuable art, and our expanding task analysis ontology is intended
to provide a scientific theory and practical tool that can help artist and stakeholders.

References

1. Qin Tan, L.: Singularity Art - How Technology Singularity Will Impact Art. China Machine Press (2018).

2. LuXun academy of fine arts (LAFA), China http://www.lumei.edu.cn

3. Exquisite Corpse – Visual Arts. http://uhbmi.ee.uh.edu/portfolio/ec-m-2/

4. Hurwitz, J.: THE FRAGILE GIANT - The smallest man-made object ever to be filmed. (2015) https://jontyhurwitz.com/fragile-giant

5. IBM: A Boy and His Atom - The world's smallest movie. (2013)
https://www-03.ibm.com/press/us/en/photo/40982.wss

6. Li, D., van der Veer, G. (2017). From Painter to Interaction Designer: The Evolution of Visual Art Things. In: Anirudha Joshi, Devanuj Balkrishan, Girish Dalvi, Marco Winckler (Eds.): Adjunct Proceedings INTERACT 2017 Mumbai. Springer: pp. 139-149

7. Kristin Hugo. Newsweek: EXCLUSIVE: WHATEVER HAPPENED TO THE MOUSE WITH THE EAR ON ITS BACK? (2017) https://www.newsweek.com/tissue-surgeon-ear-mouse-human-organs-transplant-cell-phones-666082

8. Diemut Strebe.: Sugababe (2014) http://diemutstrebe.altervista.org

9. Candy, L. (1997) Computers and Creativity Support: Knowledge, Visualization and Collaboration. Knowledge-Based Systems, 10, (1), pp. 3-13.

10. Candy, L., Edmonds, E.A.: Modeling Co-Creativity in Art and Technology. Conference: Proceedings 4th Conference on Creativity & Cognition, Loughborough, United Kingdom, 2002. https://dblp.uni-trier.de/db/conf/candc/candc2002.html

11. Li, D., van der Veer G.C.: Cognitive ergonomics on the move. ECCE 2019 workshop. UK. (2019)

12. Nijholt, A. (Ed) (2019) Brain Art: Brain-computer Interfaces for Artistic Expression. Springer, Switzerland.

13. Wang, U., Yu, C., Yang, X.D., He, W., Shi, Y.: EarTouch: Facilitating Smartphone Use for Visually Impaired People in Mobile and Public Scenarios. (2019). CHI 2019, ACM Digital Library

14. Smart glove works out what you are holding from its weight and shape. Nature, DOI: 10.1038/s41586-019-1234-z. published 8 June 2019.
https://www.newscientist.com/article/2204736-smart-glove-works-out-what-youre-holding-from-its-weight-and-shape/

15. Van der Veer G.C., Kulyk O., Vyas D., Kubbe O., Ebert A. (2011) Task Modeling for Collaborative Authoring. In: Anke Dittmar & Peter Forbrig (Eds) Designing Collaborative Activities - Proceedings of ECCE 2011. ACM Digital Library, pp. 171-178