GA2013 – XVI Generative Art Conference

Tatsuo Unemi

Topic: Interactive
Installation
Authors:
Tatsuo Unemi
Soka University,
Department ofInformation SystemsScienceJapanwww.intlab.soka.ac.jp/~unemi/

Installation: Non-stop Evolutionary Art You are Embedded in
Abstract:
In a similar mechanism as the huge variations of complex organisms
appeared and disappeared on the earth through the evolutionary
process for billions of years, it is possible to organize a process of nonstop development in the computer to produce unpredictable complex
patterns, utilizing a technique of evolutionary computation. This visuallyinteractive installation is one of the extended variations of author’s
previous work entitled fully automated evolutionary art [1]. It
continuously produces abstract images and animations by a type of
evolutionary computation based on a minimum gap of generationalchanges and combination of computational aesthetic measures as thefitness criteria. It was purely autonomous without any effect from theoutside, the author extended it using Kinect® camera to embed thevisitors’ figure in the visual output. The information captured by thecamera does not affect to the evolutionary process, but it changes theaudio output because the process of sound synthesis is based on thefeatures extracted from the visual output [2].
This is the shallowest connection between the production process andthe action of visitors, but it’s more enjoyable than the oneway effectwhere the visitors are just observing the production. And, it is the startpoint toward deeper interaction and mixture between the life in themachine and the life in the nature.

References:[1] Tatsuo Unemi, “A
Fully AutomatedEvolutionary Art”,
GA2012, Lucca, Italy,
2012[2] Tatsuo Unemi,“Synthesis of Sound
Effects for GenerativeAnimation”, GA2012,
Lucca, Italy, 2012
Example: Image visual output.

Contact:
Keywords:
unemi@iss.soka.ac.j Evolutionary art, visual interaction, interactive installation,
pautomatic art

page # 247

h16 Generative Art Conference GA2013

Non-stop Evolutionary Art You are Embedded inProf. T. Unemi, BEng, MEng, DEng.
Department of Information Systems Science, Soka University, Hachioji, Japanwww.intlab.soka.ac.jp/~unemi/
e-mail: unemi@iss.soka.ac.jp

PremiseTo extend an evolutionary art to be interactive with visitors, we introduced amechanism to mix the visitor’s image captured by the live camera and the image
produced by the automated evolutionary process. This is the shallowest connection
between the production process and the action of visitors, but it’s more enjoyablethan the one-way effect where the visitors are just observing the production. It is a
start point toward deeper interaction and mixture between the life in the machine andthe life in the nature.

1. Introduction
In a similar mechanism as the huge variations of complex organisms have appearedand disappeared on the earth through the evolutionary process for billions of years, itis possible to organize a process of non-stop development in the computer toproduce unpredictable complex patterns, utilizing a technique of evolutionarycomputation. Evolutionary art is one of the generative approaches to produce
interesting complex patterns inspired from the evolutionary process of naturalorganisms. The framework of Interactive Evolutionary Computing has been used byseveral number of artists and researchers in order to generate these kinds ofpatterns that fit with human’s preferences, just similarly to the breeding process ofornamental plants and pet animals following a type of aesthetic criteria. This method
is obviously based on a type of cooperative mechanism between man and machine,where a human is always taking a necessary role for creation.
On the other hand, owing to the recent improvement of image processing technology
in both software and hardware, the automated production of similar type of patterns
became possible. This visually interactive installation is one of the extended
variations of author’s previous work entitled fully automated evolutionary art [1]. It
continuously produces abstract images and animations by a type of evolutionary
computation based on a minimum gap of generational changes and combination ofcomputational aesthetic measures as the fitness criteria. It was purely autonomous
without any effect from the outside. We have received many times of comments fromthe visitors that it would be more interesting and playful if it could have a type ofvisual interaction with visitors. These comments should be agreed considering fromour experience with flocking orchestra [2] and identity-SA [3] that are installations ofvisually interactive swarm utilizing a live camera and generating sounds. The visualfeedback easily and effectively attracts the visitors of wide generations from theelementary school kids to elder persons if the response is quick enough. Even
though the information captured by the camera does not affect to the evolutionary
process, it is effective if it changes the result visuals. At the same time, the audio
output is also affected because the process of sound synthesis is based on thefeatures extracted from the visual output [4].

page # 248

h16 Generative Art Conference GA2013

There are several types of possible methods to mix these two images. Our approach
introduced here is relatively simple but effective enough to attract visitors.

2. Mixing Methods
SBArt is using functional expression as the genotype that maps a coordinate in thespatiotemporal space to a color value in the HSB (hue, saturation and brightness)color space. Each of the frame images of animation is rendered with x and ycoordinates and the same value for the time variable t for each pixel. The visitor's
real time image captured by the live camera is usually a distribution of color values ina two-dimensional lattice. There are several possible methods to combine thegenerated image and the captured image, but we need to design a method to make iteasy for visitors to recognize both images in the display at same time. One of theeasiest methods is to calculate the average color value between values extracted
from corresponding position in the two images for each pixel, but it is uninteresting
because mutual affection is too obvious. It is easy for visitors to separately recognize
each of the original images as shown in Figure 1.
Another method to keep the color combination of the generated image is to shift thetime value by the brightness, or another scalar measure, extracted from the captured
image. Figure 2 shows a sample result image. This method provides an effective
mixture of two different images in different features, the color distribution fromgenerated image and shape from captured image. However, it doesn’t seem suitablebecause the smooth gradation typically drawn in expression-based evolutionary art
has gone from the result image due to a spatial noise in the background part ofcamera input.
To achieve the requirement that it should present both features of evolutionary artand recognizable visitors figure, we introduced a method to eliminate the effect ofcamera image from the part of background by extracting the part of visitor’s body. It
is not difficult by using a simple method to take a difference of pre-recorded
background image and the current captured image, if the background of theinstallation environment is stable, that means nothing moving behind the visitors andno fluctuation happens at the lighting condition. To adapt to an unstable environment,
we introduced a Kinect® camera instead of ordinary web camera to use depth,
distance between a camera and an object, but not the brightness. Kinect® is atrademark owned by Microsoft Corporation®. Adjusting the threshold depth todistinguish the object and the background, it is easy to extract the visitor’s figure fromthe captured image. Figure 3 shows a typical example of the result image processedby this method. Only the depth values shorter than the adjustable threshold value
affect the mixing process with the generated image.
SBArt, itself, has two types of alternative modes to embed external images andmovies, namely deformation and discoloration [5]. These methods are also available
to introduce a type of interactivity by using camera image as the external one.
However the disadvantage is difficulty on calculation of the fitness value for eachcandidate in the evolutionary process. Though the evaluation should be processedon the image mixed with the captured image, the selection must be done before
displaying it to the visitor. Some minutes of delay between capture and display isunavoidable. Considering an expected staying time of visitors, the time delay should
be shorter than ten seconds or so, to make them aware of the interactivity.

page # 249

h

16 Generative Art Conference GA2013

Figure 1. A simple mixture of generated image and captured image by taking theaverage color values for each pixel.

Figure 2. Shifting the value of time variable t by the brightness of captured image foreach pixel.

page # 250

h

16 Generative Art Conference GA2013

Figure 3. Mixing visitor’s figure with generated image using Kinect® camera.

3. Implementation
The latest version of SBArt is using compiled code of each genotype to render theimage that enable to utilize hi-speed parallel processing by graphics processing unit.
Owing to this extension, it achieved the real time rendering of smooth animation
faster than 24 frames per second [6]. To continue to take this advantage, we
designed a modification process of shader code in order to embed a captured image.
A new argument was added to the main function of fragment shader to which theimage sampler for the depth values is assigned, and a new code was inserted tomodify the time value t. Figure 4 is an example of modification of shader code, wherethe extended parts are indicated in bold face with underlines. The depth values
greater than the threshold value are replaced with zeros before it forms a sampler
object from camera input.
Because this extension doesn't affect the evolutionary process but only in therendering process for display, we implemented this functionality in the player module
but not in the evolving side. Figure 5 illustrates the system configuration of computers,devices and software modules. The player module receives a source fragment ofshader code from the evolution module, and then modifies it to be applicable to themixing process described above. This module was extended to capture thedistribution of depth values by Kinect® camera and feed the data in a form ofsampler object to the compiled code.
The controller module for Kinect® is written in C language utilizing an open source
software library named freenect by OpenKinect project [7].
Original code:
kernel vec4 individual01(float T, float width, float height){
float t=T*-2.018997+0.769912;

page # 251h16 Generative Art Conference GA2013
vec2 p=(destCoord()/vec2(height,height)*2.-vec2(width/height,1.))*...;
vec3 v=((power(abs((-(vec3(t,p.x,p.y)))-(cnst(t,vec3(-0.375667,...
...
return vec4(v,1.);)

Modified code:
kernel vec4 individual01(float T, float width, float height, sampler cmr){
vec4 C=sample(cmr,destCoord());
float t=(T+(C.x+C.y+C.z)/3.-.5)*-2.018997+0.769912;
vec2 p=(destCoord()/vec2(height,height)*2.-vec2(width/height,1.))*...;
vec3 v=((power(abs((-(vec3(t,p.x,p.y)))-(cnst(t,vec3(-0.375667,...
...
return vec4(v,1.);)Figure 4. An example of shader code modification.

Figure 5. Configuration of computers, audio visual devices and software modules.

4. Concluding Remarks
We implemented a new functionality for visual interaction between a generative
image and a captured image in the display module of fully automated evolutionary art.
This is the shallowest connection between the production process and the action ofvisitors, but it’s more enjoyable than the one-way effect where the visitors are justobserving the production. There are a number of the other places into which we canintroduce some type of interaction, such as selection of the set of functions forgenotype, parameters in the evolutionary process, projection methods for finalvisuals, and so on. We hope the installation design described here will be one of thestart points toward deeper interaction and mixture between the life in the machine
and the life in the nature.

page # 252

h16 Generative Art Conference GA2013

References[1] Tatsuo Unemi, “SBArt4 for an Automatic Evolutionary Art,” Proceedings of theIEEE World Congress on Computational Intelligence (WCCI 2012 - IEEE CEC2012), pp. 2014-2021, June 10-15, Brisbane, QLD, Australia, 2012.
[2] Tatsuo Unemi and Daniel Bisig, “Flocking Orchestra - to play a type of generative
music by interaction between human and flocking agents.” in C. Soddu ed.
Proceedings of the eighth Generative Art Conference, Milan, Italy, December 1517, pp. 19-21, 2005.
[3] Tatsuo Unemi and Daniel Bisig, “Identity SA - an interactive swarm-basedanimation with a deformed reflection.” in C. Soddu ed. Proceedings of the 10th
Generative Art Conference, Milan, Italy, pp. 269-279, 2007.
[4] Tatsuo Unemi, “Synthesis of sound effects for generative animation,” in C. Soddu
ed. Proceedings of the 15th Generative Art Conference, pp. 364-376, December
10-13, Lucca, Italy, 2012.
[5] Tatsuo Unemi, “SBART 2.4: an IEC Tool for Creating 2D Images, Movies, andCollage,” Leonardo, Vol. 35, No. 2, pp. 171, 189-191, MIT Press, 2002.
[6] Tatsuo Unemi, “SBArt4 - Breeding Abstract Animations in Realtime,” Proceedingsof the IEEE World Congress on Computational Intelligence (WCCI 2010 - IEEE
CEC 2010), pp. 4004-4009, July 18-23, Barcelona, Spain, 2010.
[7] OpenKinect, http://openkinect.org , 2011-2013.

page # 253