th

9 Generative Art Conference GA2006

Flocking Messengers
Prof. T. Unemi, BEng, MEng, DEng.
Department of Information Systems Science, Soka University, Tokyo, Japan.
e-mail: unemi@iss.soka.ac.jp
Dr. D. Bisig, BSc, MSc, PhD.
Artificial Intelligence Laboratory, University of Zurich, Switzerland.
e-mail: dbisig@ifi.unizh.ch

Abstract
Flocking Messengers is an entirely new type of real-time communication tool, which
utilizes a technique from artificial life to allow two remote persons to speak to eachother. Messages that travel through a computer network, cross a virtual 3D space thatis inhabited by flocking agents. Attracted by the speaker's motions, agents approachand memorize his/her voice. Subsequently, they move towards the opposite spaceboundary and recite the speakers' message in their own voice. The rules controllingeach agent are simple, but the reactions of the entire system are complex and
unpredictable since they combine acoustic and visual interaction with collective
behaviour. Agents possess their own emotional state, which they occasionallyexpress through their own words. Thereby they guide the visitor's reactions andincrease his/her enjoyment of the communication.

1. Introduction
When I was a shy boy at school age, I often experienced difficulty in talking to pretty girls ina face-to-face conversation. I soon noticed how different communication tools, such as atelephone or postcard, affect my capability for communication. These tools create formal
protocols and cause different time delays in prompting for a response. We sometimes rely onthese tools to tell something, which is difficult to say directly. Nowadays, the Internet hascreated a wide variety of communication tools such as e-mail, blogs, chat etc. Along withthese technical improvements and an increasing popularity of computers, new styles ofcommunication develop.
Flocking Messengers is an entirely new type of real-time communication tool, which utilizes
a technique from artificial life to allow two remote persons to speak to each other. The system
has been realized on two iMacs (1.83 GHz Intel Core Duo) equipped with built-in cameras
and external microphones. One machine runs our application DT1 (version 1.6) in server
mode. The other runs the same application in client mode. DT1 simulates the flocking
behaviour of agents in a virtual 3D world. In the current version, 128 agents act as messengers.
This number can be increased arbitrarily as long as the employed computer is sufficiently
powerful.
The motion of users is derived from a sequence of captured video frames and causes theagents to move towards the source of the motion. By this way, users can summon agents inorder to tell them their own messages or to listen to the messages of the communication
partner. Both the captured video images and the audio messages are transmitted between thetwo connected computers. The video image obtained on one machine is displayed on the
back-wall of the virtual 3D space on the other machine as shown in Figure 1. The audio
Page 1th

9 Generative Art Conference GA2006

message is not directly played back but rather rendered as individual agent voices. Agents
possess an emotional state, which they can express through their own acoustic messages.
Different agents speak at different time delays and in different pitches that create a chorus ofvoices. This may render the message more difficult to understand but is generally appreciatedas funny and enjoyable.
The following sections describe the technical features of the system concerning flocking
simulation, visual interaction between agents and human, network communication, audio
processing, and visitors’ reaction in an experimental exhibition.

Figure 1. An example view of the screen.

2. Flocking Simulation
The implementation of the flocking simulation and visual interaction methods are largely
based on our previous artwork entitled “Flocking Orchestra [1, 2].” This previous artwork is
an interactive installation, which allows visitors to play music by conducting flocking agents
via gestures.
Each agent is controlled by a set of forces. Some of these forces implement standard boids
type of flocking rules [3] whereas other forces result from the user's visual interaction. The
flocking forces cause the following behaviours:
1.
2.
3.
4.

collision avoidance between agents,
velocity alignment with neighbouring agents,
flock cohesion by attraction towards the centre of the neighbouring agents, andcollision avoidance with the boundaries of the agent world.

The repelling forces for collision avoidance are proportional to the inverse of the square of the
distance between the agent and the obstacle. Based on the sum of all the forces that affect anagent, a goal angle and a goal speed are calculated. The agent tries to meet these goal values
Page 2th

9 Generative Art Conference GA2006

by modifying its current orientation and velocity within the allowed limitations of its steering
angle and acceleration.
To calculate these forces, it is necessary to organize the set of neighbouring agents within an
effective distance from each agent. If we used an exhaustive algorithm to check the distance
between every possible pair of agents, the computational time complexity would be
proportional to square of the number of agents. To reduce this complexity as much as possible,
we introduced a method that divides the 3D space into a number of sub-areas to manage theinformation which agent is located in which sub-area. Since we maintain information about
the locations of agents within these sub-areas, we can restrict distance calculations between
agents that reside within neighbouring sub-areas.

3. Visual Interaction
The flock's behavioural response to user input is based on the calculation of interaction forces.
This implementation is derived from our previous work. The interaction forces lead to thefollowing behaviours:·

movement towards a particular target position on the front plane when user motion isdetected, and movement away from the front plane in absence of user motion.

The target position is calculated in the following way:··a difference image is calculated by summing over the absolute differences of all pixel
RGB values between the current and previous capture images, andfor each agent an individual attractor position is calculated.

This position is derived by multiplying the RGB difference values of all pixels that lie within
the neighbourhood circle of an agent with their corresponding x and y position in the camera
image.
The system differs from the previous implementation by the fact that interaction dependent
behaviours cause agents to move either towards the front or back plane of the agent world.
The attraction force for an agent is calculated by summing the vectors towards both surfaces.
The distribution of repulsion forces that push agents away from the surfaces is modified asshown in Figure 2.

Figure 2. The distribution of repulsion forces to take agents away from the front and backsurfaces.

Page 3th

9 Generative Art Conference GA2006

4. Network Communication
We examined two types of methods to combine two machines for computation andcommunication. In either method, the images captured by the camera and the sounds capturedby the microphone are transmitted from one computer to the other through the local network.
The two methods differ as to which machine calculates the flocking simulation. One method
is to simulate flocking behaviour independently on each machine. The other method is tosimulate flocking only on one machine and to send agent position and pose information to thesecond machine. Prior to rendering the agents, the second machine rotates every agent by 180
degrees.
In the former method, the main part of the software is symmetrical in terms of computation
and amount of data to be transmitted. To establish a connection between the two machines,
one machine acts as a server and the other acts as a client for the TCP/IP connection.
Otherwise, both machines exchange the same type and amount of data. This data consists ofimage and sound data as well as the 2D motion patterns from visual interaction. Based on the
combination of two motion patterns derived by each machine, each flocking simulation
calculates the corresponding forces that act on the agents. The resulting behaviours of theagents are sufficiently similar in both simulations to provide a communication between theusers.
In the latter method, the flocking simulation is executed only on the server side and the result
is sent to the client side. In order to perform the simulation, the server needs to acquire its
own and the clients motion patterns. Thus, the client sends its motion pattern to the server.
The server on the other hand sends the pose and position information of the agents to theclient. This method guarantees that the same agent behaviours are displayed on both machines,
as long as the network connection is sufficiently fast. Unfortunately, whenever the network
communication deteriorates, the client's rendering of the agents becomes jerky. For thisreason, we are currently employing the former method for exhibition setups. It would be
preferable to implement a more robust communication method, which is able to deal with
network fluctuations in a more robust manner.
For a future installation version, we will evaluate an alternative communication setup, which
consists of one server running the simulation and several clients that each renders the visualand acoustic output. Such a setup would allow a communication between of more than two
remote sites.
Each machine displays a scaled version of the live video image captured by the other machineon the back-wall of the agent world. In the current implementation, the resolution of thecaptured image is 640 × 480 pixels, and that of the wallpaper image is 320 × 240 pixels. Eachpixel of the captured image contains three times eight bits of data to represent an RGB value.
By reducing this amount to three times five bits per RGB value and by employing a loss-lesscompression, the memory requirements for each frame drop to 64 KB or less. The visualmotion pattern possesses a resolution of 160 × 120 pixels. The amount of motion is stored as
8 bit greyscale values. Again, by employing a loss-less compression, these data are reduced toabout 4 KB or less per frame. For a frame rate of at least 15 frames per second, the amount ofimage data that needs to be transmitted equals at most 1 MB per second.
Audio data is captured at a rate of 44.1 kHz and a sample resolution of 16 bits in LPCM
(linear pulse code modulation) format. This data is sampled down to 22.05 kHz, which is ofsufficient quality for the human voice. The total amount of data for video and audio materialsums up 1.4 MB per second. Therefore, 10 Base-T Ethernet or IEEE 802.11b wirelessconnections provide insufficient speed. On the other hand, 100 Base-TX, IEEE 1394, or IEEE
802.11g wireless connections are capable of handling the required amount of data. All recent
iMacs fulfil these requirements.

Page 4th

9 Generative Art Conference GA2006

5. Audio Processing
5.1 Capturing messages
When an agent is close to the front plane of the virtual space, usually because it has been
attracted by a visitor's motion, it is ready to listen to a voice input from the microphone.
Symmetrically, an agent will start to listen to the voice of a remote visitor when it appears
close to the back-wall. An agent starts recording when it detects a sufficiently high soundlevel. It stops recording as soon as two seconds of silence have passed. If the recording is atshortest one second in duration it is memorized, otherwise it is discarded. Instead of wasting
memory by recording sound data for each agent, only the time-stamp of the beginning of therecording and its duration are memorized by each agent. The recording information of thevoice from the microphone is used only to control the aural and visual response of the agent,
but the information for the voice from the remote machine is used additionally to render audio
output.
When the agents are listening to a human voice, they visually straighten their ears and orient
their faces towards the visitor even when moving into another direction (see Figure 3). They
blink their eyes in random intervals, and close their eyes when the input voice is too loud.
After an agent has stopped listening, it turns into the opposite direction and moves away byadding a constant force towards the remote listener and ignoring any interaction based on
attraction forces towards the speaker.
For an exhibition setup, directional microphones have to be employed in order to avoid audio
feedback loops. For this reason, an external cardioid type of microphone is used instead of theiMacs internal microphone. The usage of an external microphone has the additional benefit ofproviding a visual cue that a visitor should speak towards it. The iMacs internal microphonemight be used in combination with audio output via headphones. Such a setup is suitable forprivate use or in order to avoid problematic sound situations at an exhibition. We will look
into a software-based sound filtering methods that would allow the usage of the iMacs
internal microphone and speakers. Such a solution would be beneficial since any iMac owner
can employ the system without a need for additional equipment.
5.2 Modification of human voice
As described above, each agent listens to a voice message, moves to the other side, and thenspeaks the message in its own voice. This modification simply consists of shifting the originalaudio material to a higher pitch. The easiest way to lift a sound's pitch consists of increasing
its playback speed. Unfortunately, the corresponding shortening in the sounds duration causes
the message to become difficult to understand. To maintain the sounds original duration, ouraudio processing algorithm duplicates a cycle of the waveform to fill in the gaps in the higher
pitched sound. The computational cost this method is too expensive to calculate individual
voices for each agent. Instead, we calculate and store only three frequency shifts. These shifts
correspond to a frequency ratio of 4/3, 5/3 and 2. These ratios represent a triad of a musicalchord in the major scale. Accordingly, the resulting chorus of voices is not only cheap to
calculate but exhibits harmonic relationships.
5.3 Speaking messages
The voices spoken by the agents are created by extracting the corresponding parts from thebuffers containing the frequency shifted audio inputs and by mixing all voices together into
the final sounds output from the loud speaker. Based on our preliminary experiments, weobserved that the content of a voice message becomes very difficult to understand if morethan five agents are simultaneously speaking at different time delays. To reduce the number of
speaking agents, we implemented an agent priority queue. In the current implementation, thefirst five agents that arrive at the frontal plane of the agent world contribute to the audio
Page 5th

9 Generative Art Conference GA2006

mixing. Within this group of five agents, the amplitude of every successive agent's audio
signal is multiplied with a factor of 2/3. The audio outputs of all other agents are ignored. The
resulting audio output still sounds like an echo, but the message's content remains clearly
understandable. Figure 4 illustrates the entire process from capturing a visitor’s voice atmachine A to generating the sound played by the loud speaker connected to machine B. The
voices captured at machine B are processed symmetrically to generate a sound output at
machine A.
When an agent is speaking, it moves its mouth. The size of mouth is adjusted to the loudnessof the voice. A cartoon type of speech bubble appears above the agent as shown in figure 5.
These bubbles aid the visitors to recognize the agents that are speaking. The panning of theaudio output depends on the horizontal distribution of the speaking agents.

Figure 3. Agents listening to the voice. They straighten their ears and blink their eyes.

Figure 4. The entire process for audio data.

Page 6th

9 Generative Art Conference GA2006

Figure 5. Agents speaking messages. They move their mouths and the bubble signs are drawn
above them.

5.4 Words of agents
When we performed our first experiments with an early version of the system, we noticed thatsome visitors did not say anything but simply enjoyed conducting flock. We therefore
concluded that it was necessary to embed some functionality to invite visitors to speak
something into the microphone. We came up with a solution that causes agents to speak somewords in order to motivate and prompt the visitors to say something even if they don’t know
how the system works. The generation of agent words is based on the following rules. If no
motion is detected for longer than twenty seconds, agents become bored and say things like“boring” or “nobody there?” Once they detect a motion, they greet the visitor with statements
like “hi” or “how are you?” If motion is sustained but no acoustic input is detected, agentsreact by saying, “say something” or “message please.” As soon as agents detect a sufficiently
high sound level, they start listening. The agents record any sound input that lasts for at least
one second and is followed by a pause of at least two seconds. They acknowledge a
successfully recorded message by saying “OK” or “I got it” and subsequently start to movetowards the opposite boundary of the virtual 3D space. This opposite boundary corresponds tothe front wall of the agent world on the second computer station. Once agents reach thisboundary, they announce the presence of a message by saying “you got a message” or “pleaselisten.” Subsequently, they speak the message in their own voice and finally add statements
such as “that's all” or “that's it.”
These words stem from recordings of a human’s voice. This voice has been modified byshifting its frequency by factor two. The words are stored in audio files, which form part of
the application bundle. When the system renders the output sound, the words are loaded fromthe audio files, and are modified again by pitch shifting. This time, the shifting ratio is
specific for each agent and varies uniformly from 0.8 to 1.3. The final pitch of the words istherefore higher by a factor of 1.6 to 2.6 than the original human voice.

Page 7th

9 Generative Art Conference GA2006

6. Visitors’ Reaction
We organized two experimental exhibitions at Soka University. The first one was held in theOpen Campus in August, and the other one was at the Campus Festival in October 2006. Bothof the events gathered thousands of people and more than one hundred persons haveexperienced our installation during each event. In the first experiment, the software was stillat an early stage and the agents did not say their own words. This feature was present in the
newer software version, which was shown during the second exhibition. This time, theexperiment clearly showed that the agents’ words are effective to invite more visitors andpromote them to speak into the microphone. During both exhibitions, a short instruction wasprovided that helped visitors to understand what was going on and how they could enjoy thesystem in their own style.
One typical visitor reaction was expressed a group of young boys ranging in age from four to
ten years. They were often moving their arms to examine the effect on the agents' movements.
They grasped the microphone very close to their mouths and shouted meaningless words.
Older visitors often exhibited a different type of reaction. They told us that the agents are very
cute and seemed to enjoy listening to the agents’ voices rather than speaking themselves withthe remote person. Some visitors didn't know how to end a conversation. When we are talking
on the phone, it is obvious to say “goodbye” to stop the conversation. But for this newcommunication tool, there doesn't yet exist a common rule to end a session.

7. Concluding Remarks
So far, the Flocking Messengers project has succeeded in surprising many visitors. Some ofthese visitors told us that they encountered a magical experience that they had never seen
before. One of the essential aspects of the system is its complexity. The behaviour of thesystem is unpredictable because it combines a flocking simulation with acoustic and visualinteraction. The agents and visitors influence each via their movements and voices. Anotherimportant aspect is the system's real-time response to a large amount of input data from its
physical environment. The default input channels of standard personal computers used to be
restricted to the keyboard and mouse. Normally, when working on a PC, a user always
completely knows what data he/she had input into the computer. Any unintended input could
clearly be attributed to a user mistake. However, the input patterns from a camera and a
microphone are too rich to be completely controlled. It is difficult to realize the same input
data again, and some data are often input regardless of the operator’s intention. This type ofrichness in input data provides the computer with a source of complexity.
The system works with MacOS X 10.3.9 or later running on a PowerPC G4, G5 or Intel CPU.
The source code was written in Objective-C and C by using the Xcode programming
environment. The compiled application is available on the Internet by accessing the following
URL:
http://www.intlab.soka.ac.jp/~unemi/DT1/
In it's current version, network communication is based on a simple BSD socket library. This
leads to the drawback that a user has to manually enter the IP address of the remote machine.
A more user-friendly approach would employ a plug-and-play type of communication service
such as Apple's Bonjour. We would like to distribute our software to as many people aspossible. For this reason, it is very important to make Flocking Messenger very user friendly
and simple to use.
A drawback of the current system consists of the fact that it doesn’t distinguish between ahuman voice and other input sounds. Sometimes, the agents started to transmit a message
when a user breathed loudly in front of the microphone. In such a situation, the agent says
“OK,” leaves away toward the back-wall, and makes a noisy sound at the remote side. To
Page 8th

9 Generative Art Conference GA2006

avoid such undesirable agent reactions, an algorithm needs to be implemented that
discriminates between different types of sounds. For the next software version, we willaddress this problem.
In summary, each individual technique that we have implemented is very simple and rather
straightforward. But the combination of these techniques is very novel and provides uniqueuser experiences. In addition, these techniques needed to be carefully integrated in a complex
real time system that works smoothly and reliably. In particular, we needed to make sure thatthe interactive experience is not obstructed by slow response times. As technology progresses,
we are looking forward to realize further interactive installations that provide uniqueexperiences for visitors.

References[1] Unemi, T. and Bisig, D.: Playing Music by Conducting BOID Agents - A Style ofInteraction in the Life with A-Life, Proceedings of A-Life IX, pp. 546-550, 2004.
[2] Unemi, T. and Bisig, D.: Music by Interaction among Two Flocking Species and Human,
in T. Innocent ed. Proceedings of the Third International Conference on Generative
Systems in Electronic Arts, Melbourne, Australia, pp. 171-179, 2005.
[3] Reynolds, C. W.: Flocks, herds, and schools: A distributed behavioural model, Computer
Graphics, 21(4): 25-34, (SIGGRAPH '87 Conference Proceedings) 1987.

Page 9