From: bobp@hal.com (Bob Pendelton)
Subject: TECH: Sound API?
Originator: hlab@milton.u.washington.edu
Organization: HaL Computer Systems, Inc.
Date: Thu, 1 Oct 1992 22:46:11 GMT



What I want to talk about is what people would want in a sound API for
writing VR applications. I don't really want to talk about
implementing anything. I find this kind of discussion works best if
you think about what you want and let reality settle you down later.

After this discussion maybe we can get talking about glove/body suit
APIs, VR graphics APIs, smell APIs... Or even some important things
like object interactions.

When I think of sound in VRs I think in terms of sound sources,
motions of sound sources, and physical properties of materials that
affect the propagation of sound.

I also worry about synchronizing sound with the actions of things that
I can see. There are a lot of problems here. You can get away with
slowing down the frame rate for the graphics, but once you start a
sound, you're pretty committed to it. Starting and stopping sounds or
stretching them out just doesn't work to well. Or does it?

1) What makes a sound.

So, what is the basic sound producing object called? I'm fond of
calling them "voices." 

You need to be able to create and destroy voices

        voice CreateVoice(void);

creates a voice.

        DestroyVoice(voice);

Destroys a voice.

        SetVoiceVolume(voice, volume);

How loud is it.

        SetVoiceAt(voice, location);
        SetVoiceWith(voice, object);

Where location can be just an x,y,z coordinate for disembodied voices
or the id of an other object, such as a polygon, so that you can
associate a voice with the picture of a speaker on a virtual radio, or
telephone, or some things mouth.

The mention of telephones reminds me that you have to be able to
simulate microphones too. But, hey, you need virtual microphones to
"pick up" the virtual sound and convert it to real sound so you can
hear it, just like you need virtual eyes so you convert virtual light
to real light so you can see it.

        SetVoiceEnvelope(voice, envelope);

The "shape" of the sound being generated. The sound may be stronger in
one direction than in others. It might have certain frequencies damped
out depending on the direction relative to the envelope. 

For example, depending on the distance, the "shape" of the envelope
and the intensity of sound and the frequency spectrum of the that you
hear will vary from location to location. If the envelope is spherical
for all frequencies then only distance will affect what you hear...

In a virtual environment we could simulate voices that are like
lasers, perfectly focussed that can be heard at any distance, or
voices that can be heard everywhere in the univers with equal clarity,
or voices that have a constant intensity for a fixed distance from the
voice that then cut of completely.

It looks like these special effects may be easier to implement than
normal, real world, sound would be.

Envelope is probably a bad term for this, but I couldn't think of
anything else.

The envelope of the voice is a very complex thing. I don't know how to
represent it well. It could be represented with a set of tables that
are interpolated in a standard way, but that might not work out. I
expect that the "envelope" argument will have to be a procedure that
is called with a standard set of arguments when an envelope based test
needs to be performed.


        SetVoiceDirection(voice, at, up);

A pair of vectors, the direction the envelope is pointing at, and the
up direction. Rather like the direction of a spot light. You need both
at, and up because the envelope can be most any shape.


        SetVoiceMotion(voice, ...)

Motion deserves a section all it's own. To specify motion you really
need to specify a starting location, a velocity, an acceleration, and
a jerk (no not me, I mean the rate of change of acceleration). Each of
these is a vector quantity. You also need to specify a starting time,
or event, and an ending time, or event.

I guess I forgot about specifying rotation as it moves along it's
trajectory. Hum, I think specifying the location, orientation, and
motion of the envelope of a voice is the same as specifying those
things for any other virtual object or field, so I'll punt them from
here. 

2)Propagation

Sound, like light, is affected by what it travels through. Some
materials reflect sounds, some absorb them. Sound goes around corners
and through solid objects, but doesn't travel through empty space.
Unlike light, sound moves so slowly that it's velocity can be measured
using a pair of binoculars, two people, and a stop watch. Also, it's
affected by Doppler shifting which isn't visible in light in day to
day life.

This all means that a good sound renderer (I'll use the term renderer
here for lack of a better word) will have all the complexity of a ray
tracer and then some. Doing detailed sound rendering in real time is
not going to be easy.

Please treat the lack of detail in the following sections as a plea
for information. What do YOU want?

2.1)Surface Properties.

Just like light, sound reflects off of the surfaces of objects. When
we talk about light bouncing off of something we talk about the color
of the object and whether the reflection is diffuse or specular.

An object has a color because the material it is made of absorbs some
frequencies of light and reflects others. Materials have sound
"colors" too. They absorb some frequencies of sound and reflect
others.

But, the eye can be fooled by using mixtures of just 3 colors of light
to produce most any visible color. I don't think there is anything
like this for sound. The ear directly detects many different
frequencies of sound with the little hairs in the inner ear.

So, I expect that we need to have a way of specifying how sound is
reflected off of a surface, diffuse versus specular, and the
absorption/reflection spectrum of the surface.

This is also the problem of transmission of sound across the
interfaces between different materials.

2.2)Volume Properties.

Sound is affected by the medium that it is propagating through. It
looks like we would need to specify the speed of sound in each
material, what happens to sounds as they are carried by a medium, what
else?

This also raises the question of simulating air. In most simulations
I've seen the space between objects is treated as being a vacuum. This
won't work for detailed sound simulations.

2.3)Going around corners and through windows...

These are implementation related problems. Let's skip over them for now

3)What sound to make.

There are at least 3 different kinds of sound sources.

3.1) Canned sounds.

These are typically digitized sounds that are just played back
verbatim.

        PlayBack(voice, soundId);

Generate the sound using the state of the specified voice. You might
want to be able to specify a time at which to start the play back.

        PlayBack(voice, soundId, time);

You might also want to be notified when the sound is finished.

        PlayBack(voice, soundId, time, callback, tag)

Where the callback is a routine that will be called with the argument
tag when the sound is finished playing.

3.2) Scripted sounds.

Scripts are things like midi files or text files that you want
converted into sound by a synthesiser. This could generate music, or
synthesised human speech.

        Play(voice, script, time, callback, tag)
        Say(Voice, script, time, callback, tag)


3.3) Real world sounds.

Let's say you want to have a real radio connected to a voice in your
virtual world, or you want to simulate a telephone, or you just want
to be able talk and have some other participant hear you.

You have things like radio and TV tuners, and microphones that you
want to connect to a voice in your virtual world. The sound has to be
converted to match the position and envelope of the voice.

        Connect(voice, device, callback, tag)

connects a voice to a real world device. If the connection is broken
the callback will be called with the tag as it argument. 

        Disconnect(voice)

Break the connection between the voice and its sound source.

We'll need an API for opening and closing devices and querying for the
presense and capabilites of devices. 

4)A minimal API.

Ok, I think that the few routines I've specified in any detail give a
good start for a minimal sound API. I'm sure I've missed a million
things. Tell me what I have wrong.

                                Bob P

-- 
Bob Pendleton             | As an engineer I hate to hear:
bobp@hal.com              |   1) You've earned an "I told you so."
Speaking only for myself. |   2) Our customers don't do that.
                   <<< Odin, after the well of Mimir. >>>