# Presentation: 556463

## Sonification Data Analysis Tool Using JavaSoundby Anton Schertenleib & Robert Candey

**Sonification Data Analysis Tool Using JavaSound****by Anton Schertenleib & Robert Candey**

- http://spdf.gsfc.nasa.gov/research/sonification

**Notes:**
Hello and welcome to this talk about sonification. 

I have the honor today to clarify the expression Sonification and to present the prototype for data analysis tool called “xSonify”. 

I would be more than pleased if I am able to implant this idea of displaying and exploring data as sound in a hidden corner of you mind. So maybe one day you are faced with a problem where you consider Sonification as one of the solutions for it.

## Overview

- Overview

- Introduction
- Definition Soni-/ Audification
- Benefits Of  Auditory Display
- Example Of Use
- ICAD
- Sonification Tool: ***xSonify*** (Prototype)
- Sonification Process
- Technical Background
- Demonstration Of xSonify

**Notes:**
What’s coming up the next 30 minutes shows my overview. 

It consists mainly of two parts: 

The Introduction where I clarify the expression “Sonification” and try to give an insight of the opportunities of this way to display data.

The second part is focused on our project here at Goddard where the brand new Sonification prototype “xSonify” contains some methods of Sonification and builds the first official framework for Sonification here at Goddard.

## Definition Sonification

- Definition Sonification

- Three groups for Sonification
-  Iconic Sonification
-  Direct Conversion Audification
-  Musical Sonification
- John M. Keller, 2003, “Private Communication”

**Notes:**
According to Mr. John Keller, a scientist in the fields of Auditory Display and founder member of ICAD, there are different types of Sonification and they  can be categorized in three different  groups.

Iconic sonification: 
This type of sonification applies when someone maps data to sound that are associated with certain phenomena. 
For example, if we gathered weather data, such as cloud cover, temperature, and humidity, to calculate the probability of rain tomorrow, then using the sound of rain to indicate when there is a high probability of rain would be an iconic sonification.
Or another example: Everybody knows how it sounds if you click on your trash on your desktop of your computer to empty it.

Direct Conversion Audification: 
This type of Auditory Display applies if someone maps data to sound to listen for patterns that are represented in the data. 
For example, space scientists map data of electromagnetic waves to sound waves. 
This direct conversion can be as simple as taking the frequencies of the waves and making sound waves with the same frequencies, which is most useful as long as the frequencies are at pitches that our ears can hear. 
Audification is a subgroup of  Sonification. The two terms are very often mixed up however their meaning is slightly diverse. 

Musical Sonification: 
This type of sonification is when someone maps data to sound in a musical way.   
This is the category in which our software tool appears. We map basically data and play it supported by MIDI. But more information later on. 

## Definition Sonification

- Definition Sonification

_**Example:**__** **_
- A church bell as the simplest
- 	form of Sonification.
- The amount of peals conveys
- 	information about the current
- 	time.

**Notes:**
First click on image!

Sonification is data controlled sound. The data are like the notes of a music play. 

The numerical data are mapped and the result of the mapping is the input for a device which turns the information into sound according their values. I will talk about the technical background in about 5 slides.

As a very basic example for Sonification can be seen a church bell which conveys the current time.

## Definition Audification

- Definition Audification

_**Example: **_
- Cassini spacecraft crossed the bow shock of Saturn
- Cassini’s Radio and Plasma Wave Science (RPWS) instrument

- http://www-pw.physics.uiowa.edu/space-audio/cassini/bow-shock/

**Notes:**
In comparison to Sonification, Audification is more the audible playback of a data sample. 

During the Audification process the frequencies are converted into audible or hearable frequencies.

As you can see in here in the plot the Cassini spacecraft crossed the bow shock of Saturn at 9.45 Universal Time on June 27, 2004.

>>The bow shock is a discontinuity that forms in the solar wind when the supersonic solar wind encounters the magnetic field of a planet, very similar to the shock wave that forms upstream of an aircraft moving at a supersonic speed. <<

The color frequency-time spectrogram in the diagram shows the electric field intensities detected by the Cassini’s Radio and Plasma Wave Science (RPWS) instrument. The bow shock crossing is indicated by the abrupt burst of electric field noise at the time marked by the arrow. 

The electric field noise is caused by electrical currents that flow in the shock. Time on this recording has been compressed such that these 10 seconds corresponds to 28 minutes.

The data what we can see here visually look like the following acoustical sequence for our ears.

Click on image!

## Benefits Of Auditory Display

- Uncovering Patterns
- Identifying New Phenomena
- Improving data exploration
- Analyzing Complex, Rapidly Or Temporally Changing Data
- Monitoring Data While Looking At Something Else
- Improving Visual Perception When Accompanied By
- 	Audio Cues

**Notes:**
So what are actually the benefits of Auditory Display?

Sonification can improve and increase the bandwidth of the interface “human-computer” and can find a lot of applications in the wide range of information technology.
The human ears provide a very good alternative to the visual way of reception information for visualization and understanding complex scientific data. 
With Sonification it is possible to display effectively large and multidimensional datasets which could help in finding otherwise hidden correlations and patterns. 
This will provide users with alternative and additional ways of identifying and extracting physical signatures represented in the data, including selection and inter comparison between datasets. 
In addition to improving data exploration and analysis for most researchers, the use of sound is beneficial as an supporting technology for visually impaired people and non- visually-oriented people.
As a third big advantage Sonification can also make science and math more exciting especially for young students because people learn in many different ways.

Beside these advantages there are more like:

- Uncovering patterns masked in visual displays
- Identifying new phenomena current display techniques miss
- Improving data exploration of large, multi-dimensional datasets
- Analyzing complex, rapidly or temporally changing data
- Monitoring data while looking at something else as kind of background event-finding
- Improving visual perception when accompanied by audio cues

## Example Of Use

- Example Of Use

- Voyager 2 Displayed Micrometeorids As
- “Hailstorm” Sounds  When Traversing Saturn’s rings.

**Notes:**
The best known example in space physics was the use of sound for detecting micrometeoroids impacting Voyager 2 when traversing Saturn’s rings. These impacts were obscured in the plotted data but were clearly evident as Hailstorm sounds.

Especially in this example there can be seen the high significance of using audio to display data.

## ICAD

- ICAD

- ICAD - _I_nternational _C_ommunity For _A_uditory _D_isplay
- Founded 1992, Santa Fe, CA
- http://www.icad.org/

**Notes:**
In the context of Sonification I also have to mention a community which was came together to build the framework for Auditory Display - The ICAD.

ICAD is a forum for presenting research on the use of sound to display data, monitor systems, and provide enhanced user interfaces for computers and virtual reality systems. 

It was founded in 1992 in Santa Fe, California

The research areas covered by ICAD include:

・Auditory exploration of data via sonification and audification 
・Real Time monitoring of multivariate data
・Perceptual issues in Auditory Display
・Sound in generalized computer interfaces
・Technologies supporting Auditory Display creation
・Data handling for Auditory Display systems
・Applications of Auditory Display

Included within each of these areas are many issues concerning application, hardware/software, and human factors.

## Sonification Attributes

- Sonification Attributes

**Notes:**
Now I would like to tell a little bit more about the technical background of Sonification.

As you can see, sound has different attributes where Sonification can take advantage of.
Attributes like the Location, Loudness, Pitch, Timbre, Duration and Order of sound. Also the Rhythm and the time to reach its minimum or maximum can be chosen as attributes to display the data.

It is however necessary to keep in mind that some attributes are just not effective to display certain kind of data.

For example it is hard for the human ear to distinguish a huge amount of different data values with the “Order of tones” as sonification attribute.  

## Sonification Process

- Sonification Process

- Sylvain Daude, Laurence Nigay, 2003, “Design Process For Auditory Interfaces”

- Data Pipeline:

**Notes:**
Sylvain Daude and Laurence Nigay, two french computer scientists from Grenoble/France, presented a sonification process which transforms and prepares the raw data to be finally sonified. The process can be considered as some sort of “Data Pipeline”. 

The single steps of this process are:

From Data To Data View: Data Transformation
In this step the raw data will be mapped value by value. In xSonify’s case into a range of values between 0 and 1. This process can be considered as a kind of standardization of values which is necessary to make the data available independent from their unit and scale.

From Data To Abstract Sound Space: Sonification Transformation
During this step the data are prepared according to the chosen Sonification modus. xSonify provides 3 different Sonification modes like the Pitch, Volume and Rhythm. But more details about that later on. Every value will be assigned to a certain position at a “time line”. This “time line” represents the time line of the sonification sequence which will be played in xSonify in the MIDI player.

From Abstract Sound Space To Sonic: Auditory Display Transformation
This is the part where the signal is finally displayed on a physical device. In xSonify this is the part where the MIDI sequence will be transmitted to the MIDI player and be played. 

## Sonification Tool: xSonify (Prototype)

- Sonification Tool: ***xSonify*** (Prototype)

- Facts:
-  Java 1.5
-  Java Sound API / MIDI
-  WebStart Technology
-  Speech Support
-  3 Different Sonification Modes

**Notes:**
After we heard about the technical background of Sonification I would like to switch over to our recently developed data analysis tool - xSonify.

xSonify in the current state is still a prototype. It was developed in a relatively short term effectively 3 months and is beside a few bugs almost fully functional.

It was developed in Java Version 1.5 and takes use of the Java Sound API. In order to make it easily accessible for everyone the application is based on Java’s WebStart Technology. Java’s WebStart is a technology to deploy Java application in a convenient way via networks like the Internet. It brings in all the security of the “Java Sandbox” which means that you can run an application with the confidence that it will not over step and meddle with files on the client machine. Java WebStart is designed to download all the files required the first time. It is invoked from a Web page. After the first download the user has even the option to include a shortcut to the application directly on the windows start menu or at the Desktop if he uses Microsoft Windows. The application can be designed to check for updates on the server to refresh the existing local copy with the changes made since the last access.

In order to support blind scientists, we included our own speech support with Java Speech API. That way we are independent from expensive screen reader software. 

xSonify includes three different Sonification Modes like the Pitch, Volume and Rhythm. I will demonstrate the effect of these three modes later on.

## Technical Background

- Technical Background

**Notes:**
Our sonification tool -xSonify- was developed with Java 1.5.

The main reason for choosing Java Version 1.5 was mainly the enhanced Java Sound API beside all the the other goodies which came with Java Tiger or better known as version 1.5. 

- Optimized direct audio access is implemented on all platforms.
- The new real-time Sequencer works with all MIDI devices and allows unlimited 		  
   transmitters.
- The Sequencer interface is extended with loop methods for seamless looping of specific  
   portions of a MIDI sequence.
- Java Sound no longer prevents the VM from exiting.

## Technical Background

- Technical Background

**Notes:**
The Java Sound API includes support for both digital audio and MIDI data. These two major modules of functionality are provided in separate packages.

In the left area you can see the two “sampled” packages which are handling digital audio. Java Sound API refers to it as sampled audio. Samples are successive snapshots of a analog signal. I will not explain the use of this package further since it is not used in this software application. In the right area there are the two “midi” packages for MIDI synthesis, sequencing and event transport. 
The two lower packages ending with “spi” permit service providers to create custom software components that extend the capabilities of an implementation of the Java Sound API. “spi” stands for service provider interface.

The Java Sound API does not assume a specific audio hardware configuration; it is designed to allow different sorts of audio components to be installed on a system and accessed by the API. The Java Sound API supports common functionality such as input and output from a sound card (e.g. for recording and playback of sound files) as well as mixing of multiple streams of audio.

## Demonstration Of xSonify

- Demonstration Of xSonify

_**Demo I:**_
- Simple Numerical Data
- 40 Values
- Values From 0 To 10 Repeating Four Times
_**Demo II:**_
- Data From “Cluster” Spacecrafts
- 1500 Values
- Values Show Display

**Notes:**
And now to the demonstration of xSonify.

I prepared two samples.

A very simple one first with a couple of numerical data to show the functions of

## Demonstration Enhanced Of xSonify

- Demonstration Enhanced Of xSonify

- 4 Cluster Satellites Over North Pole

**Notes:**
In order to show the second demonstration with data from the “Cluster” spacecraft I would like to ask Mona to give a brief information.

## Cluster 1 |B|

- Cluster 2 |B|

- Cluster 3 |B|

- Cluster 4 |B|

- Cluster SP |B|

## Demonstration Of xSonify

- Demonstration Of xSonify

_**Demo I:**_
- Simple Numerical Data
- 40 Values
- Values From 0 To 10 Repeating Four Times
_**Demo II:**_
- Data From “Cluster” Spacecrafts
- 1500 Values
- Values Show Display

## Thank You For Your Attention 

**Thank You For Your Attention**** **

- http://spdf.gsfc.nasa.gov/research/sonification
- anton@mail630.gsfc.nasa.gov
- robert.m.candey@nasa.gov

**Notes:**
Thank you for your attention. 

I hope I was able to clarify the expression “Sonification” and maybe we could arouse the one or the others interests with our talk. If you have any suggestions, ideas or comments about our project,  Bobby and I would welcome to receive any thought via email .

I would also like to mention that the whole project can be found under this URL. You  will also find this presentation soon on this web page. 

Thank you!