# Presentation: 816515

## Sonification Prototype for Space PhysicsRobert M. Candey(Robert.M.Candey@nasa.gov)NASA Goddard Space Flight Center, Code 612.4, Greenbelt MD 20771Anton M. SchertenleibUniversity Of Applied Sciences Hof/GermanyWanda L. Diaz MercedLaboratory of Astronomy Shirohisa Ikeda Project, University of Puerto Rico, Rio Piedras, Puerto Rico

- http://spdf.gsfc.nasa.gov/research/sonification/

- 2005 Fall AGU ED43B-0850

- 2005 December 8

## Abstract

- 	As an alternative and adjunct to visual displays, auditory exploration of data via sonification (data controlled sound) and audification (audible playback of data samples) is promising for complex or rapidly/temporally changing visualizations, for data exploration of large datasets (particularly multi-dimensional datasets), and for exploring datasets in frequency rather than spatial dimensions (see also International Conferences on Auditory Display <http://icad.org/>).
- 	Besides improving data exploration and analysis for most researchers, the use of sound is especially valuable as an assistive technology for visually-impaired people and can make science and math more exciting for high school and college students.
- 	Only recently have the hardware and software come together to make a cross-platform open-source sonification tool feasible.
- 	We have developed a prototype sonification data analysis tool using the JavaSound API and NASA GSFC’s ViSBARD software  <http://nssdcftp.gsfc.nasa.gov/selected_software/visbard/>.
- 	Wanda Diaz Merced, a blind astrophysicist from Puerto Rico, is instrumental in advising on and testing the tool.

## Familiar Sonifications

- The number of rings of a church bell conveys information about the current time
- Ticks of a Geiger-Mueller counter indicates radiation
- Sonar pings
- Audification of Jupiter radio-astronomy data presents whistlers, hiss and chorus

**Notes:**
???
First click on image!

Audification is basically the audible playback of data sample. 

The Cassini spacecraft crossed the bow shock of Saturn at 9.45 Universal Time on June 27, 2004. The bow shock is a discontinuity that forms in the solar wind when the supersonic solar wind encounters the magnetic field of a planet, very similar to the shock wave that forms upstream of an aircraft moving at a supersonic speed. 

The color frequency-time spectrogram in the above illustration shows the electric field intensities detected by the Cassini’s Radio and Plasma Wave Science (RPWS) instrument. The bow shock crossing is indicated by the abrupt burst of electric field noise at the time marked by the arrow. 

The electric field noise is caused by electrical currents that flow in the shock. Time on this recording has been compressed such that these 10 seconds corresponds to 28 minutes.

## Sonification is the use of non-speech audio to convey information

- Auditory display can be grouped into several types:
- Audification: playing data directly as a binary sound file (after perhaps shifting frequency and time step into the audible range, as with radio-astronomy data)
- Iconic Sonification: mapping data to sounds associated with certain phenomena (use sound of rain for rain data, empty trash sound on computer desktop)
- Musical sonification: mapping data to musical parameters, such as pitch and loudness
- 	Research community for audio representation of data: International Community on Auditory Display <http://www.icad.org/>
- References:
- International Conference on Auditory Display (ICAD) members, “Sonification: A status report and research agenda,” white paper to National Science Foundation, (1998) <http://icad.org/websiteV2.0/References/nsf.html>
- Gregory Kramer, Auditory Display: Sonification, Audification, and Auditory Interfaces, Addison-Wesley, Reading MA, 1994.
- E. S. Yeung, “Pattern recognition by audio representation of multivariate analytical data,” *Anal. Chem.,* 52, 7, (1980), pp. 1120-1123.
- R. S. Wolff, “Sounding out images,” *Computers in Physics,* 6, 3, (1992), pp. 287-289.

## Musical Sonification Attributes

- Reference: J. B. Krygier <http://go.owu.edu/~jbkrygie/krygier_html/krysound.html>

- Sonification Through Physical-based Models
- While musical sonification requires significant training and may elicit emotional or musical connotations rather than scientific data patterns, representing data through varying parameters to physical models may be more direct and clearly understood.
- Models include drums (varying drum size, impact force and type, etc.) and footsteps (varying person’s size, speed, floor type, footwear, etc.)

**Notes:**
???
Now I would like to go on with the technical background of Sonification.

As you can see, sound has different attributes. However it is necessary   provides us with different 

## Sonification Benefits

- Provide assistive technology for visually-impaired and non-visually-oriented people
- Make science and math more exciting for high school and college students
- Analyze complex or rapidly/temporally changing visualizations, especially in complex multi-dimension and multi-dataset research
- Explore large datasets (particularly multi-dimensional datasets)
- Explore datasets in frequency rather than spatial dimensions
- Identify new phenomena that current display techniques miss
- Find otherwise hidden correlations and patterns masked in visual displays
- Monitor data while looking at something else (background event-finding)
- Complement existing visual displays (ear is sensitive to different frequency bands and patterns than the eye)
- Improve visual perception when accompanied by audio cues
- Increase “bandwidth” of human-computer interface

**Notes:**
???
Our sonification tool -xSonify- was developed Java 1.5.

Technical background: why java just started with java sound, why it neglected

For xSonify the main reason for choosing Java Version 1.5.0 was mainly the enhanced  Java Sound API beside all the the other goodies which came with Java Tiger. 
- Ports are now available on all platforms
- MIDI device i/o is now available on all platforms
- Optimized direct audio access is implemented on all platforms.
- The new real-time Sequencer works with all MIDI devices and allows unlimited Transmitters
- The  sound.properties configuration file allows choice of default devices
- MidiDevices can query connected Receivers and Transmitters
- The Sequencer interface is extended with loop methods for seamless looping of specific portions of a MIDI sequence
- Java Sound no longer prevents the VM from exiting 

## Accessibility and Sonification of Data

- Sonification provides greatly-improved accessibility to space science data for visually-impaired scientists and students
- Common non-visual methods for examining data are often inferior and more costly than visualization methods
- Reading aloud and remembering data values is a difficult way to analyze data
- Raised plots require specialized and expensive hardware and are not readily tied to web site services
- In addition to visually-impaired users, many others are aurally-oriented rather than visually-oriented and sonification provides them a more meaningful mode for doing science
- Also appeals to the educational community, making science more exciting to students and the general public

## Space Science And Sonification 

- With sonification, a large fraction of the space physics data collection is opened to a completely new and now excluded audience (both professional and public).
- Some examples of sonification in space physics:
- Cassini’s Radio and Plasma Wave Science (RPWS) crossing the bow shock of Saturn 	<http://www-pw.physics.uiowa.edu/space-audio/cassini/bow-shock/>
- Playing data as sound for detecting micrometeoroids impacting Voyager 2 when traversing Saturn’s rings; these impacts were obscured in the plotted data but were clearly evident as hailstorm sounds [F. L. Scarf, *et al.,* “Voyager 2 Plasma Wave Observations at Saturn,” *Science*, 215, (29 January 1982), pp. 587-594.]
- Using an early Beatnik/JavaScript-based tool, we tested several sonification techniques on Hawkeye magnetic field and plasma measurements for magnetopause, bow shock and cusp crossings [Candey, R. M., R. L. Kessel, J. R. Plue, “Web-based sonification of space science data*”*,* SIGGRAPH98 Conf. Abstracts and Applications,* SK98-166, 1998].
- Our new Java-based tool, xSonify, presents 1-dimensional data from text files and from all of CDAWeb’s space physics holdings(via web services). We will expand it to handle multi-dimensional data. We expect that distinctive signatures will be especially apparent above the background noise, particularly for bow shock and magnetopause crossings. These boundary signatures appear as characteristic changes in the whole spectrum (including the background noise) while other emissions appear as tones at distinctive frequencies or time-frequency spectrums (such as electron plasma oscillations, magnetic noise bursts in the magnetosheath, and whistler mode emissions marking regions of currents).

**Notes:**
???
Data is the outcome of a set of measurements or observations. After scientists place instruments on a spacecraft that flies out to space, these instruments can take measurements of things in space, such as charged particles. These measurements are recorded and sent to Earth. 

This information is data that can help scientists understand more about the space environment, such as the solar wind or regions of Earth's outer most magnetic field. 

New user interface...

## xSonify

## xSonify Implementation

- Based on:
- Java 1.5
- Java Sound API / MIDI
- WebStart Technology
- Java Speech Support
- CDAWeb and other SPDF web services
- JavaSound in Java 1.5 currently uses Headspace AudioEngine, but it will be removed since the rights to teh source code are not available. This leaves only the not fully-functional JavaSound AudioEngine as the only available AudioDevice (disabled by default). Sun plans to encourage the development community to implement an AudioEngine using Java Bindings for OpenAL / Sound3D Toolkit (JOAL) <https://joal.dev.java.net/>.
- xSonify currently features 3 different sonification modes:
- Pitch
- Loudness
- Rhythm
- Controls:
- Play, Stop, Loop
- Speed
- Time point
- Preprocess input data:
- Limits
- Invert
- Logarithm
- Averaging

## Sonification Process

- Data Pipeline Approach:
- Data Transformation: to appropriate geophysical units
- Normalization: convert to abstract normalized view
- Sonification Transformation: to abstract sound parameter space
- Auditory Display Transformation: sonic rendering
- [from Sylvain Daude, Laurence Nigay, ICAD 2003, “Design Process For Auditory Interfaces”]
- Evaluation Criteria
- Usability testing, including participatory design by local scientists and focus groups
- Determine usability of the various sonification techniques and our implementations (with comparison to current visualization tools) in three ways:
- effectiveness (improvement in the amount and quality of users’ science knowledge derived from displaying the data using these new techniques and accuracy/errors in detecting geophysical events in the data)
- efficiency (learning time and time/effort to analyze the data)
- satisfaction (attitude and comfort rating surveys, users’ preferences)

**Notes:**
Ed Chi worked on a way how to display a visualization process and named it the “Data Pipeline”. Analogously to princip of Ed Chi, ...xxx..., 
Sylvain Daude and Laurence Nigay presented a sonification process which transforms and prepares the raw data to be finally sonified. The process  can be considered as some sort of “Data Pipeline”. 

The single points of this process are:
???
From Data To Data View: Data Transformation
In this step the raw data will be mapped value by value into a range of values between 0 < x < 1. This process can be considered as a kind of standardization of values which is necessary to make the data available independent from their unit and scale.

From Data To Abstract Sound Space: Sonification Transformation
During this step the data are prepared according to the chosen Sonification modus. Every value will be assigned to a certain position at a “time line”. This “time line” is represents the time line of the sonification sequence which will be played in xSonify in the MIDI player.

From Abstract Sound Space To Sonic: Auditory Display Transformation
This is the part where the signal is finally displayed on a physical device. In xSonify this is the part where the MIDI sequence will be transmitted to the MIDI player and played. 

## Current Plans

- Form a user’s group of interested users and experts to define requirements and evaluate the software
- Apply research in psychoacoustics, semiotics, ergonomics, commercial sound design, and music
- Add additional sound modes:
- audification (play data directly, scaled into audible range
- multiple oscillators (for frequency/energy band as plotted in spectrograms)
- physical parameter models (drum size, footsteps, etc.)
- Add stethoscope mode where the user mouses over the plot of multispectral data playing sounds correlated with some of the data parameters (as proposed by Wolff and others)
- Add variations in timbre, phase, and direction for representing additional information about main variables, such as variations in anisotropy, noisiness, uncertainty, and polarization
- Add SGT plot library Scientific Graphics Toolkit <http://www.epic.noaa.gov/java/sgt/index.html>
- Tie in TIPSOD display of 3D orbits, synced so spacecraft moves in coordination with sounds
- Improve GUI, add options
- Represent multi-dimensional datasets (for instance, represent plasma wave data with sound loudness representing intensity, pitch for frequency (or energy in particle data), and left-right amplitude and phase for polarization differences)

## Technical Issues

- How to make JAWS work in Java:
- Install Sun's Java Access Bridge 1.2
- <http://www.freedomscientific.com/fs_support/BulletinView.cfm?QC=685>
- or 2.0 beta 2 <http://java.sun.com/developer/earlyAccess/accessbridge/>
- Put all windows in initial Jframe
- <http://www.freedomscientific.com/fs_support/BulletinView.cfm?QC=567>
- Perhaps Alt-Tab out and back
- <http://www.freedomscientific.com/fs_support/BulletinView.cfm?QC=724>
- Make sure using Sun's Java
- <http://www.freedomscientific.com/fs_support/BulletinView.cfm?QC=749>
- Use the Java Accessibility API methods setAccessibleName() and setAccessibleDescription() to provide component names and context help

**Notes:**
???
The Java Sound API includes support for both digital audio and MIDI data. These two major modules of functionality are provided in separate packages.

In the left area of figure 3.2 you can see the two “sampled” packages which are handling digital audio. Java Sound API refers to it as sampled audio. Samples are successive snapshots of a analog signal. I will not explain the use of this package further since it is not used in this software application. In the right area there are the two “midi” packages for MIDI synthesis, sequencing and event transport. 
The two lower packages “spi” in figure 3.2 permit service providers to create custom software components that extend the capabilities of an implementation of the Java Sound API. “spi” stands for service provider interface.
The Java Sound API does not assume a specific audio hardware configuration; it is designed to allow different sorts of audio components to be installed on a system and accessed by the API. The Java Sound API supports common functionality such as input and output from a sound card (e.g. for recording and playback of sound files) as well as mixing of multiple streams of audio.