# Information Accessibility Lab

## Basic Proposal Information

  ---------------------- ------------------------------------------------
  Title:                 Information Accessibility Lab

  Brief Description:     An assistive technologies lab to evaluate and
                         develop immersive, assistive technologies
                         applied to NASA scientific content

  Partners:              RRTC University of Mississippi

  Center:                Johnson Space Center

  Endorsee:              Dr. Robert Shelton

  Primary Contact Name:  Dr. Robert Shelton

  Primary Contact Phone: 281-483-2901

  Primary Contact Email: <robert.o.shelton1@jsc.nasa.gov>

  Secondary Contact      Stephanie Smith
  Name:                  

  Secondary Contact      512-291-1454
  Phone:                 

  Secondary Contact      <stephanie.l.smith1@jsc.nasa.gov>
  Email:                 
  ---------------------- ------------------------------------------------

## Budget and Milestones

Milestones:

1\. Research and understand basic principles and concepts involved in
the proposed math description engine and it's application to 3-D
immersive technology

2\. Formulate the technology concept and application infrastructure by
identifying requirements and rules for math description engine

3\. Analyze and develop a proof of concept for the proposed math
description engine including the determination of an initial phrase
library.

4\. Identify interactive science demonstration as a prototype for
acoustical interface (sonification technology)

5\. Develop, integrate and validate the components of the sonified
prototype technology in a closed laboratory environment

## Project Purpose

NASA offers a wealth of interactive multimedia content ranging from
terrestrial observations to planetary surfaces and views of the universe
in spectra ranging from radio to x-ray imagery. The proliferation of
inexpensive, multi-sensory display and interface technology makes it
possible for us to bring the excitement and realism of scientific
exploration into homes and classrooms on a scale never imagined. We
propose to add a significant and necessary component to this exciting
agenda in the form of assistive instructional technology tools.

As stunning and effective as new multimedia technologies are for people
with normal senses, they can be a frustrating barrier for those with
sensory impairments. Assistive technology has made important progress in
giving disabled computer users access to standard computer
applications.It has empowered a historically under-employed segment of
the population with access to technology and provided an opportunity to
participate in the information economy. Ironically, advances that make
computers more capable and friendly for people with normal senses often
pose significant challenges as well as opportunities for those who
design and use assistive technology. The new multi-sensory immersive
technologies sponsored by The NASA Learning Technologies Project
initiative will yield many products for which there is no existing means
of accessibility for people with sensory impairments. We propose to
apply and develop emerging multi-sensory technology, along with rule and
computation based AI, to provide new assistive educational tools and
alternate pathways to hands-on science activities for vision impaired
students and their teachers.

It is well recognized that one of the greatest hurdles in teaching
science,engineering and particularly, mathematics is the difficulty that
many students have with spatial visualization. Many projects ranging
from classroom use of graphing calculators to full-blown virtual reality
have been undertaken to address this problem for non-disabled students.
It is humbling to realize that even the rudimentary capabilities of a
simple graphing calculator are beyond the reach of a blind student in a
mathematics class. The traditional way of conveying graphical content to
a blind student is to use raised line drawings either in Braille
textbooks,manual drawing boards, or computer-driven Braille displays.
Recorded textbooks often include careful descriptions of Cartesian and
polar graphs, but all of these methods suffer from significant problems
including lack of interactivity, portability and expense.

A simple approach that would provide significant value to students,
blind and otherwise would be to provide a descriptive math assistant
that couples a rule base with the code for a graphing calculator which
synthesizes interactive, text-based descriptions of graphs of functions
that are entered by the student using a standard text interface. For
example: A student enters this equation in the calculator: y = x\^2 -
4\*x + 5The calculator system parses the input and recognizes the curve
by an algebraic reduction process and comparison of a canonical
representation with a library of stored examples. The calculator
responds:"The curve is called a parabola. It rises progressively more
steeply to both left and right. It's lowest point is at the point x = 2,
y = 1. Press "more" for details."

This text-based descriptive math assistant described above is the first
component of the Information Access Lab. As a complementary approach, we
will employ "sonification", an analysis tool used by scientists as a
data mining technique, to create audible interfaces that will be capable
of going beyond simple planar graphs to render complex,
multi-dimensional, time series data to a blind user. We propose to build
NASA oriented, interactive models of physical systems with text-based
descriptive output, augmented by 3-dimensional sound to represent the
graphical content. The result will be a way to couple interactive
simulations based on the NASA mission to a dynamic interface capable of
rendering multi-dimensional dynamic data in a form accessible to a blind
student. As an aside, simulation controls will be available through the
keyboard so that it will be possible for the user to handle the activity
in real time, much like a video game. We would make no attempt to
reproduce the sound of the actual system, but rather would exploit
pitch, spatial direction, intensity and wave form (timbre) to provide
the most descriptive possible representation of the underlying physical
process.

As a simple example, over 10 years ago, one of the team, a blind
mathematician, used output from a PC speaker (no sound card, just a
monotone monaural PC speaker) to analyze output from a structural
vibration controller. Coordinates of sample points in the system were
coded as tones with pitches varying in time to represent the motion of
the structure. Using this Stone Age approach in comparison to what is
now available, it was possible for him to "tune" the controller in the
literal sense. We propose to exploit sound synthesis capabilities of
ordinary commodity class computers to open the door to simulations for
blind students. It is conceived that haptic tools could be incorporated
into this immersion scenario to provide 3-D acoustic digital objects
that can be "felt" by blind computer users. This effort builds upon work
conducted by Sound2Vison, BAT, Phillips and Certec however our plan has
a different agenda than many of these efforts because we intend to
specifically target comprehension of time series data and control of
simulations rather than generic rendition of visible scenes.

## Innovation

The innovations proposed in this project are of a revolutionary nature.
The first step in delivering this product is the creation of a 2-D graph
description engine which will be based on an algebraic parser and AI
simplification techniques of the type used in products like Mathmatica
and earlier, Macsyma. The goal-driven strategy is to match the
mathematical formula input by the user to a library of standard
equations. Though solving this problem in total generality is not
possible, knowing the kinds of equations that are commonly seen in
secondary school mathematics will make it possible to cover many
educationally relevant cases with pattern matching techniques. Beyond
the simple known cases, the system will also describe graphs of
expressions that do not fit a pattern that the application understands.
If no match is found, the software will evaluate the user's input
expression at several hundred example points to create a tabular
representation of the curve. In addition to plotting and connecting the
points on the screen, the description engine will use computational
AI---fuzzy reasoning---to generate linguistic descriptions for various
parts of the curve. Examples of typical descriptions would be phrases of
the type "rising steeply with increasing slope for x greater than 2,
reaching an inflection point at x = 5, and rising progressively more
slowly to a peak at x = 7..." The information to drive this kind of
natural language output is derived from the raw data generated by
evaluating the input expression.

The next innovative step will be a companion activity to generate
sonifications of physical systems that cannot be easily conveyed by a
2-dimensional plot. This will connect acoustical displays to simulations
thereby making simulations accessible to blind and low vision students.
This will involve considerably deeper research into how to best use the
sound capabilities of commodity PCs to best effect. The most basic PC
sound cards provide bit-by-bit control over two independent channels,
making any sound experience available for simulation. Previous
experiments with real data using simple equipment show that effective
rendition of multidimensional, time-varying output and control of a
dynamical system is possible for a blind user with only a fraction of
this capability. We will combine simple pitch direction and intensity
sonification with interesting and meaningful simulation activities to
demonstrate a practical, low-cost means for blind students to work with
interactive systems as they might do subsequently in a scientific or
technical occupation. Our ultimate goal is to see how far we can push
this technology to provide an alternate and meaningful view of natural
systems.

As interesting and challenging as the purely technical side of this
effort may be, the important intellectual innovation of this project is
the concept that mathematics, sciences and engineering are not off
limits to students with sensory disabilities just because there is a
communication issue. Our history contains important examples of human
beings with sensory disabilities who somehow made pioneering
contributions to hard science with no assistive technology whatsoever.
As inspiring as these examples are, one can only wonder how many others
would have made even greater contributions if they had possessed even
the simplest of tools. We view mathematics and science as potential
equalizers for persons with disabilities because ones contributions in
the sciences are determined by intellectual not physical activities. The
core technologies that we will bring to bear on this project are
intentionally inexpensive, readily available and well understood. It is
their revolutionary application as assistive technologies that is a
fundamental innovation in this proposal.

## Diffusion Potential

This effort will result in tools and examples of basic and fundamental
importance to blind students and their teachers.We will work through our
partner, the Research Rehabilitation and Training Center (RRTC) on
Blindness and Low Vision as we have done on previous projects to fully
evaluate and disseminate our products and the results of our research.
We expect to file a technology disclosure on the graphical description
engine by the end of the first year and will work through NASA's
Technology Transfer and Commercialization organization to make this tool
available to anyone who can benefit. We expect to publish the results of
connecting acoustical displays to simulations in journals and conference
proceedings. We anticipate that the techniques we develop, evaluate and
prove effective will be of great interest to groups throughout the
federal government, academia and private industry who are grappling with
the problem of providing alternate access to dynamic information. In
particular, the role of our team in working groups for issues related to
Section 508 of the Americans with Disabilities Act, both at the center
and agency levels, position us ideally to offer the technology we
develop to organizations and projects that will have a critical need for
it.

RRTC researchers will disseminate findings in a consumer-friendly format
through the Center's newsletter and website. The RRTC website receives
approximately 5000 hits monthly. Journal articles will be submitted to
major consumer publications, such as the Braille Forum and the Braille
Monitor. Research results will also be disseminated in relevant
professional journals, such as the Journal of Visual Impairment and
Blindness, the Journal of Rehabilitation, Teaching Exceptional Children,
and RE:View. All publications will be shared with the National
Clearinghouse of Rehabilitation Training Materials, the National
Rehabilitation Information Center, and the National Center for the
Dissemination of Disability Research. Project staff will also present
project findings at RRTC-sponsored national training conferences,
national, regional, and state consumer meetings, and other
professional/consumer conferences. Materials will be produced in
Braille, large print, on audio tape and on computer disk to ensure full
access for consumers who are blind or severely visually impaired.

The graphical description engine is also of significant value to
non-disabled students and the potential for dissemination to this group
of students is unlimited. It is well documented that many students and
particularly females have difficulty in the areas of spatial
visualization,spatial mechanical skills and spatial sense. This tool
could have a positive effect on graph comprehension and spatial
representation and reasoning skills.

## Project Feasibility

The technology components underlying the 2-D graphical description
engine are well known to the team. There are examples of calculator
parser software available on the web, as well as in our team library,
and the team is knowledgeable in the kind of pattern matching required
to accurately classify equations entered by a user from a list of
possible candidates. We plan to go beyond a strict rule based parser to
account for common typing mistakes, recognizing that many users will not
be accustomed to doing their math homework on a keyboard. The rule-based
AI

needed to reduce expressions and match them to known forms is a
well-known feature of existing software packages. We will investigate
tradeoffs

between using existing routines in our inventory, open source, or
commercial software development kits. On a complexity scale for AI
applications, the pattern matching and manipulation required would fall
somewhere in the middle.

The technology behind the computationally based AI - fuzzy logic
description engine - which will cover cases that the rules can't handle
is somewhat less well known. Fortunately, NASA (and probably lots of
others) developed similar descriptive rules for other purposes. At JSC,
this technique was pioneered by Dr. Shelton as part of a Mission Control
Center application to detect "events" on shuttle electrical buses by
monitoring time-sampled values of electric current from the telemetry
stream. The remainder of the description engine would consist of modules
to synthesize correct linguistic phrases to describe sections of the
curve. This is an area in which we intend to exercise extreme care to be
sure that the language not only fits together syntactically, but also
conveys the maximum descriptive content for the situation at hand. As a
final touch, we would like this output to be rendered as natural speech
rather than through a text-to-speech synthesizer only. Since the phrases
will be known in advance, this pleasant esthetic touch can be added with
little additional effort.

The second phase of the project will involve significantly deeper
research into the way human beings process acoustical patterns. Although
there is extensive literature on sonification, from previous experience,
we anticipate that this effort will produce significant new research. We
are fortunate to have available to us persons knowledgeable in
mathematics, teaching mathematics and music theory in addition to
software design and system integration. Further, our partner, the RRTC,
will provide feedback, validation and correction to be sure that our
results are appropriate to the widest possible audience. Our approach is
to begin with systems that illustrate science concepts e.g. motion
through space, dynamics of masses under Newtonian mechanics, but which
are subject to easy sonification techniques. As we refine, validate and
improve our techniques, we will build toward greater complexity. Given
that sonification is a technique that the sighted world uses to handle
data that is too complicated for visible display, it is difficult to
guess how far this could go. From our own experience, we are certain
that this project will result in a beneficial tool to convey science
content to blind students, but it could conceivably go far beyond,
leading to a class of entertainment, gaming and practical consumer
applications. When a fundamentally new method of perception becomes
available, the end results are difficult to predict.

## Partnerships

This project will be a partnership between the JSC LTP team, teachers,
technology specialists and NASA scientists. We intend to work with the
Research, Rehabilitation and Training Center (RRTC) on Blindness and Low
Vision situated at Mississippi State University. The RRTC is the
nation's leader in conducting applied research and training activities
focusing on the employment and educational needs of persons who are
blind or visually impaired. For more than 20 years, the RRTC has worked
collaboratively with major blindness consumer and professional groups
and will draw upon these long-standing relationships in evaluating and
disseminating technology developed in this project.

Our plan is to work with RRTC as a gateway to a broader community in
order to shape our products to best serve societal needs. The RRTC is
involved with:

-   National Federation of the Blind (Computer Science Division, Science
    and Engineering Division, and Association of Blind Students) -
    largest membership organization of blind persons

-   American Council of the Blind (Alliance of Blind Students, Data
    Processors International) - leading membership organization of blind
    individuals

-   National Association of Parents of the Visually Impaired

-   national parent group concentrating on educational needs of students
    with visual impairments

-   Association for the Education and Rehabilitation of the Blind and
    Visually Impaired (AER) - major professional organization for
    rehabilitation and education practitioners; through AER's listserv,
    AERNET, hundreds of teachers of blind students throughout the United
    States can be recruited to evaluate proposed technology

-   Association of Higher Education and the Disabled - AHEAD's Special
    Interest Group on Blindness and Visual Impairments promotes
    accessibility to higher education by blind and visually impaired
    persons

-   Disabilities, Opportunities, Internetworking and Technology
    (DO-IT) - funded by the National Science Foundation through the
    College of Engineering at the University of Washington; purpose is
    to recruit and retain students with disabilities into science,
    engineering, and mathematics academic programs and careers

-   Association of Computing Machinery - special interest group on
    computers and persons with disabilities

-   California Transcribers and Educators of the Visually Handicapped -
    promotes acceptable practices and technology to enhance the total
    educational program for students who are visually impaired

-   Texas School for the Blind and Visually Impaired - leader in
    developing accessible math materials

In addition to our collaboration with RRTC, we intend to continue our
policy of making aggressive use of open source and other free software
as opportunities become available. There also may be possibilities for
corporate, academic or government collaboration that arise as the
project develops, and we will make every effort to identify and exploit
such opportunities.

## Evaluation

The Information Access Lab will use emerging technologies to provide
interactive, student-centered activities and technology-based teaching
tools focused on NASA's mission. To achieve these outcomes, the JSC LTP
team will develop the Information Access Lab with design guidance and
evaluation from the Research, Rehabilitation and Training Center on
Blindness and Low Vision located at Mississippi State University.

The Information Access Lab outcomes will be measured using multi-level
assessments. Program metrics will be derived from the LTP Management
Plan which includes datapoints such as accomplishments, awards,
recognition, Web statistics, public outreach, conferences, publications
and schools impacted. These metrics will be tracked with the NASA-wide,
data collection and reporting system EDCATS. Monthly EDCATS reports will
track basic elements of the program, help improve program quality and
report on program performance.

The effectiveness of the project will be evaluated externally in field
tests conducted through the RRTC and internally by tracking
Internet-based dissemination, use and feedback on the product
applications. Additionally, JSC assessment activities will include user
surveys, educator evaluations, evaluation of Web logs, and review of
online feedback forms.