# Presentation: 922709

## Modality Translation on the Grid

- Network-Enabled Services on Demand
- For People With and Without Disabilities

**Notes:**
Good morning,

My name is Gottfried Zimmermann.  The Trace R&D Center which I work for, is part of the PACI partnerships in the context of the Education, Outreach and Training program (EOT).  Our contribution is to assist the PACI partners in applying Universal Design practices. 
The “Modality Translation on the Grid” concept is a variety of remote services available anywhere and anytime.  These services address the needs of people with and without disabilities.  The services are becoming possible as a result of recent technological advancements in the area of high-bandwidth networking and wireless communications.

## Agenda

- Introduction to Universal Design
- A Rainbow of Services on Demand
- Considerations for the Access Grid

**Notes:**
This presentation has 3 parts:
At first I will give a short introduction into Universal Design and how to address the needs of people with limited abilities.
In the second part - the main part - I will unfold the various aspects of modality translation services.
At the end it would be nice to contemplate about what the Access Grid could do in terms of offering value-added services for their users.

## Functional Limitations as a Function of Age

**Notes:**
This chart shows a statistic from the Bureau of the Census, from 1995.  Each pie represents the statistical amount of people with any or severe functional limitations, as a proportion to the overall population in a distinct age range.

Do you want to become 65 years old? Or even 80?

It is obvious: Statistically spoken it is just a matter of time until you will be concerned of being functionally limited in one or the other form.

## What to Do?

- Make STANDARD services and devices directly usable by people with a wide range of abilities
- Best way to create access to most technologies
- Create standard extension hooks for services and devices
- Alternative interface
- Enhanced capability
- Use new technologies for adaptive solutions where existing solutions don’t cope

**Notes:**
What can we do to prevent that the products and services we design will become inaccessible for people as they age or are functionally limited for other reasons? This is what Universal Design means:

Where possible, we try to make standard services (and devices) directly usable by a wide range of different users. Usually we achieve this by designing multi-modal user interfaces so that there are different ways of doing the same task, but by the use of different user capabilities.  This is the best way of enabling access. This is particularly true for aging people who become gradually functionally limited and don’t want to switch over to new products and services as they age.
If this approach doesn’t work, we can still make products and services accessible by providing extension hooks to accommodate alternative interfaces and enhanced capabilities. In a second presentation I will say something about the alternative interface approach.  An example for an enhanced capability would be a hook in a word processor in order to harness any spell-checking service that could be provided locally or remotely. By this we enhance the capability of the word processing program.  This hook could be used by a variety of service programs, including dictionaries and thesauri. Another example would be a hook in the IP telephony infrastructure so that if I click a button on my computer screen and a window pops up text-captioning everything what the other person says to me over the IP-phone.  More to that later.
The third approach is kind of an intelligent way to combine the first two approaches.  This means that the service or device itself adapt to the user’s needs.  For instance, the text captioning window could pop up automatically the next time I make a phone call via IP.  Or a speech recognition service could adapt to the strange language of a non-native speaker (like me) or to a user with a speech deficiency.  However, some initial training would be necessary.

## Network-Enabled Services on Demand

**Notes:**
As the user’s need vary, we need a variety of services to meet the user’s demands.  The “Modality Translation” concept includes a whole bunch of services, which we have laid out as rays on a rainbow.  Each of these services provides a translation of a specific presentation mode to another.  Wherever information is presented in a mode that a certain person cannot access or understand, we need a translation as an intermediary service between the information provider and the user.  For example, if I am deaf I cannot hear you talking; thus I need a translation from speech to text or from speech to sign language.  Or if I would talk in German, you would probably need an interpreting service that translates my speech into the English language.
Before talking about these services in detail, let me first explain an underlying principle of this “rainbow of services”.  As you can see, the rainbow consists of three layers.  Basically every service can be provided in three different manners (or implementations):
as a local automatic service (e.g. on your computer), 
as a network-enhanced service (i.e. another automatic implementation but with more computational power and larger resources than any local implementation), or
as a human-assisted service (i.e. somebody is providing the service itself or verifies the result of an automatic service).
The two last layers rely on a globally available network that enables the remote implementation of the services.  This highly increases the efficiency of these services.
The “Try Harder” feature basically means a “multiple stage” concept.  
If a simple and inexpensive service implementation is not sufficient, you can switch over to a more sophisticated, but also more expensive one. The most expensive is the human-assisted service stage.
“Try Harder” offers a convenient migration path for new and emerging technologies which are not yet mature enough to meet the requirements for a full-fledged service implementation.  As these technologies mature human assistance can be used to verify the automated result, and later be not needed any more.  At the end a light-weight service implementation can be placed on the user’s side (i.e. local), and the user will only go “try harder” if she needs special capabilities or features that aren’t provided by the local implementation.
These services must be accessible anytime and anywhere. This means that we are interested in a variety of Service Access Devices, and we are particularly interested in the mobile devices.  Examples include desktop or laptop computers, wearable computers, handheld devices, cell phones; these are outfitted with earbuds, clip-on monitors for eye-glasses, or buttonhole microphones, etc.
All these devices are connected by a globally available high-bandwidth network providing access to services and computational resources.

## Who are the Beneficiaries?

**Who are the Beneficiaries?**

- People with permanent functional limitations such as hearing, visual and cognitive impairments;
- People with temporary functional limitations like a car driver;
- People using mobile devices with restricted input and output capabilities;
- People needing special help or world-class expertise.

**Notes:**
Who are the beneficiaries of these services? 
First they are people with permanent functional limitations (hearing, visual, or cognitive) - which is just a matter of time according to statistics.
Also people with temporary functional limitations are addressed by these services.  Examples for this condition are a car driver (who cannot use his eyes for reading because they are busy), or a worker in a noisy environment (who cannot hear anything other than the noise), or a person in a library (where nobody should talk or listen to any audio), or a surgeon in need of medical expertise knowledge during a surgery.
A third group of beneficiaries are people using mobile devices with restricted input and output capabilities, e.g. handheld computers and cell phones.  In fact, I have a hard time writing when creating a meeting item on my palm pilot.  However, an accurate speech recognition service could help...
The last group of beneficiaries are people needing special help or world-class expertise.  This refers particularly to the International Language Service and the Assistance Mentoring Service.

## Text to Speech

**Text to Speech**

**Applications**
- People with visual impairments
- Eyes-busy tasks
- Phone call - TTY to regular phone
- People with low reading skills
**Technology**
- Speech synthesis (local or network)
- Synchronized voice data (local or network)

**Notes:**
This and the following slides reflect the different services of the rainbow in more detail.

The probably most mature service is the text-to-speech service.  They are also known as “document readers”.  
They are useful not only for visually impaired persons, but also for people whose eyes are busy (e.g. a driving person, or a surgeon).  Text to speech translation also could be used in telephone relay services for connecting a TTY to a regular phone.  And finally people with low or no reading skills can use it.
Today we can provide automatic text-to-speech service implementations either locally or on the network.  However, regarding the Talking Book standard (which uses synchronized voice data) we still rely on human speakers for the recording of voice data.

## Speech to Text

**Applications**
- People with hearing impairments
- Ears-free interaction for everyone
- Phone call - regular phone to TTY
- People who cannot type or cannot use a keyboard
- Speech interfaces (voice portals)
**Technology**
- Use of stenographic keyboard (human-assisted)
- Voice recognition (local or network)
- Revoicing with speech recognition (human-assisted)

**Speech to Text**

**Notes:**
The speech to text service helps:
People with hearing impairments: In face-to-face conversations, meetings, or collaborative sessions with speaking persons. Or as a safety net for accessing multimedia web content, when no captions are given.
The speech to text service addresses also ears-free interaction for everyone.  For example, a worker in a noisy factory building.  Or a student with ear infection in a classroom.  In this case the small clip-on monitor providing the teacher’s words in text would be a great help.
Again, speech to text translation facilitates automatic relay services.
People who cannot type or cannot use a keyboard benefit from this service.   We can hereby think of many possible circumstances:People with limited manual dexterity, or with learning disabilities.The car driver accessing Web-based information.People with a small wireless device.Or a manager in a business meeting who needs accurate minutes of the meeting.
A further application are speech interfaces to the web, also known as “voice portals”. 

This service still relies on human assistance if you want it in a reliable and accurate manner.  An interesting technology is certainly the re-voicing. Here is how it works: the audio signal of a meeting is sent to a remote dedicated speaker, who revoices everything she hears into a high-quality speech recognition system.  The text is then shipped back to the speaker’s location.  By this you will get much higher accuracy than with a direct voice recognition approach because the revoicer has extensively trained the system before and speaks very distinct.  This kind of service is already provided by a company in Madison, Wisconsin.

## Speech to Sign

**Applications**
- People with hearing impairments (understanding sign language)
- Phone call - regular phone to phone with video output
**Technology**
- Sign language interpretation (human-assisted)
- Automatic translation & signing avatar (network or local)

**Speech to Sign**

**Notes:**
The speech to sign service is usable by people who understand sign language.  This service could be used for face-to-face conversations, meetings, or collaborative sessions with speaking persons.  Or it could be used in an video relay service (as an offspring of today’s telephone relay service).  This would need a video screen attached to one of the participant’s phones.

Usually deaf people (who are congenitally deaf) prefer signing rather than English text because signing is their native language, and it can express more than words (e.g. it can express emphasis, or give a certain timing of words).

Technology
We still rely on sign language interpretation as a human-assisted service implementation provided by sign language interpreters. However, the Grid could provide the infrastructure for a service on demand implementation that would be much more efficient than the local presence of sign language interpreters.
Research is done in the area of signing avatars.  As a network advanced service this could be in the range of the next 2 or 3 years.

## Sign to Speech

**Applications**
- People with hearing impairments
- Phone call - phone with camera to regular phone
**Technology**
- Sign language interpretation (human-assisted)
- Sign recognition & language translation (network or local)

**Sign to Speech**

**Notes:**
The sign to speech service addresses people who sign.
This service could be used in face-to-face conversations, meetings, or collaborative sessions with hearing persons.  
Another usage would be an automated video relay service as an offspring of the telephone relay service.  This would need a camera connected to the signing participant’s phone.

Technology
Sign language interpretation today is in the realm of human-assisted services.  Today this is almost only done in a local fashion meaning that the sign interpreter has to waste her time with traveling a lot.  Again the Grid with its high-bandwidth capability would be a perfect infrastructure for a real-time sign-to-speech translation in a remote fashion.
It will still take a long time until signing recognition can be implemented as a reliable automated service.  There is research in this area.  This technology needs much computational power for image recognition, and machine translation activities.

## International Language

**International Language**

**Applications**
- People with different languages
- Document, e-mail or web page translation
- Phone call, face-to-face, or collaborative session
**Technology**
- Language interpretation (human-assisted)
- Machine translation (network or local)

**Notes:**
Applications
The international language service addresses people with different languages (for example me - and you if I would talk to you in German).  As a text-to-text translation it could be used for any kind of documents.
As a real-time speech-to-speech translation service it would be useful for intermediary translation of phone calls, face-to-face conversations or collaborative sessions with participants of different languages and cultures.

Technology
Today language interpretation is done by humans.
Machine translation is a topic of research, and there is remarkable progress in this area.  The Babel fish on the AltaVista web site is an example of the state-of-the-art in this area.

## Assistance Mentoring

**Applications**
- Learners and knowledge workers
- CACTUS Help Desk
- Job Counselor
**Technology**
- Help system - F1 key (local)
- Tech Support Knowledge Base (network)
- Help desk (human-assisted)

**Assistance Mentoring**

**Notes:**
Applications
The Assistance Mentoring service addresses people who are in a learning process or who need knowledge expertise from time to time.
The CACTUS Help Desk or the Job Counselor are examples for service applications of this kind.  

Technology
This service already exists in local, network-based and human-assisted fashions - as local help system, tech support knowledge base or human-assisted help desk.  Here the “Try harder” feature is very apparent and everybody in this room has at some point gone through all 3 stages.

## Language Level

**Applications**
- People with cognitive impairments
- Reference aide (e.g. GuruNet)
- Researcher writing for a general audience
- Medical patient consultation
**Technology**
- Language level transformation (human-assisted, network, or local)
- Thesaurus / “Term checker” (local or network)

**Language Level**

**Notes:**
Language (or cognitive) level translation is the art of transforming words and sentences to express the same meaning in a lower language (abstraction) level.

Applications
This service addresses people with cognitive impairments as well as other people in certain situations.
For example I sometimes use GuruNet if I need an explanation of technical terms or abbreviations.  GuruNet is what I would call an “Internet-based thesaurus”.  It is a sample application for a language level transformation service.
Another application would be a checking service that would work like a spell-checker and alert me when I use technical terms or abbreviations in my presentation manuscript that are not common to my expected audience.
Another application would be medical patient consultation, for example in an emergency room.

Technology
Implementations of services that transform documents or spoken language are human-assisted today.  However, it is the network that facilitates a highly efficient service implementation “on demand”, by remotely linking to these human resources.
A degenerated form of this service is a thesaurus implementation providing canned explanations for terms and abbreviations.  The “term checker”, another variant of the spell checker, is also feasible today as a local and network-based service implementation.

## Image / Video Description

**Applications**
- People with visual impairments
- Indexing legacy archives for search
**Technology**
- Verbal description (human-assisted)
- Image analysis (network)

**Image / Video Description**

**Notes:**
Applications
The image/video description service addresses people with visual impairments.  As an example, a blind participant in a meeting would get a verbal description of a visual diagram that is currently presented.  This assumes that the service has access to the document in an electronic form (which is a great advantage of a virtual meeting in a collaborative environment). Or, that the participant uses a small camera that is directed to the diagram and transmits the image to the service. 
Another application of the image/video description service is to use a verbal description for indexing an image database or video streams.

Technology
Today this service can be provided solely by humans. 
At some very late point we might have this service in an automated fashion by means of image analysis.  This would involve high computational demands.

## Speech to Text ServiceA Short-Term Goal for the Access Grid?

**Speech to Text Service****A Short-Term Goal for the Access Grid?**

- Text captioning on demand for collaborative sessions with hard-of-hearing participants
- Accurate minutes of a meeting
- Demo by Trace at SC2000 with standard software
- Ultratec providing speech-to-text translation on a 24/7 basis

**Notes:**
How could these services be valuable for the Access Grid?

The Access Grid enables people to collaborate and meet together in a virtual fashion.  I see a great potential for the Access Grid in integrating these translation services as an additional value to their users.  What could be a first step in this direction?

The speech to text service is probably one of the candidates to be implemented in today’s Access Grid infrastructure.  This would facilitate deaf or hard-of-hearing participants to follow a collaborative session and ask questions himself if he could speak.  However he would still need to type questions if he cannot speak.
In addition, you would get accurate minutes of a meeting.
At SC2000 Trace demonstrated this service for an Access Grid collaborative session with a standard software. It was ICQ in this case. However we would like to have a closer integration into the Access Grid infrastructure, e.g. direct hooks to tap the voice channel.
The service itself is already provided by a company in Madison, Ultratec. So the goal would be just to connect to this service via Internet.

## Sign Interpretation ServiceA Long-Term Goal for the Access Grid?

- Speech to sign and sign to speech translation on demand for collaborative sessions with sign language users
- Demo by Trace at SC99 with human sign interpreters
- Human assistance still needed
- Signing avatar technology emerges

**Notes:**
A more long-term goal could be an implementation of a sign interpretation service into the Access Grid. This would imply translation in both directions, i.e. speech to sign and sign to speech translation.
A remote sign interpretation service was already demonstrated by Trace at the SC99 conference in Portland, Oregon.  There, human sign language interpreters at Madison, Wisconsin, were connected over the Internet to the conference location.
But current research in the area of machine translation could help to bridge the gap between spoken language and sign language.  Signing avatar and sign recognition technology could finally deliver such automated sign interpretation service in both directions.

## Open Issues

- Communication infrastructure: H.323 or Mbone?
- Platform-independent implementation: Applet or Application?
- Close integration into the Access Grid (interface)
- _[Thank you](file:///C:/Program%20Files/Qualcomm/Eudora%20Mail/Imap/Dominant/INBOX/012901AGRetreat2001/SigningAvatar%20(Vcom3D)/AvatarStunt/ThankYou.HTM)_

**Notes:**
I’d like to finish my presentation, not without mentioning that there are some open issues. 
The first open question is: What communication infrastructure (or protocols) should we rely on? H.323 or Mbone?
Another issue is how to achieve a platform-independent implementation with the lowest effort for client installation and configuration. We would like to use an applet architecture, but are afraid there could be some obstacles.
The final questions is how to closely integrate these services into the Grid, or into the Access Grid in particular.  This means where is the place in an globally available interface to put in that button that would connect you to all these wonderful services.

It would be nice to talk to you about these issues and hear your opinion and advise.

Let me close this presentation with this nice guy talking in his own language.
Thank you for your attention.