# Presentation: 630872

## PERVASIVE COMPUTINGAT NIST

**Martin Herman, Chief**
**INFORMATION ACCESS & USER INTERFACES DIVISION**
**INFORMATION TECHNOLOGY LABORATORY**
**NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY**

## VISION

-  Small inexpensive computers and sensors
-  in every device, appliance and piece of equipment
-  in buildings, offices, classrooms, homes, cars, factories
-  people carrying portable and wearable computers
-  networked to each other and the Internet
-  “A trillion devices connected to the Net!"
-  sensing and reacting intelligently to the environment
-  Information instantly accessible anywhere and anytime
-  An emerging new paradigm for IT
-  Enormous new global market
-  transforming the way U.S. industry does business
-  transforming the way people live and learn
-  requiring the development of a measurement and standards
-    infrastructure ensuring that U.S. manufacturers are not at a
-    disadvantage

## NIST’s Role

- Pervasive computing is in infancy
- Many required technologies are immature & have high risk
- NIST’s unique role
- develop metrics, tests and standards to push forward underlying technologies and help pervasive computing mature and grow
- develop test data and test scenarios/tasks
- provide impartial, large-scale evaluations across many different systems and companies
- develop interoperability specifications and test methods
- perform & provide tools for conformance testing

## Current Industry Efforts

- Information appliances
- Web TVs, screen phones, net-enabled smart handheld computers, internet gaming consoles, internet automotive systems, electronic books
- Integration into Internet
- wireline high-speed access (high-speed data over phone lines; cable)
- wireless high-speed access (terrestrial/LMDS; satellite services)
- short-range wireless LANs (HomeRF, Bluetooth)
- networking/communication protocols for information appliances (Jini, Universal Plug & Play, Inferno, Odyssey, T spaces, JetSend)
- Applications
- networked household (TV set-top boxes, handheld communicators, PDAs, PCs, laptops, printers/scanners, digital cameras)
- networked automobile (GPS, PDAs, cell phones, video games, e-mail, voice mail, web surfing, navigation  assistance)
- BodyLANs (e.g., FedEx integration of handheld computers, scanners & printers to determine status of packages)

## Proposed NIST Efforts

- Measurements and standards for long-term as well as short-term needs.
- Rich, natural human-computer interaction
- Extensive information presentation & visualization
- Dynamic networking
- Sophisticated multi-media information extraction
- Collaborative working environments
- Security & reliability
- System integration and interoperability
- Tight integration of IT with electronic, semiconductor, optical, manufacturing, chemical, materials, building and fire technologies

## Worldwide Information Appliance ForecastUnit Shipments (000)

## Appliances vs PCsUS Shipments, Consumer Devices,In Millions

## Applications & Scenarios

**     ****Personal Assistants**

-  Extension of today’s pocket organizers, smart cards, smart mobile phones
-  New functions: electronic money clip, locator, visual communicator
-  Provide access to personal data, monitor offices & homes, transfer funds, make secure transactions
-  Increase effectiveness of people’s time; extend the range of their actions

- Personal assistant for youth/elderly
- Provide ID information
- Vital data
- GPS-based location
- Remote transmission of location
- “Help me” communication function
- Access to medical information
- Funds accessible for emergencies
- Voice I/O
- Embedded camera for visual interaction

- Personal assistant for adult
- Secure access to information
- Embedded biometric sensors
- Coordinate user’s schedule
- Retrieve & store phone, fax, e-mail
- Personal e-business
- Electronic purchasing of tickets
- Electronic toll payments
- Electronic banking
- Integration with office computer,
-     printer, in-car digital maps, WWW
- Voice I/O, voice-to-text, text-to-voice
- Embedded camera for visual interaction

## Applications & Scenarios

**           ****Smart Spaces**

- Smart Spaces are work spaces embedded with computers, information appliances, and multi-modal sensors that allow people to work efficiently, together or individually, through unprecedented access to information and help from computers.

- Smart Space Examples
- Business meeting room
- Medical consultation room
- Training/education facility
- Military command center
- Crisis management center

-  Identify and perceive users and their actions and goals
-  Speech, natural language, computer vision input
-  Interaction with information-rich sources
-  Provide for the use of mobile devices and receivers
-  Provide extensive information-presentation capabilities
-  Understand and anticipate user needs during task performance
-  Distributed and local collaboration, including collaboration with field personnel & mobile workers
-  Provide improved memory and summaries of activities and deliberations for later use

## Technologies for ITL Involvement

- Rich, natural HCI
- 	Speech & natural language 	understanding
- 	Interactive dialogue
- 	Text-to-speech generation
- 	Detecting, localizing, & tracking 	people
- 	Recognizing people
- 	Activity & gesture recognition
- 	Situation awareness
- 	Summarizing activities & events 	recorded by sensors
- 	Computer vision processing
- 	Distributed multi-sensor fusion & 	integration
- Integrated information presentation & visualization

- Dynamic networking
- 	Mobile/wireless communications
- 	Real-time system & 		communications protocol
- 	Data compression
- 	Distributed databases
- Multi-media information extraction
- 	Document retrieval
- 	Question answering
- Collaborative working environments
- 	Distributed & local collaboration
- Security
- 	Computer/Network Security
- 	Biometric authentication
- System integration technologies & interoperability

## Slide 11

## Perceptive Interface Overview

**Video**
**DSP**

**World**
**Model**

**Pattern**
**Classifiers**

**Acoustic**
**DSP**

**Display**
**Services**

## Smart Space Test Bed

**Camera **
**Elements**

**Large Screen Display**

**Microphone**
**Phased Array**

- Array
- Beams

- Equipment
- Room

- 64 ADCs/Microphones
- TI 320C32 60MHz with 512k of RAM
- 3 cameras/video capture
- Gigabit ethernet
- Servers for data
- Pervasive devices
- Palm tops
- Tablets
- Wireless LAN

## One-dimensional Microphone Array Geometry

**IAUI Prototype**
**Analog**** ****Arrays**

**59 Microphones at**
**15 mm Inter-sensor**
**distances**

**45 Microphones at**
**20 mm**
**Inter-sensor**
**distances**

## Frontal Face Detection

**Skin pixel extraction**

**Face localization**

**Template matching**

**Largest cluster of skin pixel coordinate**

**Template**

## Face feature extraction

- Cropping

- Edge detector

- Hough Transform

**Face feature extraction**

**Crop largest skin pixel cluster from frame**
**Apply: **
**Edge detector to emphasize circular shapes**
**Hough Transform for eye-extraction**
**Face Normalization to gallery conditions**

## Slide 17