Computer

Systems

Technology

U.S. DEPARTMENT OF

COMMERCE

National Institute of Standards and

Technology

Nisr

NAT'L INST. OF STAND & TECH R.I.C.

NIST Special Publication 500-198

Monitoring and Reporting Techniques

for Error Rate and Error Distribution

in Optical Disic Systems

Fernando L. Podio

A111D3 S310ME

NIST

PUBLICATIONS

100

.U57

500-19f

1991

LC.2
NATIONAL INSTITUTE OF STANDARDS k

TECHNOLOGY

Research Monnation Caiter

Gaithersburg, MD 20899
NO

NIST Special Publication 500-198 QCi

Monitoring and Reporting Techniques '^<- for Error Rate and Error Distribution

in Optical Disk Systems

Fernando L. Podio, Editor

Proceedings of a Workshop

Held in Colorado Springs, Colorado

on August 5, 1991

Advanced Systems Division

Computer Systems Laboratory

National Institute of Standards and Technology

Gaithersburg, MD 20899

October 1991

U.S. DEPARTMENT OF COMMERCE

Robert A. Mosbacher, Secretary

NATIONAL INSTITUTE OF STANDARDS

AND TECHNOLOGY

John W. Lyons, Director
Reports on Computer Systems Technology

The National Institute of Standards and Technology (NIST) has a unique responsibility for computer

systems technology within the Federal government. NIST's Computer Systems Laboratory (CSL) develops standards and guidelines, provides technical assistance, and conducts research for computers and

related telecommunications systems to achieve more effective utilization of Federal information technology resources. CSL's responsibilities include development of technical, management, physical, and administrative standards and guidelines for the cost-effective security and privacy of sensitive unclassified

information processed in Federal computers. CSL assists agencies in developing security plans and in improving computer security awareness training. This Special Publication 500 series reports CSL re- search and guidelines to Federal agencies as well as to organizations in industry, government, and

academia.

National Institute of Standards and Technology Special Publication 500-198

Natl. Inst. Stand. Technol. Spec. Publ. 500-198, 89 pages (Oct. 1991)

CODEN: NSPUE2

U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON: 1991

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, DC 20402

i
Preface

This report constitutes tine proceedings the workshop on Monitoring and Reporting

Techniques for Error Rate and Error Distribution in Optical Disk Systems held on

August 5, 1991, in Colorado Springs, Colorado. The workshop was the first step in a Computer Systems Laboratory of the National Institute of Standards and Technology

(CSL/NIST) program of work to identify Federal Government needs and the state of the

art on error rate monitoring and reporting techniques of optical disk systems. This

program is aimed towards promoting discussions on possible future optical disk drive

implementations and to promote consensus among drive manufacturers in the way of

reporting error rate information through an interface.

Thirty-five participants from Government and industry attended the workshop. Three

members of different agencies in the Federal Government gave presentations. Industry

speakers included two representatives from error correction and detection (ECC) chip

manufacturers and two representatives from drive manufacturers. The workshop

included discussions on (a) Government needs on error detection and correction, (b) error measurement capabilities on current logic chips, (c) error management strategies,

(d) error recovery techniques, (e) error reporting capabilities of current drives, and (f) plans for a Government/industry working group.

There was a panel discussion during the afternoon session. A preliminary set of user requirements was identified. These are as follows: (a) Report on the consumption of

spare sectors and the rate of change, (b) Report correction (on record base) rate

above some threshold (corrections/capacity) at the codeword level, (c) Capability to

display the uncorrected sector content at the host level, (d) Capability of reporting all soft errors on demand, (e) Capability of reporting all hard errors on demand, and (f) Report on errors encountered reading header information.

Discussions took place on how to reach consensus on reporting error rate information

to the host to satisfy the user's requirements. It was proposed that a working group or

similar structure sponsored by CSL/NIST be organized. This idea was well received by

the participants. It was also suggested that the approach to achieve consensus on

standardized method (s) of reporting error rate information to the host should be the
documentation of a Small Computer Systems Interface (SCSI) common command set

for error rate monitoring and reporting. This suggestion was also well received by the

majority of the participants.

Specific vendors and commercial products were cited during the workshop by

speakers and participants. The inclusion or omission of a particular company or product does not imply either endorsement or criticism by NIST.

I express appreciation to the speakers for their input to the workshop. I gratefully

acknowledge also the assistance of all those who made the workshop a success.

Fernando L. Podio, Editor

iv

4
Contents

Preface iii

Abstract vii

1 The NIST Program for Investigating Error Reporting Capabilities

of Optical Disk Drives

F. Podio, National Institute of Standards and Technology (NIST) 1

2 Data Management in NOAA

W. Callicott, National Oceanic and Atmospheric

Administration (NOAA) 9

3 Determining Error Management Strategies (*)

W. Brown, A. Barman, Eastman Kodak Co 31

4 Error Measurement and Recovery Capabilities on

Cirrus Logic Chip Set

N. Syrimis, Cirrus Logic 43

5 Optical Disk System Error Detection/Correction: A Records

Management Perspective

C. Obermeyer, National Archives and Records

Administration (NARA) 51

6 Advanced Burst Error Processor

B. Chang, Advanced Micro Devices 53

7 Error Handling and Reporting in the Kodak System

6800 Optical Drive

S. Gerger, Eastman Kodak Co 61

8 Error Reporting in the LD41 00\LF4500 SCSI Event Log

R. Glissmann, Laser Magnetic Storage 75

(*) Reproduced with permission from SPIE - The International Society

for Optical Engineering. Presentation given by W. Brown.

v

Monitoring and Reporting Techniques for Error

Rate and Error Distribution in Optical Disk Systems

Abstract

This report constitutes the proceedings of the workshop on Monitoring and Reporting

Techniques for Error Rate and Error Distribution in Optical Disk Systems held on August

5, 1991, in Colorado Springs, Colorado. The objectives of this workshop were to identify

the state of the art on error rate monitoring and reporting techniques in optical disk

systems and to promote discussions on possible future implementations.

The workshop presentations included the description of a Computer Systems Laboratory

of the National Institute of Standards and Technology (CSUNIST) program for investigating error reporting capabilities of optical disk drives, Federal Government needs on error detection, correction and reporting, and the state of the art on error reporting capabilities

in current generation drives. Presentations also included the description of the capabilities

of error correction and detection current chips and discussions on error management

strategies.

The participants noted the need for tools for reporting error rate activity through interfaces

with the host and emphasized the importance of this subject to data managers in the

Federal Government.

During a discussion panel, the participants identified a preliminary set of user requirements

and it was suggested that a Government/industry working group be organized in order to

document a consensus position between the the Federal Government and industry in error reporting capabilities for future generation drives.

Keywords: error detection and correction chips; error recovery techniques; optical disks, error detection and correction; optical disks, error distribution; optical disks, error management techniques; optical disks, error rate.

vii

The NIST Program for Investigating Error Reporting

Capabilities of Optical Disk Drives

Fernando L. Podio

Computer Systems Laboratory

National Institute of Standards and Technology

Federal Government data managers and many private companies are already using

optical digital data disks for long term storage of valuable data. Optical disk drives are designed with strong but not unlimited error corrections capabilities. If the level of errors overcome the error detection and corrections mechanisms implemented in the optical

disk drive controllers, uncorrectable errors may occur. Most of the current optical disk

systems do not provide the host with sufficient error rate information. Federal

Government data managers are interested in being able to monitor error rate activity in optical disk drive systems. This could be achieved through the interface, if the drives

would provide information on correctable error rate activity, such as maximum number

of correctable errors, maximum number of errors per interleave, and the location of those

errors. Reports on errors encountered while reading header information is also

necessary. This information would provide data managers with a better understanding

of the status of their data and would allow them to design more efficient recopying

policies to copy the data to similar or different media in a timely and economic fashion.

The Computer Systems Laboratory of the National Institute of Standards and Technology

(CSL/NIST) organized a program of work to identify Federal Government needs and the

state of the art on error rate monitoring and reporting techniques of optical disk systems

to the host. The first stage of this program was to organize a workshop to identify the

state of the art on error rate monitoring and reporting techniques and promote

discussions on possible future implementations. CSL/NIST also proposes the

organization of a working group to discuss user requirements, to achieve consensus

between the Federal Government and industry on standardized method (s) of reporting

error rate information to the host through an interface, and to document these methods.

Another stage of the CSL/NIST program is to implement a demonstration platform at NIST

to show the Federal Government users the state of the art on error rate monitoring and

reporting techniques as well as the interface capabilities. The platform will consist of two

hosts, a SUN SPARCstation IPC and a DOS engine. When drives become available with

the recommendations achieved during the working group, the demonstration platform will be adapted to accommodate this next generation of drives with enhanced error reporting

methodologies. Any information that the working group may develop that may be used

for the next generation standards (such as the Small Computer Systems Interface (SCSI)

standard will be provided to the relevant standards committees.

1
(0>

is

.O) 03

COo °

>*- CO 2 Q.

0. 03

CO O) ^

tr

CD o

DCO

LU

Q-2

Q- (0 0 ^ 1 ^

U. Q.

sz2

(0

0)

tr

0)O)

COo

CO t

C\JI m

O CO

O)

_ooc

CD ^ P o o 2

CT3 ^

CO ^

c5 ^ S ^

H—• CO O 1- B g

.i Q- ^ o _ ^

COco

{5 2
COcowmm g«cooCo

Q.0

LJJ

(/) >

(0 Qo coEoc

I—

c^

(/) o o o to

=6 o ^

-D

g E

OJ CO w Q)

CO

O

CO c

.y o n ^ o n

eg CD O

-I—• r"

Q. o

0)O CD

CD CO ^ O

CO o

CO

-»— CD' ^—

CD

C E

CO ro E o

§.1 o D E ^ o _ 5 0) 2 o cc

C^ CD £ ^ +^ o o

CD o cd'^

CD CO o E

t=

D

CD

(D

-Q

c;3

CO

0)

O)

COc

COF

CO "co to^^

CD

CD

"O3 « O 0)

CO

C

E^O CO - 2

a5

.c^ -o

x: c

•- 6) m o O CD

lo£

.y CD o

-* a

C CD o to

Q- o

CD ."tl £ cc o

CD

O

(f)

n

D i=: CO O ^

H^ CD

>^ ^

O)^ CO

_cg o 3
EO

CO

CD

COOo

CD

QQ.DOC

CLO

CO

-is::

C/)

"D

_C

+^c

CDEc

a5>oO o1

CLCg

•—

COcoE

CD Q

CO

CD

CDEEoo

CO D

c6

"Dc

03 +-

CO +-

c

cn>

Q)

CD

CDo

cnQ.

"o

"tr

03 4
O

0)

CO D

CD

(DC

+-

cEc>oO

Q)

"D

CD

CD

CO

CD o 13

O"

0) "c co o

CD

+-' CO U)

CD c

"t

-1—

o o ep

CD

T3

-1—

C

CO CO

CD O)

-4—' _c

-1—

c c

CD o D E

COcg

-»—

c

CDE

0)

Q.E

1-

CD ^

Q) Q

CO

COo

Q.co

COcg

CO

COo

CO

CD

+-> oEo\—

QCLOc

COD

TDC

+^C

CDEc

Q)>oO

COocg

"toN

"c

CO

D) o 5
(D

to

Q.2

O)c

COE

0)>oO

13 DCc

CO0Ec

CD>OOEo

CO

CD>c0

CO

CD^

CL

CDo

'c

x:o

CD 0

D)C

"tr o

Q.0

c6

"a "o

X: CT3 0N

D)

"D ^ C CO

Cn "7

CO ^c o

ii

CO0EEoo

CO

"D

c6

"Dc

-I—

CO

+-» c

CC>

_00

COcn

-I—

coo

CQ

COXo^

CO >> CO ^

CO oc _ coQ

P"c6

.b

-•—

O» — 0

CO ^ o

><c0o

Q_

CD

(3)

1^

1—0n

*—o

oO

D)C

"-»—

00E

CMDc0g

Q.

CO

O ;^

CO

LL

•*—» 0

CO

"a

-I—

0

O)

CO
Eo1

CLCo

"(5

i_ +—

COco£

CD Q

J;; CD

05 ^ c h C CD

§^2 o O 0)

t= CD

CD >

§^

<D o

Co Hto o

CD C/)

CD

ItO o

CD

CO E o c

oQ.cn Q 2 E

CO

-H-•

COo O

Q_Cg

-I—•

CO O

CL

CO

CD

LO

CO

CO 5 CO <

CD

Q- z E 3 O

COO

CO

CD>

CO

'"O

Id

(J

'*-> CL o

"52 • c ^

CD ID O JZ E ^ E E 8 o) o g

CD "t^ O O

CD

£2o ^

CD "D

Q_ ^C C CD

4—1

CD CD

COcoE

CD a

CO

CD>a £ o

-»—' i_

Q. CD

CD

C

"D CD < O) 7
0)0EEoO

0) a

(5 ac

CO

CO

CO b

CO

-e—

03 Q-f—

b

Q. O

CD

COXo

CO

CDo

+-» CO

0)

iZ

CDo

05

CD

_ -I—

= c

T3C

05

Q)

05 m

COXo

CD>

CD

CD

ai

\-

coXO 8
DATA MANAGEMENT IN NOAA

William M. Callicott

August 5, 1991

9
ABSTRACT

NOAA has 1 1 terabytes of digital data stored on 240,000 computer tapes. There are an additional 1 00 terabytes (TB) of geostationary satellite data stored in digital form on

specially configured SONY U-Matic video tapes at the University of Wisconsin. There

are over 90,000,000 non-digital form records in manuscript, film, printed and chart

form which are not easily accessible. The three NOAA Data Centers service 6,000

requests per year and publish 5,000 bulletins which are distributed to 40,000

subscribers. Seventeen CD-ROMs have been produced. Thirty thousand computer

tapes containing polar satellite data are being copied to 1 2 inch WORM optical disks

for research applications. The present annual data accumulation rate of 1 0 TB will grow to 30 TB in 1 994 and to 1 00 TB by the year 2000. The present storage and

distribution technologies with their attendant support systems will be overwhelmed by

these increases if not improved. Increased user sophistication coupled with more

precise measurement technologies will demand better quality control mechanisms,

especially for those data maintained in an indefinite archive. There is optimism that

the future will offer improved media technologies to accommodate the volumes of

data. With the advanced technologies, storage and performance monitoring tools will be pivotal to the successful long-term management of data and information.

10
TABLE OF CONTENTS

1 .0. Data Management in NOAA

2.0. NOAA Data Management Operations

2.1. National Climate Data Center

2.2. National Geophysical Data Center

2.3. National Oceanographic Data Center

2.4. NOAA Centers of Data

3.0. Digital Request History and Projection

4.0. Mass Storage/File Management Requirements

4.1. Hierarchical File Director

4.2. Hierarchy of Storage Levels

4.3. File Management Software

4.4. Network Access and Networking

5.0. Considerations

6.0. Assumptions and Constraints

7.0. Conclusion

11
1.0. Data Management in NOAA

Management of environmental data and information resources is becoming an

increasingly visible and important issue for the scientific community. This is of

particular importance to the National Oceanic and Atmospheric Administration (NOAA),

which routinely measures and collects large amounts of environmental data and

information in its own work, and is also officially charged with maintaining

environmental records for the Nation. Through its activities over time, NOAA has

become the steward of a treasury of Earth systems data and information--the most

comprehensive, long-term, and up-to-date environmental description of the Earth that

exists today. This treasury contains answers to urgent environmental questions facing

the Nation. The success of all NOAA's scientific work, and the national priorities which

it supports, depends on the accountability and accessibility of environmental data and

information. These data and information must be accurate, complete, stable and fullyintegrated across the spectrum of NOAA's organizational functions, and they must be

made easily accessible, in a timely and cost-efficient way. Throughout this effort to

provide resources to meet the NOAA missions of managing data for global change

research and for enhancing warning and forecast services, there will be a continual

inherent process to migrate and protect data held in the NOAA archives.

2.0. NOAA Data Management Operations:

The NOAA Centers respond to requests from a broad community of research, legal,

engineering, individuals, insurance, business, consultants, and manufacturing. About

6,000 requests per year are received for digital data. Within the next two years as

global change research increases, this should grow to 9,000. The current data files stored on computer readable media have a volume of 1 1 terabytes stored primarily on

240,000 computer tapes. There are 1 00 TB of GOES data at the University of

Wisconsin copied in digital form on specially recorded SONY U-matic video tapes.

There are analog, i.e., non-digital, holdings having a volume equivalent to over 50 TB

in a digital domain. With the conversion of some of the analog data to digital format,

and with new sources of digital data, the digital holdings will grow by 1 996 to about 35

TB not counting source level data from GOES. The rapidly expanding quantity of data

will force a different approach to managing data and information within the centers.

The use of an integrated mass storage systems with appropriate file management will be essential to manage the archive, storage hierarchy and data migration processes.

New mass storage hardware and software technologies will have to be developed and

adopted and the current archives copied. If a mass storage system is not developed,

the data centers will require immense tape storage areas and reduce the access

mechanism from data granules to physical volumes of data.

In accordance with the NASA/NOAA Memorandum of Understanding for remotely

sensed earth observations, data processing, distribution, archiving and related science

support, EOS data is intended to be archived at the NOAA data centers. The EOS

12
data from prototype operational instruments used for NOAA operational purposes, will be an inherent part of NOAA's data archives. As part of the EOS pathfinder activity,

NOAA is migrating the environmental satellite data from the operational polar orbiting

satellites from computer tapes to 12 inch SONY optical write once, read many

(WORM) disks. Also, there are selected special sensor microwave data from the Air

Force Defense Meteorological Satellite Program (DMSP) of interest to EOS scientists.

Initially, 8 terabytes of data will be migrated to optical disk.

The computing capacity at the centers is provided by mainframes, workstations, and

personal computers (PCs). In the aggregate, the systems do not have the capacity to

handle the anticipated archive, the growth in data ingest and dissemination, and

expanded quality control, analysis and reprocessing requirements in the coming

years. The computational capacity will need to grow from 10 MFLOPS at the

beginning of 1991 to over 300 MFLOPS by 1996, i.e., a factor of 30:1. On-line disk

storage should grow from 60 GB to over 500 GB during this period. The

configurations are evolving from a central processor surrounded by dedicated

terminals to a fully distributed client-server architecture which can expand in response

to workload demands.

2.0. NOAA Data Centers : There are three National Data Centers and over a dozen

centers of data in NOAA. The data centers are structured as formal archive centers

and serve at this Nation's world data centers for their respective disciplines. The

following provides a description of the centers and their activities.

2.1. The National Climate Data Center (NCDC) in Asheville, North Carolina was

established in 1950. The National Archives and Records Service, in compliance with

the Federal Records Act of 1 950, specified that NCDC's climatological records be

permanently retained. It has been designated as a World Data Center (WDC)-A for

Meteorology. It also operates the Satellite Data Service Division which manages the

high volume satellite data. The Center is responsible for ingest, quality control,

archiving, managing, providing user access, and performing analysis of data which

describes the global climate system. The NCDC also supports major new programs

such as the National Weather Service Modernization, Climate and Global Change, the

Coast Watch Initiative, and the Level-0 Earth Observing Systems (EOS) path finder

effort. In 1992, new data sources from foreign satellites will be introduced. The center

has a staff of 290 full time employees (PTEs), 100 contract and 190 federal employees.

Each working day results in 1 60 orders from 360 user contacts. Annually, about 5,000

bulletins are prepared and supplied to 40,000 subscribers. Much of NCDC's data (by

physical volume) is in the form of paper records such as ship logs, and although

manually accessible, is not readily usable, and they appear to be deteriorating. There

are fifty thousand cubic feet of paper records at NCDC. There are also film and other

non-machine readable information stored in the National Archives.

13
NOAA has a contract with the University of Wisconsin to collect and archive data from

the NOAA geostationary operational satellites (GOES). The GOES data collected from

1978 to the present, is recorded and stored at the University of Wisconsin's Space

Science and Engineering Center. To date, the GOES data are stored on 25,000

Sony U-matic beta video (19mm commercial video standard) video cassettes in 4 GB

increments. Access to the data is not highly efficient because some of the cartridges are offsite in the state records office, and because the data must be reingested as a

satellite readout. The GOES archive represents the largest amount of data anywhere

in the NOAA system to be rescued and be made more readily accessible. To improve

access, the center is engaged in a pathfinder study on mechanisms to improve the

access to the data.

2.2. The National Geophysical Data Center (NGDC) in Boulder, Colorado was

established in 1 972. Its mission is to manage solid earth and marine geophysical data

as well as ionospheric, solar and other space environmental data; and to provide

facilities for World Data Center-A for Geophysics which encompasses Solid Earth

Geophysics, Solar-Terrestrial Physics, Marine Geology and Geophysics, and

Glaciology. The center has a staff of 60 PTEs.

NGDC has over 300 databases including 54 million (M) lonograms, 2.5M

magnetograms, 12 million flight miles of aeromagnetic data and 10 million miles of

ship track data. There are 25,000 magnetic tapes partly at NGDC in Boulder and

partly in Asheville at NCDC. About 2000 tapes per year arrive from originators outside

of NGDC. Their goal is to keep all of the ingest tapes and maintain two additional

copies for use in normal center activities (3 copies total). About 14,000 tapes have no

backup. There is a requirement to copy 12,000 tapes a year for routine migration.

NGDC relies on the error checking provided by the tape copying system and

supplements this with printouts of beginning/end of record data and record counts.

NGDC began the NOAA CD-ROM program four years ago. Its first CD-ROM title was

"Geophysics of North America". The center now has a total of 12 CD-ROMs

completed or underway. This effort should change the distribution system for data

from its tape orientation and probably deflect some use of the network to obtain data.

During 1990, distribution of data by CD-ROMs has increased both the number of

requests for digital data and the annual amount of data distributed by the center. The

distribution of data by CD-ROM puts the data into a form highly convenient and useful

to PCs and workstations.

The National Snow and Ice Data Center (NSIDC) under contract to NGDC operates

the World Data Center-A for Glaciology. The role of NSIDC is to acquire, archive and

disseminate data relating to all forms of snow and ice. It provides data to about 500

requesters per year from a digital archive data base of about 15 GB (300 standard

tapes); 7GB are from the NIMBUS-7 Scanning Multi-channel Microwave Radiometer,

and 3 GB are Special Sensor Microwave/Image (SSM/I) data. Many of the NSIDC

14
datasets are redundantly held at other NOAA data centers. A daily volume of 1 GB of

data from the Defense Meteorological Satellite Program (DMSP) Operational Linescan

System (managed by NGDC) will be forwarded from the DMSP readout site to NSDIC

on EXABYTE tape. A similar means is being developed to distribute a weekly data

volume of 800MB between the Joint Ice Center (JIC), in Suitland and the National

Snow and Ice Data Center (NSIDC) in Boulder. The NSIDC has been designated as an EOSDIS Distributed Active Archive Center (DAAC). As such, it will build up its computational and archival ability to meet EOSDIS defined mission goals. As a DAAC,

the NSIDC will relocate to the University of Colorado campus.

2.3. The National Oceanoqraphic Data Center (NODC) has been in operation since

1961 as an interagency, facility under the U.S. Navy, and became a part of NOAA in

1 970. Its mission is to manage oceanographic data. It has operated as a World Data

Center-A (WDC-A) for Oceanography since 1 962. NODC has a staff of 85 PTEs.

NODC's files include data collected by U.S. federal agencies; state and local

government agencies; universities and research institutions; foreign government

agencies and institutions; and private industry. Currently, NODC maintains a digital

archive of both in-situ and satellite-sensed ocean data in excess of 30 GB. A potential

equivalent of 10 GB of digital data are currently maintained in analog form such as

data reports, manuscripts, and analog instrument recordings. With the establishment

of NODC data management responsibilities for ocean observing satellites, including

non-NOAA geostationary and orbiting platforms; new global collection efforts,

including the Global Ocean Flux Study (GOFS), the World Ocean Climate Experiment

(WOCE), and the Climate and Global Change Project; and new U. S. coastal ocean

studies, including CoastWatch and the Coastal Ocean Program, the archive is expected to increase twenty fold between FY90 and FY95.

NODC has a large amount of analog data. A tablet digitizer is used to annually

process 1 0,000 expendable bathythermograph traces (XBT) to 2 MB of data. NODC

has a contract with the Navy to annually process 100,000 similar traces. Mechanical

bathythermographs (MBT) on glass slides (300,000) await conversion and are expected to result in 1.5 GB of data. Acoustic Doppler Profiling, done from University

and NOAA ships, is expected to be a new source of data. There are perhaps 20

ships that may be equipped with these profilers. At the present, only a few are capturing the data for archiving purposes. Each profiler should provide 10 MB per

ship month. In the future, 100 ship months per year of these data could be archived.

The NOAA Coastwatch Data Management, Archive, and Access System (NCAAS) now

under development will result in expansion of the archive, related quality control, and

retrieval and distribution activities based on SONY 12 inch WORM Optical Disks.

NODC also is responsible for the NOAA Library and there is interest in digitizing some

of its data and metadata holdings.

15
NODC is responsible for the NOAA Earth System Data Directory, and interfacing it with

the larger NASA based master directory effort. This is part of the catalog

interoperability effort which is underway among other government agencies and

foreign data centers. The master directory is also needed to fit with the EOSDIS

version 0 effort where the catalog interoperability will perform relevant IMS (Information

Management System) functions. NOAA's master directory is VAX based, with

specially written software, and the ORACLE Data Base Management System.

A prototype database system has been developed to provide NODC users with direct

access to an on-line, interactive data archive. It maintains a data base of over 23

million marine observations from 31 0,000 ocean stations. Access to the data is obtainable through spatial or temporal searches with arbitrary combinations of

instruments, platforms, and parameters. By FY 1993, NODC plans to add all of its vertical profile data (Nansen, Bathythermograph, C/STD, etc.) to the POSEIDON

system.

2.4. NOAA Centers of Data include those data collection and operations elements

performing observations and monitoring services as part of NOAA's recurring mission

responsibilities. Listed below are the principal centers:

Discipline Title Location

Bathymetry,

Nautical charts.

Geodesy

Climate

Fisheries

Ice

Lake Data

Oceanography

Oceanography

Charting and Geodetic

Services

Rockville, Maryland

Climate Analysis Center Camp Springs, Maryland

National Marine

Fisheries Service

Navy Joint Ice Center

Seattle, WA; Woods

Hole, MA; Miami, FL;

Bay St. Louis, MS;

San Diego, CA

Suitland, Maryland

Great Lakes

Environmental Research Lab

Center for Ocean

Analysis and Prediction

Ann Arbor, Ml

Ocean Products Center

Monterey, CA

Suitland, Maryland

16
Pacific Ocean

Data

Equatorial Pacific Seattle, WA

Information Collection

Particle

Deposition Data

Air Resources Lab Silver Spring, MD

Sea Level University of Hawaii Honolulu, HI

Snow and Ice National Snow and Ice

Data Center

Boulder, CO

Tides National Tide and

and Water Level Data Base

Rockville, MD

Trace Gases Global Monitoring for

Climate Change

Boulder, CO

3.0. Digital Data Request History and Projection

The support for global change by the NSF has increased about 35% per year since

1987 and is expected to increase for FY 1992. NOAA's global change funding has

roughly doubled each year since 1989. Other agencies are also increasing their

global change funding. The overall funding for all agencies has increased seven times

from FY89 to 1 991 . From this, one could expect a large increase in the number of

data requests at the centers. However, in the aggregate, there has only been a modest increase in the number of requests for each of the last two years ('89 and

'90), and the projections are, therefore, based on this modest rate of increase.

Another view is that the secondary distribution of data from scientist to scientist may

be on the increase because of the ease of transmission over networks, coupled with a

desire to obtain a dataset that has had use in a familiar science project. Possibly this

secondary distribution masks the size of the actual data dissemination.

The biggest impact on the number of requests and volume of data distributed has

been from the introduction of CD-ROMs. This indicates that the increased use of

CD-ROMs for data distribution provides a means for rapid deployment of the data

among members of the research community. Secondary distribution of data from

scientist to scientist may be on the increase because of the ease of transmission over

networks, coupled with a desire to obtain a dataset that has had use in a familiar

science project.
4.0. Mass Storage/File Management Requirements

As the amount of data increases and the NOAA mission focuses on improving

accessibility of data for global change research, there is an urgent requirement to

develop mass storage systems with file management software at the centers to

improve archive management, provide vastly improved access mechanisms, and to

reduce the amount of media and associated space requirements. Moreover, the mass

storage system is the heart of a file management system. For the immediate purpose

at hand, a mass storage system should include the following:

4.1. Hierarchical File Director

A hierarchical file directory is needed that permits, as a minimum, the acceptance of

UNIX file names. The directory needs to maintain the access and update history

information for the file. The directory should allow for handling mixed media within a

single search, for cross indexing between devices, and for recording data set

utilization records for future knowledge based system applications. This directory

must interoperate with a number of different data base systems passing query

information through during interoperable data searches.

4.2. A hierarchy of storage levels

The mass storage system should support a hierarchy of storage levels with increasing

physical capacity and decreasing performance at the bottom, and decreasing physical

capacity and high transfer rates at the top (as viewed from the user client processes).

At the top, this permits the evolution to direct access electronic storage, so that the

mass storage becomes a truly integrated part of a computing environment.

4.3. File Management Software

The file management software should offer options for data compression. It should

permit the use of checksums as an overall error control mechanism. Data conversion

software should be available to migrate the data from one physical media to another, as generic files, without disturbing the data content. The software would sample files on a statistical basis reading them to verify that they were still intact and that the

media had not deteriorated beyond the point where only soft data checks were

obtained. In the event that sufficient degradation was detected during this sampling

process, the files would be migrated to new media with a corresponding directory

update. Migration would also be triggered during normal accesses whenever too

many soft errors occur.

18
Migration of files from arcliival or working media to a buffer storage area would occur

following the initial access to permit data to be more rapidly retrieved from the faster

devices in the storage hierarchy. The migration and actual location of the data should

be presented to the user/application programs in a transparent manner including, as an option, presentation to client processes in a manner simulating direct retrieval from

the ingest media if desired. To accomplish this transparency, the file management

software should provide for the ingest of data from existing media and distributed to: standard half-inch magnetic tapes in all densities and formats, EXABYTE, DAT, CDROMs, optical disks, video cassette recording technologies, digital optical media, etc. An encapsulation of the ingested media's data should record the presence and

location of permanent data errors, physical record lengths in bytes, the presence and

location of ingest media specific flags, such as tape marks, end of tape flags, etc., so

that upon access, the data can be handed to processing programs that need to be

aware of the different media. The file management software should be able to handle

any of the existing data formats and to invoke conversion routines to a standard if one

is adopted. The ability to move files from the mass storage to a requester's media in the original format should be provided.

4.4. Network Access and Networking

The NOAA centers should be coupled to the internet, at internet backbone rates, and

eventually to the National Research and Education Network (NREN). With 740

universities, laboratories and industrial sites now on the network, and 75 more

expected in the 2nd half of 1991, the internet is widely available to the scientists

involved in global change and EOS. There are 16 NSFnet backbone sites. Two of

these, the University of Maryland and NCAR, are in close proximity to NOAA data

centers. Where large data volume data transfers are required, conventional

conveyance services would probably suffice with economic considerations determining

the mode of conveyance.

5.0. Considerations

The system life under the NOAA mandate to manage data for long-term global

change research purposes is open ended. The value of data increases with age for

use in performing long term environmental change research. Global patterns are known to be subtle over time, even when viewed in a rapidly changing environment.

Today's collection of environmental data is pitifully small and of too short a duration

compared to the amount required to filter out the statistical noise over an extended

time domain.

The operational requirements are influenced by incremental science requirements

established as the knowledge of the relationships between instrument responses and

conversion to physical units became better known from the results of research and

19
development of more sophisticated processing algorithms. Because the development

of sufficient knowledge to fully understand the earth observation responses is an

accretive and repetitive process, the entire data set will require repeated reprocessing

to adequately describe the data for long-term documentation and preservation.

Aggregation of similar but different and sometimes disparate data types is also an

important feature to include. A well understood aggregation principle will allow for

compartmenting the data across the media domain in a "most" convenient form for

vastly improving the access mechanisms. This will become increasingly important as

the longevity of the archive increases. Aggregation implies some degree of

redundancy, but in a positive sense, in that redundancy of particularly critical data sets

reduces the risk of data loss over time.

Volume management may require compression mechanisms to reduce the over- volume of data as it ages. Critical data and information will require the application of

lossless data compression where data sets are reduced in volume. When permitted,

other means can be used to reduce data volume with controlled data loss as achieved

through sampling, or through small-loss data compression techniques or a

combination of the techniques to yield much higher compression ratios. This may

particularly attractive for managing very high volume image data sampled in the visible

spectra.

The technological gallop of the last several years continues to accelerate and new

storage and processing technologies are obviating the need to consider destruction of

cumbersome data through full scene sampling, scan sampling or reduction to gross

descriptive parameters which describe the sum of the parts in abbreviated form.

In order to take advantage of improved and less costly storage technologies, there is a plan to migrate the data periodically as the volumes dictate and the technology

allows and with each migration to yet developed capabilities, it becomes even more

feasible to consider placing all of the data in a near-line access environment. The

migration process will require content processing to re-develop the cross reference

inventory information to include additional content description information as part of

the inventory metadata file to increasingly document the data as it ages. Data

migration is an essential element in developing a never-ending data life for the sake of

offering an extended time baseline data set essential for detecting global scale

changes.

A wide variety of media will be used for distributing data and information to users. The large number of formats used by the NOAA data centers means that many

conversion procedures will be needed. It would be better not to convert the data in the archives themselves, but to write procedures which can be invoked in a demand

fashion. In this way, the data can be left in its original form while confidence is gained

in the accuracy of the conversion. If any problems arise in the converted data, the

original data will not be contaminated. The problem becomes one of reworking the

conversion routine and alerting previous users of the defect rather than trying to fix a

20
partially scrambled dataset. The downside is that there will be an additional

processing cost when the data is requested. Another problem with the standard data

format concept is that many researchers who submit data to the data centers will probably never conform to a standard format. Insistence on a format may become an

impediment to releasing the data to a center for distribution.

6.0. Assumptions and Constraints

Factoring today's technology advancement timescale for the purpose of being both

realistic and conservative, the period of migration to exploit new technologies and

avoid system obsolescence to extend the validity of the data and information is established at no less than every 10 years. A suggestion was recently made that the

migration rate be a function of the expected half life of the medium used. For the

3480 tapes, the manufacturers agree that 10 years of full performance life should be

expected, thus the half life for migration purposes would be five years. The criteria for

accepting a new technology as a candidate for data migration is; improved archival

qualities, the per data byte storage cost must be one half the previous, the physical

storage requirements be at least five times less, and the data transfer rate to move

the data from the media be no slower than that of the older media. And, finally, the

data migration step will include data processing to derive or extend content

description data to be used for the purpose of validating the preservation state of the

data and for reinventoring the data to add content information developed through the

accretion of user knowledge and experience.

Another assumption is that all of the data will be reprocessed three times during a 25

year cycle. This reprocessing cycle may coincide with a data migration step since all of the migration will include a content review during the passage of data from one

medium to another. The development of reprocessing algorithms will not be charged

as a data management system task but will require that the data management system

be able to put significant quantities of data on-line or at-hand for "live" ingest mode

processing.

7.0. Conclusion

To continue managing data as we do today would eventually require a facility to

accommodate an enormous number of media units to hold the data volumes

projected for the future. As the new data continues, the added function of migrating

data from degenerating media (from a systems as well as physical viewpoint), will compound the storage requirements as the annual data volume accumulates by the

hundreds of trillions of data samples each year. If acceptable mass store systems are not continually developed to match the data growth and data management

requirements, the logistics would be overwhelmed and the system would fall apart never to be recovered again because of the enormous cost to recover an inevitable

21
backlog.

The data only has value to the research community if it is conveniently and efficiently

accessible. If the data were placed in a warehouse environment, which would

ultimately have to happen if nothing was done, it would soon lose its value and

possibly its identity because of the cost to acquire it and eventually would be lost

because of the huge cost to locate, ship back, copy and return the data copies as the

data volume grows beyond manageable proportions. This is aside to the issue of

data loss due to media deterioration. The only acceptable solution would be through

development of a system capability to provide highly efficient and sophisticated data

management capabilities which would accommodate online data sources. In order for

this to happen, advanced media technologies have to be employed along with

advanced sophisticated data management software to eliminate the manual interfaces

where possible to provide the data in a ready mode for user interaction at the

subsetting level. The data value increases dramatically when placed in this type of

environment as the access to the system can provide instant gratification and

encourages repeated and expanded data query activities. This, in turn, accelerates

the research progress and enhances the research results thus broadening the value

of the data to the advancement of science and knowledge.

To physically compress the data through the implementation of high capacity media

coupled with the systems capability to control and index these data in an online or

near-line environment offers a significant reduction in the requirements to house the

archives both in terms of physical space and recurring energy and labor expenses.

The closer on line the data are placed, the less labor is required to physically mount

data either in the appropriate archive slot or onto the processing system. As

electronic access become more fully integrated into the system, direct labor service

categories will be eliminated. A major cost avoidance to be reckoned with is the cost

of adding increasing large physical facilities as the data volume grows at the projected

rates. The pace of implementation of new technologies should allow shrinking of the

space requirements to match the increase of data accumulating in the facilities. An indirect benefit of space compression through improved storage technologies

would be realized form compressing the facilities requirements sufficiently to consider

replicating the data in distributed locations as a risk reduction measure.

The broadest benefits are in terms of what the value of the data is to the world of

environmental change. Without a responsibly managed data record of scientific

measurements over time, there would be no baseline to objectively determine if the

environment we live in is really changing, how much, and at what rate. Without these

data, economies would be based on subjective opinions and in some cases, hear say.

Public policy would more often than not be misguided and consequences of

enormous proportions could occur to our physical well being either through economic

collapse or through direct physical changes. Even today, global change scenarios

portend potential devastating effects to our coastal cities and this country's agroeconomies. But, do we build dikes and seek alternate water sources if we are not

22
really sure what, if any, impacts there are? Without the data, no one knows, so any

investment in mitigating a potential problem is an economic risk. The other question

is; even if we know, can we afford to take avoidance action? Or better yet, is the

cause due to environmental causes or due to a much broader cyclic processes. One

thing is certain, there is a great potential for change based on the knowledge at hand

today, and sound economic planning based on knowledge may be sufficient to avoid

economic collapse. In a Nation with a trillion dollar economy, the risks are too great

not to invest an insignificantly small percent of this economy to acquire the maximum

amount of knowledge possible and establish this knowledge base as soon as

possible. In the case of environmental data, data is knowledge; there can never be

enough data, and the data record can never be long enough. But where there is data, it must be systematically managed to be of any value at all.

23
zoH to

DATKM1

TCM

to

to >- H ot/> zHz UJ to Q H < o z UJ u U. -1 o \~ H < H H to u o

h- to -J UJ UJ

H

Q Z H -J UJ X 3 H

h- in > UJ

U Q.

O Q. s

1- Z H

H to HO o O < > O -J

LU

UH o

-J (0

00 z Z U H o

-J

u z -Jt>0< UJ GE > UJ < z 1— to < lU 1- Z Q o tOi-H O Z o o <3 > UJ a Z H iZ ^ < • tot/) z o H J- H LO z z < o z 1- 3

tJ

H o Z H to CD Q. hJ- z Z ^ O H t>o to < < (0 UJ H o < Q. tt H > z > U. UJ o UJ H- O O < -J t/) UJ z z u to O 3 O -JH u. <

LU

H OO §^ o z O -J O < < z < < tt UJZ z < 2 O o u o o o H < H < H Z tt

U- h- 1- HLlI

u. < < < < z

MONITO

Rates

and

24
25
•r-CO

LU

CD

-i oo • o • • < <

if) z o

UJ

X

O

J

Urn 1 hH

O z

llJ o zX

a < - (/) UJ

UJ ^ Q

Q -J 3 H 1 O Q > CQ

UJ OC Z Ul 1 o CO h- < Z

Ul Ul • O 1-z • < Ul

—J UJ h-O Q zo < z UJ 1- < o <

C/) < H • • h- Z -J <

llJ < 0. <

hO < U < 2 H Q

llJ Ul

O (/)

u 1- o >- U. < z z o

kJ Z < Q. z H UJ O H -J U UJ

-1 ooo UJ

0.

H

H -J -J -J Z U < < < UJ

(/) z z z u H o o o Q H H H z H H H Ul

UJ < < < UJ

UJ h-XH

00

O o QHH

4/T

ujS 3 <

CQ

I—I O

(OS

H 0 0. H zco 3 H

OQ Q. H Q. Z

ttCO <

(/) tt H UJ I O >

-J

Q

H UJ ZOO H < Z I Ul

UJU -I H < > O

CC

H 0H

(0

to

UJU UJ oo

-J a. o

UJ < z o

IH UJ < < H O UJ UJ

UJH

J -I < <

-J z z

-I o o

UJHH

I- HH < < <

oozz

UJS

-J

-J « H to > o ^ z U H o oc

Q. I Q.

to z

UJ < uo>

UJ

i/^ oc

UJ

U

H

H Z

K- UJ UIO Qo to

UJH o to O -I z < < z<oZ UJ H II- <z < H Z -J

UJH o

OQ < >

UJ

cc

UJH

UJ

-IH

I-<

UJ 00 3

Ul 00^ <

00

UJuH>

UJ

</)

to

UJH

a.

UJz

toH

UJzH<

-J

I-

<:

OQ * o Qz < < UJ

-I to H UJ

00 <

CC Ml UJ z H Z z o

UJ O

H>

UJ z

ULU

(0

h- UJ zH <

O-J

"-3

H > < > UJ <ZC9 oH

I-H So

UJ ^

Ct

Q

a. z< Q -J Z H <

tooo H

to I >

-I< UJ Z I- < z

UJ zo<

UJ to U I- o uQ o o

U.

tt

UJ z H < Z UJ

UJ

u oo o o H Z

IH O

UJ Q.

-JOO

-JO QC O Ul> Z -J O H H t/) H< I za > O

u. o Z H

l-H < o o H OQ

IJ- < u< to

Q_ UJu

-J CL < 3 H O

tt to O Ul <O H

LU < « UJ 3 H

-J z 3 UJ -JO oz < O I- 3= <oI UJ H U H HH <3 Q < z = <

u. 2 o o

^00 H -J to <

oc z

Ul o > H H I- z <

UJ

-I-JO HO 3 >

-) ^ ^ u oc ^ O UJ ococ o

UJ -J Q I 3

-I o

UJ QQ

to< I CQ

UJ < C9

I- z < < o zo-J

UJ -J > <

UJ CD -J o o

CQI

UJ

I- oc < o 3 u- Q O z z < H

UJ o O H H<Z -I o < H GQ

hO < -I

26
o o o o o o

27
t- (/) O hoe: >-

UJ u.

:s: o ^ (/) o z > oH ^ -J < -I h- H

hH CO

I- UJ < z O -J LU O

as

CM o o CSJ rH o

ro rH

COo o o <—

CO 00 ro o

1—

CM o% VO

CSJ in

CSJ o o o

VO CO

O

CO CM CO

iH

CM o oO m o

00 CM so

CO

rH o O o

r> VO

O

in in CM

CM

in o

ro

m

Oo

VOo

CMOo

VO

roooo

CMoo

rooo* o u

»- to z UJ UJ u 1- u z H

UJ > u < o

h- UJ UJ Z < < > u o H o < z o o u H < X -J o < < 0. < o u UJ < H h- 1-

llJ o (0

H <

1- o >- -J o < z z z < a LJ 00 H UJ o 1- LU o UJ < o o o o

ro

00o

00

00mo

VO

roo

00

CMo

00

-J<o

UJ>H

I-<

-J3U<

28
CO

CO • • HH

CO LU

OO

CQ UJ OQ H

LU (/) LU 1 UJ 1—

Q > LU

21 >- z H _J

SI HH < z

LU LU

u >- LU LU h— < o

CL

O HH

H <

Z! o >-

1—

HH >- UJ CL CD

Lu -1 z Ul

O

IS LUUu -J o O 1- Q o

2S

H z

</> HH o

LU o H UJ < o O LU U. z o LU 1 Ul UJ 1 z 1P— 1 1r- Cu o < C/) HH Om z ^ ^ O _iJ H 1 Ul c

QC LU Ul

III UJ

O z H

O z

(/> I 1 UJ 2^

u > \J

H z _ 1 1 HH H Ul III UJ III UJ

(/) o z vJ

OS t/l >- z

LU UJ HH (0 o H Z ^ UJ >-

</) -J Ul z >

LU (/) < Ul

H

(/) < (0

Q Z

Q. UJ

H o O

u o z Q: UJ u UJ O < >-

O < z UJ I 1 r- p— U. HH H z LU UJ o UJ UJ H l_ L_

00 LU o tt

H OG^ LU < u <

l-Hl- z (/)U u.O UJ 00 HH < < z UJ H H 0. UJ

00 -J UJ SO

l-l<

O

00 o HH 00

SP 1

Lu <

o< -J< 1 o o

29

III1z o

iii f 1 k.1 uV 1 J J Km UJHa 1 L3

III Mi L_

u z z ^> 1 1 III r\ _J L_i ry

Ul z o CO U III III iiwmm Ul (J

CD (/)

V 1 1 H L_1 z

u.1 z Q u.

Q UJ u.

UlQ > Ul Ul

w # 1u

Ul 1 T t/)

Q to Ul J CO 1u 1 ry iiJ ry v# III

111

III MM1 t/> VI

ryMi

U 1 Ulmmm <—

u1 1 rH UJ L.1 Ul

L.1 Ul

U. UJ (/) 1U z o UJ UJ z UJ > u. u1 o z H z H o Q. H o CL o I Ul UJ > H O o C/) -1 Of z H UJ

(/) (/)

J

UJH 1 UJ UJ UJ L. £V o H o ft 1 L- z H O UJ

1- 111 n_ z

CD < (J Ul z z o < z H H H H 1 I Ul (0 iU (/) z z > UJ < H H H H Z OQ Ul a ~» w O o UJ UJ u H UJ UJ Q. o

UJ o UJ z < X z <

-J^ o < U Q. • • • • •• •• •• ••
(0H o z >-

bJZ bJ o

III UJ

IM 1-

to

HUJ in 1-

MI o

EQ

iiw < > o o III UJ < O IH

bJ vJ

O to

ER

CO

Oz

bJ bJ He GES ITY ECI z H X 1- -1 o o O H to bJ H H H (0 (0 U 1 

CO

Q o H X bJ H vj o < -1 a H to H mm z H H H Z H H % H -J

CR RE RC

-1H 1 IIP RE ED S3 UR XH

XE AC

U III UJ TO

r\

u bl iimm s. Ufa C vl

h- bJ < to

u1 1 VI bJ

l-l< CL bJ b. to VI 1 H ZCQ O bJ (0 < bJ VJ H bl h- O vi H bJ Z z Q U III UJ Cfl H III UJ -J O H O > III UJ bJ H O O < O 1- H III UJ >-< « \ w III UJ L_ > 1- Z bJ

-JO. P"^ Z < bJ bJ

O

J

w > z < z U

O < <z 1- to H

III UJ < O

H < z -1 bJ < to

III Z 1—

[ Q VI bJ 1 hJ CL Z to u 1 

< z < III UJ Z bJ

1— 

Q bJ

O

Q to o to o U

tn > u Z > z u.

O

.J Z tt < to o Ii bJ < OO 111 1 bJ H O 1 u bJ

Uu > bJ b. to VI

O Q. b. LUZ I1 Ul <9 < O O < 1- O

Ot-H

U

V

J o L—1TM bJ < w III 1 -J < w z mJ 1 H <3 <z III UJ .< ^- O III UJ o O u Q O 1 

o 1- < H L»r" H 1 1 

UJ < bJ -J

U -J < o (0 !-

H o Z Oh- III UJ u Q 2

1,1 UJ z

Li-O VI MM1 U bJ

u H- bJ VI to bJ

H

LU t/\ IrJ 1 Q z > O

O bJ l_l~ bJ

N o

LU bJ 3 H

O < -1 CO >

Q.C^O liJ 1- .J

X tt D. < bJ z >- -1

H z O O z 1 O h- 0. Z > < bJ h- < H <o o z H Z o Q bJ o z Q. l-H Z O <

CO b. LU

O < OO <

LU Z 1 1 1 1 1 1 o 1 1 1 o o o

30
A Reprint from the

PROCEECai^

Of SPIE-The Intematovy Society fbr Optical Engineenns

Volume 899

Optical Storage Technology and Applications

12-15 January 1988

Los Angeles, California

Determining error management strategies

Winton E. Brown, Allen M. Herman

Eastman Kodak Company, Rochester, New York 14650

>1 988 by the Society of Photo-Optical Instrumentation Engineers Box 10. Bellingham. Washington 98227 USA. Telephone 206/676-3290

31
Determining error management strategies

Winton E. Brown and Allen M. Eartnan Eastman Kodak Company

Rochester, New York 14650

ABS TRACT

A statistical model has been designed to interpret experimental burst error data in determining an optimal error management strategy for optical media. The model is based on the theory of a Non- homogeneous Poisson Point Process. The assumptions for the evaluation

of defects capable of causing error bursts in a user's data stream (burst starts and burst

lengths) will be discussed. With this model It is possible to determine the optimal error management strategy to achieve any desired system performance. Including depth of

interleaving, error correcting capability and certification fencing for excessively long error bursts

JL. iuomary

A stat i St i ca I - theoret i ca I model for

based on the theory of Non- homogeneou

first step in the design of statistica

control end certification of productio

have been gathered frotr a variety of o assess the performance of several poss

available empirical data (7, 8). Part

development of the model. The purpose

Rate for five correction codes with av allow the Certification Fencing Limit

different levels of clustering.

a clustered error burst process has been designed

Compound Poisson Point Processes. This model is the

I control algorithms for the purpose of quality

n disks. Detailed empirical error burst-gap data

ptical disks. At this stage, we use the model to ibie EDAC strategies In conjunction with the

I (Discussion, Appendix) discusses the theoretical

here is to evaluate the Mean Corrected Byte Error

a i I ab I e empirical burst length distributions where we and the mean cluster burst frequency to vary for

Note Figure

empirical data,

a plot of the frequency distribution of error burst data of available

The data is presented on a double log axis. The y-axis is logiQ

BURST LENGTH DISTRIBUTION

CORRECTED BYTE ERROR RATE

(POISSON CASE)

BURST LENGTH

Figure 1 BITS

'5 15 25 35 45 55 65 75

CERTIFICATION FENCING LIMIT IN BYTES

Figure 2

SPIE Vol. 899 Optical Storage Technology and Applications ( J 988) / 93

32
probability from a base line of .00001 to 1. The x-axis Is log^g bits from 1 tc 3200.

The continuous curve Is a plot of one minus the cumulative distribution. Ninety percent ot all bursts are of size 11 bits or less, 99 percent are of size 58 bits or less. The

relatively large pulse at 448 is a sampling artifact. Although not readily evident from

this plot, the distribution is actually highly skewed toward burst sizes less than 10. The mean burst is 6.98 bits, the variance Is 1871.1 bits^.

Figure §2 shows the plot of contours for a Corrected Byte Error Rate equal to 10"'^

under the assumption that burst starts are a purely random phenomenon (Poisson Process) for

each of the five codes under consideration in this report:

(5,8)* code

(32.3) code

(32.4) code

(16,6) code

(8,8) code

The y-axis displays the mean frequency of burst starts per 1024 byte data b I ocl< from 10"-^ to 10*^. The X-axis is the Certification Fencing Limit in bytes from 5 to 75 and refers

to the maximum length of bursts that will be allowed on the disk after Certification.

Blocks which have bursts longer than the Certification Fencing Limit will be removed from

user accessibility In the Certification process. The heavy horizontal line at Am = .03484 is the mean frequency of Curst starts per 1024 byte data block as estimated from

available data

Notice that the (5,8) code requires a minimum Certification Fencing Limit of 22 bytes in order to achieve a 10"'^ byte error rate at the available estimate of mean burst start

frequency (Intersection point of the 1 &~ ' 2 contour line with the heavy horizontal line

at K^ = .03484) whereas the (32,3) code requires a minimum Certification Fencing Limit

of 38 bytes. This is due to the 6.4X greater interleaving of the (32,3) code as opposed to the (5,8) code; i.e., long bursts are chopped up into a greater number of independent code

blocks, thus the chance of a single burst affecting more than two to three bytes per code

block is considerably reduced. The Increased interleaving of the (32,3) code, however, is traded for a significantly lower code rate: 0.842 (ratio of user bytes to total bytes per

code word); because the (32,3) code has more than twice as many parity bytes per block as the (5,8) code which has a code rate of 0.927.

The other disadvantage of the (32,3) code evident from the plot is that it has a threshold in burst frequency at about A = .064 at which point no further decrease in Certification Fencing Limit will offer an improvement in Corrected Byte Error Rate. This

is due to the reduced error correcting capability of the (32,3) code. The random error correcting capability of these codes increases geometrically by the number of bytes the

code will correct. The (5,8) code will achieve a 10"'^ Corrected Byte Error Rate up to

0.9 mean burst start frequency provided we are willing to fence out all blocks with bursts

in excess of 5 bytes.

Figure /2 also displays the 10"'^ contour for three other codes which achieve various

combinations of interleaving and error correcting power between the above mentioned two. The (32,4) code with a code rate of 0.80 will require a minimum Certification Fencing Limit

of 57 bytes at the available estimate of mean burst frequency ( ^ m

~ .03484), but has a burst frequency threshold at A = 0.25. The (16,6) code with a code rate of 0.842 has a minimum Certification Fencing Limit of 48 bytes and a mean burst start frequency threshold

at A = 0.82. The (8,8) code has a minimum Certification Fencing Limit of 33 bytes and a mean burst start frequency threshold In excess of A = 1.0. Figures 3, 4, and 5 show the

progressive reduction of minimum Certification Fencing Limit and mean cluster burst start

frequency threshold in these five codes as we allow for greater degrees of clustering of

the bursts over the disk surface, k = 2, 5, 10 respectively. I.e., some data blocks have

higher rates of error bursts than the mean cluster burst start frequency specified, whereas

most blocks have logarithmically smaller rates of error burst starts than the mean. The

proportion of blocks with various levels of mean block burst start frequency are

* These are non-standard code designations which the authors have adopted to describe the

salient statistical features of these codes as they would be Implemented on a 1024 byte

block of user data. The first number. Is the degree of Interleaving, i.e., the extent

to which the codes are 'wrapped' Into the data block which is done to protect against

the possibility that a long burst could exceed the error correcting power of any one of

the codes. The second number is the error correcting power of the codes. For instance

the (5,8) designation means 5-fold interleaved, 8-byte correcting. For a Reed-Solomon

error correcting code this designation translates to a (n = k + 2*t, k = 1025/nL)

where nL = 5, the degree of Interleaving, and t = 8, the error correcting power of the

code (9). All of the codes considered here use the Reed-Solomon algorithm approach.

For the (5,8) code this designation corresponds to: ((n,k): (221,205)) for each of five

interleaved codewords (2).

/ SPIE Vol. 899 Optical Storage Technology andApplications (1988)

33
represented by a Gamma dIstrlbutJon where the reciprocal mean square to variance ratio Is given by k. I.e., increasing k signifies a greater degree of clustering. See Figure #6.

COfiRECTED BYTE ERROR RATE

K - 2.0

CERTinCATION FENCING LIMIT IN BYTES

Figure 3 CORRECTED BYTE ERROR RATE

(K - 10.0)

5 15 25 35 45 55 65 75

CERTIFICATION FENCING LIMIT IN BYTES

Figure 5

CORRECTED BYTE ERROR RATE

K - 5.0

CERTIFICATION FENCING LIMIT IN BYTES

Figure 4

GAMMA PROBABILITY FUNCTIONS

(K-1,2,5,10,20)

0.001 0.01 0.1 1 BLOCK MEAN BURST START FREQUENCY

Figure 6

SPIE Vol. 899 Optical Storage Technology andApplications (19881 / 95

34
Table I summarizes the minimum Certification Fencing Limit at the available estimate of mean frequency of burst starts (1) and the mean cluster burst start frequency threshold

(2), for the five codes considered:

Table 1 Code Po 1 sson Case k = 2 k == 5 k

== 10

( 1

)

(2) ( 1 ) (2) (1 ) (2) ( 1 ) (2)

(5,8) 22 .9 18 . 1 4 16 .067 7 .037

(32,3) 38 .064 .02 .012 .0071

(32,4) 57 .25 52 .06 5 5 .034 .021

(16,6) 48 .82 47 .16 41 .08 34 .045

(8,8) 33 1 .0 32 .20 28 .098 26 .055

We estimate that the clustering on our current disks is represented by k between 2 and

5. At these levels of clustering of the burst start frequency parametier, the (32,3) code

simply has insufficient burst correcting capability to achieve a 10"'^ Corrected Byte

Error Rate at the available estimate of mean burst start frequency for any level of

Certification Fencing Limit. The (32,4) code is marginally acceptable provided we assume

that production disks will In every case have a mean burst start frequency better than the

available estimate of mean burst start frequency = .03484. The (16,6) code and

the (8,8) code will achieve a minimum Certification Fencing Limit of 41 and 28

respectively. These two codes also maintain reasonable minimum Certification Fencing

Limits at mean burst start frequencies up to 2.6 times greater than the available estimate

of mean burst start frequency at clustering levels as high as that represented by k = 5.0.

The (5,8) code will achieve a threshold In mean cluster burst start frequency of A = .067

(1.9 times the available estimate of mean burst start frequency) but requires Certification

down to 16 bytes.

2-M CQ nc.l u s I o n 5 The two-stage Compound Polsson Point process model utilizes a burst-error density

data-set generated by empirical evaluation of prototypical Kodak Optical Disks. This model

Is a dynamic Instrument for the development and refinement of a high-reliability error- management strategy for Kodak Optical recording products and provides updated results and

conclusions as the data-set Is revised.

The system performance of our optical disk as perceived by the user after

coding/decoding through these EDAC strategies depends very much on the statistical

structure of the error burst process on production disks as modified by Certification

Fencing, i.e., the distribution of burst length and the cluster distribution of block mean

cluster burst start frequency. However, unless we can demonstrate that production disks

have a radically different burst length distribution or the block burst start frequency

parameter has a mean value above 1.5 times the available estimated mean frequency of burst

starts or Is clustered to an extent greater than that represented herein by k=2.0 I.e. a^^ = 2.0'\i^ then the following conclusions are warranted:

a. The (5,8) code will no doubt work. Note however, that it has a narrow Certification

Fencing Window requiring Certification possibly down to five bytes or less depending

on clustering level k and the mean cluster burst start frequency.

b. The (32,3) code does not have sufficient error correcting capability to handle disks

with a mean cluster burst start frequency equal to the available estimate of mean burst start frequency, where k, the clustering coefficient = 2.0.

c. The (32,4) code will work for disks with a mean cluster burst start frequency equal

to the available estimate of mean burst start frequency, where k, the clustering

coefficient = 2.0 and has a fencing limit of 52. It should be noted however, that

if the mean block burst start frequency Is on the order of 3-5 times that of the

available estimate or has a burst clustering greater than k = 2.0, the (32,4) code

will not have sufficient error correcting capability.

d. The (16,6) code recommended by the authors has a fencing limit of 47 at mean cluster burst start frequencies equal to the available estimate of mean burst start

frequency where k, the clustering coefficient, = 2.0. The code also exhibits a sufficient Certification Fencing Window at high levels of mean cluster burst start

frequency: 30 bytes at 3.8 times the available estimate of mean burst start

frequency (however the Certification Fencing Window falls off rapidly to 1 byte at mean frequencies above this to five times the available estimate of mean burst start

frequency).

96 / SPIE Vol. 899 Optical Storage Technology and Applications f 19881 35
e. A close second would be the (8,8) code which has a minimum Certification Fencing

Limit of 32 at mean cluster burst start frequencies equal to the available estimate

of mean burst start frequency where k, the clustering coefficient, = 2.0. It also

has the advantage of protecting against higher mean block burst start frequencies up to 5.7 times the available estimate of mean burst start frequency. Notice that the

Certification Fencing Window Is greater than that of the (5,8) code but less than

that of the (16,6) code.

The above conclusions are based on the available estimate of burst length distribution

and mean frequency of burst starts and under the assumption that the degree of clustering

of the block mean burst start frequency is adequately represented by a Gamma distribution

for which the reciprocal mean square to variance ratio = 2.0; however, they do not

necessarily represent what may be produced on a daily basis. Accordingly it will be

necessary to gather additional data from the Certification development experiments and from

the spir-stand test equipment that represent the actual manufacturing process charactei

Istics in order to evaluate the ultimate code performance.

if production disks are significantly better than indicated by the preliminary data (the

disks measured in (7, 8) were net produced under strict quality control procedures), any of

these EDAC strategies may work satisfactorily; the Issue then would focus on how much

interleaving of the code block should be done to insure a suitably large minimum

Certification Fencing Window, i.e., a tradeoff in cost of Certification vs. percentage of

disk committed to parity bytes. It should be noted, however, that there are other sources

of error that must be handled by the EDAC system. These may Include future contamination

or degradation to the disk. Because of these one may need to provide additional margin In Uncorrected Byte Error Rate to cover for a possible two-fold Increase in mean cluster burst

frequency and possible increase in burst length to insure meeting end of life quality

specifications. This capability could be achieved with tighter Certification Fencing

Limits, higher Interleaved codes or increased power in each code.

3.1. Theory The error model used for these analyses is a 2-stage Non-homogeneous Poisson

•Point Process model, essentially the error burst process embedded In a Poisson process (see

10). To follow this development, we need first to distinguish between a shower of defects

on the disk surface, presumably measured by reflecting laser light off the surface, and a measurable burst of errors in a user's data stream wherein there is at least one bit in error for every byte Interval cf user's data. Under this scenario. It would be possible,

for instance, to have two defect showers which overlap resulting in one long burst of errors In a user's data stream. This model conceptually differs from the 2-stage burst-gap

renewal model ciscussed in (11). It is difficult to assess the occasion of overlap in the

defect showers from detected errors in a user's data stream alone, because an overlap of

two defect showers would cause a succession of error bursts which would be counted as one

long error burst under reasonable measurement and estimation procedures. However, there

are Important reasons for dealing with the error burst problem in this manner in spite of

the difficulties In measurement:

3.1.1. A model built around the defect shower problem as opposed to the error burst

problem is in principle capable of 'bridging the conceptual gap' between quality control of

materials and the manufacturing process, to certification of finished disks, to optimal

EDAC strategy and to user perceived performance over the lifetime of the product

( I ongev i ty )

3.1.2. The parameters of such a model are statistically I nterpr etab I e ; i.e., confidence

limits can be placed around key parameter estimates and Bayesian sampling statistics, for

instance, can be Incorporated with stated risks.

3.1.3. The explication of the model is mathematically and computationally tractable; i.e.,

the mathematics and computations can be done with a minimum of simpi ifying assumptions.

The one or two assumptions are at a conceptual level rather than at an algor"ithmic level so we can readily ascertain their maximal effect.

The development here proceeds with the (5,8) code interleaving structure and parity byte

structure but these results are easily generalized to the other codes. In the 5-fold

Interleaved 8-byte correcting code, there are 5 interleaved code blocks, with 5 bytes = 40

bits per Interleaved set. The t = 8 byte correcting code will detect and correct 8 bytes

In error in each of the 5 interleaved code blocks. For a Reed-Solomon code, there are 2't bytes (16 bytes for the (5,8) code) of parity for each code block (3). A 1025 byte

data block divided by 5 = 205 bytes in each user code block. (A user data block is of size

SPIE Vol. 899 Optical Storage Technology and Applications (1968 97

36
1024 bytes; the 1025th byte Is made blank). 205 bytes per user block plus 16 bytes of

parity = 221 bytes for each of 5 code blocks 1105 bytes total. The rate for the code

0.927. error code block #

12 3 4

J \.

1 -L-J \ J ^L__J Li__l.

1 2 3

._-J i—1-

4

Interleaved set § Figure 7 Figure 7 shows the first three and one-half Interleaved sets of a typical data block set up for the (5,8) code. The bytes are each numbered according to the error code block to which

they belong: 1-5. We also show a typical defect shower which starts at x = 1 9 bits and is capable of causing an error burst (at least one defective bit In every user byte) to bit

number 98. The defect shower Is z - x + 1 =80 bits long.

Suppose we pick one of these 221 Interleaved sets, say the first, and assume that a single shower of physical defects starts 'randomly' In this Intj^rval distributed Polsson

with a parameter equal to the mean burst start frequency = .03484 per 1024

byte block (A per byte = .000034023, XT = .000017013 per 5-byte interleaved set, T = 5). The arrival of a single shower of physical defects In an Interleaved set of T = 5 bytes has a probability of XTe" = .0001701 0. Now, J_f no other defect showers arrive

on this Interleaved set (an event which has probability 1.447 » 10"^), we can show (by

Integrating the exponential distribution) that the marginal distribution of the start time

(t) of the shower in C0,T3 is uniform over CO,T]]. Note: We require that the second event

together with all the ensuing events arrive after T.

-Xx^ -X(x--x^)

Ae ^Xe 11

0 T

dx2dx^

(1)

If we permit more than one shower to start in this interval of T = 5 byte Interleaved set,

then we are immediately faced with our first conceptual difficulty in the development of

this model for which an assumption greatly simplifies further progress. The essential

question addressed in its simplest form Is: If two defect showers arrive In the first

interleaved set, which one came first? An application of the theory of Order Statistics In this context will show that the first defect shower out of two has a starting time

distribution of

'1

0 and the 2nd defect shower has a starting time distribution of;

(2)

(3)

These may be seen as simple straight line functions over [0,T]i

2

98 / SPIE Vol. 899 Optica/ Storage Technology and Applications (1988)

37
In general. If we have n defect showers, starting In the 1st Interleaved set ZO,ll, the

starting time of the J + f' one Is Beta distributed (See (4)):

All of this Is necessary If we are to concern ourselves with overlap of the defect

showers which might start within our interleaved set. If we choose to ignore the Issue of

defect shower overlap by making the assumption of uniformly distributed defect shower start

times, we Ir effect permit our estimates to be biased by double counting which for mean shower sta'rt frequencies on the order of A = .000170 happens infrequently,*

Assume a single shower of defects starts at location x somewhere in the first

interleaving set and proceeds along with sufficient density to involve at least one bit in every user byte and then stops z - x + 1 bits later (see Figure #7). Then the conditional

probability density of such a defect shower capable of causing an error burst of length z - X + 1 bits given that there is exactly one shower starting somewhere in the first

Interleaved set is:

where f^ii represents the distribution of defect shower lengths as measured by available

burst length statistics (7).

The event that the shower misses the first and second byte of the first error code

block, (I.e., It must miss byte #1 and #6 completely) requires that It starts after bit #8

and stops on or before bit /40. The probability of this event is the sum of the Joint

probability of all possible events (subject to the constraints) of defect showers starting

at bit X and ending at bit z of length z - x + 1.

, 40 z 91(0) -io I I ^„(z-^^l)

^ z=9 x=9 for the available burst length distributional data fenced to 18 bytes, gi(0) has a value

of gi(0) = 0.72861 see Table II. This can be rewritten as:

, 40 z , 40 8

The first term is the probability that the burst stops after bit #8 and is contained within

the first interleaved set which has a probability of .77259 and the latter term is the

probability that the shower starts In the 1st byte of the error code block and stops

between bit /9 and bit /40 (.04398).

In general, we can show that If the burst starts in the first interleaved set, the

probability of it involving J of the bytes in the m'*''^ code block, m = 1, 2, 3, 4 or 5 is;

(nL = number of bytes in an interleaved set, i.e. nL = 5)

, (j«nL+m-l)'8 min(z,8-m)

"1 "L-S z=((j.i).nL+m-l)-8+l x=l

"

We believe that the overlap probability Is small (approximately one percent) due to the

relatively low frequency of burst starts and in the case of evaluating the EDAC

strategies it is Insignificant. Presumably in Certification of the discs, the ordering

of defect shower start times and possible overlap will be important. The next phase of

this development will be to challenge the assumption that order of arrival and possible

overlap Is unimportant.

SPIE Vol. 899 Optical Storage Technology and Applications (1 988) / 99

38
, ((j+l)«nL-Hii-l)-8 nL-8 "-^ z=(j-nL-Hii-l)-8+l x=l "

1 nL-8

(( j+1) 'nL+fn-l) -8 min(z,8«m)

I I fM-KH) (8)

z=(j-nL+ffl-l)-8+l x=l "

This expression Includes all defect showers which cover the m'*^'^ byte of the 1st

Interleaved set and those which do not cover this byte but cover an additional m^^ byte

In the (J + 1)st interleaved set down the block.

The first term Includes all events where the m'''^ byte of the first Interleaved set Is covered by the defect shower and J - 1 m"*^*^ bytes of the J - 1 succeeding Interleaved sets

are covered by the same defect shower. The second term Includes all events where the J

m"*"*^ bytes of the J succeeding Interleaved sets are covered by the same defect shower.

The third term subtracts out of the second term the events which cover the m"*^^ byte of

the first Interleaved set.

To compute these sums, we merely need to change variable to y = z - x + 1 and col lect

all factors of common terms being careful to respect the summation limits which also

undergo a change of variable. Table II lists gm(J^ where m = 1 and for the burst length

variable fenced to 18 bytes for the 5-fold Interleaved code. Notice that since the bursts

are fenced to 18 bytes the first error code block cannot be affected by this one burst by more than four bytes. I.e., the probability of gj(j) = 0, for j > 5. Ignoring the Issue of defect shower overlap (essentially double counting the overlap)

we can now show that If the defect showers arrive as a clustered Polsson process over the

data block where X , the block frequency of defect shower arrivals is chosen from a Gamma

distribution, the probability of J bytes of error code block m being affected over the

entire 1105 byte block Is:

n=l

CO n+a-1 -A(T-h|)

' ^ ^ d.\ (9)

a , S are parameters of the Gamma distribution

T Is the length of the block: 1105 bytes In the case of the (5,8) code.

The expression under the Integral sign permits us to deal with the clustering aspects of

the defect shower frequency variable. We assume that the defect shower frequency for each

block is distributed according to a Gamma distribution. The integral expression is recognized as Z(j"\-T), the n"*"" derivative of the moment generating function of a Gamma

distribution. g"*(J) Is the n^^" fold convolution of the gm(j) distribution derived

In equation (6)

.

Computing this expression requires that we prove three lemmas which will permit us to generate recursively the h{j) (see Appendix). h(J) Is listed in Table III along with

j Log]Q(1.0 -

\ h(J)) In the case where gmCJ) 's fenced to 18 bytes and where the block

i = l defect shower freque^ncy parameter A has a mean u. equal to the available mean burst start

frequency estimate Am = .03484 and a variance of = (2 • My^^ ) = .0024279.

The Expected Value of the Corrected Byte Error Rate (per byte of user Jata) Is computed

under the following assumption:

When the number of byte errors (n) In the code word exceeds the byte correcting

capability (t) of the error correcting code. I.e., n >, t, the resulting number of

Incorrect bytes after decoding will be twice the byte correcting capability (2 • t) plus

the number of byte errors In excess of the byte correcting capability of the code (n - t)

which yields (t + n) bytes (6).

This result occurs for n 2 t because the Error Correcting Code syndrome calculator Is forced to solve t-dependent equations for n variables which results In erroneous

100 / SPIE Vol. 899 Optical Storage Technology andApplications (1988)

39
solutions. The n + t resulting by+c errors Is the maximum value for miscorrect Ion; yet,

the probability of maximum m i scorrect Ion is relatively high given the total number of bytes

to correct in a user data block.

This discussion has developed a two-stage compound Poisson process model for a defect

burst error process for a set of nL independent Interleaved Reed-Solomon code blocks

divided equally into a 1024 byte data block where the mean frequency of burst starts for

each data block can be considered as an independent sample from a Gamma burst start

frequency clustering distribution. We have provided three lemmas (see Appendix) necessary

to compute the system performance probability distribution. Certain expectations on this

probability function, i.e., the expected number of errors per block, the mean corrected

byte error' rate (per byte of user data) and the mean ECC processing rate are readily

calculable from these algorithms.

Table II Probab i I I I I ty of N bytes of error code block #1 becoming affected by a defect shower which

starts in the first Interleaved set. The shower is distributed as an empirical burst

length variable fenced to 18 bytes. The code used is 5-fold interleaved, 8-byte correcting.

N PROB(N)

0 0.7286147773721488979339420535

1 0.2649306530624980491300045521

2 0.0052848762831913630579725813

3 0.0009641486473836936936719594

4 0.0002055446347779961844088538

5 0. 000 0000000000000000000000000*

6 0.0000000000000000000000000000

7 0.0000000000000000000000000000

8 0.0000000000000000000000000000

* Note that the fencing I Imit does not al low bursts which would yield an N > 5.

?

APP£NfiiZ

1. The n'''^' derivative of the moment generating function of the Gamma distribution can be

generated recursively as:

Z(n)(-T) - z(;-^)(-T) . i^^J^ (10)

2. By redefining our gm(J)# J = 0, 1, 2, 3 ... function on the set of positive

i ntegers

gmU) = gm(J)/(1-g(0)) J = 1, 2, 3 ... (11)

Then we can show that (see 5)

a. h(o) = Z^°^-T(l-g (0))) =

^ j = 0

•^A (l + T(l-g^(o)).6)''

(12)

j (T(l-g^(o)))"

b. h(j) =

I (J) Z ^-T(l-g^(o))) j f 0 n= 1 A

SPtE Vo/. 899 Optical Storage Technology and Applications (1988) / 101

40
TabI e I I t Probability of N bytes In error code block #1 becoming affected by multiple defect showers

which might arrive In a 1105 byte block of data plus parity. Each shower Is distributed

independently as an empirical burst length variable fenced to 18 bytes. The shower

frequency parameter lambda Is distributed gamma with mean = .0348416 and variance = .0024279 i.e. (K=2.0). The code used Is 5-fold Interleaved, 8-byte correcting.

N PROB(N) -CUM(PROB( N) )

nU 0 .9899501 16306 044674 9378004194031 4661 -1 .998

I 0 .0096 63457 81 1 0628506429306 59377825 99 -3 ,413

0 .00033426 3741 446 5795271845764347 57 7 8 -4 .283

t,J 0 .0000431 1 491 31 398444 023427 26083 5 9284 -5 .043

4 0 . 00000876 05972787284598047 46791 51 1 95 -6 .543

5 0 .00000027 181 0894906621 7 0544772931 222 -7 .829

6 0 . 00000001 338039851 1 1 42225 5 067292381 4 -8 .842

7 0 .00000000128882 82 03281 57 349391 5 1 05 45 -9 .821

8 0 .000 0000001 43361 7 5737891 5067 071006 74 1 1 . 1 22

9 0 . 000000000007 083944241 901 026167957 98 1 2 .337

1 0 0 . 0000000000004216 067 31 568967891 4083 4 1 3 .416

1 1 0 ,0000000000000351341 1 029005161853348 1 4 .487

1 o 0 .00000000000000305782532566343318285 1 5 ,697

1 J 0 .0000000000000001 87 323271 0591861 1 22 0 16 ,874

1 L 0 .00000000000000001 23348725 42327891 04 1 7 ,984

11 ^J 0 .0000000000000000009576246856101 6906 1 9. 093

1 f\ 1 o 0 .00000000000000000007 52035901 046 491

20 .263

1 7 0 . 00000000000000000000507 86 49083 09931 21 ,41 9

1 R 0 .0000000000000000000003525 02079896 53 22 .543

1 Q 0 . 00000000000000000000002647 41 4723923 23 .669

z u 0 .0 00000000000000000000001 99331 697798 24 .820

0 . 0000000000000000000000001 406641 8431 25 .965

22 0 .00000000000000000000000001 0031 4431

27 .096

23 0 .0000000000000000000000000007425 0636 28 .227

24 0 . 00000000000000000000000000005498398 29 ,370

25 0 ,0000000000000000000000000000039597 3 30 .509

26 0 .000000000000000000000000000000286 38 31 ,629

27 0 .000000000000000000000000000000021 08 32 .601

28 0 000000000000000000000000000000001 5 5 33 ,016

29 0 .0000000000000000000000000000000001

33 .016

30 0 00000000000000000000000000000000001 33 .016

31 0 00000000000000000000000000000000000 33 .316

Corrected Byte error rate (loglO) = - 12.202

3. This form of our expression now allows us to compute h(j) recursively using Brown's

adaptation (1982) of Adelson's routine (1966) for recursive generation of Compound

Poisson Probabilities under the case of Gamma variation In the Polsson Parameter (see

4, 1 ) .

let C

Then

T-(l-g^(o))-6

l+3T(l-g^(o))

h(0) = (1+61(1-9^(0)))"

h(l) = C-a-g|^(l)-h(o)

W2 / SPIE Vol. 899 Optical Storage Technology andApplications 11988)

41
h(2) = C-(1+-^1J-). L(l)-h(l) + Ca-a (2).h(o)

h(j) = c.{l+i^).g_^(l).h(j-l) + C.(l + 2.-5^^).g^(2).h(j-2)

+ ^ C.(H-^i^lUi^^).g^(j-l)h(l)

+ C-a.g^{j)-h(o) (13)

4. eib l lography

1. R. M. Adetson, "Compound Polsson Distributions," Operations Research Quarterly . Vol. 17 pp. 73-75, (March 1966).

2. E. R. Berlekamp, "Final Report on VLSI Plans for 25 mips/channel EDAC, Expandable for

up to 5 Channels," Report from Cyclotomlc's Inc., unpublished.

3. E. R. Berlekamp, "Algebraic Coding Theory - Revised 1984 Edition," Aeglan Park Press,

Cal Ifornia, 1 984.

4. W. E. Brown, "Low Order Rate Inventory Control," Paper presented to TIMS Conference,

Detroit, Ml, April 1982.

5. W. E. Brown, "Notes on a Revision of Adelson's Routine for Computing Compound Polsson

Probabilities for Functions Defined on the Set of Non-negative Integers," unpublished.

6. A. Merchant, "Preliminary Analysis of PRO-200 Error Rates," Letter to J. Wheeler,

unpub I I shed

7. M. Melchle, "Notes," Regarding measured burst gap data on PRO-200 discs, unpublished.

8. M. Melchle, "Preliminary," Regarding measured burst and gap data for PRO 200 discs,

unpubl ished.

9. W. W. Peterson and E. J. Weldon, Jr., "Error Correcting Codes - Second Edition," MIT

Press, Massachusetts, 1972.

10. M. Thomas, "A Generalization of Poisson's Binomial Limit for Use In Ecology,"

Blometr lea , pg. 18, (1948).

11. E. J. Weldon, Jr., "Selection and implementation of an Interleaved Reed-Solomon Code

for Use with an Optical Disk," unpublished.

42

SPIE Vol. 899 Optical Storage Technology andApplications (1988) / 103
Error Measurement and

Recovery Capabilities on

Cirrus Logic

Optical Disk Controller

Chip-set

Nicos Syrimis

Cirrus Logic, Inc

Tel: 415-226 2153 Fax: 415-226 2170

rCIRXUS LOGIC

Agenda

1. Optical Disk Controller Chip-set Overview

2. Error Recovery/Reporting Capabilities

3. Defect Management

4. Software Reporting Schemes

EnorMMMnoMnlNS ItovlJ) ar2«1-2 ZIRXUSLOGIC

43
2. Error Recovery/Reporting Capabilities

3. Defect i\/lanagement

4. Software Reporting Schemes

CHUlUSLOGfC

Optical Disl< Controller - Summary

• High integration - two cliip soiutlon

CL-Siyi330 Optlcai Disit ENDEC/ECC

• ENDEC/Forniatter

- EGG

CL-SM331 SCSI Opticai Disk Gontroiler

- SCSI interfaca

- Buffer manager

- Disl( formatter

• 100 pin QFP pacl(aglng

• CMOS tectinoiogy

CIRRUS LOCK

44
Optical Disk Controller - System Block Diagram

Micro proCMMr ROM

(8018a) RAM

SCSI

Intartac*

sea Control

£

AddroM/Data E

CL-3U331

Butlar' Data .

Diak Control

NRZData

. Serial Vactors

Status

Ct-9M330 RLL Intarfaca

Command/Btatua

Buffar Addiaaa

Buffar RAM

CIRJiUSLOGIC

Optical Disk Controller implementation

SM330/331 controller controls the datapath between host and drh^e

- Data transfers between host and drive, via buffer

- Error tolerance and recovery on header and data

High level control performed by software In \iC

- Command decode and Interpretation

• Logical to physical address translation

- Defect management (sector slipping, replacement, retirement)

- Error recovery control

Error measurement/logging

- Under software control In |j.C

- Use ha'rdware logging/recovery capabilities of SM330/331

nlNSIto«14 ara«i-« CIRXUSLOCIC

45
1. Optical Disk Controlier Chip-set Overview

3. Defect l\/lanagement

4. Software Reporting Schemes

IrrBrlHiiiiiiBinlN»Wir1J mvtvt CmUSWGK

>r Format 90mm. 512 Bvtes/sector

TRACK No ID No SECTOR No CRC

SM VF01 AM ID1 VF02 AM ID2 VF02 AM 103 PA OOF - HEADER (S2 BytM)

GAP VFOslsYNC DATA FIELD (DATA. ECC. CRC, VU, RESYNC) PA BUF

RECOHDINO RELD (872 BytM) --TM—-— «>

Error llMHraiMntNSRMljO «ai9M CIRRUS LOCK

46
Error Tolerance on Optical Format

Elfild SM

(Sector Mark)

IDs

SYNC

ReSync

Error Toleranca

Redundant Pattern

Redundant (triple)

Redundant Pattern

(12 nibbles)

N/A

SM330/331 Capabllttv

Voting on Marks/Spaces

Asymmetry/Relaxation control

Pseudo Sector Mark generation

Majority voting

ID skip

Programmable voting threshold

Automatic sync on 1st/2nd ReSync

Recovery: sync on 3rd/4th ReSync

Resynchronize to ReSync

Automatic window control

Protection on false ReSync detect

Missing ReSync threshold

rciRRUSLOGK

Error Tolerance on Optical Format - ECC

Elfiid Error Tolerance

Data 51 2 Byte Sectors

Field 5 ECC Interleaves

(max corr. 5x8=40 bytes)

1024 Byte Sectors

10 ECC Interleaves

(max corr. 10x8=80 bytes)

SM330/331 Caoabllltv

On-the-fly error correction

Extended recovery: 16 bytes/IV

(max corr. 5x16=60 bytes)

On-the-fly error correction

Extended recovery: 16 bytes/IV

(max corr. 10x16=160 bytes)

URRUSWaC

47
Error Reporting Capability

• SM330/331 controller reports error conditions for logging

- SM not found (Pseudo Sector Mark Generated)

- ID voting threshold not met - details of bad AM/CRC per ID

- Data SYNC error

• ReSync threshold exceeded

- Error/Interleave ECC correction threshold exceeded

- Total errors/sector ECC correction threshold exceeded

• SM330/331 ECC error monitoring capability - use for error log

- Interrupt generated at the end of every sector read/corrected

- Report available on:

Total errors/sector detected

Maximum errors/Interleave detected

1. Optical Disk Controller Chip-set Overview

2. Error Recovery/Reporting Capabilities

4. Software Reporting Schemes

CIRX USLOGIC

48
Defect Management

• Defect management defined by ANSI/ISO/ECMA standards

• Two methods are available

1. Sector Slipping

Media needs certification Optional

Primary Defect List (P-List - PDL)

2. Linear Replacement

Compulsory

Grown Defect List (G-Llst - SDL)

• Defect management algorithms performed by software In ^C

SM330/331 controller supports efficient sector slipping operation

1. Optical Disk Controiier Chip-set Overview

2. Error Recovery/Reporting Capabilities

3. Defect l\/lanagement

4i, S®l?tiwsiar(& lFl©|)@QrtiacRiD i(§[ni@iEn@©

rciRRUS LOGIC

mrMMMraiBMitNSNavlJ) VM\^^* ZIRRUS LOGIC

49
SCSI-2 Error Loqaina/Measurement Capabilities

• Read/Write Error Recovery Parameters

- Allows setting of error thresholds, correction spans, etc

• Read Defect Data

- Allows reading of P-Llst (PDL) and G-Ust (SDL)

• Mode Sense Command

- Read Error Log (Standard Header, Vendor Unique data)

- Buffer overflow/underflow condition

- Error counter information

- Last "n" error events page

- Non-media error page

- Vendor Specific information

tircrlliiii»iMiiitNl>lwr1J iratVIS ^^^^^IRRUSLOCK

50
Optical Disk System Error Detection/Correction:

A Records Management Perspective

August 5, 1991

Charles Obermeyer II

Office of Archival Research and Evaluation

National Archives and Records Administration

Washington, DC 20408

Error detection and correction mechanisms are designed to provide

data integrity so that information stored on media can be retrieved

with a 100% confidence level. Mechanisms used today are very good

at achieving this within the limits of error correctability . As a consequence, users are lulled into a false sense of security

regarding the permanence and retrievability of their data.

A weakness in the system is that errors can be introduced later as

a function of system operation and storage environment. Some of

these errors are temporary because they are caused by components in

need of maintenance. For instance, disk drives have moving parts

that wear out, optical lasers drift out of calibration, and

atmospheric contaminants can cause media read errors. Other errors

are permanent such as those which cause destructive deterioration

of the media. For instance, environmental conditions can cause

ageing artifacts to appear in the media which destroy the

readability of the disk in isolated regions. There is a point of no

return from which the optical information storage and retrieval

system can no longer correct all errors that were detected for a particular block of data. Current systems generally inform the

user "after-the-fact" that the point of no return has been reached.

This is equivalent to a doctor telling you on your death bed that

you are going to die. The information is explanatory but it doesn't

really help the situation.

Records managers and archivists in both the federal and private

sectors are concerned about the integrity of their optical disk

systems and the permanence of the data written onto media by these

systems. It is important for operations managers to have early

warning signs which indicate that equipment is behaving in an

abnormal way even if the equipment is taking care of errors that

are being detected. Early problem detection in operational systems

allows corrective action to be taken during non-production hours.

51
A similar situation exists with optical media. The system that

writes and reads optical media takes care of correcting errors that

are detected. However, all media are subject to degradation over

time. Records managers and archivists must be able to determine

the permanence of data recorded on optical media as a function of

time in order to detect contamination or degradation.

Contamination can often be corrected by cleaning the disk and

determining/eliminating the source of the contamination product.

Degradation over time can be corrected by a copy operation to a new

media. The copy operation must be done before the media reaches

the point of non-correctability

Clearly, analytical tools are needed to provide managers and

archivists with early warning mechanisms for detection of problem

areas in both optical drive subsystems and in optical media. Such

tools could be used to create profiles which could serve as a

device or media specific "health" record. It is even more important

that these tools be standardized in terms of the information

captured so that reporting utilities can have the flexibility

needed to provide useful as well as informative reports.

Such tools require at least two components. First, the optical

storage and retrieval subsystem must be able to capture, store, and

make available to the I/O bus standardized information relating to

errors detected in its own subsystem and errors detected from the

media. Second, a set of software utilities should be developed to

capture and report this information in a standardized way so that

meaningful subsystem and media profiles can be established.

Industry, the user community, and the government can work together

to define and develop the analytical tools needed and to stimulate

the creation of utilities that will provide both short and long

term benefit to the optical disk user and provider communities.

Systems operations will clearly benefit from the tools that provide

early warning of systems failures. The concept of "permanence of

records" can be strengthened by utilities that can monitor the

"health" of deployed optical media. These tools can strengthen the

acceptance of optical disk technology as a permanent records

management methodology.

We need to take a leading role in this area if optical disk

technology is to become the permanent electronic record's tool of

choice. Right now, a small but crucial part of the optical disk

technology highway is in the dark. The darJcness is affecting the

comfort factor of the user community and could become a drawback to

the wider acceptance and use of the technology if not corrected.

The definition and development of optical subsystem and optical

media analytical tools will help illuminate the path of progress

for optical technology.

52

rco

"5o<Eow>

C/) oQO

COc

Ex GO

Q<

54

rEo

0.

LU

CD< o

CO

Q<

56
rooo0oc

COE0

Q.

Q-oEo

_a>3OoD

CO3

0)

*+— m o

0)

4-* >

00oDC

0)c

(0

"(5

0)

CD

N

T3 c z: m CQ

COo

CD

CO3^ O

CD 0) oo

To

CO C4

"5) ^ 0 o

T3C CD CM

CO

CO ^5 3 o o E

H~ ~"

*5 < 0)

CQ

CO3o

CDc

CD

0)c

CD

CD ^ C O CD O 3

c.ti

CD CO E c ^ o

CD CDo

CD 1. O <i> C CD

CO

CDO

T3

CO

CDO

"55 o • CD ^^ CD

LU ^

CO DCoo

CD

COooEo

CO

"Ocoo

COooEo o

</) CO

CD3

JQCo

CO

CD

O COc

CQ

O ^

CD

0>

CO

CD

CDo

CO

T3

CD

MB3

o0)

O

CDoo c

0) CD

CQ

0) o

0)3E3Oo

CDO

*- CD

CD ''^

(0 >i

_o <o w «

(0 o 2

(D CO

coo

CD CO

Q)

gos >- SB

D<

57
rcoo

0)oo

LUO

CO

CO

CO

0)Ec

mwam oO

Q)OO

O4-* o>coo

0)oo

(0

0)

OS

0)c

D)

(0 DCoo

a>

(0oo

Q.

LUm<

en

m

.at:

(0 o >o

Q.2C £ o E S w E

CO

O

*- o

CO o

§^

CO0 Q-^

.Si

ET3 ^

CO ^ oo 1 CO ^

CO

0)

0)>o

COoo

0)

CO

Q.

COc

CO

COooO +

CO

"D

CO^

COo

CO3E3

COcooc

CO

COq n o

E:t=

|i^

CO

0) <

58
roo

0)ooo

ui

0)

c5

CO ooo

a:

LU

CD<

<DE

COo

CO

CO

0)coCou

0)ooo

LU

COEo

a.coEo

"D

CD

CD oo o 8 O LU

LU+

CO

CO

CO

<])

0)N

CO o X <l> o

COcE

Q.E

LU +

CO

CO

2^

CO

<D

in In

O «^

CO

M <N o *-

*-* 00

00 CM

CM TCOC>

CD ^ < i

0)N

CO » CO u.o

0)

COc>

CO

CO 0)

0)

0)c

>»

COI

CO

CO0oo

CL

0)o- CO

<D

<D

—

CO CO

0) 0)

CO

X

Ei> H

CO £>

CO d)

CO o < o

COo

CO

CD

0)

0)

COEE

COoh

t: 0)

CD

C

— >» O c

75 "o C CD

.2-g o CO

CD

CO

CO

T3

0)c

CDE

Q>

a.Eco

COo0>c

COc

mmm o

COc

0)oOo

>»

CO <

59

Error Handling and Reporting

In The

Kodak 6800 Optical Recording System

By Scott A. Gerger

Small Format Optical Media Development Group

Eastman Kodak Company

Rochester, New York

5 August 1991

61
... or

"What in the world is a

'184', '185', '216', or '204'?"

62
Outline:

L Problem Statement

II. Decision ofManagement and Engineering

III. System 6800 Error Reporting

IV. EDAC System Messages

V. Behind the EDAC System Messages

VI. Questions?

63
Problem Statement:

Given that errors occur during the operation

of any complex opto-mecha-tronic system^ how

much information regarding errors should be

reported to the host system ?

64
Decision ofManagement and Engineering

''All of it!''

The Kodak 6800 System should make available

to the host as much information regarding the

operation of the drive as possible .

65
System 6800 Error Reporting

^^^There are 210 message types reported via

the SCSI Interface.

^^^There are multiple error logs for

examining the recent history andfrequency

of errors which have occurred.

^^^There is a special error logfor monitoring

the focus and tracking processor.

66
Error reporting in the System 6800

Errors are grouped in the following manner:

* 128 Hardware Errors

* 40 Media Errors

* 10 Unit Attention Errors

(strictly speaking not errors)

* 5 Aborted Command Errors

* 7 Illegal Request Errors

* 20 Miscellaneous Messages

67
EDAC System

Certification a key step in data integrity

assurance

Reed-Solomon Coding Scheme

Interleaving to level 5 with 16 bytes of

redundant check bytes per subcode

(a total of 80 bytes per 1024 byte sector)

68
o

I

O

o

o

9<<fcocaOH

69
EDAC System Messages

Ofthe 40 Media Error Messages^ 4 are related

to EDAC system operation.

^^204 Marginal Sector Detected

^^216 Retry Attempted

"^"^ 184 EDAC Pause

185 EDAC Uncorrectable

70
Behind the EDAC System Messages

Question: Is there a way to monitor the

activity of the EDAC system prior to

receiving the 184, 185y 204, and 216

messages?

Answer: Yes

71
Behind the EDAC System Messages

Question: Is there a way to monitor the

activity of the EDAC system prior to

receiving the 184, 185, 204, and 216 error

messages?

Answer: No

72
Standard Answers:

1. LeVs check with Steve about that...

(Steve Pope^ Kodak Berkley Research)

2. Give me your card and Fll [get\ back to

you on that one.

3. You really need to talk to Dan Nelson^

Optical Drive Software Engineering, about

that (716'588'0662).

4. Fm sure that Fred Rakvica, Optical Drive

EDAC Systems Engineering, can answer

your question (716'588'0590).

73

•St Q

O

-J

O

liJ Z K- H Z

UJ Z > H UJ Z CO H O o

Q.

UJ

UJ<oH

COoHH

UJzo<

UJ< <oz<

Ofo

LU

75
76
lU

u

SH

UJ

(0

0.

UJ

00<

Q.

LU

LUOH

O

O 0^

UJ >

I-M < < LU ^ HH LJ- < </> Li. " O D

UJ 00 CO

LUO

LU

I-O

LU

0^

OS, oo

01

LU

H

LU

I-<U

-I< <o<oM>a

LU

hUJ3O

LU

01o

LU

LU

CO o z

01

H 0 X

01 <

01 s:

01

H o (0

01 01 (/) o 01 o

01 LU 01

01

01

LU ^ o

LU < <

LU LU 01

00

01

LU LU

01

H 3 LJ. H <

O.< a O LU

-I ^ H O < < Ll.

01

H- ^

LU

O LU Z 00

01o

01 oo

LU

LU i i

CO 6

>-

Li. M

01

LU

M

01

CM CO

77
(/)

o o

u o 1-

ct cc <

UJ LU H

UJ o CO

H O O z ct <

liJ b4 H UJ O liJ u. 3 Z O 0^0^

liJ O H O! O < O Q. H- Z UJ

UJ H (/) O liJ UJ

^(/) UJ

20^HHZ<O< ^ujtto^>-ttoa

UJ

H

H


o

liJ z

a:o

liJ ao<

lU

liJ <I

lUozH

CO

i/l Q

bJ

bJO

O

X

liJ bl o c/) HM O z CC o Ct

IS bJ

b.

liJ

O >

liJ o —J bl

liJ GQ u z o bJ 3 H Z H O o z CLm o H o

bJ z z (0

i/l

M bJ bl

1- 01

O h- z

bJ

H

b. o tt

Ill III O H bl 01 O 01 < bJ > M -1 < o Q b. o

111 bJ < bl

01 • f-H • CM • CO
UJ

a

UJ

I00oH

I-<

UJ

o.<

UJ oH

I-<oo<oH

in>X

UJ

UJ ^

UJ o ^

UJ

UJ

CO

Ul o

DC

0^

H

O

X<

UJ ozH O (/)

0^ (/) O

UJ ^ o o < <

UJ DC

DC

UJ UJ

DC

H S U. H < a< o O UJ ^ H u < < u.

DC

H- ^

UJ

O UJ Z CO

0^o

DC

DC

DCO

DC

DC

LU<< HO

UJ

u. o

UJ DC O DC

UJ o> o ^ 2 2 UJ >- 5 00 <

UJ

DCo

-I

DC DC tn

JJ o <

Otto Z LU UJ H DC ^

i/l DC

O 3 O < O O

tt

O Z

UJ

O

IOK

u

U.z<

tt

tto

tt DC

DC

O

tt

tt

>- UJ HM >

DC

H <

a. DC<

DC Q.

UJ

U. hH

U. (/) 3 O

OQ CO

UJ>M

tt

CVJ CO

82
oHHO >

II-ooHHOUoo

UJaao

bb III

lb Ui

lib III Ha

ft

v#

#yu* llj

111 L.

CO

u Ui III ft ft

/ybA bl a III oa

III III

w

w

u bl

Hi 1 1

MB iyUi ^»

u ft 1 1 1

111 Ui o o (J

X bJ bl o

1" 1" III Ui H H H H < 3 </) tt Q z III mSm v# III Hb ft

Ui BMB

La 111 ft ft ft ft

(J ft

UJ HI o 111 bl bJ bl

(/) </) o CC

u. ll. ll. b. ba u. o o o o O O o

tt tt tt tL Of u bJ UI bJ bJ bl bl

ca CQ GQ CQ CQ CQ CQ z Z z Z Z Z z 3 3 3 3

NIST-114A U.S. DEPARTMENT OF COMMERCE

(REV. 3-90) NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

BIBLIOGRAPHIC DATA SHEET

1. PUBUCATION OR REPORT NUMBER

NIST/SP-500/198

2. PERFORMING ORGANIZATION REPORT NUMBER

3. PUBUCATION DATE October 1991

TITUEAND SUBTITLE Monitoring and Reporting Techniques for Error

Rate and Error Distribution in Optical Disk Systems

5. AUTHOR(S)

Fernando L. Podio, Editor

6. PERFORMINQ ORQANIZATiON (IF JOINT OR OTHER THAN NIST, SEE INSTRUCTIONS)

U.S. DEPARTMENT OF COMMERCE

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

OAITHERSBURQ, MD 20899

7. CONTRACT/QRANT NUMBER

e. TYPE OF REPORT AND PERIOD COVERED

Final

9. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (STREET, CITY, STATE, ZIP) National Institute of Standards and Technology

Building 225, Room A61

Gaithersburg, MD 20899

10. SUPPLEMENTARY NOTES

11. ABSTRACT (A 200-WORD OR LESS FACTUAL SUMMARY OF MOST SIGNIFICANT INFORMATION. IF DOCUMENT INCLUDES A SIGNIFICANT BIBUOGRAPHY OR

LITERATURE SURVEY, MENTION IT HERE.) This report constitutes the proceedings of the workshop on Monitoring and Reporting

Techniques for Error Rate and Error Distribution in Optical Disk Systems held on August

5, 1991, in Colorado Springs, Colorado. The objectives of this workshop were to identify

the state of the art on error rate monitoring and reporting techniques in optical disk

systems and to promote discussions on possible future implementations.

The workshop presentations included the description of a Computer Systems Laboratory of the

National Institute of Standards and Technology (CSL/NIST) program for investigating error

reporting capabilities of optical disk drives. Federal Government needs on error detection,

correction and reporting, and the state of the art on error reporting capabilities in current

generation drives. Presentations also included the description of the capabilities of error correction and detection current chips and discussions on error management strategies.

The participants noted the need for tools for reporting error rate activity through

interfaces with the host and emphasized the importance of this subject to data managers in

the Federal Government.

During a discussion panel, the participants identified a preliminary set of user requirements

and it was suggested that a Government/industry working group be organized in order to

document a consensus position between the Federal Government and industry in error reporting

capabilities for future generation drives.

12. KEY WORDS (6 TO 12 ENTRIES; ALPHABETICAL ORDER; CAPITAUZE ONLY PROPER NAMES; AND SEPARATE KEY WORDS BY SEMICOLONS) error detection and correction chips; error recovery techniques; optical disks, error detection and correction; optical disks, error distribution; optical disks, error management

techniques; optical disks, error rate.

13. AVAILABIUTY UNUNITED

FOR OFFICIAL DISTRIBUTION. DO NOT RELEASE TO NATIONAL TECHNICAL INFORMATION SERVICE (NTIS). ORDER FROM SUPERINTENDENT OF DOCUMENTS, U.S. GOVERNMENT PRINTING OFFICE, WASHINGTON, DC 2O402. ORDER FROM NATIONAL TECHNICAL INFORMATION SERVICE (NTIS), SPRINGFIELD, VA 22161.

14. NUMBER OF PRINTED PAGES

89

15. PRICE

ELECTRONIC FORM * U.S. G.P.O.:1991-311-891:40054

ANNOUNCEMENT OF NEW PUBLICATIONS ON

COMPUTER SYSTEMS TECHNOLOGY

Superintendent of Documents

Government Printing Office

Washington, DC 20402

Dear Sir:

Please add my name to the announcement list of new publications to be issued in the series: National Institute of Standards and Technology Special Publication 500-.

Name

Company

Address

City State Zip Code

(Notification key N-503)

llM,kjJL Technical Publications

Periodical

Journal of Research of the National Institute of Standards and Technology— Reports NIST

research and development in those disciplines of the physical and engineering sciences in which

the Institute is active. These include physics, chemistry, engineering, mathematics, and computer

sciences.

Papers cover a broad range of subjects, with major emphasis on measurement methodology and

the basic technology underlying standardization. Also included from time to time are survey

articles on topics closely related to the Institute's technical and scientific programs. Issued six times a year.

Nonperiodicals

Monographs — M^or contributions to the technical literature on various subjects related to the

Institute's scientific and technical activities. Handbooks — Recommended codes of engineering and industrial practice (including safeh' codes)

developed in cooperation with interested industries, professional organizations, and regulatory

bodies.

Special Publications — Include proceedings of conferences sponsored by NIST, NIST annual

reports, and other special pubncations appropriate to this grouping such as wall charts, pocket

cards, and bibliographies.

Applied Mathematics Series — Mathematical tables, manuals, and studies of special interest to

physicists, engineers, chemists, biologists, mathematicians, computer programmers, and others

engaged in scientific and technical work.

National Standard Reference Data Series — Provides quantitative data on the physical and chemical

properties of materials, compiled from the world's literature and critically evaluated. Developed

under a worldwide program coordinated by NIST under the authority of the National Standard

Data Act (Public Law 90-396). NOTE: The Journal of Physical and Chemical Reference Data

(JPCRD) is published bi-monthly for NIST by the American Chemical Society (ACS) and the

American Institute of Physics (AIP). Subscriptions, reprints, and supplements are available from

ACS, 1155 Sixteenth St., NW., Washington, DC 20056.

Building Science Series — Disseminates technical information developed at the Institute on building

materials, components, systems, and whole structures. The series presents research results, test methods, and performance criteria related to the structural and environmental functions and the

durability and safety characteristics of building elements and systems.

Technical Notes — Studies or reports which are complete in themselves but restrictive in their treatment of a subject. Analogous to monographs but not so comprehensive in scope or definitive

in treatment of the subject area. Often serve as a vehicle for final reports of work performed at NIST under the sponsorship of other government agencies.

Voluntary Product Standards — Developed under procedures published by the Department of

Commerce in Part 10, Title 15, of the Code of Federal Regulations. The standards establish

nationally recognized requirements for products, and provide all concerned interests with a basis

for common understanding of the characteristics of the products. NIST administers this program

as a supplement to the activities of the private sector standardizing organizations.

Consumer Information Series — Practical information, based on NIST research and experience,

covering areas of interest to the consumer. Easily understandable language and illustrations provide useful background knowledge for shopping in today's technological marketplace.

Order the above NIST publications from: Superintendent of Documents, Government Printing Office,

Washington, DC 20402.

Order the following NIST publications— FIPS and NISTIRs—from the National Technical Information

Service, Springfield, VA 22161.

Federal Information Processing Standards Publications (FIPS PUB) — Publications in this series

collectively constitute the Federal Information Processing Standards Register. The Register serves

as the official source of information in the Federal Government regarding standards issued by

NIST pursuant to the Federal Property and Administrative Services Act of 1949 as amended,

Public Law 89-306 (79 Stat. 1127), and as implemented by Executive Order 11717 (38 FR 12315,

dated May 11, 1973) and Part 6 of Title 15 CFR (Code of Federal Regulations).

NIST Interagency Reports (NISTIR)— A special series of interim or final reports on work

performed by NIST for outside sponsors (both government and non-government). In general,

inifial distribution is handled by the sponsor; public distribution is by the National Technical

Information Service, Springfield, VA 22161, in paper copy or microfiche form.
3? O

92. Q

S" CDc

TJ W

i-

=>" S (D

<D (0 c(A

(D

02C

?v 5-. V) 3- O • (D 3 _

c 3 w

ISJ -»

^1CO 3

{u <D ^8

©

o3-O

o

(Q

•<