# ABSTRACT

The crux of the archive problem today is that the core to everything
that technology requires and, yet provides as a product, is data and
information, which, in turn, is used to foster future development. This
brief summary deals with how archives started, their progression in
society, and the demands and issues archivists face in today's
technological explosion as unimaginable data generation powers have now
been cast among us with the ubiquitous high performance computers. It
took civilization, once it became advanced, hundreds of years to record
enough information to warrant the first libraries established in ancient
Greece. Now an ever expanding enlightened global population are
exploiting massive observational and computer powers capable of
generating many orders of magnitude more data or information in just a
few seconds than was generated in the past two thousand years. This new
era has instigated a major change for archivists from a world of
"human-readable" data to one of "computer- ciphered" data introducing a
completely new set of issues and processes.

**THINKING BIG**

The magnificence and magnitude of the universe is incomprehensible. Just
as amazing, is the power of our gift from heaven, our brain. Dr. Ralph
Merkle of Xerox Palo Alto Research Center estimates the human brain
stores about 10^18^ bits of information and processes these bits at
about 10^16^ bits per second (of course, these numbers are not exact,
similar to our understanding of the universe, but are close to within a
power of 10 or so). Your memory is just one small database that you can
access at the speed of light, and your intelligence is derived from the
ability to access and retrieve data from this prodigious database.
Another big thought: astronomers estimate that there are as many as 100
billion (10^11^) galaxies, and, based on our own Milkyway galaxy, each
galaxy is estimated to contain more than 200 billion (2 x 10^11^) stars,
resulting in the Universe that is populated by as many as 2 x 10^22^
stars, or 20 billion trillion stars. At the same time, if the rate of
processing power and the amount of data it generates continues to
progress at its current rate of doubling every 18 months (Moore's law),
we can anticipate data collection volumes to grow to a yottabyte (10^24^
or 2^80^) in volume by the year 2015. This is two hundred times more
bytes of data than all the stars in the Universe. Where the Greeks were
"thinking big" to conceive of libraries containing treasures of
knowledge, we must begin "thinking big" to conceive of how we can
exploit all the knowledge that we have accumulated over the millenium
and that now is multiplying at exponential rates with today's technology
explosion.

**WHAT IS AN ARCHIVE?**

Ancient civilizations maintained records for commerce (and probably
taxes or tolls), but it was the Greek civilization that conceived of a
repository of data as an archive. Until recently, the word "archive" was
a noun for describing public records. There was no transitive verb to
"archive". The latin root word "archiva" came from the Greek word,
"arkhein" which meant to rule. The Greeks regarded knowledge from which
they could better understand their subjects and trade partners as a
prerequisite for ruling. The Romans adopted many of the Hellenistic
traits when Julius Caesar overthrew the Ptolemics in 48 B.C. in the
battle of the delta and evolved a word for archive, "archiva" as they
were the first real bureaucrats who retained records necessary for
public services. Incidentally, it was in 48 B.C. that Ceasar supposedly
burned the great library of Alexandria, though the act of burning the
400,000 scrolls was probably unintentional, as the scrolls were
inappropriately stored in grain depots at the docks, which were the
intended target of Caesar's conflagration. In fact, the Romans
subsequently adapted their libraries based on the Greek libraries that
originated with Aristotle. In answering "what is an archive", in today's
world, as archive is a collection of records and documents, or just
about anything else.

# THE POWER OF DATA

The Greeks as mentioned above were the first people who felt the
preservation of knowledge had value for the betterment of their society.
Aristotle established a collection of documents for his Lyceum in Athens
for the enlightenment of his students. The first library of Alexandria
was established by Demetrius, a follower of Aristotle, for Ptolemy I and
handed down to Ptolmey II in 283 B.C., beginning the practice of
establishing a library for attracting scholars. Incidentally, the first
"metadata" event was established when Callimachus the poet created the
first known card catalog called the *Pinakes* for the Royal Library in
240 B.C. following Aristotle's divisions of knowledge into subdivisions
of observational and deductive sciences. So, as the Greeks developed the
legendary library of Alexandria for the sake of collecting all the known
world's knowledge, the Romans, who followed after their conquest of the
Ptolemics in 48 B.C., continued the concept of a "Royal" or government
library for the purpose of keeping public records necessary to govern
their empire. After the fall of the Roman Empire, collections on
manuscripts were primarily maintained in the monasteries scattered
throughout the kingdoms of Europe. The monasteries, which flourished in
a non-secular environment, coveted the knowledge they held in their
libraries for the sake of control through religious tenets. During the
ensuing centuries, Europe entered and endured the "Dark Ages" when
religious tenets and public ignorance stymied any expansion of
scholastic holdings. In fact, many civilization records were destroyed
during these times and in subsequent conquests in the name of religious
heresy. During these same periods, the far eastern civilizations
continued to record data and collect documents as they were rather a
secular society in which fear of supernatural beings did not unduly
restrict their thinking. After the 15^th^ century, record keeping
reemerged with astronomy, as astronomical knowledge evolved despite the
dangers of knowledge considered to counter the monastic beliefs of the
period (more than one lost their life due to theories which countered
the perceived perfection and order of the universe guarded by religious
tenets). Learning institutions began to flourish and independent archive
collections were developed. Even as early as 1472, the library at the
Queen's College in Cambridge, England, had 199 books reflecting almost
the entire record of western civilization to date.

# WHY ARCHIVE?

When the new Republic of the United States of America moved its capitol
from Philadelphia to Washington in 1800, President John Adams
appropriated a hefty sum in those times of \$5,000 to purchase "such
books as may be necessary for the use of Congress". The books purchased
from London consisted of 740 volumes of primarily law books and three
maps. For even then, almost all literature and art was in private hands.
When Thomas Jefferson succeeded Adams as President in 1802, he expanded
the role of the Adams library as the first library of the American
government, now known as the Library of Congress. When the British
sacked and burned Washington in 1814, the 3,000 volume library was
destroyed. In 1815, the government bought Jefferson's private collection
of 6,487 volumes for almost \$24,000 to restore the library. In 1850,
the Library evolved into a National Library as Joseph Henry, the
secretary of the Smithsonian Institution, argued that the Library of
Congress should be viewed as an appropriate foundation for "a collection
of books worthy of a Government whose perpetuity principally depends on
the intelligence of the people". This is a reflection of the Greek
feelings almost two millenium before. The copyright law of 1870 which
stipulated that two copies of every book, pamphlet, map, print,
photograph, and piece of music registered for copyright be deposited in
the Library, gave cause for expanding the Library. Finally the Library
was funded to be housed in its own building. In the argument for support
of funding to build a Library building, Senator Voorhees of Indiana
eloquently expressed his Jeffersonian belief in the essential moral
value of books and intellectual property by stating: "Knowledge is
power, the power to maintain free government and preserve constitutional
liberty. Without it, the world grows dark and the human race takes up
its backward race to the regions of barbarism." So from that, the
Library of Congress became the National Library to preserve not only the
intellectual property of this country, but also to preserve an
intellectual enlightenment for the betterment of society.

For the first 150 years of our country's history, most government
records were stored haphazardly, falling victim to fire, water, and
neglect. Historians and federal officals realized that the records
needed a permanent home that would be accessible to the government and
the public. Government operations at all levels were collecting
information regarding population and demography (census), property deed
records, marriage records, and birth and death certificates. Finally in
1934, the National Archives and records Administration became a reality.
Where the Library of Congress preserved an archive of documents
originally intended for reference for the Congress and as well as
documents with cultural value, the National Archives and Records
Administration was chartered to preserve the official records produced
by the operations of the government. In this role, the National Archives
ended up with many historical documents and artifacts, to such an extent
that it's function as an official government records repository extended
to co-function as a museum and historical reference library.

# A CHANGE IS IN THE AIR

Star catalogs developed by Hipparchus around 150 B.C. and preserved as
an early scientific record became a valuable record to detect change
when novae or "stellar guests" appeared. Only recently (just the last
three or four decades) have we begun to realize the value of data and
information as the world has begun to undergo change at an alarming pace
as technologies began to evolve. Until recently, little attention was
given to preserving information outside of public records and historical
documents. Much of the actual data and information which was used in
developing and publishing scientific study results were withheld, as
data were guarded by scientists and industry as proprietary for economic
advantage or coveted by researchers for personal claim. Other data
records were simply regarded as working copy data without consideration
of disposition for archive. Until now, access to data was generally
restricted to those few who had the capability or access privilege to
parse through data repositories. Even then, since data and documents
were generally shelved versus archived, the processing of parsing
information from the "stacks" was time consuming and required prior
knowledge of where and how to look, even with access paradigms to assist
in the "hunt, search, and browse" processes. As a result, the value of
an archive wasn't generally appreciated and received little economic
support. As the stacks grew to overflowing and treasures of information
became buried under the pile, the problem of finding data was
exacerbated by the rate of influx of new volumes. It was as if the
forest of data grew so much as to block out the light with which to see
under the canopy. Then, just in this decade, came the Internet and its
prodigious outreach capabilities. Suddenly, the value of data increased
as more people suddenly had a method for readily exchanging information.
The Universities first, followed by the government, seized on the
concept of placing data and information into accessible realms to fully
exploit the original intent of the Internet. As an outgrowth, search
engines evolved to enable more efficient and broader community
utilization of data and information sources, and now anyone with
Internet capability has access to prodigious volumes of data and
information. As the populace became more in tuned to the world and
environment around them, a greater appreciation of understanding the
changes that were now appearing so rapidly aroused interest in the study
of change. It's as if we suddenly realized we were running out of space,
as the new age of computers and data have shrunk the world. Now data in
repositories have increased value as an asset for studying change. The
driving force behind the efforts to collect almost unimaginable amounts
of data is to understand and appreciate change. What we archive is
longer just a question of "are we enlightened?", but now, it's a
question of "what's going to happen to us?" Perhaps historical data can
offer some clues to answer these questions, and as we look at more
information at a quicker pace, answers and many more new questions will
begin to surface even faster. And, of course, the pile of data will
become increasingly big.

## JUST COLLECTING DATA

In 1934, T.S. Eliot foresaw a fundamental problem with what seemed to be
a data explosion even then when he published the dramatic poem, "The
Rock", where he lamented, "*Where is the wisdom we have lost in
knowledge? Where is the knowledge we have lost in information?*"
Archivists are reluctant to throw anything away. Keeping everything was
reasonable in the past, as there were few copies available as compared
to today. After all, there was only a few stone tablets produced (only
90 or so have been recovered) and if it had not been for the Rosetta
Stone discovered in 1799, we would have not been able to interpret most
of them. Just a few more hand-scribed manuscripts survived, and then,
many more circulated copies after the Gutenberg printing press was
developed in 1472. At that time, the library at the Queen's College in
Cambridge, England, had only 199 books. Now, almost a half million books
and journals are published annually. The Library of Congress has
accumulated 113 million items in the 200 years since its inception. The
National Archives as accrued over a billion pages of texts and documents
in its short life of 65 years.

The computer revolution that evolved in just the last four decades have
changed our entire concept of collecting and managing data and
information. In the opening paragraph, a statement was made that the
core to everything that technology requires and, yet provides as a
product, is data and information. As computer technologies advanced, the
capability to observe, assimilate, and output large amounts of data
accelerated at even a faster pace. The result is, today data are being
provided at rates beyond which we are prepared to manage. The
information explosion is happening. Perhaps the explosion just began to
happen only an instant ago in the time of humans, and, in an analogy to
the creation of the universe, we are in the early phase of the explosion
and have a universe void yet to fill.

The amounts of data collected from natural and science-induced events
were limited to the capability to observe events and to transcribe the
events into a recordable form. After all, there were only so many stone
tablets. There were only so many blank papers, quills, and powdered ink
for Lewis and Clark to record the events and discoveries of their
amazing journey across the North American continent. Even early
computers were restricted to assimilating just a few inputs and
producing even fewer results. But as recording capabilities expanded,
the ability to detect and produce data from events and the ability to
capture the sensed information expanded, and the amounts of information
started growing at more than the exponential rate of technology
advancement. Weather observations are a good example. Just a couple of
decades ago, weather observations were transcribed to digital form on an
hourly basis, though analog recorders provided a continuous data record.
And because this required human intervention, there were a limited
number of stations, 154 stations in 1888 reporting once a day, to 250 or
so in the last decade reporting hourly in the U.S. In 1997, the
Automatic Surface Observation System (ASOS) was installed, and when
fully deployed can provide as many 1700 sites. The ASOS system captures
observed weather events in digital form on a minute by minute basis. The
data gathering potential thus increases by a factor of 400:1. At the
same time, the U.S. national Doppler radar system was installed,
providing a digital data record at even more prodigious rates. This is
nothing compared to the amounts of data being generated by particle
super-collider systems used for atomic research. They accumulate data at
petabyte (million billion) volumes in the matter or a year or so. The
reason space borne sensors do not collect more data than they do (the
civil earth observing environmental satellites are expected to collect
about less than five petabytes of data from their inception in the 1960s
through the year 2015), is because of cost and political constraints
limiting power resources and broadcast options.

## SUDDENLY EVERYTHING IS DIGITAL

Digital television is on its way. The entire collection of commercially
audio-recorded data are now available in digital form. And, telephony is
becoming all-digital, publications are migrating to digital, and as more
people come on-line, messaging is evolving to become all digital. This
has a compounding effect on the amounts of data be collected and saved
in digital form. Technology enables everyone to place their data on-line
so people can have access to it. The Internet is a perfect example. It
is estimated that today, there are over 50 million web sites, and 300
billion pages of information available over the web. Since everything
associated with printing these days is now in digital form from
inception to the press, it is easy to place material in a data base and
provide it over the Internet. But, the digital information explosion has
resulted in entirely new paradigms for managing information from dealing
with "human readable" records to that of dealing with information that
requires a translator tool -- the computer. The concept is now longer
keeping track of documents held on a shelf, but controlling or losing
control of indecipherable information. Suddenly, the methods of archive
developed over the last two millenium are forced to undergo a radical
change.

## IS THERE A DIGITAL CRISIS?

First of all, no technology is finite, in fact, as the technology curve
bends at a more exponential rate, the half-life of systems and their
interface requirements will become increasing shorter, placing data at
risk. Even when dealing with "human readable" documents, archivists have
problems with preserving data because new lower cost, mass produced
paper products are acidic resulting in deterioration problems. The same
is true for film products as the more active emulsions tend to fade
faster (emulsions never cease to be active). In the past, the major
concern of the digital archivist for the loss of data, aside from fire
or storm damage to the storage facility, was because of media
deterioration. However now, at least for the digital archivist, the
problem has an added dimension of system obsolescence despite the fact
that media technologies have succeeded in producing highly stable
materials which have significant shelf life. For example, magnetic tape
used to develop physical problems in ten years or so. New encapsulated
magnetic tapes now are expected to have a shelf life of 25 or more
years. Optical media is touted to have a shelf life of up to 100 years.
With the technology curve being what it is, even the short lived 25 year
magnetic tapes will face system obsolescence problems long before the
media experiences physical changes putting data at risk. This is no more
apparent than to personal computer users who find that documents created
only a few years ago cannot be read by new, supposedly improved
applications. Suddenly, everyone is at the mercy of manufacturers for
preserving their data. The manufacturers look at their products as a
short half-life, throw-a-way item, as they continually advance their
technological capabilities to maintain market position. In the market
place, the majority rules. If the data archivist is not in the majority,
as they obviously aren't, they lose, and the data are at risk.

## CAN WE MITIGATE OBSOLESCENCE?

The answer is probably not entirely. Unfortunately, no one is willing to
make decisions on what data should be preserved versus what is not
essential. Most say everything should be saved, but when faced with the
sometimes enormous cost of migrating data from one form to another, a
choice has to be made on what is saved. This problem is compounded by
the volumes of data being captured. Three major issues are brought to
bare when considering the prospect of preserving data indefinitely. One
is simply keeping track of what you have. Another is being able to read
the information in a new systems environment. And the last is, the cost
in time of moving data from a prior technology to a new one. For
example, if one collects data for ten years onto a media sized to the
rate data are being collected, then presumably it would require ten
years to re-record it onto a new media even if the new media system is
an order of magnitude faster than the prior system (as the truck bumper
sticker stated, "I may be slow, but I'm in front of you"). In this case,
one would have to go to the expense of replicating the initial recording
capability by whatever speed up factor they wished to migrate the data.
For example, if the effort is intended to migrate a ten year data set in
one year, ten of the original units would have to be acquired to
complete the task in one year. Then there is the problem of acquiring
the older technologies, often no longer produced. The result is
inevitable, there will be data that will not be recovered. Even the most
perfectly controlled and documented data will face this problem, and the
problem severity is directly proportional to the volume of data in
question. For example, a recent Washington Post article on March 12,
1999, pointed out that the Library of Congress intends to digitize
10,000 out of the 113 million texts they will have on hand by the year
2001. At the same time they are accumulating new texts at the rate of
10,000 every two weeks. This is equivalent of trying to empty a
reservoir fed by a fast moving river with a two gallon bucket.

## WHAT NOW COACH?

Technology seems to be following a curve that promises to make the media
more indelible in some instances mitigating, and others complicating,
the data storage problem. This places a greater emphasis on adhering to
standards for recording data, and for extensive indexing of the data to
ensure future access. Hopefully, access will be a key element in the
data archive paradigm, as that should be the criteria for saving data in
the first place. The question of whether data and information is a
national asset or a national heritage has to be addressed. Using a
historical model, data always starts out as an asset (having economic,
military, or political advantage) and ends up as a heritage trove
(historic value for science and education). Generally, there are
economic reasons to fund the management of data while it is an asset
mode and little appreciation or understanding of the value of
maintaining data once it enters a heritage mode. Preservation of
heritage data suffers from lack of support because it's beyond a "budget
cycle", or "it's not on my watch", or the historical value is not fully
appreciated or understood. The result, data continues to accumulate, now
exponentially, because no one will decide on its fate. More often than
not, the decision of whether to preserve data or not is likely to be
made for us as a result of media deterioration or systems obsolescence.

## WILL WE BECOME LOST IN THE FOREST OF DATA AND INFORMATION?

In 1985, David Byrne wrote a song called "In the Future", in which he
lampoons those making predictions, where the song goes, "*In the future
there will be so much going on that no one will be able to keep track of
it."* As the market place expands, the progression of advances in
computing capabilities continues to double about every 18 months with
the same physical and cost constraints (a fact attributed to Gordon
Moore, Chairman Emeritus of Intel). Part of the reason for this rapid
evolution is the fact that technology tends to feed on itself. As far as
the future is concerned, and my personal crystal ball is myopic at best,
and may even be clouded by a cataract, but there is no reason that
Moore's law will not continue to hold. The pace of development and
capabilities will compound the rate at which data will be generated. The
major obstacle to advancement will be managing data for access. So as we
compute at teraflop rates, communicate at terabit rates, and store data
in petabyte or even exabyte (billion billion) or yotta bytes (billion
trillion) quantities, will we be able to parse information from all the
data that will be generated as a result? I wonder. Also, how can we
afford the cost and effort to save it forever? My guess is that we can't
and won't, and data will be lost. This brings up another point, with so
much data streaming from so many sources, how is the credibility of all
these data assessed? Will the proportion of misinformation grow at the
same rate as accredited information, and how will we ever know? As a
result, future development must now focus on data management processes
extending the data warehousing, and data mining just started to enable
future generations to even begin to extract value from what will be the
heritage data sets in their time. Or will the new generation's "throw
away mentality" be content to abandon data and information out of a
perpetual fear of obsolescence? Or, with increased use of "snippets" of
information in today's increasing fast pace of life, will the new
generation be content with only "snippets" of heritage data? I suspect
there may be no other choice.

Then, there is a more disturbing factor. Large data producers are more
concerned with fast access to data than the preservation of data. The
new "hot button" in data management is Storage Area Networks (SANs)
where data are physically distributed but managed as a logical whole in
order to accelerate access to the data. Some even refer to SANs to stand
for Storage Access Networks. All of the data management paradigms are
concentrating on instant access. In this mode, data are striped across
many storage devices to relieve access contention while at the same time
providing a high degree of access fault tolerance to mitigate data loss.
But this process parses data into segments of files or objects only for
the purpose of access and the data no longer is resides as a whole. This
places all the control of managing data in the realm of access taking
away any form of control for the data in a repository environment. In
this fast paced world of providing data services, there usually is no
consideration given to archive. So, not only is support for archiving
data eroded as its heritage value comes second to its asset value, now
systems are focusing on access with no consideration for archive. When
the on-line caches fill up, old data will spill over onto the floor.
This is happening now because access drives the market place.

**AND FINALLY**

In April 19, 1998, there was an article in the Washington Post newspaper
by Joel Achenback, addressing the approaching singularity of technology.
It states: \"*The moment in the future when so many technologies have
converged - computers, miniaturization, bio-medicine - that they become
\"auto-catalytic\", driving one another to yet greater sophistication,
\"hyper-accelerating.\" Predictions will be worthless because everything
is changing so fast - an event horizon beyond which we can detect
nothing!*\" And, as a subsequent Washington Post article pointed out,
with the information glut, the world is not ending, it is just becoming
incomprehensible.