One appproach has been to analyze the data as soon as possible, reducing the amount of information that has to be stored. A second approach has been to create discrete data sets for each experiment, which are stored on tape and subsequently analyzed when time and funds are available. When data is immediately reduced in size, it is not possible to take a second look at the raw data and apply revised algorithms to better understand the observed phenomena. When each experiment is analyzed separately, it becomes very difficult to analyze correlations between data sets. The need for such a system is being driven by the enormous amount of scientific data that is already being collected.
Radio astronomers can collect a terabyte of raw data per day.
Oceanographers are generating single 600-GB data sets for the analysis of the average temperature of the world's oceans. Future data sets will be even larger. The Earth Observing Satellite system is expected to store 7 petabytes of data by year 2007. Synthetic Aperture Radar has the projected capability to map the Earth's surface to a resolution of 10 centimeters square. Even at the coarse resolution of 1 byte per point, this is a 50 petabyte data set.