Architecture of the new MARS server
Baudouin Raoult
Meteorological Applications
ECMWF
1. Introduction
Since 1985, the ECMWF Meteorological Archival and Retrieval System (MARS) has grown both in size anddiversity. At its start in 1985, the operational archive was growing at 70 Mbytes/day. In 1996, the growth rate was 125
Gbytes/day of operational data and 7 Gbytes/day of research data, the total archive was 22 Tbytes. The number ofindividual items archived is also very large: 120,000 operational fields/day, 140,000 reports/day and 200,000 research
fields/day.
Figure 1 shows the various data types that are in MARS and the year in which they have been introduced. We actually
have more data, for example we have observations since 1979, but the data type “observation” was added to MARS injuly 1990. Earlier data was them back-archived. Since this chart was drawn, new types have been added: 4Dvariational analysis, climatological simulations, seasonal forecasting, ensemble tubes and much more. As you cansee, the curve shown is exponential, and we add new types more and more often.
Figure 1: MARS - Data Types
Wave Monthly Means
Analysis Feedback
Sensitivity Forecast
Monthly Means
Ensemble Means
Re-Analysis
ERS1
Forecast Probability
Time Series
Satellite Images
Ensemble
Waves
Error in FG
Other Centres
Observations
Error in Analysis
TogaModel86

87

88

89

90

91

92

93

94

95

We still have to add ocean models, coupled models, private datasets,... The list is endless.
MARS is used in batch from the super-computers, interactively through Metview and remotely from the Member
States. Every day, 30,000 requests are processed, 1,500,000 fields and 100 Gbytes are moved.
In order to cope with this growing archive and the ever changing requirements, a project was set up in 1996, called theDHS project (Data Handling System). MARS was totally redesigned to run on a Unix platform. It is written in C++
and uses ObjectStore to store its metadata and ADSM to store the actual data.
The new MARS had to be compatible with the old system. It has to provide a service for the next 10 years, and likeevery computer program, it has to be efficient, scalable and flexible. This paper describes how we try to achieve thosegoals.

2. Decoupling physical and logical organisation of the data
From the experience gained with the previous MARS systems, we knew we could build a system scalable it we could
decouple the physical organisation of the data from its logical organisation. For that we split the system in two parts.
The first part, the MARS Server, has a semantic knowledge of the data. It knows what a meteorological field is, whata forecast is. The second part, the Data Server, has a physical knowledge of the data. It knows if a piece of data in ontape, on disk or cached. Figure 2 shows the architecture of MARS.

ObjectStoreFigure 2: Architecture

RequestRequest
Data

MetaData

MetaData

References

References

MARSServer

References
Data

Files

DataServer
Data

Files

ADSM

The MARS Server does not handle files but data references. When a user request is processed, the MARS Servertranslates it in a list of data references, that are passed to the Data Server. The Data Server translates data references inactual files and returns the data.
By using this design, we have a system that is independent from the underlying hardware and from the underlying
software (ADSM). The data can be physically re-organised without any impact on the system. Data files are split orjoined, moved from disk to tape without a need for the MARS Server to know about it. Something else we have learnt
from the previous system is that the fewer the files we have, the more manageable the system is. Most of the existing
systems have problems managing more that a few million files. With this architecture, we can reduce the number offiles by merging them into larger files. Nowadays, we have more than 30 Tbytes in less than 200,000 files.

3. The metadata
The metadata is “the data that describes the data”. In MARS, the data are fields or observations. The metadata is thedata that knows where is which field or observation. This metadata is stored in an object oriented database, managed
by a commercial product called ObjectStore. As we saw earlier, we need to split the logical organisation of the data
form its physical organisation. This split will be reflected in the metadata. In order to define this metadata, we firstneed to have a look at the data and answer the following question:

3.1 What is a meteorological field?
For MARS, a meteorological field is the smallest addressable object. It is defined by various attributes such as thoseshown in the following table:
AttributeValueClassVersion
StreamParameter
Level
Date
Base TimeTime StepMember

Operational
1
Daily archive
Temperature1000 hPa
1993-08-10
12Z
120 H
42

The attributes may be different for different data types. For example, the number attribute is only used to select aspecific forecast number in an EPS (Ensemble Prediction System).
3.2 Archive objects
There are too many fields to store them individually. We need to group them into logical entities such as “a forecast”,“a month of analyses” or “a research experiment”. This provides a natural data co-location for a faster access torelated data. Those entities are called archive objects. We can stack all the fields for a forecast in a “cube” as shown in
Figure 3. We group the fields using three attributes: level, parameter and step. This would represent a single
forecast, for a given date, a given base time, and a given version. The three attributes are the three dimensions of thearchive object.

StepsLevels

Figure 3: Archive objectParameters

Because we group the fields using more than three attributes, the archive objects are actually hypercubes. Figure 4 is
an attempt to draw a six the dimension hypercube that would represent all the fields of all the forecasts of an EPS fora month.

St

eps

Levelsep
St
eps

Levels

St

epss
St
ep
eps

Levels

St

epss
St
ep
St

ep
eps

Parameters

ep

ep

St

Sts
tes

DaSt

epss
St
ep
St

eps
St
ep

Levels

ep
Sts

eps

Parameters

Sts

Parameters

Parameters

Sts
eps
St

eps

LevelsLevelssStepSt

LevelsLevelsLevels
St

eps
St
ep

Parameters

Parameters
LevelsSt

ParameterssLevelssParameters

Parameters

Parameters

Parameters

Parameters

ParameterssLevelsep
St

LevelsLevels

s
St
ep
St

ep

Levels

s
ep

s
St
ep

LevelsParameters

s
St

ep

Parameters

Parameters

Parameters

Parameters

Parameters

Parameters

St

Parameters
LevelssLevelss
St
ep

Levels

s

ep

s
St

ep

LevelsParameters

Parameters

Parameters

Sts

LevelssLevelsep
St

LevelsParameters

LevelsParameters

LevelsLevelsLevelsLevelsep

LevelsLevelsSt

Parameters
Parameters

LevelsTimes

Parameters
Parameters

Parameters

Parameters

Parameters

Steps

LevelsStLevels

LevelsParameters

ParameterssLevelss
ep
StLevelsFigure 4: Archive objects are hypercubes

Parameters

Members

3.3 Shapes and Layouts
We split the description of an archive object into its shape and its layout. The shape will be:•••

Its meteorological content.
Its logical organisation.
Part of MARS metadata.

And its layout will be:•••

Its physical organisation.
Its physical location.
Part of the Data Server metadata.

Figure 5 shows how we perform this split: the archive object is represented as a cube. The MARS Server metadata issimply the list of each dimension of the cube. In this case, three axes and their labels. This is called the “shape” of thearchive object.
The Data Server metadata will contain the size of each field, the file it is in, and its offset in this file. This is the“layout” of the archive object. Each layout is given a unique identifier that will be save by the MARS Server with theshape.
Figure 5: Shapes and Layouts
MARS Server
Metadata

Data Server
Metadata

Shape

LayoutSteps
12Length2448

Levels

32000
57000
32000

850 500 200...

Parameters

32000

ZT

57000

Files3.4 MARS Server metadata
All the shapes are organised in a tree fashion, as shown in Figure 6. The nodes of the tree are defined by the attributes
that are not used to describe the dimensions of the archive objects.
When a user request is received, it is used to navigate the tree. The highlighted path in Figure 6 corresponds to arequest for some fields from the operational analysis of february 1996.
Figure 6: MARS Tree
CLASSOPERATIONRESEARCH

YEAR

EXPERIMENT
1997

1995

ZQWI

1996
VERSION
2TYPE
1TYPE

FORECASTFORECASTANALYSISANALYSISDATEMONTHJANUARY

950101950102FEBRUARY

Because the design is object oriented, each node and each shape can be different (polymorphism). New nodes can beadded later, as new attributes are invented and new shapes can be added latter, as new data types are created. The
branches of the tree can all be different and the tree can span several physical databases
3.5 Data Server Metadata
The layouts are all organised in a large table, using their unique identifier as an index (Figure 7). A layout can be fully
expanded, in this case it is itself a three column table. The first line correspond to the first field, the second line to thesecond field and so on. A layout can also be compacted. Most of the time, the fields have a fixed size, so we can use a
simple run-length encoding scheme to reduce the size of the metadata. In Figure 7, the layout 12349 represents 8400
lines compacted into 3. The first one is the repeating factor of the two next lines.

Figure 7: Data Server Metadata
12345

12346

OffsetLengthFile032000Length

32000

57000

(4200)

89000

32000

57000

1210005700032000

032000

File

32000

57000

89000

32000

121000

57000

0

32000

32000

57000

89000

32000

121000

57000

...

12347

12348

12349

...

48400

4

4

Again, the design being object-oriented, the layouts are polymorphic objects. Future type of layout may be introduced
without any change code. The files also are polymorphic objects: support for new media or file storage systems will
be done without difficulty. A layout can span several files, tape or disk and a file can be shared amongst several
layouts.
Each file knows how many layout lines point to them (the numbers shown in Figure 7). This is called reference
counting and is use to perform garbage collection: files that are not pointed to are automatically deleted.
3.6 Adding a new fields in an existing archive object
The design described above gives us a new feature that did not exist in the previous system: incremental archiving. A
field can be added latter to an existing archive object. If one of is attributes has a new value, it automatically inserted.
Is an archive object has three dimensions, there are three ways it can grow, as shown in Figure 8:sepStSteps

LevelsLevelsFigure 8: Reshaping hypercubes

Parameters

Steps

LevelsParameters

ParametersIn general there a n ways to grow a n dimension hypercube, by inserting a n-1 dimensions hyperplan. It is a lot easierto perform the same insertion on the metadata once it is split. If we archive a new parameter w in an archive object(Figure 9), we simply insert a string in a array (MARS Server) and some empty slots in a other array (Data Server).
Figure 9: Updating metadata
MARS Server
Metadata(Shape)

Data Server
Metadata(Layout)
LengthSteps122448

Files

57000

96

a32000Levels

45700032000

850 500 300 200

Parameters
ZT

W

32000
Ub57000

V

332000...

4. MARS in action
4.1 Archiving
When an archive request is issued from a client, the request visits the tree (Figure 6) until it reaches a shape. If it does
not, the tree is grown accordingly, and a new empty shape is added. Figure 10 shows an incremental archive. The
names at the top are the various components involved. The time flows from top to bottom. The dashed arrows are themessages sent between the components. The dotted arrows are the data transfers.
File a and b contains data that has been been archived earlier. The data is transferred from the client machine into adisk file c. This file contains two 57000 byte long fields. The layout is resized to accommodate the new fields, and it
is updated to point to file c. The data held in c in still on disk, and said to be in “pre-archive” stage.
Figure 10: Incremental archiving
ClientMARS

Data Server

Archive

57000
32000

reshaoea57000
32000
57000
32000

TransferTarget = File3c

57000
32000
57000
...

UpdateIndex = (5,9)
Length = (57000,57000)
ID = 42
Source = File3b4.2 Retrieving from disk
While the data is still in “pre-archive” stage, retrieving is very fast. As in archive, the MARS tree is visited in order tofind one or more shape matching the user request. The request is translated into a list of indexes in the layout, and thedata is read directly from the disk files and sent to the user.
Figure 11: Retrieving, data is on disk
ClientMARS

Retrieve(Date = 1997-10-10,
Param = (z,t,w))

Data Server

57000
Get(Index = (2,3,7),
ID = 42)a

32000
45700032000

Result((File= a,Offset = 57000,Length = 89000),(File= b,Offset = 32000,Length = 57000))

57000

b

32000

357000
32000
57000

c

...

2Receive(Data = Result)

4.3 Flushing
Flushing occurs when an archive object is complete, or when the disks are full. A flush request write all the fields thatare in the “pre-archive” stage on tape. The layout will be updated to reflect the change, an the data on-line will bedeleted. The layout is then compacted. Figure 12 shows that the layout 42 is composed of files a, b, c and more.
Those files are merged into a single tape file x. The layout is compacted from 8400 lines to three lines, noticing thatthe field sizes are 4200 times 57000 and 32000 bytes long. Once the tape exist, the archive object is no longer in the“pre-archive” stage.
Figure 12: Flushing
ClientMARS

Flush(Date=1997-10-10)

Data Server

Flush(ID = 42)

57000

ADSM

Write(Source = (a,b,c,...),
Target = x)a3200057000

432000
57000

b

32000

357000x

32000
57000

c...

2(4200)

8400

57000
32000

4.4 Retrieving from tape
When the data is on tape, it must be copied first to a set of disks that are the MARS cache. ADSM is very useful as it
can read only portions of a tape file, so only the requested data will be cached. When data is retrieve from a tape, atemporary layout, called the “cache layout” is created. This layout contains the pointers to all the data that has beencached for a particular archive object. Figure 13 show the flow of request and data. The requested data is copied fromtape y into cache file z. The data is then sent to the user as describe previously. The cache is emptied using a least
recently used algorithm.
Figure 13: Retrieving data that is on tape
ClientMARS

Retrieve(Date = 1997-10-10,
Param = (z,t,w))

Data Server

ADSM

Get(Index = (2,3,7),
ID= 42)Read(Source = y,Target = z,
Offset = (57000,89000,267000),
Length = (57000,32000,57000))y(4200)

8400

57000
32000-z

57000
32000

3

Result(
File= z,Offset = 0,
Length = 146000)

57000
...

Receive(
Data = Result)

4.5 Patching
To correct a field that is wrong, the user simply need to archive it again. If the layout has been flushed, it is mutated inits expanded shape again, and the field is simple added as for a normal archive (Figure 14). The reference counting of
the tape file is decremented. If it reaches zero, that mean that all the fields have been replaced, and the tape file isautomatically deleted. The layout can be flushed again. In this case, only the new data will be written to tape. The
layout will now point to several tape files. When a layout is too fragmented (it points to too many tape files) it can be
defragmented by copying all of its data back to disk and flushing it again, into a single tape file.

Figure 14: Patching
ClientMARS

Data Server

ArchiveTransfer(Target = File3)

p

57000
32000

1

57000
32000
57000
32000
57000

Update(Index = 3,
Length = 57000,
ID= 42,
Source = Patch1)

8399

32000
57000
...

5. Conclusion
After one year of service, the new MARS keeps is promises. It now contains 520 millions of meteorological fields,
representing 32 Tbytes in only 180,000 files. More than 7 millions requests have been processed. The metadatadatabase represents 0.03% of the size of the archive.
This would not have been possible without an object oriented design, the metadata being too complex for a relational
approach, and yet very simple for an object oriented approach. The system is extendable, new data types can besupported and the underlying storage management can evolve.