DEIM Forum 2013 F4-4

Measuring the Effect of Node Joining and Leaving in a Hash-based
Distributed Storage System
Nam DANG† and Haruo YOKOTA†† Department of Computer Science, Graduate School of Information Science and Technology
Tokyo Institute of Technology 2–12–1 Oookayama, Meguro–ku, Tokyo 152–8552, Japan
Abstract Distributed storage systems have become the core of many large-scale applications today. Much research
has focused on the distribution of data in such systems, with two main approaches of tree-based and hash-based algorithms. Hash-based algorithms tend to provide a good distribution based on the randomness of the hash function
while maintaining a good performance if correctly implemented. However, the approach also incurs performancecost when a node joins or leaves the system, especially in the context of high-performance datacenter clusters. This
paper investigates such costs with a simulation model and discusses the effect of a node joining and leaving for avariant of hash-based algorithms for datacenter-oriented storage clusters.
Key words hash-based index, distributed storage systems, distributed index, scalable hashing

1. Introduction
Distributed storage systems are a current research focus in

Another challenge is to place data’s replicas on uncorrelated
devices to avoid loss of two replicas in case of failure on correlated devices.

computer systems and data engineering today. Such systems

This paper discusses the effect of node leaving and joining

provide many attractive features such as scalability, avail-in hash-based distributed storage systems. We also present

ability, and fault tolerance by harnessing collective capacity

an approach to measure node joining and leaving costs inof multiple storage nodes. The design of distributed storagea hash-based distributed storage system designed for data-systems also comes with a number of difficulties due to thecenter networks. The algorithm in this approach is CRUSH

distributed nature; oftentimes a good design is a compromise(Controlled Replication Under Scalable Hashing) [2], the coreof scalability, availability, and fault tolerance based on the

distribution algorithm employed by Ceph [3], targeted forrequirements of the system.

its proven effectiveness in both theoretical and industrial

Most storage systems write data to underutilized device.

practice. CRUSH shares many similar characteristics with

The problem with this approach is that once written, data

other hash-based distribution algorithms including high par-is rarely, if ever, moved. Even a perfect distribution canallelism, fast look-up, and randomized distribution of data.

become imbalanced because the storage is expanded. More-

The algorithm implements a deterministic hashing function

over, it is difficult to equally distribute the load across thethat maps an input value typically an object identifier to asystems since data is distributed to node gradually over time,

list of devices on which to store object replicas. Only a com-which often yields unexpected hotspots.

pact, hierarchical description of the devices comprising the

A robust solution for the above issues is to randomly dis-storage cluster and the replica placement policy is required

tribute data among available storage devices. The result is ato determine the location of an object’s replica; therefore,

probabilistically balanced distribution and uniformly mixes

the approach provides a high level of parallelism by elimi-

of old and new data. In this approach, all device will havenating any shared structure or central communication point

similar load, enabling the system to perform well under vari-and near-random data distribution.

ous workloads [1]. Furthermore, random distribution across a

The structure of the paper is organized as follows. The

large number of devices offer a high degree of parallelism andsecond section presents several notable data distribution ap-

better aggregate bandwidth, thus exploiting the distributed

proaches. We review the design CRUSH in details in section

nature of the system. However, simple hash-based methods

3. Section 4 introduces our simulation model for evaluating

are unable to cope with changes in the number of partici-

the costs of node joining and leaving. Section 5 presents thepating nodes, often incurring a massive reshuffling of data.

results of our model from simulating a large cluster. Finally,conclusion and future work are discussed in section 6.

2. Related Work
We present several related works of data distribution for
distributed storage systems. Each of them offers a differ-is mainly designed for and mostly used only in peer-to-peer
distributed systems.

3. The CRUSH Algorithm
CRUSH is the basic distribution algorithm of Ceph, an

ent paradigm and address different requirements in today’s

object-based distributed storage system.

It shares manystorage systems.

common characteristics with other hash-based distribution

2. 1 Hadoop Distributed File System (HDFS)

approaches such as high parallelism, fast look-up and bal-

HDFS [4] is a popular distributed file system designed

anced data distribution. At the same time, similar to otherbased on Google File System [5] for MapReduce [6]. Withhash-based algorithms, CRUSH also faces with the difficulty

one single central namenode server (metadata server), HDFS

of node joining and leaving. The inherent costs of mitigat-

utilizes several techniques to guarantee a robust and highly

ing such changes in the cluster are translated into compu-

scalable storage platform. Files are divided into small blocks

tational costs and exchanged messages. This paper inves-of 64MB and the blocks are distributed to data nodes under

tigates CRUSH in terms of node joining and leaving costs

the supervision of the namenode. Since the namenode han-as an example of hash-based storage systems, and discusses

dles every operation except data read/write, it also becomesmeasurements for such costs.

the central point of failure of the system. There has been

The fundamental of CRUSH is a per-device weight-basedsome research [7] [8] to address this weakness with proposals

distribution of data object that approximates a uniform

including mirrored namenode (or secondary namenode) or

probabilistic distribution.

using BigTable-like architecture [9].

cluster map that represents the hierarchy of the cluster and

CRUSH utilizes a hierarchical2. 2 Fat-Btree

the weight of individual devices to control the distribution

Fat-Btree [10] is a variant of parallel B-Tree structureof data. In addition, the distribution is also based on pol-widely used in many distributed Database Management Sys-

icy, which is defined in terms of placement rules that specify

tems (DBMS’s). In a Fat-BTree, each Processing Element

how many replica targets are selected from the cluster and(PE) has a subtree of the whole B-tree consisting of the root

what restrictions are imposed on replica selection. Given anode and all intermediate nodes between the root and theleaf nodes allocated to that PE. As a result, the lower level

single integer input value x, CRUSH will output an ordered−→
list R of n distinct storage targets. Relying upon a strong

index nodes with high update frequency have fewer copies

multi-input integer hash function with x as a parameter, thecompared to upper level index nodes with lower update fre-algorithm provides a deterministic and independently calcu-

quency. This technique reduces synchronization costs amonglable mapping scheme using cluster map, placement rules,

PEs when the Btree structure is modified by data operations

and x.

or node additions and removals. Nevertheless, there is still

3. 1 Hierarchical Cluster Map

significant performance cost due to the strict synchroniza-

CRUSH organizes the cluster topology according to de-tion of the shared nodes among the PEs, especially when thevices and buckets, both of which have numerical identifiers

number of nodes increases.

and weight values associated with. A bucket may contain

2. 3 Chord

any number of devices or other buckets; they can form inte-

Chord [11] is a protocol for a peer-to-peer distributed hashrior nodes in a hierarchical manner with the storage devices

table that uses a variant of consistent hashing to assign keys

at the leaves. Bucket weights are defined as the sum of theto nodes. In Chord, each node requires routing information

weight of the items they contain. By combining buckets into

about O(log n) nodes, and performs lookups via O(log n)a hierarchy, the structure as in a data-center cluster can bemessages to other nodes. Chord provides good support as

flexibly represented.

node joins or leaves the system, with each event resulting in
2

There are four different types of buckets, each with a differ-O(log n) messages. The algorithm also scales well with theent selection algorithm to address different data movement

number of nodes, and is able to recover from large numbers

patterns resulting from the addition or removal of devicesof simultaneous node failures and joins, and provide correct

and overall computation complexity.

answers for lookups even during recovery. One big drawback

3. 2 Replica Placementof Chord in the context of distributed storage systems is its

CRUSH is designed to address the placement of data withmessage-based exchange nature, which makes lookups in a

regards to replication in a heterogeneous cluster. The place-datacenter-oriented file system rather expensive compared toment of replicas on storage devices in the hierarchy also has

HDFS’s centralized namenode or Fat-Btree. In fact, Chorda significant effect on data safety. By projecting the struc-in case of node addition or remove. A summary of the dif-root

ference among the four bucket types is presented in Table
row1

row3

1.

row4

row2

cab22

cab24

Uniform

List

Tree

Straw

Speed

O(1)

O(n)

O(log n)O(n)Additionspoor

optimalgood

optimalRemovals

poor

poor

good

optimal

cab23
cab21

ActionTable 1 Summary of mapping speed and data reorganization ef…..
…..
…..

…..
…..
…..

…..

…..

…..

ficiency of different bucket types when items are added

…..
…..
…..

Figure 1 A partial view of a four-level cluster map hierarachy
consisting of rows, cabinets, and shelves of disks.

or removed from a bucket

In this study, we focus on only Tree Bucket and Straw
Bucket for their performance and high potential for practical applications. The other two, Uniform and List Bucket,

ture of the system into the cluster map, CRUSH can separate

lack adequate performance for a cluster with frequent nodestorage devices into failure domains according to the physi-

additions and removals and are mainly for theoretical discus-

cal location of the devices. For example, devices on the same

sion.

shelf belong to the same failure domain since they are more

3. 4 Implementation of CRUSH

likely to fail together. Although placing replicas on devices in

The implementation of CRUSH is presented in details in

distant locations ensures better data safety, communication[12] as part of RADOS. In addition to storage nodes, the de-costs also increases due to lower bandwidth between nodes

sign also includes monitors which provide information aboutacross long distance than nodes in vicinity.

the current state of the cluster and manage node joining,

CRUSH accommodates a wide range of scenarios in data-

leaving, and failures. Clients access one of the replicatedcenter storage clusters with different data replication strate-monitors first to retrieve a cluster map before accessing any

gies and underlying hardware configurations with placement

objects; later updates in cluster map can be retrieved fromrules that allow the system administrator to specify how ob-

any storage nodes. Within the storage cluster, changes injects are place with fine granularity. Figure 1 shows a sys-

cluster map are propagated rather than directly transferredtem of four level of hierarchy, the algorithm starts from thefrom the monitor to all the nodes. This approach enables

root that has pointers to four rows. The placement policy

a small number of replicated monitors to support a large

then requires the selection of one row, and within the row

number of storage nodes in datacenter clusters.

it chooses three cabinets. For each cabinet, the placement
rule selects a disk to store a replica. In this example, three

Storage nodes

Monitorsdisks are selected in three separate cabinet of the same row
to store three corresponding replicas. This approach is usefulto spread replicas across separated failure domains (separate
cabinets), but constrain them within some physical boundary (the same row).

I/O

Map update,
Failure reporting

Clients

3. 3 Bucket Types
CRUSH is designed to reconcile two competing goals: efficiency and scalability of the mapping algorithm, and minimal
data migration to maintain a balanced distribution of data.
In order to achieve these goals, CRUSH utilizes four different

Figure 2 A cluster of thousands of storage nodes. A small num-types of buckets to represent internal (non-leaf) nodes in theber of tightly coupled, replicated monitors collectively

cluster hierarchy: uniform buckets, list buckets, tree buckets,manage the cluster map that defines cluster memeber-and straw buckets. Each bucket type is based on a different

ship and placement rules. Each client performns direct

data structure and utilizes a different hashing function to

IO operations on data objects with storage nodes.

pseudo-randomly select nested item during the replica placement process. The performance of each bucket is a trade-off
between speed and efficiency of minimizing data movement

4. Node Joining and Leaving Costs

locate date in storage nodes.
In a cluster with n storage nodes, it is obvious to see that

A critical element of data distribution in a large storage

the minimum number of messages required to update thesystem is the response to the addition or removal of storage

cluster map on every node is also n. In case of CRUSH,

nodes. Hash-based algorithms typically exploit the random

as discussed in [12] the hard upper bound on the number ofnature of hash functions to achieve a uniform distribution of

copies of a single update a node might receive is proportional

data and workload to avoid asymmetries in data placement.

to µ, the number of objects on the storage nodes. The reason

When an individual device fails, the algorithm needs to redis-behind this value is that each node has to communicate withtribute the data to maintain such balance. The total data to

the nodes that store the replicas of the object. Simulation

be remapped is minimized to as much as wf ailed /W , where

results in [12] indicate that the actual number of duplicated

W is the total weight of all devices. However, when the clus-

messages is at most as much as 20% of µ. The number of

ter hierarchy is modified, as in adding or removing a storage

intra-cluster messages is therefore bounded and does not varydevice, the situation is much more complex. The mapping

significantly across different systems with the same number

process with hashing often struggle to minimize data move-of nodes and objects.

ment and can result in additional data movement beyond the4w.
W

On the other hand, the client also requires the information

For example, in CRUSH changesof an up-to-date cluster map to access data objects. Givenin the cluster map also leads to propagation of the map tothere are often more clients than storage nodes, the number

all storage nodes, which takes a O(log n) time for a clusterof messages between clients and the cluster imposes signif-of n nodes [12]. In worst case scenario, the implementation

icant costs on the system. In this paper, we focus on theof CRUSH in RADOS [12] offers a performance of less thannumber of messages exchanged between clients and storage

20% of duplicated massages. Clients that access the data innodes as the indicator for the costs of exchanged messages

a CRUSH cluster also needs to update their local copy of thein the system. In the event that a client with an out-of-datecluster map in order to correctly locate data objects in thismap tries to access the cluster, the client will have to updatecase.

its map from the incorrectly accessed node, and access thetheoretical optimum

Another performance cost incurred when a node joins ordata again to a correct storage node. The number of mes-leaves the cluster hierarchy is from re-calculating the location

sages incurred from such events is 2 for an access, which isof every data object. As discussed in 3.3, the randomness ofthe number used in our simulation.

data distribution in has-based algorithms relies on the qual-

4. 2 Re-computation Costs

ity of hashing functions, which can be expensive when there

Hash-based distribution algorithms rely on the assumption

are a large number of objects. Given that in a large-scalethat the hash function provides a good degree of randomness

storage clusters with thousands of nodes, node removals andto achieve a balanced distribution. An object’s location is de-additions occur frequently, re-calculation of objects’ position

termined by feeding this multi-input hash function a number

each time a change occurs is rather costly and can have sig-of parameters, including the object identifier and the cluster

nificant impact on the overall performance. Some algorithms

map. This approach enables the clients and storage nodes

can avoid calculating the position of every data object, while

to independently calculate the placement of objects withoutsome other must perform a full check to ensure the correct-

referring to a central management. At the same time, whenness of data placement.

the cluster map changes due to node joining or leaving, it isThis paper discusses the costs of node joining and leaving

imperative to update the local copy of the map. Given thatwith respect to two aspects: number of messages exchanged

hashing function can take up 45% of a hash-based algorithm’sand number of re-computations. We then present our ap-

computation time [2], in a system with a large number ofproach to model the costs and evaluate the effect of node

objects, node joining and leaving can lead to considerable

joining and leaving with simulation of Tree Bucket and Straw

performance penalty in terms of delay of access.

Bucket.
4. 1 Exchanged Messages

4. 3 Simulator
The simulator first builds a hierarchy map with a specified

The number of messages exchanged can be classified into

number of devices. This cluster map is used as a physicaltwo categories: intra-cluster and client-cluster. Intra-cluster

cluster for the simulation to evaluate data placement. The

messages are exchanges that occur among the storage devices

simulator then generates a large number of objects, each withwithin the cluster; client-cluster messages are exchanges be-a unique integer identifier. The objects are then assigned totween clients and the storage nodes in case of map change.

the storage devices based on the CRUSH distribution algo-

Note that clients also require an up-to-date map to correctly

rithm. A large number of clients randomly requesting objects

from the storage clusters are then created. Here, we assume

gorized in to two categories: number of computations andthat the clients access objects in a random manner, as ob-

computation time. We examine the performance of Tree

ject identifiers are typically randomly assigned in an actual

Bucket and Straw Bucket over thirty millions ojbects withsystem. During the access, the simulator then subsequently

different cluster sizes, ranging from 512 nodes to 2048 nodes.

changes the hierarchy map by removing or adding a number

The results are shown in Fig. 4 and Fig. 3.

of devices randomly. Since we assume that intra-cluster message exchanged is necessary to maintain the uniform state

Computation Time for Different Number ofNodes and Objects

throughout the cluster, the simulator only tracks the number of messages introduced by the change, including map
2,500

track of the number of re-calculation performed due to thechange in topology. The results are recorded for different
cluster sizes, number of clients, and number of objects. The
setup of the environment for the implementation is shown in
Table 2.
CPUIntel Xeon E5620 (×2)

CPU Speed

2.40 GHzCores

4 (×2) (Hyper-Threading)Memory24 GB

OSUbuntu 11.10

Programming Language

C/C++

Computation Time (ms)updates and failed client requests. The simulator also keeps

Straw 512
Straw 1024

2,000

Straw 2048
1,500

Tree 512
Tree 1024

1,000

Tree 2048
500

0

0

5

10

15

20

Number of Objects

25

30

35

Millions

Figure 3 The largest number of computation that a node must
perform in case of topology change.

Table 2 Environment Setup for the Simulator

Figure 3 reports the number of computations in different
The simulator measures the number of re-calculations required in the event of node joining and leaving with different topology sizes and configurations to evaluate the costs.
There are two aspects of this cost that are measured in thisreport. The first one is computation time, which depends
on both the number of computations and the speed of the
distribution algorithm. The second one only concerns with
the number of computations that are executed to re-evaluate
the existing distribution.
In order to simulate exchanged messages, each client is assumed to randomly access a random object from a pool of
object identifier. Although this random access pattern may
not correlate to one of a real system, it is adequately generic
to demonstrate the effect of node joining and leaving. Also,
since data objects are distributed randomly, this pattern inthe simulation is not too far off from actual access pattern.
If the client tries to access a data object with a non-up-to
date map, the storage node will sends back a new map, andthe client may need to re-access the data according to the
new information. The simulator then records the numberof exchanged map messages in the system when a node issimulated to join or leave, resulting in topology change.

5. Results

cluster sizes. In this paper, the maximum value of objects
that a node in a cluster must re-evalutated in case of topology
change is considered as the maximum number of computations per node in the given cluster. Due to the probablistically uniform distribution of both Tree and Straw Bucket
algorithms, the maximum number of computations in eachcase for the same cluster sizes varies litte. Note that withlarger clusters, with the same number of objects this costshould decrease since the number of objects per node decreases accordingly.
Fig. 4 discusses the time required for a cluster to re-evaluteall the objects. Since objects are distributed in a probablistic
uniform manner accross the nodes, the computation time ismeasured as the time from the start of the computation till
the finish of every node in the cluster. However, due to themap propapation process to propagate changes in the cluster,in reality the starting time may vary from systems to systems
with different data and cluster configuration. Therefore, tosimplify our measure, this paper reports the longest timethat some node in the cluster must undertake to re-evalutethe distribution of its objects. Such values are representative of the performance of the cluster in terms of computation
time. In Fig 4, the difference in terms of performance canbe easily observed, with the maximum of computation time

5. 1 Computation Costs
Computation cost of a distribution algorithm can be cate-going as high as 2 seconds in a Straw-based cluster of size
512 nodes.

This cost is shown in Fig. 5 with over thirty thousand clients

Maximum Number of Computeration Per Nodeclient with an out-of-date cluster map tries to access the200

Number of Computations (Thousands)

and different cluster sizes. A message is incurred when aStraw 1024

180

storage nodes. Figure 5 shows a near-linear graph of theStraw 2048

160

number of messages exchanged with different cluster sizes.

Straw 512140

Since the update is done per storage node and client basis,

Tree 2048

120
100
80

Tree 1024

the total number of messages only depend on the number ofTree 512

clients. As the number of clients increases, typically alongwith the amount of data, this message exchange mechanism

60

may lead to network congestion and performance degration.

40
20

6. Conclustion and Future Work

0
0

10

20

30

In this paper, we have presented an evaluation of the cost

Number of Objects (Millions)of node joining and leaving in a hash-based distributed storFigure 4 The time required for CRUSH to re-evaluate the posi-

age system, specifically, CRUSH. The result shows a signif-tions of every object with different cluster size. Storage

icant increase in computation cost of node addition or re-nodes are assumed to perform such calculations in par-moval when the amount of data increases. Moreover, as

allel; the graph shows the longest time of some given

the amount of data increases, the number of clients tend

nodes with the largest nubmer of objects.

to increase accordingly, leading to higher costs in terms of
messages exchanged in the system. Along with the cost ofBoth graphs show the effect of cluster sizes and number ofinter-cluster communication, the two categories of costs dis-

objects with regard to node joining and leaving costs. No-

cussed in this paper can lead to considerable performance

tice that such costs are without the cost of communication

degration in datacenter-oriented storage clusters.

within the cluster itself, which vary depending on the config-

In the future, we plan to apply similar measurements touration and the amount of data). This computation cost candifferent hashing algorithms and evaluate the trade-off be-

result in significant delay from the client’s side when trying

tween performance costs and node joining or leaving. We also

to access a server undergoing distribution evaluation, espe-

plan to propose an approach that can significantly reduce

cially when the amount of data increases at a much fastersuch costs while maintaining the favorable characteristics ofrate than capacity of storage systems.

hash-based approaches. The approach can be improved byNumber of Messages Exchanged(Thousands)

5. 2 Exchanged Messages

incorporating real-world access pattern to better simulate

Number of Exchanged Messages Between Clientsand the Cluster in case of Node Joining/Leaving

the effect of exchanging messages in a system. Finally, the35

metrics for evaluating the performance of other datacenter-

30

Straw 2048

Straw 512

20Tree 2048

10

oriented distribution algorithms in many other scenarios.

Straw 1024

25

15

measurements described in this paper can be used as the

7. Acknowledgements
This work is partly supported by Grants-in-Aid for Scien-

Tree 1024

tific Research from Japan Science and Technology Agency

Tree 512(A) (#22240005)

5
0
1,024

2,048

4,096

8,192

16,38432,768

Number of Clients
Figure 5 Number of messages exchanged in different topologies
with different number of clients. The cost increases linearly with the number of the clients

As previously discussed, our work focuses on the number ofmessages exchanged between clients and the storage nodes.

References[1] Jose Renato Santos, Richard R. Muntz, and BerthierRibeiro-Neto, “Comparing random data allocation and data
striping in multimedia servers”, ACM SIGMETRICS Performance Evaluation Review, vol. 28, no. 1, pp. 44–55, June2000.

[2] Sage A Weil, Scott A Brandt, and Ethan L Miller,“CRUSH : Controlled , Scalable , Decentralized Placement of Replicated Data”, in SC ’06 Proceedings of the2006 ACM/IEEE conference on Supercomputing, Tampa,
Florida, 2006, ACM.
[3] Sage A Weil, Scott A Brandt, Ethan L Miller, and Darrell
D E Long, “Ceph : A Scalable , High-Performance Distributed File System”, in Proceedings of the 7th symposium
on Operating systems design and implementation (OSDI’06), Seattle, Washington, 2006, pp. 307–320, USENIX Association.
[4] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, andRobert Chansler, “The Hadoop Distributed File System”,
in Proceedings of the 2010 IEEE 26th Symposium on Mass
Storage Systems and Technologies (MSST). May 2010, pp.
1–10, IEEE Computer Society.
[5] Sanjay Ghemawat, Howard Gobioff, and Shun-tak ST Leung, “The Google file system”, ACM SIGOPS Operating
Systems Review, vol. 37, no. 5, pp. 29–43, Dec. 2003.
[6] Jeffrey Dean and Sanjay Ghemawat, “MapReduce: simplified data processing on large clusters”, Communications of
the ACM, vol. 51, no. 1, pp. 107–113.
[7] Feng Wang, Jie Qiu, Jie Yang, Bo Dong, Xinhui Li, andYing Li, “Hadoop high availability through metadata replication”, in Proceedings of the first international workshop
on Cloud data management (CloudDB ’09), New York, New
York, USA, 2009, pp. 37–44, ACM Press.
[8] Anup Suresh Talwalkar, HadoopT - Breaking the Scalability Limits of Hadoop, PhD thesis, Rochester Institute ofTechnology, 2011.
[9] Fay A Y Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C
Hsieh, Deborah A Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E Gruber, “Bigtable : A
Distributed Storage System for Structured Data”, ACM
Trans. Comput. Syst., vol. 26, no. 2, pp. 4:1—-4:26, 2008.
[10] Haruto Yokota, Yasuhiko Kanemasa, and Jun Miyazaki,“Fat-Btree: An Update-Conscious Parallel Directory Structure”, in Proceed of International Conference on Data Engineering, ICDE, 1999, pp. 448–457.
[11] Ion Stoica, Robert Morris, David Karger, M. Frans
Kaashoek, and Hari Balakrishnan, “Chord: A scalable
peer-to-peer lookup service for internet applications”, SIGCOMM Comput. Commun. Rev., vol. 31, no. 4, pp. 149–
160, Aug. 2001.
[12] Sage A Weil, Andrew W Leung, Scott A Brandt, and Carlos
Maltzahn, “RADOS : A Scalable , Reliable Storage Service
for Petabyte-scale Storage Clusters”, PDSW ’07 Proceedings of the 2nd international workshop on Petascale data
storage, pp. 35–44, 2007.