Power-Aware Autonomous Distributed Storage
Systems for Internet Hosting Service Platforms
Jumpei Okoshi, Koji Hasebe, and Kazuhiko Kato
Department of Computer Science, University of Tsukuba, Japan{oks@osss.,hasebe@,kato@}cs.tsukuba.ac.jp
Abstract. We present a power-saving method for large-scale distributed
storage systems of Internet hosting services, whose prime example is avideo/photo sharing service. The idea behind our method is to periodically exchange stored data among disks in an autonomous way so as to
skew the workload towards a small number of disks while not overloadingthem. The objective of this paper is to explore a power-saving methodthat is adaptable to both constant massive inﬂux of data and changes
in data popularity. The performance is measured both in simulation andprototype implementation using a real access pattern of 20,000 public
photos on Flickr. In the experiments, we observed that our method saved
14.5–39.7% of energy, while the overall average response time was 133
ms, where 6.8–19.1% of total accesses were of disks in low-power mode.
Key words: power-saving, distributed storage systems, autonomous
control.

1 Introduction
Energy eﬃciency has become a central issue in today’s cloud computing. In
particular, as a high percentage of the total computing system’s energy is used by
the data storage systems, various attempts at reducing power use in cloud storage
systems have been proposed; e.g., [2, 3, 4, 9, 10]. These techniques are essentiallybased on an idea commonly considered in studies on power-saving in storage
systems such as MAID [1] and PDC [5]. That is, they skew the workload towards
a small number of disks and thereby enable other disks to be in low-power mode.
However, when applying this idea to recent Internet hosting service platforms
whose prime examples are YouTube1 and Flickr2 , several important issues thathave not been thoroughly investigated may arise owing to the following dynamic
aspects of the stored data.
First, most previous studies explicitly or implicitly assumed that the numberof stored data is ﬁxed, but in a real situation, enormous data quantities arecontinuously uploaded. In addition, their popularity (i.e., frequency of access)may vary at any moment. Second, previous studies often assumed that there isa speciﬁc type of central controller for data allocation to eﬀectively skew the12

http://www.youtube.com
http://www.ﬂickr.com/

2

Jumpei Okoshi et al.

workload, but this technique cannot be directly applied to large-scale storage
systems owing to a scalability issue. The objective of this paper is to addressthese issues and explore a power-saving method that is adaptable to a typicalenvironment of Internet hosting services, i.e., constant massive inﬂux of dataand changes in data popularity.
Our method is based on the idea behind MAID and PDC systems, which is
the migration of frequently accessed data to a subset of the disks. However, toenhance scalability, our method periodically exchanges data among disks in anautonomous way such that frequently accessed disks tend to gather frequently
accessed data from neighboring disks up to their capacity, and the opposite
occurs for rarely accessed disks so as to extend their time in low-power mode.
In this paper, we also consider several types of restrictions on the exchange ofdata and evaluate their eﬀect on performance in terms of power consumption,
response time, and data migration cost.
To evaluate the eﬀectiveness of our method in a more realistic situation, we
measured the performance both in simulation and prototype implementation
using a real access pattern of 20,000 public photos uploaded to Flickr, which areobservable outside the website. In the experiments, we observed that our method
resulted in energy savings of 14.5–39.7%, while the overall average response timewas 133 ms. According to our experimental results, our method eﬀectively skewed
the workload even if the data migration was conducted autonomously. On theother hand, accesses of data stored on disks in low-power mode totaled 6.8–
19.1% of all accesses, and these accesses required extra time for the spinning-up
of disks. As a major factor of this problem, we observed that the number ofaccesses rapidly decreased after one week from the upload for most of the ﬁles,and such infrequent accesses were evenly distributed. Thus, in our method, it
was diﬃcult to gather such unpopular data on some speciﬁc disks completely.
This results in a trade-oﬀ between the idleness threshold (i.e., the period fornever-accessed disks to spin down) and response time.
This paper is organized as follows. Section 2 presents related work. Section 3
describes the design of the proposed storage system. Section 4 gives the resultsof preliminary investigations of access patterns of public uploaded photos on
Flickr, which are used in both simulations and in experiments with a prototype.
Sections 5 and 6 present the simulation results and the evaluation of the prototype implementation. Finally, Section 7 concludes the paper and presents futurework.

2 Related work
There have been a number of studies on reducing storage power consumption.
A common feature of many of the techniques proposed in the literature is thatthey skew the workload, and they can be classiﬁed into the following categories
according to variations in their approach.
The ﬁrst category, which includes MAID [1] and PDC [5], focuses on the
popularity and concentrates popular data on speciﬁc disks. The second category,Power-Aware Autonomous Distributed Storage Systems

3

as typiﬁed by Pergamum [8], uses NVRAM to extend the low-power mode period
by caching data to a write store. The ﬁnal category considers redundancy (i.e.,
data replication). In DIV [6], original and redundant data are separated onto
diﬀerent disks, thereby allowing read/write requests to be concentrated on thedisks with the original data. In Hibernator [13] and PARAID [11], data arecollected or spread to adapt to changes in operational loads.
Although the above studies restricted their scope to storage systems consisting of a relatively small number of disks (typically, up to several dozen), recent
works such as those of Harnik et al. [2], Kaushik et al. [4], Verma et al. [9],Vrbsky et al. [10], and our previous work [3] address power-saving in large-scale
distributed storage on the basis of the existing skewing technique. Our research
can be thought of as a direct successor to these studies based on the approachtaken in the ﬁrst category, but the main motivation is to explore power-saving inan environment where a huge number of data are continuously uploaded and thedata access frequency varies at any moment, whose prime example is Internet
hosting services.

3 System Design
Our proposed storage system is composed of several thousand (possibly heterogeneous) disks, each of which is classiﬁed into one of three groups: Group A,
Group B, and the Empty disk pool. (See also Fig. 1 for the graphical presentation.) Each disk autonomously travels among these groups (depicted by the
thick arrows in the ﬁgure) depending on its capacity in the following way.

Clients......
Write (Upload )ReadGroup A
Disk

Group B
Disk......

Disk

Disk......

Disk

Disk

Disk

Disk

. . . . . . Disk

Exchange data

Supply empty disks

Disk

Move
full disks

Disk
Remove empty disks

Empty Disk PoolFig. 1. Architecture of the System

The system is assumed to have many empty disks (with unique IDs), which
are stored in the Empty disk pool for future increases in stored data. Initially,some of the empty disks in the pool are moved to Group A, and the data uploaded

4

Jumpei Okoshi et al.

by the clients are always written to these disks. If a disk in Group A becomes
full, then it moves to Group B and a new empty disk is supplied to Group A
from the disk pool.
In Group B, to skew the workload, each disk periodically (e.g., once everyhour) exchanges part of its stored data one-for-one following four steps.
Step 1: The disk (say, D1 ) with the smallest ID randomly chooses a disk (say,D2 ).
Step 2: D1 and D2 exchange information of their current workloads.
Step 3: If D1 is more popular than D2 , the least frequently accessed ﬁle (say,fL ) on D1 is exchanged for the most frequently accessed ﬁle (say, fH ) on
D2 , and this process is repeated while the frequency of access of fL is lessthan that of fH .
Step 4: D1 issues an instruction to the next disk.
When implementing this process, the access frequency of popular disks gradually
increases, until the capacity of the disks is exceeded, by gathering popular datafrom the neighboring disks, and the opposite process is carried out for unpopulardisks so as to extend their time spent in low-power mode. During this exchange
process, if a disk eventually becomes empty, it is removed from Group B andqueued in the Empty disk pool for the supply of disks to Group A.
Here, we may consider restrictions on the exchange to reduce migration cost,such as limits on the number of exchangeable ﬁles during any one exchange, thenumber of migrations for each ﬁle, and the scope of exchangeable disks. In thisstudy, we evaluated the eﬀects of restrictions on the energy performance of disks,
which are described in later sections.
The underlying lookup service used by clients to access data is managed bya distributed hash table, such as Chord [7]. However, for adaption to our system
in which data are frequently moved among disks, we use pointers in a list of
key-value pairs to indicate the current disk on which data are stored. That is, if
a ﬁle f1 stored on disk D1 migrates to D2 , the key of f1 is associated with thedestination of migration (i.e., D2 ) in D1 ’s list of key-value pairs. In addition,
to avoid multiple hops from pointer to pointer, each ﬁle is assumed to haveinformation of the ID of the disk originally storing the ﬁle as metadata, and if
f1 originally stored on D1 and moved to D2 is moved further to D3 , the pointerof D1 is rewritten so as to directly indicate the current position.

4 Data Access Tracing in Flickr
To evaluate the eﬀectiveness of our method in a realistic situation, as a preliminary study, we traced access patterns of photo data uploaded to Flickr, which
is one of the largest photo sharing services in the world.
In this preliminary study, we randomly selected 20,000 photos and traced thecumulative number of accesses for each ﬁle every hour over two weeks with API
s provided by the website. Owing to limitations of accessible observable data, allPower-Aware Autonomous Distributed Storage Systems

5

(A)
150

(B)
2000

The number of acceses

The number of accesses

the selected photos are public, although the website has supposedly around fourtimes as many private photos as public photos (according to a Flickr report andour observations).

100

50

0
0

0.5

1
Photos

1.5

2
x 10

4

1500
1000
500
0
0

100

200
300
Time (hour)

400

Fig. 2. (A) Distribution of popularity and (B) cumulative number of total accesses of
20,000 ﬁles on Flickr

Figure 2-(A) shows the distribution of popularity for all photos at a lapse of
200 hours, while (B) shows the change in the cumulative number of total accessesof all the ﬁles over 400 hours. Figure 2-(A) shows that the ﬁle access frequencies
were highly skewed, with 69.3% of ﬁles never being accessed after upload, while
the number of accesses of the most popular ﬁle is 133. Figure 2-(B) shows strong
negative correlation between the frequency of access and elapsed time. Moreprecisely, the result shows that the frequency of access became highest (1748accesses per hour) at a lapse of 3 hours and then rapidly decreased, eventuallyreaching 219 accesses per hour at a lapse of 17 hours. However, the frequency ofaccess did not change after 200 hours, with most ﬁles being accessed once perhour or not at all. We here note that it is diﬃcult to forecast which ﬁles will beaccessed in a short time from the past access pattern. This makes it diﬃcult tocompletely gather the accesses to some speciﬁc disks in an eﬀort to avoid accessof disks in low-power mode.

5 Simulation Results
To understand the eﬀectiveness of our method for storage systems consisting ofseveral thousands of disks, we ﬁrst evaluated the running time of disks and thefrequency of access of disks in low-power mode.
Parameters and settings. In the evaluation presented in this section, we considered the following system. Group A consisted of a single disk and Group B
consisted of up to 1500 disks whose number increases depending on the upload. Each disk in Group B required 5 seconds for spin-up where it had been in
low-power mode. We set the idleness threshold as 30, 60, 90, and 120 seconds.

6

Jumpei Okoshi et al.

The workload in the simulations was based on the access traces obtained by
the observation of Flickr described in Section 3. In our simulation, to determine
the trace of each ﬁle, we chose at random from the set of 20,000 real traces when
uploaded. However, as our traces were observed for 400 hours, we expanded thereal traces to 2400 hours by repeating the access pattern from 200 hours to 400
hours for the period after 400 hours.
In the simulations, we ﬁrst measured the running time to evaluate the impact
of both the conﬁgurations of idleness threshold and restrictions of data exchangeon the power consumption then measured the number of accesses of disks in lowpower mode in some conﬁgurations.

1
0.95

Running time

0.9
0.85
0.8
0.75
0.7
0.65
0

no−exchange
30
60
90
120
500

1000
1500
Time (hour)

2000

Fig. 3. Running time in some conﬁgurations of idleness threshold

Impact of exchange and idleness threshold on power consumption.
Figure 3 shows the change in running time for diﬀerent conﬁgurations of idleness threshold. The ﬁgure shows the relative time when the running time in thecase that all the disks are always active is equal to one. In this ﬁgure, the “noexchange” conﬁguration means that the disks spin down after an idle time (30seconds) without data exchange and, in other conﬁgurations, disks spin down
after an idle time (30, 60, 90, and 120 seconds) with data exchange. We observed that power consumption after 2400 hours was reduced by 14.5–39.7% ineach conﬁguration. From this result, we observed that the data exchange was
eﬀective in reducing power consumption as demonstrated by comparing the “noexchange” conﬁguration and all the conﬁgurations with data exchange, among
which the “30” conﬁguration was the most eﬀective.
Impact of the restrictions of data exchange on power consumption.
Figure 4 shows the change in running time for four conﬁgurations, which diﬀeredin terms of the application of two restrictions. To denote these conﬁgurations,Power-Aware Autonomous Distributed Storage Systems

7

1

Running time

0.95
0.9
0.85
0.8
0.75
0.7
0

00
01
10
11
500

1000
1500
Time (hour)

2000

Fig. 4. Running time for some restrictions of data exchange

we use two digits b1 and b2 , where b1 indicates application of a limit on theexchange range (in the case that b1 = 1, one disk can change ﬁles with only 20
speciﬁc disks), while b2 indicates application of a limit on data migration (in thecase that b2 = 1, the migration capacity is limited to 10% of each disk space).
The ﬁgure shows the relative time when the running time in the case that all
the disks are active is equal to one. Here, the idleness threshold is ﬁxed as 60
seconds in each conﬁguration. In this simulation, we observed that our method
still reduced power even if one of the restrictions was introduced, which may
reduce the migration cost. In addition, under these parameter settings, the limiton the exchange range (i.e., b1 ) was stronger than the limit on data migration(i.e., b2 ), which would change depending on a given trace.
Accesses of disks in low-power mode. Figure 5 shows the change in ratioof the access of disks in low-power mode among all accesses in the same conﬁguration as in the case of Figure 3. We observed that 6.8–28.7% of accesses wereof disks in low-power mode. This result means that 6.8–28.7% of accesses needextra time for spinning-up. However, it should be emphasized that the data exchange realized a 9.5% reduction in accesses of disks in low-power mode, as seen
by comparing the “no-exchange” conﬁguration and “30” conﬁguration, which
had the same idle time but diﬀered in terms of there being data exchange.

6 Experiments on an Implementation
We conducted an experiment on the current prototype implementation of our
proposed system to evaluate the applicability of our method to a real system. We
measured the response time and the cost of data migration in an environment
where the system workload was the same as that in the simulation.

Jumpei Okoshi et al.

Ratio of access to the spin−down disks

8

0.35
0.3
0.25
0.2

no−exchange
30
60
90
120

0.15
0.1
0.05
0
0

500

1000
1500
Time (hour)

2000

Fig. 5. Ratio of access of disks in low-power mode

Our prototype consisted of 50 PC servers (where one server was used for theclient and the other servers were used for Group B), each of which was equippedwith a Dual Xeon 3.60 GHz CPU, 2 GB memory, and a single 36 GB SCSI disk.
For our prototype, owing to the limitation of our experimental environment (i.e.,the bandwidth of diﬀerent servers), we evaluated the response time of data accessby measuring the time from sending a request until the data were loaded into
the memory of the server. In addition, no underlying lookup service to accessdata was implemented in our prototype. Thus, in the experiments, the data wereaccessed by their storing server.
Parameters and settings. In our experimental environment, although the realcapacity of a disk was 36 GB, we assumed that the capacity was 500 GB, which
was emulated by only accessing one ﬁle. In addition, because it was diﬃcult tospin up or spin down disks in the current system conﬁguration, we realized theseactions by letting the server wait before accessing the disk. The two parameters
were spin-up time of 5 seconds and idle time before spin-down of 60 seconds.
(According to simulation analysis, this conﬁguration was the best in terms of
response time and power consumption.)
Response time. Figure 6 shows the change in response time. The ﬁgure showsthat the overall average response time is 133 ms. However, we observed thatsome responses were delayed and required more than 5 seconds after 15 hours,
among which the worst took 18,307 ms. This result means that some accesses
were of the spin-down disk and had to wait for the disk to spin up or for thecompletion of another ﬁle access.
Migration cost. Table 1 shows the migration cost evaluated by measuring thenumber of ﬁles that migrated per data exchange and the elapsed time. Note that
this result was for 48 hours and the migration cost would decrease gradually afterPower-Aware Autonomous Distributed Storage Systems

Response time (msec)

2

1.5

x 10

9

4

minimum
averagemaximum

1

0.5

0

10

20
30
Time (hour)

40

Fig. 6. Response time
Table 1. Migration cost
The number of exchanged ﬁlesTime (s)minimum median average maximum
21691 13583 163079
0.2
170
1426
17123

48 hours. In addition, owing to the limitation of our experimental environment,
times in Table 1 are predicted values. This result shows that the required time
for most of the migration was short relative to the frequency of exchange (i.e.,
once every hour) and would have no adverse impact on the performance.

7 Conclusions and Future Work
We presented a power-saving method for large-scale distributed storage systems, especially those on Internet hosting service platforms. The idea to skew
the workload is similar to that of MAID and PDC, but to adapt to the constantmassive inﬂux of data and changes in data popularity, our approach periodicallyexchanges data among disks autonomously, instead of introducing a central controller to manage the relocations. We also evaluated the performance both insimulations and prototype implementation for a real access pattern of 20,000
public photos on Flickr. We observed that our method saved 14.5–39.7% energy,
even if we introduced several restrictions on data migration. The ratio of accessesof disks in low-power mode was 6.8–19.1%, and the overall average response timewas 133 ms.
The results of this research suggest that our autonomous approach can eﬀectively skew the workload, but some accesses require extra time to wait for the
spinning-up of disks in low-power mode. This results in a trade-oﬀ between the10Jumpei Okoshi et al.

idleness threshold of disks, which may increase the power consumption, and theperformance of the response time. To more eﬀectively skew the workload, weshould investigate a radical solution that possibly considers eﬀective allocationsof replicas. In addition, we should thoroughly estimate the cost of underlying
lookup services. These topics shall be investigated in future work.

References
1. D. Colarelli and D. Grunwald. Massive arrays of idle disks for storage archives. In
ACM/IEEE Conference on Supercomputing, pp.1-11, 2002.
2. D. Harnik, D. Naor, and I. Segall. Low power mode in cloud storage systems. Paralleland Distributed Processing Symposium, International, pp.1-8, 2009.
3. K. Hasebe, T. Niwa, A. Sugiki, and K. Kato. Power-Saving in Large-Scale Storage
Systems with Data Migration. IEEE International Conference on Cloud Computing
Technology and Science (CloudCom’10), pp.266-273, 2010.
4. R. T. Kaushik and M. Bhandarkar. GreenHDFS: towards an energy-conserving,storage-eﬃcient, hybrid Hadoop compute cluster. In 2010 international conference
on Power aware computing and systems (HotPower’10), pp.1-9, 2010.
5. E. Pinheiro and R. Bianchini. Energy conservation techniques for disk array-based
servers. In International Conference on Supercomputing, pp.68-78, 2004.
6. E. Pinheiro, R. Bianchini, and C. Dubnicki. Exploiting redundancy to conserve
energy in storage systems. In ACM SIGMETRICS Conference on Measurement
and modeling of computer systems, pp.15-26, 2006.
7. I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A
scalable peer-to-peer lookup service for internet applications. In ACM SIGCOMM,
pp.149-160, 2001.
8. M. Storer, K. Greenan, E. Miller, and K. Voruganti. Pergamum: Replacing tapewith energy eﬃcient reliable, disk-based archival storage. In USENIX Conference
on File and Storage Technologies (FAST’08), pp.1-16, 2008.
9. A. Verma, R. Koller, L. Useche, and R. Rangaswami. SRCMap: energy proportional
storage using dynamic consolidation. In 8th USENIX Conference on File and Storage
Technologies (FAST’10), pp.154-168, 2010.
10. S. V. Vrbsky, M. Lei, K. Smith, and J. Byrd. Data Replication and Power Consumption in Data Grids. In 2010 IEEE Second International Conference on Cloud
Computing Technology and Science (CloudCom’10), pp.288-295, 2010.
11. C. Weddle, M. Oldham, J. Qian, A. Wang, P. Reiher, and G. Kuenning. PARAID:
A gear-shifting power-aware RAID. In USENIX Conference on File and Storage
Technologies (FAST’07), pp.245-260, 2007.
12. X. Yao and J. Wang. RIMAC: a novel redundancy-based hierarchical cachearchitecture for energy eﬃcient, high performance storage systems. In ACM
SIGOPS/EuroSys European Conference on Computer Systems, pp.249-262, 2006.
13. Q. Zhu, Z. Chen, L. Tan, Y. Zhou, K. Keeton, and J. Wilkes. Hibernator: helping
disk arrays sleep through the winter. In ACM symposium on Operating systems
principles, pp.177-190, 2005.