Facility Information Management on HBase:
Large-Scale Storage for Time-Series Data
Hideya Ochiai
Hiroyuki Ikegami
Yuuichi Teranishi
Hiroshi Esaki
The University of Tokyo / NICT The University of Tokyo NICT / Osaka University The University of Tokyo
jo2lxq@hongo.wide.ad.jp
ikegam@hongo.wide.ad.jp teranisi@cmc.osaka-u.ac.jp
hiroshi@wide.ad.jp
Abstract—A very large number of sensors on facilities suchas HVAC, light control systems and electric power meters, periodically submit their status information to Cloud platforms thesedays. As the amount of data can easily get petabyte scale, we must
consider the use of distributed application layer storage for managing such facility information, which is often formatted on timeseries data. This paper describes FIAPStoragePeta, petabyte scalestorage for facility information access protocol (FIAP), proposing
the architecture and the scheme of such data management on
HBase. In this work, we have identified three requirements tothe design of HBase row keys for implementing this storage using
HBase. Though, we have not finished petabyte scale experiments,
our preliminary evaluation results have shown good performancefor managing large scale facility information. It has achieved
scalable data retrieval on the data of 10 million sensors withproperly balancing loads on distributed data storages.
Keywords—Internet of Things, Time-Series Data, HBase,
HadoopI.

I NTRODUCTION

In the age of the Internet of Things, a very large numberof sensor readings gather at Cloud data platforms. In facility
management applications, the platform should collect frommillions of sensors on thousands of buildings every minute.
The amount of sensor readings collected in the platform caneasily get one petabyte.
HBase[5] is known as distributed petabyte scale storage,that is widely-applicable, highly-available and highly-dynamic.
System administrators can operate it from gigabyte scalestorage, but by adding computers, they can extend it even toa petabyte scale. Here, the use of HBase and the design of
HBase keys are open to the users. HBase itself just allows
to persistently store values with lexicographic-ordered keys.
Whether the user can take the advantages of HBase or not, istotally dependent on the manner of use for their applications.
In this paper, we focus on the use of HBase for facility
information management, especially time-series data collectedfrom sensors or facilities such as HVAC (Heating, Ventilation
and Air-Conditioning), light control systems, environmental
monitors and electric power meters. Targetting at managing
thousands of buildings on a Cloud platform, we have launcheda project of developing FIAPStoragePeta – intended ”petabyte
scale storage for facility information access protocol (FIAP)”.
This paper describes the architecture and design of FIAPStoragePeta, proposing a facility information management schemeon HBase.

Fig. 1. Facilities generate sequences of ”time-value” pairs: i.e., time-series
data. FIAP manages each sequence by a unique identifier called ”Point ID”.

FIAP[7] is a communication protocol for data-centricbuilding automation systems. It provides data storage formanaging time-series data collected from facilities, whichlooks like Figure 1. In FIAP, a sensor (or an element of a
facility) provides a sequence of ”time-value” pairs. The value
may be a string type, for example, if the sequence represents
the change of HVAC working mode. It may be a decimal typeif the sequence represents the change of temperature. Eachsequence has a unique identifier, which we call ”Point ID”, corresponding to the sensor. Though the examples are monitoringvalues, FIAP applies this data structure even to actuator status
and calculated values: e.g., periodically summarized values byhour or day. The time intervals between values are not always
the same.
This paper proposes the scheme of putting such timeseries data onto HBase. Here, we design HBase keys sothat FIAPStoragePeta (1) distributes the values on HBase
appropriately, (2) optimizes the performance of sequential-readand (3) gets scalability for the increase of managed Point IDs.
In this work, we have carried out preliminary experimentswith five rack mount servers, and evaluated the performance
regarding to these aspects.
We recognize that OpenTSDB[9] allows the managementof time-series data on HBase. However, the major applicationof OpenTSDB is the management of working metrics ofcomputer systems (such as network gear, operating systems
and applications), and the design of HBase keys are forthese applications, and not optimized for FIAP’s data model.
And, though it may just an implementation issue, OpenTSDB
does not support string type, which is necessary in facility
information management. This was another motivation that
we started FIAPStoragePeta development with studying theimplementation of OpenTSDB.
This paper is organized as follows. The next section
describes FIAP’s time-series data model, which we target at inthis paper. Section III provides the architecture and the designof FIAPStoragePeta. We provide our preliminary evaluation
works in Section IV. Section V shows the related works anddiscussions. Section VI provides the conclusion of this paper.
II.

FIAP T IME -S ERIES DATA M ODEL

Facility information access protocol (FIAP)[7] defines amodel of time-series data called ”Point”. In the operation of
FIAP, a Point is associated with a sensor or an actuator of a
facility, which values change over time.
Let P be a set of Point IDs that data storage of FIAP
manages. A Point p ∈ P has a set of values, which we denoteby V (p) in this paper. Here, a value v ∈ V (p) has two parameters: time and content. In the following discussions, v.time
represents the timestamp of the value, which is unique in V (p).
v.content represents the value associated to the timestamp. In
the operation, Point IDs are provided by universal resource
identifiers (URIs) often with path-based data structures. For
example,p = http://gutp.jp/EngBldg2/10F/102B1/Humidity
may represent a sequence of humidity values of room 102B1,
10th floor of Engineering Building 2 in the University of Tokyo(operated by gutp.jp).
In FIAP, we specify a range of datasets by a Query (wedenote it by q in this paper). A Query q may contain multiple
keys. Here, we denote the set of the keys associate to q as
K(q). Each key k ∈ K(q) specifies a point ID, and time
range. We denote the point ID specified by a key as k.id,
and the time-range is specified with the combinations of k.eq,
k.lt, k.gt, k.lteq, k.gteq, k.neq, k.select. Here, eq represents”equal to”, lt represents ”less than”, neq represents ”not equalto”. Thus, gteq represents ”greater than or equal to”. selectspecifies one of {maximum, minimum} to select only one
value at the maximum time edge or the minimum time edge inthe specified time range by the other parameters. In this way,
we can specify and get values from FIAP’s data storage (wecall this data storage FIAPStorage) using queries. For example,•

Query q1 (k.id = p) specifies all the values of point p.

•

Query q2 (k.id = p ∧ k.select = maximum) specifies
the latest value of point p.

•

Query q3 (k.id = p ∧ k.gt =2014-01-01∧k.lt =201402-01) specifies the values between 2014-01-01 and2014-02-01 of point p.

•

Query q4 (k0 .id = p0 ∨ k1 .id = p1 ) specifies all thevalues of points {p0 , p1 }.

The original FIAP[7] defines WRITE and FETCH procedures. However, to make the discussion simple in this paper,
we define SET and GET methods instead of WRITE andFETCH. Then, a Gateway sends sensor data to a FIAPStorage
by SET, and an UI-terminal retrieves stored data by GET (seeFigure 2). The Gateway and UI-terminal are the terms definedin FIAP[7]. Formally, we can define SET and GET as follows.

Fig. 2. FIAP time-series data model. FIAPStorage provides data storage with
GET and SET methods. The Gateway sends the values to FIAPStorage by SET
method, then it adds the values to the persistent memory. The UI-terminal can
GET data with specifying the range of data by query. This paper redefine
WRITE and FETCH procedures of FIAP as SET and GET for simplicity.

•

SET: Let S be a set of points and associated values
stored in FIAPStorage, and Let G be a set of points
and associated values sending by a Gateway. The SET
method is a process that adds G to S, that is:
S := S ∪ G•(1)

GET: The GET method is a process that takes points
and values match to the query q, that is:
A := q × S(2)

Here, A is a set of points and associated values
returned to the UI-terminal which has issued q. ”×”
cuts out the dataset by the query.
III.

FIAPS TORAGE P ETA

We have designed large-scale time-series data storage on
HBase in the FIAP architecture. FIAPStoragePeta (intending
peta-scale storage for facility information access protocol) is
a code name of our project for developing such data storage.
A. Architecture
FIAPStoragePeta is organized with a FIAP protocol stack,driver for HBase, and a HBase platform (see, Figure 3).

Fig. 3. Architecture of FIAPStoragePeta. The driver stores the sensor data SET from Gateways onto their HBase platform. UI-terminals GET data retrieved
from the HBase by the driver.

The driver implements SET and GET methods, handling theprocess of data locating, storing and retrieving on HBase.
Gateways send their data to FIAPStoratePeta by SET, and UIterminals retrieve data by GET.
Here, HBase is operated by multiple servers providing keyvalue store. Though the key of HBase is three dimensional(i.e., row, column, time)[1], we decided to use only the row as
the key of the FIAP’s data values for the following reasons;(1) v.content can be uniquely pointed by p and v.time;(2) the time of HBase is basically used for content version
management (and time-range search seems not supported);(3) v.time is not necessarily mapped to columns though
OpenTSDB puts the second of v.time for columns. Thus, inprinciple, we can use p and v.time as the row key of HBase
to uniquely coordinate v.content.
B. HBase Row Key Design for Time-Series Data
In the principle of HBase, the design of the row key
structure is open to the users. Whether we can get full
performance or not is totally depend on the design of it. We
have identified the following three items as important featuresthat FIAPStoragePeta should have to get full performance.
••

Do not make hotspots on the distributions of therow keys: HBase divides the whole row-key space
into tablets (some amount of range of row keys), andassigns those tablets to data nodes. If there are some
hotspots in the distributions of the row keys, somespecific data nodes get in charge of, and often suffer
from, archiving larger amount of data and processing
many requests. Thus, it is important to make sure thatthe distribution of the row keys does not have such a
hotspot.
Put the values of each point locally, and keep themsorted by time on the row keys: The row of HBase
is a ordered key. For the following reasons, the timeseries data of the same point should be put closely

and sorted by time-order; (1) it allows the driver todirectly specify the range of row keys corresponding
to the time-range of the query, (2) it allows data readin block, (3) it allows the data to be kept sorted after
the retrieval from HBase.
•Get values without data filtering process evenon a large number of points: We generally apply
Hash to identifiers (i.e., Point IDs) to make nearuniform key distributions. But, as it is recommended
to keep the length of row keys short to decrease theoverhead, the bit-length of Hash should be kept short.
However, this leads to hash-conflict, especially when
the number of identifiers become large, and wehave to apply filtering process when retrieving data. If
filtering process is necessary to remove such conflicts,
it may result in the bottleneck of read performance.

Based on the discussions above, we have designed thestructure of the row key as Figure 4.
To generate row keys, we have defined two functions:
Hash and Seq for point p. Hash(p) takes top 16bit of the
MD5 hash value of p, and Seq(p) is the unique 24bit serial
number corresponding to p. v.time is given in second in
Unix timestamp in UTC. From the most significant bit (MSB),
the row key is organized with Hash(p), Seq(p) and v.time.
The least significant bit (LSB) represents one second; i.e., theincrease of the last bit means the increase of one second.
Under this definition, P should be distributed to the spaceof HBase row keys by Hash, meaning that in many cases wecan avoid the hotspots discussed above. As v.time is placed at
LSB-side, the values of p (i.e., V (p)) should be put locally, andthe time-order is kept on the HBase row’s order. As Seq(p)is a unique identifier corresponding to p, we can guarranteethat the pair of Hash(p) and Seq(p) (with 40 bits) should becorresponding to p. Though the maximum limit of points that
this FIAPStoragePeta can manage is 224 , if we have applied
40bit Hash instead of these pairs, we must worry about theFig. 4. Row key design for value v of point p. This row key locates v.content
with Hash(p), Seq(p) and v.time on HBase. Under this definision, row keys
should be distributed by Hash, but never conflict because of Seq, and V (p)should be put locally.

hash conflicts because 40 bits are not large enough for 224
elements.
C. Tables for FIAPStoragePeta
FIAPStoragePeta has three major tables on HBase: point
table (TP ), ID map table(TID ) and value table (TV ). TP has
one key and two columns named id and maxtime. TP .key isa point p managed in this FIAPStoragePeta, TP .id is Seq(p)corresponding to the point, and TP .maxtime is the maximum
timestamp of the values associated to the point. TID is forlooking up p from Seq(p). This has one key and one column
named pi. TID .key is Seq(p) and TID .pi is the point p
corresponding to Seq(p). TV contains all the values associatedto all the points of TP . It has one key and one column
named value. TV .key is the row key defined in the previous
sub section (i.e., Hash(p), Seq(p), v.time), and TV .value
contains the value (i.e., v.content).
D. SET and GET Procedures
As HBase allows parallel execution of processes on data,
we must carefully consider that SET and GET processes may
be done in parallel. For example, a client may call a SET
method adding new point ID – during another SET process
adding another new point ID. We must guarrantee that Seq(p)should be unique even when this kind of conflicts happen.
With carefully considering this issue, we have implemented
the procedures of SET and GET as follows.
••

SET procedure: For point p and associated value
v, SET first tries to lookup the row key from TP .
If it succeeds, it generates a row key for TV with
Hash(p), Seq(p) = TP .id, v.time and put v.content
to TV .value. Here, if TP .maxtime is smaller thanv.time, it updates TP .maxtime with v.time. If it fails
to lookup the row key from TP , it gets last id, puts
TP .id = last id + 1, TP .maxtime = v.time where
TP .key = p and increments last id with locked, andputs TID .pi = p where TID .key = last id + 1. Here,last id is a globally synchronized integer.
GET procedure: For query key k ∈ K(q), GET specifies the range of row keys based on k.eq, k.lt, k.gt,
k.lteq, k.gteq, k.neq and Hash(k.id) and Seq(k.id),and scan the row keys on TV . If the k specifies thelatest value of p (e.g., with select = maximum
property only), GET specifies the time range after
TP .maxtime, and picks up the latest one after the retrieval. This is because TP .maxtime may not always
be synchronized to the latest one in the distributed
environment.

Fig. 5. Experiment configuration for FIAPStoragePeta preliminary evaluation.
The name node operated HBase master, ZooKeeper and Hadoop name node.
The data nodes operated HRegion and Hadoop data node. The web nodeoperated the driver of FIAPStoragePeta (see, Figure 3). The performance of
GET was measured in the web node.

IV.

P RELIMINARY E VALUATIONThis section provides our preliminary evaluation result
regarding to (1) data distribution on HBase, (2) speed ofsequential read and (3) scalability to point numbers. These
three evaluation metrics corresponds to the discussion ofSection III.B.
A. Experiment Setting
In our evaluation, we carried out experiments with five rack
mount servers: using three servers as data nodes, one server
as name node and the other server as web node. We installed
CentOS 6.4 Linux 2.6.32-358 on those servers and software
components as Figure 5. Here, each server was connected with
Gigabit Ethernet.
Each server had a Xeon E5-1410 (2.8GHz 4Core) processor, 32GByte main memory and 2TByte hard drive for web
and name nodes, and 8TByte hard drive for data nodes.
We have implemented SET and GET procedures in Java,
carried out experiments and measured the execution timeon the web node. In the experiment, we have changedsome configuration parameters of HBase from the default
settings as Dfs.replication=2 and hbase.snapshot.enable=true.
IN MEMORY option for HBase tables was false, and theblock size of HadoopFS was 64KByte.
B. Data Distribution on HBase Platform
As we have designed to avoid hotspots in the distributionsof row keys, the data should be distributed onto data nodes
equally. To evaluate this, we have generated points and values
as Table I, SET them to the FIAPStoragePeta, and checked
the disk usage. Case 1 generated 1000 points and each point
had 525600 values. This corresponds to one year data of 1000
points with every minute sampling. Case 2 generated 5000
points in addition to the dataset of Case 1, which means that it
had 6000 points totally. In the same way, we generated dataset
for Case 3 and Case 4.
Figure 6 shows the disk usage of the data nodes. Data
were distributed to those nodes almost equally. The differencesamong data nodes were within 2.5%.

TABLE I.
Add toCase 1Case 2
Case 3
Case 4

Case 1Case 2
Case 3

SET TING OF P OINTS AND VALUES INTO FIAPS TORAGE P ETA FOR DATA DISTRIBUTION EVALUATION : C ASE 1 - 4
Point ID Format
http://a.hongo.wide.ad.jp/MPM/VA[x]http://fiap-gw.gutp.ic.i.u-tokyo.ac.jp/EngBldg2/10F/102A1[x]
http://fiap-gw.gutp.ic.i.u-tokyo.ac.jp/EngBldg2/10F/102A1[x]
http://www.hongo.wide.ad.jp/MPM/VA[x]Fig. 6. The result of data distribution on data nodes for Case 1 - 4. The
differences among data nodes were within 2.5% for all the cases.

Generation Rule[x]=0000-0999[x]=0000-4999[x]=5000-9999[x]=0000-0999

Total Point IDs
1000
6000
11000
12000

Total Values
525600000
3153600000
5781600000
6307200000

Fig. 7. Execution time of 10000 values sequential read. The average execution
time was 48.8[ms], which was 3.01 × 103 times faster than random-read.

C. Execution Time of Sequential Read
As we have designed to put the same point’s values locallyand kept sorted, the sequential read should be done quickly. To
evaluate this, we have carried out read experiments as follows.
With the data set of the Case 4 of the previous experiment,
we have posted a time-range query that specifies 10000 values
on a single point ID. We measured the time spent for thequery execution. We carried out this measurement 30 times
continuously on different point IDs; we call each measurement”SEQ-READ Test”. To compare this sequential read process,we have also made random read experiment. In the random
read, we have specified 10000 points with 10000 keys andone value for each point with eq property; we call eachmeasurement ”RND-READ Test”.
Figure 7 and 8 shows the experiment results of sequentialread and random-read respectively. The average execution
times were 48.8[ms] for sequential case and 147[sec] forrandom case. Here, in this calculation, we have excluded
the execution time of the first tests, because the first tests
have clearly taken longer time compared to the successive
tests: probably because of loading necessary software on main
memory at the first execution. The result shows that sequential
cases performed 3.01×103 times faster execution than randomcases, though in this preliminary experiment, we can say thatthe improvement for the random read is still required.
D. Scalability to the number of managed points
As we have designed HBase row key to allow data retrieval
without filtering process even Hash conflict occurs for a large
number of points, FIAPStoragePeta should have scalability
especially in GETting values. To evaluate this, we have carriedout an experiment as follows. We have cleared the HBase andFig. 8. Execution time of 10000 values random read. The averate execution
time was 147[sec].

set up points as ”http://gutp.jp/FSP/[x]”, where [x] = 00000009999999. In the experiment, we have generated first 10000
points and tested the performance of read, then generated additional 90000 points (totally 100000 points) and tested the read
performance. In the same way, we have generated 10000000(i.e., 10 million) points finally and tested the performance onthat dataset. Here, each point had 60 values.
In the read process, we chose 1000 points at random andput them in a query. We did not specify any other parameters
for the keys, meaning that the query specified all the values
associated to the points. The total number of values to retrieve
by one query was 60000.
Figure 9 shows the result of the average query execution
time on the number of managed points in FIAPStoragePeta.
The execution time was about 1.1[sec] when it was less than
1 million points. From about 1 million points, the execution
time increased up to 1.9[sec] on 10 million points. This result
indicates that FIAPStoragePeta has great scalability on the

VI.

Fig. 9. The average execution time on a large number of managed points in
FIAPStoragePeta.

increase of point IDs.

V.

In this paper, we presented FIAPStoragePeta, proposing
a facility information management scheme on HBase. We
described a model of time-series data used in facility management applications, and proposed the design of HBase row
keys, table schemas and SET/GET procedures. To implement
the following three requirements identified in this work; (1)no hotspots should be made in HBase row keys; (2) values ofthe same point should be put locally and sorted; and (3) allow
to get values without filtering even on large number of point
IDs, we used Hash(p), Seq(p) and v.time in this order for
the HBase row key of the value table TV . According to ourpreliminary evaluation, (1) it appropriately balanced the loads
of storing time-series data, (2) it achieved fast execution on
the sequential read of the same point, (3) and it showed great
scalability in retrieval time even on 10 million point IDs.
R EFERENCES[1]R ELATED W ORK AND D ISCUSSION

Time-series databases (TSDB) have been developed often
in the context of network and system monitoring applications.
One of the most traditional TSDBs would be the rrdtool[8],which is popular among network system administrators. With
the rapid increase of time-series data with the advent of new IT
systems, fine-grained real-time monitoring from thousands ofhosts has become necessary. OpenTSDB[9] and tsdb[2] havebeen developed to satisfy these demands. Kacznmarski et al.
[6] have studied the application of OpenTSDB to the realtime monitoring of content delivery networks. Groves et al. [3]have used OpenTSDB for distributed and push-based network
monitoring at Yahoo!. OpenTSDB is built on top of HBase[5],which works like Google’s Bigtable[1] on Hadoop[4].
FIAP[7] was originally developed for the delivery of timeseries data over TCP/IP networks. However, FIAP defines
time-series data for facility information management such as
HVAC, light control systems, environmental monitors andpower meters. Here, the context of time-series data is not
the same as the network monitoring applications. Though
Prasad et al.[10] have tested the applicability of smart meterdata into OpenTSDB, we have needed another approach forstoring FIAP time-series data on HBase. OpenTSDB manages
data with ”metrics” and ”tags”. But, FIAP manages themwith ”Point IDs”. Instead of developing the mapping of Point
IDs into metrics and tags of OpenTSDB, in this work, wehave designed FIAPStoragePeta on HBase directly. We alsoneeded to store string type values which were not supported
by OpenTSDB.
According to our preliminary evaluation, FIAPStoragePeta
used HBase platform quite efficiently though we still needfurther analysis on it. It appropriately balanced the load oftime-series data onto the distributed data nodes, it achieved
fast data retrieval in the sequential read cases, and it showed
great scalability in retrieval time even on 10 million point IDs.
In the future, we should evaluate it with larger number of data
nodes and larger amount of values, and carry out measurementwith many types of queries and data.

C ONCLUSION

[2]

[3]

[4]
[5]
[6]
[7]

[8]
[9]
[10]

F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach,
M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A
distributed storage system for structured data. In Seventh Symposium
on Operating System Design and Implementation, 2006.
L. Deri, S. Mainardi, and F. Fusco. tsdb: A compressed databasefor time series. In Traffic Monitoring and Analysis, pages 143–156.
Springer, 2012.
T. Groves, D. Arnold, and Y. He. In-network, push-based network
resource monitoring: scalable, responsive network management. In
ACM NDM 2013, 2013.
Apache Hadoop. http://hadoop.apache.org/.
Apache HBase. http://hbase.apache.org/.
K. Kaczmarski and M. Pilarski. Real-time content delivery networks
monitoring. Przeglad Teleinformatyczny, 19(1):75–85, 2013.
H. Ochiai, M. Ishiyama, T. Momose, N. Fujiwara, K. Ito, H. Inagaki,
A. Nakagawa, and H. Esaki. FIAP: facility information access protocol
for data-centric building automation systems. In IEEE INFOCOM
M2MCN workshop, 2011.
T. Oetiker.
Rrdtool: Round robin database tool, 2013.
http://oss.oetiker.ch/rrdtool/.
OpenTSDB - a distributed, scalable monitoring system, 2014.
http://opentsdb.net/.
S. Prasad and S. Avinash. Smart meter data analytics using opentsdb
and hadoop. In Innovative Smart Grid Technologies-Asia (ISGT Asia),
2013 IEEE, pages 1–6. IEEE, 2013.