# MHBase: A Distributed Real-Time Query Scheme for Meteorological Data Based on HBase
###### Abstract  
Meteorological technology has evolved rapidly in recent years to provide enormous, accurate and personalized advantages in the public service. Large volumes of observational data are generated gradually by technologies such as geographical remote sensing, meteorological radar satellite, _etc._ that makes data analysis in weather forecasting more precise but also poses a threat to the traditional method of data storage. In this paper, we present MHBase, (Meteorological data based on HBase (Hadoop Database), a distributed real-time query scheme for meteorological data based on HBase. The calibrated data obtained from terminal devices will be partitioned into HBase and persisted to HDFS (the Hadoop Distributed File System). We propose two algorithms (_the Indexed Store_ and _the Indexed Retrieve Algorithms_) to implement a secondary index using HBase Coprocessors, which allow MHbase to provide high performance data querying on columns other than rowkey. Experimental results show that the performance of MHBase can satisfy the basic demands of meteorological business services.  
meteorological data; distributed storage; HBase; secondary index +
Footnote †: journal: Journal of the American Statistical Association  
distributed file system designed to be operated on commodity hardware while HBase (Hadoop Database) [6] is an open source, non-relational, distributed database modeled after Google's BigTable [7] based on HDFS. The combination of Hadoop and HBase enables reliable and expandable data storage.  
It is common to use self-designed architecture to solve specific problem in specialized fields, especially in Meteorology. For example, NASA (National Aeronautics and Space Administration) uses Hadoop and Hive in RCMES (The Regional Climate Model Evaluation System) [8] to provide services and analysis based on observation data and also uses Sqoop [9] to migrate data. In another NASA project, MERRA/AS (Modern Era Retrospective-analysis for Research and Applications Analytic Services) [10], Cloudera's CDH cluster is used to store 80 TB of data to provide data sharing and data analysis services.  
In this paper, we propose a platform called MHBase (Meteorological data based on Hbase) for meteorologically structured data based on HBase in order to satisfy the business requirements of safe storage and efficiently improve the query of large meteorological data. Our other meteorological operational system uses Hadoop to store data and other components of \"Hadoop ecosystem\" to implement functions like Hive and Zookeeper; thus, HBase is the best choice as its features are closely combined with Hadoop. Our system is based on existing techniques such as distributed architecture and data partition strategy. Several experiments have been done to show the performance improvement for our optimized method and combined architecture. The reliability of data is ensured by a multi-copying mechanism through HDFS. In accordance with the conventional query cases, especially on HBase, we designed rowkey and to realize a secondary index.  
The rest of this paper is organized as follows. Section 2 introduces some related works about our project and analyses some indexed schemes already in existence. In Section 3, we present two algorithms and detail the design and implementation of MHBase. We also evaluate performance by experimentation in Section 4. Finally, we present conclusions and future work in Section 5.