# MHBase: A Distributed Real-Time Query Scheme for Meteorological Data Based on HBase
## 3 Proposed Design
### Data Partitioning  
Hadoop ecosystem is highly extensible; therefore, HBase, unlike in a RDBMS, data distribution is done at design time [26]. HBase can be configured to use different policies for data partitioning, but for the sake of simplicity, we will use the default approach and we also optimize system performance by designing the tables' structure.  
During the horizontal partitioning phase, undesirable _RegionServer_ hotspotting will occur at write time if the rowkey is used, which, in turn, starts with \"observation time\". HBase sorts records lexicographically using the rowkey, which allows quick access to individual records, and it is always bigger, so the write operation will always be positioned in the region with the upper bound and this results particularly to _RegionServer_ overloading. To circumvent this issue, we salted rowkey in the user table by prefixing \"station number\" in front of observation time. Meanwhile, rowkeys' length should be fixed in order to improve retrieval performance.  
In vertical partitioning, for a logical table in HBase, data in each family persist in a file called _Store_, which separates physically. Thus, we put relevant attributes into the same family so that they can be stored together on a disk at design time. For example, we put regular meteorological parameters such as rainfall, temperature, and clouds into a family named \"w_info\", while the remaining unrelated information is stored in a separate family. This can improve the systems performance because non-relevant families (for current query) are not read [27]. The _Store_ will be split into two when it is larger than a given threshold. The formula below shows how this threshold is set by default split policy.  
\[split.threshold=\min(max.size,R^{2}*flush.size) \tag{2}\]  
Figure 4: Architecture of MHBase.  
The split threshold is defined as the minimum of: (1) a constant value of a _Store's_ max size; and (2) a function of the number of regions in its corresponding _RegionServer_ (\(R\)) and the flush size of _memstore_.  
From the beginning, the secondary element is used and data are split at a faster pace regardless of max.size. Ultimately, the split threshold will increase until it exceeds max.size. After a certain number of splits, the splitting step will remain constant from then on.  
Physically, each family is separately stored in a _Store_; therefore, the pace of horizontal partitioning depends on the vertical design. As shown in Figure 5, with the given data, the larger the number of families, the more difficult it becomes for a _Store_ to reach its splitting threshold. Conversely, if each family contains more attributes, it will be faster to reach the splitting threshold.  
With the aim of reducing communication cost across the different _RegionServers_, we assumed that one server could contain both regions of the actual table (user regions for short) and their corresponding indexed tables (index regions for short) so that it can only scan the corresponding index regions while scanning user regions for one index. To achieve this goal, we used an enhanced load balancer presented by [PERSON] (2016) to collocate the user regions and corresponding index regions.