Data Management and Data Processing Support on Array-Based Scientific Data

Scientific simulations are now being performed at finer tempo ral and spatial scales, leading to an explosion of the output data (mostly in array-b sed formats), and challenges in effectively storing, managing, querying, disseminatin g, analyzing, and visualizing these datasets. Many paradigms and tools used today for large-sca l ientific data management and data processing are often too heavy-weight and have inhe re t limitations, making it extremely hard to cope with the ‘big data’ challenges in a var iety of scientific domains. Our overall goal is to provide high-performance data manage ment and data processing support on array-based scientific data, targeting data-int ensive applications and various scientific array storages. We believe that such high-perfor mance support can significantly reduce the prohibitively expensive costs of data translati on, data transfer, data ingestion, data integration, data processing, and data storage involv ed in many scientific applications, leading to better performance, ease-of-use, and responsiv eness. On one hand, we have investigated four data management topic s as follows. First, we built a light-weight data management layer over scientific d atasets stored in HDF5 format, which is one of the popular array formats. Unlike many popula r data transport protocols such as OPeNDAP, which requires costly data translationand data transferbefore accessing remote data, our implementation can support serv er-side flexible subsetting and aggregation, with high parallel efficiency. Second, to avoi d the high upfrontdata ingestion costs of loading large-scale array data into array database s lik SciDB, we designed a ii system referred to as SAGA, which can provide database-like support over native array storage. Specifically, we focused on implementing a number o f structural (grid, sliding, hierarchical, andcircular) aggregations, which are unique in array data model. Third, we proposed a novel approximate aggregation approach over arr ay data using bitmap indexing. This approach can operate on the compact bitmap in dices rather than the original raw datasets, and can support fast, accurate and fle xible aggregations over any array or its subset without data reorganization. Fourth, we ext nded bitmap indexing to assist the data mining task subgroup discoveryover array data. Like the aggregation approach, our algorithm can operate entirely on bitmap indi ces, and it can efficiently handle a key challenge associated with array data – a subgrou p identified over array data can be described by value-based and/or dimension-based att ributes. On the other hand, we focused on both fflineandin-situdata processing paradigms in the context of MapReduce. To process disk-resident scienti fic data in various data formats, we developed a customizable MapReduce-like frame work, SciMATE, which can be adapted to support transparent processing on any of the sc ientifi data formats. Thus, unnecessarydata integration and data reloading incurred by applying traditional MapReduce paradigm to scientific data processing can be avoi ded. We then designed another MapReduce-like framework, Smart, to support effici nt n-situ scientific analytics in both time sharingand space sharingmodes. In contrast to offline processing, our implementation can avoid, either completely or to a very lar ge extent, bothdata transfer anddata storagecosts.