Aggregating Queries Against Large Inventories of Remotely Accessible Data
Creators: name: Fulker, David
name: Gallagher, James
Publisher: name: Zenodo
Year: 2016

description: This poster introduces an enhanced OPeNDAP feature for aggregating data from multiple files.

Problem: Existing data systems often allow access to individual files only, even when they belong to a collection. This granularity is often unrelated to user needs, making it difficult to retrieve data efficiently.

Solution: The authors propose a new web service that:



Allows users to specify:



Files to be aggregated: Users can choose specific files from a collection for aggregation.

Output format: They can choose between receiving the aggregated data as source files, a single archive, or a concatenated table (for specific array variables).

Subsetting criteria: Users can apply constraints to filter the data within the chosen files.



Supports various data formats: The service works with netcdf3, netcdf4, ASCII, and CSV formats.

Handles large datasets: The service utilizes HTTP POST to avoid limitations with long URLs when dealing with numerous files.


Benefits:



Users can acquire data from thousands of files with a single request, improving efficiency.

The service offers flexibility in choosing the desired output format and applying filtering criteria.

It supports various data formats commonly used in scientific research.


Implementation:



The proposed service is implemented as a web service accessible through HTTP POST.

It can be integrated with existing servers like Hyrax with some modifications.


Overall, this enhanced OPeNDAP aggregation feature simplifies data access for users by allowing them to efficiently retrieve and combine data from large collections of files.
description: Original Abstract

Those seeking to discover data for a specific purpose often encounter search results that are so large as to be useless without computing assistance. This situation arises, with increasing frequency, in part because repositories contain ever-greater numbers of granules, and their granularities may well be poorly aligned or even orthogonal to the data- selection needs of the user. This presentation describes a recently developed service for simultaneously querying large lists of OPeNDAP- accessible granules to extract specified data. The specifications include a richly expressive set of data-selection criteria—applicable to content as well as metadata—and the service has been tested successfully against lists naming hundreds of thousands of granules.

Querying such numbers of local files (i.e., granules) on a desktop or laptop computer is practical (by using a scripting language, e.g.), but this practicality is diminished when the data are remote and thus best accessed through a Web-services interface. In these cases, which are increasingly common, scripted queries can take many hours because of inherent network latencies. Furthermore, communication dropouts can add fragility to such scripts, yielding gaps in the acquired results. In contrast, OPeNDAP's new aggregated-query services enable data discovery in the context of very large inventory sizes. These capabilities have been developed for use with OPeNDAP's Hyrax server, which is an open-source realization of DAP (for "Data Access Protocol," a specification widely used in NASA, NOAA and other data-intensive contexts). These aggregated-query services exhibit good response times (on the order of seconds, not hours) even for inventories that list hundreds of thousands of source granules.