Four dimensional weather data storage and access

Storage of weather data in four dimensions in a mass storage data cube for ready access. Weather data to be stored is ingested and processed with respect to location and time to generate one or more tiles each characterized by a geographic location index and a time index. The tiles are stored in a mass storage data cube in accordance with the geographic location index and the time index of each tile. The tiles containing the stored weather data can be readily accessed and retrieved from the mass storage data cube.

BACKGROUND

In today's computer industry, the term Big Data is often used to refer to large volumes of collected information from past events. Some use cases include user behavior data on web sites or medical patient information. These Big Data are then queried to look for statistical understanding from the data, for example, trends, means or norms. The results of the queries are often referred to as Big Data Analytics. To date, there has been no effective use of Big Data to efficiently update, process, and make use of weather data that is continuously being collected and that is disparate in time and space.

DETAILED DESCRIPTION

Weather Data

The Big Data of interest are weather related or meteorological data that are continuously being collected, in some cases is very near real-time in nature, is volatile (new data replaces old data), and is disparate in time and space (location). In these ways, the vast amount of weather data that must be understood and used is quite different from usage of other Big Data, in which old data and trends must be analyzed and understood. The Big Data challenges of weather data scalability and the need to efficiently manage the storage and access to highly volatile, voluminous weather data with minimal processing latency is addressed herein.

In the field of meteorology, there are a larger number of different data types to help a human or computer application understand the atmospheric conditions in order to make an accurate forecast, to review observation data, to analyze past performance of computer application, to understand trends, to perform statistical analysis, etc. The data vary inSpatial type (point, contour, grid, image, etc.)Timeframe of the data (valid at a specific time or over a range of time)Spatial resolution of the dataWhether it is observation data/parameter or forecast data/parameterGeographic projection
The terms data, weather data, atmospheric data and the like used herein, refer to forecast or observation data of weather or meteorological forecasts or conditions.

In order to be able to use this disparate data at the same time for a given scenario, be it a forecast, analysis, etc., the data must be in a common geographic framework and be time-matched. The framework for the data is referred to as the “Data Cube.” The Data Cube stores data that is time- and geographic location-indexed, and accordingly uses a geographic tile-based or Quad key geographic framework. For a given geographical area on the earth, the area can be separated into four tiles (quad key) as the area becomes more specific or more “zoomed in” the area is sub-divided into four more tiles. This geographic framework is used by Google and Bing, for example, for their interactive mapping. The use of integer math in a quad key tile based location index allows for an efficient referencing method for quick access, grouping of data from one “zoom level” to another and is useful if trying to display the data in a map form.

The data also must be time-matched (time indexed) before it is stored in the Data Cube. Some data involved is observation data, valid at the time that the data was collected. Some data involved is forecast data, valid at some point in the future when the data are first created. Forecast data can take on many forms, but, for example, a numerical model is a computer program made up of equations of motion and numerical schemes to simulate atmospheric conditions.

Process of “Converting” to Tile Data

Surface observations consist of a number of variables at a given point. For example, at a location at an airport, the surface observation “station” will have a thermometer (to measure temperature), a hygrometer (to measure moisture content of the air), a rain gauge (to measure the amount of falling precipitation, an anemometer (to measure wind speed), etc. These observations are taken at this location at regular time intervals such as every 30 minutes. Across the United States (and the world) similar observation stations exist and report their data at similar time intervals.

The data are transmitted via local networks to a central distribution point. The data are then sent to users who are interested in the data. These data are ingested, decoded (if it is in a coded format) and then the latitude/longitude of the data associated with the nearest quad key in the tile system. The data is then stored in the Data Cube and accessed by the tile. The tile of data can be provided back to users in various formats (XML, JSON, etc.).

Some applications (mapping for example) need to have a visual representation of weather, more than just the numbers, such as temperature and wind speed. The data need to be shown as a colorized contoured field of data. To create this field of data the initial raw data have to be objectively analyzed. This objective analysis is fitting the data to a grid by performing interpolation between the raw data points. Once data is stored as a grid in the Data Cube in the quad keys that make up the organizational structure of the data in the Data Cube, the data can be easily retrieved. The new gridded form of the data can then be queried from the Data Cube and similar data returned from the raw form previously explained by valid at points “between” the initial raw observation locations.

To complete the visual representation of the observation data (think of a USA Today temperature map on the weather page), the gridded data then has to be contoured where each contour (say every 1-2 degrees F.) is drawn, then color shading is applied to the contours and the contours are then rendered as an image. The image is then broken into tiles (like a puzzle) so that the large image can be put back together by the requesting application.

Examples of Data to be Inserted Into the Data Cube

The numerical model is executed from an initial time, T0, and is run for a predetermined amount of time with results produced at some convenient interval of time, such as thirty minutes or one hour. Variables that define the atmospheric conditions, such as temperature, humidity, pressure, wind speed, and wind direction are determine at various locations, typically on a grid of predefined locations in three dimensions and at each time interval of the forecast time, thus making the resulting numerical model data four dimensional (x, y, z, t). The numerical model data are then put into the Data Cube valid at times, T0, T30 min, T60 min, T90 min, . . . TN min.

Weather satellite observation data can take a few forms. For example, weather satellites can have cameras attached to produce photographs of the earth taken from high above the earth. These photographs show where clouds are located, can indicate what types of clouds are part of the earth's weather system at a given time and when the photographs are shown in a series, speed and direction of movement can be determined. Weather satellites can also use infrared sensors to determine the temperature of the clouds that are observed at the top of the clouds. Using assumptions about the depth and temperature profile of the atmosphere, one can estimate the height of clouds using infrared satellite data. These various forms of weather satellite data may be compiled into imagery or pixel data valid at a given time. These data are then put into the Data Cube valid at the time of observation.

Radiosondes are instrumentation packages containing weather sensors, e.g., thermometer, barometer, hygrometer, along with a radio tracking device. These radiosondes are typically attached to a balloon and released into the atmosphere to take observations of the weather variables. The data are collected at a predefined interval during a balloons assent, e.g., every 10 seconds. These data are then put into the Data Cube valid at the time of the observation of the variables.

Examples Uses of the Data Cube

Meteorologists and other weather data users, often need to visualize and analyze weather data to help understand the weather situation for making a forecast, for doing analysis on past data to get a better understanding of how variables changed so as to help make a better forecast in the future or to understand statistically what the data can provide as information. With all the various data in the Data Cube, access to the necessary data can be simplified for the user to facilitate their data needs. The data may be accessed through an Application Programming Interface (API) that allows the user to specify what types of data are needed, for where the data are needed and for when the data are needed. Some examples that the Data Cube facilitates are:Power utility companies during the summer months often have to make decisions about a particular day's power needs for their area of concern. They may have plenty of energy available to service the power needs, they may have to bring on additional resources to generate more energy to meet demand or they may have to purchase energy to meet demand. The decision process involves not only things like the day of the week and whether most businesses are open for business, but is greatly influenced by the weather and weather forecast, particularly the temperature, humidity and wind conditions. Based on the forecast of a very hot day, a power utility company may begin producing or buying more energy to meet the forecast demand due to the hot weather. Buying more energy from another power utility company can be a very expensive part of the utility company's business and thus optimizing the needs is necessary to keep costs down. With all the data in the Data Cube, a user can make requests for observation and forecast data for various times in the forecast period to best understand how the weather situation is expected to play out. Additionally, as time goes by in the forecast period additional data added to the Data Cube may reveal the need to buy less power or the need to buy more power due to changing weather information. A user may request information through the API from the Data Cube that reveals that the timing of a precipitation event is 1 hour before the last forecast indicated, thus meaning that temperatures will cool down earlier than was originally forecast and thus the power needs will be less than originally thought. The user can then make decisions based on this new information.High voltage power lines are often run from tower to tower built in large open areas to move produced energy from the generation location to where the power needs are. In the western United States, these high voltage power lines can be roosting places for large wild birds. The birds will often leave bird excrement on the power lines. When high humidity and light wind conditions, like a dense fog situation occurs, the excrement can serve as a conductor and cause an arc to occur between power lines and this can result in an outage of the power lines cutting off the movement from the generation location to the location of need. Such an outage can be costly to the power utility company with lost revenue and fines. When moderate to heavy rain occurs, the excrement is washed off the power, thus eliminating the potential for an arcing and outage situation. If it has been a while since the excrement has been washed off the power lines, a power utility company may fly helicopters armed with power washers to wash off the excrement. With all the data in the Data Cube, a user can make a request to determine the length of time since the last moderate to heavy rain events to determine if the company needs to intercede and wash the lines themselves or if weather conditions will wash the lines for them.Farmers depend on weather data to understand how their crops will mature, to determine what diseases the crops maybe be susceptible to and to predict when crops will be ready to harvest. Much of the weather information that is needed is crop dependent, thus requesting data from the Data Cube is not general but specific to a set of criteria for a given crop. Farmers us a calculation of Growing Degree Days (GDD) that is a measure of the average temperature above a base temperature. For example, if the base temperature for a particular crop is 65 degrees Fahrenheit and if the high temperature on a given day is 80 degrees and the low temperature is 60 degrees, the average temperature is 70 degrees and thus the GDD is 5 degrees (5 degrees above the base temperature). Each crop requires a certain number of GDDs to be ready to harvest.Pilots need weather forecasts at a series of locations and altitudes for flight planning. The Data Cube stores weather forecasts from many different forecast models as well as human derived forecasts. Through an API request to the Data Cube, a user can request the “average” or “consensus” forecast at all points along a route or the request could be for the “most severe” or the “range” of forecasts.

FIG. 1shows the flow of information into and out of the Data Cube for some example inputs and example outputs along with pre-processing processes necessary to organize data to be put into the Data Cube. The data are transmitted via local networks to a central distribution point. The data are then sent to users who are interested in the data. WDT ingests these data, decodes the data (it is in a coded format) and then associates the latitude/longitude with the nearest quad key in the tile system. The data is then stored in the Data Cube and accessed by the tile. The tile of data can be provided back to users in various formats (XML, JSON, etc.). This is reflected in the drawings in which Raw Data Ingest110occurs, followed by Pre-Processing130of the data, storage in the Data Cube140, access to the stored and indexed data through API layer150, and one or more Applications170that may wish to access the data indexed and stored in Data Cube140.

Data may be received through satellite data ingest112, observations ingest122, lightning ingest124, contour data ingest126and model data grids128. The Data Cube storage has inputs of various weather data types and raw data examples on the left of the drawing can take on various forms (raster data, vector data, observations data, lightning data, model data). These data enter the Data Cube140through their respective processing methods130depending on the nature of the data, its spatial type, timeframe of the data, spatial resolution of the data, whether that data are observation or forecast in nature and the geographic projection of the data.

As shown in the figure, for example, some data types are raster in form, such as satellite data112. To be stored as data tiles they are converted to vector data through vectorization pre-processors132and then further organized (filtered and sorted) by a process commonly referred to as MapReduce134.

Surface observations, are data such as temperature, humidity, winds (wind speed and wind direction), etc. observed by a number of meteorological sensors that are co-located at a given site (point data) and, together with Lightning data and Contour data, for example, are ingest as Point data Ingest120. These point data often arrive asynchronously as Observations Ingest122and are put on a “queue” to be moved into the Data Cube, as shown by Queue-based ingest136. Lightning data are point data, are data valid at a given location, the location of a lightning strike as estimated by a network of sensors and are ingest by Lightning Ingest124. Lightning, like surface observations, are processed by Queue-based ingest136.

Some applications (mapping, for example) need to have a visual representation of weather, more than just the numbers, such as temperature and wind speed. The data need to be shown as a colorized contoured field of data. To create this field of data the initial raw data, collected by Contour Data Ingest126, have to be objectively analyzed. This objective analysis is fitting the data to a grid by performing interpolation between the raw data points. Once stored as grid data in the Data Cube, the grid data is comprised of the quad keys that make up the organizational structure of the data in the Data Cube. As previously mentioned, the new grid form of the data can then be queried from the Data Cube and get back similar data as was returned from the raw form previously explained by valid at points “between” the initial raw observation locations.

Model data grids128, or numerical model data, are the raw output data from numerical models. The numerical models are computer programs that use assumptions about the state of the atmosphere and solve equations of motion of the atmosphere. The grids are four dimensional grids (x, y, z, t) of each variable (e.g., temperature, humidity, winds speed, wind direction). These model data grids tend to be very large datasets (many gigabytes of data) and thus have to be broken into smaller parts (slices) through a parallelized ingest process138. The smaller parts of data allow the data to be then stored in the Data Cube more efficiently. Since model data tends to be very large and there are numerous different models that can vary significantly from each other, storing the statistical variations of the model solutions is valuable as well. Thus, specific data (slices) or all the model data can be pulled from the Data Cube, processed through statistical techniques and the results also stored in the Data Cube

The Data Cube uses a quad key organizational structure so that the data can be queried out of storage relatively fast. For example, an image tile request system, shown as Image tile services152, pulls data from the Data Cube for a given location (point or area) and data are then rendered into a series of tiled images that can then be pieced together by a downstream application172,174,176, such as a mapping application that places the tiled images on the map (a web app172or mobile app174). Similarly, some applications may want actual data rather than imagery and thus the tiles are returned to a requesting Data tile services154as a “data tile” where the data is valid at the tile location.

Once stored in the Data Cube, various systems and users will want access to the data, through an API, such as Data Access API156. The API allows users to specify the type of data they are requesting, the location (e.g., region, point, entire world), the valid date/time of the data, the source of the data, and any input that may be needed for a calculation. Two examples are shown where an Image tile request system (shown as Image tile services152) is making requests for data from the Data Cube140to generate quad key (tiles) forms of imagery, such as <quad key>.png and Data tile services154is making requests data valid at a particular quad key location for JSON or XML format. The data in this case are accessed by end users or end user applications through a CDN (Content Delivery Network160) which serves as a caching mechanism so that identical results are not recreated over and over for each individual user.

Often times, the amount of data being requested for simultaneous users from the Data Cube is very large and many of the requests are the same. So, a Content Delivery Network (CDN)160is used to cache the query results for more rapid access to the results. Once requests for the cached data are no longer occurring, after some amount of time has passed (such as 24 hours), the data are dropped from the cache.

Other applications from clients and partners can also make requests to the API for data or image tiles. Data pulled from the Data Cube can also be processed by various statistical techniques at Statistical processing block139. As indicated in the drawing, statistical processing can be model averaging, processing of ranges, etc.

Referring now toFIG. 2, an example200of pre-processing data for the Data Cube140is shown. In this particular example, a weather radar210collects data on the intensity of the “returned” power (or reflectivity). The intensity is traditionally shown as a radar image220for a user to understand where weather events are taking place within the coverage area of the radar. To store these data in the Data Cube140, the data are sent to a pre-processor250that converts the data to a series of tiles (quad key and time indexed)240,242,244for ease of storage and access. Each tile240,242,244in the Data Cube represents a small area of the radar coverage230and is valid at a specific time. For example, it can be seen that tiles240,242,244are each valid at time 12:01:00.

FIG. 3illustrates an implementation of the upstream processing to store data in the Data Cube. To move data from one process to another or to notify a process that data is available for use, a data queuing and transfer mechanism310exists between many of the steps in the process. The data queuing and transfer block310can be a client server application that servers as a client notification system, for example.

The Data Access API320serves as the interface for external applications to pull data from the Data Cube330and in some cases, as shown here, the data from the Data Cube are pulled into a model averaging process340. A model, in this case, is a numerical weather prediction model. Each model (Raw Model Data350) is moved via the data queuing and transfer mechanism310to the Data Ingest and Pre-processing module360and then stored in the Data Cube330. Each individual model represents a different solution (prediction) of the atmospheric conditions. A statistical averaged solution is then also stored in the Data Cube for external distribution. The Data Ingest module360preprocesses the data (turning it into tiled data that is time indexed) to allow the data to be stored in the Data Cube.

Oftentimes, model data are used by forecasters as a first guess for their “final” or “official” forecast. Forecasters can edit the raw model data grids using a model grid editor370and the resulting edited model data380can also serve as an input into the Data Cube350.

Referring now toFIG. 4, an example implementation400in which a Data Cube provides Forensics (past) data through an API to users is shown. The Data Cube implementation contains both data in a NoSQL form (for unstructured data) in NoSQL Data store412as well as in a relational (SQL) database form (for structured data) in SQL Data store414. Various derived data types420(surface data analysis, hail contours, tornado tracks, etc.) as well as raw real-time data460(surface observations, Watches/Warnings, etc.) are stored in the Data Cube410. Users can make an API request430for archived or past data from the Data Cube. If the request is authenticated as allowed by the Authentication system440, the results450of the request are then returned to the user in various data formats such as XML, JSON, GeoJSON, and KML.

FIG. 5provides a detailed example implementation of data processing, image tile generation and storage in the Data Cube510for access via an API layer520. Pre-processors530ingest various weather data types540-558(Watches/Warnings540, Lightning Strikes542, Surface Analysis544, Hail contour data546, etc.), decode and organize the data to be stored in the Data Cube510in its quad key indexed and time indexed structure. Through an Application Programmer Interface (API) a user can make a tile request560for a single tile or an area of tiles, for a single time or series of times. The request may require that the user is authenticated at Authentication block570. Once authenticated, the request is routed to a Content Delivery Network (CDN) cache580to see if the same request has already been made and results stored previously. If so, the results are simply returned from the cache and no additional computing or data transfer takes place. If the request is new or unique, the request is then passed on to the API/Web layer520where the query is sent to the Data Cube510and the results are calculated and returned as tiled images590to the user.

FIG. 6illustrates the use of the Data Cube for weather alerting processes.

Various weather data types (Heavy Rain data, Lightning Prediction data, lightning data, watch/warning data, etc.) are ingest into the Data Cube620through pre-processors610(Heavy Rain contour Pre-processor612, Predicted Lightning contour Pre-processor614, Lightning Data Pre-processor616, Watch/Warning Pre-processor618) that decode and organize the data to store it in the Data Cube620. As a means of putting data in or pulling data out of the Data Cube, an Application Programming Interface (API)625sits between the external processes610and the Data Cube620.

The Alerting Engine630performs checks with the Registration System API640to see if a given location (such as the user location stored in the Locations database644and reflected in the user's account stored in the User Accounts database642, for example) is in an alert condition and a message (alert) needs to be sent to a user through a number of various alerting mechanisms650(SMS Alerter, E-mail Alerter, Mobile Push Alerter, Social Media Alerter, etc.). Once an alert condition is detected, the Alert Engine630generates the message and sends the message to the Message Queue660along with the necessary user information (such as alerting mechanism). The Message Queue then routes the alert to the proper alerting mechanism to send the alert to the user as shown.

Upon receiving an alert, a user may then select to make a request for data from the Data Cube through the API, the API Request670, to get more information about the data that generated the alert condition.

For retrospective investigation and analysis, alerts that are generated are logged via an Alert Logging process662and stored in an Alert Log database664.

Those skilled in the art will recognize that the present invention has been described in terms of exemplary embodiments based upon use of a programmed processor. However, the invention should not be so limited, since the present invention could be implemented using hardware component equivalents such as special purpose hardware and/or dedicated processors which are equivalents to the invention as described and claimed. Similarly, general purpose computers, microprocessor based computers, micro-controllers, optical computers, analog computers, dedicated processors and/or dedicated hard wired logic may be used to construct alternative equivalent embodiments of the present invention.

Those skilled in the art will appreciate that a program flow and associated data used to implement the embodiments described above can be implemented using various forms of storage such as Read Only Memory (ROM), Random Access Memory (RAM), Electrically Erasable Programmable Read Only Memory (EEPROM); non-volatile memory (NVM); mass storage such as a hard disc drive, floppy disc drive, optical disc drive; optical storage elements, magnetic storage elements, magneto-optical storage elements, flash memory, core memory and/or other equivalent storage technologies without departing from the present invention. Such alternative storage devices should be considered equivalents.

Various embodiments described herein are implemented using programmed processors executing programming instructions that are broadly described above in flow chart form that can be stored on any suitable electronic storage medium or transmitted over any suitable electronic communication medium. However, those skilled in the art will appreciate that the processes described above can be implemented in any number of variations and in many suitable programming languages without departing from the present invention. For example, the order of certain operations carried out can often be varied, additional operations can be added or operations can be deleted without departing from the invention. Error trapping can be added and/or enhanced and variations can be made in user interface and information presentation without departing from the present invention. Such variations are contemplated and considered equivalent.