DOI: 10.5061/dryad.v41ns1rtt
title: MERRA-2 subset for evaluation of renewables with merra2ools R-package: 1980-2020 hourly, 0.5° lat x 0.625° lon global grid

description: Renewable variable energy resources (VER) - solar and wind energy are
 becoming increasingly important sources of electricity worldwide.
 Assessing the potential and the reliability of the resources requires
 long-term historical data. Directly measured solar radiation and wind
 speed are limited to locations of weather stations, and even when
 available, the observations are not directly suitable for the evaluation
 of VERs potential (as an example, the wind speed is rarely
 measured at wind turbines heights).  Reanalysis data
 based on satellite imagery and Earth system models, such as MERRA-2 offer
 a broad set of long-term time series on a global grid. 
 `merra2ools` is a preprocessed subset of MERRA-2 variables and a software
 (R-package) designed for quick estimation of hourly output of solar
 photovoltaics and wind turbines. The grid of the MERRA-2 dataset has
 0.625° step length along longitude (- 180° to 180°) and 0.5° along
 latitude (- 90° to 90°), making 576 x 361 grid or 207936 locations. The
 subset of the hourly data covers the period from 1980-Jan-01 00:30 UTC to
 2020-Jan-31 23:30 UTC. It includes eight variables: wind speed at 10- and
 50-meters height (W10M and W50M), wind direction (WDIR), the atmospheric
 temperature at 10 meters height (T10M), surface incoming shortwave flux
 (SWGDN), surface albedo (ALBEDO), bias-corrected total precipitation
 (PRECTOTCORR), and air density at the surface (RHOA). The dataset’s key
 variables are date-time in Coordinated Universal Time timezone (UTC) and
 location identifiers (locid). In total, the subset has 290,357,084,160
 data points (362,946,355,200 including the key variables). To reduce the
 dataset’s memory footprint (~3TB uncompressed), the original MERRA-2
 variables have been rounded, scaled, and stored as integers in highly
 compressed data format with high speed full random access (`fst` package
 for R). The resulting dataset is saved in separate files by months (41
 years x 12 months, 492 data-files in total). Additionally, some summary
 statistics such as mean values of each variable by month and location ID,
 annual spatial correlations with the nearest neighbors have been
 calculated for wind speed and solar irradiance and added to the dataset.
description: The `merra2ools` dataset has been assembled through the following
 steps: The MERRA-2
 collections <em>tavg1_2d_flx_Nx </em>(Surface Flux
 Diagnostics), <em>tavg1_2d_rad_Nx</em> (Radiation
 Diagnostics), and <em>tavg1_2d_slv_Nx </em>(Single-level
 atmospheric state variables) downloaded from NASA Goddard Earth Sciences
 (GES) Data and Information Services Center (DISC)
 (https://disc.gsfc.nasa.gov/datasets?project=MERRA-2) using GNU Wget
 network utility (https://disc.gsfc.nasa.gov/data-access). Every of the
 three collections consist of daily netCDF-4 files with 3-dimensional
 variables
 (<em>lon </em>x <em>lat</em> x <em>hour</em>).  The following variables obtained from the netCDF-4 files and merged into long-term time-series: Northward (<em>V</em>) and Eastward (<em>U</em>) wind at 10 and 50 meters (<em>V10M</em>, <em>V50M</em>, <em>U10M</em>, <em>U50M</em>, respectively), and 10-meter air temperature (<em>T10M)</em> from the <em>tavg1_2d_slv_Nx</em><strong> </strong>collection; Incident shortwave land (<em>SWGDN</em>) and Surface albedo (<em>ALBEDO</em>) from the <em>tavg1_2d_rad_Nx</em> collection; Bias corrected total precipitation (<em>PRECTOTCORR</em>) and Air density at the surface (<em>RHOA</em>) from the <em>tavg1_2d_flx_Nx </em>collection. 3. The representation of wind speed and direction has been changed from Northward (<em>V</em>) and Eastward (<em>U</em>) components to the wind speed (<em>W</em>) and direction (<em>DIR</em>). Wind speed has been calculated for both 10- and 50-meters height (W10M and W50M). The wind direction is reported for 50 meters height only due to the variable’s marginal differences between the two heights. This conversion also reduces the dataset’s size and the computational burden of calculating wind power capacity factors (which require wind speed rather than direction). 4. The obtained time series have been merged, scaled, grouped by year and month, and stored in compressed “<em>fst” </em>file-format (file names mask: “merra2_YYYYMM.fst” where YYYY is a year, MM - two-digit number of a month). The resulting <em>merra2ools</em> dataset is stored in 492 data files (12 months x 41 years) and has the following hourly time-series: <strong>locid</strong> - location IDs (key variable), an index of locations in MERRA-2 dataset, from 1 to 207936, integer; <strong>UTC</strong> - date and time (key variable) in Coordinated Universal Time (UTC) timezone (“POSIXct/POSIXt” format); <strong>W10M</strong> - wind speed at 10-meter height (calculated sqrt(V10M<sup>2</sup> + U10M<sup>2</sup>) where V10M and U10M are northward and eastward wind at 10-meter, m/s, rounded to the first decimal place, stored as an integer variable named “W10M.e1”, where suffix “.e1” indicates scale factor = 10); <strong>W50M</strong> - wind speed at 50-meter heigh (calculated sqrt(V50M<sup>2</sup> + U50M<sup>2</sup>) where V50M and U50M are northward and eastward wind at 50-meter, m/s, rounded to the first decimal place, stored as an integer variable named “W50M.e1”, where “.e1” indicates scale factor = 10); <strong>WDIR</strong> - Direction of wind at 50-meter height (calculated atan2(V50M/U50M), converted to degrees, rounded to tens, stored as an integer variable named “WDIR.e_1”, where “.e_1” indicates scale factor = 0.1); <strong>T10M</strong> - 10-meter air temperature (Celsius, converted from original Kelvin units (C = K - 273.15), rounded to the nearest integer, stored as an integer variable “T10M.C”, where “.C” indicates Celsius units); <strong>SWGDN</strong> - surface incoming shortwave flux (W/m<sup>2</sup>, rounded to the nearest integer, stored as an integer variable with the same name); <strong>ALBEDO</strong> - Surface albedo (index [0, 1], rounded to the second decimal place, stored as an integer variable named “ALBEDO.e2”, where “.e2” indicates scale factor = 100); <strong>PRECTOTCORR</strong> - Bias corrected total precipitation (kg/m<sup>2</sup>/hour, rounded to the first decimal place, stored as an integer variable named “PRECTOTCORR.kg_m2_h.e1”, where ".kg_m2_h" indicates chages in units from the original kg/m<sup>2</sup>/s units in MERRA-2 to kg/m<sup>2</sup>/hour, and “.e1” indicates the scale factor = 10); <strong>RHOA</strong> - Air density at the surface (kg/m<sup>2</sup>, rounded to the second decimal place, stored as an integer variable named “RHOA.e2”, where “.e2” indicates scale factor = 100). All variables are hourly averages. UTC-time is given for the middle of every hour. Additional files: <strong>locid.RData </strong>- R-Data-file with `data.frame` of MERRA-2 locations identification, which contains the following fields: <em>lon</em> - longitude of the location; <em>lat</em> - latitude of the location; <em>locid</em> - location ID, a variable created based on longitude and latitude grid indexes (V1 and V2), from 1 to 207936; <em>V1 </em>- longitude location index, from the MERRA-2 database netCDF-4 files; <em>V2 </em>- latitude location index, from the MERRA-2 database netCDF-4 files. <strong>locid_neighbr.RData </strong>- R-Data-file with `data.frame` of locations identification of first neighbors for every location on the grid: <em>locid</em> - location ID of a cell of the grid for which neighbors are reported in the following columns; <em>N</em> - <em>locid </em>of the closest Northern neighbor; <em>NE</em> - <em>locid </em>of the closest North-Eastern neighbor; <em>E</em> - <em>locid </em>of the closest Eastern neighbor; <em>SE</em> - <em>locid </em>of the closest South-Eastern neighbor; <em>S</em> - <em>locid </em>of the closest Southern neighbor; <em>SW</em> - <em>locid</em> of the closest South-Western neighbor; <em>W</em> - <em>locid </em>of the closest Western neighbor; <em>NW</em> - <em>locid </em>of the closest North-Western neighbor. <strong>merra2_stat_ym_mean.fst</strong> - monthly means by location for each of the eight variables. <strong>merra2_stat_ym_sd.fst</strong> - monthly standard deviations by location for each of the eight variables. <strong>merra2_stat_y_W50M_nbcor.fst</strong> - annual spatial autocorrelation of W50M in <em>locid</em> with every of the first neighbors (<em>N, NE, E, SE, S, SW, W, N</em>W) and the average across all neighbors (<em>Z</em>). <strong>merra2_stat_y_SWGDN_nbcor.fst</strong> - annual spatial autocorrelation of SWGND in <em>locid </em>with every of the first neighbors (<em>N, NE, E, SE, S, SW, W, NW</em>) and the average across all neighbors (Z).
description: The straightforward way of working with the dataset is with the
 `merra2ools` package that offers functions to read, automatically rescale,
 merge, and subset the data for particular locations and time-periods.
 Alternatively, the data can be accessed directly using `fst`
 package-function “read_fst()” and resaved in other formats. For meaningful
 units, the data must be rescaled (dividing scaled variables by their scale
 factors - see the “Methods”).