DOI: 10.5281/zenodo.17105306
title: Project HF-EOLUS. Task 2. Aggregated SAR and HF-Radar Radial Metrics for Wind-Inversion Model Training

description: This record presents the results obtained by applying the open, reproducible workflow described in HF-EOLUS Geospatial Processing Tools to the HF-EOLUS Sentinel‑1 SAR Derived Data Bundle and to the VILA & PRIO Radial Metrics GeoParquet datasets. It provides analysis-ready geospatial data products generated to train HF-Radar wind-inversion models. HF-EOLUS Geospatial Processing Tools documents the methodology, software environment, and parameterization; the dataset records contain the data used as input. This record includes the concrete outputs for reuse and citation.

Wind-inversion models are trained using supervised learning methods. In the HF-EOLUS project, we tackle this task as a regression problem where, given a set of features (HF-Radar data), we train a model to predict a numerical output (wind speed and direction). Due to the dense nature of the SAR and HF-Radar data and that they are not initially co-located, we need to aggregate them into a grid that allows us to get paired time-series of HF-Radar input <-> SAR output on a grid. HF-EOLUS Geospatial Processing Tools provides a workflow that we use here to 1) compute the coverage area of the HF-Radar data, 2) build a regular grid inside that area, 3) map each SAR and HF-Radar record into that grid, 4) aggregate the features and outputs into time-series for each node in the grid, 5) standarize the result for easy access and downstream analysis.

Step 1 — Intersection of coverage hulls (VILA ∩ PRIO)

This step computes per-station convex hulls from the VILA and PRIO HF-Radar radial metrics stored in Athena, intersects them to isolate the shared coverage area, and produces both a GeoJSON polygon and an interactive HTML map for quick inspection. The resulting intersection defines the spatial domain used for training and subsequent gridding.



Inputs: Athena tables HF_INGESTION_VILA.radial_metrics and HF_INGESTION_PRIO.radial_metrics (described in VILA & PRIO Radial Metrics GeoParquet).

Method: scripts/hulls/convex_hulls.sh computes per‑station convex hulls and their intersection; scripts/hulls/view_hull.sh renders an interactive preview map.

Commands:

./scripts/hulls/convex_hulls.sh --profile default --tables "HF_INGESTION_VILA.radial_metrics,HF_INGESTION_PRIO.radial_metrics" --output-file VILA_PRIO_hull.json --output-location s3://hf-eolus/athena-query-results/ --operation intersection

./scripts/hulls/view_hull.sh --input VILA_PRIO_hull.json --output VILA_PRIO_hull_map.html --title "VILA ∩ PRIO"


Outputs:



VILA_PRIO_hull.json: GeoJSON FeatureCollection (EPSG:4326) with a single Polygon representing the intersection of the VILA and PRIO coverage hulls.

VILA_PRIO_hull_map.html: Leaflet-based interactive map preview titled “VILA ∩ PRIO”.

HF_INGESTION_VILA_radial_metrics_wkb.csv: CSV export with a wkb column (WKB geometries) for VILA radial metrics points used in hull computation.

HF_INGESTION_PRIO_radial_metrics_wkb.csv: CSV export with a wkb column (WKB geometries) for PRIO radial metrics points used in hull computation.




Step 2 — Grid generation

This step builds a regular analysis grid over the VILA ∩ PRIO coverage, using 20 km spacing with a 10 km buffer, optionally appending manual nodes (e.g., buoys), and exports the resulting grid nodes and a preview map. The grid nodes serve as the common index where SAR outputs (labels) and HF-Radar inputs (features) are aligned.



Inputs: VILA_PRIO_hull.json (from Step 1); optional buoys_grid_csv.csv (also provided as an output for reproducibility); Athena database ANN_TRAINING.

Method: scripts/grids/create_grid_table.sh materializes the grid into the Athena table; the first call creates a regular grid constrained to the hull + buffer, the second call appends nodes from a CSV; scripts/grids/view_grid.sh produces an interactive preview.

Commands:







./scripts/grids/create_grid_table.sh \ --profile default \ --database ANN_TRAINING \ --hull-file VILA_PRIO_hull.json \ --output-table grid \ --table-location s3://hf-eolus/ann_training/grid \ --output-location s3://hf-eolus/athena-query-results/ \ --node-prefix VILA_PRIO \ --grid-spacing-km 20 \ --buffer-km 10 

./scripts/grids/create_grid_table.sh \ --profile default \ --database ANN_TRAINING \ --manual-csv buoys_grid_csv.csv \ --mode append \ --output-table grid \ --table-location s3://hf-eolus/ann_training/grid \ --output-location s3://hf-eolus/athena-query-results/ 

./scripts/grids/view_grid.sh --input grid_nodes_1757320574.parquet --input grid_nodes_1757320746.parquet --output grid_map.html --title "Grid Preview"





Outputs:



grid_nodes_1757320574.parquet: Parquet with grid nodes generated from the hull-based regular grid (first command).

grid_nodes_1757320746.parquet: Parquet with nodes appended from the manual CSV (second command).

grid_map.html: Leaflet-based interactive map preview titled “Grid Preview”.

buoys_grid_csv.csv: CSV with manual grid nodes used for appending (columns: node_id, longitude, latitude).



Notes: The grid table ANN_TRAINING.grid is stored at s3://hf-eolus/ann_training/grid. The --node-prefix VILA_PRIO is applied to node identifiers.


The final grid consists of 55 nodes in a regular mesh plus one buoy node.

Step 3 — Geospatial mapping to a grid

This step links each observation to its nearest grid node within a 10 km radius, creating row‑to‑node mappings for the SAR and HF-Radar datasets and materializing them as Parquet‑backed Athena tables. These mappings enable joining inputs and outputs by node_id and time to form aligned samples for supervised training.



Inputs: grid table ANN_TRAINING.grid (Step 2); data tables:



SAR_INGEST.SAR (described in HF-EOLUS Sentinel‑1 SAR Derived Data Bundle)

HF_INGESTION_VILA.radial_metrics and HF_INGESTION_PRIO.radial_metrics (described in VILA & PRIO Radial Metrics GeoParquet)



Method: scripts/mapping/geo_mapping.sh creates an Athena CTAS that emits (rowid, node_id) pairs for observations whose geodesic distance to a grid node is ≤ 10 km, materialized as Parquet on S3.

Commands:







./scripts/mapping/geo_mapping.sh --profile default --db-name SAR_INGEST --data-db-name SAR_INGEST --grid-db-name ANN_TRAINING --data-table SAR --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_sar --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_sar 

./scripts/mapping/geo_mapping.sh --profile default --db-name HF_INGESTION_VILA --data-db-name HF_INGESTION_VILA --grid-db-name ANN_TRAINING --data-table radial_metrics --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_vila --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_vila 

./scripts/mapping/geo_mapping.sh --profile default --db-name HF_INGESTION_PRIO --data-db-name HF_INGESTION_PRIO --grid-db-name ANN_TRAINING --data-table radial_metrics --grid-table grid --bucket-name hf-eolus --output-prefix ann_training/geo_join_prio --output-table GRID_JOIN --distance 10 --log-dir ./geo_join_output_prio





Outputs:



Athena tables:



SAR_INGEST.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_sar/

HF_INGESTION_VILA.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_vila/

HF_INGESTION_PRIO.GRID_JOIN → s3://hf-eolus/ann_training/geo_join_prio/



Logs: geo_join_output_sar/geo_mapping.sh.log, geo_join_output_vila/geo_mapping.sh.log, geo_join_output_prio/geo_mapping.sh.log.



Notes: --distance 10 is in kilometers. Existing target tables and S3 prefixes are dropped/cleared before creation to ensure clean outputs.


Step 4 — Aggregation and GeoParquet consolidation

This step aggregates observations by time and grid node (and optional partitions), applying circular statistics for wind direction and projecting radar velocities along the line of sight when required, and then consolidates the outputs with GeoParquet metadata. The resulting per‑node, per‑time aggregates—produced separately for SAR (outputs) and HF-Radar (features)—can be joined downstream on (timestamp, node_id) to build training and validation datasets.



Method:



SAR: aggregate_direction_wrapper.sh applies circular statistics to owiwinddirection (optionally weighted by owiwindspeed), then aggregates; finalize_geoparquet.sh merges files and writes GeoParquet metadata.

VILA/PRIO: aggregate_projection_wrapper.sh projects VELO onto the line joining each grid node to a reference point (radar location), then aggregates; finalize_geoparquet.sh consolidates and repairs partitions.



Commands:







./scripts/aggregation/aggregate_direction_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name SAR_INGEST --grid-db-name ANN_TRAINING --mapping-db-name SAR_INGEST --data-table SAR --grid-table grid --mapping-table GRID_JOIN --columns "owiwinddirection,owiwindspeed" --bucket-name hf-eolus --output-prefix ann_training/aggregation_sar --output-table SAR_AGGREGATED --log-dir ./aggregation_output_sar --timestamp-col date --direction-cols "owiwinddirection" 

./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_sar --output-table SAR_AGGREGATED --log-dir ./aggregation_output_sar 

./scripts/aggregation/aggregate_projection_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name HF_INGESTION_VILA --grid-db-name ANN_TRAINING --mapping-db-name HF_INGESTION_VILA --data-table radial_metrics --grid-table grid --mapping-table GRID_JOIN --columns "Pwr,VELO" --bucket-name hf-eolus --output-prefix ann_training/aggregation_vila --output-table VILA_AGGREGATED --log-dir ./aggregation_output_vila --partition-cols pos_bragg --projection-col "VELO" --point-lat 43.1588833 --point-lon -9.2108333 

./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_vila --output-table VILA_AGGREGATED --log-dir ./aggregation_output_vila --partition-cols pos_bragg 

./scripts/aggregation/aggregate_projection_wrapper.sh --profile default --db-name ANN_TRAINING --data-db-name HF_INGESTION_PRIO --grid-db-name ANN_TRAINING --mapping-db-name HF_INGESTION_PRIO --data-table radial_metrics --grid-table grid --mapping-table GRID_JOIN --columns "Pwr,VELO" --bucket-name hf-eolus --output-prefix ann_training/aggregation_prio --output-table PRIO_AGGREGATED --log-dir ./aggregation_output_prio --partition-cols pos_bragg --projection-col "VELO" --point-lat 43.5680000 --point-lon -8.3140000 

./scripts/aggregation/finalize_geoparquet.sh --profile default --db-name ANN_TRAINING --bucket-name hf-eolus --output-prefix ann_training/aggregation_prio --output-table PRIO_AGGREGATED --log-dir ./aggregation_output_prio --partition-cols pos_bragg





Outputs:



Athena tables and S3 datasets (consolidated as GeoParquet):



ANN_TRAINING.SAR_AGGREGATED → s3://hf-eolus/ann_training/aggregation_sar/

ANN_TRAINING.VILA_AGGREGATED → s3://hf-eolus/ann_training/aggregation_vila/ (partitioned by pos_bragg)

ANN_TRAINING.PRIO_AGGREGATED → s3://hf-eolus/ann_training/aggregation_prio/ (partitioned by pos_bragg)



Logs and saved SQL (attached as ZIPs): aggregation_output_sar.zip, aggregation_output_vila.zip, aggregation_output_prio.zip (contain wrapper logs and the key SQL statements generated during the process).



Notes: finalize_geoparquet.sh downloads the dataset, merges Parquet files per partition, adds GeoParquet metadata for column geometry, syncs back to S3, and runs MSCK REPAIR TABLE when --partition-cols is provided.


Step 5 — STAC catalog and GeoParquet packaging

This step packages the aggregated GeoParquet datasets as portable STAC catalogs by staging assets locally, generating Items and a Collection with the provided property templates, and creating ZIP bundles for distribution. These catalogs facilitate discovery, reuse, and ingestion into analysis or model‑training pipelines.



Inputs: consolidated GeoParquet datasets from Step 4 synced locally; STAC property templates:



stac_properties_item_sar.json, stac_properties_collection_sar.json

stac_properties_item_hf.json, stac_properties_collection_hf.json



Method: sync each aggregated dataset from S3 to a local folder; then run scripts/aggregation/build_stac_catalog.sh to assemble a STAC collection with Items pointing to local GeoParquet assets and produce a ZIP bundle.

Commands:



aws s3 sync s3://hf-eolus/ann_training/aggregation_sar ./sar_aggregated/ --profile default 

aws s3 sync s3://hf-eolus/ann_training/aggregation_vila ./vila_aggregated/ --profile default 

aws s3 sync s3://hf-eolus/ann_training/aggregation_prio ./prio_aggregated/ --profile default 

# Build self-contained STAC catalogs for locally synced aggregates 

./scripts/aggregation/build_stac_catalog.sh \ --collection SAR_AGGREGATED \ --local-source-dir ./sar_aggregated \ --output-dir ./sar_catalog \ --zip-file ./sar_catalog.zip \ --stac-item-properties-json ./stac_properties_item_sar.json \ --stac-collection-properties-json ./stac_properties_collection_sar.json 

./scripts/aggregation/build_stac_catalog.sh \ --collection VILA_AGGREGATED \ --local-source-dir ./vila_aggregated \ --output-dir ./vila_catalog \ --zip-file ./vila_catalog.zip \ --stac-item-properties-json ./stac_properties_item_hf.json \ --stac-collection-properties-json ./stac_properties_collection_hf.json 

./scripts/aggregation/build_stac_catalog.sh \ --collection PRIO_AGGREGATED \ --local-source-dir ./prio_aggregated \ --output-dir ./prio_catalog \ --zip-file ./prio_catalog.zip \ --stac-item-properties-json ./stac_properties_item_hf.json \ --stac-collection-properties-json ./stac_properties_collection_hf.json



Outputs:



STAC catalogs (folders) and ZIP bundles ready to attach:



sar_catalog/ and sar_catalog.zip

vila_catalog/ and vila_catalog.zip

prio_catalog/ and prio_catalog.zip



STAC property JSONs included alongside the catalogs:



stac_properties_collection_hf.json, stac_properties_item_hf.json

stac_properties_collection_sar.json, stac_properties_item_sar.json





Notes: catalogs are self-contained for offline use; Items reference local GeoParquet assets under the catalog’s assets/ directory and include additional properties from the provided JSON templates.


Table statistics

Total rows in each aggregated table:



VILA: 11,197,374

PRIO: 7,439,566

SAR: 7,353


Number of rows by node in each aggregated table:




Node ID
SAR n
VILA n
PRIO n




VILA_PRIO2
129
200,592
24,796


VILA_PRIO11
129
215,430
22,423


VILA_PRIO12
129
240,065
65,294


VILA_PRIO13
129
259,771
73,995


VILA_PRIO14
129
286,584
72,840


VILA_PRIO15
129
282,672
15,168


VILA_PRIO21
129
227,948
51,762


VILA_PRIO22
129
256,650
77,228


VILA_PRIO23
129
280,174
87,157


VILA_PRIO24
129
280,623
105,682


VILA_PRIO25
129
283,889
115,792


VILA_PRIO26
129
232,556
102,868


VILA_PRIO31
129
241,219
55,463


VILA_PRIO32
129
252,427
77,025


VILA_PRIO33
129
253,449
85,888


VILA_PRIO34
129
253,465
114,046


VILA_PRIO35
129
272,197
123,962


VILA_PRIO36
129
258,016
143,172


VILA_PRIO37
129
139,492
184,669


VILA_PRIO41
129
218,703
59,621


VILA_PRIO42
129
212,609
90,817


VILA_PRIO43
129
230,476
111,725


VILA_PRIO44
129
190,620
147,695


VILA_PRIO45
129
256,603
178,326


VILA_PRIO46
129
243,969
198,894


VILA_PRIO47
129
227,560
212,324






VILA_PRIO48






129






0






216,702




VILA_PRIO51
129
164,288
61,374


VILA_PRIO52
129
191,728
117,335


VILA_PRIO53
129
191,794
139,647


VILA_PRIO54
129
166,493
170,011


VILA_PRIO55
129
238,227
190,413


VILA_PRIO56
129
219,641
201,207


VILA_PRIO57
129
220,627
212,314


VILA_PRIO58
129
183,865
167,566


VILA_PRIO62
129
172,199
131,575


VILA_PRIO63
129
149,007
145,094


VILA_PRIO64
129
155,628
165,541


VILA_PRIO65
129
211,356
179,106


VILA_PRIO66
129
183,756
200,832


VILA_PRIO67
129
182,008
183,382


VILA_PRIO68
129
171,708
135,059


VILA_PRIO69
129
121,474
208,998


VILA_PRIO72
129
132,360
111,845


VILA_PRIO73
129
117,432
141,909


VILA_PRIO74
129
131,752
136,387


VILA_PRIO75
129
175,633
180,928


VILA_PRIO76
129
158,828
183,215


VILA_PRIO77
129
152,795
153,647


VILA_PRIO78
129
150,911
125,660


VILA_PRIO79
129
123,177
202,434


VILA_PRIO83
129
72,122
103,911


VILA_PRIO84
129
109,102
149,200


VILA_PRIO85
129
145,492
158,288


VILA_PRIO86
129
127,539
146,056


VILA_PRIO87
129
103,087
100,134


Vilano_buoy
129
275,586
121,164


Total
7,353
11,197,374
7,439,566




 

Acknowledgements

This work has been funded by the HF-EOLUS project (TED2021-129551B-I00), financed by MICIU/AEI /10.13039/501100011033 and by the European Union NextGenerationEU/PRTR - BDNS 598843 - Component 17 - Investment I3. Members of the Marine Research Centre (CIM) of the University of Vigo have participated in the development of this repository.

Disclaimer

This software is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages, or other liability, whether in an action of contract, tort, or otherwise, arising from, out of, or in connection with the software or the use or other dealings in the software.