DOI: 10.5281/zenodo.17670939
title: Atmopheric Machine Learning Emulation Challenge, 1st Ed. (AMLEC-1)

description: Note: Unzip files to have access to the original data files (.h5, .csv, .xml)

This dataset contains the train and test data used for the first edition of the Atmospheric Machine Learning Emulation Challenge (AMLEC-1), presented at ECMLPKDD 2025 (https://ecmlpkdd.org/2025/discovery-challenges/) and carried out within the EU ELIAS project (https://elias-ai.eu/opportunities/amlec/).

The dataset contains a series of .h5 storing MODTRAN6 spectral simulations (transmittances, spherical albedo, path radiance), computed at a with various atmospheric and geometric conditions and two scenarios --atmospheric correction of hyperspectral data (A) and CO2 retrieval (B)-- each associated with its own spectral configuration.

The training data (i.e., inputs and outputs of RTM simulations) is stored in HDF5 format with the following structure:






Dimensions
 




Name
Description


n_wl
Number of wavelengths for which spectral data is provided


n_funcs
Number of atmospheric transfer functions


n_comb
Number of data points at which spectral data is provided


n_param
Dimensionality of the input variable space










Data Components
 
 
 




Name
Description
Dimensions
Datatype


LUTdata
Atmospheric transfer functions (i.e. outputs)
n_funcs*n_wvl x n_comb
single


LUTHeader
Matrix of input variable values for each combination (i.e., inputs)
n_param x n_comb
double


wvl
Wavelength values associated with the atmospheric transfer functions (i.e., spectral grid)
n_wvl
double





Note: Participants may choose to predict the spectral data either as a single vector of length n_funcs*n_wvl or as n_funcs separate vectors of lenght n_wvl.

Testing input datasets (i.e., input for predictions) are stored in a tabulated .csv format with dimensions n_param x n_comb. During the challenge, participants only had access to this .csv data, while here we also provide the reference spectral simulations using for evaluation

The training and testing dataset will be organized organized into scenario-specific folders: scenarioA (Atmospheric Correction), and scenarioB (CO2 Column Retrieval). Each folder will contain:



A train with multiple .h5 files corresponding to different training sample sizes (e.g. train2000.h5contains 2000 samples).

A reference subfolder containg two test files (refInterp and refExtrap) referring to the two aforementioned tracks (i.e., interpolation and extrapolation).


Here is an example of how to load each dataset in python:



import h5py
import pandas as pd
import numpy as np

# Replace with the actual path to your training and testing data
trainFile = 'train2000.h5'
testFile = 'refInterp.csv'

# Open the H5 file
with h5py.File(file_path, 'r') as h5_file
Ytrain = h5_file['LUTdata'][:]
Xtrain = h5_file['LUTHeader'][:]
wvl = h5_file['wvl'][:]

# Read testing data
df = pd.read_csv(testFile)
Xtest = df.to_numpy()



in Matlab:



# Replace with the actual path to your training and testing data
trainFile = 'train2000.h5';
testFile = 'refInterp.csv';

# Open the H5 file
Ytrain = h5read(trainFile,'/LUTdata');
Xtrain = h5read(trainFile,'/LUTheader');
wvl = h5read(trainFile,'/wvl');

# Read testing data
Xtest = importdata(testFile);



and in R language:



library(rhdf5)

# Replace with the actual path to your training and testing data
trainFile <- "train2000.h5"
testFile <- "refInterp.csv"

# Open the H5 file
lut_data <- h5read(file_path, "LUTdata")
lut_header <- h5read(file_path, "LUTHeader")
wavelengths <- h5read(file_path, "wvl")

# Read testing data
Xtest <- as.matrix(read.table(file_path, sep = ",", header = TRUE))