Package: otbench
otbench- Effective Benchmarks for Optical Turbulence ModelingOverviewotbenchis a Python package for benchmarking optical turbulence models.The package currently covers regression and forecasting for modeling the strength of optical turbulence as measured during diverse operational studies.otbenchis centered around "Tasks" which specify a dataset, target variable, data splits, evaluation metrics, and operational metadata. The package provides a consistent interface for running models on these tasks, and for comparing the results of different models.otbenchis designed to be easy to use, and to provide a consistent interface for comparing different models. It is also designed to be extensible, so that new tasks and datasets can be added easily. The package is built on top ofxarray,scikit-learnandpandas, and is designed to be compatible with the broader Python data science ecosystem.FeaturesEnvironmentally Diverse Datasetsotbenchincludes a variety of datasets from different operational studies. These datasets include measurements of optical turbulence strength, as well as environmental data such as wind speed, temperature, and relative humidity. The datasets are stored asxarrayobjects, which provide a consistent interface for working with the data.Consistent Interface for Model EvaluationComparing different models can be difficult, especially when the models are trained and evaluated on different datasets.otbenchprovides a consistent interface for evaluating models on different tasks. This makes it easy to compare the performance of different models, and to compare the performance of a single model on different tasks.Extensibleotbenchis designed to be extensible, so that new tasks and datasets can be added easily. We welcome any and all outside contributions, including new datasets, tasks, and baseline models. As the number of datasets grows,otbenchwill become an increasingly useful tool for comparing different models, the applicability of specific models across propagation environments and over time, and identifying best practices for across the optics and machine learning communities.InstallationYou can installotbenchusing a Python package manager such aspip.The package is available on PyPi, versioned using the date of release. We will work to ensure compatibility with the last three major versions of Python, but have designed the package to be compatible with Python 3.6 and above.From PyPiPyPi is the recommended installation method. You can install the latest release usingpip:pipinstallotbenchFrom SourceIf you want to install the latest development version, or would like to make a contribution you can install from source. You can clone the repository usinggit:gitclonehttps://github.com/CDJellen/otbench.gitcdotbench
pipinstall-e.RequirementsThe current dependencies are listed inrequirements.txt. We strongly recommend installing the package in a virtual environment.Requirements:numpy>=1.22
pandas==1.3.5
scikit-learn>=1.0.1
xarray>=2022.3.0
netcdf4>=1.6.0
matplotlib==3.5.3
requests==2.22.0
astral==2.2You can install them usingpip:pipinstall-rrequirements.txtIf you would like to make a contribution, please install the development dependencies as well:pipinstall-rrequirements-dev.txtUsageThe package seeks to balance ease of use with evaluation extensibility across ecosystems. Examples of how to use the package are provided in thenotebooksdirectory.TasksA deeper overview of the package structure, the concept of a "Task", and the available tasks is provided inthis notebook.The task object, either aRegressionTaskor aForecastingTask, provides a consistent interface for obtaining the full underlying dataset, specific data slices, evaluating models, and comparing newly-developed models against existing benchmarks. It specifies the dataset, target variable, data splits, evaluation metrics, and operational metadata. The package provides a consistent interface for running models on these tasks, and for comparing the results of different models.Tasks are obtained by name using theTaskApi:fromotbenchimportTaskApitask_api=TaskApi()# use the task which removes missing measurementsmlo_cn2=task_api.get_task("regression.mlo_cn2.dropna.Cn2_15m")# obtain the full datasetds=mlo_cn2.get_dataset()# obtain the task-specific training dataX_train,y_train=mlo_cn2.get_train_data()# evaluate a model on the taskmodel=...mlo_cn2.evaluate_model(predict_call=model.predict)# compare a model against the task's benchmarksmlo_cn2.get_benchmark_info()# add the model to the task's benchmarksmlo_cn2.evaluate_model(predict_call=model.predict,include_as_benchmark=True,model_name="my_model",)The full set of tasks, along with their complete definition and metadata, is available as aJSON file.RegressionWe provide an overview of the process for training a regression model on themlo_cn2dataset^1using theLightGBMpackage. We develop a Gradient Boosted Regression Tree (GBRT)model^2for predicting the strength of optical turbulence inthis notebook.ForecastingWe provide a similar overview for the process of developing a forecasting model specific to the fullusna_cn2_smforecasting task. Under this task, the next measurement of $C_n^2$ is predicted using the prior 6 observations of $C_n^2$ and the available environmental variables. The example model and forecasting baseline are available inthis notebook.ContributingAll contributions are welcome! We are especially interested in contributions that add new datasets, tasks, and benchmark models. We are also interested in contributions that improve the documentation, add new examples, or improve the codebase. When contributing, please follow thecontributing guidelines, and ensure that your code passes the tests and linting. If you have any questions, please feel free to open an issue or reach out to the maintainers.Adding new datasetsNew datasets can be added by following the instructions in thedata documentation. We strongly prefer datasets to conform to the NetCDF4 format, and to be stored in thedata/directory. If you have a dataset that you would like to add, but are unable to convert it to NetCDF4, please reach out to the maintainers.Adding new tasksNew tasks, if they are based on existing datasets, can be added directly tothe task specifications file.  If you would like to add a new task which uses a new dataset, please include the new dataset first or on the same pull request. The existing tasks provide a good template for how to specify a new task. More information is available in thedata documentation. If you have any questions, please reach out to the maintainers.We strive to avoid a combinatorial explosion of tasks, and to ensure that the tasks are representative of the broader optical turbulence modeling problem. If you have any questions about whether a new task is appropriate, please reach out to the maintainers.Adding new benchmark modelsThe set of benchmark models in the current release ofotbenchare relatively sparse, and serve mostly to gauge whether a new model provides any improvement over standard climatology or persistence techniques. Some literature models are also included to asses their performance against new data collected over a diverse set of measurement campaigns. We welcome contributions of new benchmark models, especially those that are designed to be more representative of the broader optical turbulence modeling problem, and which can generate predictions across multiple datasets.Other contributions to the codebaseThe set of evaluation metrics and plotting functions is relatively sparse, and we welcome contributions of new metrics and plots. We also welcome contributions that improve the documentation, add new examples, or improve the codebase. If you have any questions, please feel free to open an issue or reach out to the maintainers.CitingotbenchIf you useotbenchin your research, please cite the package using the following BibTeX entry:@misc{jellen2023otbench,author={Jellen, Christopher and Nelson, Charles and Burkhardt, John and Brownell, Cody}.title={otbench:EffectiveBenchmarksforOpticalTurbulenceModeling},howpublished = {\url{github.com/CDJellen/otbench}},year = {2023},}In addition to the citation reference and any other acknowledgements:Please acknowledge NCAR/EOL in your publications with text such as:Data provided by NCAR/EOL under the sponsorship of the National Science Foundation.https://data.eol.ucar.edu/Please acknowledge ONR and DE-JTO in your publications with text such as:Data provided by the United States Naval Academy under the sponsorship of the Office of Naval Research and the Directed Energy Joint Technology Office.