Repository: gammasim/simtools (language: python)

Issue:
Change references from CTA to CTAO throughout the code
              This made me realize that we still have a lot of references to "CTA" in the code, whereas it should now be "CTAO". Should we open an issue to change that?

_Originally posted by @VictorBarbosaMartins in https://github.com/gammasim/simtools/pull/928#discussion_r1613014014_
            


Discussion / hints:

Fix patch:
diff --git a/docs/source/api-reference/data_model.md b/docs/source/api-reference/data_model.md
index fe6be668b..6efa0a2c2 100644
--- a/docs/source/api-reference/data_model.md
+++ b/docs/source/api-reference/data_model.md
@@ -3,7 +3,7 @@
 # Data Model
 
 Modules in the `data_model` sections provide functionality for reading, writing, and validation of data.
-Data products ingested or produced by simtools generally follows the CTA data model.
+Data products ingested or produced by simtools generally follows the CTAO data model.
 
 (datareader)=
 
diff --git a/docs/source/developer-guide/continuous_integration.md b/docs/source/developer-guide/continuous_integration.md
new file mode 100644
index 000000000..c220e28bc
--- /dev/null
+++ b/docs/source/developer-guide/continuous_integration.md
@@ -0,0 +1,52 @@
+# Continuous Integration
+
+The simtools continuous integration tests, lints, and builds code, documentation and binaries. The CI is based on GitHub Actions and defined in the `.github/workflows` directory.
+
+## CI Units and Integration tests
+
+Tests are implemented in the CI as:
+
+- [unit tests](testing.md#unit-tests) in the [CI-unittests.yml](.github/workflows/CI-unittests.yml) workflow.
+- [integration tests](testing.md#integration-tests) in the [CI-integrationtests.yml](.github/workflows/CI-integrationtests.yml) workflow.
+
+Unit tests are running for different python versions  and installation types (mamba, pip).
+There is a scheduled run of unit tests over the main branch each night, which includes also a random-order execution of the tests (to find inter-dependencies between modules).
+
+:::{important}
+Simtools uses the CTAO sonar quality gates for code quality. Passing those gates is a requirement for merging code into the main branch.
+See the [simtools sonar project page](https://sonar-cta-dpps.zeuthen.desy.de/dashboard?id=gammasim_simtools_AY_ssha9WiFxsX-2oy_w) for the current status.
+:::
+
+Integration tests are running for two different model versions.
+
+## CI Linting
+
+Consistent code style is enforced by running the following linters:
+
+- linters defined as pre-commit workflow, see [.pre-commit-config.yaml](../.pre-commit-config.yaml)
+- files are checked for non-ascii characters
+- linting of pyproject.toml
+- linting of CITATION.cff
+- running pylint on the code
+- linting docker files using [hadolint](https://github.com/hadolint/hadolint)
+- validate env files
+- check for natural language
+
+## CI Schema files
+
+All schema files are validated using the `validate_file_using_schema.py` tool of simtools.
+Schema CI is define in [CI-schema-validation.yml](.github/workflows/CI-schema-validation.yml).
+
+## CI Documentation
+
+Documentation is built using sphinx after every merge to main.
+The documentation is built into the [CI-docs.yml](.github/workflows/CI-docs.yml) workflow.
+
+## CI Binary builds
+
+Binary builds are done in two ways:
+
+- containers for CORSIKA and sim_telarray, see [build-docker-corsika-simtelarray-image.yml](.github/workflows/build-docker-corsika-simtelarray-image.yml)
+- containers for simtools, see [build-docker-images.yml](.github/workflows/build-docker-images.yml
+
+Builds are done for the platforms: linux/amd64,linux/arm64/v8.
diff --git a/docs/source/developer-guide/developer_guidelines.md b/docs/source/developer-guide/developer_guidelines.md
index 35bcff841..553b6f828 100644
--- a/docs/source/developer-guide/developer_guidelines.md
+++ b/docs/source/developer-guide/developer_guidelines.md
@@ -38,7 +38,7 @@ The following steps outline how to contribute code to simtools:
 01. Set up your coding environment as outlined in the [getting started](getting_started.md#getting-started) section.
 02. Start a new feature branch from the main branch (`git checkout -b new-branch-name`).
 03. Implement your code changes.
-04. Add unit tests for new modules and functions.
+04. Add unit tests for new modules and functions (see [unit test section](testing.md#unit-tests)).
 05. Commit your code changes (use meaningful commit messages) and push them to GitHub.
 06. Create a draft pull request on GitHub when all features are implemented.
 07. Wait for the CI tests to finish and address any issues that arise.
@@ -48,38 +48,6 @@ The following steps outline how to contribute code to simtools:
 
 Note the [guidelines on pull requests](pull_requests.md#pull-requests).
 
-## Unit and Integration Testing
-
-The pytest framework is used for testing:
-
-- unit tests should be written for every module and function
-- integration tests should be written for every application and cover the most important use cases. Integration tests should follow the logic of the existing tests in [simtools/tests/integration_tests](https://github.com/gammasim/simtools/tree/main/tests/integration_tests/).
-
-The test modules are located in
-[simtools/tests](https://github.com/gammasim/simtools/tree/main/tests) separated
-by unit and integration tests.
-It is recommended to write unit tests in parallel with the modules to assure that the code is testable.
-
-General service functions for tests (e.g., DB connection) can be found in
-[conftest.py](https://github.com/gammasim/simtools/blob/main/tests/conftest.py).
-This should be used to avoid duplication.
-
-:::{note}
-Developers should expect that code changes affecting several modules are acceptable in case unit tests are successful.
-:::
-
-The [pytest-xdist](https://pytest-xdist.readthedocs.io/en/latest/) plugin is part of the developer environment
-and can be used to run unit and integration tests in parallel (e.g., `pytest -n 4` to run on four cores in parallel).
-
-Tests might pass just because they run after an unrelated test. In order to test the independence of unit tests, use the
-[pytest-random-order](https://pypi.org/project/pytest-random-order/) plugin with `pytest --random-order`.
-
-Check the test coverage with `pytest -vv -n auto tests/unit_tests/ tests/integration_tests/ --cov`.
-Add the `--cov-report html` option to generate a coverage report in HTML format.
-
-It might be quite useful to test a single application while developing.
-This can be done with e.g. `pytest -v -k "simtools-convert-all-model-parameters-from-simtel" tests/integration_tests/test_applications_from_config.py` to run all the integrations tests for the `simtools-convert-all-model-parameters-from-simtel` application.
-Alternatively, run `pytest -v -k "simtools-convert-all-model-parameters-from-simtel_num_gains" tests/integration_tests/test_applications_from_config.py` to run the tool for the specific test called `num_gains`. Test names are found in the yml files in the `tests/integration_tests/config` directory.
 ## Writing Applications
 
 Applications are command-line tools that should be built of the simtools library.
diff --git a/docs/source/developer-guide/index.md b/docs/source/developer-guide/index.md
index 985a85527..eb767959f 100644
--- a/docs/source/developer-guide/index.md
+++ b/docs/source/developer-guide/index.md
@@ -1,9 +1,11 @@
 # Developer guidelines
 
 ```{toctree}
-:maxdepth: 2
+:maxdepth: 1
 developer_guidelines.md
+testing.md
 documentation.md
 pull_requests.md
 coding_guidelines.md
+continuous_integration.md
 ```
diff --git a/docs/source/user-guide/getting_started.md b/docs/source/user-guide/getting_started.md
index 2b7b86eea..fcd56b1b5 100644
--- a/docs/source/user-guide/getting_started.md
+++ b/docs/source/user-guide/getting_started.md
@@ -140,8 +140,8 @@ CORSIKA and sim_telarray are external tools to simtools and are required depende
 Recommended is to use the Docker environment, see description in  [Docker Environment for Developers].
 
 For a non-Docker setup, follow the instruction provided by the CORSIKA/sim_telarray authors for installation.
-CTA users can download both packages from the [sim_telarray web page](https://www.mpi-hd.mpg.de/hfm/CTA/MC/Software/Testing/)
-(CTA password applies) and install the package with:
+CTAO users can download both packages from the [sim_telarray web page](https://www.mpi-hd.mpg.de/hfm/CTA/MC/Software/Testing/)
+(CTAO password applies) and install the package with:
 
 ```console
 tar -czf corsika7.7_simtelarray.tar.gz


Test patch:
diff --git a/docs/source/developer-guide/testing.md b/docs/source/developer-guide/testing.md
new file mode 100644
index 000000000..d4ca2566f
--- /dev/null
+++ b/docs/source/developer-guide/testing.md
@@ -0,0 +1,130 @@
+# Testing
+
+Automatic code testing is absolutely essential for the development of simtools. It ensures that the code is working as expected and that new features do not break existing functionality.
+
+:::{important}
+Developers should expect that code changes affecting several modules are acceptable in case unit tests are successful.
+:::
+
+The testing should be done at two levels:
+
+- unit tests: test individual modules and functions
+- integration tests: test applications as a whole using typical use cases
+
+The [pytest](https://docs.pytest.org) framework is used for testing.
+The test modules are located in
+[simtools/tests](https://github.com/gammasim/simtools/tree/main/tests) separated
+by unit and integration tests.
+
+## Pytest fixtures
+
+General service functions for tests (e.g., DB connection) are defined as `pytest.fixtures` in
+[conftest.py](https://github.com/gammasim/simtools/blob/main/tests/conftest.py).
+This should be used to avoid duplication.
+
+Define fixtures as local as possible: fixtures which are used in a single test module should be defined at the top of the the same module.
+
+All fixtures can be listed with:
+
+```bash
+pytest --fixtures
+```
+
+Fixtures should therefore include a docstring.
+
+## Unit tests
+
+Unit tests should be written for every module and function and should test the result of individual functions ("units").
+It is recommended to write unit tests in parallel with the modules to assure that the code is testable.
+
+:::{warning}
+The simtools project aims for a high test coverage.
+Code lines without coverage should be the exception and the aim should be to achieve a coverage greater than 90%
+(the CTAO quality requirements aim for a coverage of at least 80%).
+:::
+
+Check the test coverage with `pytest -n auto --cov-report-html  tests/unit_tests/ tests/integration_tests/`.
+Coverage reports can be accessed through the `htmlcov/index.html` file.
+
+Good practice:
+
+- at least one test for every function in a module clearly testing the function's behavior (including edge cases).
+- sort tests in the testing module in the same sequence as the functions in the module.
+- unit tests need to be fast.
+
+## Integration tests
+
+Integration tests should be written for every application and cover the most important use cases.
+Integration tests should follow the logic of the existing tests in [simtools/tests/integration_tests](https://github.com/gammasim/simtools/tree/main/tests/integration_tests/).
+Any configuration file in the [simtools/tests/integration_tests/config](https://github.com/gammasim/simtools/tree/main/tests/integration_tests/config) directory is used for testing.
+
+For testing a single applications, use the `-k` option to select the test by name:
+
+```bash
+pytest -v -k "simtools-convert-all-model-parameters-from-simtel" tests/integration_tests/test_applications_from_config.py
+```
+
+To run a specific tests, use the `-k` option with the test name:
+
+```bash
+pytest -v -k "simtools-convert-all-model-parameters-from-simtel_num_gains" tests/integration_tests/test_applications_from_config.py
+```
+
+This runs to run the tool for the specific test called `num_gains`.
+
+The integration test module allows for additional output tests (see the [simtools/tests/integration_tests/config](https://github.com/gammasim/simtools/tree/main/tests/integration_tests/config) directory for examples).
+
+Test that a given output file exists:
+
+```text
+  INTEGRATION_TESTS:
+    - OUTPUT_FILE: ray-tracing/results/ray-tracing-North-LSTN-01-d11.0km-za20.0deg_validate_optics.ecsv
+```
+
+Compare tool output with a given reference files (in the following example with a tolerance of 1.e-2 for floating point comparison):
+
+```text
+REFERENCE_OUTPUT_FILES:
+  - REFERENCE_FILE: ray-tracing/results/ray-tracing-North-LSTN-01-d11.0km-za20.0deg_validate_optics.ecsv
+    TOLERANCE: 1.e-2
+```
+
+This is implemented for any astropy table-like file.
+
+Test for a certain file type:
+
+```text
+- FILE_TYPE: json
+```
+
+### Model versions for integration tests
+
+The integration tests run for two different model versions, defined in the matrix in the [CI-integrationtests.yml](https://github.com/gammasim/simtools/blob/main/.github/workflows/CI-integrationtests.yml) workflow.
+
+As model versions and definitions change, not all integration tests will work with all model versions.
+Use the entry `MODEL_VERSION_USE_CURRENT` in the configuration file to fix the model version to the one defined in the configuration file.
+
+## Testing utilities
+
+The [pytest-xdist](https://pytest-xdist.readthedocs.io/en/latest/) plugin is part of the developer environment
+and can be used to run unit and integration tests in parallel (e.g., `pytest -n 4` to run on four cores in parallel).
+
+Tests might pass just because they run after an unrelated test. In order to test the independence of unit tests, use the
+[pytest-random-order](https://pypi.org/project/pytest-random-order/) plugin with `pytest --random-order`.
+
+## Profiling tests
+
+Tests should be reasonably fast.
+
+To identify slow tests, run the tests with the `--durations` option to list e.g. the 10 slowest tests:
+
+```bash
+pytest --durations=10
+```
+
+For profiling, use the [pytest-profiling](https://pypi.org/project/pytest-profiling/) plugin.
+Example:
+
+```bash
+pytest --no-cov --profile tests/unit_tests/utils/test_general.py
+```