Repository: theroggy/cropclassification (language: python)

Issue:
ENH: add feature cross-prediction-models
The cross-prediction-models feature can be used to avoid predicting a marker on a parcel where the parcel was part of the training dataset.

When the number of cross_pred_models = 2 for example, the input data is split in 2 (per class). Each 50% of the data will be used to train one model, and this model will be used to predict the other 50% of the parcels.


Discussion / hints:
## [Codecov](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) Report
Attention: Patch coverage is `52.70270%` with `35 lines` in your changes missing coverage. Please review.
> Project coverage is 60.42%. Comparing base [(`8c08a6b`)](https://app.codecov.io/gh/theroggy/cropclassification/commit/8c08a6be6a5f39261ee4e4f2c4867c79fe8b956c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) to head [(`316b9c3`)](https://app.codecov.io/gh/theroggy/cropclassification/commit/316b9c376d5846f457ebc29f74018c272d4a8357?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) | Patch % | Lines |
|---|---|---|
| [cropclassification/calc\_marker.py](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&filepath=cropclassification%2Fcalc_marker.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans#diff-Y3JvcGNsYXNzaWZpY2F0aW9uL2NhbGNfbWFya2VyLnB5) | 56.36% | [24 Missing :warning: ](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) |
| [cropclassification/predict/classification.py](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&filepath=cropclassification%2Fpredict%2Fclassification.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans#diff-Y3JvcGNsYXNzaWZpY2F0aW9uL3ByZWRpY3QvY2xhc3NpZmljYXRpb24ucHk=) | 55.55% | [4 Missing :warning: ](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) |
| [...tion/preprocess/\_classification\_preprocess\_BEFL.py](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&filepath=cropclassification%2Fpreprocess%2F_classification_preprocess_BEFL.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans#diff-Y3JvcGNsYXNzaWZpY2F0aW9uL3ByZXByb2Nlc3MvX2NsYXNzaWZpY2F0aW9uX3ByZXByb2Nlc3NfQkVGTC5weQ==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) |
| [...sification/preprocess/classification\_preprocess.py](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&filepath=cropclassification%2Fpreprocess%2Fclassification_preprocess.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans#diff-Y3JvcGNsYXNzaWZpY2F0aW9uL3ByZXByb2Nlc3MvY2xhc3NpZmljYXRpb25fcHJlcHJvY2Vzcy5weQ==) | 50.00% | [2 Missing :warning: ](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) |
| [cropclassification/helpers/pandas\_helper.py](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&filepath=cropclassification%2Fhelpers%2Fpandas_helper.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans#diff-Y3JvcGNsYXNzaWZpY2F0aW9uL2hlbHBlcnMvcGFuZGFzX2hlbHBlci5weQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
- Coverage   60.72%   60.42%   -0.30%     
==========================================
  Files          35       35              
  Lines        4461     4516      +55     
==========================================
+ Hits         2709     2729      +20     
- Misses       1752     1787      +35     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/theroggy/cropclassification/pull/142?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Pieter+Roggemans).


Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 52c3f3f..0bce8c8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,7 +26,7 @@
 ### Improvements
 
 - Add feature "cross-prediction-models" to avoid markers being calculated on parcels
-  using a model that used this parcel in its training (#142)
+  using a model that used this parcel in its training (#142, #143)
 - Consolidated some landcover pre-processing ignore codes (#120)
 - Add some extra global accuracies (precision, recall, f1) to report (#119)
 - Add option `images.on_missing_images` to be able to ignore errors in the
diff --git a/bin_util/recalc_reporting.py b/bin_util/recalc_reporting.py
index 4eb2670..323b5c4 100644
--- a/bin_util/recalc_reporting.py
+++ b/bin_util/recalc_reporting.py
@@ -4,7 +4,7 @@
 import cropclassification.helpers.config_helper as conf
 from cropclassification.postprocess import classification_postprocess as class_post
 from cropclassification.postprocess import classification_reporting as class_report
-from cropclassification.preprocess import classification_preprocess as class_pre
+from cropclassification.preprocess import prepare_input as class_pre
 
 
 def main():
@@ -109,7 +109,7 @@ def main():
     classtype_to_prepare = conf.preprocess["classtype_to_prepare"]
 
     # Prepare parcel input file
-    class_pre.prepare_input(
+    class_pre.prepare(
         input_parcel_path=input_parcel_nogeo_path,
         input_parcel_filetype=input_parcel_filetype,
         timeseries_periodic_dir=timeseries_periodic_dir,
@@ -166,7 +166,7 @@ def main():
             run_dir
             / f"{input_groundtruth_path.stem}_classes{input_groundtruth_path.suffix}"
         )
-        class_pre.prepare_input(
+        class_pre.prepare(
             input_parcel_path=input_groundtruth_path,
             input_parcel_filetype=input_parcel_filetype,
             timeseries_periodic_dir=timeseries_periodic_dir,
diff --git a/cropclassification/calc_marker.py b/cropclassification/calc_marker.py
index 9b3864f..4d0ba16 100644
--- a/cropclassification/calc_marker.py
+++ b/cropclassification/calc_marker.py
@@ -9,19 +9,16 @@
 from pathlib import Path
 
 # Import geofilops here already, if tensorflow is loaded first leads to dll load errors
-import geofileops as gfo
-import pandas as pd
+import geofileops as gfo  # noqa: F401
 import pyproj
 
-import cropclassification.helpers.pandas_helper as pdh
 from cropclassification.helpers import config_helper as conf
 from cropclassification.helpers import dir_helper, log_helper
-from cropclassification.helpers import model_helper as mh
 from cropclassification.postprocess import classification_postprocess as class_post
 from cropclassification.postprocess import classification_reporting as class_report
 from cropclassification.predict import classification
 from cropclassification.preprocess import _timeseries_helper as ts_helper
-from cropclassification.preprocess import classification_preprocess as class_pre
+from cropclassification.preprocess import prepare_input
 from cropclassification.preprocess import timeseries as ts
 
 # -------------------------------------------------------------
@@ -235,7 +232,7 @@ def calc_marker_task(
     min_parcels_in_class = conf.preprocess.getint("min_parcels_in_class")
     parcel_path = run_dir / f"{input_parcel_filename.stem}_parcel{data_ext}"
     base_filename = f"{input_parcel_filename.stem}_bufm{buffer:g}_weekly"
-    class_pre.prepare_input(
+    prepare_input.prepare(
         input_parcel_path=input_parcel_nogeo_path,
         input_parcel_filetype=input_parcel_filetype,
         timeseries_periodic_dir=timeseries_periodic_dir,
@@ -262,146 +259,27 @@ def calc_marker_task(
         parceldata_aggregations_to_use=parceldata_aggregations_to_use,
     )
 
-    # STEP 4: Train and test if necessary... and predict
-    # -------------------------------------------------------------
-    markertype = conf.marker.get("markertype")
-    parcel_predictions_proba_all_path = (
-        run_dir / f"{base_filename}_predict_proba_all{data_ext}"
-    )
-    parcel_predictions_proba_test_path = (
-        run_dir / f"{base_filename}_predict_proba_test{data_ext}"
-    )
-    cross_pred_models = conf.classifier.get("cross_pred_models", 1)
-    cross_pred_models = (
-        1
-        if cross_pred_models is None or int(cross_pred_models) < 1
-        else int(cross_pred_models)
+    # STEP 4: Train a model and predict
+    # ---------------------------------
+    cross_pred_models = conf.classifier.getint("cross_pred_models", 1)
+    test_size = conf.classifier.getfloat("test_size")
+
+    (
+        parcel_predictions_proba_all_path,
+        parcel_predictions_proba_test_path,
+        parcel_train_path,
+        parcel_test_path,
+    ) = classification.classify(
+        classifier_type=conf.classifier["classifier_type"],
+        parcel_path=parcel_path,
+        parcel_classification_data_path=parcel_classification_data_path,
+        output_dir=run_dir,
+        output_base_filename=base_filename,
+        test_size=test_size,
+        cross_pred_models=cross_pred_models,
+        input_model_to_use_path=input_model_to_use_path,
+        force=False,
     )
-    classifier_ext = conf.classifier["classifier_ext"]
-    cross_pred_model_id_column = conf.columns["cross_pred_model_id"]
-    parcel_test_path = None
-    parcel_train_path = None
-
-    # If the predictions file doesn't exist, do the classification
-    if not parcel_predictions_proba_all_path.exists():
-        if cross_pred_models > 1:
-            # Use "cross models": train multiple models, so prediction of a parcel is
-            # always done with a model where the parcel wasn't used to train it.
-
-            # Assign each parcel to a model where it will be predicted with.
-            gfo.add_column(
-                path=parcel_path,
-                name=cross_pred_model_id_column,
-                type="int",
-                expression=f"ABS(RANDOM() % {cross_pred_models})",
-            )
-            if input_model_to_use_path is not None:
-                raise ValueError(
-                    "cross_pred_models not supported with input_model_to_use_path"
-                )
-
-        pred_test_files = []
-        pred_all_files = []
-        for cross_pred_model_idx in range(cross_pred_models):
-            if cross_pred_models == 1:
-                # Only one model, so no need to create subdirectory
-                model_dir = run_dir
-
-                # Output can be the final file immediately
-                parcel_preds_proba_test_model_path = parcel_predictions_proba_test_path
-                parcel_preds_proba_all_model_path = parcel_predictions_proba_all_path
-
-                # No need to filter percels based on cross model indexes
-                training_query = None
-                predict_query = None
-            else:
-                # Create a subfolder for each model
-                model_dir = run_dir / f"cross_pred_model_{cross_pred_model_idx}"
-                model_dir.mkdir(parents=True, exist_ok=True)
-
-                # Temporary output files in the subdirectory
-                parcel_preds_proba_test_model_path = (
-                    model_dir / parcel_predictions_proba_test_path.name
-                )
-                pred_test_files.append(parcel_preds_proba_test_model_path)
-                parcel_preds_proba_all_model_path = (
-                    model_dir / parcel_predictions_proba_all_path.name
-                )
-                pred_all_files.append(parcel_preds_proba_all_model_path)
-
-                # The training will be done on all parcels with another index
-                training_cross_pred_model_ids = [
-                    idx
-                    for idx in range(cross_pred_models)
-                    if idx != cross_pred_model_idx
-                ]
-                training_query = (
-                    f"{cross_pred_model_id_column} in {training_cross_pred_model_ids}"
-                )
-                predict_query = (
-                    f"{cross_pred_model_id_column} == {cross_pred_model_idx}"
-                )
-
-            # Check if a model exists already
-            if input_model_to_use_path is None:
-                best_model = mh.get_best_model(model_dir, acc_metric_mode="min")
-                if best_model is not None:
-                    input_model_to_use_path = best_model["path"]
-
-            # If we cannot reuse an existing model, train one!
-            parcel_train_path = model_dir / f"{base_filename}_parcel_train{data_ext}"
-            parcel_test_path = model_dir / f"{base_filename}_parcel_test{data_ext}"
-            classifier_basepath = model_dir / f"{markertype}_01_mlp{classifier_ext}"
-            if input_model_to_use_path is None:
-                # Create the training sample.
-                # Remark: this creates a list of representative test parcel + a list of
-                # (candidate) training parcel
-                balancing_strategy = conf.marker["balancing_strategy"]
-                class_pre.create_train_test_sample(
-                    input_parcel_path=parcel_path,
-                    output_parcel_train_path=parcel_train_path,
-                    output_parcel_test_path=parcel_test_path,
-                    balancing_strategy=balancing_strategy,
-                    training_query=training_query,
-                )
-
-                # Train the classifier and output predictions
-                classification.train_test_predict(
-                    input_parcel_train_path=parcel_train_path,
-                    input_parcel_test_path=parcel_test_path,
-                    input_parcel_all_path=parcel_path,
-                    input_parcel_classification_data_path=parcel_classification_data_path,
-                    output_classifier_basepath=classifier_basepath,
-                    output_predictions_test_path=parcel_preds_proba_test_model_path,
-                    output_predictions_all_path=parcel_preds_proba_all_model_path,
-                    predict_query=predict_query,
-                )
-            else:
-                # There exists already a classifier, so just use it!
-                classification.predict(
-                    input_parcel_path=parcel_path,
-                    input_parcel_classification_data_path=parcel_classification_data_path,
-                    input_classifier_basepath=classifier_basepath,
-                    input_classifier_path=input_model_to_use_path,
-                    output_predictions_path=parcel_preds_proba_all_model_path,
-                    predict_query=predict_query,
-                )
-
-        # Merge all prediction to single output file
-        if cross_pred_models > 1:
-            # Merge all test predictions to single output file
-            pred_test_df = None
-            for path in pred_test_files:
-                df = pdh.read_file(path)
-                pred_test_df = pd.concat([pred_test_df, df], ignore_index=True)
-            pdh.to_file(pred_test_df, parcel_predictions_proba_test_path, index=False)
-
-            # Merge all "all" predictions to single output file
-            pred_all_df = None
-            for path in pred_all_files:
-                df = pdh.read_file(path)
-                pred_all_df = pd.concat([pred_all_df, df], ignore_index=True)
-            pdh.to_file(pred_all_df, parcel_predictions_proba_all_path, index=False)
 
     # STEP 5: if necessary, do extra postprocessing
     # -------------------------------------------------------------
@@ -411,14 +289,10 @@ def calc_marker_task(
 
     # STEP 6: do the default, mandatory postprocessing
     # -------------------------------------------------------------
-    # If it was necessary to train, there will be a test prediction... so postprocess it
+    # If there is a test dataset, so postprocess it
     parcel_predictions_test_path = None
     parcel_predictions_test_geopath = None
-    if (
-        input_model_to_use_path is None
-        and parcel_test_path is not None
-        and parcel_predictions_proba_test_path is not None
-    ):
+    if input_model_to_use_path is None and parcel_test_path is not None:
         parcel_predictions_test_path = (
             run_dir / f"{base_filename}_predict_test{data_ext}"
         )
@@ -459,7 +333,7 @@ def calc_marker_task(
             run_dir
             / f"{input_groundtruth_path.stem}_classes{input_groundtruth_path.suffix}"
         )
-        class_pre.prepare_input(
+        prepare_input.prepare(
             input_parcel_path=input_groundtruth_path,
             input_parcel_filetype=input_parcel_filetype,
             timeseries_periodic_dir=timeseries_periodic_dir,
diff --git a/cropclassification/general.ini b/cropclassification/general.ini
index 964071b..36b29ea 100644
--- a/cropclassification/general.ini
+++ b/cropclassification/general.ini
@@ -213,15 +213,14 @@ min_parcels_in_class = 50
 
 # The number of cross-prediction-models to use for this marker.
 #
-# If cross_pred_models is None (the default) or <= 1, , no cross-prediction models are
-# used.
+# If cross_pred_models is 0 (the default), no cross-prediction models are used.
 #
 # The cross-prediction-models feature can be used to avoid predicting a marker on a
 # parcel where the parcel was part of the training dataset.
 # When cross_pred_models = 2, the input data is split in 2 (per class). Each 50% of
 # the data will be used to train one model, and this model will be used to
 # predict the other 50% of the parcels.
-cross_pred_models
+cross_pred_models = 0
 
 # The classifier type to use. Currently supported types:
 #     * histgradientboostingclassifier (sklearn)
@@ -241,6 +240,19 @@ classifier_type = keras_multilayer_perceptron
 #     }
 classifier_sklearn_kwargs
 
+# The fraction of the training data to be used to evaluate the model performance.
+# Float between 0 and 1.
+#
+# Use depends on some factors:
+#   - For keras classifiers, this will be used as validation data: it will not be
+#     trained on, but it will be used to evaluate the loss and any model metrics on this
+#     data at the end of each epoch.
+#   - For sklearn classifiers, this will only be used during reporting.
+#   - If cross_pred_models > 1, and an sklearn classifier is used, this parameter is
+#     ignored as the predictions will happen on 100% unused data anyway, so the model
+#     performance can be evaluated safely on the end result.
+test_size = 0.2
+
 # The extension of the file format to save the trained model to
 classifier_ext = .hdf5
 
diff --git a/cropclassification/predict/classification.py b/cropclassification/predict/classification.py
index 79fc880..e228513 100644
--- a/cropclassification/predict/classification.py
+++ b/cropclassification/predict/classification.py
@@ -9,24 +9,248 @@
 from pathlib import Path
 from typing import Optional
 
+import geofileops as gfo
 import pandas as pd
 
 import cropclassification.helpers.config_helper as conf
 import cropclassification.helpers.pandas_helper as pdh
+from cropclassification.helpers import model_helper as mh
+from cropclassification.predict import data_balancing
 
-# -------------------------------------------------------------
-# First define/init some general variables/constants
-# -------------------------------------------------------------
 # Get a logger...
 logger = logging.getLogger(__name__)
-# logger.setLevel(logging.DEBUG)
 
-# -------------------------------------------------------------
-# The real work
-# -------------------------------------------------------------
+
+def classify(
+    classifier_type: str,
+    parcel_path: Path,
+    parcel_classification_data_path: Path,
+    output_dir: Path,
+    output_base_filename: str,
+    test_size: float,
+    cross_pred_models: int,
+    input_model_to_use_path: Optional[Path],
+    force: bool = False,
+) -> tuple[Path, Path, Optional[Path], Optional[Path]]:
+    """Classify the parcels.
+
+    Args:
+        classifier_type (str): the classifier to use.
+        parcel_path (Path): path to the parcels to classify.
+        parcel_classification_data_path (Path): path to the classification data.
+        output_dir (Path): output directory.
+        output_base_filename (str): base filename for the output files.
+        test_size (float): size of the test set to use. Float between 0 and 1.
+        cross_pred_models (int): the number of models to use for cross prediction.
+            If <= 1, no cross prediction models are used.
+        input_model_to_use_path (Optional[Path]): path to an existing model to use.
+        force (bool, optional): True to force training a new model. Defaults to False.
+
+    Returns:
+        Tuple with the following paths:
+            - output_proba_all_path: file with the predictions for all parcels
+            - output_proba_test_path: file with the predictions for the test parcels
+            - parcel_train_path: file with the training parcels
+            - parcel_test_path: file with the test parcels
+
+    """
+    # Prepare output filenames
+    data_ext = conf.general["data_ext"]
+    classifier_ext = conf.classifier["classifier_ext"]
+    output_proba_all_path = (
+        output_dir / f"{output_base_filename}_predict_proba_all{data_ext}"
+    )
+    output_proba_test_path = (
+        output_dir / f"{output_base_filename}_predict_proba_test{data_ext}"
+    )
+
+    # If the predictions file doesn't exist, do the classification
+    if not force and output_proba_all_path.exists():
+        return (output_proba_all_path, output_proba_test_path, None, None)
+
+    cross_pred_model_id_column = conf.columns["cross_pred_model_id"]
+
+    if cross_pred_models <= 1:
+        # Simplifies code further on to set to 1 when not using cross prediction models
+        cross_pred_models = 1
+    else:
+        # Use "cross models": train multiple models, so prediction of a parcel is
+        # always done with a model where the parcel wasn't used to train it.
+        if input_model_to_use_path is not None:
+            raise ValueError(
+                "cross_pred_models not supported with input_model_to_use_path"
+            )
+
+        # Assign each parcel to a model where it will be predicted with.
+        # To avoid impacting the data balancing, the are divided evenly over the models
+        # based on the class_balancing_column.
+        class_balancing_column = conf.columns["class_balancing"]
+        add_cross_pred_model_id(
+            parcel_path=parcel_path,
+            cross_pred_models=cross_pred_models,
+            columnname=cross_pred_model_id_column,
+            class_balancing_column=class_balancing_column,
+        )
+
+        # No need to split the data in test and train when using cross prediction models
+        # with an sklearn model.
+        if not classifier_type.lower().startswith("keras_"):
+            test_size = 0.0
+
+    pred_all_files = []
+    for cross_pred_model_idx in range(cross_pred_models):
+        if cross_pred_models <= 1:
+            # Only one model, so no need to create subdirectory
+            model_dir = output_dir
+
+            # Output can be the final file immediately
+            parcel_preds_proba_test_model_path = output_proba_test_path
+            parcel_preds_proba_all_model_path = output_proba_all_path
+
+            # No need to filter percels based on cross model indexes
+            training_query = None
+            predict_query = None
+        else:
+            # Create a subfolder for each model
+            model_dir = output_dir / f"cross_pred_model_{cross_pred_model_idx}"
+            model_dir.mkdir(parents=True, exist_ok=True)
+
+            # Temporary output partial files in the subdirectory
+            parcel_preds_proba_test_model_path = model_dir / output_proba_test_path.name
+            parcel_preds_proba_all_model_path = model_dir / output_proba_all_path.name
+            pred_all_files.append(parcel_preds_proba_all_model_path)
+
+            # The training will be done on all parcels with another index
+            training_cross_pred_model_ids = [
+                idx for idx in range(cross_pred_models) if idx != cross_pred_model_idx
+            ]
+            training_query = (
+                f"{cross_pred_model_id_column} in {training_cross_pred_model_ids}"
+            )
+            predict_query = f"{cross_pred_model_id_column} == {cross_pred_model_idx}"
+            # TODO: is there a use of keeping test data seperate if
+            # cross-prediction-models are used?
+
+        # Check if a model exists already
+        if input_model_to_use_path is None:
+            best_model = mh.get_best_model(model_dir, acc_metric_mode="min")
+            if best_model is not None:
+                input_model_to_use_path = best_model["path"]
+
+        # If we cannot reuse an existing model, train one!
+        parcel_train_path = model_dir / f"parcel_train{data_ext}"
+        parcel_test_path = model_dir / f"parcel_test{data_ext}"
+        classifier_basepath = model_dir / f"model{classifier_ext}"
+        if input_model_to_use_path is None:
+            # Create the training sample.
+            # Remark: this creates a list of representative test parcel + a list of
+            # (candidate) training parcels
+            balancing_strategy = conf.marker["balancing_strategy"]
+            data_balancing.create_train_test_sample(
+                input_parcel_path=parcel_path,
+                output_parcel_train_path=parcel_train_path,
+                output_parcel_test_path=parcel_test_path,
+                balancing_strategy=balancing_strategy,
+                test_size=test_size,
+                training_query=training_query,
+                force=force,
+            )
+
+            # Train the classifier and output predictions
+            train_test_predict(
+                classifier_type=classifier_type,
+                input_parcel_train_path=parcel_train_path,
+                input_parcel_test_path=parcel_test_path,
+                input_parcel_all_path=parcel_path,
+                input_parcel_classification_data_path=parcel_classification_data_path,
+                output_classifier_basepath=classifier_basepath,
+                output_predictions_test_path=parcel_preds_proba_test_model_path,
+                output_predictions_all_path=parcel_preds_proba_all_model_path,
+                predict_query=predict_query,
+                force=force,
+            )
+        else:
+            # There exists already a classifier, so just use it!
+            predict(
+                input_parcel_path=parcel_path,
+                input_parcel_classification_data_path=parcel_classification_data_path,
+                input_classifier_basepath=classifier_basepath,
+                input_classifier_path=input_model_to_use_path,
+                output_predictions_path=parcel_preds_proba_all_model_path,
+                predict_query=predict_query,
+                force=force,
+            )
+
+    # Merge all predictions to a single output file
+    if cross_pred_models > 1:
+        # Merge all "all" predictions to single output file.
+        # The "test" predictions are not very useful when cross prediction models
+        # are used, as the "all" predictions are independent of the training.
+        pred_all_df = None
+        for path in pred_all_files:
+            df = pdh.read_file(path)
+            pred_all_df = pd.concat([pred_all_df, df], ignore_index=True)
+        pdh.to_file(pred_all_df, output_proba_all_path, index=False)
+
+        return (output_proba_all_path, output_proba_test_path, None, None)
+
+    return (
+        output_proba_all_path,
+        output_proba_test_path,
+        parcel_train_path,
+        parcel_test_path,
+    )
+
+
+def add_cross_pred_model_id(
+    parcel_path: Path,
+    cross_pred_models: int,
+    columnname: Optional[str] = None,
+    class_balancing_column: Optional[str] = None,
+):
+    """Add a column to the parcel file that assigns each parcel to a model.
+
+    Args:
+        parcel_path (Path): path with parcels to add the column to.
+        cross_pred_models (int): the number of models to divide the parcels over.
+        cross_pred_model_id_column (Optional[str], optional): the column name for the
+            column to add to `parcel_path`. If None, conf.columns["cross_pred_model_id"]
+            is used. Defaults to None.
+        class_balancing_column (Optional[str], optional): the column in the
+            `parcel_path` file to use for balancing the model ids. If None,
+            conf.columns["class_balancing"] is used. Defaults to None.
+    """
+    if columnname is None:
+        columnname = conf.columns["cross_pred_model_id"]
+    if class_balancing_column is None:
+        class_balancing_column = conf.columns["class_balancing"]
+    assert columnname is not None
+    assert class_balancing_column is not None
+
+    layer = gfo.get_only_layer(parcel_path)
+    gfo.add_column(path=parcel_path, layer=layer, name=columnname, type="int")
+    # Remarks:
+    #   - ORDER BY rowid is used to get a deterministic order
+    #   - row_number() starts at 1, so we subtract 1 to get a 0-based number
+    sql = f"""
+            WITH tmp AS
+              (SELECT rowid
+                     ,row_number() OVER
+                        (PARTITION BY "{class_balancing_column}" ORDER BY rowid)
+                      AS rownum
+                 FROM "{layer}"
+              )
+            UPDATE "{layer}"
+               SET "{columnname}" = (
+                     SELECT (rownum -1) % {cross_pred_models} AS model_id FROM tmp
+                      WHERE tmp.rowid = "{layer}".rowid
+                   )
+        """
+    gfo.execute_sql(path=parcel_path, sql_stmt=sql)
 
 
 def train_test_predict(
+    classifier_type: str,
     input_parcel_train_path: Path,
     input_parcel_test_path: Path,
     input_parcel_all_path: Path,
@@ -40,6 +264,7 @@ def train_test_predict(
     """Train a classifier, test it and do full predictions.
 
     Args
+        classifier_type: the type of classifier to use.
         input_parcel_classes_train_path: the list of parcels with classes to train the
             classifier, without data!
         input_parcel_classes_test_path: the list of parcels with classes to test the
@@ -91,6 +316,7 @@ def train_test_predict(
 
     # Train the classification
     output_classifier_path = train(
+        classifier_type=classifier_type,
         input_parcel_train_path=input_parcel_train_path,
         input_parcel_test_path=input_parcel_test_path,
         input_parcel_classification_data_path=input_parcel_classification_data_path,
@@ -100,15 +326,16 @@ def train_test_predict(
     )
 
     # Predict the test parcels
-    predict(
-        input_parcel_path=input_parcel_test_path,
-        input_parcel_classification_data_path=input_parcel_classification_data_path,
-        input_classifier_basepath=output_classifier_basepath,
-        input_classifier_path=output_classifier_path,
-        output_predictions_path=output_predictions_test_path,
-        force=force,
-        input_parcel_classification_data_df=input_parcel_classification_data_df,
-    )
+    if input_parcel_test_path.exists():
+        predict(
+            input_parcel_path=input_parcel_test_path,
+            input_parcel_classification_data_path=input_parcel_classification_data_path,
+            input_classifier_basepath=output_classifier_basepath,
+            input_classifier_path=output_classifier_path,
+            output_predictions_path=output_predictions_test_path,
+            force=force,
+            input_parcel_classification_data_df=input_parcel_classification_data_df,
+        )
 
     # Predict all parcels
     predict(
@@ -124,6 +351,7 @@ def train_test_predict(
 
 
 def train(
+    classifier_type: str,
     input_parcel_train_path: Path,
     input_parcel_test_path: Path,
     input_parcel_classification_data_path: Path,
@@ -170,20 +398,25 @@ def train(
     train_df = train_df.join(input_parcel_classification_data_df, how="inner")
 
     # Read the test/validation data
-    logger.info(f"Read test file: {input_parcel_test_path}")
-    test_df = pdh.read_file(
-        input_parcel_test_path, columns=[conf.columns["id"], conf.columns["class"]]
-    )
-    if test_df.index.name != conf.columns["id"]:
-        test_df.set_index(conf.columns["id"], inplace=True)
-    logger.debug("Read test file ready")
+    test_df = None
+    if input_parcel_test_path.exists():
+        logger.info(f"Read test file: {input_parcel_test_path}")
+        test_df = pdh.read_file(
+            input_parcel_test_path, columns=[conf.columns["id"], conf.columns["class"]]
+        )
+        if test_df.index.name != conf.columns["id"]:
+            test_df.set_index(conf.columns["id"], inplace=True)
+        logger.debug("Read test file ready")
 
-    # Join the columns of input_parcel_classification_data_df that aren't yet in test_df
-    logger.info("Join test sample with the classification data")
-    test_df = test_df.join(input_parcel_classification_data_df, how="inner")
+        # Join the columns of input_parcel_classification_data_df that aren't in test_df
+        logger.info("Join test sample with the classification data")
+        test_df = test_df.join(input_parcel_classification_data_df, how="inner")
 
     # Train
-    if conf.classifier["classifier_type"].lower() == "keras_multilayer_perceptron":
+    if classifier_type.lower() == "keras_multilayer_perceptron":
+        if test_df is None:
+            raise ValueError("test_df is mandatory when using a keras classifier")
+
         import cropclassification.predict.classification_keras as class_core_keras
 
         return class_core_keras.train(
@@ -301,9 +534,3 @@ def predict(
             classifier_path=input_classifier_path,
             output_parcel_predictions_path=output_predictions_path,
         )
-
-
-# If the script is run directly...
-if __name__ == "__main__":
-    logger.critical("Not implemented exception!")
-    raise Exception("Not implemented")
diff --git a/cropclassification/preprocess/classification_preprocess.py b/cropclassification/predict/data_balancing.py
similarity index 79%
rename from cropclassification/preprocess/classification_preprocess.py
rename to cropclassification/predict/data_balancing.py
index e62fc69..2dbfc95 100644
--- a/cropclassification/preprocess/classification_preprocess.py
+++ b/cropclassification/predict/data_balancing.py
@@ -1,133 +1,26 @@
 """
-Module with helper functions to preprocess the data to use for the classification.
+Module with helper functions to prepare the data to use for the classification.
 """
 
 import logging
-import os
-import shutil
 from pathlib import Path
 from typing import Optional
 
-import geofileops as gfo
 import pandas as pd
 
 import cropclassification.helpers.config_helper as conf
 import cropclassification.helpers.pandas_helper as pdh
-import cropclassification.preprocess._classification_preprocess_BEFL as befl
 
 # Get a logger...
 logger = logging.getLogger(__name__)
 
 
-def prepare_input(
-    input_parcel_path: Path,
-    input_parcel_filetype: str,
-    timeseries_periodic_dir: Path,
-    base_filename: str,
-    data_ext: str,
-    classtype_to_prepare: str,
-    classes_refe_path: Path,
-    min_parcels_in_class: int,
-    output_parcel_path: Path,
-    force: bool = False,
-):
-    """
-    Prepare a raw input file by eg. adding the classification classes to use for the
-    classification,...
-    """
-
-    # If force == False Check and the output file exists already, stop.
-    if not force and output_parcel_path.exists():
-        logger.warning(
-            f"prepare_input: output file exists + force is False: {output_parcel_path}"
-        )
-        return
-
-    # If it exists, copy the refe file to the run dir, so we always keep knowing which
-    # refe was used
-    if classes_refe_path is not None:
-        shutil.copy(classes_refe_path, output_parcel_path.parent)
-
-    if input_parcel_filetype == "BEFL":
-        # classes_refe_path must exist for BEFL!
-        if not classes_refe_path.exists():
-            raise Exception(f"Input classes file doesn't exist: {classes_refe_path}")
-
-        df_parceldata = befl.prepare_input(
-            input_parcel_path=input_parcel_path,
-            classtype_to_prepare=classtype_to_prepare,
-            classes_refe_path=classes_refe_path,
-            min_parcels_in_class=min_parcels_in_class,
-            output_dir=output_parcel_path.parent,
-        )
-    else:
-        message = f"Unknown value for input_parcel_filetype: {input_parcel_filetype}"
-        logger.critical(message)
-        raise Exception(message)
-
-    # Load pixcount data and join it
-    parcel_pixcount_path = (
-        timeseries_periodic_dir / f"{base_filename}_pixcount{data_ext}"
-    )
-    if parcel_pixcount_path.exists():
-        logger.info(f"Read pixcount file {parcel_pixcount_path}")
-        df_pixcount = pdh.read_file(parcel_pixcount_path)
-        logger.debug(f"Read pixcount file ready, shape: {df_pixcount.shape}")
-    else:
-        # Find the s1 image having the largest percentage of pixcount >= 1, and use that
-        logger.info("Determine pixel count based on s1 timeseries data")
-        pixcount_path = None
-        nb_pixcount_ok_max = -1
-        sql_stmt = """
-            SELECT COUNT(*) AS aantal_pixcount_gte1 FROM info WHERE info.count >= 1
-        """
-        for path in timeseries_periodic_dir.glob("*s1*.sqlite"):
-            nb_pixcount_ok = gfo.read_file(path, sql_stmt=sql_stmt)[
-                "aantal_pixcount_gte1"
-            ][0].item()
-            if nb_pixcount_ok > nb_pixcount_ok_max:
-                nb_pixcount_ok_max = nb_pixcount_ok
-                pixcount_path = path
-        if pixcount_path is None:
-            raise RuntimeError(
-                f"No valid timeseries found to get pixel count for {input_parcel_path}"
-            )
-
-        df_pixcount = pdh.read_file(pixcount_path)
-        df_pixcount = df_pixcount[[conf.columns["id"], "count"]].rename(
-            columns={"count": conf.columns["pixcount_s1s2"]}
-        )
-        pdh.to_file(df_pixcount, parcel_pixcount_path)
-
-    if df_pixcount.index.name != conf.columns["id"]:
-        df_pixcount.set_index(conf.columns["id"], inplace=True)
-
-    df_parceldata.set_index(conf.columns["id"], inplace=True)
-    df_parceldata = df_parceldata.join(
-        df_pixcount[conf.columns["pixcount_s1s2"]], how="left"
-    )
-    df_parceldata.fillna({conf.columns["pixcount_s1s2"]: 0}, inplace=True)
-
-    # Export result to file
-    output_ext = os.path.splitext(output_parcel_path)[1]
-    for column in df_parceldata.columns:
-        # if the output asked is a csv... we don't need the geometry...
-        if column == conf.columns["geom"] and output_ext == ".csv":
-            df_parceldata.drop(column, axis=1, inplace=True)
-
-    logger.info(f"Write output to {output_parcel_path}")
-    # If extension is not .shp, write using pandas (=a lot faster!)
-    if output_ext.lower() != ".shp":
-        pdh.to_file(df_parceldata, output_parcel_path)
-    else:
-        df_parceldata.to_file(output_parcel_path, index=False)
-
-
 def create_train_test_sample(
     input_parcel_path: Path,
     output_parcel_train_path: Path,
     output_parcel_test_path: Path,
     balancing_strategy: str,
+    test_size: float,
     training_query: Optional[str] = None,
     force: bool = False,
 ):
@@ -176,22 +69,28 @@ def create_train_test_sample(
     #     groups above the data and evades having to do
     #     .reset_index(level=class_balancing_column_NAME, drop=True)
     #     to get rid of the group level
-    test_df = (
-        df_in.groupby(class_balancing_column)
-        .apply(pd.DataFrame.sample, frac=0.20, include_groups=False)
-        .reset_index()
-    )
-    if not test_df.empty:
-        test_df.set_index("level_1", inplace=True)
-        test_df.index.name = None
+    if test_size > 0.0:
+        test_df = (
+            df_in.groupby(class_balancing_column)
+            .apply(pd.DataFrame.sample, frac=test_size, include_groups=False)
+            .reset_index()
+        )
+        if not test_df.empty:
+            test_df.set_index("level_1", inplace=True)
+            test_df.index.name = None
 
-    logger.debug(
-        f"df_test after sampling 20% of data per class, shape: {test_df.shape}"
-    )
+        logger.debug(
+            f"df_test after sampling a fraction of {test_size} of the data per class, "
+            f"shape: {test_df.shape}"
+        )
+
+        # The candidate parcel for training are all non-test parcel
+        train_base_df = df_in[~df_in.index.isin(test_df.index)]
+    else:
+        train_base_df = df_in
+        test_df = None
 
-    # The candidate parcel for training are all non-test parcel
-    train_base_df = df_in[~df_in.index.isin(test_df.index)]
-    logger.debug(f"df_train_base after isin\n{train_base_df}")
+    logger.debug("df_train_base after splitting off test data\n%s", train_base_df)
 
     # Remove parcels with too few pixels from the train sample
     min_pixcount = conf.marker.getfloat("min_nb_pixels_train")
@@ -656,22 +555,27 @@ def create_train_test_sample(
             )
 
     # Log the resulting numbers per class in the test sample
-    with pd.option_context("display.max_rows", None, "display.max_columns", None):
-        count_per_class = test_df.groupby(class_balancing_column, as_index=False).size()
-        logger.info(
-            "Number of elements per class_balancing_column in test dataset:\n"
-            f"{count_per_class}"
-        )
-        if class_balancing_column != class_column:
-            count_per_class = test_df.groupby(class_column, as_index=False).size()
+    if test_df is not None:
+        with pd.option_context("display.max_rows", None, "display.max_columns", None):
+            count_per_class = test_df.groupby(
+                class_balancing_column, as_index=False
+            ).size()
             logger.info(
-                "Number of elements per class_column in test dataset:\n"
+                "Number of elements per class_balancing_column in test dataset:\n"
                 f"{count_per_class}"
             )
+            if class_balancing_column != class_column:
+                count_per_class = test_df.groupby(class_column, as_index=False).size()
+                logger.info(
+                    "Number of elements per class_column in test dataset:\n"
+                    f"{count_per_class}"
+                )
 
     # Write to output files
     logger.info("Write the output files")
     train_df.set_index(conf.columns["id"], inplace=True)
-    test_df.set_index(conf.columns["id"], inplace=True)
     pdh.to_file(train_df, output_parcel_train_path)  # The ID column is the index...
-    pdh.to_file(test_df, output_parcel_test_path)  # The ID column is the index...
+
+    if test_df is not None:
+        test_df.set_index(conf.columns["id"], inplace=True)
+        pdh.to_file(test_df, output_parcel_test_path)  # The ID column is the index...
diff --git a/cropclassification/preprocess/_classification_preprocess_BEFL.py b/cropclassification/preprocess/_prepare_input_BEFL.py
similarity index 100%
rename from cropclassification/preprocess/_classification_preprocess_BEFL.py
rename to cropclassification/preprocess/_prepare_input_BEFL.py
diff --git a/cropclassification/preprocess/prepare_input.py b/cropclassification/preprocess/prepare_input.py
new file mode 100644
index 0000000..548963c
--- /dev/null
+++ b/cropclassification/preprocess/prepare_input.py
@@ -0,0 +1,121 @@
+"""
+Module with helper functions to preprocess the data to use for the classification.
+"""
+
+import logging
+import os
+import shutil
+from pathlib import Path
+
+import geofileops as gfo
+
+import cropclassification.helpers.config_helper as conf
+import cropclassification.helpers.pandas_helper as pdh
+import cropclassification.preprocess._prepare_input_BEFL as befl
+
+# Get a logger...
+logger = logging.getLogger(__name__)
+
+
+def prepare(
+    input_parcel_path: Path,
+    input_parcel_filetype: str,
+    timeseries_periodic_dir: Path,
+    base_filename: str,
+    data_ext: str,
+    classtype_to_prepare: str,
+    classes_refe_path: Path,
+    min_parcels_in_class: int,
+    output_parcel_path: Path,
+    force: bool = False,
+):
+    """
+    Prepare a raw input file by eg. adding the classification classes to use for the
+    classification,...
+    """
+
+    # If force == False Check and the output file exists already, stop.
+    if not force and output_parcel_path.exists():
+        logger.warning(
+            f"prepare_input: output file exists + force is False: {output_parcel_path}"
+        )
+        return
+
+    # If it exists, copy the refe file to the run dir, so we always keep knowing which
+    # refe was used
+    if classes_refe_path is not None:
+        shutil.copy(classes_refe_path, output_parcel_path.parent)
+
+    if input_parcel_filetype == "BEFL":
+        # classes_refe_path must exist for BEFL!
+        if not classes_refe_path.exists():
+            raise Exception(f"Input classes file doesn't exist: {classes_refe_path}")
+
+        df_parceldata = befl.prepare_input(
+            input_parcel_path=input_parcel_path,
+            classtype_to_prepare=classtype_to_prepare,
+            classes_refe_path=classes_refe_path,
+            min_parcels_in_class=min_parcels_in_class,
+            output_dir=output_parcel_path.parent,
+        )
+    else:
+        message = f"Unknown value for input_parcel_filetype: {input_parcel_filetype}"
+        logger.critical(message)
+        raise Exception(message)
+
+    # Load pixcount data and join it
+    parcel_pixcount_path = (
+        timeseries_periodic_dir / f"{base_filename}_pixcount{data_ext}"
+    )
+    if parcel_pixcount_path.exists():
+        logger.info(f"Read pixcount file {parcel_pixcount_path}")
+        df_pixcount = pdh.read_file(parcel_pixcount_path)
+        logger.debug(f"Read pixcount file ready, shape: {df_pixcount.shape}")
+    else:
+        # Find the s1 image having the largest percentage of pixcount >= 1, and use that
+        logger.info("Determine pixel count based on s1 timeseries data")
+        pixcount_path = None
+        nb_pixcount_ok_max = -1
+        sql_stmt = """
+            SELECT COUNT(*) AS aantal_pixcount_gte1 FROM info WHERE info.count >= 1
+        """
+        for path in timeseries_periodic_dir.glob("*s1*.sqlite"):
+            nb_pixcount_ok = gfo.read_file(path, sql_stmt=sql_stmt)[
+                "aantal_pixcount_gte1"
+            ][0].item()
+            if nb_pixcount_ok > nb_pixcount_ok_max:
+                nb_pixcount_ok_max = nb_pixcount_ok
+                pixcount_path = path
+        if pixcount_path is None:
+            raise RuntimeError(
+                f"No valid timeseries found to get pixel count for {input_parcel_path}"
+            )
+
+        df_pixcount = pdh.read_file(pixcount_path)
+        df_pixcount = df_pixcount[[conf.columns["id"], "count"]].rename(
+            columns={"count": conf.columns["pixcount_s1s2"]}
+        )
+        pdh.to_file(df_pixcount, parcel_pixcount_path)
+
+    if df_pixcount.index.name != conf.columns["id"]:
+        df_pixcount.set_index(conf.columns["id"], inplace=True)
+
+    df_parceldata.set_index(conf.columns["id"], inplace=True)
+    df_parceldata = df_parceldata.join(
+        df_pixcount[conf.columns["pixcount_s1s2"]], how="left"
+    )
+    df_parceldata.fillna({conf.columns["pixcount_s1s2"]: 0}, inplace=True)
+
+    # Export result to file
+    output_ext = os.path.splitext(output_parcel_path)[1]
+    for column in df_parceldata.columns:
+        # if the output asked is a csv... we don't need the geometry...
+        if column == conf.columns["geom"] and output_ext == ".csv":
+            df_parceldata.drop(column, axis=1, inplace=True)
+
+    logger.info(f"Write output to {output_parcel_path}")
+    # If extension is not .shp, write using pandas (=a lot faster!)
+    if output_ext.lower() != ".shp":
+        pdh.to_file(df_parceldata, output_parcel_path)
+    else:
+        df_parceldata.to_file(output_parcel_path, index=False)
diff --git a/sample_marker_basedir/_tasks/local_overrule.ini b/sample_marker_basedir/_tasks/local_overrule.ini
index b202eaf..8d4753d 100644
--- a/sample_marker_basedir/_tasks/local_overrule.ini
+++ b/sample_marker_basedir/_tasks/local_overrule.ini
@@ -1,12 +1,5 @@
 # Configuration file with base local overrules for reuse in other configuration files.
 
-# Configuration of marker parameters regarding the preprocessing of input files.
-[preprocess]
-
-# Minimum amount of parcels that need to be in a specific class or it will get moved
-# into IGNORE_NOT_ENOUGH_SAMPLES.
-min_parcels_in_class = 1
-
 # Configuration regarding the region of interest (roi) we want to use.
 [roi]
 


Test patch:
diff --git a/tests/test_classification.py b/tests/test_classification.py
new file mode 100644
index 0000000..7449e91
--- /dev/null
+++ b/tests/test_classification.py
@@ -0,0 +1,26 @@
+import shutil
+
+import geofileops as gfo
+
+from cropclassification.predict import classification
+from tests import test_helper
+
+
+def test_add_cross_pred_model_id(tmp_path):
+    parcel_path = test_helper.SampleData.inputdata_dir / "Prc_BEFL_2023_2023-07-24.gpkg"
+    test_path = tmp_path / parcel_path.name
+    shutil.copy(parcel_path, test_path)
+    columnname = "test_column"
+    cross_pred_models = 4
+
+    classification.add_cross_pred_model_id(
+        test_path,
+        cross_pred_models=cross_pred_models,
+        columnname=columnname,
+        class_balancing_column="GWSCOD_H",
+    )
+
+    result_gdf = gfo.read_file(test_path)
+    assert columnname in result_gdf.columns
+    assert result_gdf[columnname].max() == cross_pred_models - 1
+    assert result_gdf[columnname].min() == 0
diff --git a/tests/test_end2end.py b/tests/test_end2end.py
index e4e0b6a..d6ea5b9 100644
--- a/tests/test_end2end.py
+++ b/tests/test_end2end.py
@@ -11,18 +11,17 @@
 
 
 @pytest.mark.parametrize(
-    "task, balancing_strategy",
+    "balancing_strategy, cross_pred_models",
     [
-        ("calc_periodic_mosaic", None),
-        ("calc_marker", "BALANCING_STRATEGY_MEDIUM"),
-        ("calc_marker", "BALANCING_STRATEGY_MEDIUM2"),
-        ("calc_marker", "BALANCING_STRATEGY_PROPORTIONAL_GROUPS"),
-        ("calc_marker", "BALANCING_STRATEGY_UPPER_LIMIT"),
-        ("calc_marker", "BALANCING_STRATEGY_EQUAL"),
+        ("BALANCING_STRATEGY_MEDIUM", 0),
+        ("BALANCING_STRATEGY_MEDIUM2", 0),
+        ("BALANCING_STRATEGY_PROPORTIONAL_GROUPS", 2),
+        ("BALANCING_STRATEGY_UPPER_LIMIT", 0),
+        ("BALANCING_STRATEGY_EQUAL", 0),
     ],
 )
-def test_end2end_task(tmp_path, task, balancing_strategy):
-    if task == "calc_marker" and not HAS_QGIS:
+def test_task_calc_marker(tmp_path, balancing_strategy, cross_pred_models):
+    if not HAS_QGIS:
         pytest.skip("QGIS is not available on this system.")
 
     marker_basedir = tmp_path / test_helper.SampleData.marker_basedir.name
@@ -31,7 +30,7 @@ def test_end2end_task(tmp_path, task, balancing_strategy):
     # Create configparser and read task file!
     tasks_dir = marker_basedir / "_tasks"
     ignore_dir = tasks_dir / "ignore"
-    task_ini = f"task_test_{task}.ini"
+    task_ini = "task_test_calc_marker.ini"
 
     shutil.copy(src=ignore_dir / task_ini, dst=tasks_dir / task_ini)
 
@@ -39,55 +38,51 @@ def test_end2end_task(tmp_path, task, balancing_strategy):
         tasksdir=tasks_dir,
         config_overrules=[
             f"marker.balancing_strategy={balancing_strategy}",
+            f"classifier.cross_pred_models={cross_pred_models}",
+            "preprocess.min_parcels_in_class=1",
         ],
     )
 
-    if task == "calc_marker":
-        run_dir = (
-            marker_basedir
-            / "2024_CROPGROUP"
-            / f"Run_{datetime.now().strftime('%Y-%m-%d')}_001"
-        )
-        assert run_dir.exists()
-        assert (run_dir / "CROPGROUP_01_mlp.hdf5").exists()
-        assert (
-            run_dir / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_all.gpkg"
-        ).exists()
-        assert (
-            run_dir
-            / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_all.sqlite_accuracy_report.html"  # noqa: E501
-        ).exists()
-        assert (
-            run_dir
-            / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_test.sqlite_accuracy_report.txt_groundtruth_pred_quality_details.gpkg"  # noqa: E501
-        )
-        df_predict = gfo.read_file(
-            path=run_dir
-            / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_test.sqlite_accuracy_report.txt_groundtruth_pred_quality_details.gpkg"  # noqa: E501
-        )
-        assert is_numeric_dtype(df_predict["LAYER_ID"].dtype)
-        assert is_numeric_dtype(df_predict["PRC_ID"].dtype)
-        assert is_numeric_dtype(df_predict["pixcount"].dtype)
-        assert is_numeric_dtype(df_predict["pred1_prob"].dtype)
-        assert is_numeric_dtype(df_predict["pred2_prob"].dtype)
-        assert is_numeric_dtype(df_predict["pred3_prob"].dtype)
-        assert (
-            run_dir
-            / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_all.sqlite_accuracy_report.txt_groundtruth_pred_quality_details.gpkg"  # noqa: E501
-        )
-        df_predict = gfo.read_file(
-            path=run_dir
-            / "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_all.sqlite_accuracy_report.txt_groundtruth_pred_quality_details.gpkg"  # noqa: E501
-        )
-        assert is_numeric_dtype(df_predict["LAYER_ID"].dtype)
-        assert is_numeric_dtype(df_predict["PRC_ID"].dtype)
-        assert is_numeric_dtype(df_predict["pixcount"].dtype)
-        assert is_numeric_dtype(df_predict["pred1_prob"].dtype)
-        assert is_numeric_dtype(df_predict["pred2_prob"].dtype)
-        assert is_numeric_dtype(df_predict["pred3_prob"].dtype)
-
-    if task == "calc_periodic_mosaic":
-        # Check if a log file was written
-        log_dir = marker_basedir / "log"
-        assert log_dir.exists()
-        assert len(list(log_dir.glob("*.log"))) == 1
+    today_str = datetime.now().strftime("%Y-%m-%d")
+    run_dir = marker_basedir / f"2024_CROPGROUP/Run_{today_str}_001"
+    assert run_dir.exists()
+    base_stem = "Prc_BEFL_2023_2023-07-24_bufm5_weekly_predict_all"
+    assert (run_dir / f"{base_stem}.gpkg").exists()
+    assert (run_dir / f"{base_stem}.sqlite_accuracy_report.html").exists()
+    details_gpgk_path = (
+        run_dir
+        / f"{base_stem}.sqlite_accuracy_report.txt_groundtruth_pred_quality_details.gpkg"  # noqa: E501
+    )
+    assert details_gpgk_path.exists()
+
+    df_predict = gfo.read_file(path=details_gpgk_path)
+    assert is_numeric_dtype(df_predict["LAYER_ID"].dtype)
+    assert is_numeric_dtype(df_predict["PRC_ID"].dtype)
+    assert is_numeric_dtype(df_predict["pixcount"].dtype)
+    assert is_numeric_dtype(df_predict["pred1_prob"].dtype)
+    assert is_numeric_dtype(df_predict["pred2_prob"].dtype)
+    assert is_numeric_dtype(df_predict["pred3_prob"].dtype)
+
+    if cross_pred_models <= 1:
+        assert (run_dir / "model.hdf5").exists()
+    else:
+        for model_id in range(cross_pred_models):
+            assert (run_dir / f"cross_pred_model_{model_id}").exists()
+
+
+def test_task_calc_periodic_mosaic(tmp_path):
+    marker_basedir = tmp_path / test_helper.SampleData.marker_basedir.name
+    shutil.copytree(test_helper.SampleData.marker_basedir, marker_basedir)
+    # Create configparser and read task file!
+    tasks_dir = marker_basedir / "_tasks"
+    ignore_dir = tasks_dir / "ignore"
+    task_ini = "task_test_calc_periodic_mosaic.ini"
+
+    shutil.copy(src=ignore_dir / task_ini, dst=tasks_dir / task_ini)
+
+    cropclassification.cropclassification(tasksdir=tasks_dir)
+
+    # Check if a log file was written
+    log_dir = marker_basedir / "log"
+    assert log_dir.exists()
+    assert len(list(log_dir.glob("*.log"))) == 1