Repository: leferrad/pyoselm (language: python)

Issue:
Add functional tests
This library needs testing regarding behavior to ensure the models are working fine in some given scenarios. A BDD testing approach is suggested to cover that with some functional tests.


Discussion / hints:

Fix patch:
diff --git a/.travis.yml b/.travis.yml
index 322c42f..998bb33 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,10 +1,10 @@
 language: python
 
 python:
-  - "3.5"
   - "3.6"
   - "3.7"
   - "3.8"
+  - "3.9"
 
 notifications:
   email:
@@ -22,7 +22,7 @@ install:
 
 script:
   # run tests
-  - pytest --cov=pyoselm --cov-report=term --cov-report=html:tests/coverage tests/
+  - pytest --cov=pyoselm --cov-report=term --cov-report=xml --cov-report=html:tests/coverage -m "not expensive" tests/unit
 
 after_success:
   - codecov # submit coverage
diff --git a/docs/source/setup.rst b/docs/source/setup.rst
index d1c1e9b..cf33b83 100644
--- a/docs/source/setup.rst
+++ b/docs/source/setup.rst
@@ -1,7 +1,7 @@
 Setup
 =====
 
-``pyoselm`` supports Python 3.5+.
+``pyoselm`` supports Python 3.6+.
 
 Installing
 ----------
diff --git a/setup.py b/setup.py
index ac63345..ef08eb4 100644
--- a/setup.py
+++ b/setup.py
@@ -19,6 +19,7 @@
             'pytest==6.0.1',
             'pytest-pep8==1.0.6',
             'pytest-cov==2.10.1',
+            'pytest-bdd==3.4.0',
         ],
         'docs': [
             'sphinx==3.2.1',


Test patch:
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000..1dabb52
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,54 @@
+# Testing
+
+This folder contains all the tests to be handled for this library. There are two types of tests supported:
+- **Unit testing:** To validate each module or block of code during development, normally done by the programmer who writes the code. Here they are implemented as simple functions executed by `pytest`.
+- **System testing:** To validate the complete and fully integrated software, normally done by a tester on the completed software before it is released. Here we focus on *functional* tests, so they are written in Gherkin language and implemented with functions executed with `pytest-bdd`.
+
+## Project structure
+                            
+      │
+      │ ...   
+      │
+      ├── tests                        <- Folder for tests
+      │    ├── system                  <- Core folder for System testing                         
+      │    │	├── features           <- Contains .feature files where scenarios are written in Gherkin language
+      │    │    │   `-- *.feature
+      │    │	├── step_defs          <- Contains the definitions of the steps in Python methods     
+      │    │    │   |-- __init__.py
+      │    │    │   |-- utils          <- Package with utils for steps in tests    
+      │    │    │   `-- conftest.py    <- Steps that are shared between scenarios                          
+      │    │    │   `-- test_*.py      <- Steps for tests                                
+      │    │	├── pytest.ini         <- Configuration file (e.g. to register markers) 
+      │    │
+      │    ├── unit                    <- Core folder for Unit testing                         
+      │         `-- test_*.py│         <- Scripts for tests
+      │
+      │── README.md                    <- Documentation for testing
+
+## Installation
+
+Install the dependencies for `tests` through:
+
+    pip install -e .[tests] 
+
+## Execution
+                                     
+- To execute all the tests: 
+
+    `pytest tests/`
+    
+- To execute only unit tests
+
+    e.g: `pytest tests/unit`
+    
+- To execute only system tests
+
+    e.g: `pytest tests/system`
+
+
+- To execute system tests with specific markets: "pytest -m <tag> tests/system/" where tag is a tag added in the test case or .feature file
+
+    e.g.: `pytest -m elm tests/system/`
+
+
+
diff --git a/tests/system/features/elm.feature b/tests/system/features/elm.feature
new file mode 100644
index 0000000..2af7848
--- /dev/null
+++ b/tests/system/features/elm.feature
@@ -0,0 +1,43 @@
+
+Feature: Fit and validate an ELMRegressor model with input data
+  As a data scientist,
+  I want to fit and validate an ELMRegressor with input data
+  in order to get a regression model with proper results
+
+  @regression @elm
+  Scenario: Fit an ELMRegressor with toy data (easy)
+    Given the dataset 'boston'
+    And a pre-processing pipeline
+    And an ELMRegressor model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets 
+    And the results are very good
+
+  @regression @elm
+  Scenario: Fit an ELMRegressor with real data (hard)
+    Given the dataset 'california'
+    And a pre-processing pipeline
+    And an ELMRegressor model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets 
+    And the results are good enough
+
+  @classification @elm
+  Scenario: Fit an ELMClassifier with toy data (easy)
+    Given the dataset 'iris'
+    And a pre-processing pipeline
+    And an ELMClassifier model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets
+    And the results are very good
+
+  @classification @elm
+  Scenario: Fit an ELMClassifier with real data (hard)
+    Given the dataset 'covertype'
+    And a pre-processing pipeline
+    And an ELMClassifier model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets
+    And the results are good enough
+
+
diff --git a/tests/system/features/oselm.feature b/tests/system/features/oselm.feature
new file mode 100644
index 0000000..f4286ad
--- /dev/null
+++ b/tests/system/features/oselm.feature
@@ -0,0 +1,61 @@
+
+Feature: Fit and validate an OSELMRegressor model with input data
+  As a data scientist,
+  I want to fit and validate an OSELMRegressor with input data
+  in order to get a regression model with proper results
+
+  @regression @oselm
+  Scenario: Fit an OSELMRegressor with toy data (easy)
+    Given the dataset 'boston'
+    And a pre-processing pipeline
+    And an OSELMRegressor model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets 
+    And the results are very good
+
+  @regression @oselm
+  Scenario: Fit an OSELMRegressor with real data (hard)
+    Given the dataset 'california'
+    And a pre-processing pipeline
+    And an OSELMRegressor model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets 
+    And the results are good enough
+
+  @classification @oselm
+  Scenario: Fit an OSELMClassifier with toy data (easy)
+    Given the dataset 'iris'
+    And a pre-processing pipeline
+    And an OSELMClassifier model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets 
+    And the results are very good
+
+  @classification @oselm @expensive
+  Scenario: Fit an OSELMClassifier with real data (hard)
+    Given the dataset 'covertype'
+    And a pre-processing pipeline
+    And an OSELMClassifier model
+    When I fit the pipeline and the model
+    Then I compute the score in train and test sets
+    And the results are good enough
+
+  @regression @oselm @expensive
+  Scenario: Fit an OSELMRegressor with toy data in online fashion, row by row
+    Given the dataset 'boston'
+    And a pre-processing pipeline
+    And an OSELMRegressor model
+    When I fit the pipeline and the model in online fashion, row by row,
+    Then I compute the score in train and test sets
+    And the results are good enough
+
+  @regression @oselm  @expensive
+  Scenario: Fit an OSELMRegressor with toy data in online fashion, chunk by chunk
+    Given the dataset 'boston'
+    And a pre-processing pipeline
+    And an OSELMRegressor model
+    When I fit the pipeline and the model in online fashion, chunk by chunk,
+    Then I compute the score in train and test sets
+    And the results are good enough
+
+
diff --git a/tests/system/pytest.ini b/tests/system/pytest.ini
new file mode 100644
index 0000000..ee48fbd
--- /dev/null
+++ b/tests/system/pytest.ini
@@ -0,0 +1,6 @@
+[pytest]
+markers =
+    elm: Tests for ELM algorithm
+    oselm: Tests for OS-ELM algorithm
+    classification: Tests for classification models
+    regression: Tests for regression models
diff --git a/tests/system/step_defs/__init__.py b/tests/system/step_defs/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/system/step_defs/conftest.py b/tests/system/step_defs/conftest.py
new file mode 100644
index 0000000..847f718
--- /dev/null
+++ b/tests/system/step_defs/conftest.py
@@ -0,0 +1,54 @@
+"""Steps that are shared between scenarios"""
+
+import pytest
+from pytest_bdd import given, when, then, parsers
+
+from .utils.loaders import load_dataset, load_pipeline, load_elm_model, load_oselm_model, Results
+
+
+@given(parsers.parse("the dataset '{key}'"))
+def dataset(key):
+    yield load_dataset(key)
+
+
+@given("a pre-processing pipeline")
+def pipeline(dataset):
+    key = dataset.name
+    yield load_pipeline(key)
+
+
+@given(parsers.parse("an {model_type} model"))
+def model(model_type, dataset):
+    key = dataset.name
+    loader = load_elm_model if model_type.startswith("ELM") else load_oselm_model
+    yield loader(key)
+
+
+@pytest.fixture
+@when("I fit the pipeline and the model")
+def pipeline_model(dataset, pipeline, model):
+    pipeline.steps.append(("model", model))
+    pipeline.fit(dataset.X_train, dataset.y_train)
+    yield pipeline
+
+
+@pytest.fixture
+@then("I compute the score in train and test sets")
+def score_model(pipeline_model, dataset):
+    score_train = pipeline_model.score(dataset.X_train, dataset.y_train)
+    score_test = pipeline_model.score(dataset.X_test, dataset.y_test)
+    yield Results(score_train, score_test)
+
+
+@then("the results are very good")
+def results_really_good(score_model):
+    # Thresholds valid for r1 and accuracy score
+    assert score_model.train > 0.7
+    assert score_model.test > 0.7
+
+
+@then("the results are good enough")
+def results_good_enough(score_model):
+    # Thresholds valid for r1 and accuracy score
+    assert score_model.train > 0.5
+    assert score_model.test > 0.5
diff --git a/tests/system/step_defs/test_elm_steps.py b/tests/system/step_defs/test_elm_steps.py
new file mode 100644
index 0000000..9d0ece3
--- /dev/null
+++ b/tests/system/step_defs/test_elm_steps.py
@@ -0,0 +1,6 @@
+"""Steps for elm.feature"""
+
+from pytest_bdd import scenarios
+
+
+scenarios("../features/elm.feature")
diff --git a/tests/system/step_defs/test_oselm_steps.py b/tests/system/step_defs/test_oselm_steps.py
new file mode 100644
index 0000000..03e6358
--- /dev/null
+++ b/tests/system/step_defs/test_oselm_steps.py
@@ -0,0 +1,43 @@
+"""Steps for oselm.feature"""
+
+import pytest
+from pytest_bdd import scenarios, when
+
+scenarios("../features/oselm.feature")
+
+
+def pipeline_model_online_chunk(dataset, pipeline, model, chunk_size):
+    # First fit the pipeline
+    pipeline.fit(dataset.X_train, dataset.y_train)
+
+    # Get data
+    X_train = pipeline.transform(dataset.X_train)
+    y_train = dataset.y_train
+
+    # Model should be fitted with a batch of at least model.n_hidden rows the first time
+    n = model.n_hidden
+    model.fit(X_train[:n], y_train[:n])
+    N = len(y_train)
+
+    # Now fit the model chunk-by-chunk
+    for i in range(n, N, chunk_size):
+        X = X_train[i:i+chunk_size]
+        y = y_train[i:i+chunk_size]
+        model.fit(X, y)
+
+    # Add the model to the pipeline
+    pipeline.steps.append(("model", model))
+
+    return pipeline
+
+
+@pytest.fixture
+@when("I fit the pipeline and the model in online fashion, chunk by chunk,")
+def pipeline_model(dataset, pipeline, model):
+    yield pipeline_model_online_chunk(dataset, pipeline, model, chunk_size=100)
+
+
+@pytest.fixture
+@when("I fit the pipeline and the model in online fashion, row by row,")
+def pipeline_model(dataset, pipeline, model):
+    yield pipeline_model_online_chunk(dataset, pipeline, model, chunk_size=1)
diff --git a/tests/system/step_defs/utils/__init__.py b/tests/system/step_defs/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/system/step_defs/utils/loaders.py b/tests/system/step_defs/utils/loaders.py
new file mode 100644
index 0000000..d0f1d7e
--- /dev/null
+++ b/tests/system/step_defs/utils/loaders.py
@@ -0,0 +1,105 @@
+"""Module to share utils for the tests regarding loaders of data and models"""
+
+from sklearn import datasets
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+
+from pyoselm import ELMRegressor, OSELMRegressor, ELMClassifier, OSELMClassifier
+
+# Datasets to use for tests
+AVAILABLE_DATASETS = {
+    "boston": datasets.load_boston,  # regression
+    "california": datasets.fetch_california_housing,  # regression
+    "iris": datasets.load_iris,  # classification
+    "covertype": datasets.fetch_covtype,  # classification
+}
+
+# Pipelines to use for tests
+AVAILABLE_PIPELINES = {
+    "boston": lambda: Pipeline([("scaler", StandardScaler())]),
+    "california": lambda: Pipeline([("scaler", StandardScaler())]),
+    "iris": lambda: Pipeline([("scaler", StandardScaler())]), 
+    "covertype": lambda: Pipeline([("scaler", StandardScaler())]),
+}
+
+# Models from pyoselm.elm to use for tests
+AVAILABLE_ELM_MODELS = {
+    "boston": lambda: ELMRegressor(n_hidden=50, activation_func="sigmoid", random_state=123),
+    "california": lambda: ELMRegressor(n_hidden=50, activation_func="sigmoid", random_state=123),
+    "iris": lambda: ELMClassifier(n_hidden=20, activation_func="sigmoid", random_state=123),
+    "covertype": lambda: ELMClassifier(n_hidden=50, activation_func="sigmoid", random_state=123),
+}
+
+# Models from pyoselm.oselm to use for tests
+AVAILABLE_OSELM_MODELS = {
+    "boston": lambda: OSELMRegressor(n_hidden=70, activation_func="sigmoid", random_state=123),
+    "california": lambda: OSELMRegressor(n_hidden=50, activation_func="sigmoid", random_state=123),
+    "iris": lambda: OSELMClassifier(n_hidden=20, activation_func="sigmoid", random_state=123),
+    "covertype": lambda: OSELMClassifier(n_hidden=50, activation_func="sigmoid", random_state=123),
+}
+
+
+class Dataset:
+    """
+    Util to store a dataset for tests. 
+    It splits data in train and test sets for cross validation.
+
+    Args:
+        name (string): name of dataset to load
+        X (np.array): features
+        y (np.array): target
+        test_size (float, optional): Ratio for test set. Defaults to 0.3.
+    """
+    def __init__(self, name, X, y, test_size=0.3):
+        self.name = name
+        X_train, X_test, y_train, y_test = train_test_split(X, y,
+                                                            test_size=test_size,
+                                                            random_state=123)
+        self.X_train = X_train
+        self.X_test = X_test
+        self.y_train = y_train
+        self.y_test = y_test
+
+        self.y = y
+
+
+class Results:
+    def __init__(self, train, test):
+        self.train = train
+        self.test = test
+
+
+def load_dataset(key):
+    """Load a dataset based on key"""
+    if key not in AVAILABLE_DATASETS:
+        raise ValueError(f"Not supported dataset for key '{key}'")
+
+    loader = AVAILABLE_DATASETS[key]
+    X, y = loader(return_X_y=True)
+
+    return Dataset(key, X, y)
+
+
+def load_pipeline(key):
+    """Load a pipeline based on key"""
+    if key not in AVAILABLE_PIPELINES:
+        raise ValueError(f"Not supported pipeline for key '{key}'")
+
+    return AVAILABLE_PIPELINES[key]()
+
+
+def load_elm_model(key):
+    """Load an ELM model based on key"""
+    if key not in AVAILABLE_ELM_MODELS:
+        raise ValueError(f"Not supported ELM model for key '{key}'")
+
+    return AVAILABLE_ELM_MODELS[key]()
+
+
+def load_oselm_model(key):
+    """Load an OSELM model based on key"""
+    if key not in AVAILABLE_OSELM_MODELS:
+        raise ValueError(f"Not supported OSELM model for key '{key}'")
+
+    return AVAILABLE_OSELM_MODELS[key]()
diff --git a/tests/unit/test_elm.py b/tests/unit/test_elm.py
index 9f55791..4c96d9f 100644
--- a/tests/unit/test_elm.py
+++ b/tests/unit/test_elm.py
@@ -10,11 +10,11 @@
 from pyoselm.layer import RBFRandomLayer
 
 
-@pytest.mark.parametrize("hidden_layer", [None, RBFRandomLayer()])
+@pytest.mark.parametrize("hidden_layer", [None, RBFRandomLayer(random_state=123)])
 @pytest.mark.parametrize("regressor", [None, LinearRegression()])
 def test_gen_elm_regressor(hidden_layer, regressor):
     # get data
-    X, y = make_regression(n_samples=100, n_targets=1, n_features=10)
+    X, y = make_regression(n_samples=100, n_targets=1, n_features=10, random_state=123)
 
     # build model
     model = GenELMRegressor(hidden_layer=hidden_layer, regressor=regressor)
@@ -33,10 +33,10 @@ def test_gen_elm_regressor(hidden_layer, regressor):
     assert -1.0 < score <= 1.0, "Score of model is not in expected range"
 
 
-@pytest.mark.parametrize("hidden_layer", [None, RBFRandomLayer()])
+@pytest.mark.parametrize("hidden_layer", [None, RBFRandomLayer(random_state=123)])
 @pytest.mark.parametrize("binarizer", [None,
-                                       LabelBinarizer(0, 1),
-                                       LabelBinarizer(-1, 1)])
+                                       LabelBinarizer(neg_label=0, pos_label=1),
+                                       LabelBinarizer(neg_label=-1, pos_label=1)])
 @pytest.mark.parametrize("regressor", [None, LinearRegression()])
 def test_gen_elm_classifier(hidden_layer, binarizer, regressor):
     # get data
@@ -67,13 +67,13 @@ def test_gen_elm_classifier(hidden_layer, binarizer, regressor):
 @pytest.mark.parametrize("regressor", [None, LinearRegression()])
 def test_elm_regressor(n_hidden, activation_func, regressor):
     # get data
-    X, y = make_regression(n_samples=100, n_targets=1, n_features=10)
+    X, y = make_regression(n_samples=100, n_targets=1, n_features=10, random_state=123)
 
     # build model
     model = ELMRegressor(n_hidden=n_hidden,
                          activation_func=activation_func,
-                         regressor=regressor)
-
+                         regressor=regressor,
+                         random_state=123)
     # fit model
     model.fit(X, y)
 
@@ -102,7 +102,8 @@ def test_elm_classifier(n_hidden, activation_func, binarizer, regressor):
     model = ELMClassifier(n_hidden=n_hidden,
                           activation_func=activation_func,
                           binarizer=binarizer,
-                          regressor=regressor)
+                          regressor=regressor,
+                          random_state=123)
 
     # fit model
     model.fit(X, y)
@@ -125,7 +126,7 @@ def test_elm_classifier(n_hidden, activation_func, binarizer, regressor):
 
 def test_elm_regressor_reproducible_results():
     # get data
-    X, y = make_regression(n_samples=100, n_targets=1, n_features=10)
+    X, y = make_regression(n_samples=100, n_targets=1, n_features=10, random_state=123)
 
     # build model 1
     model = ELMRegressor(random_state=123)
diff --git a/tests/unit/test_oselm.py b/tests/unit/test_oselm.py
index 559ae9a..c8338be 100644
--- a/tests/unit/test_oselm.py
+++ b/tests/unit/test_oselm.py
@@ -14,12 +14,13 @@
 @pytest.mark.parametrize("use_woodbury", [False, True])
 def test_oselm_regressor(n_hidden, activation_func, use_woodbury):
     # get data
-    X, y = make_regression(n_samples=100, n_targets=1, n_features=10)
+    X, y = make_regression(n_samples=100, n_targets=1, n_features=10, random_state=123)
 
     # build model
     model = OSELMRegressor(n_hidden=n_hidden,
                            activation_func=activation_func,
-                           use_woodbury=use_woodbury)
+                           use_woodbury=use_woodbury,
+                           random_state=123)
 
     # fit model
     model.fit(X, y)
@@ -60,7 +61,8 @@ def test_oselm_classifier(n_hidden, activation_func, binarizer):
     # build model
     model = OSELMClassifier(n_hidden=n_hidden,
                             activation_func=activation_func,
-                            binarizer=binarizer)
+                            binarizer=binarizer,
+                            random_state=123)
 
     # fit model
     model.fit(X, y)
@@ -139,7 +141,7 @@ def test_oselm_bad_path():
 @pytest.mark.skip("Very expensive test")
 def test_oselm_fit_woodbury_large_input():
     n = 20e3
-    X, y = make_regression(n_samples=int(n), n_targets=1, n_features=10)
+    X, y = make_regression(n_samples=int(n), n_targets=1, n_features=10, random_state=123)
 
     model = OSELMRegressor(use_woodbury=True)