Repository: NatLibFi/Annif (language: python)

Issue:
Use Schemathesis to test REST API
For testing the REST API there have been manually written tests since abandoning the unmaintained swagger-tester in #551. Already at that time Schemathesis was considered as a replacement of swagger-tester, but there was a [dependency conflict](https://github.com/NatLibFi/Annif/pull/551#issuecomment-1016506375) that prevented using it.

Now Schemathesis can be installed to restore testing that the API actually conforms to the OpenAPI specification. Also the specification will now be validated.

With the default settings Schemathesis takes 12-15 s to run the tests in `tests/test_swagger.py`, but using `@settings(max_examples=10)`  only 1-2 s.

~Due to a [bug](https://github.com/schemathesis/schemathesis/issues/1718) the pyproject.yaml includes pinned starlette installation (`starlette = ">=0.13,<0.21"`).~ Edit: Pinning starlette is not necessary when pinning to recent version Schemathesis `3.18.*`.

When fixing the errors noted by Schemathesis I marked some properties in the specification as nullable. `is_trained` and `modification_time` should be nullable, but `backend_id` probably should not be:
https://github.com/NatLibFi/Annif/blob/6fc315763a395dff86420f3d1f07e3b3fe3ca869/annif/openapi/annif.yaml#L173
 I marked this only to make the test pass. One project for the testing config is deliberately missing the backend specification:
https://github.com/NatLibFi/Annif/blob/6fc315763a395dff86420f3d1f07e3b3fe3ca869/tests/projects.cfg#L70-L74

I'm not really sure what to do for that.



Discussion / hints:
## [Codecov](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) Report
Patch coverage: **`100.00`**% and project coverage change: **`-0.01`** :warning:
> Comparison is base [(`e9fd177`)](https://codecov.io/gh/NatLibFi/Annif/commit/e9fd1771b36925bd68492346b6c4dde378dfd88a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) 99.57% compared to head [(`08bf4ce`)](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) 99.57%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #682      +/-   ##
==========================================
- Coverage   99.57%   99.57%   -0.01%     
==========================================
  Files          88       88              
  Lines        6170     6138      -32     
==========================================
- Hits         6144     6112      -32     
  Misses         26       26              
```


| [Impacted Files](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) | Coverage Δ | |
|---|---|---|
| [tests/conftest.py](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-dGVzdHMvY29uZnRlc3QucHk=) | `100.00% <ø> (ø)` | |
| [tests/test\_openapi.py](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-dGVzdHMvdGVzdF9vcGVuYXBpLnB5) | `100.00% <100.00%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/NatLibFi/Annif/pull/682?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi).

> is_trained and modification_time should be nullable, but backend_id probably should not be:

Agree, backend_id should not be nullable. We should ensure that the API never returns data like that.
Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=NatLibFi_Annif&pullRequest=682)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=682&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=NatLibFi_Annif&pullRequest=682) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=NatLibFi_Annif&pullRequest=682&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=NatLibFi_Annif&pullRequest=682&metric=new_duplicated_lines_density&view=list)


> > is_trained and modification_time should be nullable, but backend_id probably should not be:
> 
> Agree, backend_id should not be nullable. We should ensure that the API never returns data like that.

I marked `nobackend` project as private, which I think is enough for this PR.
I started to feel that maybe some of the manually written tests should have not been deleted. 

Schemathesis does use the examples defined in the API schema (indirectly stated in the [CLI part of the documentation](https://schemathesis.readthedocs.io/en/stable/cli.html#tests-configuration) for the `--hypothesis-phases=explicit` option). This is true also when using it via pytest: I removed dummy-fi project from Annif test projects and 404 responses from the API schema, and this made those tests fail that use dummy-fi as an example project id in the schema. Reason for fails was shown to be

    Received a response with a status code, which is not defined in the schema: 404

and in details `"Project 'dummy-fi' not found"`.

But still this does not mean that e.g. the query parameters for `/suggest-batch` method are correctly passed and processed in `rest.py`. However it seems to need some amount of effort to make up an error that either tests in `test_openapi.py` or `test_rest.py` wont catch.


Fix patch:
diff --git a/.github/workflows/cicd.yml b/.github/workflows/cicd.yml
index c799228b3..b9ea4cea9 100644
--- a/.github/workflows/cicd.yml
+++ b/.github/workflows/cicd.yml
@@ -77,6 +77,9 @@ jobs:
     - name: Test with pytest
       run: |
         poetry run pytest --cov=./ --cov-report xml
+        if [[ ${{ matrix.python-version }} == '3.9' ]]; then
+          poetry run pytest --cov=./ --cov-report xml --cov-append -m slow
+        fi
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@81cd2dc8148241f03f5839d295e000b8f761e378  # v3.1.0
     - name: Save cache
diff --git a/annif/openapi/annif.yaml b/annif/openapi/annif.yaml
index 117faef9c..09d26a2a0 100644
--- a/annif/openapi/annif.yaml
+++ b/annif/openapi/annif.yaml
@@ -224,6 +224,7 @@ components:
         notation:
           type: string
           example: "42.42"
+          nullable: true
         score:
           type: number
           example: 0.85
@@ -246,6 +247,8 @@ components:
           type: array
           items:
             type: object
+            required:
+              - uri
             properties:
               uri:
                 type: string
diff --git a/pyproject.toml b/pyproject.toml
index 4c7fb93a9..5a69bc4b1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -91,3 +91,6 @@ build-backend = "poetry.core.masonry.api"
 profile = "black"
 line_length = "88"
 skip_gitignore = true
+
+[tool.pytest.ini_options]
+addopts = "-m 'not slow'"


Test patch:
diff --git a/tests/test_openapi.py b/tests/test_openapi.py
index e77230fd9..91de50385 100644
--- a/tests/test_openapi.py
+++ b/tests/test_openapi.py
@@ -1,4 +1,5 @@
 """Unit tests for Annif REST API / Swagger spec"""
+import pytest
 import schemathesis
 from hypothesis import settings
 
@@ -15,3 +16,12 @@ def check_cors(response, case):
 def test_api(case, app):
     response = case.call_wsgi(app)
     case.validate_response(response, additional_checks=(check_cors,))
+
+
+@pytest.mark.slow
+@schema.parametrize(endpoint="/v1/projects/{project_id}")
+@settings(max_examples=50)
+def test_api_target_dummy_fi(case, app):
+    case.path_parameters = {"project_id": "dummy-fi"}
+    response = case.call_wsgi(app)
+    case.validate_response(response)