Repository: opendatacube/odc-tools (language: python)

Issue:
There's no integration tests
Some of the tools in this repository rely on the ODC Database to do their work.

We do mock things like SQS or datasets and run simple functional tests, but some tools like `sqs-to-dc` or `s3-to-dc` really need to be tested end-to-end.

Can we have a discussion about how to set up some integration tests?

The simplest way is to have a shell script that calls the tools, with a Docker Compose file that sets up an environment.

More complex is having an environment that can be bootstapped and then mocking things like SQS and S3 to do the testing from.

Thoughts?


Discussion / hints:
@alexgleith the bulk of the work seems to happen in functions like `queue_to_odc` which accept a `Datacube` and only really use a few functions from it. We could go a long way towards fully testing by mocking a `Datacube` and using this in our tests.

Although I agree that full testing would be ideal! but this might be a lot easier
Hey @jeremyh do you think it's plausible at all to mock a Datacube instance? It'll be a crazy amount or work, right...
not all of it though! afaik we only need:

- ` dc.index.datasets.has`
- `dc.index.products.get_by_name`
- `dc.index.datasets.bulk_get`
- `dc.index.datasets.add` (can do nothing in mocked dc)
- `dc.index.datasets.update` (can do nothing in mocked dc)
- `dc.index.datasets.archive` (can do nothing in mocked dc)

I've also done some work in stats creating test datasets products that we can leverage https://github.com/opendatacube/odc-tools/blob/fbf58611b52bb935fed52ab504214235490decb6/libs/stats/tests/test_utils.py#L192
Yeah, you could just mock those methods you use. I think the majority of bugs we hit deal with quirks in the interaction of a real ODC instances, though. Especially across different version of ODC and postgres etc.


One option is to use the pytest plugin for ODC in the digitalearthau repository -- it's what Explorer and others use.

To try it, add the plugin to your pytest `conftest.py`:
```
pytest_plugins = "digitalearthau.testing.plugin"
```

Now there are pytest fixtures available called `index` and `dea_index` (the latter includes DEA's products already added to the db). 

```
# Use in a test:
def test_with_datacube(index: Index):
    index.datasets.add(...)
    dc = Datacube(index)
    ... etc

# Or create your own fixture with preloaded data
@pytest.fixture
def my_index(index: Index):
    # Add data.
    return index

def test_with_my_index(my_index: Index):
    ....
```

The DB schema is reset individually for each test that's run (by default).

(by default, it will connect to database `dea_integration` on localhost. But you can add a `~/.datacube_integration.conf` with custom config, identical to a standard `~/.datacube.conf`... and you'd need docker compose or similar, as with Explorer, for doing this in Github Actions)

I will backport tests from `datacube-docker` and add the [docker-compose](https://github.com/opendatacube/datacube-docker/blob/main/index/docker-compose.yml) stack to test this in a way I am familiar with.

Fix patch:
diff --git a/.coveragerc b/.coveragerc
new file mode 100644
index 000000000..1cb8c51ca
--- /dev/null
+++ b/.coveragerc
@@ -0,0 +1,2 @@
+[run]
+omit = *tests*
\ No newline at end of file
diff --git a/apps/dc_tools/assets/bootstrap-odc.sh b/apps/dc_tools/assets/bootstrap-odc.sh
new file mode 100755
index 000000000..eda764340
--- /dev/null
+++ b/apps/dc_tools/assets/bootstrap-odc.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+############################################################
+# This script allows creation of all metadata/products     #
+# in an ODC instance from a given CSV catalog definition   #
+# In local development activate <odc> conda environment    #
+# In kubernetes pod executor run from <datacube-index>     #
+# container                                                #
+############################################################
+set -o errexit
+set -o pipefail
+set -o nounset
+
+product_catalog=$1
+metadata_catalog=$2
+
+datacube system init --no-default-types --no-init-users
+# Created using : datacube metadata list | awk '{print $1}' | xargs datacube metadata show
+datacube metadata add "$metadata_catalog"
+python -m wget "$product_catalog" -o product_list.csv
+tail -n+2 product_list.csv | awk -F, '{print $2}' | xargs datacube -v product add
+
+# Clean up
+rm product_list.csv


Test patch:
diff --git a/.github/workflows/test-dc-tools.yml b/.github/workflows/test-dc-tools.yml
index 298e17b70..8cccd6965 100644
--- a/.github/workflows/test-dc-tools.yml
+++ b/.github/workflows/test-dc-tools.yml
@@ -1,6 +1,7 @@
 name : Run integration tests on dc_tools
 
 on:
+  pull_request:
   push:
 
 jobs:
@@ -20,7 +21,10 @@ jobs:
     - name: Run Dockerized Tests
       timeout-minutes: 20
       shell: bash
+      # TODO: Fix superlong line below
       run: |
         cd tests
         docker-compose up -d
-        docker-compose exec -T tools-tester pytest --timeout=30
+        docker-compose exec -T tools-tester bash -c "pip install wget && /code/apps/dc_tools/assets/bootstrap-odc.sh \$PRODUCT_CATALOG \$METADATA_CATALOG" || true
+        docker-compose exec -T tools-tester datacube product add https://raw.githubusercontent.com/digitalearthafrica/config/78dfc5a1895b48225803cb0891b99eb01feb9f2e/products/esa_s2_l2a.odc-product.yaml
+        docker-compose exec -T tools-tester pytest --cov=/code/ --timeout=30
diff --git a/apps/dc_tools/tests/test_azure_to_dc.py b/apps/dc_tools/tests/test_azure_to_dc.py
new file mode 100644
index 000000000..657312768
--- /dev/null
+++ b/apps/dc_tools/tests/test_azure_to_dc.py
@@ -0,0 +1,1 @@
+# Tests using the Click framework the azure-to-dc CLI tool
\ No newline at end of file
diff --git a/apps/dc_tools/tests/test_s3_to_dc.py b/apps/dc_tools/tests/test_s3_to_dc.py
new file mode 100644
index 000000000..756f357a3
--- /dev/null
+++ b/apps/dc_tools/tests/test_s3_to_dc.py
@@ -0,0 +1,28 @@
+# Tests using the Click framework the s3-to-dc CLI tool
+
+from click.testing import CliRunner
+
+from odc.apps.dc_tools.s3_to_dc import cli
+
+def test_s3_to_dc_yaml():
+  runner = CliRunner()
+  # This will fail if requester pays is enabled
+  result = runner.invoke(cli, [
+      "--no-sign-request",
+      "s3://dea-public-data/cemp_insar/insar/displacement/alos/2010/**/*.yaml",
+      "cemp_insar_alos_displacement"
+    ])
+  assert result.exit_code == 0
+  assert result.output == 'Added 25 Datasets, Failed 0 Datasets\n'
+
+def test_s3_to_dc_stac():
+  runner = CliRunner()
+  # This will fail if requester pays is enabled
+  result = runner.invoke(cli, [
+      "--no-sign-request",
+      "--stac",
+      "s3://sentinel-cogs/sentinel-s2-l2a-cogs/2020/S2A_32NNF_20200127_0_L2A/*.json",
+      "s2_l2a"
+    ])
+  assert result.exit_code == 0
+  assert result.output == 'Added 1 Datasets, Failed 0 Datasets\n'
\ No newline at end of file
diff --git a/apps/dc_tools/tests/test_stac_api_to_dc.py b/apps/dc_tools/tests/test_stac_api_to_dc.py
new file mode 100644
index 000000000..a874b3cbb
--- /dev/null
+++ b/apps/dc_tools/tests/test_stac_api_to_dc.py
@@ -0,0 +1,1 @@
+# Tests using the Click framework the stac_api-to-dc CLI tool
\ No newline at end of file
diff --git a/tests/docker-compose.yml b/tests/docker-compose.yml
index 46c519c82..d8d9eee81 100644
--- a/tests/docker-compose.yml
+++ b/tests/docker-compose.yml
@@ -21,6 +21,8 @@ services:
       - DB_PORT=5432
       - AWS_DEFAULT_REGION=ap-southeast-2
       - STAC_API_URL=https://earth-search.aws.element84.com/v0/
+      - PRODUCT_CATALOG=https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/workspaces/prod-products.csv
+      - METADATA_CATALOG=https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/workspaces/sandbox-metadata.yaml
     depends_on:
       - postgres
     volumes: