Repository: MobileTeleSystems/onetl (language: python)

Issue:
[DOP-3137] Add tests workflow for HDFS
Added CI workflow for HDFS.

Unlike other images, HDFS requires config file to be mounted to container, but `services` field in Github Actions starts container before checkout. So instead I use `docker-compose` to start container with proper mounts.

Also new Hadoop image uses different port numbers, so I've updated `.env` and `docker-compose.yml` files.


Discussion / hints:
## [Codecov](https://codecov.io/gh/MobileTeleSystems/onetl/pull/21?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) Report
> Merging [#21](https://codecov.io/gh/MobileTeleSystems/onetl/pull/21?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) (b541d21) into [develop](https://codecov.io/gh/MobileTeleSystems/onetl/commit/6ccefeeed7dd658c61e0701e425118f1b4d1f6c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) (6ccefee) will **increase** coverage by `0.03%`.
> The diff coverage is `100.00%`.

```diff
@@             Coverage Diff             @@
##           develop      #21      +/-   ##
===========================================
+ Coverage    90.79%   90.83%   +0.03%     
===========================================
  Files          103      105       +2     
  Lines         5130     5381     +251     
  Branches       969     1030      +61     
===========================================
+ Hits          4658     4888     +230     
- Misses         376      397      +21     
  Partials        96       96              
```


| [Impacted Files](https://codecov.io/gh/MobileTeleSystems/onetl/pull/21?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) | Coverage Δ | |
|---|---|---|
| [onetl/connection/file\_connection/hdfs.py](https://codecov.io/gh/MobileTeleSystems/onetl/pull/21?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9maWxlX2Nvbm5lY3Rpb24vaGRmcy5weQ==) | `95.83% <100.00%> (ø)` | |

... and [3 files with indirect coverage changes](https://codecov.io/gh/MobileTeleSystems/onetl/pull/21/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems)



Fix patch:
diff --git a/docker-compose.yml b/docker-compose.yml
index a9be0c6a7..0add1e6a1 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -85,16 +85,14 @@ services:
     - onetl
 
   webdav:
-    image: docker.rep.msk.mts.ru/bigdata/platform/onetools/onetl/webdav:develop
-    build:
-      dockerfile: ./docker/webdav/Dockerfile
-      context: .
-    hostname: webdav
+    image: chonjay21/webdav:latest
     restart: unless-stopped
     env_file: .env.dependencies
     ports:
     - 8000:80
-    - 8443:443
+    volumes:
+    # Remove after https://github.com/chonjay21/docker-webdav/pull/3
+    - ./docker/webdav/on_post_init.sh:/sources/webdav/eventscripts/on_post_init.sh
     networks:
     - onetl
 
diff --git a/docker/webdav/Dockerfile b/docker/webdav/Dockerfile
deleted file mode 100644
index b3e29f6d2..000000000
--- a/docker/webdav/Dockerfile
+++ /dev/null
@@ -1,3 +0,0 @@
-FROM chonjay21/webdav:latest
-
-COPY ./docker/webdav/on_post_init.sh /sources/webdav/eventscripts/


Test patch:
diff --git a/.github/workflows/test-webdav.yml b/.github/workflows/test-webdav.yml
new file mode 100644
index 000000000..598b07c12
--- /dev/null
+++ b/.github/workflows/test-webdav.yml
@@ -0,0 +1,81 @@
+name: Tests for WebDAV
+on:
+  workflow_call:
+    inputs:
+      webdav-version:
+        required: true
+        type: string
+      python-version:
+        required: true
+        type: string
+      os:
+        required: true
+        type: string
+
+jobs:
+  test-webdav:
+    name: Run WebDAV tests (server=${{ inputs.webdav-version }}, python=${{ inputs.python-version }}, os=${{ inputs.os }})
+    runs-on: ${{ inputs.os }}
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v3
+
+    - name: Set up Python ${{ inputs.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ inputs.python-version }}
+
+    - name: Cache pip
+      uses: actions/cache@v3
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-python-${{ inputs.python-version }}-tests-webdav-${{ hashFiles('requirements/core.txt', 'requirements/webdav.txt', 'requirements/tests/base.txt', 'requirements/tests/spark*.txt') }}
+        restore-keys: |
+          ${{ runner.os }}-python-${{ inputs.python-version }}-tests-webdav-${{ hashFiles('requirements/core.txt', 'requirements/webdav.txt', 'requirements/tests/base.txt', 'requirements/tests/spark*.txt') }}
+          ${{ runner.os }}-python-${{ inputs.python-version }}-tests-webdav-
+
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip setuptools wheel
+
+    - name: Install dependencies
+      run: |
+        pip install -I \
+          -r requirements/core.txt \
+          -r requirements/webdav.txt \
+          -r requirements/tests/base.txt
+
+    # Replace with Github Actions' services after https://github.com/chonjay21/docker-webdav/pull/3
+    # Cannot use services because we need to mount config file from the repo, but services start before checkout.
+    # See https://github.com/orgs/community/discussions/25792
+    - name: Start WebDAV
+      run: |
+        docker compose down -v --remove-orphans
+        docker compose up -d webdav
+      env:
+        WEBDAV_IMAGE: chonjay21/webdav:${{ inputs.webdav-version }}
+        COMPOSE_PROJECT_NAME: ${{ github.run_id }}-webdav${{ inputs.webdav-version }}
+
+    - name: Wait for WebDAV to be ready
+      run: |
+        ./docker/wait-for-it.sh -h localhost -p 8000 -t 60
+
+    - name: Run tests
+      run: |
+        mkdir reports/ || echo "Directory exists"
+        sed '/^$/d' ./.env.local | sed '/^#/d' | sed 's/^/export /' > ./env
+        source ./env
+        ./pytest_runner.sh -m webdav
+
+    - name: Shutdown WebDAV
+      if: always()
+      run: |
+        docker compose down -v --remove-orphans
+      env:
+        COMPOSE_PROJECT_NAME: ${{ github.run_id }}-webdav${{ inputs.webdav-version }}
+
+    - name: Upload coverage results
+      uses: actions/upload-artifact@v3
+      with:
+        name: webdav-${{ inputs.webdav-version }}-python-${{ inputs.python-version }}-os-${{ inputs.os }}
+        path: reports/*
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index c49b19716..b8d34b53c 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -289,6 +289,25 @@ jobs:
       python-version: ${{ matrix.python-version }}
       os: ${{ matrix.os }}
 
+  tests-webdav:
+    name: Run WebDAV tests (server=${{ matrix.openwebdavssh-version }}, python=${{ matrix.python-version }}, os=${{ matrix.os }})
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - webdav-version: latest
+          python-version: '3.7'
+          os: ubuntu-latest
+        - webdav-version: latest
+          python-version: '3.11'
+          os: ubuntu-latest
+
+    uses: ./.github/workflows/test-webdav.yml
+    with:
+      webdav-version: ${{ matrix.webdav-version }}
+      python-version: ${{ matrix.python-version }}
+      os: ${{ matrix.os }}
+
   all_done:
     name: Tests done
     runs-on: ubuntu-latest
@@ -305,6 +324,7 @@ jobs:
     - tests-hdfs
     - tests-s3
     - tests-sftp
+    - tests-webdav
 
     steps:
     - name: Checkout code
diff --git a/tests/tests_integration/tests_file_connection_integration/test_webdav_integration.py b/tests/tests_integration/tests_file_connection_integration/test_webdav_integration.py
new file mode 100644
index 000000000..6fd03c743
--- /dev/null
+++ b/tests/tests_integration/tests_file_connection_integration/test_webdav_integration.py
@@ -0,0 +1,30 @@
+import logging
+
+import pytest
+
+pytestmark = [pytest.mark.webdav, pytest.mark.file_connection, pytest.mark.connection]
+
+
+def test_webdav_check(webdav_connection, caplog):
+    with caplog.at_level(logging.INFO):
+        assert webdav_connection.check() == webdav_connection
+
+    assert "type = WebDAV" in caplog.text
+    assert f"host = '{webdav_connection.host}'" in caplog.text
+    assert f"port = {webdav_connection.port}" in caplog.text
+    assert f"protocol = '{webdav_connection.protocol}'" in caplog.text
+    assert f"ssl_verify = {webdav_connection.ssl_verify}" in caplog.text
+    assert f"user = '{webdav_connection.user}'" in caplog.text
+    assert "password = SecretStr('**********')" in caplog.text
+    assert webdav_connection.password.get_secret_value() not in caplog.text
+
+    assert "Connection is available" in caplog.text
+
+
+def test_webdav_wrong_source_check():
+    from onetl.connection import WebDAV
+
+    webdav = WebDAV(user="some_user", password="pwd", host="host", port=123)
+
+    with pytest.raises(RuntimeError, match="Connection is unavailable"):
+        webdav.check()
diff --git a/tests/tests_unit/tests_file_connection_unit/test_webdav_unit.py b/tests/tests_unit/tests_file_connection_unit/test_webdav_unit.py
new file mode 100644
index 000000000..7d92d494c
--- /dev/null
+++ b/tests/tests_unit/tests_file_connection_unit/test_webdav_unit.py
@@ -0,0 +1,45 @@
+import pytest
+
+from onetl.connection import FileConnection
+
+pytestmark = [pytest.mark.webdav, pytest.mark.file_connection, pytest.mark.connection]
+
+
+def test_webdav_connection():
+    from onetl.connection import WebDAV
+
+    webdav = WebDAV(host="some_host", user="some_user", password="pwd")
+    assert isinstance(webdav, FileConnection)
+    assert webdav.host == "some_host"
+    assert webdav.protocol == "https"
+    assert webdav.port == 443
+    assert webdav.user == "some_user"
+    assert webdav.password != "pwd"
+    assert webdav.password.get_secret_value() == "pwd"
+
+    assert "password='pwd'" not in str(webdav)
+    assert "password='pwd'" not in repr(webdav)
+
+
+def test_webdav_connection_with_http():
+    from onetl.connection import WebDAV
+
+    webdav = WebDAV(host="some_host", user="some_user", password="pwd", protocol="http")
+    assert webdav.protocol == "http"
+    assert webdav.port == 80
+
+
+@pytest.mark.parametrize("protocol", ["http", "https"])
+def test_webdav_connection_with_custom_port(protocol):
+    from onetl.connection import WebDAV
+
+    webdav = WebDAV(host="some_host", user="some_user", password="pwd", port=500, protocol=protocol)
+    assert webdav.protocol == protocol
+    assert webdav.port == 500
+
+
+def test_webdav_connection_without_mandatory_args():
+    from onetl.connection import WebDAV
+
+    with pytest.raises(ValueError):
+        WebDAV()