Repository: MobileTeleSystems/onetl (language: python)

Issue:
[DOP-9653] Test onETL compatibility with Spark 3.5.0
<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->
<!-- See https://github.com/MobileTeleSystems/onetl/blob/develop/CONTRIBUTING.rst for help on Contributing -->
<!-- PLEASE DO **NOT** put issue ids in the PR title! Instead, add a descriptive title and put ids in the body -->

## Change Summary

<!-- Please give a short summary of the changes. -->

Nightly tests with latest pyspark version shown that some classes do not yet support Spark 3.5.0:
https://github.com/MobileTeleSystems/onetl/actions/runs/6336863272/job/17210825113

* MongoDB fails with sone internal error. This could be caused by used package version (10.1 then latest is 10.2).
* Excel [does not have package](https://github.com/crealytics/spark-excel/issues/787) for 3.5.0 yet: https://mvnrepository.com/artifact/com.crealytics/spark-excel

Other connection and file formats do support 3.5.0, so updated documentation of connectors & Readme.

## Related issue number

<!-- Are there any issues opened that will be resolved by merging this change? -->
<!-- WARNING: please use "fix #123" style references so the issue is closed when this PR is merged. -->

## Checklist

* [X] Commit message and PR title is comprehensive
* [X] Keep the change as small as possible
* [ ] Unit and integration tests for the changes exist
* [X] Tests pass on CI and coverage does not decrease
* [X] Documentation reflects the changes where applicable
* [X] `docs/changelog/next_release/<pull request or issue id>.<change type>.rst` file added describing change
  (see [CONTRIBUTING.rst](https://github.com/MobileTeleSystems/onetl/blob/develop/CONTRIBUTING.rst) for details.)
* [X] My PR is ready to review.



Discussion / hints:
## [Codecov](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) Report
> Merging [#159](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) (8467098) into [develop](https://app.codecov.io/gh/MobileTeleSystems/onetl/commit/08b38ff0d2f4e0147830d5dd4710ed48a982532a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) (08b38ff) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

```diff
@@             Coverage Diff             @@
##           develop     #159      +/-   ##
===========================================
+ Coverage    94.33%   94.35%   +0.02%     
===========================================
  Files          209      209              
  Lines         7925     7925              
  Branches      1490     1490              
===========================================
+ Hits          7476     7478       +2     
+ Misses         326      325       -1     
+ Partials       123      122       -1     
```


| [Files](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) | Coverage Δ | |
|---|---|---|
| [.../connection/db\_connection/clickhouse/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL2NsaWNraG91c2UvY29ubmVjdGlvbi5weQ==) | `100.00% <ø> (ø)` | |
| [onetl/connection/db\_connection/hive/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL2hpdmUvY29ubmVjdGlvbi5weQ==) | `91.07% <ø> (ø)` | |
| [onetl/connection/db\_connection/kafka/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL2thZmthL2Nvbm5lY3Rpb24ucHk=) | `98.98% <ø> (+1.01%)` | :arrow_up: |
| [onetl/connection/db\_connection/mssql/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL21zc3FsL2Nvbm5lY3Rpb24ucHk=) | `100.00% <ø> (ø)` | |
| [onetl/connection/db\_connection/mysql/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL215c3FsL2Nvbm5lY3Rpb24ucHk=) | `100.00% <ø> (ø)` | |
| [...netl/connection/db\_connection/oracle/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL29yYWNsZS9jb25uZWN0aW9uLnB5) | `86.18% <ø> (ø)` | |
| [...tl/connection/db\_connection/postgres/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL3Bvc3RncmVzL2Nvbm5lY3Rpb24ucHk=) | `97.61% <ø> (ø)` | |
| [...tl/connection/db\_connection/teradata/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9kYl9jb25uZWN0aW9uL3RlcmFkYXRhL2Nvbm5lY3Rpb24ucHk=) | `100.00% <ø> (ø)` | |
| [...ection/file\_df\_connection/spark\_hdfs/connection.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9maWxlX2RmX2Nvbm5lY3Rpb24vc3BhcmtfaGRmcy9jb25uZWN0aW9uLnB5) | `100.00% <ø> (ø)` | |
| [...tl/connection/file\_df\_connection/spark\_local\_fs.py](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems#diff-b25ldGwvY29ubmVjdGlvbi9maWxlX2RmX2Nvbm5lY3Rpb24vc3BhcmtfbG9jYWxfZnMucHk=) | `100.00% <ø> (ø)` | |
| ... and [2 more](https://app.codecov.io/gh/MobileTeleSystems/onetl/pull/159?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=MobileTeleSystems) | |



Fix patch:
diff --git a/.github/workflows/data/local-fs/matrix.yml b/.github/workflows/data/local-fs/matrix.yml
index e956169ba..4329d6582 100644
--- a/.github/workflows/data/local-fs/matrix.yml
+++ b/.github/workflows/data/local-fs/matrix.yml
@@ -16,12 +16,18 @@ min_excel: &min_excel
   java-version: 8
   os: ubuntu-latest
 
-max: &max
+max_excel: &max_excel
   spark-version: 3.4.1
   python-version: '3.11'
   java-version: 20
   os: ubuntu-latest
 
+max: &max
+  spark-version: 3.5.0
+  python-version: '3.11'
+  java-version: 20
+  os: ubuntu-latest
+
 latest: &latest
   spark-version: latest
   python-version: '3.11'
@@ -30,13 +36,13 @@ latest: &latest
 
 matrix:
   small:
+  - <<: *max_excel
   - <<: *max
-  - <<: *min_avro
-  - <<: *min_excel
   full:
   - <<: *min
   - <<: *min_avro
   - <<: *min_excel
+  - <<: *max_excel
   - <<: *max
   nightly:
   - <<: *min


Test patch:
diff --git a/tests/.coveragerc b/tests/.coveragerc
index 17cafe455..218452499 100644
--- a/tests/.coveragerc
+++ b/tests/.coveragerc
@@ -19,3 +19,4 @@ exclude_lines =
     if pyspark_version
     spark = SparkSession._instantiatedSession
     if log.isEnabledFor(logging.DEBUG):
+    if sys.version_info
diff --git a/tests/tests_integration/test_file_format_integration/test_excel_integration.py b/tests/tests_integration/test_file_format_integration/test_excel_integration.py
index de8cc9cf9..f9aaad38f 100644
--- a/tests/tests_integration/test_file_format_integration/test_excel_integration.py
+++ b/tests/tests_integration/test_file_format_integration/test_excel_integration.py
@@ -33,6 +33,8 @@ def test_excel_reader_with_infer_schema(
     spark_version = get_spark_version(spark)
     if spark_version < (3, 2):
         pytest.skip("Excel files are supported on Spark 3.2+ only")
+    if spark_version >= (3, 5):
+        pytest.skip("Excel files are not supported on Spark 3.5+ yet")
 
     file_df_connection, source_path, _ = local_fs_file_df_connection_with_path_and_files
     df = file_df_dataframe
@@ -81,6 +83,8 @@ def test_excel_reader_with_options(
     spark_version = get_spark_version(spark)
     if spark_version < (3, 2):
         pytest.skip("Excel files are supported on Spark 3.2+ only")
+    if spark_version >= (3, 5):
+        pytest.skip("Excel files are not supported on Spark 3.5+ yet")
 
     local_fs, source_path, _ = local_fs_file_df_connection_with_path_and_files
     df = file_df_dataframe
@@ -117,6 +121,8 @@ def test_excel_writer(
     spark_version = get_spark_version(spark)
     if spark_version < (3, 2):
         pytest.skip("Excel files are supported on Spark 3.2+ only")
+    if spark_version >= (3, 5):
+        pytest.skip("Excel files are not supported on Spark 3.5+ yet")
 
     file_df_connection, source_path = local_fs_file_df_connection_with_path
     df = file_df_dataframe