Repository: nchammas/flintrock (language: python)

Issue:
flintrock requires a pretty old version of Click
* Flintrock version: 2.0.0
* Python version: 3.11.6
* OS: Ubuntu 22.04

Flintrock appears to require an old version of Click, which then conflicts with a bunch of other libraries. Time for an update?

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flask 3.0.0 requires click>=8.1.3, but you have click 7.0 which is incompatible.
fiona 1.9.4.post1 requires click~=8.0, but you have click 7.0 which is incompatible.
black 23.11.0 requires click>=8.0.0, but you have click 7.0 which is incompatible.
dask 2023.6.1 requires click>=8.0, but you have click 7.0 which is incompatible.
```


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/flintrock.yaml b/.github/workflows/flintrock.yaml
index 33d1923..6a39d90 100644
--- a/.github/workflows/flintrock.yaml
+++ b/.github/workflows/flintrock.yaml
@@ -17,10 +17,11 @@ jobs:
           - ubuntu-20.04
           - macos-11
         python-version:
-          - "3.7"
           - "3.8"
           - "3.9"
           - "3.10"
+          - "3.11"
+          - "3.12"
     name: ${{ matrix.os }} / Python ${{ matrix.python-version }}
     steps:
       - uses: actions/checkout@v3
diff --git a/CHANGES.md b/CHANGES.md
index 31af175..46b34c2 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -6,15 +6,16 @@
 
 ### Changed
 
-* [#348]: Bumped default Spark to 3.2; dropped support for Python 3.6; added CI build for Python 3.10.
+* [#348], [#367]: Bumped default Spark to 3.5.0 and default Hadoop to 3.3.6; dropped support for Python 3.6 and 3.7; added CI builds for Python 3.10, 3.11, and 3.12.
 * [#361]: Migrated from AdoptOpenJDK, which is deprecated, to Adoptium OpenJDK.
-* [#362][#366]: Improved Flintrock's ability to cleanup after launch failures.
+* [#362], [#366]: Improved Flintrock's ability to cleanup after launch failures.
 * [#366]: Deprecated `--ec2-spot-request-duration`, which is not needed for one-time spot instances launched using the RunInstances API.
 
 [#348]: https://github.com/nchammas/flintrock/pull/348
 [#361]: https://github.com/nchammas/flintrock/pull/361
 [#362]: https://github.com/nchammas/flintrock/pull/362
 [#366]: https://github.com/nchammas/flintrock/pull/366
+[#367]: https://github.com/nchammas/flintrock/pull/367
 
 ## [2.0.0] - 2021-06-10
 
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ccd406f..b7dfd83 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -80,9 +80,11 @@ There are a few things you should do before diving in to write a new feature or
 
 If you are changing anything about Flintrock's dependencies, be sure to update the compiled requirements using [pip-tools]:
 
+[pip-tools]: https://github.com/jazzband/pip-tools
+
 ```shell
 function update-deps() {
-    pip install -U "pip-tools==6.8.0"
+    pip install -U "pip-tools==7.3.0"
 
     pip-compile -U requirements/user.in -o requirements/user.pip
     pip-compile -U requirements/developer.in -o requirements/developer.pip
@@ -91,19 +93,13 @@ function update-deps() {
     # Uncomment whichever set of requirements makes sense for you.
     # pip-sync requirements/user.pip
     # pip-sync requirements/developer.pip
-    pip-sync requirements/maintainer.pip
+    # pip-sync requirements/maintainer.pip
 }
 
 update-deps
 ```
 
-After doing that, make sure your environment matches what's in the compiled requirements by running `pip-sync` against the appropriate requirements file:
-
-```
-pip-sync requirements/[user|developer|maintainer].pip
-```
-
-[pip-tools]: https://github.com/jazzband/pip-tools
+`pip-compile` takes the provided set of input requirements, like `user.in` and compiles them into a full list of pinned transitive dependencies, like `user.pip`. This is similar to a lock file. `pip-sync` ensures that the current active virtual environment has exactly the dependencies listed in the provided pip file, no more and no less.
 
 #### Coordinate first
 
diff --git a/README.md b/README.md
index 623dbb3..b83a54f 100644
--- a/README.md
+++ b/README.md
@@ -30,7 +30,7 @@ Here's a quick way to launch a cluster on EC2, assuming you already have an [AWS
 ```sh
 flintrock launch test-cluster \
     --num-slaves 1 \
-    --spark-version 3.3.0 \
+    --spark-version 3.5.0 \
     --ec2-key-name key_name \
     --ec2-identity-file /path/to/key.pem \
     --ec2-ami ami-0aeeebd8d2ab47354 \
@@ -87,12 +87,12 @@ these steps:
    better performance.
 3. Make sure Flintrock is configured to use Hadoop/HDFS 2.7+. Earlier
    versions of Hadoop do not have solid implementations of `s3a://`.
-   Flintrock's default is Hadoop 3.3.4, so you don't need to do anything
+   Flintrock's default is Hadoop 3.3.6, so you don't need to do anything
    here if you're using a vanilla configuration.
 4. Call Spark with the hadoop-aws package to enable `s3a://`. For example:
    ```sh
-   spark-submit --packages org.apache.hadoop:hadoop-aws:3.3.4 my-app.py
-   pyspark --packages org.apache.hadoop:hadoop-aws:3.3.4
+   spark-submit --packages org.apache.hadoop:hadoop-aws:3.3.6 my-app.py
+   pyspark --packages org.apache.hadoop:hadoop-aws:3.3.6
    ```
    If you have issues using the package, consult the [hadoop-aws troubleshooting
    guide](http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html)
@@ -114,7 +114,7 @@ Before using Flintrock, take a quick look at the
 notice and [license](https://github.com/nchammas/flintrock/blob/master/LICENSE)
 and make sure you're OK with their terms.
 
-**Flintrock requires Python 3.7 or newer**, unless you are using one
+**Flintrock requires Python 3.8 or newer**, unless you are using one
 of our **standalone packages**. Flintrock has been thoroughly tested
 only on OS X, but it should run on all POSIX systems.
 A motivated contributor should be able to add
@@ -252,7 +252,7 @@ provider: ec2
 
 services:
   spark:
-    version: 3.3.0
+    version: 3.5.0
 
 launch:
   num-slaves: 1
diff --git a/flintrock/config.yaml.template b/flintrock/config.yaml.template
index fdf8da6..872649a 100644
--- a/flintrock/config.yaml.template
+++ b/flintrock/config.yaml.template
@@ -1,6 +1,6 @@
 services:
   spark:
-    version: 3.3.0
+    version: 3.5.0
     # git-commit: latest  # if not 'latest', provide a full commit SHA; e.g. d6dc12ef0146ae409834c78737c116050961f350
     # git-repository:  # optional; defaults to https://github.com/apache/spark
     # optional; defaults to download from a dynamically selected Apache mirror
@@ -12,7 +12,7 @@ services:
     # download-source: "s3://some-bucket/spark/{v}/"
     # executor-instances: 1
   hdfs:
-    version: 3.3.4
+    version: 3.3.6
     # optional; defaults to download from a dynamically selected Apache mirror
     #   - can be http, https, or s3 URL
     #   - must contain a {v} template corresponding to the version
diff --git a/flintrock/flintrock.py b/flintrock/flintrock.py
index bd7f353..6e722e0 100644
--- a/flintrock/flintrock.py
+++ b/flintrock/flintrock.py
@@ -292,7 +292,7 @@ def cli(cli_context, config, provider, debug):
 @click.option('--num-slaves', type=click.IntRange(min=1), required=True)
 @click.option('--java-version', type=click.IntRange(min=8), default=11)
 @click.option('--install-hdfs/--no-install-hdfs', default=False)
-@click.option('--hdfs-version', default='3.3.4')
+@click.option('--hdfs-version', default='3.3.6')
 @click.option('--hdfs-download-source',
               help=(
                   "URL to download Hadoop from. If an S3 URL, Flintrock will use the "
@@ -1121,10 +1121,13 @@ def normalize_keys(obj):
     """
     Used to map keys from config files to Python parameter names.
     """
-    if type(obj) != dict:
+    if not isinstance(obj, dict):
         return obj
     else:
-        return {k.replace('-', '_'): normalize_keys(v) for k, v in obj.items()}
+        return {
+            k.replace('-', '_'): normalize_keys(v)
+            for k, v in obj.items()
+        }
 
 
 def config_to_click(config: dict) -> dict:
diff --git a/requirements/developer.in b/requirements/developer.in
index 1728700..a1658c3 100644
--- a/requirements/developer.in
+++ b/requirements/developer.in
@@ -1,6 +1,5 @@
 -r user.pip
 pytest >= 3.5.0
 pytest-cov >= 2.5.1
-flake8 == 3.8.4
-freezegun == 1.1.0
+flake8 == 6.1.0
 # PyYAML  # requirement already covered by setup.py
diff --git a/requirements/developer.pip b/requirements/developer.pip
index f6186c1..3e84878 100644
--- a/requirements/developer.pip
+++ b/requirements/developer.pip
@@ -1,119 +1,103 @@
 #
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
 #
 #    pip-compile --output-file=requirements/developer.pip requirements/developer.in
 #
 -e file:.#egg=Flintrock
     # via -r requirements/user.pip
-attrs==22.1.0
-    # via pytest
-bcrypt==3.2.2
+bcrypt==4.0.1
     # via
     #   -r requirements/user.pip
     #   paramiko
-boto3==1.21.44
+boto3==1.29.4
     # via
     #   -r requirements/user.pip
     #   flintrock
-botocore==1.24.44
+botocore==1.32.4
     # via
     #   -r requirements/user.pip
     #   boto3
     #   flintrock
     #   s3transfer
-cffi==1.15.1
+cffi==1.16.0
     # via
     #   -r requirements/user.pip
-    #   bcrypt
     #   cryptography
     #   pynacl
-click==7.1.2
+click==8.1.7
     # via
     #   -r requirements/user.pip
     #   flintrock
-coverage[toml]==6.4.3
-    # via pytest-cov
-cryptography==37.0.4
+coverage[toml]==7.3.2
+    # via
+    #   coverage
+    #   pytest-cov
+cryptography==41.0.5
     # via
     #   -r requirements/user.pip
     #   flintrock
     #   paramiko
-flake8==3.8.4
-    # via -r requirements/developer.in
-freezegun==1.1.0
+exceptiongroup==1.2.0
+    # via pytest
+flake8==6.1.0
     # via -r requirements/developer.in
-importlib-metadata==4.12.0
-    # via
-    #   flake8
-    #   pluggy
-    #   pytest
-iniconfig==1.1.1
+iniconfig==2.0.0
     # via pytest
 jmespath==1.0.1
     # via
     #   -r requirements/user.pip
     #   boto3
     #   botocore
-mccabe==0.6.1
+mccabe==0.7.0
     # via flake8
-packaging==21.3
+packaging==23.2
     # via pytest
-paramiko==2.11.0
+paramiko==3.3.1
     # via
     #   -r requirements/user.pip
     #   flintrock
-pluggy==1.0.0
+pluggy==1.3.0
     # via pytest
-py==1.11.0
-    # via pytest
-pycodestyle==2.6.0
+pycodestyle==2.11.1
     # via flake8
 pycparser==2.21
     # via
     #   -r requirements/user.pip
     #   cffi
-pyflakes==2.2.0
+pyflakes==3.1.0
     # via flake8
 pynacl==1.5.0
     # via
     #   -r requirements/user.pip
     #   paramiko
-pyparsing==3.0.9
-    # via packaging
-pytest==7.1.2
+pytest==7.4.3
     # via
     #   -r requirements/developer.in
     #   pytest-cov
-pytest-cov==3.0.0
+pytest-cov==4.1.0
     # via -r requirements/developer.in
 python-dateutil==2.8.2
     # via
     #   -r requirements/user.pip
     #   botocore
-    #   freezegun
-pyyaml==6.0
+pyyaml==6.0.1
     # via
     #   -r requirements/user.pip
     #   flintrock
-s3transfer==0.5.2
+s3transfer==0.7.0
     # via
     #   -r requirements/user.pip
     #   boto3
 six==1.16.0
     # via
     #   -r requirements/user.pip
-    #   paramiko
     #   python-dateutil
 tomli==2.0.1
     # via
     #   coverage
     #   pytest
-typing-extensions==4.3.0
-    # via importlib-metadata
-urllib3==1.26.11
+urllib3==1.26.18
     # via
     #   -r requirements/user.pip
     #   botocore
-zipp==3.8.1
-    # via importlib-metadata
diff --git a/requirements/maintainer.in b/requirements/maintainer.in
index a5b9d26..2aafa5e 100644
--- a/requirements/maintainer.in
+++ b/requirements/maintainer.in
@@ -1,4 +1,4 @@
 -r developer.pip
 wheel >= 0.31.0
-twine == 3.8.0
-PyInstaller == 5.3
+twine == 4.0.2
+PyInstaller == 6.2.0
diff --git a/requirements/maintainer.pip b/requirements/maintainer.pip
index 8adaafb..dbc4d40 100644
--- a/requirements/maintainer.pip
+++ b/requirements/maintainer.pip
@@ -1,113 +1,112 @@
 #
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
 #
 #    pip-compile --output-file=requirements/maintainer.pip requirements/maintainer.in
 #
 -e file:.#egg=Flintrock
     # via -r requirements/developer.pip
-altgraph==0.17.2
+altgraph==0.17.4
     # via
     #   macholib
     #   pyinstaller
-attrs==22.1.0
-    # via
-    #   -r requirements/developer.pip
-    #   pytest
-bcrypt==3.2.2
+bcrypt==4.0.1
     # via
     #   -r requirements/developer.pip
     #   paramiko
-bleach==5.0.1
-    # via readme-renderer
-boto3==1.21.44
+boto3==1.29.4
     # via
     #   -r requirements/developer.pip
     #   flintrock
-botocore==1.24.44
+botocore==1.32.4
     # via
     #   -r requirements/developer.pip
     #   boto3
     #   flintrock
     #   s3transfer
-certifi==2022.6.15
+certifi==2023.11.17
     # via requests
-cffi==1.15.1
+cffi==1.16.0
     # via
     #   -r requirements/developer.pip
-    #   bcrypt
     #   cryptography
     #   pynacl
-charset-normalizer==2.1.0
+charset-normalizer==3.3.2
     # via requests
-click==7.1.2
+click==8.1.7
     # via
     #   -r requirements/developer.pip
     #   flintrock
-colorama==0.4.5
-    # via twine
-coverage[toml]==6.4.3
+coverage[toml]==7.3.2
     # via
     #   -r requirements/developer.pip
+    #   coverage
     #   pytest-cov
-cryptography==37.0.4
+cryptography==41.0.5
     # via
     #   -r requirements/developer.pip
     #   flintrock
     #   paramiko
-docutils==0.19
+docutils==0.20.1
     # via readme-renderer
-flake8==3.8.4
-    # via -r requirements/developer.pip
-freezegun==1.1.0
+exceptiongroup==1.2.0
+    # via
+    #   -r requirements/developer.pip
+    #   pytest
+flake8==6.1.0
     # via -r requirements/developer.pip
-idna==3.3
+idna==3.4
     # via requests
-importlib-metadata==4.12.0
+importlib-metadata==6.8.0
     # via
-    #   -r requirements/developer.pip
-    #   flake8
     #   keyring
-    #   pluggy
     #   pyinstaller
-    #   pytest
     #   twine
-iniconfig==1.1.1
+importlib-resources==6.1.1
+    # via keyring
+iniconfig==2.0.0
     # via
     #   -r requirements/developer.pip
     #   pytest
+jaraco-classes==3.3.0
+    # via keyring
 jmespath==1.0.1
     # via
     #   -r requirements/developer.pip
     #   boto3
     #   botocore
-keyring==23.8.2
+keyring==24.3.0
     # via twine
-macholib==1.16
+macholib==1.16.3
     # via pyinstaller
-mccabe==0.6.1
+markdown-it-py==3.0.0
+    # via rich
+mccabe==0.7.0
     # via
     #   -r requirements/developer.pip
     #   flake8
-packaging==21.3
+mdurl==0.1.2
+    # via markdown-it-py
+more-itertools==10.1.0
+    # via jaraco-classes
+nh3==0.2.14
+    # via readme-renderer
+packaging==23.2
     # via
     #   -r requirements/developer.pip
+    #   pyinstaller
     #   pytest
-paramiko==2.11.0
+paramiko==3.3.1
     # via
     #   -r requirements/developer.pip
     #   flintrock
-pkginfo==1.8.3
+pkginfo==1.9.6
     # via twine
-pluggy==1.0.0
+pluggy==1.3.0
     # via
     #   -r requirements/developer.pip
     #   pytest
-py==1.11.0
-    # via
-    #   -r requirements/developer.pip
-    #   pytest
-pycodestyle==2.6.0
+pycodestyle==2.11.1
     # via
     #   -r requirements/developer.pip
     #   flake8
@@ -115,86 +114,77 @@ pycparser==2.21
     # via
     #   -r requirements/developer.pip
     #   cffi
-pyflakes==2.2.0
+pyflakes==3.1.0
     # via
     #   -r requirements/developer.pip
     #   flake8
-pygments==2.12.0
-    # via readme-renderer
-pyinstaller==5.3
+pygments==2.17.1
+    # via
+    #   readme-renderer
+    #   rich
+pyinstaller==6.2.0
     # via -r requirements/maintainer.in
-pyinstaller-hooks-contrib==2022.8
+pyinstaller-hooks-contrib==2023.10
     # via pyinstaller
 pynacl==1.5.0
     # via
     #   -r requirements/developer.pip
     #   paramiko
-pyparsing==3.0.9
-    # via
-    #   -r requirements/developer.pip
-    #   packaging
-pytest==7.1.2
+pytest==7.4.3
     # via
     #   -r requirements/developer.pip
     #   pytest-cov
-pytest-cov==3.0.0
+pytest-cov==4.1.0
     # via -r requirements/developer.pip
 python-dateutil==2.8.2
     # via
     #   -r requirements/developer.pip
     #   botocore
-    #   freezegun
-pyyaml==6.0
+pyyaml==6.0.1
     # via
     #   -r requirements/developer.pip
     #   flintrock
-readme-renderer==36.0
+readme-renderer==42.0
     # via twine
-requests==2.28.1
+requests==2.31.0
     # via
     #   requests-toolbelt
     #   twine
-requests-toolbelt==0.9.1
+requests-toolbelt==1.0.0
     # via twine
 rfc3986==2.0.0
     # via twine
-s3transfer==0.5.2
+rich==13.7.0
+    # via twine
+s3transfer==0.7.0
     # via
     #   -r requirements/developer.pip
     #   boto3
 six==1.16.0
     # via
     #   -r requirements/developer.pip
-    #   bleach
-    #   paramiko
     #   python-dateutil
 tomli==2.0.1
     # via
     #   -r requirements/developer.pip
     #   coverage
     #   pytest
-tqdm==4.64.0
-    # via twine
-twine==3.8.0
+twine==4.0.2
     # via -r requirements/maintainer.in
-typing-extensions==4.3.0
-    # via
-    #   -r requirements/developer.pip
-    #   importlib-metadata
-urllib3==1.26.11
+typing-extensions==4.8.0
+    # via rich
+urllib3==1.26.18
     # via
     #   -r requirements/developer.pip
     #   botocore
     #   requests
     #   twine
-webencodings==0.5.1
-    # via bleach
-wheel==0.37.1
+wheel==0.41.3
     # via -r requirements/maintainer.in
-zipp==3.8.1
+zipp==3.17.0
     # via
-    #   -r requirements/developer.pip
     #   importlib-metadata
+    #   importlib-resources
 
 # The following packages are considered to be unsafe in a requirements file:
 # setuptools
diff --git a/requirements/user.pip b/requirements/user.pip
index b5df9e2..19e2d1f 100644
--- a/requirements/user.pip
+++ b/requirements/user.pip
@@ -1,28 +1,27 @@
 #
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
 #
 #    pip-compile --output-file=requirements/user.pip requirements/user.in
 #
 -e file:.#egg=Flintrock
     # via -r requirements/user.in
-bcrypt==3.2.2
+bcrypt==4.0.1
     # via paramiko
-boto3==1.21.44
+boto3==1.29.4
     # via flintrock
-botocore==1.24.44
+botocore==1.32.4
     # via
     #   boto3
     #   flintrock
     #   s3transfer
-cffi==1.15.1
+cffi==1.16.0
     # via
-    #   bcrypt
     #   cryptography
     #   pynacl
-click==7.1.2
+click==8.1.7
     # via flintrock
-cryptography==37.0.4
+cryptography==41.0.5
     # via
     #   flintrock
     #   paramiko
@@ -30,7 +29,7 @@ jmespath==1.0.1
     # via
     #   boto3
     #   botocore
-paramiko==2.11.0
+paramiko==3.3.1
     # via flintrock
 pycparser==2.21
     # via cffi
@@ -38,13 +37,11 @@ pynacl==1.5.0
     # via paramiko
 python-dateutil==2.8.2
     # via botocore
-pyyaml==6.0
+pyyaml==6.0.1
     # via flintrock
-s3transfer==0.5.2
+s3transfer==0.7.0
     # via boto3
 six==1.16.0
-    # via
-    #   paramiko
-    #   python-dateutil
-urllib3==1.26.11
+    # via python-dateutil
+urllib3==1.26.18
     # via botocore
diff --git a/setup.py b/setup.py
index 2d3f3d2..7e2ee6a 100644
--- a/setup.py
+++ b/setup.py
@@ -16,7 +16,7 @@
     author='Nicholas Chammas',
     author_email='nicholas.chammas@gmail.com',
     license='Apache License 2.0',
-    python_requires='>= 3.7',
+    python_requires='>= 3.8',
 
     # See: https://pypi.python.org/pypi?%3Aaction=list_classifiers
     classifiers=[
@@ -47,11 +47,11 @@
     # totally break Flintrock.
     # For example: https://github.com/paramiko/paramiko/issues/615
     install_requires=[
-        'boto3 == 1.21.44',
-        'botocore == 1.24.44',
-        'click == 7.1.2',
-        'paramiko == 2.11.0',
-        'PyYAML == 6.0',
+        'boto3 == 1.29.4',
+        'botocore == 1.32.4',
+        'click == 8.1.7',
+        'paramiko == 3.3.1',
+        'PyYAML == 6.0.1',
         # This is to address reports that PyInstaller-packaged versions
         # of Flintrock intermittently fail due to an out-of-date version
         # of Cryptography being used.


Test patch:
diff --git a/tests/README.md b/tests/README.md
index 6ece937..b740858 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -6,7 +6,7 @@ The instructions here assume the following things:
 
 1. You've read through our [guide on contributing code](../CONTRIBUTING.md#contributing-code) and installed Flintrock's development dependencies.
 2. You're working from Flintrock's root directory.
-3. You're running Python 3.7+.
+3. You're running Python 3.8+.
 4. You've already setup your Flintrock config file and can launch clusters.
 
 To run all of Flintrock's tests that don't require AWS credentials, just run:
diff --git a/tests/conftest.py b/tests/conftest.py
index a105728..669dc9f 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -10,9 +10,9 @@
 # External
 import pytest
 
-HADOOP_VERSION = '3.3.4'
-SPARK_VERSION = '3.3.0'
-SPARK_GIT_COMMIT = 'de351e30a90dd988b133b3d00fa6218bfcaba8b8'  # 3.1.2
+HADOOP_VERSION = '3.3.6'
+SPARK_VERSION = '3.5.0'
+SPARK_GIT_COMMIT = 'ce5ddad990373636e94071e7cef2f31021add07b'  # 3.5.0
 JAVA_VERSION = '11'
 
 
diff --git a/tests/test_core.py b/tests/test_core.py
index 5379c62..69cf8da 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -16,7 +16,7 @@
 @pytest.mark.parametrize(
     'spark_version', [
         (''),
-        ('3.3.0'),
+        ('3.5.0'),
         ('a28880f3b9c63d86368bcd6cbbaa6a9af7075409'),
     ])
 def test_templates(dummy_cluster, spark_version):
diff --git a/tests/test_ec2.py b/tests/test_ec2.py
index fd60eb7..39ed57a 100644
--- a/tests/test_ec2.py
+++ b/tests/test_ec2.py
@@ -19,7 +19,7 @@ def test_validate_tags():
 
     for test_case in positive_test_cases:
         ec2_tags = validate_tags(test_case[0])
-        assert(isinstance(ec2_tags, list))
+        assert isinstance(ec2_tags, list)
         for i, ec2_tag in enumerate(ec2_tags):
             expected_dict = test_case[1][i]
             for k in expected_dict:
diff --git a/tests/test_flintrock.py b/tests/test_flintrock.py
index ff6b7c1..4ee5fc2 100644
--- a/tests/test_flintrock.py
+++ b/tests/test_flintrock.py
@@ -15,6 +15,7 @@
     mutually_exclusive,
     get_latest_commit,
     validate_download_source,
+    normalize_keys,
 )
 
 
@@ -157,10 +158,30 @@ def test_get_latest_commit():
     raises=Error,
 )
 def test_validate_valid_download_source():
-    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz")
-    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz")
+    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz")
+    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz")
 
 
 def test_validate_invalid_download_source():
     with pytest.raises(Error):
         validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-invalid-version/hadoop-invalid-version.tar.gz")
+
+
+def test_normalize_keys():
+    config_file_settings = {
+        "java-version": 11,
+        "ec2": {
+            "spot-price": 0.05,
+            "key-name": "key.pem",
+        },
+        "tags": ["name, test-cluster"],
+    }
+    cli_settings = {
+        "java_version": 11,
+        "ec2": {
+            "spot_price": 0.05,
+            "key_name": "key.pem",
+        },
+        "tags": ["name, test-cluster"],
+    }
+    assert normalize_keys(config_file_settings) == cli_settings