Repository: nchammas/flintrock (language: python)

Issue:
New Release
Hi @nchammas, would it be possible to cut a new Flintrock release with the existing changes? I'm interested in the private VPC changes that were merged in recently.


Discussion / hints:
Thanks for the gentle bump. Will get a new release out this week. Been meaning to do it for a month or so now. 😄

Fix patch:
diff --git a/.gitignore b/.gitignore
index 8b5df0d6..18d44819 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 # Started from: https://github.com/github/gitignore/blob/master/Python.gitignore
 
 .vscode/
+.metals/
 config.yaml
 venv/
 *example.py
diff --git a/CHANGES.md b/CHANGES.md
index e6910ff7..1666b875 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -2,25 +2,44 @@
 
 ## [Unreleased]
 
-[Unreleased]: https://github.com/nchammas/flintrock/compare/v1.0.0...master
+[Unreleased]: https://github.com/nchammas/flintrock/compare/v2.0.0...master
+
+Nothing notable yet.
+
+## [2.0.0] - 2021-06-10
+
+[2.0.0]: https://github.com/nchammas/flintrock/compare/v1.0.0...v2.0.0
 
 ### Added
 
-* [#296]: Added support for launching clusters into private VPCs.
-* [#324]: Flintrock now supports S3 URLs as a download source for Hadoop or Spark. This makes it easy to host your own copies of the Hadoop and Spark release builds in a private bucket.
+* [#296]: Added support for launching clusters into private VPCs. This includes new infrastructure added in [#302] to support testing against private VPCs.
+* [#307]: Added support for Hadoop/HDFS 3.x.
+* [#315]: Added a new `--ec2-spot-request-duration` option to support setting the EC2 spot request duration.
+* [#316]: Added a new `--java-version` option and support for Java 11.
 * [#323]: Flintrock now automatically selects the correct build of Spark to use, based on the version of Hadoop/HDFS that you specify.
+* [#324]: Flintrock now supports S3 URLs as a download source for Hadoop or Spark. This makes it easy to host your own copies of the Hadoop and Spark release builds in a private bucket.
 
 [#296]: https://github.com/nchammas/flintrock/pull/296
-[#324]: https://github.com/nchammas/flintrock/pull/324
+[#302]: https://github.com/nchammas/flintrock/pull/302
+[#307]: https://github.com/nchammas/flintrock/pull/307
+[#315]: https://github.com/nchammas/flintrock/pull/315
+[#316]: https://github.com/nchammas/flintrock/pull/316
 [#323]: https://github.com/nchammas/flintrock/pull/323
+[#324]: https://github.com/nchammas/flintrock/pull/324
 
 ### Changed
 
+* [#285]: Flintrock now configures cluster nodes to use private IP addresses for internal communication. This should improve the reliability of cluster launches and restarts.
+* [#304]: Fixed a bug in how `UserData` scripts are submitted to new cluster slaves.
 * [#311]: Changed how Flintrock manages its own security groups to reduce the likelihood of hitting any limits on the number of rules per security group.
+* [#326]: Switched some internals from using host names to IP addresses, which should improve Flintrock's behavior when running from an EC2 host.
 * [#329]: Dropped support for Python 3.5 and added automated testing for Python 3.8 and 3.9.
 * [#334]: Flintrock now ensures that `python3` is available on launched clusters and sets that as the default Python that PySpark will use.
 
+[#285]: https://github.com/nchammas/flintrock/pull/285
+[#304]: https://github.com/nchammas/flintrock/pull/304
 [#311]: https://github.com/nchammas/flintrock/pull/311
+[#326]: https://github.com/nchammas/flintrock/pull/326
 [#329]: https://github.com/nchammas/flintrock/pull/329
 [#334]: https://github.com/nchammas/flintrock/pull/334
 
diff --git a/README.md b/README.md
index 329e106a..84c691b7 100644
--- a/README.md
+++ b/README.md
@@ -30,10 +30,10 @@ Here's a quick way to launch a cluster on EC2, assuming you already have an [AWS
 ```sh
 flintrock launch test-cluster \
     --num-slaves 1 \
-    --spark-version 3.0.1 \
+    --spark-version 3.1.2 \
     --ec2-key-name key_name \
     --ec2-identity-file /path/to/key.pem \
-    --ec2-ami ami-0beafb294c86717a8 \
+    --ec2-ami ami-0aeeebd8d2ab47354 \
     --ec2-user ec2-user
 ```
 
@@ -150,7 +150,7 @@ unzip it to a location of your choice, and run the `flintrock` executable inside
 For example:
 
 ```sh
-flintrock_version="1.0.0"
+flintrock_version="2.0.0"
 
 curl --location --remote-name "https://github.com/nchammas/flintrock/releases/download/v$flintrock_version/Flintrock-$flintrock_version-standalone-macOS-x86_64.zip"
 unzip -q -d flintrock "Flintrock-$flintrock_version-standalone-macOS-x86_64.zip"
@@ -252,7 +252,7 @@ provider: ec2
 
 services:
   spark:
-    version: 3.0.1
+    version: 3.1.2
 
 launch:
   num-slaves: 1
@@ -263,7 +263,7 @@ providers:
     identity-file: /path/to/.ssh/key.pem
     instance-type: m5.large
     region: us-east-1
-    ami: ami-0beafb294c86717a8
+    ami: ami-0aeeebd8d2ab47354
     user: ec2-user
 ```
 
diff --git a/flintrock/__init__.py b/flintrock/__init__.py
index 5eda4167..1f8df058 100644
--- a/flintrock/__init__.py
+++ b/flintrock/__init__.py
@@ -1,2 +1,2 @@
 # See: https://packaging.python.org/en/latest/distributing/#standards-compliance-for-interoperability
-__version__ = '2.0.0.dev0'
+__version__ = '2.0.0'
diff --git a/flintrock/config.yaml.template b/flintrock/config.yaml.template
index dc07a32e..4399bd08 100644
--- a/flintrock/config.yaml.template
+++ b/flintrock/config.yaml.template
@@ -1,6 +1,6 @@
 services:
   spark:
-    version: 3.0.1
+    version: 3.1.2
     # git-commit: latest  # if not 'latest', provide a full commit SHA; e.g. d6dc12ef0146ae409834c78737c116050961f350
     # git-repository:  # optional; defaults to https://github.com/apache/spark
     # optional; defaults to download from a dynamically selected Apache mirror
@@ -30,7 +30,7 @@ providers:
     instance-type: m5.large
     region: us-east-1
     # availability-zone: <name>
-    ami: ami-0beafb294c86717a8  # Amazon Linux 2, us-east-1
+    ami: ami-0aeeebd8d2ab47354  # Amazon Linux 2, us-east-1
     user: ec2-user
     # ami: ami-61bbf104  # CentOS 7, us-east-1
     # user: centos
diff --git a/requirements/developer.pip b/requirements/developer.pip
index 2b9cb9b5..3061f1bc 100644
--- a/requirements/developer.pip
+++ b/requirements/developer.pip
@@ -6,23 +6,23 @@
 #
 -e file:.#egg=Flintrock
     # via -r requirements/user.pip
-attrs==20.3.0
+attrs==21.2.0
     # via pytest
 bcrypt==3.2.0
     # via
     #   -r requirements/user.pip
     #   paramiko
-boto3==1.10.45
+boto3==1.17.90
     # via
     #   -r requirements/user.pip
     #   flintrock
-botocore==1.13.45
+botocore==1.20.90
     # via
     #   -r requirements/user.pip
     #   boto3
     #   flintrock
     #   s3transfer
-cffi==1.14.4
+cffi==1.14.5
     # via
     #   -r requirements/user.pip
     #   bcrypt
@@ -32,22 +32,18 @@ click==7.0
     # via
     #   -r requirements/user.pip
     #   flintrock
-coverage==5.4
+coverage==5.5
     # via pytest-cov
-cryptography==3.4.2
+cryptography==3.4.7
     # via
     #   -r requirements/user.pip
     #   flintrock
     #   paramiko
-docutils==0.15.2
-    # via
-    #   -r requirements/user.pip
-    #   botocore
 flake8==3.8.4
     # via -r requirements/developer.in
 freezegun==1.1.0
     # via -r requirements/developer.in
-importlib-metadata==3.4.0
+importlib-metadata==4.5.0
     # via
     #   flake8
     #   pluggy
@@ -85,9 +81,9 @@ pynacl==1.4.0
     #   paramiko
 pyparsing==2.4.7
     # via packaging
-pytest-cov==2.11.1
+pytest-cov==2.12.1
     # via -r requirements/developer.in
-pytest==6.2.2
+pytest==6.2.4
     # via
     #   -r requirements/developer.in
     #   pytest-cov
@@ -96,27 +92,29 @@ python-dateutil==2.8.1
     #   -r requirements/user.pip
     #   botocore
     #   freezegun
-pyyaml==5.2
+pyyaml==5.4.1
     # via
     #   -r requirements/user.pip
     #   flintrock
-s3transfer==0.2.1
+s3transfer==0.4.2
     # via
     #   -r requirements/user.pip
     #   boto3
-six==1.15.0
+six==1.16.0
     # via
     #   -r requirements/user.pip
     #   bcrypt
     #   pynacl
     #   python-dateutil
 toml==0.10.2
-    # via pytest
-typing-extensions==3.7.4.3
+    # via
+    #   pytest
+    #   pytest-cov
+typing-extensions==3.10.0.0
     # via importlib-metadata
-urllib3==1.25.11
+urllib3==1.26.5
     # via
     #   -r requirements/user.pip
     #   botocore
-zipp==3.4.0
+zipp==3.4.1
     # via importlib-metadata
diff --git a/requirements/maintainer.in b/requirements/maintainer.in
index 11cb30fb..e13e39c6 100644
--- a/requirements/maintainer.in
+++ b/requirements/maintainer.in
@@ -1,4 +1,4 @@
 -r developer.pip
 wheel >= 0.31.0
 twine == 1.15.0
-PyInstaller == 4.2
+PyInstaller == 4.3
diff --git a/requirements/maintainer.pip b/requirements/maintainer.pip
index d12d2f00..7008e5ca 100644
--- a/requirements/maintainer.pip
+++ b/requirements/maintainer.pip
@@ -10,7 +10,7 @@ altgraph==0.17
     # via
     #   macholib
     #   pyinstaller
-attrs==20.3.0
+attrs==21.2.0
     # via
     #   -r requirements/developer.pip
     #   pytest
@@ -20,19 +20,19 @@ bcrypt==3.2.0
     #   paramiko
 bleach==3.3.0
     # via readme-renderer
-boto3==1.10.45
+boto3==1.17.90
     # via
     #   -r requirements/developer.pip
     #   flintrock
-botocore==1.13.45
+botocore==1.20.90
     # via
     #   -r requirements/developer.pip
     #   boto3
     #   flintrock
     #   s3transfer
-certifi==2020.12.5
+certifi==2021.5.30
     # via requests
-cffi==1.14.4
+cffi==1.14.5
     # via
     #   -r requirements/developer.pip
     #   bcrypt
@@ -44,27 +44,24 @@ click==7.0
     # via
     #   -r requirements/developer.pip
     #   flintrock
-coverage==5.4
+coverage==5.5
     # via
     #   -r requirements/developer.pip
     #   pytest-cov
-cryptography==3.4.2
+cryptography==3.4.7
     # via
     #   -r requirements/developer.pip
     #   flintrock
     #   paramiko
-docutils==0.15.2
-    # via
-    #   -r requirements/developer.pip
-    #   botocore
-    #   readme-renderer
+docutils==0.17.1
+    # via readme-renderer
 flake8==3.8.4
     # via -r requirements/developer.pip
 freezegun==1.1.0
     # via -r requirements/developer.pip
 idna==2.10
     # via requests
-importlib-metadata==3.4.0
+importlib-metadata==4.5.0
     # via
     #   -r requirements/developer.pip
     #   flake8
@@ -117,11 +114,11 @@ pyflakes==2.2.0
     # via
     #   -r requirements/developer.pip
     #   flake8
-pygments==2.7.4
+pygments==2.9.0
     # via readme-renderer
-pyinstaller-hooks-contrib==2020.11
+pyinstaller-hooks-contrib==2021.1
     # via pyinstaller
-pyinstaller==4.2
+pyinstaller==4.3
     # via -r requirements/maintainer.in
 pynacl==1.4.0
     # via
@@ -131,9 +128,9 @@ pyparsing==2.4.7
     # via
     #   -r requirements/developer.pip
     #   packaging
-pytest-cov==2.11.1
+pytest-cov==2.12.1
     # via -r requirements/developer.pip
-pytest==6.2.2
+pytest==6.2.4
     # via
     #   -r requirements/developer.pip
     #   pytest-cov
@@ -142,11 +139,11 @@ python-dateutil==2.8.1
     #   -r requirements/developer.pip
     #   botocore
     #   freezegun
-pyyaml==5.2
+pyyaml==5.4.1
     # via
     #   -r requirements/developer.pip
     #   flintrock
-readme-renderer==28.0
+readme-renderer==29.0
     # via twine
 requests-toolbelt==0.9.1
     # via twine
@@ -154,11 +151,11 @@ requests==2.25.1
     # via
     #   requests-toolbelt
     #   twine
-s3transfer==0.2.1
+s3transfer==0.4.2
     # via
     #   -r requirements/developer.pip
     #   boto3
-six==1.15.0
+six==1.16.0
     # via
     #   -r requirements/developer.pip
     #   bcrypt
@@ -170,15 +167,16 @@ toml==0.10.2
     # via
     #   -r requirements/developer.pip
     #   pytest
-tqdm==4.56.0
+    #   pytest-cov
+tqdm==4.61.0
     # via twine
 twine==1.15.0
     # via -r requirements/maintainer.in
-typing-extensions==3.7.4.3
+typing-extensions==3.10.0.0
     # via
     #   -r requirements/developer.pip
     #   importlib-metadata
-urllib3==1.25.11
+urllib3==1.26.5
     # via
     #   -r requirements/developer.pip
     #   botocore
@@ -187,7 +185,7 @@ webencodings==0.5.1
     # via bleach
 wheel==0.36.2
     # via -r requirements/maintainer.in
-zipp==3.4.0
+zipp==3.4.1
     # via
     #   -r requirements/developer.pip
     #   importlib-metadata
diff --git a/requirements/user.pip b/requirements/user.pip
index 396a1b07..0f9f7b60 100644
--- a/requirements/user.pip
+++ b/requirements/user.pip
@@ -8,26 +8,24 @@
     # via -r requirements/user.in
 bcrypt==3.2.0
     # via paramiko
-boto3==1.10.45
+boto3==1.17.90
     # via flintrock
-botocore==1.13.45
+botocore==1.20.90
     # via
     #   boto3
     #   flintrock
     #   s3transfer
-cffi==1.14.4
+cffi==1.14.5
     # via
     #   bcrypt
     #   cryptography
     #   pynacl
 click==7.0
     # via flintrock
-cryptography==3.4.2
+cryptography==3.4.7
     # via
     #   flintrock
     #   paramiko
-docutils==0.15.2
-    # via botocore
 jmespath==0.10.0
     # via
     #   boto3
@@ -40,14 +38,14 @@ pynacl==1.4.0
     # via paramiko
 python-dateutil==2.8.1
     # via botocore
-pyyaml==5.2
+pyyaml==5.4.1
     # via flintrock
-s3transfer==0.2.1
+s3transfer==0.4.2
     # via boto3
-six==1.15.0
+six==1.16.0
     # via
     #   bcrypt
     #   pynacl
     #   python-dateutil
-urllib3==1.25.11
+urllib3==1.26.5
     # via botocore
diff --git a/setup.py b/setup.py
index fec22867..2271f84c 100644
--- a/setup.py
+++ b/setup.py
@@ -47,11 +47,11 @@
     # totally break Flintrock.
     # For example: https://github.com/paramiko/paramiko/issues/615
     install_requires=[
-        'boto3 == 1.10.45',
-        'botocore == 1.13.45',
+        'boto3 == 1.17.90',
+        'botocore == 1.20.90',
         'click == 7.0',
         'paramiko == 2.7.2',
-        'PyYAML == 5.2',
+        'PyYAML == 5.4.1',
         # This is to address reports that PyInstaller-packaged versions
         # of Flintrock intermittently fail due to an out-of-date version
         # of Cryptography being used.


Test patch:
diff --git a/test-infra/provider.tf b/test-infra/provider.tf
index f3956526..0f011851 100644
--- a/test-infra/provider.tf
+++ b/test-infra/provider.tf
@@ -1,8 +1,16 @@
-provider "aws" {
-  version = "~> 2"
-  region  = "us-east-1"
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 2"
+    }
+    http = {
+      source  = "hashicorp/http"
+      version = "~> 1"
+    }
+  }
 }
 
-provider "http" {
-  version = "~> 1"
+provider "aws" {
+  region = "us-east-1"
 }
diff --git a/tests/conftest.py b/tests/conftest.py
index 1f558914..d0dd0826 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -11,8 +11,8 @@
 import pytest
 
 HADOOP_VERSION = '3.3.0'
-SPARK_VERSION = '3.0.1'
-SPARK_GIT_COMMIT = '2b147c4cd50da32fe2b4167f97c8142102a0510d'  # 3.0.1
+SPARK_VERSION = '3.1.2'
+SPARK_GIT_COMMIT = 'de351e30a90dd988b133b3d00fa6218bfcaba8b8'  # 3.1.2
 JAVA_VERSION = '11'
 
 
diff --git a/tests/test_core.py b/tests/test_core.py
index cf3c4093..7b2a681b 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -16,7 +16,7 @@
 @pytest.mark.parametrize(
     'spark_version', [
         (''),
-        ('3.0.1'),
+        ('3.1.2'),
         ('0626b11147133b67b26a04b4819f61a33dd958d3'),
     ])
 def test_templates(dummy_cluster, spark_version):
diff --git a/tests/test_flintrock.py b/tests/test_flintrock.py
index bcdef830..8179bd0f 100644
--- a/tests/test_flintrock.py
+++ b/tests/test_flintrock.py
@@ -158,7 +158,7 @@ def test_get_latest_commit():
 )
 def test_validate_valid_download_source():
     validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz")
-    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz")
+    validate_download_source("https://www.apache.org/dyn/closer.lua?action=download&filename=spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz")
 
 
 def test_validate_invalid_download_source():