Repository: nchammas/flintrock (python)

Issue:
Allow users to set the git commit to install Spark from as 'latest'
Oftentimes, I imagine users wanting to install Spark from git may just care about getting the latest version on master to test a bug fix or new feature.

In that case, it's a hassle to have to look up the latest commit when Flintrock can just do it for you. Set the desired commit to `latest` and Flintrock will lookup the latest commit automatically.

There are 3 approaches we can take to implement a convenience feature like this:

1. Import some Python library for interacting with git and use that to query the latest version. Advantage: No implicit user dependencies. Disadvantage: It's another Flintrock dependency we have to vet.
2. Use `git` at the command line as such:

  ```
git ls-remote https://github.com/apache/spark master
  ```
  Advantage: Very simple. Disadvantage: We've created an implicit dependency on `git` being installed on the machine running Flintrock.
3. Query the GitHub API. Advantage: Still simple. Disadvantage: Now we have a dependency on GitHub for something that is strictly git functionality. This may not be a big deal since Spark itself, and likely most forks of it, are hosted on GitHub.

Depends on #25.

Pull request description:
Allow users to set the git commit to install Spark from as 'latest'
Made some minor tweaks to @BenFradet's work.

Supersedes / Fixes #98.
Fixes #61.

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index c7d2429..5a451a7 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -3,10 +3,14 @@
 
 ## [Unreleased](https://github.com/nchammas/flintrock/compare/v0.3.0...master)
 
+### Added
+
+* [#98](https://github.com/nchammas/flintrock/pull/98)/[#99](https://github.com/nchammas/flintrock/pull/99): You can now specify `latest` for `--spark-git-commit` and Flintrock will automatically build Spark on your cluster at the latest commit. This feature is only available for repos hosted on GitHub.
+
 ### Changed
 
 * [#86](https://github.com/nchammas/flintrock/pull/86): Flintrock now correctly catches when spot requests fail and bubbles up an appropriate error message.
-* [#97](https://github.com/nchammas/flintrock/pull/97): Fixed the ability to build Spark from git. (It was broken for recent commits.)
+* [#93](https://github.com/nchammas/flintrock/pull/93)/[#97](https://github.com/nchammas/flintrock/pull/97): Fixed the ability to build Spark from git. (It was broken for recent commits.)
 
 
 ## [0.3.0](https://github.com/nchammas/flintrock/compare/v0.2.0...v0.3.0) - 2016-02-14
diff --git a/flintrock/config.yaml.template b/flintrock/config.yaml.template
index ae824a6..4dcba25 100644
--- a/flintrock/config.yaml.template
+++ b/flintrock/config.yaml.template
@@ -1,8 +1,8 @@
 services:
   spark:
     version: 1.6.0
-    # git-commit:  # mutually exclusive with version
-    # git-repository:  # optional; defaults to https://github.com/apache/spark.git
+    # git-commit: latest  # if not 'latest', provide a full commit SHA; e.g. d6dc12ef0146ae409834c78737c116050961f350
+    # git-repository:  # optional; defaults to https://github.com/apache/spark
   hdfs:
     version: 2.7.1
 
diff --git a/flintrock/flintrock.py b/flintrock/flintrock.py
index 15ed5d5..40ad812 100644
--- a/flintrock/flintrock.py
+++ b/flintrock/flintrock.py
@@ -1,10 +1,13 @@
 import os
 import posixpath
 import errno
+import json
 import resource
 import sys
 import shutil
 import textwrap
+import urllib.parse
+import urllib.request
 import warnings
 
 # External modules
@@ -81,8 +84,9 @@ def option_requires(
                 required_name = option_name_to_variable_name(required_option)
                 if required_name not in scope or scope[required_name] is None:
                     raise UsageError(
-                        'Error: Missing option "{missing_option}" is required by '
-                        '"{option}{space}{conditional_value}".'.format(
+                        "Error: Missing option \"{missing_option}\" is required by "
+                        "\"{option}{space}{conditional_value}\"."
+                        .format(
                             missing_option=required_option,
                             option=option,
                             space=' ' if conditional_value is not None else '',
@@ -94,8 +98,9 @@ def option_requires(
                     break
             else:
                 raise UsageError(
-                    'Error: "{option}{space}{conditional_value}" requires at least '
-                    'one of the following options to be set: {at_least}'.format(
+                    "Error: \"{option}{space}{conditional_value}\" requires at least "
+                    "one of the following options to be set: {at_least}"
+                    .format(
                         option=option,
                         space=' ' if conditional_value is not None else '',
                         conditional_value=conditional_value if conditional_value is not None else '',
@@ -121,9 +126,10 @@ def mutually_exclusive(*, options: list, scope: dict):
         bad_option1 = used_options.pop()
         bad_option2 = used_options.pop()
         raise UsageError(
-            'Error: "{option1}" and "{option2}" are mutually exclusive.\n'
-            '  {option1}: {value1}\n'
-            '  {option2}: {value2}'.format(
+            "Error: \"{option1}\" and \"{option2}\" are mutually exclusive.\n"
+            "  {option1}: {value1}\n"
+            "  {option2}: {value2}"
+            .format(
                 option1=variable_name_to_option_name(bad_option1),
                 value1=scope[bad_option1],
                 option2=variable_name_to_option_name(bad_option2),
@@ -172,11 +178,12 @@ def cli(cli_context, config, provider):
 @click.option('--spark-version',
               help="Spark release version to install.")
 @click.option('--spark-git-commit',
-              help="Git commit hash to build Spark from. "
-                   "--spark-version and --spark-git-commit are mutually exclusive.")
+              help="Git commit to build Spark from. "
+                   "Set to 'latest' to build Spark from the latest commit on the "
+                   "repository's default branch.")
 @click.option('--spark-git-repository',
               help="Git repository to clone Spark from.",
-              default='https://github.com/apache/spark.git',
+              default='https://github.com/apache/spark',
               show_default=True)
 @click.option('--assume-yes/--no-assume-yes', default=False)
 @click.option('--ec2-key-name')
@@ -266,10 +273,15 @@ def launch(
         if spark_version:
             spark = Spark(version=spark_version)
         elif spark_git_commit:
-            print("Warning: Building Spark takes a long time. "
-                  "e.g. 15-20 minutes on an m3.xlarge instance on EC2.")
-            spark = Spark(git_commit=spark_git_commit,
-                          git_repository=spark_git_repository)
+            print(
+                "Warning: Building Spark takes a long time. "
+                "e.g. 15-20 minutes on an m3.xlarge instance on EC2.")
+            if spark_git_commit == 'latest':
+                spark_git_commit = get_latest_commit(spark_git_repository)
+                print("Building Spark at latest commit: {c}".format(c=spark_git_commit))
+            spark = Spark(
+                git_commit=spark_git_commit,
+                git_repository=spark_git_repository)
         services += [spark]
 
     if provider == 'ec2':
@@ -297,6 +309,30 @@ def launch(
         raise UnsupportedProviderError(provider)
 
 
+def get_latest_commit(github_repository: str):
+    """
+    Get the latest commit on the default branch of a repository hosted on GitHub.
+    """
+    parsed_url = urllib.parse.urlparse(github_repository)
+    repo_domain, repo_path = parsed_url.netloc, parsed_url.path.strip('/')
+
+    if repo_domain != 'github.com':
+        raise UsageError(
+            "Error: Getting the latest commit is only supported "
+            "for repositories hosted on GitHub. "
+            "Provided repository domain was: {d}".format(d=repo_domain))
+
+    url = "https://api.github.com/repos/{rp}/commits".format(rp=repo_path)
+    try:
+        with urllib.request.urlopen(url) as response:
+            result = json.loads(response.read().decode('utf-8'))
+            return result[0]['sha']
+    except Exception as e:
+        raise Exception(
+            "Could not get latest commit for repository: {r}"
+            .format(r=repo_path)) from e
+
+
 @cli.command()
 @click.argument('cluster-name')
 @click.option('--assume-yes/--no-assume-yes', default=False)


Test patch:
diff --git a/tests/test_flintrock.py b/tests/test_flintrock.py
index 09db5c5..32e2110 100644
--- a/tests/test_flintrock.py
+++ b/tests/test_flintrock.py
@@ -1,5 +1,6 @@
 # External modules
 import pytest
+import urllib.error
 
 # Flintrock modules
 from flintrock.exceptions import (
@@ -9,7 +10,8 @@ from flintrock.flintrock import (
     option_name_to_variable_name,
     variable_name_to_option_name,
     option_requires,
-    mutually_exclusive
+    mutually_exclusive,
+    get_latest_commit
 )
 
 
@@ -126,3 +128,14 @@ def test_mutually_exclusive():
                 '--option1',
                 '--option2'],
             scope=locals())
+
+
+def test_get_latest_commit():
+    sha = get_latest_commit("https://github.com/apache/spark")
+    assert len(sha) == 40
+
+    with pytest.raises(UsageError):
+        get_latest_commit("https://google.com")
+
+    with pytest.raises(Exception):
+        get_latest_commit("https://github.com/apache/nonexistent-repo")