Repository: great-expectations/great_expectations

Issue:
Docs Build --site-name bug
**Describe the bug**
If you have a local site and S3 site configured in the yml and run the command `great_expectations docs build --site-name local_site`, great expectations still tries to verify the S3 credentials and run stuff related to the S3 site in the background.

This is an issue because a user testing locally should not need access to the production S3 credentials (or any AWS credentials).

**Temporary Workaround**
If you add `--no-view` to the end of the command (so the command becomes `great_expectations docs build --site-name local_site --no-view`) you can circumvent this error and achieve the expected behavior. This isn't exactly obvious though to the end user without being told.

**To Reproduce**
Steps to reproduce the behavior:
1. Put something like the below in the grea_expectations.yml file:
```
data_docs_sites:
  # Data Docs make it simple to visualize data quality in your project. These
  # include Expectations, Validations & Profiles. The are built for all
  # Datasources from JSON artifacts in the local repo including validations &
  # profiles from the uncommitted directory. Read more at https://docs.greatexpectations.io/en/latest/features/data_docs.html
  local_site:
    class_name: SiteBuilder
    # set to false to hide how-to buttons in Data Docs
    # show_how_to_buttons: false
    store_backend:
      class_name: TupleFilesystemStoreBackend
      base_directory: uncommitted/data_docs/local_site/
    site_index_builder:
      class_name: DefaultSiteIndexBuilder
  s3_site:
    class_name: SiteBuilder
    store_backend:
      class_name: TupleS3StoreBackend
      bucket: bucket_name
      prefix: sub_directory
```
2. Run `great_expectations docs build --site-name local_site`
3. See error
```
Running great_expectations docs build --site-name local_site
Building Data Docs...
The following Data Docs sites were built:
 - local_site: file:///great_expectations/uncommitted/data_docs/local_site/index.html
Traceback (most recent call last):
  File "/usr/local/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/great_expectations/cli/cli.py", line 67, in main
    cli()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/cli/docs.py", line 40, in docs_build
    build_docs(context, site_name=site_name, view=view)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/cli/docs.py", line 147, in build_docs
    context.open_data_docs()
  File "/usr/local/lib/python3.7/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 215, in usage_statistics_wrapped_method
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 428, in open_data_docs
    data_docs_urls = self.get_docs_sites_urls(resource_identifier=resource_identifier)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 411, in get_docs_sites_urls
    url = site_builder.get_resource_url(resource_identifier=resource_identifier)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/render/renderer/site_builder.py", line 253, in get_resource_url
    return self.target_store.get_url_for_resource(resource_identifier=resource_identifier)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/store/html_site_store.py", line 168, in get_url_for_resource
    return store_backend.get_url_for_key(key)
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/store/tuple_store_backend.py", line 380, in get_url_for_key
    location = boto3.client('s3').get_bucket_location(Bucket=self.bucket)['LocationConstraint']
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the GetBucketLocation operation: The AWS Access Key Id you provided does not exist in our records.
```

**Expected behavior**
I expect that by specifying `local_site`, only the local site will be updated and surfaced. Great_expectations will not try to validate/open/update the S3 site.

**Environment (please complete the following information):**
 - python:3.7.6-slim base docker image
 - GE Version: 0.10.6

**Additional context**
I discussed this via slack with @jcampbell and he recommended opening an issue here.



Discussion:
Thanks @ddaltonrtr for this excellent and detailed write up. I plan on shipping a fix for this by mid week!

Resolving patch:
diff --git a/great_expectations/cli/docs.py b/great_expectations/cli/docs.py
--- a/great_expectations/cli/docs.py
+++ b/great_expectations/cli/docs.py
@@ -144,4 +144,4 @@ def build_docs(context, site_name=None, view=True):
     cli_message(msg)
 
     if view:
-        context.open_data_docs()
+        context.open_data_docs(site_name=site_name)
diff --git a/great_expectations/data_context/data_context.py b/great_expectations/data_context/data_context.py
--- a/great_expectations/data_context/data_context.py
+++ b/great_expectations/data_context/data_context.py
@@ -11,7 +11,7 @@
 import uuid
 import warnings
 import webbrowser
-from typing import Union, List
+from typing import Union, List, Dict, Optional
 
 from marshmallow import ValidationError
 from ruamel.yaml import YAML, YAMLError
@@ -54,6 +54,7 @@
     load_class,
     substitute_all_config_variables,
 )
+from great_expectations.render.renderer.site_builder import SiteBuilder
 from great_expectations.validator.validator import Validator
 from great_expectations.dataset import Dataset
 from great_expectations.datasource import Datasource
@@ -366,67 +367,90 @@ def _normalize_store_path(self, resource_store):
                 resource_store["base_directory"] = os.path.join(self.root_directory, resource_store["base_directory"])
         return resource_store
 
-    def get_docs_sites_urls(self, resource_identifier=None):
+    def get_docs_sites_urls(
+        self, resource_identifier=None, site_name: Optional[str] = None
+    ) -> List[Dict[str, str]]:
         """
         Get URLs for a resource for all data docs sites.
 
-        This function will return URLs for any configured site even if the sites have not
-        been built yet.
+        This function will return URLs for any configured site even if the sites
+        have not been built yet.
 
-        :param resource_identifier: optional. It can be an identifier of ExpectationSuite's,
-                ValidationResults and other resources that have typed identifiers.
-                If not provided, the method will return the URLs of the index page.
-        :return: a list of URLs. Each item is the URL for the resource for a data docs site
-        """
-
-        site_urls = []
+        Args:
+            resource_identifier (object): optional. It can be an identifier of
+                ExpectationSuite's, ValidationResults and other resources that
+                have typed identifiers. If not provided, the method will return
+                the URLs of the index page.
+            site_name: Optionally specify which site to open. If not specified,
+                return all urls in the project.
 
-        site_names = None
+        Returns:
+            list: a list of URLs. Each item is the URL for the resource for a
+                data docs site
+        """
         sites = self._project_config_with_variables_substituted.data_docs_sites
-        if sites:
-            logger.debug("Found data_docs_sites.")
+        if not sites:
+            logger.debug("Found no data_docs_sites.")
+            return []
+        logger.debug(f"Found {len(sites)} data_docs_sites.")
 
-            for site_name, site_config in sites.items():
-                if (site_names and site_name in site_names) or not site_names:
-                    complete_site_config = site_config
-                    module_name = 'great_expectations.render.renderer.site_builder'
-                    site_builder = instantiate_class_from_config(
-                        config=complete_site_config,
-                        runtime_environment={
-                            "data_context": self,
-                            "root_directory": self.root_directory
-                        },
-                        config_defaults={
-                            "module_name": module_name
-                        }
-                    )
-                    if not site_builder:
-                        raise ge_exceptions.ClassInstantiationError(
-                            module_name=module_name,
-                            package_name=None,
-                            class_name=complete_site_config['class_name']
-                        )
-                    url = site_builder.get_resource_url(resource_identifier=resource_identifier)
-                    site_urls.append({
-                        "site_name": site_name,
-                        "site_url": url
-                    })
+        if site_name:
+            if site_name not in sites.keys():
+                raise ge_exceptions.DataContextError(f"Could not find site named {site_name}. Please check your configurations")
+            site = sites[site_name]
+            site_builder = self._load_site_builder_from_site_config(site)
+            url = site_builder.get_resource_url(resource_identifier=resource_identifier)
+            return [{"site_name": site_name, "site_url": url}]
+
+        site_urls = []
+        for _site_name, site_config in sites.items():
+            site_builder = self._load_site_builder_from_site_config(site_config)
+            url = site_builder.get_resource_url(resource_identifier=resource_identifier)
+            site_urls.append({"site_name": _site_name, "site_url": url})
 
         return site_urls
 
+    def _load_site_builder_from_site_config(self, site_config) -> SiteBuilder:
+        default_module_name = "great_expectations.render.renderer.site_builder"
+        site_builder = instantiate_class_from_config(
+            config=site_config,
+            runtime_environment={
+                "data_context": self,
+                "root_directory": self.root_directory,
+            },
+            config_defaults={"module_name": default_module_name},
+        )
+        if not site_builder:
+            raise ge_exceptions.ClassInstantiationError(
+                module_name=default_module_name,
+                package_name=None,
+                class_name=site_config["class_name"],
+            )
+        return site_builder
+
     @usage_statistics_enabled_method(event_name="data_context.open_data_docs",)
-    def open_data_docs(self, resource_identifier=None):
+    def open_data_docs(
+        self, resource_identifier: Optional[str] = None, site_name: Optional[str] = None
+    ) -> None:
         """
         A stdlib cross-platform way to open a file in a browser.
 
-        :param resource_identifier: ExpectationSuiteIdentifier, ValidationResultIdentifier
-                or any other type's identifier. The argument is optional - when
-                not supplied, the method returns the URL of the index page.
+        Args:
+            resource_identifier: ExpectationSuiteIdentifier,
+                ValidationResultIdentifier or any other type's identifier. The
+                argument is optional - when not supplied, the method returns the
+                URL of the index page.
+            site_name: Optionally specify which site to open. If not specified,
+                open all docs found in the project.
         """
-        data_docs_urls = self.get_docs_sites_urls(resource_identifier=resource_identifier)
-        for site_dict in data_docs_urls:
-            logger.debug("Opening Data Docs found here: {}".format(site_dict["site_url"]))
-            webbrowser.open(site_dict["site_url"])
+        data_docs_urls = self.get_docs_sites_urls(
+            resource_identifier=resource_identifier, site_name=site_name,
+        )
+        urls_to_open = [site["site_url"] for site in data_docs_urls]
+
+        for url in urls_to_open:
+            logger.debug(f"Opening Data Docs found here: {url}")
+            webbrowser.open(url)
 
     @property
     def root_directory(self):


Test patch:
diff --git a/tests/data_context/test_data_context.py b/tests/data_context/test_data_context.py
--- a/tests/data_context/test_data_context.py
+++ b/tests/data_context/test_data_context.py
@@ -26,10 +26,10 @@
 from great_expectations.datasource.types.batch_kwargs import PathBatchKwargs
 from great_expectations.exceptions import (
     BatchKwargsError,
-    ConfigNotFoundError,
-    DataContextError,
     CheckpointError,
     CheckpointNotFoundError,
+    ConfigNotFoundError,
+    DataContextError,
 )
 from great_expectations.util import gen_directory_tree_str
 from tests.integration.usage_statistics.test_integration_usage_statistics import (
@@ -892,7 +892,6 @@ def test_data_context_do_all_uncommitted_dirs_exist(tmp_path_factory):
     uncommitted_dir = os.path.join(ge_dir, "uncommitted")
     DataContext.create(project_path)
     fixture = gen_directory_tree_str(uncommitted_dir)
-    print(fixture)
     assert fixture == expected
 
     # Test that all exist
@@ -939,7 +938,6 @@ def test_data_context_create_does_not_overwrite_existing_config_variables_yml(tm
 
     with open(config_vars_yml, "r") as ff:
         obs = ff.read()
-    print(obs)
     assert "# LOOK I WAS MODIFIED" in obs
 
 
@@ -1024,7 +1022,6 @@ def test_existing_local_data_docs_urls_returns_single_url_from_customized_local_
     assert os.path.isfile(expected_path)
 
     obs = context.get_docs_sites_urls()
-    print(obs)
     assert obs == [{'site_name': 'my_rad_site', 'site_url': "file://{}".format(expected_path)}]
 
 
diff --git a/tests/data_context/test_data_context_data_docs_api.py b/tests/data_context/test_data_context_data_docs_api.py
new file mode 100644
--- /dev/null
+++ b/tests/data_context/test_data_context_data_docs_api.py
@@ -0,0 +1,194 @@
+import mock
+import pytest
+
+from great_expectations.exceptions import DataContextError
+
+
+@mock.patch("webbrowser.open", return_value=True, side_effect=None)
+def test_open_docs_with_no_site(mock_webbrowser, context_with_no_sites):
+    context = context_with_no_sites
+
+    context.open_data_docs()
+    assert mock_webbrowser.call_count == 0
+
+
+@mock.patch("webbrowser.open", return_value=True, side_effect=None)
+def test_open_docs_with_non_existent_site_raises_error(
+    mock_webbrowser, empty_data_context
+):
+    context = empty_data_context
+    with pytest.raises(DataContextError):
+        context.open_data_docs(site_name="foo")
+    assert mock_webbrowser.call_count == 0
+
+
+@mock.patch("webbrowser.open", return_value=True, side_effect=None)
+def test_open_docs_with_single_local_site(mock_webbrowser, empty_data_context):
+    context = empty_data_context
+    obs = context.get_docs_sites_urls()
+    assert len(obs) == 1
+    assert obs[0]["site_url"].endswith(
+        "great_expectations/uncommitted/data_docs/local_site/index.html"
+    )
+    assert obs[0]["site_name"] == "local_site"
+
+    context.open_data_docs()
+    assert mock_webbrowser.call_count == 1
+    call = mock_webbrowser.call_args_list[0][0][0]
+    assert call.startswith("file:///")
+    assert call.endswith(
+        "/great_expectations/uncommitted/data_docs/local_site/index.html"
+    )
+
+
+@pytest.fixture
+def context_with_multiple_sites(empty_data_context):
+    context = empty_data_context
+    config = context._project_config_with_variables_substituted
+    multi_sites = {
+        "local_site": {
+            "class_name": "SiteBuilder",
+            "store_backend": {
+                "class_name": "TupleFilesystemStoreBackend",
+                "base_directory": "uncommitted/data_docs/local_site/",
+            },
+            "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
+        },
+        "another_local_site": {
+            "class_name": "SiteBuilder",
+            "store_backend": {
+                "class_name": "TupleFilesystemStoreBackend",
+                "base_directory": "uncommitted/data_docs/another_local_site/",
+            },
+            "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
+        },
+    }
+    config.data_docs_sites = multi_sites
+    context._project_config = config
+    obs = context.get_docs_sites_urls()
+    assert len(obs) == 2
+    assert obs[0]["site_url"].endswith(
+        "great_expectations/uncommitted/data_docs/local_site/index.html"
+    )
+    assert obs[0]["site_name"] == "local_site"
+
+    assert obs[1]["site_url"].endswith(
+        "great_expectations/uncommitted/data_docs/another_local_site/index.html"
+    )
+    assert obs[1]["site_name"] == "another_local_site"
+
+    return context
+
+
+@pytest.fixture
+def context_with_multiple_local_sites_and_s3_site(empty_data_context):
+    context = empty_data_context
+    config = context._project_config_with_variables_substituted
+    multi_sites = {
+        "local_site": {
+            "class_name": "SiteBuilder",
+            "store_backend": {
+                "class_name": "TupleFilesystemStoreBackend",
+                "base_directory": "uncommitted/data_docs/local_site/",
+            },
+            "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
+        },
+        "another_local_site": {
+            "class_name": "SiteBuilder",
+            "store_backend": {
+                "class_name": "TupleFilesystemStoreBackend",
+                "base_directory": "uncommitted/data_docs/another_local_site/",
+            },
+            "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
+        },
+        "s3_site": {
+            "class_name": "SiteBuilder",
+            "store_backend": {"class_name": "TupleS3StoreBackend", "bucket": "foo",},
+            "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
+        },
+    }
+    config.data_docs_sites = multi_sites
+    context._project_config = config
+    obs = context.get_docs_sites_urls()
+    assert len(obs) == 2
+    assert obs[0]["site_url"].endswith(
+        "great_expectations/uncommitted/data_docs/local_site/index.html"
+    )
+    assert obs[0]["site_name"] == "local_site"
+
+    assert obs[1]["site_url"].endswith(
+        "great_expectations/uncommitted/data_docs/another_local_site/index.html"
+    )
+    assert obs[1]["site_name"] == "another_local_site"
+
+    return context
+
+
+@mock.patch("webbrowser.open", return_value=True, side_effect=None)
+def test_open_docs_with_two_local_sites(mock_webbrowser, context_with_multiple_sites):
+    context = context_with_multiple_sites
+    context.open_data_docs()
+    assert mock_webbrowser.call_count == 2
+    first_call = mock_webbrowser.call_args_list[0][0][0]
+    assert first_call.startswith("file:///")
+    assert first_call.endswith(
+        "/great_expectations/uncommitted/data_docs/local_site/index.html"
+    )
+    second_call = mock_webbrowser.call_args_list[1][0][0]
+    assert second_call.startswith("file:///")
+    assert second_call.endswith(
+        "/great_expectations/uncommitted/data_docs/another_local_site/index.html"
+    )
+
+
+@mock.patch("webbrowser.open", return_value=True, side_effect=None)
+def test_open_docs_with_two_local_sites_specify_open_one(
+    mock_webbrowser, context_with_multiple_sites
+):
+    context = context_with_multiple_sites
+    context.open_data_docs(site_name="another_local_site")
+
+    assert mock_webbrowser.call_count == 1
+    call = mock_webbrowser.call_args_list[0][0][0]
+    assert call.startswith("file:///")
+    assert call.endswith(
+        "/great_expectations/uncommitted/data_docs/another_local_site/index.html"
+    )
+
+
+@pytest.fixture
+def context_with_no_sites(empty_data_context):
+    context = empty_data_context
+    context._project_config["data_docs_sites"] = None
+    return context
+
+
+def test_get_docs_sites_urls_with_no_sites(context_with_no_sites):
+    assert context_with_no_sites.get_docs_sites_urls() == []
+
+
+def test_get_docs_sites_urls_with_no_sites_specify_one(context_with_no_sites):
+    assert context_with_no_sites.get_docs_sites_urls(site_name="foo") == []
+
+
+def test_get_docs_sites_urls_with_non_existent_site_raises_error(
+    context_with_multiple_sites,
+):
+    context = context_with_multiple_sites
+    with pytest.raises(DataContextError):
+        context.get_docs_sites_urls(site_name="not_a_real_site")
+
+
+def test_get_docs_sites_urls_with_two_local_sites_specify_one(
+    context_with_multiple_sites,
+):
+    context = context_with_multiple_sites
+    obs = context.get_docs_sites_urls(site_name="another_local_site")
+    assert len(obs) == 1
+    assert obs[0]["site_name"] == "another_local_site"
+
+    url = obs[0]["site_url"]
+    assert url.startswith("file:///")
+    assert url.endswith(
+        "/great_expectations/uncommitted/data_docs/another_local_site/index.html"
+    )