Repository: great-expectations/great_expectations
Pull request: [MAINTENANCE] Remove `ipython` dependency



- [x] Description of PR changes above includes a link to [an existing GitHub issue](https://github.com/great-expectations/great_expectations/issues)
- [x] PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
- [x] Code is linted - run `invoke lint` (uses `ruff format` + `ruff check`)
- [x] Appropriate tests and docs have been updated

For more information about contributing, see [Contribute](https://docs.greatexpectations.io/docs/contributing/contributing_checklist).

After you submit your PR, keep the page open and **monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up** and [reach out on Slack](https://greatexpectations.io/slack) if you need help. Thanks for contributing!


Problem statement:
Light Weight Install
**Is your feature request related to a problem? Please describe.**
In some deployment strategies there is no need to have the Jupiter dependencies installed. 
Having the Jupiter (and associated dependencies) installed ends up prolonging the deployment cycle.

**Describe the solution you'd like**
Have a "lite" whl available on pip/conda

Hints from issue discussion:
@JeroenSchmidt Thank you for posting this suggestion! This sounds like a more general case of https://github.com/great-expectations/great_expectations/issues/2688. 

Marking this "in investigation", since this is an open discussion.
@jcampbell Please add you thoughts on the substance of this suggestion and its priority. Thank you!
> @JeroenSchmidt Thank you for posting this suggestion! This sounds like a more general case of https://github.com/great-expectations/great_expectations/issues/2688. 
> 
> 
> 
> Marking this "in investigation", since this is an open discussion.

Yes that's correct. In fact the other issue you referenced mentions almost the same use case I am dealing with. 
@JeroenSchmidt Circling back to my question in the other issue - do you have a preferred strategy?
@twcurrie I think that the pip install that omits ipywidgets should be the optional install given that great_expecations is meant to be used interactively. I foresee it causing headaches for new users if the default pip install omitted Jupiter.

We would prefer;
`pip install great_expectations[no-ipywidgets]` or `pip install great_expectations[thin-client]` 

Our use case is less concerned with the dependency issue that is encountered by #2688 and more concerned with creating small and quick deployment cycles for our applications that use great_expectations but do not need the whole Jupiter suit.  
@JeroenSchmidt Makes sense, and I agree with the comment of the headaches - exactly why I've been trying to solicit the great_expectations team for an opinion.  Thank you for sharing your team's perspective.

What are some of the other dependencies that your team has issues with or sees as heavy-weight?


I have limited experience developing expectations, but as someone who runs production infrastructure which leverages GE, I'm quite surprised this issue has (had) zero upvotes and is not more widely referenced. It seems like an oversight _not_ to have an `interactive/development` installation and a `production/server` installation which excludes dependencies required to create expectations, i.e. is limited to _only_ the set of dependencies required to run expectations. Those are two very different workflows, the former occurring on a developer's machine, the latter running on the server. Separating those dependency suites would have immense benefits for users running GE in production environments, including reducing the vulnerability surface by drastically reducing the number of transitive dependencies.
@twcurrie I don't have a concrete list that I can share but the comment by @bradgwest touches very well on what I mentioned earlier. 
As @bradgwest already mentioned earlier, having the option to exclude the whole Ipython/Jupyter suite of dependencies from great-expectations would be very helpful if you want to use great-expectations in production (e.g. as part of an Airflow job). For the simple reason that we reduce the number of (Python) dependencies. This has the advantages that: 

1. we get smaller container images when running jobs on Kubernetes (faster downloads of container images etc)
2. We have a smaller attack surface for vulnerability which means it is more likely that GE is allowed to be used in heavily regulated industry (banking, pharma, ...) from a compliance point of view.

My suggestion (and that also seems like the simplest) would be to make all Jupyter dependencies optional. Basically to install them as 
```
pip install "great_expectations[jupyter]"
```
This could be a simple PR that only requires changes to `setup.py` and `requirements.txt`
👍🏽 to this issue. 

I would also prefer the "normal" installation to install everything but have the option for a lite installer like 
`pip install great_expectations[slim]` or 
`pip install great_expectations[lite]` 
... that does not install any of the Jupyter dependencies.

My use case is when I install GX for GitHub Actions CI/CD. It only runs a few checks, re-builds the Data Docs, etc. so it does not need a full fat GX installation.

Commit message: init commit

Patch to review:
diff --git a/great_expectations/core/expectation_validation_result.py b/great_expectations/core/expectation_validation_result.py
index e2cb84f25265..de8c15b62752 100644
--- a/great_expectations/core/expectation_validation_result.py
+++ b/great_expectations/core/expectation_validation_result.py
@@ -1,6 +1,5 @@
 from __future__ import annotations
 
-import datetime
 import json
 import logging
 from copy import deepcopy
@@ -10,7 +9,6 @@
 from typing_extensions import TypedDict
 
 import great_expectations.exceptions as gx_exceptions
-from great_expectations import __version__ as ge_version
 from great_expectations._docs_decorators import public_api
 from great_expectations.alias_types import JSONValues  # noqa: TCH001
 from great_expectations.compatibility.typing_extensions import override
@@ -20,7 +18,6 @@
 )
 from great_expectations.core.id_dict import BatchSpec  # noqa: TCH001
 from great_expectations.core.run_identifier import RunIdentifier  # noqa: TCH001
-from great_expectations.core.util import in_jupyter_notebook
 from great_expectations.data_context.util import instantiate_class_from_config
 from great_expectations.exceptions import ClassInstantiationError
 from great_expectations.render import (
@@ -193,24 +190,6 @@ def __repr__(self) -> str:
         would consist of "__str__()" calling "__repr__()", while all output options are handled through state variables.
         """  # noqa: E501
         json_dict: dict = self.to_json_dict()
-        if in_jupyter_notebook():
-            if (
-                "expectation_config" in json_dict
-                and "kwargs" in json_dict["expectation_config"]
-                and "auto" in json_dict["expectation_config"]["kwargs"]
-                and json_dict["expectation_config"]["kwargs"]["auto"]
-            ):
-                json_dict["expectation_config"]["meta"] = {
-                    "auto_generated_at": datetime.datetime.now(datetime.timezone.utc).strftime(
-                        "%Y%m%dT%H%M%S.%fZ"
-                    ),
-                    "great_expectations_version": ge_version,
-                }
-                json_dict["expectation_config"]["kwargs"].pop("auto")
-                json_dict["expectation_config"]["kwargs"].pop("batch_id", None)
-            else:
-                json_dict.pop("expectation_config", None)
-
         return json.dumps(json_dict, indent=2)
 
     @override
diff --git a/great_expectations/core/util.py b/great_expectations/core/util.py
index 123500685f7e..067afbeff3dd 100644
--- a/great_expectations/core/util.py
+++ b/great_expectations/core/util.py
@@ -5,12 +5,11 @@
 import re
 import warnings
 from collections import OrderedDict
-from typing import TYPE_CHECKING, Any, Callable, Mapping, MutableMapping, Optional, TypeVar, Union
+from typing import TYPE_CHECKING, Any, Mapping, MutableMapping, Optional, TypeVar, Union
 from urllib.parse import urlparse
 
 import dateutil.parser
 import numpy as np
-from IPython import get_ipython
 
 from great_expectations import exceptions as gx_exceptions
 from great_expectations.compatibility.sqlalchemy import SQLALCHEMY_NOT_IMPORTED, LegacyRow, Row
@@ -98,38 +97,6 @@ def nested_update(
     return d
 
 
-def in_jupyter_notebook():
-    try:
-        shell = get_ipython().__class__.__name__
-        if shell == "ZMQInteractiveShell":
-            return True  # Jupyter notebook or qtconsole
-        elif shell == "TerminalInteractiveShell":
-            return False  # Terminal running IPython
-        else:
-            return False  # Other type (?)
-    except NameError:
-        return False  # Probably standard Python interpreter
-
-
-def determine_progress_bar_method_by_environment() -> Callable:
-    """
-    As tqdm has specific methods for progress bar creation and iteration,
-    we require a utility to determine which method to use.
-
-    If in a Jupyter notebook, we want to use `tqdm.notebook.tqdm`. Otherwise,
-    we default to the standard `tqdm.tqdm`. Please see the docs for more information: https://tqdm.github.io/
-
-    Returns:
-        The appropriate tqdm method for the environment in question.
-    """
-    from tqdm import tqdm
-    from tqdm.notebook import tqdm as tqdm_notebook
-
-    if in_jupyter_notebook():
-        return tqdm_notebook
-    return tqdm
-
-
 def substitute_all_strftime_format_strings(
     data: Union[dict, list, str, Any], datetime_obj: Optional[datetime.datetime] = None
 ) -> Union[str, Any]:
diff --git a/great_expectations/experimental/rule_based_profiler/rule/rule.py b/great_expectations/experimental/rule_based_profiler/rule/rule.py
index dcb8434228c7..0fa50d40183e 100644
--- a/great_expectations/experimental/rule_based_profiler/rule/rule.py
+++ b/great_expectations/experimental/rule_based_profiler/rule/rule.py
@@ -4,10 +4,9 @@
 import json
 from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
 
+from tqdm import tqdm
+
 from great_expectations.compatibility.typing_extensions import override
-from great_expectations.core.util import (
-    determine_progress_bar_method_by_environment,
-)
 from great_expectations.experimental.rule_based_profiler.config.base import (
     domainBuilderConfigSchema,
     expectationConfigurationBuilderConfigSchema,
@@ -141,7 +140,7 @@ def run(  # noqa: PLR0913
 
         rule_state.reset_parameter_containers()
 
-        pbar_method: Callable = determine_progress_bar_method_by_environment()
+        pbar_method: Callable = tqdm
 
         domain: Domain
         for domain in pbar_method(
diff --git a/great_expectations/experimental/rule_based_profiler/rule_based_profiler.py b/great_expectations/experimental/rule_based_profiler/rule_based_profiler.py
index 6441f81b7abc..bdb1a76fe7d6 100644
--- a/great_expectations/experimental/rule_based_profiler/rule_based_profiler.py
+++ b/great_expectations/experimental/rule_based_profiler/rule_based_profiler.py
@@ -8,6 +8,8 @@
 import traceback
 from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Set, Union
 
+from tqdm import tqdm
+
 import great_expectations.exceptions as gx_exceptions
 from great_expectations.core.batch import (
     Batch,
@@ -16,7 +18,6 @@
 )
 from great_expectations.core.config_peer import ConfigPeer
 from great_expectations.core.util import (
-    determine_progress_bar_method_by_environment,
     nested_update,
 )
 from great_expectations.data_context.cloud_constants import GXCloudRESTResource
@@ -302,7 +303,7 @@ def run(  # noqa: PLR0913
         sys.stdout.write("\n")
         sys.stdout.flush()
 
-        pbar_method: Callable = determine_progress_bar_method_by_environment()
+        pbar_method: Callable = tqdm
 
         rule_state: RuleState
         rule: Rule
diff --git a/requirements.txt b/requirements.txt
index 2a7a02653daa..a54280d6f73c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,5 @@
 altair>=4.2.1,<5.0.0
 cryptography>=3.2
-Ipython>=7.16.3
 ipywidgets>=7.5.1
 jinja2>=2.10
 jsonschema>=2.5.1


Code review comments:
diff_hunk: @@ -98,38 +97,6 @@ def nested_update(
     return d
 
 
-def in_jupyter_notebook():
-    try:
-        shell = get_ipython().__class__.__name__
-        if shell == "ZMQInteractiveShell":
-            return True  # Jupyter notebook or qtconsole
-        elif shell == "TerminalInteractiveShell":
-            return False  # Terminal running IPython
-        else:
-            return False  # Other type (?)
-    except NameError:
-        return False  # Probably standard Python interpreter
-
-
-def determine_progress_bar_method_by_environment() -> Callable:
-    """
-    As tqdm has specific methods for progress bar creation and iteration,
-    we require a utility to determine which method to use.
-
-    If in a Jupyter notebook, we want to use `tqdm.notebook.tqdm`. Otherwise,
-    we default to the standard `tqdm.tqdm`. Please see the docs for more information: https://tqdm.github.io/
-
-    Returns:
-        The appropriate tqdm method for the environment in question.
-    """
-    from tqdm import tqdm
-    from tqdm.notebook import tqdm as tqdm_notebook
-
-    if in_jupyter_notebook():
-        return tqdm_notebook
-    return tqdm
path: great_expectations/core/util.py
text: @user1:
I would expect removing this will negatively impact the notebook experience which many of our users still use.

I think we can still remove the dependency by moving the `ipython` imports inside these methods (and adding a catch for the import error).

https://github.com/great-expectations/great_expectations/pull/10487/files#r1792085766

@author:
This method is only ever used in RBP

diff_hunk: @@ -141,7 +140,7 @@ def run(  # noqa: PLR0913
 
         rule_state.reset_parameter_containers()
 
-        pbar_method: Callable = determine_progress_bar_method_by_environment()
path: great_expectations/experimental/rule_based_profiler/rule/rule.py
text: @author:
We don't care about RBP and I don't think it's worth adding a dep just to account for this