Repository: capitalone/DataProfiler (language: python)

Issue:
Replace snappy with cramjam



Discussion / hints:
@taylorfturner please assign reviewers
Thanks for your patience @gliptak -- we will review your proposal 
please review
https://github.com/capitalone/synthetic-data/pull/346
@gliptak rebase onto `dev` and I'll approve
rebase or `update branch` on this and I'll approve @gliptak 
green for your review @taylorfturner 
@micdavis @ksneab7 please review/approve
@gliptak are you able to restore your `cramjam1` branch? 

This change fails tests locally on this command `DATAPROFILER_SEED=0 python3 -m unittest discover -p "test*.py"`

First issues is ```dataprofiler/dataprofiler/tests/test_dp_logging.py", line 27, in test_default_verbosity
    self.assertEqual(
AssertionError: 20 != 30```

Second issue is ```dataprofiler/dataprofiler/tests/test_data_profiler.py", line 32, in test_set_seed
    self.assertEqual(dp.settings._seed, None)
AssertionError: 0 != None```

`main` testing works locally and the only diff between `main` and `dev` is this PR

We can either:
- restore your branch and revert this PR 
- or I or you can create a new PR to revert back to snappy, as it was prior to this PR 
Checking out `f8b3e5dbd4b76f0ecc291911ace9e8e21cf1ecb1` and running tests on that commit (one commit prior to this PR) ensures that tests did run on `dev` locally at that point in the history @gliptak 
Glad to ultimately do cramjam + python 3.11 -- these tests will just need to pass locally as well as remote 

Fix patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 38ac7fc2..6f25850c 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -55,7 +55,7 @@ repos:
             pyarrow>=1.0.1,
             chardet>=3.0.4,
             fastavro>=1.0.0.post1,
-            cramjam>=2.7.0,
+            python-snappy>=0.5.4,
             charset-normalizer>=1.3.6,
             psutil>=4.0.0,
             scipy>=1.4.1,
diff --git a/dataprofiler/__init__.py b/dataprofiler/__init__.py
index 5f218bd8..2e89d3e2 100644
--- a/dataprofiler/__init__.py
+++ b/dataprofiler/__init__.py
@@ -20,6 +20,22 @@
 from .validators.base_validators import Validator
 from .version import __version__
 
+try:
+    import snappy
+except ImportError:
+    import warnings
+
+    warnings.warn(
+        "Snappy must be installed to use parquet/avro datasets."
+        "\n\n"
+        "For macOS use Homebrew:\n"
+        "\t`brew install snappy`"
+        "\n\n"
+        "For linux use apt-get:\n`"
+        "\tsudo apt-get -y install libsnappy-dev`\n",
+        ImportWarning,
+    )
+
 
 def set_seed(seed=None):
     # also check it's an integer
diff --git a/requirements.txt b/requirements.txt
index 405f808b..a45dc34a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -7,7 +7,7 @@ pytz>=2020.1
 pyarrow>=1.0.1
 chardet>=3.0.4
 fastavro>=1.0.0.post1
-cramjam>=2.7.0
+python-snappy>=0.5.4
 charset-normalizer>=1.3.6
 psutil>=4.0.0
 scipy>=1.10.0


Test patch:
diff --git a/dataprofiler/tests/test_data_profiler.py b/dataprofiler/tests/test_data_profiler.py
index 9ebdfa03..ef7664ce 100644
--- a/dataprofiler/tests/test_data_profiler.py
+++ b/dataprofiler/tests/test_data_profiler.py
@@ -56,6 +56,46 @@ def test_data_profiling(self):
             self.assertIsNotNone(profile.profile)
             self.assertIsNotNone(profile.report())
 
+    def test_no_snappy(self):
+        import importlib
+        import sys
+        import types
+
+        orig_import = __import__
+        # necessary for any wrapper around the library to test if snappy caught
+        # as an issue
+
+        def reload_data_profiler():
+            """Recursively reload modules."""
+            sys_modules = sys.modules.copy()
+            for module_name, module in sys_modules.items():
+                # Only reload top level of the dataprofiler
+                if "dataprofiler" in module_name and len(module_name.split(".")) < 3:
+                    if isinstance(module, types.ModuleType):
+                        importlib.reload(module)
+
+        def import_mock(name, *args, **kwargs):
+            if name == "snappy":
+                raise ImportError("test")
+            return orig_import(name, *args, **kwargs)
+
+        with mock.patch("builtins.__import__", side_effect=import_mock):
+            with self.assertWarns(ImportWarning) as w:
+                import dataprofiler
+
+                reload_data_profiler()
+
+        self.assertEqual(
+            str(w.warning),
+            "Snappy must be installed to use parquet/avro datasets."
+            "\n\n"
+            "For macOS use Homebrew:\n"
+            "\t`brew install snappy`"
+            "\n\n"
+            "For linux use apt-get:\n`"
+            "\tsudo apt-get -y install libsnappy-dev`\n",
+        )
+
     def test_no_tensorflow(self):
         import sys