Repository: innobi/pantab
Pull request #123 (2021-02-09 18:04:04)

Allow the user to supply his own Hyper connection reading from Hyper

WillAyd: If you push a PR to fix this can you also add a “versionadded” sphinx directive for 2.0?

From: Adrian Vogelsgesang <notifications@github.com>
Reply-To: innobi/pantab <reply@reply.github.com>
Date: Tuesday, February 9, 2021 at 1:15 PM
To: innobi/pantab <pantab@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: Re: [innobi/pantab] Allow the user to supply his own Hyper connection (#123)


@vogelsgesang commented on this pull request.

________________________________

In doc/source/examples.rst<https://github.com/innobi/pantab/pull/123#discussion_r573244645>:

> +

+

+Providing your own Hyper Connection

+-----------------------------------

+

+

+In order to interface with Hyper, pantab functions need a HyperAPI `Connection<https://help.tableau.com/current/api/hyper_api/en-us/reference/py/tableauhyperapi.html#tableauhyperapi.Connection>`_ to interface with Hyper.

+For convenience, pantab creates those connections implicitly for you.

+However, establishing a connection is not for free, and by reusing the same ``Connection`` for multiple operations, we can save time.

+Hence, pantab also allows you to pass in a HyperAPI connection instead of the name / location of your Hyper file.

+

+In particular for automated scripts or if you call ``frame_to_hyper`` repeatedly in a loop, passing the ``Connection`` to pantab from is a good optimization opportunity.

+

+

+.. code-block:: python

+   import pandas as pd

missing empty line

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/innobi/pantab/pull/123#pullrequestreview-587016674>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAEU4UN7NROXAOXZDJ3PLYTS6GQWJANCNFSM4XLOSZTQ>.

pep8speaks: Hello @vogelsgesang! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pantab/_hyper_util.py`](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/_hyper_util.py):

> [Line 33:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/_hyper_util.py#L33): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 88 characters)







* In the file [`pantab/tests/test_roundtrip.py`](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py):

> [Line 83:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py#L83): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)
> [Line 85:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py#L85): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (101 > 88 characters)
> [Line 107:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py#L107): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (105 > 88 characters)
> [Line 115:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py#L115): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (96 > 88 characters)
> [Line 123:89](https://github.com/innobi/pantab/blob/19f87ed4c364791c998f8cd284738436ad89fcf2/pantab/tests/test_roundtrip.py#L123): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (100 > 88 characters)


vogelsgesang: Scoped this PR down to only "reading from Hyper", as the semantics for "bring your own connection for writing" are still unclear

Patch:
diff --git a/asv.conf.json b/asv.conf.json
index 8adec52..627498a 100644
--- a/asv.conf.json
+++ b/asv.conf.json
@@ -73,7 +73,7 @@
     //
     "matrix": {
          "pandas": [],
-         "pip+http://downloads.tableau.com/tssoftware/tableauhyperapi-0.0.9273-py3-none-macosx_10_11_x86_64.whl": [""],
+         "pip+tableauhyperapi": [],
     },
 
     // Combinations of libraries/python versions can be excluded/included
diff --git a/doc/source/api.rst b/doc/source/api.rst
index 06134a1..69ce0fd 100644
--- a/doc/source/api.rst
+++ b/doc/source/api.rst
@@ -11,11 +11,11 @@ API Reference
    :param hyper_process: A `HyperProcess` in case you want to spawn it by yourself. Optional. Must be supplied as a keyword argument.
 
 
-.. py:function:: frame_from_hyper(database: Union[str, pathlib.Path], *, table: Union[str, tableauhyperapi.Name, tableauhyperapi.TableName], hyper_process: Optional[HyperProcess]) -> pd.DataFrame:
+.. py:function:: frame_from_hyper(source: Union[str, pathlib.Path, tab_api.Connection], *, table: Union[str, tableauhyperapi.Name, tableauhyperapi.TableName], hyper_process: Optional[HyperProcess]) -> pd.DataFrame:
 
    Extracts a DataFrame from a .hyper extract.
 
-   :param database: Name / location of the Hyper file to be read.
+   :param source: Name / location of the Hyper file to be read  or Hyper-API connection.
    :param table: Table to read. Must be supplied as a keyword argument.
    :param hyper_process: A `HyperProcess` in case you want to spawn it by yourself. Optional. Must be supplied as a keyword argument.
    :rtype: pd.DataFrame
@@ -30,20 +30,22 @@ API Reference
    :param hyper_process: A `HyperProcess` in case you want to spawn it by yourself. Optional. Must be supplied as a keyword argument.
 
 
-.. py:function:: frames_from_hyper(database: Union[str, pathlib.Path], *, hyper_process: Optional[HyperProcess]) -> Dict[tableauhyperapi.TableName, pd.DataFrame]:
+.. py:function:: frames_from_hyper(source: Union[str, pathlib.Path, tab_api.Connection], *, hyper_process: Optional[HyperProcess]) -> Dict[tableauhyperapi.TableName, pd.DataFrame]:
 
    Extracts tables from a .hyper extract.
 
-   :param database: Name / location of the Hyper file to read from.
+   :param source: Name / location of the Hyper file to be read  or Hyper-API connection.
    :param hyper_process: A `HyperProcess` in case you want to spawn it by yourself. Optional. Must be supplied as a keyword argument.
    :rtype: Dict[tableauhyperapi.TableName, pd.DataFrame]
 
 
-.. py:function:: frame_from_hyper_query(database: Union[str, pathlib.Path], query: str, *, hyper_process: Optional[HyperProcess]) -> pd.DataFrame:
+.. py:function:: frame_from_hyper_query(source: Union[str, pathlib.Path, tab_api.Connection], query: str, *, hyper_process: Optional[HyperProcess]) -> pd.DataFrame:
+
+.. versionadded:: 2.0
 
    Executes a SQL query and returns the result as a pandas dataframe
 
-   :param database: Name / location of the Hyper file to be read.
-   :param query: SQL query to execute
+   :param source: Name / location of the Hyper file to be read  or Hyper-API connection.
+   :param query: SQL query to execute.
    :param hyper_process: A `HyperProcess` in case you want to spawn it by yourself. Optional. Must be supplied as a keyword argument.
    :rtype: Dict[tableauhyperapi.TableName, pd.DataFrame]
diff --git a/doc/source/examples.rst b/doc/source/examples.rst
index d04f758..1e862d5 100644
--- a/doc/source/examples.rst
+++ b/doc/source/examples.rst
@@ -113,6 +113,8 @@ Please note that ``table_mode="a"`` will create the table(s) if they do not alre
 Issuing SQL queries
 -------------------
 
+.. versionadded:: 2.0
+
 With ``frame_from_hyper_query``, one can execute SQL queries against a Hyper file and retrieve the resulting data as a DataFrame. This can be used, e.g. to retrieve only a part of the data (using a ``WHERE`` clause) or to offload computations to Hyper.
 
 .. code-block:: python
@@ -151,25 +153,31 @@ With ``frame_from_hyper_query``, one can execute SQL queries against a Hyper fil
 Providing your own HyperProcess
 -------------------------------
 
-For convenience, pantab's functions internally spawn a `HyperProcess<https://help.tableau.com/current/api/hyper_api/en-us/reference/py/tableauhyperapi.html#tableauhyperapi.HyperProcess>`_. In case you prefer to spawn your own ``HyperProcess``, you can supply it to pantab through the ``hyper_process`` keyword argument.
+.. versionadded:: 2.0
+
+For convenience, pantab's functions internally spawn a `HyperProcess <https://help.tableau.com/current/api/hyper_api/en-us/reference/py/tableauhyperapi.html#tableauhyperapi.HyperProcess>`_. In case you prefer to spawn your own ``HyperProcess``, you can supply it to pantab through the ``hyper_process`` keyword argument.
 
 By using your own ``HyperProcess``, you have full control over all its startup paramters.
 In the following example we use that flexibility to:
 
 - enable telemetry, thereby making sure the Hyper team at Tableau knows about our use case and potential issues we might be facing
-- `disable log files<https://help.tableau.com/current/api/hyper_api/en-us/reference/sql/loggingsettings.html#LOG_CONFIG>`_, as we operate in some environment with really tight disk space
-- opt-in to the `new Hyper file format<https://help.tableau.com/current/api/hyper_api/en-us/reference/sql/databasesettings.html#DEFAULT_DATABASE_VERSION>`_
+- `disable log files <https://help.tableau.com/current/api/hyper_api/en-us/reference/sql/loggingsettings.html#LOG_CONFIG>`_, as we operate in some environment with really tight disk space
+- opt-in to the `new Hyper file format <https://help.tableau.com/current/api/hyper_api/en-us/reference/sql/databasesettings.html#DEFAULT_DATABASE_VERSION>`_
 
 By reusing the same ``HyperProcess`` for multiple operations, we also save a few milliseconds. While not noteworthy in this simple example, this might be a good optimization in case you call ``frame_to_hyper`` repeatedly in a loop.
 
+
 .. code-block:: python
+
    import pandas as pd
    import pantab
    from tableauhyperapi import HyperProcess, Telemetry
+
    df = pd.DataFrame([
        ["dog", 4],
        ["cat", 4],
    ], columns=["animal", "num_of_legs"])
+
    parameters = {"log_config": "", "default_database_version": "1"}
    with HyperProcess(Telemetry.SEND_USAGE_DATA_TO_TABLEAU, parameters=parameters) as hyper:
        # Insert some initial data
@@ -178,3 +186,42 @@ By reusing the same ``HyperProcess`` for multiple operations, we also save a few
        # Append additional data to the same table using `table_mode="a"`
        new_data = pd.DataFrame([["moose", 4]], columns=["animal", "num_of_legs"])
        pantab.frame_to_hyper(df, "example.hyper", table="animals", table_mode="a", hyper_process=hyper)
+
+
+Providing your own Hyper Connection
+-----------------------------------
+
+.. versionadded:: 2.0
+
+In order to interface with Hyper, pantab functions need a HyperAPI `Connection <https://help.tableau.com/current/api/hyper_api/en-us/reference/py/tableauhyperapi.html#tableauhyperapi.Connection>`_ to interface with Hyper.
+For convenience, pantab creates those connections implicitly for you.
+However, establishing a connection is not for free, and by reusing the same ``Connection`` for multiple operations, we can save time.
+Hence, pantab also allows you to pass in a HyperAPI connection instead of the name / location of your Hyper file.
+
+.. code-block:: python
+
+   import pandas as pd
+   import pantab
+   from tableauhyperapi import HyperProcess, Telemetry, Connection, CreateMode
+
+   df = pd.DataFrame([
+       ["dog", 4],
+       ["cat", 4],
+       ["centipede", 100],
+   ], columns=["animal", "num_of_legs"])
+   path = "example.hyper"
+
+   with HyperProcess(Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU) as hyper:
+       pantab.frames_to_hyper({"animals": df}, path, hyper_process=hyper)
+
+       with Connection(hyper.endpoint, path, CreateMode.NONE) as connection:
+            query = """
+            SELECT animal
+            FROM animals
+            WHERE num_of_legs > 4
+            """
+            many_legs_df = pantab.frame_from_hyper_query(connection, query)
+            print(many_legs_df)
+
+            all_animals = pantab.frame_from_hyper_query(connection, table="animals")
+            print(all_animals)
diff --git a/pantab/_hyper_util.py b/pantab/_hyper_util.py
index 7fbb84b..70eb3c4 100644
--- a/pantab/_hyper_util.py
+++ b/pantab/_hyper_util.py
@@ -20,3 +20,10 @@ def ensure_hyper_process(hyper_process: Optional[tab_api.HyperProcess]):
         # Wrap the HyperProcess into a nullcontext such that the `with` doesn't close
         # the HyperProcess
         return nullcontext(hyper_process)
+
+
+def forbid_hyper_process(hyper_process: Optional[tab_api.HyperProcess]):
+    if hyper_process is not None:
+        raise ValueError(
+            "hyper_process parameter is useless because `Connection` is provided"
+        )
diff --git a/pantab/_reader.py b/pantab/_reader.py
index ffa0cc4..f7e68e1 100644
--- a/pantab/_reader.py
+++ b/pantab/_reader.py
@@ -9,7 +9,7 @@ import tableauhyperapi as tab_api
 
 import libpantab  # type: ignore
 import pantab._types as pantab_types
-from pantab._hyper_util import ensure_hyper_process
+from pantab._hyper_util import ensure_hyper_process, forbid_hyper_process
 
 TableType = Union[str, tab_api.Name, tab_api.TableName]
 
@@ -75,49 +75,69 @@ def _read_table(*, connection: tab_api.Connection, table: TableType) -> pd.DataF
 
 
 def frame_from_hyper(
-    database: Union[str, pathlib.Path],
+    source: Union[str, pathlib.Path, tab_api.Connection],
     *,
     table: TableType,
     hyper_process: Optional[tab_api.HyperProcess] = None,
 ) -> pd.DataFrame:
     """See api.rst for documentation"""
 
-    with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(
-        hyper_process
-    ) as hpe:
-        tmp_db = shutil.copy(database, tmp_dir)
-        with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
-            return _read_table(connection=connection, table=table)
+    if isinstance(source, tab_api.Connection):
+        forbid_hyper_process(hyper_process)
+        return _read_table(connection=source, table=table)
+    else:
+        with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(
+            hyper_process
+        ) as hpe:
+            tmp_db = shutil.copy(source, tmp_dir)
+            with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
+                return _read_table(connection=connection, table=table)
 
 
 def frames_from_hyper(
-    database: Union[str, pathlib.Path],
+    source: Union[str, pathlib.Path, tab_api.Connection],
     *,
     hyper_process: Optional[tab_api.HyperProcess] = None,
 ) -> Dict[tab_api.TableName, pd.DataFrame]:
     """See api.rst for documentation."""
     result: Dict[TableType, pd.DataFrame] = {}
-    with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(
-        hyper_process
-    ) as hpe:
-        tmp_db = shutil.copy(database, tmp_dir)
-        with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
-            for schema in connection.catalog.get_schema_names():
-                for table in connection.catalog.get_table_names(schema=schema):
-                    result[table] = _read_table(connection=connection, table=table)
+
+    if isinstance(source, tab_api.Connection):
+        forbid_hyper_process(hyper_process)
+        connection = source
+        for schema in connection.catalog.get_schema_names():
+            for table in connection.catalog.get_table_names(schema=schema):
+                result[table] = _read_table(connection=connection, table=table)
+    else:
+        with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(
+            hyper_process
+        ) as hpe:
+            tmp_db = shutil.copy(source, tmp_dir)
+            with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
+                for schema in connection.catalog.get_schema_names():
+                    for table in connection.catalog.get_table_names(schema=schema):
+                        result[table] = _read_table(connection=connection, table=table)
 
     return result
 
 
 def frame_from_hyper_query(
-    database: Union[str, pathlib.Path],
+    source: Union[str, pathlib.Path, tab_api.Connection],
     query: str,
     *,
-    hyper: Optional[tab_api.HyperProcess] = None,
+    hyper_process: Optional[tab_api.HyperProcess] = None,
 ) -> pd.DataFrame:
     """See api.rst for documentation."""
-    with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(hyper) as hpe:
-        tmp_db = shutil.copy(database, tmp_dir)
-        with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
-            with connection.execute_query(query) as result:
-                return _read_query_result(result, None)
+
+    if isinstance(source, tab_api.Connection):
+        forbid_hyper_process(hyper_process)
+        with source.execute_query(query) as result:
+            return _read_query_result(result, None)
+    else:
+        with tempfile.TemporaryDirectory() as tmp_dir, ensure_hyper_process(
+            hyper_process
+        ) as hpe:
+            tmp_db = shutil.copy(source, tmp_dir)
+            with tab_api.Connection(hpe.endpoint, tmp_db) as connection:
+                with connection.execute_query(query) as result:
+                    return _read_query_result(result, None)


Test patch:
diff --git a/pantab/tests/test_reader.py b/pantab/tests/test_reader.py
index cd169f0..81e2b2d 100644
--- a/pantab/tests/test_reader.py
+++ b/pantab/tests/test_reader.py
@@ -1,7 +1,7 @@
 import pandas as pd
 import pandas.testing as tm
 import pytest
-from tableauhyperapi import TableName, HyperProcess, Telemetry, Connection
+from tableauhyperapi import TableName
 
 import pantab
 import pantab._compat as compat
diff --git a/pantab/tests/test_roundtrip.py b/pantab/tests/test_roundtrip.py
index ac0db26..23ae08e 100644
--- a/pantab/tests/test_roundtrip.py
+++ b/pantab/tests/test_roundtrip.py
@@ -2,7 +2,8 @@ import numpy as np
 from pathlib import Path
 import pandas as pd
 import pandas.testing as tm
-from tableauhyperapi import TableName, HyperProcess, Telemetry
+import pytest
+from tableauhyperapi import Connection, TableName, HyperProcess, Telemetry, CreateMode
 
 import pantab
 import pantab._compat as compat
@@ -31,8 +32,6 @@ def test_basic(df, tmp_hyper, table_name, table_mode):
     if table_mode == "a":
         expected = pd.concat([expected, expected]).reset_index(drop=True)
 
-    expected["float32"] = expected["float32"].astype(np.float64)
-
     assert_roundtrip_equal(result, expected)
 
 
@@ -69,8 +68,65 @@ def test_roundtrip_with_external_hyper_process(df, tmp_hyper):
     with HyperProcess(
         Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU, parameters=parameters
     ) as hyper:
+        # test frame_to_hyper/frame_from_hyper
         pantab.frame_to_hyper(df, tmp_hyper, table="test", hyper_process=hyper)
-        pantab.frame_from_hyper(tmp_hyper, table="test", hyper_process=hyper)
-        pantab.frames_from_hyper(tmp_hyper, hyper_process=hyper)
+        result = pantab.frame_from_hyper(tmp_hyper, table="test", hyper_process=hyper)
+        assert_roundtrip_equal(result, df)
+
+        # test frame_from_hyper_query
+        result = pantab.frame_from_hyper_query(
+            tmp_hyper, "SELECT * FROM test", hyper_process=hyper
+        )
+        assert result.size == 63
+
+        # test frames_to_hyper/frames_from_hyper
+        pantab.frames_to_hyper(
+            {"test2": df, "test": df}, tmp_hyper, hyper_process=hyper
+        )
+        result = pantab.frames_from_hyper(tmp_hyper, hyper_process=hyper)
+        assert set(result.keys()) == set(
+            (TableName("public", "test"), TableName("public", "test2"))
+        )
+        for val in result.values():
+            assert_roundtrip_equal(val, df)
 
     assert not default_log_path.exists()
+
+
+def test_roundtrip_with_external_hyper_connection(df, tmp_hyper):
+    with HyperProcess(Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU) as hyper:
+        pantab.frames_to_hyper(
+            {"test": df, "test2": df}, tmp_hyper, hyper_process=hyper
+        )
+
+        with Connection(hyper.endpoint, tmp_hyper, CreateMode.NONE) as connection:
+            result = pantab.frame_from_hyper(connection, table="test")
+            assert_roundtrip_equal(result, df)
+
+            result = pantab.frame_from_hyper_query(connection, "SELECT * FROM test")
+            assert result.size == 63
+
+            result = pantab.frames_from_hyper(connection)
+            assert set(result.keys()) == set(
+                (TableName("public", "test"), TableName("public", "test2"))
+            )
+            for val in result.values():
+                assert_roundtrip_equal(val, df)
+
+
+def test_external_hyper_connection_and_process_error(df, tmp_hyper):
+    with HyperProcess(Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU) as hyper:
+        with Connection(hyper.endpoint, tmp_hyper, CreateMode.CREATE) as connection:
+            expected_msg = (
+                "hyper_process parameter is useless because `Connection` is provided"
+            )
+            with pytest.raises(ValueError, match=expected_msg):
+                pantab.frame_from_hyper(connection, table="test", hyper_process=hyper)
+
+            with pytest.raises(ValueError, match=expected_msg):
+                pantab.frame_from_hyper_query(
+                    connection, "SELECT * FROM test", hyper_process=hyper
+                )
+
+            with pytest.raises(ValueError, match=expected_msg):
+                pantab.frames_from_hyper(connection, hyper_process=hyper)