Repository: comet-ml/opik
Pull request: [OPIK-427]: remove pandas from the app dependencies; and add it to an unit test requirements;

## Details
* Removed pandas from the dependencies of the app;
* Added it as a dependency to unit tests;

<img width="882" alt="image" src="https://github.com/user-attachments/assets/1b7f6306-3f86-4967-9474-5d2a446bcbfb">

## Issues

Resolves #

## Testing

## Documentation


Problem statement:
[FR]: Make pandas, numpy an optional dependency for the Python SDK
### Proposal summary

We are using Opik from within AWS Lambda, and numpy impacts the package size significantly. Unfortunately, it's required as a top-level import, and we use the SDK for only tracing purposes. Can we update the SDK to support working without a numpy dependency ?

### Motivation

_No response_

Hints from issue discussion:
Hi @SanjayBysani!
That's a good suggestion. Yes, we'll exclude pandas from the dependencies (probably next week).
Thanks @alexkuzmik

Commit message: [OPIK-427]: run linter;

Patch to review:
diff --git a/.github/workflows/python_sdk_unit_tests.yml b/.github/workflows/python_sdk_unit_tests.yml
index 38e2f6ce11..2ec1b8f8e0 100644
--- a/.github/workflows/python_sdk_unit_tests.yml
+++ b/.github/workflows/python_sdk_unit_tests.yml
@@ -42,7 +42,7 @@ jobs:
       - name: Install test requirements
         run: |
           cd ./tests
-          pip install --no-cache-dir --disable-pip-version-check -r test_requirements.txt
+          pip install --no-cache-dir --disable-pip-version-check -r test_requirements.txt -r ./unit/test_requirements.txt
 
       - name: Running SDK Unit Tests
         run: python -m pytest --cov=src/opik -vv tests/unit/
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 73d513259f..3dfd76e65c 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -41,7 +41,6 @@
         "levenshtein~=0.25.1",
         "litellm",
         "openai<2.0.0",
-        "pandas>=2.0.0,<3.0.0",
         "pydantic-settings>=2.0.0,<3.0.0",
         "pydantic>=2.0.0,<3.0.0",
         "pytest",
diff --git a/sdks/python/src/opik/api_objects/dataset/converters.py b/sdks/python/src/opik/api_objects/dataset/converters.py
index 1257959f90..3a5479b01c 100644
--- a/sdks/python/src/opik/api_objects/dataset/converters.py
+++ b/sdks/python/src/opik/api_objects/dataset/converters.py
@@ -1,16 +1,34 @@
-import pandas as pd
 import json
 
-from typing import List, Callable, Any, Dict
+from typing import List, Callable, Any, Dict, TYPE_CHECKING
+import importlib.util
+import logging
+
+if TYPE_CHECKING:
+    import pandas as pd
 
 from . import dataset_item
 
 ItemConstructor = Callable[[Any], dataset_item.DatasetItem]
 
 
+LOGGER = logging.getLogger(__name__)
+IMPORT_PANDAS_ERROR = "The Python library Pandas is required for this method. You can install it with `pip install pandas`."
+
+
+def _check_if_pandas_is_available() -> None:
+    module_spec = importlib.util.find_spec("pandas")
+    if module_spec is None:
+        raise ImportError(IMPORT_PANDAS_ERROR)
+
+
 def to_pandas(
     items: List[dataset_item.DatasetItem], keys_mapping: Dict[str, str]
-) -> pd.DataFrame:
+) -> "pd.DataFrame":
+    _check_if_pandas_is_available()
+
+    import pandas as pd
+
     new_item_dicts = []
 
     for item in items:
@@ -38,10 +56,12 @@ def from_jsonl_file(
 
 
 def from_pandas(
-    dataframe: pd.DataFrame,
+    dataframe: "pd.DataFrame",
     keys_mapping: Dict[str, str],
     ignore_keys: List[str],
 ) -> List[dataset_item.DatasetItem]:
+    _check_if_pandas_is_available()
+
     result = []
     ignore_keys = [] if ignore_keys is None else ignore_keys
     for _, row in dataframe.iterrows():
diff --git a/sdks/python/src/opik/api_objects/dataset/dataset.py b/sdks/python/src/opik/api_objects/dataset/dataset.py
index 68ef044a95..b8bcb9b290 100644
--- a/sdks/python/src/opik/api_objects/dataset/dataset.py
+++ b/sdks/python/src/opik/api_objects/dataset/dataset.py
@@ -1,6 +1,6 @@
 import logging
 import json
-from typing import Optional, Any, List, Dict, Sequence, Set
+from typing import Optional, Any, List, Dict, Sequence, Set, TYPE_CHECKING
 
 from opik.rest_api import client as rest_api_client
 from opik.rest_api.types import dataset_item_write as rest_dataset_item
@@ -9,7 +9,9 @@
 
 from .. import constants
 from . import dataset_item, converters
-import pandas
+
+if TYPE_CHECKING:
+    import pandas as pd
 
 LOGGER = logging.getLogger(__name__)
 
@@ -162,8 +164,10 @@ def clear(self) -> None:
 
         self.delete(item_ids)
 
-    def to_pandas(self) -> pandas.DataFrame:
+    def to_pandas(self) -> "pd.DataFrame":
         """
+        Requires: `pandas` library to be installed.
+
         Convert the dataset to a pandas DataFrame.
 
         Returns:
@@ -298,11 +302,13 @@ def read_jsonl_from_file(
 
     def insert_from_pandas(
         self,
-        dataframe: pandas.DataFrame,
+        dataframe: "pd.DataFrame",
         keys_mapping: Optional[Dict[str, str]] = None,
         ignore_keys: Optional[List[str]] = None,
     ) -> None:
         """
+        Requires: `pandas` library to be installed.
+
         Args:
             dataframe: pandas dataframe
             keys_mapping: Dictionary that maps dataframe column names to dataset item field names.
diff --git a/sdks/python/src/opik/jsonable_encoder.py b/sdks/python/src/opik/jsonable_encoder.py
index 458ad2e204..7de2bdde1e 100644
--- a/sdks/python/src/opik/jsonable_encoder.py
+++ b/sdks/python/src/opik/jsonable_encoder.py
@@ -8,8 +8,6 @@
 from pathlib import PurePath
 from types import GeneratorType
 
-import numpy as np
-
 import opik.rest_api.core.datetime_utils as datetime_utils
 
 LOGGER = logging.getLogger(__name__)
@@ -55,13 +53,18 @@ def jsonable_encoder(obj: Any) -> Any:
                 encoded_list.append(jsonable_encoder(item))
             return encoded_list
 
-        if isinstance(obj, np.ndarray):
-            return jsonable_encoder(obj.tolist())
-
         for type_, encoder in _ENCODER_EXTENSIONS:
             if isinstance(obj, type_):
                 return jsonable_encoder(encoder(obj))
 
+        try:
+            import numpy as np
+
+            if isinstance(obj, np.ndarray):
+                return jsonable_encoder(obj.tolist())
+        except ImportError:
+            pass
+
     except Exception:
         LOGGER.debug("Failed to serialize object.", exc_info=True)
 
diff --git a/sdks/python/tests/unit/test_requirements.txt b/sdks/python/tests/unit/test_requirements.txt
new file mode 100644
index 0000000000..fb6c7ed7ec
--- /dev/null
+++ b/sdks/python/tests/unit/test_requirements.txt
@@ -0,0 +1 @@
+pandas


Code review comments:
diff_hunk: @@ -1,16 +1,34 @@
-import pandas as pd
 import json
 
-from typing import List, Callable, Any, Dict
+from typing import List, Callable, Any, Dict, TYPE_CHECKING
+import importlib.util
+import logging
+
+if TYPE_CHECKING:
+    import pandas as pd
 
 from . import dataset_item
 
 ItemConstructor = Callable[[Any], dataset_item.DatasetItem]
 
 
+LOGGER = logging.getLogger(__name__)
+IMPORT_PANDAS_ERROR = "The Python library Pandas is required for this method. You can install it with `pip install pandas`."
+
+
+def _check_if_pandas_is_available() -> None:
path: sdks/python/src/opik/api_objects/dataset/converters.py
text: @user1:
`_raise_if_pandas_is_unavailable()`

diff_hunk: @@ -55,13 +53,18 @@ def jsonable_encoder(obj: Any) -> Any:
                 encoded_list.append(jsonable_encoder(item))
             return encoded_list
 
-        if isinstance(obj, np.ndarray):
-            return jsonable_encoder(obj.tolist())
-
         for type_, encoder in _ENCODER_EXTENSIONS:
             if isinstance(obj, type_):
                 return jsonable_encoder(encoder(obj))
 
+        try:
path: sdks/python/src/opik/jsonable_encoder.py
text: @user1:
Let's import numpy at the top of the module 
```python
try:
    import numpy
except ImportError:
    numpy = None
```
and inside encoder function we'll have
```python
if numpy is not None:
    ...
```

There is not a big difference except for the fact that with the import inside this function, it will work significantly longer when called the first time. And I'd like to avoid this effect for this function.