Repository: Avaiga/taipy
Pull request:  feature/#398-expand-exposed-type-parameter
## What type of PR is this? (check all applicable)

- [ ] Refactor
- [x] Feature
- [ ] Bug Fix
- [ ] Optimization
- [ ] Documentation Update

## Description
We are also including pandas.DataFrame and numpy.ndarray as exposed type.

## Related Tickets & Documents

- Related Issue #398
- Closes #398

## How to reproduce the issue


## Checklist
_We encourage you to keep the code coverage percentage at 80% and above._

- [x] Does this solution meet the acceptance criteria of the related issue?
- [x] Is the related issue checklist completed?
- [x] Does this PR adds unit tests for the developed code? If not, why?
- [ ] End-to-End tests have been added or updated?
- [ ] Was the documentation updated, or a dedicated issue for documentation created? (If applicable)
- [ ] Is the release notes updated? (If applicable)


Problem statement:
Expand the `exposed_type` parameter of DataNode to accept actual Python types
**Description**

For now, the `exposed_type` parameter only accepts a string (`"pandas", "numpy") or a custom python class.

This ticket proposes to expand the parameter type to accept actual Python types for the predefined types:
- `numpy.array`
- `pandas.DataFrame`

Hints from issue discussion:
Can i help out here ?
Thank you @sqrt676. I assigned the ticket to you.

Let us know if you need more information on this.
Thanks @trgiangdo , will ask for information wherever needed.
Hello @sqrt676, any update on this issue?
@trgiangdo can I take this one?
Absolutely. I will assign the issue to you @ranjanmangla1.

If you have any question on the topic, feel free to ask here.
@trgiangdo just want to ask, since this issue is almost 18 months old now, what changes have been to taipy? and since modin has been deprecated in taipy, is there a need to provide support for modin.pandas.DataFrame?
Oh, you're right. Let me update the description. Other than that, there hasn't been any other change that affect this issue.
This issue has been labelled as "🥶Waiting for contributor" because it has been inactive for more than 14 days. If you would like to continue working on this issue, please add another comment or create a PR that links to this issue. If a PR has already been created which refers to this issue, then you should explicitly mention this issue in the relevant PR. Otherwise, you will be unassigned in 14 days. For more information please refer to the contributing guidelines.
This issue has been unassigned automatically because it has been marked as "🥶Waiting for contributor" for more than 14 days with no activity.

Commit message: Merge branch 'develop' into feature/#398-expand-exposed-type-parameter

Patch to review:
diff --git a/Pipfile b/Pipfile
index e308f18c4d..da1ab78ea4 100644
--- a/Pipfile
+++ b/Pipfile
@@ -36,6 +36,7 @@ boto3 = "==1.29.1"
 watchdog = "==4.0.0"
 charset-normalizer = "==3.3.2"
 numpy = "<2.0.0"
+pre-commit = "*"
 
 [dev-packages]
 freezegun = "*"
diff --git a/git b/git
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/taipy/core/config/data_node_config.py b/taipy/core/config/data_node_config.py
index 0b095fc297..a25be23187 100644
--- a/taipy/core/config/data_node_config.py
+++ b/taipy/core/config/data_node_config.py
@@ -14,6 +14,9 @@
 from datetime import timedelta
 from typing import Any, Callable, Dict, List, Optional, Union
 
+import numpy
+import pandas
+
 from taipy.common.config import Config
 from taipy.common.config._config import _Config
 from taipy.common.config.common._config_blocker import _ConfigBlocker
@@ -71,11 +74,17 @@ class DataNodeConfig(Section):
     _EXPOSED_TYPE_PANDAS = "pandas"
     _EXPOSED_TYPE_MODIN = "modin"  # Deprecated in favor of pandas since 3.1.0
     _EXPOSED_TYPE_NUMPY = "numpy"
+    _EXPOSED_TYPE_PANDAS_DATAFRAME = pandas.DataFrame
+    _EXPOSED_TYPE_NUMPY_NDARRAY = numpy.ndarray
+    _EXPOSED_TYPE_LIST = List
     _DEFAULT_EXPOSED_TYPE = _EXPOSED_TYPE_PANDAS
 
     _ALL_EXPOSED_TYPES = [
         _EXPOSED_TYPE_PANDAS,
         _EXPOSED_TYPE_NUMPY,
+        _EXPOSED_TYPE_PANDAS_DATAFRAME,
+        _EXPOSED_TYPE_NUMPY_NDARRAY,
+        _EXPOSED_TYPE_LIST,
     ]
 
     _OPTIONAL_ENCODING_PROPERTY = "encoding"
diff --git a/taipy/core/data/_tabular_datanode_mixin.py b/taipy/core/data/_tabular_datanode_mixin.py
index 54e5d04947..bd5ecd9acc 100644
--- a/taipy/core/data/_tabular_datanode_mixin.py
+++ b/taipy/core/data/_tabular_datanode_mixin.py
@@ -26,6 +26,9 @@ class _TabularDataNodeMixin(object):
     _EXPOSED_TYPE_PANDAS = "pandas"
     _EXPOSED_TYPE_MODIN = "modin"  # Deprecated in favor of pandas since 3.1.0
     _VALID_STRING_EXPOSED_TYPES = [_EXPOSED_TYPE_PANDAS, _EXPOSED_TYPE_NUMPY]
+    _VALID_OTHER_EXPOSED_TYPES = [pd.DataFrame, np.ndarray]
+    _EXPOSED_TYPE_PANDAS_DATAFRAME = pd.DataFrame
+    _EXPOSED_TYPE_NUMPY_NDARRAY = np.ndarray
 
     def __init__(self, **kwargs) -> None:
         self._decoder: Union[Callable, Any]
@@ -44,11 +47,14 @@ def __init__(self, **kwargs) -> None:
         if callable(custom_encoder):
             self._encoder = custom_encoder
 
-
     def _convert_data_to_dataframe(self, exposed_type: Any, data: Any) -> Union[pd.DataFrame, pd.Series]:
-        if exposed_type == self._EXPOSED_TYPE_PANDAS and isinstance(data, (pd.DataFrame, pd.Series)):
+        if exposed_type in [self._EXPOSED_TYPE_PANDAS, self._EXPOSED_TYPE_PANDAS_DATAFRAME] and isinstance(
+            data, (pd.DataFrame, pd.Series)
+        ):
             return data
-        elif exposed_type == self._EXPOSED_TYPE_NUMPY and isinstance(data, np.ndarray):
+        elif exposed_type in [self._EXPOSED_TYPE_NUMPY, self._EXPOSED_TYPE_NUMPY_NDARRAY] and isinstance(
+            data, np.ndarray
+        ):
             return pd.DataFrame(data)
         elif isinstance(data, list) and not isinstance(exposed_type, str):
             return pd.DataFrame.from_records([self._encoder(row) for row in data])
diff --git a/taipy/core/data/csv.py b/taipy/core/data/csv.py
index f61e1b1b55..1516584890 100644
--- a/taipy/core/data/csv.py
+++ b/taipy/core/data/csv.py
@@ -134,9 +134,9 @@ def _read_from_path(self, path: Optional[str] = None, **read_kwargs) -> Any:
             path = self._path
 
         properties = self.properties
-        if properties[self._EXPOSED_TYPE_PROPERTY] == self._EXPOSED_TYPE_PANDAS:
+        if properties[self._EXPOSED_TYPE_PROPERTY] in [self._EXPOSED_TYPE_PANDAS, self._EXPOSED_TYPE_PANDAS_DATAFRAME]:
             return self._read_as_pandas_dataframe(path=path)
-        if properties[self._EXPOSED_TYPE_PROPERTY] == self._EXPOSED_TYPE_NUMPY:
+        if properties[self._EXPOSED_TYPE_PROPERTY] in [self._EXPOSED_TYPE_NUMPY, self._EXPOSED_TYPE_NUMPY_NDARRAY]:
             return self._read_as_numpy(path=path)
         return self._read_as(path=path)
 
diff --git a/taipy/core/data/excel.py b/taipy/core/data/excel.py
index f7bb0ddfd2..0922a83acd 100644
--- a/taipy/core/data/excel.py
+++ b/taipy/core/data/excel.py
@@ -137,7 +137,7 @@ def write_with_column_names(self, data: Any, columns: List[str] = None, editor_i
 
     @staticmethod
     def _check_exposed_type(exposed_type):
-        if isinstance(exposed_type, str):
+        if isinstance(exposed_type, str) or exposed_type in [pd.DataFrame, np.ndarray]:
             _TabularDataNodeMixin._check_exposed_type(exposed_type)
         elif isinstance(exposed_type, list):
             for t in exposed_type:
@@ -154,18 +154,18 @@ def _read_from_path(self, path: Optional[str] = None, **read_kwargs) -> Any:
             path = self._path
 
         exposed_type = self.properties[self._EXPOSED_TYPE_PROPERTY]
-        if exposed_type == self._EXPOSED_TYPE_PANDAS:
+        if exposed_type in [self._EXPOSED_TYPE_PANDAS, self._EXPOSED_TYPE_PANDAS_DATAFRAME]:
             return self._read_as_pandas_dataframe(path=path)
-        if exposed_type == self._EXPOSED_TYPE_NUMPY:
+        if exposed_type in [self._EXPOSED_TYPE_NUMPY, self._EXPOSED_TYPE_NUMPY_NDARRAY]:
             return self._read_as_numpy(path=path)
         return self._read_as(path=path)
 
     def _read_sheet_with_exposed_type(
         self, path: str, sheet_exposed_type: str, sheet_name: str
     ) -> Optional[Union[np.ndarray, pd.DataFrame]]:
-        if sheet_exposed_type == self._EXPOSED_TYPE_NUMPY:
+        if sheet_exposed_type in [self._EXPOSED_TYPE_NUMPY, self._EXPOSED_TYPE_NUMPY_NDARRAY]:
             return self._read_as_numpy(path, sheet_name)
-        elif sheet_exposed_type == self._EXPOSED_TYPE_PANDAS:
+        elif sheet_exposed_type in [self._EXPOSED_TYPE_PANDAS, self._EXPOSED_TYPE_PANDAS_DATAFRAME]:
             return self._read_as_pandas_dataframe(path, sheet_name)  # type: ignore
         return None
 
@@ -202,6 +202,10 @@ def _read_as(self, path: str):
                         sheet_exposed_type = exposed_type.get(sheet_name, self._EXPOSED_TYPE_PANDAS)
                     elif isinstance(exposed_type, List):
                         sheet_exposed_type = exposed_type[i]
+                    elif exposed_type == np.ndarray:
+                        sheet_exposed_type = self._EXPOSED_TYPE_NUMPY
+                    elif exposed_type == pd.DataFrame:
+                        sheet_exposed_type = self._EXPOSED_TYPE_PANDAS
 
                     if isinstance(sheet_exposed_type, str):
                         sheet_data = self._read_sheet_with_exposed_type(path, sheet_exposed_type, sheet_name)
diff --git a/taipy/core/data/parquet.py b/taipy/core/data/parquet.py
index 698aae58a5..395e825f34 100644
--- a/taipy/core/data/parquet.py
+++ b/taipy/core/data/parquet.py
@@ -227,9 +227,9 @@ def _read_from_path(self, path: Optional[str] = None, **read_kwargs) -> Any:
         return self._do_read_from_path(path, properties[self._EXPOSED_TYPE_PROPERTY], kwargs)
 
     def _do_read_from_path(self, path: str, exposed_type: str, kwargs: Dict) -> Any:
-        if exposed_type == self._EXPOSED_TYPE_PANDAS:
+        if exposed_type in [self._EXPOSED_TYPE_PANDAS, self._EXPOSED_TYPE_PANDAS_DATAFRAME]:
             return self._read_as_pandas_dataframe(path, kwargs)
-        if exposed_type == self._EXPOSED_TYPE_NUMPY:
+        if exposed_type in [self._EXPOSED_TYPE_NUMPY, self._EXPOSED_TYPE_NUMPY_NDARRAY]:
             return self._read_as_numpy(path, kwargs)
         return self._read_as(path, kwargs)
 
@@ -249,4 +249,3 @@ def _append(self, data: Any):
 
     def _write(self, data: Any):
         self._write_with_kwargs(data)
-
diff --git a/tests/core/data/test_csv_data_node.py b/tests/core/data/test_csv_data_node.py
index 360fd62a56..fbc00b22d4 100644
--- a/tests/core/data/test_csv_data_node.py
+++ b/tests/core/data/test_csv_data_node.py
@@ -157,6 +157,16 @@ def test_pandas_exposed_type(self):
         dn = CSVDataNode("foo", Scope.SCENARIO, properties={"path": path, "exposed_type": "pandas"})
         assert isinstance(dn.read(), pd.DataFrame)
 
+    def test_pandas_dataframe_exposed_type(self):
+        path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example.csv")
+        dn = CSVDataNode("foo", Scope.SCENARIO, properties={"path": path, "exposed_type": pd.DataFrame})
+        assert isinstance(dn.read(), pd.DataFrame)
+
+    def test_numpy_ndarray_exposed_type(self):
+        path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example.csv")
+        dn = CSVDataNode("foo", Scope.SCENARIO, properties={"path": path, "exposed_type": np.ndarray})
+        assert isinstance(dn.read(), np.ndarray)
+
     def test_raise_error_invalid_exposed_type(self):
         path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example.csv")
         with pytest.raises(InvalidExposedType):
diff --git a/tests/core/data/test_excel_data_node.py b/tests/core/data/test_excel_data_node.py
index 734f72b2d8..6cc7c62519 100644
--- a/tests/core/data/test_excel_data_node.py
+++ b/tests/core/data/test_excel_data_node.py
@@ -302,6 +302,26 @@ def test_pandas_exposed_type(self):
         data = dn.read()
         assert isinstance(data, pd.DataFrame)
 
+    def test_pandas_dataframe_exposed_type(self):
+        path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example.xlsx")
+        dn = ExcelDataNode(
+            "foo",
+            Scope.SCENARIO,
+            properties={"default_path": path, "exposed_type": pd.DataFrame, "sheet_name": "Sheet1"},
+        )
+        assert dn.properties["exposed_type"] == pd.DataFrame
+        data = dn.read()
+        assert isinstance(data, pd.DataFrame)
+
+    def test_numpy_ndarray_exposed_type(self):
+        path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example.xlsx")
+        dn = ExcelDataNode(
+            "foo", Scope.SCENARIO, properties={"default_path": path, "exposed_type": np.ndarray, "sheet_name": "Sheet1"}
+        )
+        assert dn.properties["exposed_type"] == np.ndarray
+        data = dn.read()
+        assert isinstance(data, np.ndarray)
+
     def test_complex_exposed_type_dict(self):
         # ["Sheet1", "Sheet2", "Sheet3", "Sheet4", "Sheet5"]
         path = os.path.join(pathlib.Path(__file__).parent.resolve(), "data_sample/example_4.xlsx")
diff --git a/tests/core/data/test_parquet_data_node.py b/tests/core/data/test_parquet_data_node.py
index 02dd247b69..109ae3c90e 100644
--- a/tests/core/data/test_parquet_data_node.py
+++ b/tests/core/data/test_parquet_data_node.py
@@ -98,6 +98,20 @@ def test_create(self):
         assert isinstance(dn_1, ParquetDataNode)
         assert dn_1.properties["exposed_type"] == MyCustomObject
 
+        parquet_dn_config_2 = Config.configure_parquet_data_node(
+            id="bar", default_path=path, compression=compression, exposed_type=np.ndarray
+        )
+        dn_2 = _DataManagerFactory._build_manager()._create_and_set(parquet_dn_config_2, None, None)
+        assert isinstance(dn_2, ParquetDataNode)
+        assert dn_2.properties["exposed_type"] == np.ndarray
+
+        parquet_dn_config_3 = Config.configure_parquet_data_node(
+            id="bar", default_path=path, compression=compression, exposed_type=pd.DataFrame
+        )
+        dn_3 = _DataManagerFactory._build_manager()._create_and_set(parquet_dn_config_3, None, None)
+        assert isinstance(dn_3, ParquetDataNode)
+        assert dn_3.properties["exposed_type"] == pd.DataFrame
+
         with pytest.raises(InvalidConfigurationId):
             dn = ParquetDataNode("foo bar", Scope.SCENARIO, properties={"path": path, "name": "super name"})
 


Code review comments:
path: git
text: @user1:
I believe this file is redundant

diff_hunk: @@ -71,11 +74,17 @@ class DataNodeConfig(Section):
     _EXPOSED_TYPE_PANDAS = "pandas"
     _EXPOSED_TYPE_MODIN = "modin"  # Deprecated in favor of pandas since 3.1.0
     _EXPOSED_TYPE_NUMPY = "numpy"
+    _EXPOSED_TYPE_PANDAS_DATAFRAME = pandas.DataFrame
+    _EXPOSED_TYPE_NUMPY_NDARRAY = numpy.ndarray
+    _EXPOSED_TYPE_LIST = List
path: taipy/core/config/data_node_config.py
text: @user1:
The `_EXPOSED_TYPE_LIST` is defined here but is not used anywhere.

Besides, it is vague to have `List` as an exposed type, because it can be a list of anything (list of string, dataframe, ndarray, objects, etc).

diff_hunk: @@ -36,6 +36,7 @@ boto3 = "==1.29.1"
 watchdog = "==4.0.0"
 charset-normalizer = "==3.3.2"
 numpy = "<2.0.0"
+pre-commit = "*"
path: Pipfile
text: @user1:
This change is redundant since `pre-commit` is already in dev-packages section (line 58)

diff_hunk: @@ -26,6 +26,9 @@ class _TabularDataNodeMixin(object):
     _EXPOSED_TYPE_PANDAS = "pandas"
     _EXPOSED_TYPE_MODIN = "modin"  # Deprecated in favor of pandas since 3.1.0
     _VALID_STRING_EXPOSED_TYPES = [_EXPOSED_TYPE_PANDAS, _EXPOSED_TYPE_NUMPY]
+    _VALID_OTHER_EXPOSED_TYPES = [pd.DataFrame, np.ndarray]
path: taipy/core/data/_tabular_datanode_mixin.py
text: @user1:
The `_VALID_STRING_EXPOSED_TYPES` is used when serialize and deserialize the `exposed_type` property.

The `_VALID_OTHER_EXPOSED_TYPES` is defined here but not used anywhere.