Repository: Avaiga/taipy
Pull request: Added a function to support data conversion if type is not supported by the taipy.gui.data module
Resolves: #457 

So there was this issue where we thought of giving developer option to change unsupported data format to a supported type
For this I have made two functions in class Gui namely "on_invalid_data" and "set_on_invalid_data_callback"

The Second function "set_on_invalid_data_callback" accepts a callback as parameter and uses this function to transform data into supported format. If user doesn't provide a callback, it works as it used to. Nothing would change. 

Besides in data_accessor, I have modified _InvalidDataAccessor. For all methods like get_data, get_col_types, etc. It calls the first function "on_invalid_data" to transform it into a valid data type. If no callback present it will work as it used to work. 

Problem statement:
Tabular Data: support a callback to change unsupported data type to supported one
**What would that feature address**
display unsupported data type

***Description of the ideal solution***
If a dataframe holds a column with unsupported data, we could handle a user callback that would transform that column to a supported type

***Caveats***
performance ?

***Other options***
Raise an error (as of today)

**Acceptance Criteria**
- [ ] Ensure new code is unit tested, and check code coverage is at least 90%
- [ ] Propagate any change on the demos and run all of them to ensure there is no breaking change
- [ ] Ensure any change is well documented

Hints from issue discussion:
@FredLL-Avaiga to clarify, we're creating a user callback that alerts users of unsupported type errors and then suggests them to convert the data into a supported type as well? (which the user may accept or declineand fix the data by himself)
That is not what I had in mind.
The objective is to allow the dev to define a callback that would transform a specific data from an unknown data-type to a supported data-type (that would fit in a DataFrame)
At the moment, we support datetime (ie transform them to str) so that the Dataframe can be jsonified
### ![New Quest! image](https://quine-assets.s3.amazonaws.com/assets/quest-bot/quest-generic/opening28.svg) New Quest!

*A new Quest has been launched in @Avaiga’s repo.
Merge a PR that solves this issue to loot the Quest and earn your reward.*

---

Some loot has been stashed in this issue to reward the solver!

🗡 Comment `@quest-bot embark` to check-in for this Quest and start solving the issue. Other solvers will be notified!

⚔️ When you submit a PR, comment `@quest-bot loot #457` to link your PR to this Quest.

Questions? Check out [the docs](https://docs.quira.sh/for-developers/quests/solver-quests).
Can i work on this?

@ScalarJerk Do you aim to work on this issue? 

@eleensmathew You have already been assigned to another issue.
can i work on this issue? 
@Ujj1225, I am assigning you this issue. Thank you for your contribution!
@FlorianJacta  @FredLL-Avaiga  
can i get more insights on this? 
I checked into the codebase and its huge. Can you tell me in which part of the code is the data is loaded, inspected, or converted into a dataframe?

In a file _data_manager.py
I found a __create method, here there is a point where data_node_config._properties.copy() is used. I feel this might be the right place to handle unsupported data types by adding a check for data types in the props. Since the data nodes are created based on configurations, the place to handle unsupported data types might be during or right after the node's creation or configuration process, right? 

Can you please provide me more insights on this? 
I will let the R&D answer you. Maybe, @FabienLelaquais can also jump on this discussion
In the code you can check how we use data_accessors to transform different types of data into tabular data that is handled by our front end component.
The indirections are done in the class [_DataAccessors](https://github.com/Avaiga/taipy/blob/b2b7f8c8b0d677e58f8db646814833deb3d9991c/taipy/gui/data/data_accessor.py#L97)

In that class, we return an [_InvalidDataAccessor](https://github.com/Avaiga/taipy/blob/b2b7f8c8b0d677e58f8db646814833deb3d9991c/taipy/gui/data/data_accessor.py#L141) when we cannot handle the type of data.

The aim of this issue is to add a callback to Gui (like on_change or on_action) to let the developer change an unsupported data into a supported data type. I would propose `on_invalid_data` but I guess @FabienLelaquais might have a better proposition

Commit message: Linter (2)

Patch to review:
diff --git a/contributors.txt b/contributors.txt
new file mode 100644
index 0000000000..fc43f630b8
--- /dev/null
+++ b/contributors.txt
@@ -0,0 +1,22 @@
+jrobinAV
+FabienLelaquais
+florian-vuillemot
+FredLL-Avaiga
+dinhlongviolin1
+joaoandre-avaiga
+toan-quach
+trgiangdo
+gmarabout
+tsuu2092
+arcanaxion
+Dr-Irv
+enarroied
+bobbyshermi
+Forchapeatl
+yarikoptic
+Luke-0162
+Satoshi-Sh
+kushal34712
+DeepanshuProgrammer
+Ujj1225
+
diff --git a/taipy/gui/data/data_accessor.py b/taipy/gui/data/data_accessor.py
index d20b5ea8d3..23d85445c0 100644
--- a/taipy/gui/data/data_accessor.py
+++ b/taipy/gui/data/data_accessor.py
@@ -110,13 +110,14 @@ def __init__(self, gui: "Gui") -> None:
         self._register(_NumpyDataAccessor)
 
     def _register(self, cls: t.Type[_DataAccessor]) -> None:
+        """Register a new DataAccessor type."""
         if not inspect.isclass(cls):
-            raise AttributeError("The argument of 'DataAccessors.register' should be a class")
+            raise AttributeError("The argument of 'DataAccessors.register()' should be a class")
         if not issubclass(cls, _DataAccessor):
             raise TypeError(f"Class {cls.__name__} is not a subclass of DataAccessor")
         classes = cls.get_supported_classes()
         if not classes:
-            raise TypeError(f"method {cls.__name__}.get_supported_classes returned an invalid value")
+            raise TypeError(f"{cls.__name__}.get_supported_classes() returned an invalid value")
         # check existence
         inst: t.Optional[_DataAccessor] = None
         for cl in classes:
@@ -132,11 +133,21 @@ def _register(self, cls: t.Type[_DataAccessor]) -> None:
                 for cl in classes:
                     self.__access_4_type[cl] = inst  # type: ignore
 
+    def _unregister(self, cls: t.Type[_DataAccessor]) -> None:
+        """Unregisters a DataAccessor type."""
+        if cls in self.__access_4_type:
+            del self.__access_4_type[cls]
+
     def __get_instance(self, value: _TaipyData) -> _DataAccessor:  # type: ignore
         value = value.get() if isinstance(value, _TaipyData) else value
         access = self.__access_4_type.get(type(value))
         if access is None:
             if value is not None:
+                converted_value = type(self.__gui)._convert_unsupported_data(value)
+                if converted_value is not None:
+                    access = self.__access_4_type.get(type(converted_value))
+                    if access is not None:
+                        return access
                 _warn(f"Can't find Data Accessor for type {str(type(value))}.")
             return self.__invalid_data_accessor
         return access
diff --git a/taipy/gui/gui.py b/taipy/gui/gui.py
index 0a92334d9a..e25a34ed6d 100644
--- a/taipy/gui/gui.py
+++ b/taipy/gui/gui.py
@@ -174,6 +174,9 @@ class Gui:
 
     __content_providers: t.Dict[type, t.Callable[..., str]] = {}
 
+    # See set_unsupported_data_converter()
+    __unsupported_data_converter: t.Optional[t.Callable] = None
+
     def __init__(
         self,
         page: t.Optional[t.Union[str, Page]] = None,
@@ -447,7 +450,6 @@ def register_content_provider(content_type: type, content_provider: t.Callable[.
         Arguments:
             content_type: The type of the content that triggers the content provider.
             content_provider: The function that converts content of type *type* into an HTML string.
-
         """  # noqa: E501
         if Gui.__content_providers.get(content_type):
             _warn(f"The type {content_type} is already associated with a provider.")
@@ -1015,17 +1017,17 @@ def __upload_files(self):
                 complete = part == total - 1
 
         # Extract upload path (when single file is selected, path="" does not change the path)
-        upload_root = os.path.abspath( self._get_config( "upload_folder", tempfile.gettempdir() ) )
-        upload_path = os.path.abspath( os.path.join( upload_root, os.path.dirname(path) ) )
-        if upload_path.startswith( upload_root ):
-            upload_path = Path( upload_path ).resolve()
-            os.makedirs( upload_path, exist_ok=True )
+        upload_root = os.path.abspath(self._get_config("upload_folder", tempfile.gettempdir()))
+        upload_path = os.path.abspath(os.path.join(upload_root, os.path.dirname(path)))
+        if upload_path.startswith(upload_root):
+            upload_path = Path(upload_path).resolve()
+            os.makedirs(upload_path, exist_ok=True)
             # Save file into upload_path directory
             file_path = _get_non_existent_file_path(upload_path, secure_filename(file.filename))
-            file.save( os.path.join( upload_path, (file_path.name + suffix) ) )
+            file.save(os.path.join(upload_path, (file_path.name + suffix)))
         else:
             _warn(f"upload files: Path {path} points outside of upload root.")
-            return("upload files: Path part points outside of upload root.", 400)
+            return ("upload files: Path part points outside of upload root.", 400)
 
         if complete:
             if part > 0:
@@ -1061,9 +1063,7 @@ def __upload_files(self):
                     if not _is_function(file_fn):
                         file_fn = _getscopeattr(self, on_upload_action)
                     if _is_function(file_fn):
-                        self._call_function_with_state(
-                            t.cast(t.Callable, file_fn), ["file_upload", {"args": [data]}]
-                        )
+                        self._call_function_with_state(t.cast(t.Callable, file_fn), ["file_upload", {"args": [data]}])
                 else:
                     setattr(self._bindings(), var_name, newvalue)
         return ("", 200)
@@ -1143,6 +1143,41 @@ def __update_state_context(self, payload: dict):
             for var, val in state_context.items():
                 self._update_var(var, val, True, forward=False)
 
+
+    @staticmethod
+    def set_unsupported_data_converter(converter: t.Optional[t.Callable[[t.Any], t.Any]]) -> None:
+        """Set a custom converter for unsupported data types.
+
+        This function allows specifying a custom conversion function for data types that cannot
+        be serialized automatically. When Taipy GUI encounters an unsupported type in the data
+        being serialized, it will invoke the provided *converter* function. The returned value
+        from this function will then be serialized if possible.
+
+        Arguments:
+            converter: A function that converts a value with an unsupported data type (the only
+              parameter to the function) into data with a supported data type (the returned value
+              from the function).</br>
+              If set to `None`, it removes any existing converter.
+        """
+        Gui.__unsupported_data_converter = converter
+
+    @staticmethod
+    def _convert_unsupported_data(value: t.Any) -> t.Optional[t.Any]:
+        """
+        Handles unsupported data by invoking the converter if there is one.
+
+        Arguments:
+            value: The unsupported data encountered.
+
+        Returns:
+            The transformed data or None if no transformation is possible.
+        """
+        try:
+            return Gui.__unsupported_data_converter(value) if callable(Gui.__unsupported_data_converter) else None  # type: ignore
+        except Exception as e:
+            _warn(f"Error transforming data: {str(e)}")
+            return None
+
     def __request_data_update(self, var_name: str, payload: t.Any) -> None:
         # Use custom attrgetter function to allow value binding for _MapDict
         newvalue = _getscopeattr_drill(self, var_name)
@@ -2637,6 +2672,7 @@ def __bind_default_function(self):
             self.__bind_local_func("on_exception")
             self.__bind_local_func("on_status")
             self.__bind_local_func("on_user_content")
+            self.__bind_local_func("on_invalid_data")
 
     def __register_blueprint(self):
         # add en empty main page if it is not defined
diff --git a/tests/gui/data/test_accessors.py b/tests/gui/data/test_accessors.py
new file mode 100644
index 0000000000..e24cb9c12d
--- /dev/null
+++ b/tests/gui/data/test_accessors.py
@@ -0,0 +1,79 @@
+import typing as t
+from unittest.mock import Mock
+
+from taipy.gui import Gui
+from taipy.gui.data.data_accessor import (
+    _DataAccessor,
+    _DataAccessors,
+    _InvalidDataAccessor,
+)
+from taipy.gui.data.data_format import _DataFormat
+from taipy.gui.utils.types import _TaipyData
+
+
+class MyDataAccessor(_DataAccessor):
+    @staticmethod
+    def get_supported_classes() -> t.List[t.Type]:
+        return [int]
+
+    def get_data(
+        self,
+        var_name: str,
+        value: t.Any,
+        payload: t.Dict[str, t.Any],
+        data_format: _DataFormat,
+    ) -> t.Dict[str, t.Any]:
+        return {"value": 2 * int(value)}
+
+    def get_col_types(self, var_name: str, value: t.Any) -> t.Dict[str, str]:  # type: ignore
+        pass
+
+    def to_pandas(self, value: t.Any) -> t.Union[t.List[t.Any], t.Any]:
+        pass
+
+    def on_edit(self, value: t.Any, payload: t.Dict[str, t.Any]) -> t.Optional[t.Any]:
+        pass
+
+    def on_delete(self, value: t.Any, payload: t.Dict[str, t.Any]) -> t.Optional[t.Any]:
+        pass
+
+    def on_add(
+        self,
+        value: t.Any,
+        payload: t.Dict[str, t.Any],
+        new_row: t.Optional[t.List[t.Any]] = None,
+    ) -> t.Optional[t.Any]:
+        pass
+
+    def to_csv(self, var_name: str, value: t.Any) -> t.Optional[str]:
+        pass
+
+
+def mock_taipy_data(value):
+    """Helper to mock _TaipyData objects."""
+    mock_data = Mock(spec=_TaipyData)
+    mock_data.get.return_value = value
+    return mock_data
+
+
+def test_custom_accessor(gui: Gui):
+    """Test if get_data() uses the correct accessor."""
+    data_accessors = _DataAccessors(gui)
+    data = mock_taipy_data(123)
+
+    # Testing when accessor is not registered
+    data_accessor = data_accessors._DataAccessors__get_instance(mock_taipy_data)  # type: ignore
+    assert isinstance(
+        data_accessor, _InvalidDataAccessor
+    ), f"Expected _InvalidDataAccessor but got {type(data_accessor)}"
+    result = data_accessors.get_data("var_name", data, {})
+    assert result == {}
+
+    # Testing when accessor is not registered
+    data_accessors._register(MyDataAccessor)
+
+    result = data_accessors.get_data("var_name", data, {})
+    assert isinstance(result, dict)
+    assert result["value"] == 246
+
+    data_accessors._unregister(MyDataAccessor)
diff --git a/tests/gui/data/test_unsupported_data_converter.py b/tests/gui/data/test_unsupported_data_converter.py
new file mode 100644
index 0000000000..54db6ce34f
--- /dev/null
+++ b/tests/gui/data/test_unsupported_data_converter.py
@@ -0,0 +1,73 @@
+import logging
+
+import pytest
+
+from taipy.gui import Gui
+
+
+def test_handle_invalid_data_no_callback():
+    result = Gui._convert_unsupported_data("invalid_data")
+    assert result is None
+
+
+def test_unsupported_data_converter_returns_none():
+    def convert(value):
+        return None  # Simulates a failed transformation
+
+    Gui.set_unsupported_data_converter(convert)
+
+    result = Gui._convert_unsupported_data("invalid_data")
+    assert result is None
+
+    # Reset converter
+    Gui.set_unsupported_data_converter(None)
+
+
+def test_unsupported_data_converter_applied():
+    def convert(value):
+        return "converted"  # Successful transformation
+
+    Gui.set_unsupported_data_converter(convert)
+
+    result = Gui._convert_unsupported_data("raw data")
+    assert result == "converted"
+
+    # Reset converter
+    Gui.set_unsupported_data_converter(None)
+
+
+def test_unsupported_data_converter_raises_exception(capfd, monkeypatch):
+    def convert(value):
+        raise ValueError("Conversion failure")  # Simulate an error
+
+    def mock_warn(message: str):
+        logging.warning(message)  # Ensure the warning goes to stderr.
+
+    # Patch the _warn function inside the taipy.gui._warnings module.
+    monkeypatch.setattr("taipy.gui._warnings._warn", mock_warn)
+
+    Gui.set_unsupported_data_converter(convert)
+
+    result = Gui._convert_unsupported_data("raw data")
+
+    out, _ = capfd.readouterr()
+
+    assert result is None  # Should return None on exception
+    assert "Error transforming data: Transformation error"
+
+    # Reset converter
+    Gui.set_unsupported_data_converter(None)
+
+
+@pytest.mark.parametrize("input_data", [None, 123, [], {}, set()])
+def test_unsupported_data_converter_with_various_inputs(input_data):
+    def convert(value):
+        return "converted"  # Always returns valid data
+
+    Gui.set_unsupported_data_converter(convert)
+
+    result = Gui._convert_unsupported_data(input_data)
+    assert result == "converted"  # Transformed correctly for all inputs
+
+    # Reset converter
+    Gui.set_unsupported_data_converter(None)


Code review comments:
path: contributors.txt
text: @user1:
how come ? has this file moved ?

@user2:
This should no have been created indeed

diff_hunk: @@ -0,0 +1,79 @@
+import typing as t
+from unittest.mock import Mock
+
+from taipy.gui import Gui
+from taipy.gui.data.data_accessor import (
+    _DataAccessor,
+    _DataAccessors,
+    _InvalidDataAccessor,
+)
+from taipy.gui.data.data_format import _DataFormat
+from taipy.gui.utils.types import _TaipyData
+
+
+class MyDataAccessor(_DataAccessor):
+    @staticmethod
+    def get_supported_classes() -> t.List[t.Type]:
+        return [int]
+
+    def get_data(
+        self,
+        var_name: str,
+        value: t.Any,
+        payload: t.Dict[str, t.Any],
+        data_format: _DataFormat,
+    ) -> t.Dict[str, t.Any]:
+        return {"value": 2 * int(value)}
+
+    def get_col_types(self, var_name: str, value: t.Any) -> t.Dict[str, str]:  # type: ignore
+        pass
+
+    def to_pandas(self, value: t.Any) -> t.Union[t.List[t.Any], t.Any]:
+        pass
+
+    def on_edit(self, value: t.Any, payload: t.Dict[str, t.Any]) -> t.Optional[t.Any]:
+        pass
+
+    def on_delete(self, value: t.Any, payload: t.Dict[str, t.Any]) -> t.Optional[t.Any]:
+        pass
+
+    def on_add(
+        self,
+        value: t.Any,
+        payload: t.Dict[str, t.Any],
+        new_row: t.Optional[t.List[t.Any]] = None,
+    ) -> t.Optional[t.Any]:
+        pass
+
+    def to_csv(self, var_name: str, value: t.Any) -> t.Optional[str]:
+        pass
+
+
+def mock_taipy_data(value):
+    """Helper to mock _TaipyData objects."""
+    mock_data = Mock(spec=_TaipyData)
+    mock_data.get.return_value = value
+    return mock_data
+
+
+def test_custom_accessor(gui: Gui):
+    """Test if get_data() uses the correct accessor."""
+    data_accessors = _DataAccessors(gui)
+    data = mock_taipy_data(123)
+
+    # Testing when accessor is not registered
+    data_accessor = data_accessors._DataAccessors__get_instance(mock_taipy_data)  # type: ignore
+    assert isinstance(
+        data_accessor, _InvalidDataAccessor
+    ), f"Expected _InvalidDataAccessor but got {type(data_accessor)}"
+    result = data_accessors.get_data("var_name", data, {})
+    assert result == {}
+
+    # Testing when accessor is not registered
path: tests/gui/data/test_accessors.py
text: @user1:
```suggestion
    # Testing when accessor is registered
```

diff_hunk: @@ -1143,6 +1143,41 @@ def __update_state_context(self, payload: dict):
             for var, val in state_context.items():
                 self._update_var(var, val, True, forward=False)
 
+
+    @staticmethod
+    def set_unsupported_data_converter(converter: t.Optional[t.Callable[[t.Any], t.Any]]) -> None:
+        """Set a custom converter for unsupported data types.
+
+        This function allows specifying a custom conversion function for data types that cannot
+        be serialized automatically. When Taipy GUI encounters an unsupported type in the data
+        being serialized, it will invoke the provided *converter* function. The returned value
+        from this function will then be serialized if possible.
+
+        Arguments:
+            converter: A function that converts a value with an unsupported data type (the only
+              parameter to the function) into data with a supported data type (the returned value
+              from the function).</br>
+              If set to `None`, it removes any existing converter.
+        """
+        Gui.__unsupported_data_converter = converter
+
+    @staticmethod
+    def _convert_unsupported_data(value: t.Any) -> t.Optional[t.Any]:
+        """
+        Handles unsupported data by invoking the converter if there is one.
+
+        Arguments:
+            value: The unsupported data encountered.
+
+        Returns:
+            The transformed data or None if no transformation is possible.
+        """
+        try:
+            return Gui.__unsupported_data_converter(value) if callable(Gui.__unsupported_data_converter) else None  # type: ignore
path: taipy/gui/gui.py
text: @user1:
use `_is_function` instead of `callable`

@user2:
I would recommend not to. 
A class (with `__call__()` is to be a valid value here.
Tell me.

@user1:
I thought `_is_function` was created for this :-(

@user1:
but a class without `__call__` would be `callable`

@user2:
Whaat?

@user2:
Oh I see. `MyClass` class is always callable since you can write `MyClass()`... which invokes the constructor.

diff_hunk: @@ -2637,6 +2672,7 @@ def __bind_default_function(self):
             self.__bind_local_func("on_exception")
             self.__bind_local_func("on_status")
             self.__bind_local_func("on_user_content")
+            self.__bind_local_func("on_invalid_data")
path: taipy/gui/gui.py
text: @user1:
I forgot to remove this