Repository: Avaiga/taipy
Pull request: GUI - Frontend Decoupling
Resolves #446 .
Resolves #656.

Problem statement:
SPIKE-Decouple the front-end from the Taipy GUI backend
**What would that feature address**
We want to allow for the use of other technologies for the front-end side of Taipy.

That requirement is mandatory if we plan to deliver a Page Builder application, hosted on the client, using some other technology that may not even be React or TypeScript.

***Description of the ideal solution***
Say we use some technology called '3rdParty'.
This is integrated in Taipy with a `taipy-3rdparty` package that provides a few classes that Taipy GUI application can use as a replacement for raw Taipy GUI classes, as well as the front-end code for the application that ultimately runs on the client.

Ex:
```
from taipy.gui import Gui
from taipy_3rdparty import Page

page = Page(whatever_content_makes_sense_for_the_3rdparty_package)

Gui(page=page).run()
```

- A page definition for '3rdParty' could be raw text, or the content of a file, or whatever.
- A page definition would allow for using variable binding, just like it is done in Taipy GUI vanilla.
- When the `Gui` instance runs, it could detect that no pure Taipy Gui page is to be served, therefore not using the `taipy-gui` JavaScript bundle.
- It would detect, though, that '3rdParty' *is* used, and make sure that the 3rdParty bundle is properly loaded by the application.
- One of the complex points of that move is the creation of the Web Service API so that communication can be set up.
   -   A new message must be installed to require the exposed variables and their scopes (a tree, I suppose).
       That's a change compared to what Taipy GUI needs today, where variables are queried when pages are rendered.
   -   The 3rdParty library must explicitly set up the client side of the communication WebService in order to both receive and send messages to the server application. This can be done only if its API is exposed, documented, and written in raw JavaScript.
- the rendering of the age itself is delegated to the 3rdParty library, which can convert the definition of a page into whatever is expected in the browser.

***Caveats***

Users are prevented from using Taipy's components, and even page definitions, that rely on these.

**Acceptance Criteria**
- [ ] A path to allow for using third-party front-end code and support bi-directional communication.


API for custom front-end implementation
**What would that feature address**

So far, when one wants to use the `taipy.gui.custom.Page` class and implement Taipy's front-end with some JS-based technology, we have to implement the WebSocket protocol entirely: essentially the connection and message management part.
This code is likely to be identical no matter what the front-end technology is, so it would make sense to create such an API and expose it. The resulting JavaScript bundle could be reused by programmers with a far lower development effort.

***Description of the ideal solution***

Taipy GUI could provide (in taipy/gui/custom?) a JS bundle that the use would simply import in the custom front-end code as a module.
This bundle would provide a 'taipy' namespace where reusable code would be immediately accessible.

This API would allow for:
- Connect to the WebSocket and get a notification when the connection is done.
- Disconnect from the WebSocket and get a notification when the connection is broken.
- Notify on WebSocket errors.
- Update the module context and variable scopes so the front-end code can just rely on it.
- Pre-parse the WebSocket messages that come from the Taipy back-end (invoking overloaded functions defined by the programmer).<br/>
   i.e. : `on_update(variableName, variableValue)` when receiving a "U" or "MU" message.
- Sending message to the back-end with properly named functions with documented arguments.<br/>
   i.e. : `update(variableName, variableValue)` -> `sendWsMessage(socket, "U", variableName, { value: variableValue}, context);`.

**Acceptance Criteria**
- [ ] Ensure new code is unit tested, and check code coverage is at least 90%
- [ ] Update private demos based on this bundle.

Hints from issue discussion:

Commit message: register payload before handle ws message

Patch to review:
diff --git a/frontend/taipy-gui/dom/package-lock.json b/frontend/taipy-gui/dom/package-lock.json
index dac125b852..d9afc84b08 100644
--- a/frontend/taipy-gui/dom/package-lock.json
+++ b/frontend/taipy-gui/dom/package-lock.json
@@ -14,6 +14,7 @@
       }
     },
     "../packaging": {
+      "name": "taipy-gui",
       "version": "3.1.0"
     },
     "node_modules/js-tokens": {
diff --git a/frontend/taipy-gui/src/components/Taipy/Navigate.tsx b/frontend/taipy-gui/src/components/Taipy/Navigate.tsx
index 0c98541de5..adbbddd827 100644
--- a/frontend/taipy-gui/src/components/Taipy/Navigate.tsx
+++ b/frontend/taipy-gui/src/components/Taipy/Navigate.tsx
@@ -38,6 +38,9 @@ const Navigate = ({ to, params, tab, force }: NavigateProps) => {
                     navigate(0);
                 } else {
                     navigate({ pathname: to, search: `?${searchParams.toString()}` });
+                    if (searchParams.has("tprh")) {
+                        navigate(0);
+                    }
                 }
             } else {
                 window.open(`${to}?${searchParams.toString()}`, tab || "_blank")?.focus();
diff --git a/taipy/gui/external/__init__.py b/taipy/gui/external/__init__.py
new file mode 100644
index 0000000000..63f12fad87
--- /dev/null
+++ b/taipy/gui/external/__init__.py
@@ -0,0 +1,12 @@
+# Copyright 2023 Avaiga Private Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+
+from ._custom_page import CustomPage, ResourceHandler
diff --git a/taipy/gui/external/_custom_page.py b/taipy/gui/external/_custom_page.py
new file mode 100644
index 0000000000..49098fff37
--- /dev/null
+++ b/taipy/gui/external/_custom_page.py
@@ -0,0 +1,82 @@
+# Copyright 2023 Avaiga Private Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+import typing as t
+from abc import ABC, abstractmethod
+
+from ..page import Page
+from ..utils.singleton import _Singleton
+
+if t.TYPE_CHECKING:
+    from ..gui import Gui
+
+
+class CustomPage(Page):
+    """A custom page for external application that can be added to Taipy GUI"""
+
+    def __init__(self, resource_handler: ResourceHandler, binding_variables: t.List[str] = None, **kwargs) -> None:
+        if binding_variables is None:
+            binding_variables = []
+        super().__init__(**kwargs)
+        self._resource_handler = resource_handler
+        self._binding_variables = binding_variables
+
+
+class ResourceHandler(ABC):
+    """Resource handler for custom pages
+    User can implement this class to provide custom resources for the custom pages
+    """
+
+    id: str = ""
+
+    def __init__(self) -> None:
+        _ExternalResourceHandlerManager().register(self)
+
+    def get_id(self) -> str:
+        return self.id if id != "" else str(id(self))
+
+    @abstractmethod
+    def get_resources(self, path: str) -> t.Any:
+        raise NotImplementedError
+
+
+class _ExternalResourceHandlerManager(object, metaclass=_Singleton):
+    """Manager for resource handlers
+    This class is used to manage resource handlers for custom pages
+    """
+
+    def __init__(self) -> None:
+        self.__handlers: t.Dict[str, ResourceHandler] = {}
+
+    def register(self, handler: ResourceHandler) -> None:
+        """Register a resource handler
+        Arguments:
+            handler (ResourceHandler): The resource handler to register
+        """
+        self.__handlers[handler.get_id()] = handler
+
+    def get(self, id: str) -> t.Optional[ResourceHandler]:
+        """Get a resource handler by its id
+        Arguments:
+            id (str): The id of the resource handler
+        Returns:
+            ResourceHandler: The resource handler
+        """
+        return self.__handlers.get(id, None)
+
+    def get_all(self) -> t.List[ResourceHandler]:
+        """Get all resource handlers
+        Returns:
+            List[ResourceHandler]: The list of resource handlers
+        """
+        return list(self.__handlers.values())
diff --git a/taipy/gui/gui.py b/taipy/gui/gui.py
index cead6ad5c6..70271789f0 100644
--- a/taipy/gui/gui.py
+++ b/taipy/gui/gui.py
@@ -25,13 +25,13 @@
 import warnings
 from importlib import metadata, util
 from importlib.util import find_spec
-from types import FrameType, SimpleNamespace
+from types import FrameType, FunctionType, LambdaType, ModuleType, SimpleNamespace
 from urllib.parse import unquote, urlencode, urlparse
 
 import __main__
 import markdown as md_lib
 import tzlocal
-from flask import Blueprint, Flask, g, jsonify, request, send_file, send_from_directory
+from flask import Blueprint, Flask, g, has_app_context, jsonify, request, send_file, send_from_directory
 from taipy.logger._taipy_logger import _TaipyLogger
 from werkzeug.utils import secure_filename
 
@@ -53,6 +53,7 @@
 from .data.data_format import _DataFormat
 from .data.data_scope import _DataScopes
 from .extension.library import Element, ElementLibrary
+from .external import CustomPage
 from .gui_types import _WsType
 from .page import Page
 from .partial import Partial
@@ -537,6 +538,8 @@ def _get_client_id(self) -> str:
     def __set_client_id_in_context(self, client_id: t.Optional[str] = None, force=False):
         if not client_id and request:
             client_id = request.args.get(Gui.__ARG_CLIENT_ID, "")
+        if not client_id and (ws_client_id := getattr(g, "ws_client_id", None)):
+            client_id = ws_client_id
         if not client_id and force:
             res = self._bindings()._get_or_create_scope("")
             client_id = res[0] if res[1] else None
@@ -583,10 +586,12 @@ def _manage_message(self, msg_type: _WsType, message: dict) -> None:
             if msg_type == _WsType.CLIENT_ID.value:
                 res = self._bindings()._get_or_create_scope(message.get("payload", ""))
                 client_id = res[0] if res[1] else None
-            self.__set_client_id_in_context(client_id or message.get(Gui.__ARG_CLIENT_ID))
+            expected_client_id = client_id or message.get(Gui.__ARG_CLIENT_ID)
+            self.__set_client_id_in_context(expected_client_id)
+            g.ws_client_id = expected_client_id
             with self._set_locals_context(message.get("module_context") or None):
+                payload = message.get("payload", {})
                 if msg_type == _WsType.UPDATE.value:
-                    payload = message.get("payload", {})
                     self.__front_end_update(
                         str(message.get("name")),
                         payload.get("value"),
@@ -600,6 +605,10 @@ def _manage_message(self, msg_type: _WsType, message: dict) -> None:
                     self.__request_data_update(str(message.get("name")), message.get("payload"))
                 elif msg_type == _WsType.REQUEST_UPDATE.value:
                     self.__request_var_update(message.get("payload"))
+                elif msg_type == _WsType.GET_MODULE_CONTEXT.value:
+                    self.__handle_ws_get_module_context(payload)
+                elif msg_type == _WsType.GET_VARIABLES.value:
+                    self.__handle_ws_get_variables()
             self.__send_ack(message.get("ack_id"))
         except Exception as e:  # pragma: no cover
             _warn(f"Decoding Message has failed: {message}", e)
@@ -1024,6 +1033,54 @@ def __request_var_update(self, payload: t.Any):
                     )
             self.__send_var_list_update(payload["names"])
 
+    def __handle_ws_get_module_context(self, payload: t.Any):
+        if isinstance(payload, dict):
+            # Get Module Context
+            if mc := self._get_page_context(str(payload.get("path"))):
+                self._bind_custom_page_variables(
+                    self._get_page(str(payload.get("path")))._renderer, self._get_client_id()
+                )
+                self.__send_ws(
+                    {
+                        "type": _WsType.GET_MODULE_CONTEXT.value,
+                        "payload": {"data": mc},
+                    }
+                )
+
+    def __handle_ws_get_variables(self):
+        # Get Variables
+        self.__pre_render_pages()
+        # Module Context -> Variable -> Variable data (name, type, initial_value)
+        variable_tree: t.Dict[str, t.Dict[str, t.Dict[str, t.Any]]] = {}
+        data = vars(self._bindings()._get_data_scope())
+        data = {
+            k: v
+            for k, v in data.items()
+            if not k.startswith("_")
+            and not callable(v)
+            and "TpExPr" not in k
+            and not isinstance(v, (ModuleType, FunctionType, LambdaType, type, Page))
+        }
+        for k, v in data.items():
+            if isinstance(v, _TaipyBase):
+                data[k] = v.get()
+            var_name, var_module_name = _variable_decode(k)
+            if var_module_name == "" or var_module_name is None:
+                var_module_name = "__main__"
+            if var_module_name not in variable_tree:
+                variable_tree[var_module_name] = {}
+            variable_tree[var_module_name][var_name] = {
+                "type": type(v).__name__,
+                "value": data[k],
+                "encoded_name": k,
+            }
+        self.__send_ws(
+            {
+                "type": _WsType.GET_VARIABLES.value,
+                "payload": {"data": variable_tree},
+            }
+        )
+
     def __send_ws(self, payload: dict, allow_grouping=True) -> None:
         grouping_message = self.__get_message_grouping() if allow_grouping else None
         if grouping_message is None:
@@ -1865,10 +1922,12 @@ def __pre_render_pages(self) -> None:
         for page in self._config.pages:
             if page is not None:
                 with contextlib.suppress(Exception):
-                    page.render(self)
+                    if isinstance(page._renderer, CustomPage):
+                        self._bind_custom_page_variables(page._renderer, self._get_client_id())
+                    else:
+                        page.render(self)
 
-    def __render_page(self, page_name: str) -> t.Any:
-        self.__set_client_id_in_context()
+    def _get_navigated_page(self, page_name: str) -> t.Any:
         nav_page = page_name
         if hasattr(self, "on_navigate") and callable(self.on_navigate):
             try:
@@ -1889,8 +1948,26 @@ def __render_page(self, page_name: str) -> t.Any:
             except Exception as e:  # pragma: no cover
                 if not self._call_on_exception("on_navigate", e):
                     _warn("Exception raised in on_navigate()", e)
-        page = next((page_i for page_i in self._config.pages if page_i._route == nav_page), None)
+        return nav_page
+
+    def _get_page(self, page_name: str):
+        return next((page_i for page_i in self._config.pages if page_i._route == page_name), None)
 
+    def _bind_custom_page_variables(self, page: CustomPage, client_id: t.Optional[str]):
+        """Handle the bindings of custom page variables"""
+        with self.get_flask_app().app_context() if has_app_context() else contextlib.nullcontext():
+            self.__set_client_id_in_context(client_id)
+            with self._set_locals_context(page._get_module_name()):
+                for k in self._get_locals_bind().keys():
+                    if (not page._binding_variables or k in page._binding_variables) and not k.startswith("_"):
+                        self._bind_var(k)
+
+    def __render_page(self, page_name: str) -> t.Any:
+        self.__set_client_id_in_context()
+        nav_page = self._get_navigated_page(page_name)
+        if not isinstance(nav_page, str):
+            return nav_page
+        page = self._get_page(nav_page)
         # Try partials
         if page is None:
             page = self._get_partial(nav_page)
@@ -1901,6 +1978,19 @@ def __render_page(self, page_name: str) -> t.Any:
                 400,
                 {"Content-Type": "application/json; charset=utf-8"},
             )
+        # Handle custom pages
+        if (pr := page._renderer) is not None and isinstance(pr, CustomPage):
+            if self._navigate(
+                to=page_name,
+                params={
+                    _Server._RESOURCE_HANDLER_ARG: pr._resource_handler.get_id(),
+                },
+            ):
+                # proactively handle the bindings of custom page variables
+                self._bind_custom_page_variables(pr, self._get_client_id())
+                return ("Successfully redirect to external resource handler", 200)
+            return ("Failed to navigate to external resource handler", 500)
+        # Handle page rendering
         context = page.render(self)
         if (
             nav_page == Gui.__root_page_name
diff --git a/taipy/gui/gui_types.py b/taipy/gui/gui_types.py
index 2f03161eb9..87f15bc772 100644
--- a/taipy/gui/gui_types.py
+++ b/taipy/gui/gui_types.py
@@ -45,6 +45,8 @@ class _WsType(Enum):
     DOWNLOAD_FILE = "DF"
     PARTIAL = "PR"
     ACKNOWLEDGEMENT = "ACK"
+    GET_MODULE_CONTEXT = "GMC"
+    GET_VARIABLES = "GVS"
 
 
 NumberTypes = {"int", "int64", "float", "float64"}
diff --git a/taipy/gui/page.py b/taipy/gui/page.py
index 9dbe163bd7..0cfa1e4ea3 100644
--- a/taipy/gui/page.py
+++ b/taipy/gui/page.py
@@ -34,6 +34,8 @@ class Page:
     """
 
     def __init__(self, **kwargs) -> None:
+        from .external import CustomPage
+
         self._class_module_name = ""
         self._class_locals: t.Dict[str, t.Any] = {}
         self._frame: t.Optional[FrameType] = None
@@ -42,6 +44,8 @@ def __init__(self, **kwargs) -> None:
             self._frame = kwargs.get("frame")
         elif self._renderer:
             self._frame = self._renderer._frame
+        elif isinstance(self, CustomPage):
+            self._frame = t.cast(FrameType, t.cast(FrameType, inspect.stack()[2].frame))
         elif len(inspect.stack()) < 4:
             raise RuntimeError(f"Can't resolve module. Page '{type(self).__name__}' is not registered.")
         else:
diff --git a/taipy/gui/server.py b/taipy/gui/server.py
index ae09a8945e..907702981d 100644
--- a/taipy/gui/server.py
+++ b/taipy/gui/server.py
@@ -22,9 +22,10 @@
 import webbrowser
 from importlib import util
 from random import randint
+from urllib.parse import parse_qsl, urlparse
 
 import __main__
-from flask import Blueprint, Flask, json, jsonify, render_template, send_from_directory
+from flask import Blueprint, Flask, json, jsonify, render_template, request, send_from_directory
 from flask_cors import CORS
 from flask_socketio import SocketIO
 from gitignore_parser import parse_gitignore
@@ -34,6 +35,7 @@
 
 from ._renderers.json import _TaipyJsonProvider
 from .config import ServerConfig
+from .external._custom_page import _ExternalResourceHandlerManager
 from .utils import _is_in_notebook, _is_port_open, _RuntimeManager
 
 if t.TYPE_CHECKING:
@@ -45,6 +47,7 @@ class _Server:
     __RE_CLOSING_CURLY = re.compile(r"(\})([^\"])")
     __OPENING_CURLY = r"\1&#x7B;"
     __CLOSING_CURLY = r"&#x7D;\2"
+    _RESOURCE_HANDLER_ARG = "tprh"
 
     def __init__(
         self,
@@ -144,6 +147,18 @@ def _get_default_blueprint(
         @taipy_bp.route("/", defaults={"path": ""})
         @taipy_bp.route("/<path:path>")
         def my_index(path):
+            resource_handler_id = dict(parse_qsl(urlparse(request.referrer or "").query)).get(
+                _Server._RESOURCE_HANDLER_ARG
+            )
+            resource_handler_id = resource_handler_id or request.args.get(_Server._RESOURCE_HANDLER_ARG, None)
+            if resource_handler_id is not None:
+                resource_handler = _ExternalResourceHandlerManager().get(resource_handler_id)
+                if resource_handler is None:
+                    return (f"Invalid value for query {_Server._RESOURCE_HANDLER_ARG}", 404)
+                try:
+                    return resource_handler.get_resources(path)
+                except Exception:
+                    raise RuntimeError("Can't get resources from external resource handler")
             if path == "" or path == "index.html" or "." not in path:
                 try:
                     return render_template(


Code review comments:
diff_hunk: @@ -0,0 +1,82 @@
+# Copyright 2023 Avaiga Private Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+import typing as t
+from abc import ABC, abstractmethod
+
+from ..page import Page
+from ..utils.singleton import _Singleton
+
+if t.TYPE_CHECKING:
+    from ..gui import Gui
+
+
+class CustomPage(Page):
+    """A custom page for external application that can be added to Taipy GUI"""
+
+    def __init__(self, resource_handler: ResourceHandler, binding_variables: t.List[str] = None, **kwargs) -> None:
path: taipy/gui/external/_custom_page.py
text: @user1:
t.Optional[t.List[str]] ?