Repository: pyodide/pyodide

Issue:
Improving console.html
`console.html`is a great tool. It's a showcase for Pyodide and [the devel version](https://pyodide-cdn2.iodide.io/dev/full/console.html) allows developers to track issues and make quick tests.

Unfortunately, IMHO, it lacks some important features like:

1. sync behavior (do not set next prompt until current command finished, e.g. with imports)
2. full stdout/stderr support (e.g. import warnings are not visible)
3. correct result representation with `repr` (type `"1"` and it shows `1`, type `globals()` and it shows `[object Object]`)

While 1. and 2. are addressed in this PR, 3. seems tricky, at least with the current setup (JQuery's terminal + `copy.py` interactive console + `runPythonAscyn`) but not impossible (see [Basthon](https://basthon.fr)).

Getting correct representations for basic types (like strings) are addressed in this PR. Difficulty to get correct `repr` for complex objects comes from the fact that, when you get a Python object on the JS side (e.g. runPython[Async]'s result), it seems impossible to get it back into Python. As an illustration, compare the result of the two following code snippets:

```javascript
// returns "[object Object]"
pyodide.runPython("import js ; js.window.x = {1: 2} ; repr(js.window.x)")
// returns "{1: 2}"
pyodide.runPython("x = {1: 2} ; repr(x)")
```
Long story short, 

```javascript
pyodide.runPython("repr({1: 2})")
// is not equivalent to:
pyodide.pyimport("repr")(pyodide.runPython("{1: 2}"))
```

Any idea to fully address 3.with the current setup?

Should I edit changelog ?

Closes https://github.com/iodide-project/pyodide/issues/897


Discussion:
Thanks a lot for improving `console.html` @casatir !

For the point 3 could you open an issue about it? I think it's a general issue with py ⟷ JS conversions, possibly related to https://github.com/iodide-project/pyodide/issues/780.

Yes, please add a changelog entry with a link to this PR.

Hello @casatir, thank you for your contribution.

I'm also of the same opinion, the REPL is a very important place to start with. I've just compiled and tested your branch, it runs well, but I found about another problem: 
```python
s = input()
```
shows a JavaScript prompt() in 0.15.0 (anyway, I doesn't return anything), but now yields in an error
```
>>> s = input()
Error: Traceback (most recent call last):
  File "/lib/python3.8/site-packages/pyodide.py", line 66, in eval_code
    exec(compile(mod, "<exec>", mode="exec"), ns, ns)
  File "<exec>", line 1, in <module>
ValueError: I/O operation on closed file.
```
Is there a way to resolve this? Please do also update the CHANGELOG.md.
I will update the changelog and put console stuff in `pyodide.py`.

Are you sure about the `input` issue? My tests shows that it works like in 0.15.0...
> Are you sure about the `input` issue? My tests shows that it works like in 0.15.0...

you're right... I've just built it again and it works. Then forget about it.


Here is an attempt to fix #897 . Seems OK but namespace is polluted by imports and variables from the REPL code. Not tab completion on filepath (not supported by `Jedi` from what I understand).
> Here is an attempt to fix #897 . Seems OK but namespace is polluted by imports and variables from the REPL code. Not tab completion on filepath (not supported by Jedi from what I understand).

Great thanks! We can look into improving that situation in a separate PR.

> Answered to #896 where your proposal makes sense. However, should this be blocking for this PR?

I'll try to make a PR to create that package soon, but it's not a blocker. However we need tests for those classes to merge this PR, which for now probably means putting them into `src/pyodide.py`.
I think I have a working console with:

- sync behavior
- full stdout/stderr support
- correct result representation
- clean namespace
- integration in a new environment made easy
- tests

It is fully implemented in Python and demonstrates the Pyodide ability to interpret itself.

That's a lot of changes that I'll split into several PR.

See you soon!

Resolving patch:
diff --git a/src/pyodide-py/pyodide/console.py b/src/pyodide-py/pyodide/console.py
--- a/src/pyodide-py/pyodide/console.py
+++ b/src/pyodide-py/pyodide/console.py
@@ -1,8 +1,40 @@
-from typing import Optional, Callable
+from typing import Optional, Callable, Any
 import code
 import io
 import sys
 import platform
+from contextlib import contextmanager
+import builtins
+
+# this import can fail when we are outside a browser (e.g. for tests)
+try:
+    import js
+
+    _dummy_promise = js.Promise.resolve()
+    _load_packages_from_imports = js.pyodide.loadPackagesFromImports
+
+except ImportError:
+
+    class _FakePromise:
+        """A promise that mimic the JS promises.
+
+        Only `then is supported` and there is no asynchronicity.
+        execution occurs when then is call.
+
+        This is mainly to fake `load_packages_from_imports`
+        and `InteractiveConsole.run_complete` in contexts
+        where JS promises are not available (tests)."""
+
+        def __init__(self, args=None):
+            self.args = (args,) if args is not None else ()
+
+        def then(self, func, *args):
+            return _FakePromise(func(*self.args))
+
+    _dummy_promise = _FakePromise()
+
+    def _load_packages_from_imports(*args):
+        return _dummy_promise
 
 
 __all__ = ["InteractiveConsole"]
@@ -62,7 +94,9 @@ class InteractiveConsole(code.InteractiveConsole):
     """Interactive Pyodide console
 
     Base implementation for an interactive console that manages
-    stdout/stderr redirection.
+    stdout/stderr redirection. Since packages are loaded before running
+    code, `runcode` returns a JS promise. Override `sys.displayhook` to
+    catch the result of an execution.
 
     `self.stdout_callback` and `self.stderr_callback` can be overloaded.
 
@@ -91,12 +125,16 @@ def __init__(
         self._stderr = None
         self.stdout_callback = stdout_callback
         self.stderr_callback = stderr_callback
-        self._persistent_stream_redirection = persistent_stream_redirection
-        if self._persistent_stream_redirection:
+        self._streams_redirected = False
+        if persistent_stream_redirection:
             self.redirect_stdstreams()
+        self.run_complete = _dummy_promise
 
     def redirect_stdstreams(self):
         """ Toggle stdout/stderr redirections. """
+        # already redirected?
+        if self._streams_redirected:
+            return
 
         if self._stdout is None:
             # we use meta callbacks to allow self.std{out,err}_callback
@@ -127,27 +165,126 @@ def meta_stderr_callback(*args):
         # actual redirection
         sys.stdout = self._stdout
         sys.stderr = self._stderr
+        self._streams_redirected = True
 
     def restore_stdstreams(self):
         """Restore stdout/stderr to the value it was before
         the creation of the object."""
-        sys.stdout = self._old_stdout
-        sys.stderr = self._old_stderr
-
-    def runcode(self, code):
-        if self._persistent_stream_redirection:
-            super().runcode(code)
+        if self._streams_redirected:
+            sys.stdout = self._old_stdout
+            sys.stderr = self._old_stderr
+            self._streams_redirected = False
+
+    @contextmanager
+    def stdstreams_redirections(self):
+        """Ensure std stream redirection.
+
+        This supports nesting."""
+        if self._streams_redirected:
+            yield
         else:
             self.redirect_stdstreams()
-            super().runcode(code)
+            yield
             self.restore_stdstreams()
 
+    def flush_all(self):
+        """ Force stdout/stderr flush. """
+        with self.stdstreams_redirections():
+            sys.stdout.flush()
+            sys.stderr.flush()
+
+    def runsource(self, *args, **kwargs):
+        """Force streams redirection.
+
+        Syntax errors are not thrown by runcode but here in runsource.
+        This is why we force redirection here since doing twice
+        is not an issue."""
+
+        with self.stdstreams_redirections():
+            return super().runsource(*args, **kwargs)
+
+    def runcode(self, code):
+        """Load imported packages then run code, async.
+
+        To achieve nice result representation, the interactive console
+        is fully implemented in Python. This has a major drawback:
+        packages should be loaded from here. This is why this
+        function sets the promise `self.run_complete`.
+        If you need to wait for the end of the computation,
+        you should await for it."""
+        parent_runcode = super().runcode
+        source = "\n".join(self.buffer)
+
+        def load_packages_and_run(*args):
+            def run(*args):
+                with self.stdstreams_redirections():
+                    parent_runcode(code)
+                    # in CPython's REPL, flush is performed
+                    # by input(prompt) at each new prompt ;
+                    # since we are not using input, we force
+                    # flushing here
+                    self.flush_all()
+                return _dummy_promise
+
+            return _load_packages_from_imports(source).then(run)
+
+        self.run_complete = self.run_complete.then(load_packages_and_run)
+
     def __del__(self):
-        if self._persistent_stream_redirection:
-            self.restore_stdstreams()
+        self.restore_stdstreams()
 
     def banner(self):
         """ A banner similar to the one printed by the real Python interpreter. """
         # copyied from https://github.com/python/cpython/blob/799f8489d418b7f9207d333eac38214931bd7dcc/Lib/code.py#L214
         cprt = 'Type "help", "copyright", "credits" or "license" for more information.'
-        return f"Python {platform.python_version()} {platform.python_build()} on WebAssembly VM\n{cprt}"
+        version = platform.python_version()
+        build = f"({', '.join(platform.python_build())})"
+        return f"Python {version} {build} on WebAssembly VM\n{cprt}"
+
+
+def repr_shorten(
+    value: Any, limit: int = 1000, split: Optional[int] = None, separator: str = "..."
+):
+    """Compute the string representation of `value` and shorten it
+    if necessary.
+
+    If it is longer than `limit` then return the firsts `split`
+    characters and the last `split` characters seperated by '...'.
+    Default value for `split` is `limit // 2`.
+    """
+    if split is None:
+        split = limit // 2
+    text = repr(value)
+    if len(text) > limit:
+        text = f"{text[:split]}{separator}{text[-split:]}"
+    return text
+
+
+def displayhook(value, repr: Callable[[Any], str]):
+    """A displayhook with custom `repr` function.
+
+    It is intendend to overload `sys.displayhook`. Note that monkeypatch
+    `builtins.repr` does not work in `sys.displayhook`. The pointer to
+    `repr` seems hardcoded in default `sys.displayhook` version
+    (which is written in C)."""
+    # from https://docs.python.org/3/library/sys.html#sys.displayhook
+    # If value is not None, this function prints repr(value) to
+    # sys.stdout, and saves value in builtins._. If repr(value) is not
+    # encodable to sys.stdout.encoding with sys.stdout.errors error
+    # handler (which is probably 'strict'), encode it to
+    # sys.stdout.encoding with 'backslashreplace' error handler.
+    if value is None:
+        return
+    builtins._ = None  # type: ignore
+    text = repr(value)
+    try:
+        sys.stdout.write(text)
+    except UnicodeEncodeError:
+        bytes = text.encode(sys.stdout.encoding, "backslashreplace")
+        if hasattr(sys.stdout, "buffer"):
+            sys.stdout.buffer.write(bytes)
+        else:
+            text = bytes.decode(sys.stdout.encoding, "strict")
+            sys.stdout.write(text)
+    sys.stdout.write("\n")
+    builtins._ = value  # type: ignore


Test patch:
diff --git a/src/tests/test_console.py b/src/tests/test_console.py
--- a/src/tests/test_console.py
+++ b/src/tests/test_console.py
@@ -1,6 +1,7 @@
 import pytest
 from pathlib import Path
 import sys
+import io
 
 sys.path.append(str(Path(__file__).parents[2] / "src" / "pyodide-py"))
 
@@ -23,15 +24,13 @@ def callback(string):
 
 
 @pytest.fixture
-def safe_stdstreams():
-    stdout = sys.stdout
-    stderr = sys.stderr
+def safe_sys_redirections():
+    redirected = sys.stdout, sys.stderr, sys.displayhook
     yield
-    sys.stdout = stdout
-    sys.stderr = stderr
+    sys.stdout, sys.stderr, sys.displayhook = redirected
 
 
-def test_interactive_console_streams(safe_stdstreams):
+def test_interactive_console_streams(safe_sys_redirections):
 
     my_stdout = ""
     my_stderr = ""
@@ -66,6 +65,12 @@ def stderr_callback(string):
     shell.push("print('foobar')")
     assert my_stdout == "foo\nfoobar\n"
 
+    shell.push("print('foobar')")
+    assert my_stdout == "foo\nfoobar\nfoobar\n"
+
+    shell.push("1+1")
+    assert my_stdout == "foo\nfoobar\nfoobar\n2\n"
+
     shell.restore_stdstreams()
 
     my_stdout = ""
@@ -94,3 +99,91 @@ def stderr_callback(string):
 
     print("bar")
     assert my_stdout == "foobar\n"
+
+    shell.push("print('foobar')")
+    assert my_stdout == "foobar\nfoobar\n"
+
+    shell.push("import sys")
+    shell.push("print('foobar', file=sys.stderr)")
+    assert my_stderr == "foobar\n"
+
+    shell.push("1+1")
+    assert my_stdout == "foobar\nfoobar\n2\n"
+
+
+def test_repr(safe_sys_redirections):
+    sep = "..."
+    for string in ("x" * 10 ** 5, "x" * (10 ** 5 + 1)):
+        for limit in (9, 10, 100, 101):
+            assert len(
+                console.repr_shorten(string, limit=limit, separator=sep)
+            ) == 2 * (limit // 2) + len(sep)
+
+    sys.stdout = io.StringIO()
+    console.displayhook(
+        [0] * 100, lambda v: console.repr_shorten(v, 100, separator=sep)
+    )
+    assert len(sys.stdout.getvalue()) == 100 + len(sep) + 1  # for \n
+
+
+@pytest.fixture
+def safe_selenium_sys_redirections(selenium):
+    selenium.run("_redirected = sys.stdout, sys.stderr, sys.displayhook")
+    yield
+    selenium.run("sys.stdout, sys.stderr, sys.displayhook = _redirected")
+
+
+def test_interactive_console(selenium, safe_selenium_sys_redirections):
+    def ensure_run_completed():
+        selenium.driver.execute_async_script(
+            """
+        const done = arguments[arguments.length - 1];
+        pyodide.globals.shell.run_complete.then(done);
+        """
+        )
+
+    selenium.run(
+        """
+    from pyodide.console import InteractiveConsole
+
+    result = None
+
+    def displayhook(value):
+        global result
+        result = value
+
+    shell = InteractiveConsole()
+    sys.displayhook = displayhook"""
+    )
+
+    selenium.run("shell.push('x = 5')")
+    selenium.run("shell.push('x')")
+    ensure_run_completed()
+    assert selenium.run("result") == 5
+
+    selenium.run("shell.push('x ** 2')")
+    ensure_run_completed()
+    assert selenium.run("result") == 25
+
+    selenium.run("shell.push('def f(x):')")
+    selenium.run("shell.push('    return x*x + 1')")
+    selenium.run("shell.push('')")
+    selenium.run("shell.push('[f(x) for x in range(5)]')")
+    ensure_run_completed()
+    assert selenium.run("result") == [1, 2, 5, 10, 17]
+
+    selenium.run("shell.push('def factorial(n):')")
+    selenium.run("shell.push('    if n < 2:')")
+    selenium.run("shell.push('        return 1')")
+    selenium.run("shell.push('    else:')")
+    selenium.run("shell.push('        return n * factorial(n - 1)')")
+    selenium.run("shell.push('')")
+    selenium.run("shell.push('factorial(10)')")
+    ensure_run_completed()
+    assert selenium.run("result") == 3628800
+
+    # with package load
+    selenium.run("shell.push('import pytz')")
+    selenium.run("shell.push('pytz.utc.zone')")
+    ensure_run_completed()
+    assert selenium.run("result") == "UTC"