Repository: randy3k/radian (language: python)

Issue:
Error when pasting long code containing multibyte characters into the radian console and executing it
When I paste a long code of multibyte characters into the radian console and try to run it, I get the following error and cannot run it.
This does not occur with the default R terminal.

```
Error: EOF whilst reading MBCS char at line 5
Error: invalid multibyte character in parser at line 1
```

The following code reproduces this in my environment.
(The following strings are randomly generated.)

```r
a <- c("おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ
おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ
おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ
おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ
おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ")
```


Discussion / hints:
I was able to reproduce the issue. 

Actually, base R has a similar bug if the string contains no new line symbols, like this

```
a <- c("おゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとみぺととびばぶゐぺぴのるぃゅぽぐやじぶみあいさざゔぅまだおゎがごなめかぅさんんぴしもあみぺえぅぞかおもづどいたぼぺぁりうもぞさせめにげさほぅぅもあぶまゆぜゎまやつりゅびけぁむれぴりゃぎぴせぐゑきぢぷぅるおみらへつにすぅょよへょこぺぁひおべいけとゎべぢでみぷゃぜてこごねぼぶづなぺれぶゑげらぷぼだぱんかこべとこるぐゔなへさばろすてへわたなさゐぜぺてぉでてずけゆぼれきゅはむぇだぴづにをぉわえじごふちりっぅぷあぜぷやべろうまぞてすやなげぅずでょぱけんぬくやわもぱぼゑどぱげをらひゕぜこ")
```

~~But radian does handle this case correctly.~~ Actually, radian will fail if we change the variable name from `a` to `ab` depends on how a character is cut.

I will need to investigate how the new lines break radian.
I think I understand what it fails. It is basically because we need to break a long text into chunks of exact size of 4094 bytes. However, for multibytes characters, it is obviously problematic because we might cut the character to halves. Base R works around it by breaking text at line breaks.

However, radian doesn't simply work by breaking long text at new lines because radian parses the whole long text at once. I will need to rework on the parsing logic.

Here is a workaround for this situation. To support both base R and radian, one could save the string in a `.R` file and source the R file.
Thank you for digging into this!
At the moment I get around this by using the default R console for projects that need to deal with long strings, but not being able to use radian is stressful.
It is partly due to a bug in [rchitect](https://github.com/randy3k/rchitect).  It will still require more work to make it work in all cases.


Fix patch:
diff --git a/radian/console.py b/radian/console.py
index 2224823..09f9b65 100644
--- a/radian/console.py
+++ b/radian/console.py
@@ -5,7 +5,6 @@
 from .settings import radian_settings as settings
 from rchitect import console
 
-
 TERMINAL_CURSOR_AT_BEGINNING = [True]
 
 SUPPRESS_STDOUT = False
@@ -120,8 +119,34 @@ def _read_console(message, add_history=1):
 
         return text
 
+    _text = [""]
+    startpos = [0]
+
     def read_console(message, add_history):
-        return _read_console(message, add_history)
+        if session.current_mode in ["r", "browse"]:
+            # this code is needed to allow new line breaks with strings, see #377
+            if _text[0]:
+                text = _text[0][startpos[0]:]
+            else:
+                text = _read_console(message, add_history)
+                if text and "\n" in text:
+                    # make sure the text is evaluated at once
+                    text = "{\n" + text + "\n}"
+                _text[0] = text
+                startpos[0] = 0
+
+            if text:
+                index = text.find('\n')
+                if index >= 0:
+                    startpos[0] += index + 1
+                    text = text[:index]
+                else:
+                    _text[0] = ""
+                    startpos[0] = 0
+        else:
+            text = _read_console(message, add_history)
+
+        return text
 
     return read_console
 
diff --git a/radian/key_bindings.py b/radian/key_bindings.py
index 77d8c63..54adc4e 100644
--- a/radian/key_bindings.py
+++ b/radian/key_bindings.py
@@ -140,14 +140,14 @@ def newline(event, chars=["{", "[", "("]):
         event.current_buffer.insert_text(" " * tab_size)
 
 
-def create_prompt_key_bindings(prase_text_complete):
+def create_prompt_key_bindings(parse_text_complete):
     kb = KeyBindings()
     handle = kb.add
 
     @Condition
     def prase_complete():
         app = get_app()
-        return prase_text_complete(app.current_buffer.text)
+        return parse_text_complete(app.current_buffer.text)
 
     @handle('c-j', filter=insert_mode & default_focused)
     @handle('enter', filter=insert_mode & default_focused)
@@ -288,7 +288,7 @@ def _(event):
         should_eval = data and data[-1] == "\n" and \
             len(event.current_buffer.document.text_after_cursor) == 0
         # todo: allow partial prase complete
-        if should_eval and prase_text_complete(data):
+        if should_eval and parse_text_complete(data):
             data = data.rstrip("\n")
             event.current_buffer.insert_text(data)
             event.current_buffer.validate_and_handle()
@@ -299,8 +299,8 @@ def _(event):
 
 
 # keybinds for both r mond and browse mode
-def create_r_key_bindings(prase_text_complete):
-    kb = create_prompt_key_bindings(prase_text_complete)
+def create_r_key_bindings(parse_text_complete):
+    kb = create_prompt_key_bindings(parse_text_complete)
     handle = kb.add
 
     # r mode
diff --git a/radian/prompt_session.py b/radian/prompt_session.py
index 0e88951..1b89232 100644
--- a/radian/prompt_session.py
+++ b/radian/prompt_session.py
@@ -18,10 +18,10 @@
 from pygments.styles import get_style_by_name
 
 from rchitect import rcopy, rcall, robject
-from rchitect.interface import roption, setoption, process_events, peek_event, polled_events
+from rchitect.interface import roption, setoption, parse_text_complete, \
+    process_events, peek_event, polled_events
 
 from . import shell
-from .rutils import prase_text_complete
 from .key_bindings import create_r_key_bindings, create_shell_key_bindings, create_key_bindings
 from .completion import RCompleter, SmartPathCompleter
 from .io import CustomInput, CustomOutput
@@ -168,7 +168,7 @@ def message():
     if settings.highlight_matching_bracket:
         input_processors.append(HighlightMatchingBracketProcessor())
 
-    r_key_bindings = create_r_key_bindings(prase_text_complete)
+    r_key_bindings = create_r_key_bindings(parse_text_complete)
 
     session.register_mode(
         name="r",
diff --git a/radian/reticulate/__init__.py b/radian/reticulate/__init__.py
index 80fcf7d..9f67619 100644
--- a/radian/reticulate/__init__.py
+++ b/radian/reticulate/__init__.py
@@ -45,7 +45,7 @@ def reticulate_prompt_hook(*args):
     source_file(os.path.join(os.path.dirname(__file__), "key_bindings.R"))
 
 
-def prase_text_complete(code):
+def parse_text_complete(code):
     if "\n" in code:
         try:
             return compile_command(code, "<input>", "exec") is not None
diff --git a/radian/reticulate/key_bindings.R b/radian/reticulate/key_bindings.R
index a756aaa..cccb9bb 100644
--- a/radian/reticulate/key_bindings.R
+++ b/radian/reticulate/key_bindings.R
@@ -54,8 +54,8 @@ kb$add("~", filter = main_mode & insert_mode & default_focused & cursor_at_begin
     })
 
 
-prase_text_complete <- radian$reticulate$prase_text_complete
-pkb <- radian$key_bindings$create_prompt_key_bindings(prase_text_complete)
+parse_text_complete <- radian$reticulate$parse_text_complete
+pkb <- radian$key_bindings$create_prompt_key_bindings(parse_text_complete)
 
 pkb$add("c-d", filter = insert_mode & default_focused & cursor_at_begin & text_is_empty)(
     function(event) commit_text(event, "exit", FALSE)
diff --git a/radian/rutils.py b/radian/rutils.py
index 0bddbc8..46d88cc 100644
--- a/radian/rutils.py
+++ b/radian/rutils.py
@@ -1,21 +1,8 @@
 import os
 import sys
 from rchitect import rcopy, reval, rcall
-from rchitect._cffi import ffi, lib
-from rchitect.interface import roption, protected, rstring_p
+from rchitect.interface import roption
 from .key_bindings import map_key
-from .console import suppress_stderr
-
-
-def prase_text_complete(text):
-    status = ffi.new("ParseStatus[1]")
-    s = rstring_p(text)
-    orig_stderr = sys.stderr
-    sys.stderr = None
-    with protected(s), suppress_stderr():
-        lib.R_ParseVector(s, -1, status, lib.R_NilValue)
-        sys.stderr = orig_stderr
-    return status[0] != 2
 
 
 def package_is_loaded(pkg):
diff --git a/setup.py b/setup.py
index 8cc201a..c00c72b 100644
--- a/setup.py
+++ b/setup.py
@@ -42,8 +42,8 @@ def get_version(package):
     package_data={'radian': ['reticulate/*.R']},
     python_requires='>=3.6',
     install_requires=[
-        # 'rchitect@git+https://github.com/randy3k/rchitect',
-        'rchitect>=0.3.36,<0.4.0',
+        'rchitect@git+https://github.com/randy3k/rchitect',
+        # 'rchitect>=0.3.38,<0.4.0',
         'prompt_toolkit>=3.0.15,<3.1',
         'pygments>=2.5.0'
     ],


Test patch:
diff --git a/tests/test_readline.py b/tests/test_readline.py
index 42a2d16..6b25016 100644
--- a/tests/test_readline.py
+++ b/tests/test_readline.py
@@ -1,3 +1,7 @@
+import sys
+import pytest
+
+
 def test_readline(terminal):
     # issue #106
     terminal.current_line().assert_startswith("r$>")
@@ -13,3 +17,52 @@ def test_askpass(terminal):
     terminal.current_line().assert_startswith("askpass>")
     terminal.write("answer\n")
     terminal.previous_line(2).assert_contain("\"answer\"")
+
+
+def test_strings(terminal):
+    # issue #377
+    terminal.current_line().assert_startswith("r$>")
+    terminal.write("x <- 'a'\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(x)\n")
+    terminal.previous_line(2).assert_startswith("[1] 1")
+
+
+@pytest.mark.skipif(sys.platform.startswith("win"), reason="windows doesn't support bpm.")
+def test_strings_bracketed(terminal):
+    terminal.current_line().assert_startswith("r$>")
+    terminal.write("\x1b[200~x <- '" + 'a'*10 + "'\x1b[201~\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(x)\n")
+    terminal.previous_line(2).assert_startswith("[1] 10")
+
+    terminal.write("\x1b[200~x <- '" + 'a'*5000 + "'\x1b[201~\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(x)\n")
+    terminal.previous_line(2).assert_startswith("[1] 5000")
+
+    terminal.write("\x1b[200~x <- '" + 'a'*2000 + '\n' + 'b'*2000 + "'\x1b[201~\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(x)\n")
+    terminal.previous_line(2).assert_startswith("[1] 4001")
+
+    s = '中'*1000 + '\n' + '文'*1000 + '\n' + '中'*1000 + '\n' + '文'*1000
+
+    terminal.write("\x1b[200~x <- '" + s + "'\x1b[201~\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(x)\n")
+    terminal.previous_line(2).assert_startswith("[1] 4003")
+
+    # different padding
+    terminal.write("\x1b[200~xy <- '" + s + "'\x1b[201~\n")
+    terminal.current_line().strip().assert_equal("r$>")
+    terminal.write("nchar(xy)\n")
+    terminal.previous_line(2).assert_startswith("[1] 4003")
+
+
+def test_early_termination(terminal):
+    terminal.current_line().assert_startswith("r$>")
+    terminal.write("stop('!')\x1b\rd = 1\n")
+    terminal.previous_line(2).assert_startswith("Error")
+    terminal.write("d\n")
+    terminal.previous_line(2).assert_startswith("Error: object 'd' not found")