Repository: abs-lang/abs
Pull request #130 (2019-01-24 15:15:19)

Fix issues 63+101
Hi Alex, for your consideration...

In issues #63 and #101, the root problem with strings that contain escaped control characters lies in the way the lexer parses the strings into tokens. In Go and Python, `CR`, `LF`, and `TAB` are replaced with their ASCII control codes only when they are created from literal strings such as `"\n"`. If we read in the characters `'\'` and `'n'` as a sequence of bytes from IO and just convert them into a string they are not automatically expanded into an ASCII control code. Therefore, we have to explicitly do the conversion when the string token is created in `lexer.readString()`. This also affects how `split()` and `join()` work when matching the delimiter.

The second problem is that when `lexer.readString()` is handling double escaped quotes it also creates a side effect on any other double escaped sequence where it removes the double escape. Thus both `'\' '\' 'n'` and `'\' 'n'` always comes out as `"\n"`.

So, based on our earlier discussion this PR implements escaped control character expansion into ASCII control codes in `lexer.readString()` only when the strings are double quoted while single quoted strings preserve the literal escape sequence. Also, double quoted strings may contain a mix of expanded and double escaped control characters. This also allows `a = split("a\nb\nc", "\n"); join(a, "\n")` to match the expanded LF control code while `a = split('a\nb\nc', '\n');join(a, '\n')` will match the literal string. Note that in the absence of control characters, double and single quoted strings are otherwise interchangeable.

You can see a use case for this model in the tests/test-strings.abs file where the test echo's the fully escaped string before it executes it as an unescaped string.

Also, I have fixed issues with the docs for error line location from the previous PR.

Rick

odino: @ntwrick I'm going to add Go tests for this functionality just to make sure those are covered. Again, thanks!

BTW I'm now using version branches, so I changed the base branch of this PR to `1.0`.

Patch:
diff --git a/docs/misc/error.md b/docs/misc/error.md
index ba56eba..a1e4909 100644
--- a/docs/misc/error.md
+++ b/docs/misc/error.md
@@ -12,7 +12,7 @@ When you are running ABS interactively in the Run, Eval, Print Loop (REPL) the l
 
 However, when you are running ABS over a script file (even a small one) locating errors requires more help from the interpreter. ABS now provides `[line:column]` positions as well as the error line itself following the error message.
 
-For example, a file with syntax errors might look like this when the first syntax error is in line 4 somewhere around column 11.
+For example, a file with syntax errors might look like this when the first syntax error is in line 4 at column 5.
 ```
 $ cat examples/error-parse.abs
 # there are multiple parser errors in this file
@@ -30,20 +30,20 @@ c/c = $(command)
 b %% c
 
 $ abs examples/error-parse.abs
- parser errors:
+  parser errors:
 	no prefix parse function for '=' found
-	[4:11]	m.a = 'abc'
+	[4:5]	m.a = 'abc'
 	no prefix parse function for '=' found
-	[7:16]	d/d = $(command);
+	[7:5]	d/d = $(command);
 	no prefix parse function for '=' found
-	[10:16]	c/c = $(command)
+	[10:5]	c/c = $(command)
 	no prefix parse function for '%' found
-	[13:6]	b %% c
+	[13:4]	b %% c
 
 $ echo $?
 99
 ```
-Furthermore, a file with evaluation errors might look like this when the first error encountered is in line 2 somewhere around column 11:
+Furthermore, a file with evaluation errors might look like this when the first error encountered is in line 2 at column 3:
 ```
 $ cat examples/error-eval.abs
 # there is an evaluation error on line 2
@@ -52,7 +52,7 @@ echo("should not reach here")
 
 $ abs examples/error-eval.abs
 ERROR: type mismatch: NUMBER + STRING
-	[2:11]	1 + "hello"
+	[2:3]	1 + "hello"
 
 $ echo $?
 99
diff --git a/docs/misc/technical-details.md b/docs/misc/technical-details.md
index 2d6260d..cd7f44e 100644
--- a/docs/misc/technical-details.md
+++ b/docs/misc/technical-details.md
@@ -31,59 +31,59 @@ With `make run` you can get inside a container built for ABS'
 development, and `make test` will run all tests.
 
 ## Testing
-You can run the interpreter error location tests by invoking this bash script: `tests/test-abs.sh`.
+### Interpreter Error Location
+You can run the interpreter error location tests by invoking this bash script: `tests/test-abs.sh`. This script iterates over the `test-parser.abs` and `test-eval.abs` test scripts.
 ```
 $ tests/test-abs.sh
-
 =======================================
 Test Parser
 tests/test-parser.abs
  parser errors:
 	no prefix parse function for '=' found
-	[4:11]	m.a = 'abc'
+	[4:5]	m.a = 'abc'
 	no prefix parse function for '=' found
-	[7:16]	d/d = $(command);
+	[7:5]	d/d = $(command);
 	no prefix parse function for '=' found
-	[10:16]	c/c = $(command)
+	[10:5]	c/c = $(command)
 	no prefix parse function for '%' found
-	[13:6]	b %% c
+	[13:4]	b %% c
 	no prefix parse function for '&&' found
-	[22:4]	&&||!-/*5;
-	no prefix parse function for 'OR' found
-	[22:5]	&&||!-/*5;
+	[22:1]	&&||!-/*5;
+	no prefix parse function for '||' found
+	[22:3]	&&||!-/*5;
 	no prefix parse function for '/' found
-	[22:8]	&&||!-/*5;
+	[22:7]	&&||!-/*5;
 	no prefix parse function for '<=>' found
-	[25:3]	<=>
-	expected next token to be ], got , instead
-	[44:3]	[1, 2];
+	[25:2]	<=>
+	expected next token to be NUMBER, got , instead
+	[44:2]	[1, 2];
 	no prefix parse function for ',' found
-	[44:5]	[1, 2];
+	[44:3]	[1, 2];
 	no prefix parse function for ']' found
-	[44:7]	[1, 2];
+	[44:6]	[1, 2];
 	no prefix parse function for '%' found
 	[68:2]	~%
 	no prefix parse function for '-=' found
-	[70:2]	-=
+	[70:1]	-=
 	no prefix parse function for '/=' found
-	[72:2]	/=
+	[72:1]	/=
 	no prefix parse function for '%=' found
-	[74:2]	%=
+	[74:1]	%=
 	no prefix parse function for '^' found
-	[79:4]	&^>><<
+	[79:2]	&^>><<
 	no prefix parse function for '<<' found
-	[79:6]	&^>><<
+	[79:5]	&^>><<
 	Illegal token '$111'
-	[80:4]	$111
-	no prefix parse function for 'ILLEGAL' found
-	[76:7]	1.str()
+	[80:1]	$111
+	no prefix parse function for '$111' found
+	[80:1]	$111
 Exit code: 99
 
 =======================================
 Test Eval()
 tests/test-eval.abs
 ERROR: type mismatch: STRING + NUMBER
-	[8:35]	    s = s + 1   # this is a comment
+	[8:11]	    s = s + 1   # this is a comment
 Exit code: 99
 
 ERROR: invalid property 'junk' on type ARRAY
@@ -91,10 +91,59 @@ ERROR: invalid property 'junk' on type ARRAY
 Exit code: 99
 
 ERROR: index operator not supported: f(x) {x} on HASH
-	[19:29]	    {"name": "Abs"}[f(x) {x}];  
+	[19:20]	    {"name": "Abs"}[f(x) {x}];  
 Exit code: 99
-
 ```
+### String tests
+String handling tests can be run from `abs tests/test-strings.abs`
+```
+$ abs tests/test-strings.abs
+=====================
+>>> Testing string with expanded LFs:
+echo("a\nb\nc")
+a
+b
+c
+=====================
+>>> Testing string with expanded TABs:
+echo("a\tb\tc")
+a	b	c
+=====================
+>>> Testing string with expanded CRs:
+echo("a\rb\rc")
+c
+=====================
+>>> Testing string with mixed expanded LFs and escaped LFs:
+echo("a\\nb\\nc\n%s\n", "x\ny\nz")
+a\\nb\\nc
+x
+y
+z
+
+=====================
+>>> Testing string with multiple escapes:
+echo("hel\\\\lo")
+hel\\\\lo
+=====================
+>>> Testing split and join strings with expanded LFs:
+s = split("a\nb\nc", "\n")
+echo(s)
+[a, b, c]
+ss = join(s, "\n")
+echo(ss)
+a
+b
+c
+=====================
+>>> Testing split and join strings with literal LFs:
+s = split('a\nb\nc', '\n')
+echo(s)
+[a, b, c]
+ss = join(s, '\n')
+echo(ss)
+a\nb\nc
+```
+
 ## Roadmap
 
 We're currently working on [1.0](https://github.com/abs-lang/abs/milestone/5).
diff --git a/docs/types/string.md b/docs/types/string.md
index 7574dde..5bcdeb7 100644
--- a/docs/types/string.md
+++ b/docs/types/string.md
@@ -47,6 +47,72 @@ a value that evaluates to `false` when casted to boolean:
 !!"" # false
 ```
 
+## Special characters embedded in strings
+Double and single quoted strings behave differently if the string contains escaped special ASCII line control characters such as `LF "\n"`, `CR "\r"`, and `TAB "\t"`. 
+
+If the string is double quoted these characters will be expanded to their ASCII codes. On the other hand, if the string is single quoted, these characters will be considered as escaped literals. 
+
+This means, for example, that double quoted LFs will cause line feeds to appear in the output:
+
+```bash
+⧐  echo("a\nb\nc")
+a
+b
+c
+⧐  
+```
+Conversely, single quoted LFs will appear as escaped literal strings:
+
+```bash
+⧐  echo('a\nb\nc')
+a\nb\nc
+⧐  
+```
+
+And if you need to mix escaped and unescaped special characters, then you can do this with double escapes within double quoted strings:
+```bash
+⧐  echo("a\\nb\nc")
+a\\nb
+c
+⧐  
+```
+### Working with special characters in string functions
+Special characters also work with `split()` and `join()` and other string functions as well.
+
+1) Double quoted expanded special characters:
+```bash
+⧐  s = split("a\nb\nc", "\n")
+⧐  echo(s)
+[a, b, c]
+⧐  ss = join(s, "\n")
+⧐  echo(ss)
+a
+b
+c
+⧐ 
+```
+
+2) Single quoted literal special characters:
+```bash
+⧐  s = split('a\nb\nc', '\n')
+⧐  echo(s)
+[a, b, c]
+⧐  ss = join(s, '\n')
+⧐  echo(ss)
+a\nb\nc
+⧐  
+```
+
+3) Double quoted, double escaped special characters:
+```bash
+⧐  s = split("a\\nb\\nc", "\\n")
+⧐  echo(s)
+[a, b, c]
+⧐  ss = join(s, "\\n")
+⧐  echo(ss)
+a\\nb\\nc
+⧐  
+```
 ## Supported functions
 
 ### len()
diff --git a/evaluator/functions.go b/evaluator/functions.go
index b68556a..6c7b7ea 100644
--- a/evaluator/functions.go
+++ b/evaluator/functions.go
@@ -176,6 +176,11 @@ func getFns() map[string]*object.Builtin {
 		"echo": &object.Builtin{
 			Types: []string{},
 			Fn: func(args ...object.Object) object.Object {
+				if len(args) == 0 {
+					// allow echo() without crashing
+					fmt.Println("")
+					return NULL
+				}
 				var arguments []interface{} = make([]interface{}, len(args)-1)
 				for i, d := range args {
 					if i > 0 {
diff --git a/lexer/lexer.go b/lexer/lexer.go
index ccb8e76..9e0aff5 100644
--- a/lexer/lexer.go
+++ b/lexer/lexer.go
@@ -381,33 +381,55 @@ func (l *Lexer) readLogicalOperator() string {
 // character itself ("\\").
 func (l *Lexer) readString(quote byte) string {
 	var chars []string
+	esc := byte('\\')
 	doubleEscape := false
 	for {
 		l.readChar()
 
-		if l.ch == '\\' && l.peekChar() == '\\' {
-			chars = append(chars, string('\\'))
+		if l.ch == esc && l.peekChar() == esc {
+			chars = append(chars, string(esc))
 			l.readChar()
-			doubleEscape = true
+			// be careful here, there may be double escaped LFs in the string
+			if l.peekChar() == quote {
+				doubleEscape = true
+			} else {
+				// this is not a double escaped quote
+				chars = append(chars, string(esc))
+			}
 			continue
 		}
-
-		// If we encounter a \, let's check whether
-		// we're trying to escape a ". If so, let's skip
-		// the / and add the " to the string.
-		if l.ch == '\\' && l.peekChar() == quote {
+		// If we encounter an escape, let's check whether
+		// we're trying to escape a quote. If so, let's skip
+		// the escape and add the quote to the string.
+		if l.ch == esc && l.peekChar() == quote {
 			chars = append(chars, string(quote))
 			l.readChar()
 			continue
 		}
-
-		// The string ends when we encounter a "
-		// and the character before that was not a \,
-		// or the \ was escaped as well ("string\\").
-		if (l.ch == quote && (l.prevChar(2) != '\\' || doubleEscape)) || l.ch == 0 {
+		// If this is a double quoted string we need to expand embedded
+		// LF, CR, and TAB to ASCII and add the ASCII code to the string
+		// NB. single quoted strings don't expand special characters to ASCII
+		if quote == '"' {
+			if l.ch == esc && l.peekChar() == 'n' {
+				chars = append(chars, "\n")
+				l.readChar()
+				continue
+			} else if l.ch == esc && l.peekChar() == 'r' {
+				chars = append(chars, "\r")
+				l.readChar()
+				continue
+			} else if l.ch == esc && l.peekChar() == 't' {
+				chars = append(chars, "\t")
+				l.readChar()
+				continue
+			}
+		}
+		// The string ends when we encounter a quote
+		// and the character before that was not an escape,
+		// or the escape was escaped as well ("string\\").
+		if (l.ch == quote && (l.prevChar(2) != esc || doubleEscape)) || l.ch == 0 {
 			break
 		}
-
 		chars = append(chars, string(l.ch))
 		doubleEscape = false
 	}


Test patch:
diff --git a/lexer/lexer_test.go b/lexer/lexer_test.go
index 0b4890a..f2b1191 100644
--- a/lexer/lexer_test.go
+++ b/lexer/lexer_test.go
@@ -237,12 +237,12 @@ $111
 		{token.IDENT, "tree"},
 		{token.STRING, "hel\"lo"},
 		{token.STRING, "hel\\lo"},
-		{token.STRING, "hel\\\\lo"},
+		{token.STRING, `hel\\\\lo`},
 		{token.STRING, "\"hello\""},
 		{token.STRING, "\"he\"\"llo\""},
 		{token.STRING, "hello\\"},
-		{token.STRING, "hello\\\\"},
-		{token.STRING, "\\\\hello"},
+		{token.STRING, `hello\\\`},
+		{token.STRING, `\\\\hello`},
 		{token.EXPONENT, "**"},
 		{token.NUMBER, "1"},
 		{token.RANGE, ".."},
@@ -310,7 +310,7 @@ func TestRewind(t *testing.T) {
 		}
 
 		if tok.Literal != tt.expectedLiteral {
-			t.Fatalf("tests[%d] - literal wrong. expected=%q, got=%q", i, tt.expectedLiteral, tok.Literal)
+			t.Errorf("tests[%d] - literal wrong. expected=%q, got=%q", i, tt.expectedLiteral, tok.Literal)
 		}
 	}
 
diff --git a/tests/test-strings.abs b/tests/test-strings.abs
new file mode 100644
index 0000000..afdff95
--- /dev/null
+++ b/tests/test-strings.abs
@@ -0,0 +1,67 @@
+# tests/tests-strings.abs
+echo("=====================")
+msg = 'string with expanded LFs'
+ta = 'echo("a\nb\nc")'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+echo("a\nb\nc")
+
+echo("=====================")
+msg = 'string with expanded TABs'
+ta = 'echo("a\tb\tc")'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+echo("a\tb\tc")
+
+echo("=====================")
+msg = 'string with expanded CRs'
+ta = 'echo("a\rb\rc")'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+echo("a\rb\rc")
+
+echo("=====================")
+msg = 'string with mixed expanded LFs and escaped LFs'
+ta = 'echo("a\\nb\\nc\n%s\n", "x\ny\nz")'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+echo("a\\nb\\nc\n%s\n", "x\ny\nz")
+
+echo("=====================")
+msg = 'string with multiple escapes'
+ta = 'echo("hel\\\\lo")'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+echo("hel\\\\lo")
+
+echo("=====================")
+msg = 'split and join strings with expanded LFs'
+ta = 's = split("a\nb\nc", "\n")'
+tb = 'echo(s)'
+tc = 'ss = join(s, "\n")'
+td = 'echo(ss)'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+s = split("a\nb\nc", "\n")
+echo("%s", tb)
+echo(s)
+echo("%s", tc)
+ss = join(s, "\n")
+echo("%s", td)
+echo(ss)
+
+echo("=====================")
+msg = 'split and join strings with literal LFs'
+ta = 's = split(\'a\nb\nc\', \'\n\')'
+tb = 'echo(s)'
+tc = 'ss = join(s, \'\n\')'
+td = 'echo(ss)'
+echo(">>> Testing %s:", msg)
+echo("%s", ta)
+s = split('a\nb\nc', '\n')
+echo("%s", tb)
+echo(s)
+echo("%s", tc)
+ss = join(s, '\n')
+echo("%s", td)
+echo(ss)