Repository: mozman/steputils

Issue:
.step file parsing error
I am unable to read a step file.
These .step file are coming from Rhino & Creo professional tools
I keep getting the following error.

```
File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 1025, in readfile
    return load(fp)
  File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 1019, in load
    return loads(content)
  File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 1003, in loads
    return Parser(lexer).parse()
  File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 633, in __init__
    self.tokens = list(lexer.parse())
  File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 505, in parse
    yield step_decoder(self.string())  # str
  File "D:\Progam Files\Python\python v3.7.0\lib\site-packages\steputils\p21.py", line 557, in string
    raise ParseError(f'Found invalid character in string "{current}".')
steputils.exceptions.ParseError: Found invalid character in string "
".
```

When I open the .stp files in notepad++ text editor, the encoding is UTF-8. 
I tried changing the file encoding to ISO 8859-1 without any success. 


Discussion:
w0rm: @mozman fair enough, this doesn't seem to follow the official spec... 

I my case I don't have control over the files that I have to parse. Plus I saw an open issue https://github.com/mozman/steputils/issues/1 and thought it wouldn't hurt to relax the parser a bit to allow multiline strings.
w0rm: @mozman this seems to be allowed in the standard, but the newline should be ignored instead of included in the string.

http://www.steptools.com/stds/step/IS_final_p21e3.html#clause-5-6

> NOTE      Space is the only whitespace character that separates tokens. Line-delimiters such as line feed or carriage return and other control characters such as form feed or character tabulation (tab) may appear in the exchange structure but are required by [5.2](http://www.steptools.com/stds/step/IS_final_p21e3.html#clause-5-2) to be ignored when processing the exchange structure. Consequently, line breaks may appear anywhere within the structure, including within tokens.

Patch:
diff --git a/src/steputils/p21.py b/src/steputils/p21.py
index c8dd28b..a7f3de1 100644
--- a/src/steputils/p21.py
+++ b/src/steputils/p21.py
@@ -33,6 +33,7 @@ FIRST_KEYWORD_CHAR = ascii_uppercase + '_'
 KEYWORD_CHARS = ascii_letters + digits + '_-'  # should accept ISO-10303-21 and lower case letters
 
 STRING_CHARS = ascii_letters + digits + ' _' + SPECIAL + BACKSLASH
+IGNORE_DELIMETERS = "\n\r\f\t"
 FIRST_NUMBER_CHARS = '-+01234567890'
 NUMBER_CHARS = FIRST_NUMBER_CHARS + '.eE'
 FIRST_ENUM_CHARS = ascii_uppercase + '_.'
@@ -553,6 +554,8 @@ class Lexer:
                     return ''.join(s)
             elif current in STRING_CHARS:
                 s.append(b.get())
+            elif current in IGNORE_DELIMETERS:
+                b.skip()
             else:
                 raise ParseError(f'Found invalid character in string "{current}".')
 


Test patch:
diff --git a/tests/p21/test_p21_lexer.py b/tests/p21/test_p21_lexer.py
index 53cda6c..787b453 100644
--- a/tests/p21/test_p21_lexer.py
+++ b/tests/p21/test_p21_lexer.py
@@ -63,9 +63,7 @@ def test_lexer_string_with_enclosed_comments():
 
 
 def test_lexer_string_across_lines():
-    # strings across lines are not allowed
-    with pytest.raises(p21.ParseError):
-        list(p21.Lexer("'\n',"))
+    assert list(p21.Lexer("'multi\nline',")) == ['multiline', ',']
 
 
 def test_lexer_empty_string():