Repository: fonttools/fonttools

Issue:
[feaLib] Lexer for double quote strings should not stop at newline?
I have fonts and fea files with a name table where some nameid strings have newlines in the string. According to the spec 

> A string is composed of 1-byte ASCII characters enclosed by ASCII double quote characters ("). Newlines embedded within the string are removed from the character sequence to be stored.

Looking at a ttx dump these strings have `&#13;` inserted for the newlines so ttx seems to have no issue, but loading the fea file with feaLib fails because in lexer.py there is `self.scan_until_('"\r\n')`
If I remove  `\r\n` it loads fine, but then test_string fails in lexer_test.py because it doesn't raise an Exception at the embedded newline. Maybe I'm missing something, but I don't see a reason for the `\r\n` in the double quote lexer. 

[feaLib] Lexer for double quote strings should not stop at newline?
I have fonts and fea files with a name table where some nameid strings have newlines in the string. According to the spec 

> A string is composed of 1-byte ASCII characters enclosed by ASCII double quote characters ("). Newlines embedded within the string are removed from the character sequence to be stored.

Looking at a ttx dump these strings have `&#13;` inserted for the newlines so ttx seems to have no issue, but loading the fea file with feaLib fails because in lexer.py there is `self.scan_until_('"\r\n')`
If I remove  `\r\n` it loads fine, but then test_string fails in lexer_test.py because it doesn't raise an Exception at the embedded newline. Maybe I'm missing something, but I don't see a reason for the `\r\n` in the double quote lexer. 



Discussion:
Many thanks for the report; sounds like a bug. (I'm traveling next week so it may take a few days until I've a fix ready, sorry about that).

@brawer FWIW, i'm almost done fixing it. I'll send a PR and you can have a look at it next week.

Many thanks for the report; sounds like a bug. (I'm traveling next week so it may take a few days until I've a fix ready, sorry about that).

@brawer FWIW, i'm almost done fixing it. I'll send a PR and you can have a look at it next week.


Resolving patch:
diff --git a/Lib/fontTools/feaLib/lexer.py b/Lib/fontTools/feaLib/lexer.py
--- a/Lib/fontTools/feaLib/lexer.py
+++ b/Lib/fontTools/feaLib/lexer.py
@@ -142,10 +142,12 @@ def next_(self):
             return (Lexer.SYMBOL, cur_char, location)
         if cur_char == '"':
             self.pos_ += 1
-            self.scan_until_('"\r\n')
+            self.scan_until_('"')
             if self.pos_ < self.text_length_ and self.text_[self.pos_] == '"':
                 self.pos_ += 1
-                return (Lexer.STRING, text[start + 1:self.pos_ - 1], location)
+                # strip newlines embedded within a string
+                string = re.sub("[\r\n]", "", text[start + 1:self.pos_ - 1])
+                return (Lexer.STRING, string, location)
             else:
                 raise FeatureLibError("Expected '\"' to terminate string",
                                       location)


Test patch:
diff --git a/Lib/fontTools/feaLib/lexer_test.py b/Lib/fontTools/feaLib/lexer_test.py
--- a/Lib/fontTools/feaLib/lexer_test.py
+++ b/Lib/fontTools/feaLib/lexer_test.py
@@ -82,7 +82,9 @@ def test_comment(self):
     def test_string(self):
         self.assertEqual(lex('"foo" "bar"'),
                          [(Lexer.STRING, "foo"), (Lexer.STRING, "bar")])
-        self.assertRaises(FeatureLibError, lambda: lex('"foo\n bar"'))
+        self.assertEqual(lex('"foo \nbar\r baz \r\nqux\n\n "'),
+                         [(Lexer.STRING, "foo bar baz qux ")])
+        self.assertRaises(FeatureLibError, lambda: lex('"foo\n bar'))
 
     def test_bad_character(self):
         self.assertRaises(FeatureLibError, lambda: lex("123 \u0001"))