Repository: Instagram/LibCST (language: python)

Issue:
Escaping quote characters in raw string literals causes a tokenizer error
This string breaks the native parser: `rf"\"{feature_name.lower()}\""`

```
$ echo 'rf"\"{feature_name.lower()}\""' | LIBCST_PARSER_TYPE=native python -m libcst.tool print - > /dev/null
Traceback (most recent call last):
  File "<string>", line 49, in <module>
  File "<string>", line 47, in __run
  File "lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "libcst/__main__.py", line 25, in <module>
    sys.exit(fbcode_main())
  File "libcst/__main__.py", line 19, in fbcode_main
    return libcst.tool.main(
  File "libcst/tool.py", line 834, in main
    return lookup.get(args.action or None, _invalid_command)(proc_name, command_args)
  File "libcst/tool.py", line 278, in _print_tree_impl
    tree = parse_module(
  File "libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "libcst/_parser/entrypoints.py", line 55, in _parse
    return parse(source_str)
libcst._exceptions.ParserSyntaxError: Syntax Error @ 0:1.
tokenizer error: unexpected characters after a line continuation

rf"\"{feature_name.lower()}\""
^
$
```

```
$ echo 'rf"\"{feature_name.lower()}\""' | python -m libcst.tool print - > /dev/null
$
```


Discussion / hints:
Not sure if is the same issue, but this also breaks the native parser: `f"regexp_like(path, '.*\{file_type}$')"`
This is another example: `f"\{{\}}"` which fails both libcst parsers but `python3 -m py_compile` parses it.


Right, so after digging into this a bit, there are two bugs:
1. f-strings and escaping `{`. I opened #699 and I think I have a fix for that.
2. the original example in this issue has nothing to do with f-strings, and is a tokenizer bug where it doesn't respect escaping `'` and `"` quotes in raw strings.

Apparently it's possible to write `r"\""` in Python, and this string has two characters, a backslash and a double quote.

Fix patch:
diff --git a/native/libcst/src/tokenizer/core/mod.rs b/native/libcst/src/tokenizer/core/mod.rs
index 9a8695556..a746be0fe 100644
--- a/native/libcst/src/tokenizer/core/mod.rs
+++ b/native/libcst/src/tokenizer/core/mod.rs
@@ -953,6 +953,13 @@ impl<'t> TokState<'t> {
                         }
                     }
                 }
+                (Some('\\'), _) if is_raw_string => {
+                    self.text_pos.next();
+                    if let Some('"' | '\'') = self.text_pos.peek() {
+                        // these aren't end of string markers, skip them
+                        self.text_pos.next();
+                    }
+                }
                 (Some('{'), _) => {
                     if is_in_format_spec {
                         // don't actually consume the {, and generate an OP for it instead


Test patch:
diff --git a/native/libcst/src/tokenizer/tests.rs b/native/libcst/src/tokenizer/tests.rs
index 300e6d299..503b54f98 100644
--- a/native/libcst/src/tokenizer/tests.rs
+++ b/native/libcst/src/tokenizer/tests.rs
@@ -519,6 +519,36 @@ fn test_string_prefix() {
             (TokType::String, "''"),
         ]),
     );
+
+    // raw string escapes
+    assert_eq!(
+        tokenize_all("r'\\''", &default_config()),
+        Ok(vec![(TokType::String, "r'\\''")]),
+    );
+    assert_eq!(
+        tokenize_all(r#"r"\"""#, &default_config()),
+        Ok(vec![(TokType::String, r#"r"\"""#)]),
+    );
+    let config = TokConfig {
+        split_fstring: true,
+        ..default_config()
+    };
+    assert_eq!(
+        tokenize_all("rf'\\''", &config),
+        Ok(vec![
+            (TokType::FStringStart, "rf'"),
+            (TokType::FStringString, "\\'"),
+            (TokType::FStringEnd, "'"),
+        ]),
+    );
+    assert_eq!(
+        tokenize_all(r#"rf"\"""#, &config),
+        Ok(vec![
+            (TokType::FStringStart, "rf\""),
+            (TokType::FStringString, r#"\""#),
+            (TokType::FStringEnd, "\""),
+        ]),
+    );
 }
 
 #[test]