Repository: Instagram/LibCST (language: python)

Issue:
Rust parser fails on trailing indentation
Repro by @tushar-deepsource:

```pycon
Python 3.10.0
>>> import libcst
>>> code = '''\
... if x == type(y):
...     pass
...     ''' # This indent whitespace is important
>>> libcst.parse_statement(code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../libcst/_parser/entrypoints.py", line 138, in parse_statement
    result = _parse(
  File ".../libcst/_parser/entrypoints.py", line 55, in _parse
    return parse(source_str)
libcst._exceptions.ParserSyntaxError: Syntax Error @ 3:5.
parser error: error at 3:4: expected one of (, *, +, -, ..., AWAIT, DEDENT, False, NAME, NUMBER, None, True, [, break, continue, lambda, match, not, pass, ~

    pass
       ^
>>>
```

_Originally posted by @tushar-deepsource in https://github.com/Instagram/LibCST/issues/568#issuecomment-1002574953_


Discussion / hints:
This is handled in the pure parser by automatically appending a newline here:
https://github.com/Instagram/LibCST/blob/c44ff0500b52dd78716c36c7d1efd3016e20005f/libcst/_parser/detect_config.py#L161-L171

which gets removed in https://github.com/Instagram/LibCST/blob/c44ff0500b52dd78716c36c7d1efd3016e20005f/libcst/_nodes/module.py#L109

Unfortunately this hack doesn't really work well unless `parse_module` is used, and then `code` is called on the returned `Module` object, which is why I decided not to port the hack over to Rust. Instead, the Rust tokenizer adds a fake newline token automatically without mutating the input (this is important because the Rust API's `String` input is immutable)
https://github.com/Instagram/LibCST/blob/c44ff0500b52dd78716c36c7d1efd3016e20005f/native/libcst/src/tokenizer/core/mod.rs#L337-L344
And then during inflation (i.e. whitespace parsing) this fake newline is detected here:
https://github.com/Instagram/LibCST/blob/c44ff0500b52dd78716c36c7d1efd3016e20005f/native/libcst/src/tokenizer/whitespace_parser.rs#L248-L253


Fix patch:
diff --git a/native/libcst/src/tokenizer/core/mod.rs b/native/libcst/src/tokenizer/core/mod.rs
index 5e8c29a15..3ff97e7db 100644
--- a/native/libcst/src/tokenizer/core/mod.rs
+++ b/native/libcst/src/tokenizer/core/mod.rs
@@ -342,7 +342,20 @@ impl<'t> TokState<'t> {
                         self.missing_nl_before_eof = false;
                         Ok(TokType::Newline)
                     } else {
-                        Ok(TokType::EndMarker)
+                        let hanging_indents = self.indent_stack.len() as i32;
+                        if self.pending_indents == 0 && hanging_indents != 0 {
+                            // We've reached EOF but there are still pending indents not
+                            // accounted for. Flush them out.
+                            self.pending_indents = -hanging_indents;
+                            self.indent_stack.clear();
+                            self.alt_indent_stack.clear();
+                            self.missing_nl_before_eof = false;
+                        }
+                        if let Some(t) = self.process_pending_indents() {
+                            Ok(t)
+                        } else {
+                            Ok(TokType::EndMarker)
+                        }
                     }
                 }
 


Test patch:
diff --git a/native/libcst/src/tokenizer/tests.rs b/native/libcst/src/tokenizer/tests.rs
index 6b4dfbb1a..db437857d 100644
--- a/native/libcst/src/tokenizer/tests.rs
+++ b/native/libcst/src/tokenizer/tests.rs
@@ -687,3 +687,43 @@ fn test_add_dedents_after_fake_newline() {
         ])
     );
 }
+
+#[test]
+fn test_add_dedents_for_dangling_indent() {
+    assert_eq!(
+        tokenize_with_end_marker("if 1:\n  if 2:\n    ", &default_config()),
+        Ok(vec![
+            (TokType::Name, "if"),
+            (TokType::Number, "1"),
+            (TokType::Op, ":"),
+            (TokType::Newline, "\n"),
+            (TokType::Indent, ""),
+            (TokType::Name, "if"),
+            (TokType::Number, "2"),
+            (TokType::Op, ":"),
+            (TokType::Newline, "\n"),
+            (TokType::Dedent, ""),
+            (TokType::EndMarker, "")
+        ])
+    );
+}
+
+#[test]
+fn test_add_dedents_for_dangling_indent_with_comment() {
+    assert_eq!(
+        tokenize_with_end_marker("if 1:\n  if 2:\n    # foo", &default_config()),
+        Ok(vec![
+            (TokType::Name, "if"),
+            (TokType::Number, "1"),
+            (TokType::Op, ":"),
+            (TokType::Newline, "\n"),
+            (TokType::Indent, ""),
+            (TokType::Name, "if"),
+            (TokType::Number, "2"),
+            (TokType::Op, ":"),
+            (TokType::Newline, "\n"),
+            (TokType::Dedent, ""),
+            (TokType::EndMarker, "")
+        ])
+    );
+}
diff --git a/native/libcst/tests/fixtures/dangling_indent.py b/native/libcst/tests/fixtures/dangling_indent.py
new file mode 100644
index 000000000..32ac62307
--- /dev/null
+++ b/native/libcst/tests/fixtures/dangling_indent.py
@@ -0,0 +1,3 @@
+if 1:
+    pass
+    
\ No newline at end of file