Repository: Instagram/LibCST (language: python)

Issue:
Native parser does not allow indentation mismatch at EOF
Short reproducer: (Only happens when the `LIBCST_PARSER_TYPE=native`)
```py
import ast
import libcst

src = "if bleh:\n  print()\n "
ast.parse(src) # works perfectly
libcst.parse_module(src) # INDENT of 2 space does not match with indent of 1 space
```

```
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,2:            NAME           'if'           
1,3-1,7:            NAME           'bleh'         
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,1:            INDENT         '\t'           
2,1-2,6:            NAME           'print'        
2,6-2,7:            OP             '('            
2,7-2,8:            OP             ')'            
2,8-2,9:            NEWLINE        '\n'           
3,0-3,0:            DEDENT         ''             
3,0-3,0:            ENDMARKER      ''             
```


Discussion / hints:
Another way to repro, including the error message:
```
❯ printf 'if bleh:\n  print()\n ' | cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/parse`
Parse error for stdin: tokenizer error: no matching outer block for dedent
```
Seems like this is only a problem at the end of the file. Mixed indents are otherwise completely fine AFAICT.

Fix patch:
diff --git a/native/libcst/src/tokenizer/core/mod.rs b/native/libcst/src/tokenizer/core/mod.rs
index 3ff97e7db..0dd60944e 100644
--- a/native/libcst/src/tokenizer/core/mod.rs
+++ b/native/libcst/src/tokenizer/core/mod.rs
@@ -533,9 +533,13 @@ impl<'t> TokState<'t> {
             }
         }
 
-        // Lines with only whitespace and/or comments and/or a line continuation character shouldn't
-        // affect the indentation and are not passed to the parser as NEWLINE tokens.
-        self.blank_line = matches!(self.text_pos.peek(), Some('#') | Some('\n') | Some('\\'));
+        // Lines with only whitespace and/or comments and/or a line continuation
+        // character shouldn't affect the indentation and are not passed to the parser
+        // as NEWLINE tokens.
+        self.blank_line = matches!(
+            self.text_pos.peek(),
+            Some('#') | Some('\n') | Some('\\') | None
+        );
 
         if self.blank_line || !self.paren_stack.is_empty() {
             return Ok(());


Test patch:
diff --git a/native/libcst/src/tokenizer/tests.rs b/native/libcst/src/tokenizer/tests.rs
index db437857d..2be070d13 100644
--- a/native/libcst/src/tokenizer/tests.rs
+++ b/native/libcst/src/tokenizer/tests.rs
@@ -727,3 +727,20 @@ fn test_add_dedents_for_dangling_indent_with_comment() {
         ])
     );
 }
+
+#[test]
+fn test_inconsistent_indentation_at_eof() {
+    assert_eq!(
+        tokenize_all("if 1:\n  pass\n ", &default_config()),
+        Ok(vec![
+            (TokType::Name, "if"),
+            (TokType::Number, "1"),
+            (TokType::Op, ":"),
+            (TokType::Newline, "\n"),
+            (TokType::Indent, ""),
+            (TokType::Name, "pass"),
+            (TokType::Newline, "\n"),
+            (TokType::Dedent, ""),
+        ])
+    )
+}