Repository: tree-sitter/tree-sitter-java (language: python)

Issue:
Danzat/fix class literal precedence

Checklist:
- [x] All tests pass in CI.
- [x] There are sufficient tests for the new fix/feature.
- [x] Grammar rules have not been renamed unless absolutely necessary.
- [x] The conflicts section hasn't grown too much.
- [x] The parser size hasn't grown too much (check the value of STATE_COUNT in src/parser.c).
      

Danzat/fix class literal precedence

Checklist:
- [x] All tests pass in CI.
- [x] There are sufficient tests for the new fix/feature.
- [x] Grammar rules have not been renamed unless absolutely necessary.
- [x] The conflicts section hasn't grown too much.
- [x] The parser size hasn't grown too much (check the value of STATE_COUNT in src/parser.c).
      

Variable declaration with scoped generic type parsed as binary expression
### Problem description

The variable declaration `util.List<X> x = null;` gets parsed as nested binary expressions with operators `<`, `>` and `=`, but it should be a "local variable declaration".
This worked with v0.20.0 but does not anymore with v0.20.1. Also, it works if the type is not scoped, e.g. `List<x> x = null;`.

```java
class X {
	void foo() {
    	util.List<X> x = null;
    }
}
```

feat: support distinguishing escape sequences inside strings, fix multiline strings
Checklist:
- [x] All tests pass in CI.
- [x] There are sufficient tests for the new fix/feature. (Existing literals.txt reflects updates)
- [x] Grammar rules have not been renamed unless absolutely necessary.
- [x] The conflicts section hasn't grown too much.
- [x] The parser size hasn't grown too much (check the value of STATE_COUNT in src/parser.c). (1282 -> 1297)
      



Discussion / hints:
This closes #121 

This closes #121 

`git bisect` tells me that b042baa283d4e840361fd90d69f751606d42166c introduces the problem.
the relevant change to the grammar must be this then: https://github.com/tree-sitter/tree-sitter-java/pull/125

not sure how to make the CI run, but on my end running tree-sitter test and ./script/parse-examples is ok

edit: nvm it's since I'm a first time contributor
cc @bmahe 
Note that I derived a lot of inspiration from the way the javascript parser handles escape sequences [here](https://github.com/tree-sitter/tree-sitter-javascript/blob/15e85e80b851983fab6b12dce5a535f5a0df0f9c/grammar.js#L906)
Interesting changes!
@amaanq can we add some test cases to exercise the new cases being handled?
> Interesting changes!
> @amaanq can we add some test cases to exercise the new cases being handled?

I do believe the multiline string changes are covered with the tests you added, but I can add some for the escape sequences soon :)
@bmahe  Sorry, got caught up with life...added a test with the escape sequences

Let me know what you think of that.
oops, didnt mean to remove the request
Don't mean to be hasty or annoying...but I'd like to get this merged :slightly_smiling_face: 

ping @bmahe @aryx 
@amaanq can we add tests that demonstrate the intent of this PR? 
The escape sequence provided in your latest commit is successfully parsed.

So what problem is this pr trying to solve?


On the latest in the main branch:
```bash
[bruno@buo tree-sitter-java]$ cat test.java 
String a = "\n\a\b\fhi im a piece of text\t\v and im some more text \\\'\"\?";
```
```bash
[bruno@buo tree-sitter-java]$ tree-sitter parse test.java 
Warning: You have not configured any parser directories!
Please run `tree-sitter init-config` and edit the resulting
configuration file to indicate where we should look for
language grammars.

(program [0, 0] - [2, 0]
  (local_variable_declaration [0, 0] - [0, 78]
    type: (type_identifier [0, 0] - [0, 6])
    declarator: (variable_declarator [0, 7] - [0, 77]
      name: (identifier [0, 7] - [0, 8])
      value: (string_literal [0, 11] - [0, 77]))))
```
> @amaanq can we add tests that demonstrate the intent of this PR? The escape sequence provided in your latest commit is successfully parsed.
> 
> So what problem is this pr trying to solve?
> 
> On the latest in the main branch:
> 
> ```shell
> [bruno@buo tree-sitter-java]$ cat test.java 
> String a = "\n\a\b\fhi im a piece of text\t\v and im some more text \\\'\"\?";
> ```
> 
> ```shell
> [bruno@buo tree-sitter-java]$ tree-sitter parse test.java 
> Warning: You have not configured any parser directories!
> Please run `tree-sitter init-config` and edit the resulting
> configuration file to indicate where we should look for
> language grammars.
> 
> (program [0, 0] - [2, 0]
>   (local_variable_declaration [0, 0] - [0, 78]
>     type: (type_identifier [0, 0] - [0, 6])
>     declarator: (variable_declarator [0, 7] - [0, 77]
>       name: (identifier [0, 7] - [0, 8])
>       value: (string_literal [0, 11] - [0, 77]))))
> ```

output on my end (with my changes):
ts p test.java

(program [0, 0] - [1, 0]
  (local_variable_declaration [0, 0] - [0, 78]
    type: (type_identifier [0, 0] - [0, 6])
    declarator: (variable_declarator [0, 7] - [0, 77]
      name: (identifier [0, 7] - [0, 8])
      value: (string_literal [0, 11] - [0, 77]
        (escape_sequence [0, 12] - [0, 14])
        (escape_sequence [0, 14] - [0, 16])
        (escape_sequence [0, 16] - [0, 18])
        (escape_sequence [0, 18] - [0, 20])
        (string_fragment [0, 20] - [0, 41])
        (escape_sequence [0, 41] - [0, 43])
        (escape_sequence [0, 43] - [0, 45])
        (string_fragment [0, 45] - [0, 68])
        (escape_sequence [0, 68] - [0, 70])
        (escape_sequence [0, 70] - [0, 72])
        (escape_sequence [0, 72] - [0, 74])
        (escape_sequence [0, 74] - [0, 76])))))

with this, editors can differentiate pieces of regular text vs escape sequences, like the image below:
![image](https://user-images.githubusercontent.com/29718261/217375641-f960214d-4817-41e8-a479-d830d0296f8e.png)

the current parser doesn't allow for this. tree-sitter-python has a similar rule [here](https://github.com/tree-sitter/tree-sitter-python/blob/9e53981ec31b789ee26162ea335de71f02186003/grammar.js#L936), and tree-sitter-c [here](https://github.com/tree-sitter/tree-sitter-c/blob/7175a6dd5fc1cee660dce6fe23f6043d75af424a/grammar.js#L959)



> Please make a separate PR for the styling/spacing/ and all the other stuff that are independent of what the title of this PR says.
> 
> In this PR we should just see modifications to distinguish escape sequence inside strings, and related test files, and nothing else. One this is merged you can make another PR to add those eslint and ci checks for styling issue.

Gotcha, I believe I've removed everything unrelated to the point of the PR. Anything else @aryx?
There's a tons of stuff not related to the PR title still. all those semicolons, $ to _, etc.
Those are good changes, but they need to be in a separate PR.
The modifications in this PR should just be related to the title of the PR, so only the stuff that help distinguish escape sequences
inside strings
tidied up history a bit (so ugly commits dont get merged)
ping @aryx 

Fix patch:
diff --git a/grammar.js b/grammar.js
index dd5e8fa..7de9257 100644
--- a/grammar.js
+++ b/grammar.js
@@ -91,7 +91,6 @@ module.exports = grammar({
       $.false,
       $.character_literal,
       $.string_literal,
-      $.text_block,
       $.null_literal
     ),
 
@@ -361,7 +360,7 @@ module.exports = grammar({
       optional($.class_body)
     )),
 
-    field_access: $ => prec.dynamic(PREC.OBJ_ACCESS, seq(
+    field_access: $ => seq(
       field('object', choice($.primary_expression, $.super)),
       optional(seq(
         '.',
@@ -369,7 +368,7 @@ module.exports = grammar({
       )),
       '.',
       field('field', choice($.identifier, $._reserved_identifier, $.this))
-    )),
+    ),
 
     array_access: $ => seq(
       field('array', $.primary_expression),
diff --git a/src/grammar.json b/src/grammar.json
index ac49b90..8dd9cc3 100644
--- a/src/grammar.json
+++ b/src/grammar.json
@@ -2749,76 +2749,72 @@
       }
     },
     "field_access": {
-      "type": "PREC_DYNAMIC",
-      "value": 16,
-      "content": {
-        "type": "SEQ",
-        "members": [
-          {
-            "type": "FIELD",
-            "name": "object",
-            "content": {
-              "type": "CHOICE",
+      "type": "SEQ",
+      "members": [
+        {
+          "type": "FIELD",
+          "name": "object",
+          "content": {
+            "type": "CHOICE",
+            "members": [
+              {
+                "type": "SYMBOL",
+                "name": "primary_expression"
+              },
+              {
+                "type": "SYMBOL",
+                "name": "super"
+              }
+            ]
+          }
+        },
+        {
+          "type": "CHOICE",
+          "members": [
+            {
+              "type": "SEQ",
               "members": [
                 {
-                  "type": "SYMBOL",
-                  "name": "primary_expression"
+                  "type": "STRING",
+                  "value": "."
                 },
                 {
                   "type": "SYMBOL",
                   "name": "super"
                 }
               ]
+            },
+            {
+              "type": "BLANK"
             }
-          },
-          {
+          ]
+        },
+        {
+          "type": "STRING",
+          "value": "."
+        },
+        {
+          "type": "FIELD",
+          "name": "field",
+          "content": {
             "type": "CHOICE",
             "members": [
               {
-                "type": "SEQ",
-                "members": [
-                  {
-                    "type": "STRING",
-                    "value": "."
-                  },
-                  {
-                    "type": "SYMBOL",
-                    "name": "super"
-                  }
-                ]
+                "type": "SYMBOL",
+                "name": "identifier"
               },
               {
-                "type": "BLANK"
+                "type": "SYMBOL",
+                "name": "_reserved_identifier"
+              },
+              {
+                "type": "SYMBOL",
+                "name": "this"
               }
             ]
-          },
-          {
-            "type": "STRING",
-            "value": "."
-          },
-          {
-            "type": "FIELD",
-            "name": "field",
-            "content": {
-              "type": "CHOICE",
-              "members": [
-                {
-                  "type": "SYMBOL",
-                  "name": "identifier"
-                },
-                {
-                  "type": "SYMBOL",
-                  "name": "_reserved_identifier"
-                },
-                {
-                  "type": "SYMBOL",
-                  "name": "this"
-                }
-              ]
-            }
           }
-        ]
-      }
+        }
+      ]
     },
     "array_access": {
       "type": "SEQ",
diff --git a/src/parser.c b/src/parser.c
index 321012d..d8165f5 100644
--- a/src/parser.c
+++ b/src/parser.c
@@ -5228,7 +5228,7 @@ static inline bool sym_identifier_character_set_1(int32_t c) {
                   : c <= 67826)
                 : (c <= 67829 || (c < 67872
                   ? (c >= 67840 && c <= 67861)
-                  : c <= 67880)))))))))))))));
+                  : c <= 67883)))))))))))))));
 }
 
 static inline bool sym_identifier_character_set_2(int32_t c) {
@@ -5742,7 +5742,7 @@ static inline bool sym_identifier_character_set_2(int32_t c) {
                   : c <= 67826)
                 : (c <= 67829 || (c < 67872
                   ? (c >= 67840 && c <= 67861)
-                  : c <= 67880)))))))))))))));
+                  : c <= 67883)))))))))))))));
 }
 
 static inline bool sym_identifier_character_set_3(int32_t c) {
@@ -6256,7 +6256,7 @@ static inline bool sym_identifier_character_set_3(int32_t c) {
                   : c <= 67826)
                 : (c <= 67829 || (c < 67872
                   ? (c >= 67840 && c <= 67861)
-                  : c <= 67880)))))))))))))));
+                  : c <= 67883)))))))))))))));
 }
 
 static inline bool sym_identifier_character_set_4(int32_t c) {
@@ -6770,7 +6770,7 @@ static inline bool sym_identifier_character_set_4(int32_t c) {
                   : (c <= 67742 || (c >= 67808 && c <= 67826)))
                 : (c <= 67829 || (c < 67872
                   ? (c >= 67840 && c <= 67861)
-                  : c <= 67880)))))))))))))));
+                  : c <= 67883)))))))))))))));
 }
 
 static bool ts_lex(TSLexer *lexer, TSStateId state) {
@@ -66657,10 +66657,10 @@ static const TSParseActionEntry ts_parse_actions[] = {
   [1255] = {.entry = {.count = 2, .reusable = false}}, REDUCE(aux_sym_annotation_type_body_repeat1, 2), SHIFT_REPEAT(687),
   [1258] = {.entry = {.count = 1, .reusable = true}}, SHIFT(261),
   [1260] = {.entry = {.count = 1, .reusable = true}}, SHIFT(219),
-  [1262] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 3, .dynamic_precedence = 16, .production_id = 31),
-  [1264] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 3, .dynamic_precedence = 16, .production_id = 31),
-  [1266] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 3, .dynamic_precedence = 16, .production_id = 32),
-  [1268] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 3, .dynamic_precedence = 16, .production_id = 32),
+  [1262] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 3, .production_id = 31),
+  [1264] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 3, .production_id = 31),
+  [1266] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 3, .production_id = 32),
+  [1268] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 3, .production_id = 32),
   [1270] = {.entry = {.count = 1, .reusable = false}}, SHIFT(1081),
   [1272] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_primary_expression, 1, .production_id = 1),
   [1274] = {.entry = {.count = 1, .reusable = false}}, SHIFT(167),
@@ -66668,10 +66668,10 @@ static const TSParseActionEntry ts_parse_actions[] = {
   [1278] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_primary_expression, 1, .production_id = 1),
   [1280] = {.entry = {.count = 1, .reusable = true}}, SHIFT(107),
   [1282] = {.entry = {.count = 1, .reusable = false}}, SHIFT(1083),
-  [1284] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 5, .dynamic_precedence = 16, .production_id = 105),
-  [1286] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 5, .dynamic_precedence = 16, .production_id = 105),
-  [1288] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 5, .dynamic_precedence = 16, .production_id = 106),
-  [1290] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 5, .dynamic_precedence = 16, .production_id = 106),
+  [1284] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 5, .production_id = 105),
+  [1286] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 5, .production_id = 105),
+  [1288] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_field_access, 5, .production_id = 106),
+  [1290] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_field_access, 5, .production_id = 106),
   [1292] = {.entry = {.count = 1, .reusable = false}}, REDUCE(sym_array_access, 4, .production_id = 72),
   [1294] = {.entry = {.count = 1, .reusable = true}}, REDUCE(sym_array_access, 4, .production_id = 72),
   [1296] = {.entry = {.count = 1, .reusable = false}}, SHIFT(218),


Test patch:
diff --git a/test/corpus/expressions.txt b/test/corpus/expressions.txt
index aba0a29..852d9ca 100644
--- a/test/corpus/expressions.txt
+++ b/test/corpus/expressions.txt
@@ -1164,3 +1164,53 @@ $cibleVisée2 = 2;
     (assignment_expression
       (identifier)
       (decimal_integer_literal))))
+
+================================================================================
+Local variable declaration with scoped type identifiers
+================================================================================
+
+util.List<Integer> x = null;
+java.util.List<Integer> x = null;
+java.util.List<java.lang.Integer> x = null;
+
+--------------------------------------------------------------------------------
+
+(program
+  (local_variable_declaration
+    (generic_type
+      (scoped_type_identifier
+        (type_identifier)
+        (type_identifier))
+      (type_arguments
+        (type_identifier)))
+    (variable_declarator
+      (identifier)
+      (null_literal)))
+  (local_variable_declaration
+    (generic_type
+      (scoped_type_identifier
+        (scoped_type_identifier
+          (type_identifier)
+          (type_identifier))
+        (type_identifier))
+      (type_arguments
+        (type_identifier)))
+    (variable_declarator
+      (identifier)
+      (null_literal)))
+  (local_variable_declaration
+    (generic_type
+      (scoped_type_identifier
+        (scoped_type_identifier
+          (type_identifier)
+          (type_identifier))
+        (type_identifier))
+      (type_arguments
+        (scoped_type_identifier
+          (scoped_type_identifier
+            (type_identifier)
+            (type_identifier))
+          (type_identifier))))
+    (variable_declarator
+      (identifier)
+      (null_literal))))