Repository: ruby/prism
Pull request #3034 (2024-09-07 18:36:04)

Fix a token incompatibility for `Prism::Translation::Parser::Lexer`
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis.

## Parser gem (Expected)

Returns `tLPAREN2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 \
-ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tLPAREN` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```

After the update, the parser now returns `tLPAREN2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```

The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem. The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`.

With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`:

```diff
-          when :tLPAREN
-            actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
```


Patch:
diff --git a/lib/prism/translation/parser/lexer.rb b/lib/prism/translation/parser/lexer.rb
index 8191bd19..f6873607 100644
--- a/lib/prism/translation/parser/lexer.rb
+++ b/lib/prism/translation/parser/lexer.rb
@@ -134,7 +134,7 @@ module Prism
           MINUS_GREATER: :tLAMBDA,
           NEWLINE: :tNL,
           NUMBERED_REFERENCE: :tNTH_REF,
-          PARENTHESIS_LEFT: :tLPAREN,
+          PARENTHESIS_LEFT: :tLPAREN2,
           PARENTHESIS_LEFT_PARENTHESES: :tLPAREN_ARG,
           PARENTHESIS_RIGHT: :tRPAREN,
           PERCENT: :tPERCENT,
@@ -187,7 +187,14 @@ module Prism
         EXPR_BEG = 0x1 # :nodoc:
         EXPR_LABEL = 0x400 # :nodoc:
 
-        private_constant :TYPES, :EXPR_BEG, :EXPR_LABEL
+        # The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem.
+        # The following token types are listed as those classified as `tLPAREN`.
+        LPAREN_CONVERSION_TOKEN_TYPES = [
+          :kBREAK, :kCASE, :tDIVIDE, :kFOR, :kIF, :kNEXT, :kRETURN, :kUNTIL, :kWHILE, :tAMPER, :tANDOP, :tBANG, :tCOMMA, :tDOT2, :tDOT3,
+          :tEQL, :tLPAREN, :tLPAREN2, :tLSHFT, :tNL, :tOP_ASGN, :tOROP, :tPIPE, :tSEMI, :tSTRING_DBEG, :tUMINUS, :tUPLUS
+        ]
+
+        private_constant :TYPES, :EXPR_BEG, :EXPR_LABEL, :LPAREN_CONVERSION_TOKEN_TYPES
 
         # The Parser::Source::Buffer that the tokens were lexed from.
         attr_reader :source_buffer
@@ -268,6 +275,8 @@ module Prism
               value.chomp!(":")
             when :tLCURLY
               type = :tLBRACE if state == EXPR_BEG | EXPR_LABEL
+            when :tLPAREN2
+              type = :tLPAREN if tokens.empty? || LPAREN_CONVERSION_TOKEN_TYPES.include?(tokens.dig(-1, 0))
             when :tNTH_REF
               value = parse_integer(value.delete_prefix("$"))
             when :tOP_ASGN


Test patch:
diff --git a/test/prism/fixtures/unparser/corpus/literal/unary.txt b/test/prism/fixtures/unparser/corpus/literal/unary.txt
index 77685cb7..6992d86b 100644
--- a/test/prism/fixtures/unparser/corpus/literal/unary.txt
+++ b/test/prism/fixtures/unparser/corpus/literal/unary.txt
@@ -6,3 +6,4 @@
 -a
 +a
 -(-a).foo
++(+a).foo
diff --git a/test/prism/ruby/parser_test.rb b/test/prism/ruby/parser_test.rb
index b60ccf93..f4433420 100644
--- a/test/prism/ruby/parser_test.rb
+++ b/test/prism/ruby/parser_test.rb
@@ -271,8 +271,6 @@ module Prism
           case actual_token[0]
           when :kDO
             actual_token[0] = expected_token[0] if %i[kDO_BLOCK kDO_LAMBDA].include?(expected_token[0])
-          when :tLPAREN
-            actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
           when :tPOW
             actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR
           end
diff --git a/test/prism/snapshots/unparser/corpus/literal/unary.txt b/test/prism/snapshots/unparser/corpus/literal/unary.txt
index be19b045..ec2115e4 100644
--- a/test/prism/snapshots/unparser/corpus/literal/unary.txt
+++ b/test/prism/snapshots/unparser/corpus/literal/unary.txt
@@ -1,10 +1,10 @@
-@ ProgramNode (location: (1,0)-(8,9))
+@ ProgramNode (location: (1,0)-(9,9))
 ├── flags: ∅
 ├── locals: []
 └── statements:
-    @ StatementsNode (location: (1,0)-(8,9))
+    @ StatementsNode (location: (1,0)-(9,9))
     ├── flags: ∅
-    └── body: (length: 8)
+    └── body: (length: 9)
         ├── @ CallNode (location: (1,0)-(1,2))
         │   ├── flags: newline
         │   ├── receiver:
@@ -211,50 +211,98 @@
         │   ├── arguments: ∅
         │   ├── closing_loc: ∅
         │   └── block: ∅
-        └── @ CallNode (location: (8,0)-(8,9))
+        ├── @ CallNode (location: (8,0)-(8,9))
+        │   ├── flags: newline
+        │   ├── receiver:
+        │   │   @ CallNode (location: (8,1)-(8,9))
+        │   │   ├── flags: ∅
+        │   │   ├── receiver:
+        │   │   │   @ ParenthesesNode (location: (8,1)-(8,5))
+        │   │   │   ├── flags: ∅
+        │   │   │   ├── body:
+        │   │   │   │   @ StatementsNode (location: (8,2)-(8,4))
+        │   │   │   │   ├── flags: ∅
+        │   │   │   │   └── body: (length: 1)
+        │   │   │   │       └── @ CallNode (location: (8,2)-(8,4))
+        │   │   │   │           ├── flags: newline
+        │   │   │   │           ├── receiver:
+        │   │   │   │           │   @ CallNode (location: (8,3)-(8,4))
+        │   │   │   │           │   ├── flags: variable_call, ignore_visibility
+        │   │   │   │           │   ├── receiver: ∅
+        │   │   │   │           │   ├── call_operator_loc: ∅
+        │   │   │   │           │   ├── name: :a
+        │   │   │   │           │   ├── message_loc: (8,3)-(8,4) = "a"
+        │   │   │   │           │   ├── opening_loc: ∅
+        │   │   │   │           │   ├── arguments: ∅
+        │   │   │   │           │   ├── closing_loc: ∅
+        │   │   │   │           │   └── block: ∅
+        │   │   │   │           ├── call_operator_loc: ∅
+        │   │   │   │           ├── name: :-@
+        │   │   │   │           ├── message_loc: (8,2)-(8,3) = "-"
+        │   │   │   │           ├── opening_loc: ∅
+        │   │   │   │           ├── arguments: ∅
+        │   │   │   │           ├── closing_loc: ∅
+        │   │   │   │           └── block: ∅
+        │   │   │   ├── opening_loc: (8,1)-(8,2) = "("
+        │   │   │   └── closing_loc: (8,4)-(8,5) = ")"
+        │   │   ├── call_operator_loc: (8,5)-(8,6) = "."
+        │   │   ├── name: :foo
+        │   │   ├── message_loc: (8,6)-(8,9) = "foo"
+        │   │   ├── opening_loc: ∅
+        │   │   ├── arguments: ∅
+        │   │   ├── closing_loc: ∅
+        │   │   └── block: ∅
+        │   ├── call_operator_loc: ∅
+        │   ├── name: :-@
+        │   ├── message_loc: (8,0)-(8,1) = "-"
+        │   ├── opening_loc: ∅
+        │   ├── arguments: ∅
+        │   ├── closing_loc: ∅
+        │   └── block: ∅
+        └── @ CallNode (location: (9,0)-(9,9))
             ├── flags: newline
             ├── receiver:
-            │   @ CallNode (location: (8,1)-(8,9))
+            │   @ CallNode (location: (9,1)-(9,9))
             │   ├── flags: ∅
             │   ├── receiver:
-            │   │   @ ParenthesesNode (location: (8,1)-(8,5))
+            │   │   @ ParenthesesNode (location: (9,1)-(9,5))
             │   │   ├── flags: ∅
             │   │   ├── body:
-            │   │   │   @ StatementsNode (location: (8,2)-(8,4))
+            │   │   │   @ StatementsNode (location: (9,2)-(9,4))
             │   │   │   ├── flags: ∅
             │   │   │   └── body: (length: 1)
-            │   │   │       └── @ CallNode (location: (8,2)-(8,4))
+            │   │   │       └── @ CallNode (location: (9,2)-(9,4))
             │   │   │           ├── flags: newline
             │   │   │           ├── receiver:
-            │   │   │           │   @ CallNode (location: (8,3)-(8,4))
+            │   │   │           │   @ CallNode (location: (9,3)-(9,4))
             │   │   │           │   ├── flags: variable_call, ignore_visibility
             │   │   │           │   ├── receiver: ∅
             │   │   │           │   ├── call_operator_loc: ∅
             │   │   │           │   ├── name: :a
-            │   │   │           │   ├── message_loc: (8,3)-(8,4) = "a"
+            │   │   │           │   ├── message_loc: (9,3)-(9,4) = "a"
             │   │   │           │   ├── opening_loc: ∅
             │   │   │           │   ├── arguments: ∅
             │   │   │           │   ├── closing_loc: ∅
             │   │   │           │   └── block: ∅
             │   │   │           ├── call_operator_loc: ∅
-            │   │   │           ├── name: :-@
-            │   │   │           ├── message_loc: (8,2)-(8,3) = "-"
+            │   │   │           ├── name: :+@
+            │   │   │           ├── message_loc: (9,2)-(9,3) = "+"
             │   │   │           ├── opening_loc: ∅
             │   │   │           ├── arguments: ∅
             │   │   │           ├── closing_loc: ∅
             │   │   │           └── block: ∅
-            │   │   ├── opening_loc: (8,1)-(8,2) = "("
-            │   │   └── closing_loc: (8,4)-(8,5) = ")"
-            │   ├── call_operator_loc: (8,5)-(8,6) = "."
+            │   │   ├── opening_loc: (9,1)-(9,2) = "("
+            │   │   └── closing_loc: (9,4)-(9,5) = ")"
+            │   ├── call_operator_loc: (9,5)-(9,6) = "."
             │   ├── name: :foo
-            │   ├── message_loc: (8,6)-(8,9) = "foo"
+            │   ├── message_loc: (9,6)-(9,9) = "foo"
             │   ├── opening_loc: ∅
             │   ├── arguments: ∅
             │   ├── closing_loc: ∅
             │   └── block: ∅
             ├── call_operator_loc: ∅
-            ├── name: :-@
-            ├── message_loc: (8,0)-(8,1) = "-"
+            ├── name: :+@
+            ├── message_loc: (9,0)-(9,1) = "+"
             ├── opening_loc: ∅
             ├── arguments: ∅
             ├── closing_loc: ∅