Repository: lark-parser/lark
Pull request #1395 (2024-02-25 06:49:18)

Fix for when providing a transformer with a Token
Fixes issue #1394

codecov-commenter: ## [Codecov](https://app.codecov.io/gh/lark-parser/lark/pull/1395?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 89.39%. Comparing base [(`7646fb3`)](https://app.codecov.io/gh/lark-parser/lark/commit/7646fb31609c6c83e0998b121eeffb908f5ec5ba?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser) to head [(`e53a0ae`)](https://app.codecov.io/gh/lark-parser/lark/pull/1395?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser).


:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master    #1395   +/-   ##
=======================================
  Coverage   89.39%   89.39%           
=======================================
  Files          52       52           
  Lines        7730     7744   +14     
=======================================
+ Hits         6910     6923   +13     
- Misses        820      821    +1     
```

| [Flag](https://app.codecov.io/gh/lark-parser/lark/pull/1395/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/lark-parser/lark/pull/1395/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser) | `89.39% <100.00%> (+<0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser#carryforward-flags-in-the-pull-request-comment) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/lark-parser/lark/pull/1395?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lark-parser).


Patch:
diff --git a/lark/visitors.py b/lark/visitors.py
index ae9d128..2d30c53 100644
--- a/lark/visitors.py
+++ b/lark/visitors.py
@@ -158,7 +158,11 @@ class Transformer(_Decoratable, ABC, Generic[_Leaf_T, _Return_T]):
 
     def transform(self, tree: Tree[_Leaf_T]) -> _Return_T:
         "Transform the given tree, and return the final result"
-        return self._transform_tree(tree)
+        res = list(self._transform_children([tree]))
+        if not res:
+            return None     # type: ignore[return-value]
+        assert len(res) == 1
+        return res[0]
 
     def __mul__(
             self: 'Transformer[_Leaf_T, Tree[_Leaf_U]]',


Test patch:
diff --git a/tests/test_trees.py b/tests/test_trees.py
index dd95f6b..1f69869 100644
--- a/tests/test_trees.py
+++ b/tests/test_trees.py
@@ -447,5 +447,20 @@ class TestTrees(TestCase):
         with self.assertRaises(AttributeError):
             merge_transformers(T1(), module=T3())
 
+    def test_transform_token(self):
+        class MyTransformer(Transformer):
+            def INT(self, value):
+                return int(value)
+
+        t = Token('INT', '123')
+        assert MyTransformer().transform(t) == 123
+
+        class MyTransformer(Transformer):
+            def INT(self, value):
+                return Discard
+
+        assert MyTransformer().transform(t) is None
+
+
 if __name__ == '__main__':
     unittest.main()