nebius/SWE-rebench-V2-PRs huggingface dataset:
Repository: lark-parser/lark
Pull request #133 (2018-04-23 22:15:59)

interpreter: default behavior changed to return the values
this behavior seems to be almost always the preferred one, now you don't need to define a handler for nodes that just visit all their children and return all values


Patch:
diff --git a/lark/tree.py b/lark/tree.py
index e6d5ed7..ad086d2 100644
--- a/lark/tree.py
+++ b/lark/tree.py
@@ -195,7 +195,7 @@ class Interpreter(object):
         return self.__default__
 
     def __default__(self, tree):
-        self.visit_children(tree)
+        return self.visit_children(tree)
 
 
 class Transformer_NoRecurse(Transformer):


Test patch:
diff --git a/tests/test_trees.py b/tests/test_trees.py
index c83b5ef..6017386 100644
--- a/tests/test_trees.py
+++ b/tests/test_trees.py
@@ -49,6 +49,14 @@ class TestTrees(TestCase):
 
         self.assertEqual(Interp2().visit(t), list('BCde'))
 
+        class Interp3(Interpreter):
+            def b(self, tree):
+                return 'B'
+
+            def c(self, tree):
+                return 'C'
+
+        self.assertEqual(Interp3().visit(t), list('BCd'))
Repository: lark-parser/lark
Pull request #260 (2018-10-24 15:35:55)

Method iter_subtrees_topdown added
I am not a big hacker, but looks working


Patch:
diff --git a/docs/classes.md b/docs/classes.md
index 3a69f73..68c133c 100644
--- a/docs/classes.md
+++ b/docs/classes.md
@@ -76,7 +76,11 @@ Returns all nodes of the tree whose data equals the given data.
 
 #### iter_subtrees(self)
 
-Iterates over all the subtrees, never returning to the same node twice (Lark's parse-tree is actually a DAG)
+Iterates over all the subtrees, never returning to the same node twice (Lark's parse-tree is actually a DAG).
+
+#### iter_subtrees_topdown(self)
+
+Iterates over all the subtrees, return nodes in order like pretty() does.
 
 #### \_\_eq\_\_, \_\_hash\_\_
 
diff --git a/lark/tree.py b/lark/tree.py
index c406f45..e892a44 100644
--- a/lark/tree.py
+++ b/lark/tree.py
@@ -5,6 +5,7 @@ except ImportError:
 
 from copy import deepcopy
 
+
 ###{standalone
 class Meta:
     pass
@@ -42,6 +43,7 @@ class Tree(object):
 
     def pretty(self, indent_str='  '):
         return ''.join(self._pretty(0, indent_str))
+
     def __eq__(self, other):
         try:
             return self.data == other.data and self.children == other.children
@@ -99,12 +101,22 @@ class Tree(object):
                 yield x
                 seen.add(id(x))
 
+    def iter_subtrees_topdown(self):
+        stack = [self]
+        while stack:
+            node = stack.pop()
+            if not isinstance(node, Tree):
+                continue
+            yield node
+            for n in reversed(node.children):
+                stack.append(n)
 
     def __deepcopy__(self, memo):
         return type(self)(self.data, deepcopy(self.children, memo))
 
     def copy(self):
         return type(self)(self.data, self.children)
+
     def set(self, data, children):
         self.data = data
         self.children = children


Test patch:
diff --git a/tests/test_trees.py b/tests/test_trees.py
index 564d02b..7e1d841 100644
--- a/tests/test_trees.py
+++ b/tests/test_trees.py
@@ -21,6 +21,17 @@ class TestTrees(TestCase):
         data = pickle.dumps(s)
         assert pickle.loads(data) == s
 
+    def test_iter_subtrees(self):
+        expected = [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z'),
+                    Tree('a', [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')])]
+        nodes = list(self.tree1.iter_subtrees())
+        self.assertEqual(nodes, expected)
+
+    def test_iter_subtrees_topdown(self):
+        expected = [Tree('a', [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')]),
+                    Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')]
+        nodes = list(self.tree1.iter_subtrees_topdown())
+        self.assertEqual(nodes, expected)
 
     def test_interp(self):
         t = Tree('a', [Tree('b', []), Tree('c', []), 'd'])
Repository: lark-parser/lark
Pull request #322 (2019-02-12 20:51:18)

Support hex escape (\xhh) in string literal
I'm trying to parse some binary opcodes with Lark and quite satisfied with what it offers so far. A minor gripe I had was limited support for string literal escape in Lark grammar.

Let's say I want to capture `\x01` in a string with a rule with embedded literal, while not keeping its terminal node. Introducing non-capturing regex literal (i.e. `_A: /x01/`) was not an option since otherwise grammar would get too complicated.

Then only left with some partial solutions:

```
A: /\x01/ ... // string literal always captured
B: "\u0001" ... // not captured, but verbose and maybe confusing to represent a byte
C: /\u0001/ ... // captured, confusing?
```

Here I want to directly use hex escape in my string literal like this:

```
D: "\x01" // string literal not captured, clearly representing a byte
```

It can be possible with a very little change in `_fix_escaping()` although I have no idea if there was any other reasons it didn't have one on the first place.

Maybe we could also add other [escape sequences](https://docs.python.org/3/reference/lexical_analysis.html) (i.e. `\ooo` for octal) if interested.


Patch:
diff --git a/lark/load_grammar.py b/lark/load_grammar.py
index bbf6fa3..2d5d547 100644
--- a/lark/load_grammar.py
+++ b/lark/load_grammar.py
@@ -346,7 +346,7 @@ def _fix_escaping(s):
             n2 = next(i)
             if n2 == '\\':
                 w += '\\\\'
-            elif n2 not in 'unftr':
+            elif n2 not in 'uxnftr':
                 w += '\\'
             w += n2
     w = w.replace('\\"', '"').replace("'", "\\'")


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
index 54fb288..7d99e20 100644
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -448,6 +448,12 @@ def _make_parser_test(LEXER, PARSER):
                         """)
             g.parse(u'\xa3\u0101\u00a3\u0203\n')
 
+        def test_hex_escape(self):
+            g = _Lark(r"""start: A B
+                          A: "\x01"
+                          B: /\x02/
+                          """)
+            g.parse('\x01\x02')
 
         @unittest.skipIf(PARSER == 'cyk', "Takes forever")
         def test_stack_for_ebnf(self):
@@ -1363,4 +1369,3 @@ for _LEXER in ('dynamic', 'dynamic_complete'):
 
 if __name__ == '__main__':
     unittest.main()
-
Repository: lark-parser/lark
Pull request #327 (2019-02-23 23:41:27)

Fix literal range escape assertion
Currently string literal range syntax doesn't work for unicode/hex escaped string because it assumes the length of literal always one which is not true for those escaped strings. It can be simply fixed by  using `_fix_escaping()` on `len()` although assert now looks a bit busy.


Patch:
diff --git a/lark/load_grammar.py b/lark/load_grammar.py
index 2d5d547..f95d0fa 100644
--- a/lark/load_grammar.py
+++ b/lark/load_grammar.py
@@ -389,7 +389,7 @@ class PrepareLiterals(Transformer_InPlace):
         assert start.type == end.type == 'STRING'
         start = start.value[1:-1]
         end = end.value[1:-1]
-        assert len(start) == len(end) == 1, (start, end, len(start), len(end))
+        assert len(_fix_escaping(start)) == len(_fix_escaping(end)) == 1, (start, end, len(_fix_escaping(start)), len(_fix_escaping(end)))
         regexp = '[%s-%s]' % (start, end)
         return ST('pattern', [PatternRE(regexp)])
 


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
index 68514a1..a61fc37 100644
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -456,6 +456,18 @@ def _make_parser_test(LEXER, PARSER):
                           """)
             g.parse('\x01\x02\xABCD')
 
+        def test_unicode_literal_range_escape(self):
+            g = _Lark(r"""start: A+
+                          A: "\u0061".."\u0063"
+                          """)
+            g.parse('abc')
+
+        def test_hex_literal_range_escape(self):
+            g = _Lark(r"""start: A+
+                          A: "\x01".."\x03"
+                          """)
+            g.parse('\x01\x02\x03')
+
         @unittest.skipIf(PARSER == 'cyk', "Takes forever")
         def test_stack_for_ebnf(self):
             """Verify that stack depth isn't an issue for EBNF grammars"""
Repository: lark-parser/lark
Pull request #339 (2019-03-13 10:40:26)

Fix v_args when used on classes with non-callable members
When `v_args` was used on an entire class but the class had static members that were not callable (such as a constant with some dictionary), it would crash on:
```
...
  File "./lark/utils.py", line 80, in smart_decorator
    return create_decorator(f.__func__.__call__, True)
AttributeError: 'dict' object has no attribute '__func__'
```

https://github.com/lark-parser/lark/issues/340


Patch:
diff --git a/lark/visitors.py b/lark/visitors.py
index 26ca242..71dfd42 100644
--- a/lark/visitors.py
+++ b/lark/visitors.py
@@ -74,6 +74,8 @@ class Transformer:
         for name, value in getmembers(cls):
             if name.startswith('_') or name in libmembers:
                 continue
+            if not callable(cls.__dict__[name]):
+                continue
 
             # Skip if v_args already applied (at the function level)
             if hasattr(cls.__dict__[name], 'vargs_applied'):


Test patch:
diff --git a/tests/test_trees.py b/tests/test_trees.py
index e5008ef..38f74d5 100644
--- a/tests/test_trees.py
+++ b/tests/test_trees.py
@@ -137,6 +137,8 @@ class TestTrees(TestCase):
             f = float
             sub = lambda self, a, b: a-b
 
+            not_a_method = {'other': 'stuff'}
+
             @v_args(inline=False)
             def add(self, values):
                 return sum(values)
Repository: lark-parser/lark
Pull request #482 (2019-11-18 09:49:54)

Allow comments in rule definitions
Fixes a bug whereby adding comments in the middle of multi line rule definitions causes a parsing error.

Example:
```
from lark import Lark

grammar = """
start: example+

example: "a"
  // Comment
  | "b"
"""

l = Lark(grammar)
```

Gives:
```
lark.exceptions.UnexpectedToken: Unexpected token Token(_OR, '|') at line 6, column 3.
```


Patch:
diff --git a/examples/lark.lark b/examples/lark.lark
index 915cf2e..7373c54 100644
--- a/examples/lark.lark
+++ b/examples/lark.lark
@@ -44,7 +44,7 @@ _NL: /(\r?\n)+\s*/
 %import common.INT -> NUMBER
 %import common.WS_INLINE
 
-COMMENT: "//" /[^\n]/*
+COMMENT: /\s*/ "//" /[^\n]/*
 
 %ignore WS_INLINE
 %ignore COMMENT
diff --git a/lark/load_grammar.py b/lark/load_grammar.py
index a65ca1e..1070f86 100644
--- a/lark/load_grammar.py
+++ b/lark/load_grammar.py
@@ -85,7 +85,7 @@ TERMINALS = {
     'REGEXP': r'/(?!/)(\\/|\\\\|[^/\n])*?/[%s]*' % _RE_FLAGS,
     '_NL': r'(\r?\n)+\s*',
     'WS': r'[ \t]+',
-    'COMMENT': r'//[^\n]*',
+    'COMMENT': r'\s*//[^\n]*',
     '_TO': '->',
     '_IGNORE': r'%ignore',
     '_DECLARE': r'%declare',


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
index caee80e..8cefcb8 100644
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -94,6 +94,16 @@ class TestParsers(unittest.TestCase):
         r = g.parse('xx')
         self.assertEqual( r.children[0].data, "c" )
 
+    def test_comment_in_rule_definition(self):
+        g = Lark("""start: a
+               a: "a"
+               // A comment
+               // Another
+                | "b"
+            """)
+        r = g.parse('b')
+        self.assertEqual( r.children[0].data, "a" )
+
     def test_visit_tokens(self):
         class T(Transformer):
             def a(self, children):
Repository: lark-parser/lark
Pull request #709 (2020-10-03 13:07:08)

Guard Lark.__doc__ assignment for tools.standalone
This change is necessary for standalones to work, otherwise you get errors like:


    kson/grammar/kson.py:2285: in Lark
        __doc__ += "\n\n" + LarkOptions.OPTIONS_DOC
    E   TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'


MegaIng: I could have sworn this guard was already there, but apparently it was only in `LarkOptions`. Also, if this is that broken, how did the tests pass? (Or did you run it with the `-O` flag?)
rec: Something about the standaloning process strips away the `__doc__` comments...?
MegaIng: Ok. We have in theory tests for this. But for some reason `exec` messes with the `__doc__` variable, so that despite the doc string being stripped away, it is still defined. But if you run it in an external file, it obviously doesn't work.
MegaIng: Ok, can you update this PR with the following code in `tests/test_tools.py` in `TestStandalone._create_standalone`, replacing the 
```
context = {]
```
line with
```
context = {'__doc__': None}
```
?

This simulates the behavior better.
rec: onnit!


Patch:
diff --git a/lark/lark.py b/lark/lark.py
index e749138..4acc9b8 100644
--- a/lark/lark.py
+++ b/lark/lark.py
@@ -310,7 +310,8 @@ class Lark(Serialize):
             with FS.open(cache_fn, 'wb') as f:
                 self.save(f)
 
-    __doc__ += "\n\n" + LarkOptions.OPTIONS_DOC
+    if __doc__:
+        __doc__ += "\n\n" + LarkOptions.OPTIONS_DOC
 
     __serialize_fields__ = 'parser', 'rules', 'options'
 


Test patch:
diff --git a/tests/test_tools.py b/tests/test_tools.py
index f7e0c47..ce995d8 100644
--- a/tests/test_tools.py
+++ b/tests/test_tools.py
@@ -25,7 +25,7 @@ class TestStandalone(TestCase):
         standalone.main(StringIO(grammar), 'start', print=pr)
         code = code_buf.getvalue()
 
-        context = {}
+        context = {'__doc__': None}
         exec(code, context)
         return context
Repository: lark-parser/lark
Pull request #759 (2020-11-15 10:28:34)

Make `LALR_CustomLexerWrapper` behave correctly + tests
This fixes the bug in #758, e.g. the example didn't work.


Note that it is probably a good idea to add a new example page for the new interface.


Patch:
diff --git a/lark/parser_frontends.py b/lark/parser_frontends.py
index 202382b..337ddeb 100644
--- a/lark/parser_frontends.py
+++ b/lark/parser_frontends.py
@@ -29,10 +29,9 @@ def get_frontend(parser, lexer):
                 def lex(self, lexer_state, parser_state):
                     return self.lexer.lex(lexer_state.text)
 
-            class LALR_CustomLexerWrapper(LALR_CustomLexer):
+            class LALR_CustomLexerWrapper(LALR_WithLexer):
                 def __init__(self, lexer_conf, parser_conf, options=None):
-                    super(LALR_CustomLexerWrapper, self).__init__(
-                        lexer, lexer_conf, parser_conf, options=options)
+                    super(LALR_CustomLexerWrapper, self).__init__(lexer_conf, parser_conf, options=options)
                 def init_lexer(self):
                     future_interface = getattr(lexer, '__future_interface__', False)
                     if future_interface:
@@ -163,13 +162,6 @@ class LALR_ContextualLexer(LALR_WithLexer):
 
 ###}
 
-class LALR_CustomLexer(LALR_WithLexer):
-    def __init__(self, lexer_cls, lexer_conf, parser_conf, options=None):
-        self.lexer = lexer_cls(lexer_conf)
-        debug = options.debug if options else False
-        self.parser = LALR_Parser(parser_conf, debug=debug)
-        WithLexer.__init__(self, lexer_conf, parser_conf, options)
-
 
 class Earley(WithLexer):
     def __init__(self, lexer_conf, parser_conf, options=None):


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
index 6d0981f..cdd2e1c 100644
--- a/tests/test_parser.py
+++ b/tests/test_parser.py
@@ -322,6 +322,22 @@ class TestParsers(unittest.TestCase):
 
     def test_alias(self):
         Lark("""start: ["a"] "b" ["c"] "e" ["f"] ["g"] ["h"] "x" -> d """)
+        
+    def test_backwards_custom_lexer(self):
+        class OldCustomLexer(Lexer):
+            def __init__(self, lexer_conf):
+                pass
+
+            def lex(self, text):
+                yield Token('A', 'A')
+        
+        p = Lark("""
+        start: A
+        %declare A
+        """, parser='lalr', lexer=OldCustomLexer)
+        
+        r = p.parse('')
+        self.assertEqual(r, Tree('start', [Token('A', 'A')]))
 
 
 
@@ -849,6 +865,8 @@ class CustomLexer(Lexer):
         self.lexer = TraditionalLexer(copy(lexer_conf))
     def lex(self, *args, **kwargs):
         return self.lexer.lex(*args, **kwargs)
+    
+    __future_interface__ = True
 
 def _tree_structure_check(a, b):
     """
Repository: chezou/tabula-py
Pull request #87 (2018-04-25 11:36:04)

Show stderr of Java process obviously
To investigate #86, this PR adds Java error.


Patch:
diff --git a/tabula/wrapper.py b/tabula/wrapper.py
index 7a642a0..297d981 100644
--- a/tabula/wrapper.py
+++ b/tabula/wrapper.py
@@ -73,7 +73,12 @@ def read_pdf(input_path,
     args = ["java"] + java_options + ["-jar", JAR_PATH] + options + [path]
 
     try:
-        output = subprocess.check_output(args)
+        output = subprocess.check_output(args, stderr=subprocess.STDOUT)
+
+    except subprocess.CalledProcessError as e:
+        print(e.output)
+        raise(ValueError)
+
     finally:
         if is_url:
             os.unlink(path)
@@ -134,7 +139,12 @@ def convert_into(input_path, output_path, output_format='csv', java_options=None
     args = ["java"] + java_options + ["-jar", JAR_PATH] + options + [path]
 
     try:
-        subprocess.check_output(args)
+        subprocess.check_output(args, stderr=subprocess.STDOUT)
+
+    except subprocess.CalledProcessError as e:
+        print(e.output)
+        raise(ValueError)
+
     finally:
         if is_url:
             os.unlink(path)


Test patch:
diff --git a/tests/test_read_pdf_table.py b/tests/test_read_pdf_table.py
index ee9aa73..d159509 100644
--- a/tests/test_read_pdf_table.py
+++ b/tests/test_read_pdf_table.py
@@ -84,6 +84,11 @@ class TestReadPdfTable(unittest.TestCase):
         with self.assertRaises(pd.errors.ParserError):
             tabula.read_pdf(pdf_path, pages=2)
 
+    def test_read_pdf_exception(self):
+        invalid_pdf_path = 'notexist.pdf'
+        with self.assertRaises(ValueError):
+            tabula.read_pdf(invalid_pdf_path)
+
     def test_convert_from(self):
         pdf_path = 'tests/resources/data.pdf'
         expected_csv = 'tests/resources/data_1.csv'