Repository: HypothesisWorks/hypothesis
Pull request: Don't error on a Lambda in a line with Unicode identifiers
Closes #1700.

@lucaswiman’s guess is correct:

> This seems to be an off-by-two error with this example (source is 'mbda x: abs(x) > 1e-5))'), so I'm guessing it's probably that col_offset is computed on the bytes of the original file (UTF-8 encoded in this case)

The `inspect` module gives you source code as Unicode strings, but computes the index with the raw bytes. Doing a quick round-trip to bytes and back (using the same encoding detection as inspect is using) seems to fix this particular issue.

Fun little bug to dig into.

Problem statement:
Error with lambda in a line containing unicode identifiers
`.filter(lambda ...)` fails when used on a line with unicode identifiers prior to the lambda declaration. For example:
```python
from hypothesis import given
import hypothesis.strategies as st
π = 3.1415

@given(st.floats(min_value=-π, max_value=π).filter(lambda x: abs(x) > 1e-5))
def test_foo(bar):
    pass
```

This seems to be an off-by-two error with this example (`source` is `'mbda x: abs(x) > 1e-5))'`), so I'm guessing it's probably that `col_offset` is computed on the bytes of the original file (UTF-8 encoded in this case):
```
(Pdb) orig_source.encode('utf-8')[lambda_ast.col_offset :]
b'lambda x: abs(x) > 1e-5))'
```

## Testing conditions
Python 3.6 and 3.7; `hypothesis==3.83.2`, `pytest==4.0.2`.

## Full error

```python-traceback
$ python3.6 -m pytest test2.py --tb=short
pyenv-implicit: found multiple python3.6 in pyenv. Use version 3.6.
================================================================ test session starts ================================================================
platform darwin -- Python 3.6.5, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/lwiman/personal/nerdsniped/sine_testing/.hypothesis/examples')
rootdir: /Users/lwiman/personal/nerdsniped/sine_testing, inifile:
plugins: hypothesis-3.83.2
collected 1 item                                                                                                                                    

test2.py F                                                                                                                                    [100%]

===================================================================== FAILURES ======================================================================
_____________________________________________________________________ test_foo ______________________________________________________________________
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/engine.py:174: in test_function
    self._test_function(data)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/core.py:625: in evaluate_test_data
    escalate_hypothesis_internal_error()
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/core.py:605: in evaluate_test_data
    result = self.execute(data)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/core.py:574: in execute
    result = self.test_runner(data, run)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/executors.py:56: in default_new_style_executor
    return function(data)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/core.py:551: in run
    args, kwargs = data.draw(self.search_strategy)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:239: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/collections.py:55: in do_draw
    return tuple(data.draw(e) for e in self.element_strategies)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/collections.py:55: in <genexpr>
    return tuple(data.draw(e) for e in self.element_strategies)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:233: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py:576: in do_draw
    result = self.pack(data.draw(self.mapped_strategy))
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:233: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/lazy.py:156: in do_draw
    return data.draw(self.wrapped_strategy)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:233: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py:576: in do_draw
    result = self.pack(data.draw(self.mapped_strategy))
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:233: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/collections.py:55: in do_draw
    return tuple(data.draw(e) for e in self.element_strategies)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/collections.py:55: in <genexpr>
    return tuple(data.draw(e) for e in self.element_strategies)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:224: in draw
    return self.__draw(strategy, label=label)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/conjecture/data.py:233: in __draw
    return strategy.do_draw(self)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py:633: in do_draw
    data.note_event("Aborted test because unable to satisfy %r" % (self,))
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py:610: in __repr__
    get_pretty_function_description(self.condition),
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/reflection.py:375: in get_pretty_function_description
    result = extract_lambda_source(f)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/reflection.py:348: in extract_lambda_source
    source = source[source.index("lambda") :]
E   ValueError: substring not found

During handling of the above exception, another exception occurred:
test2.py:7: in test_foo
    ???
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/lazyformat.py:29: in __str__
    return self.__format_string % self.__args
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py:610: in __repr__
    get_pretty_function_description(self.condition),
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/reflection.py:375: in get_pretty_function_description
    result = extract_lambda_source(f)
../../../.pyenv/versions/3.6/lib/python3.6/site-packages/hypothesis/internal/reflection.py:348: in extract_lambda_source
    source = source[source.index("lambda") :]
E   ValueError: substring not found
-------------------------------------------------------------------- Hypothesis ---------------------------------------------------------------------
You can add @seed(306900588268218898824024975885321058451) to this test or run pytest with --hypothesis-seed=306900588268218898824024975885321058451 to reproduce this failure.
============================================================= 1 failed in 0.32 seconds ==============================================================
```

Hints from issue discussion:

Commit message: Add a newline to placate black

Patch to review:
diff --git a/hypothesis-python/RELEASE.rst b/hypothesis-python/RELEASE.rst
new file mode 100644
index 0000000000..45ddff1147
--- /dev/null
+++ b/hypothesis-python/RELEASE.rst
@@ -0,0 +1,4 @@
+RELEASE_TYPE: patch
+
+This patch fixes :issue:`1700`, where a line that contained a Unicode character
+before a lambda definition would cause an internal exception.
\ No newline at end of file
diff --git a/hypothesis-python/src/hypothesis/internal/reflection.py b/hypothesis-python/src/hypothesis/internal/reflection.py
index 78680c728a..f7d8cad7e2 100644
--- a/hypothesis-python/src/hypothesis/internal/reflection.py
+++ b/hypothesis-python/src/hypothesis/internal/reflection.py
@@ -25,6 +25,7 @@
 import hashlib
 import inspect
 import re
+import tokenize
 import types
 import uuid
 from functools import wraps
@@ -347,9 +348,54 @@ def extract_lambda_source(f):
         return if_confused
     lambda_ast = aligned_lambdas[0]
     assert lambda_ast.lineno == 1
-    source = source[lambda_ast.col_offset :].strip()
 
+    # If the source code contains Unicode characters, the bytes of the original
+    # file don't line up with the string indexes, and `col_offset` doesn't match
+    # the string we're using.  We need to convert the source code into bytes
+    # before slicing.
+    #
+    # Under the hood, the inspect module is using `tokenize.detect_encoding` to
+    # detect the encoding of the original source file.  We'll use the same
+    # approach to get the source code as bytes.
+    #
+    # See https://github.com/HypothesisWorks/hypothesis/issues/1700 for an
+    # example of what happens if you don't correct for this.
+    #
+    # Note: if the code doesn't come from a file (but, for example, a doctest),
+    # `getsourcefile` will return `None` and the `open()` call will fail with
+    # an OSError.  Or if `f` is a built-in function, in which case we get a
+    # TypeError.  In both cases, fall back to splitting the Unicode string.
+    # It's not perfect, but it's the best we can do.
+    #
+    # Note 2: You can only detect the encoding with `tokenize.detect_encoding`
+    # in Python 3.2 or later.  But that's okay, because the only version that
+    # affects for us is Python 2.7, and 2.7 doesn't support non-ASCII identifiers:
+    # https://www.python.org/dev/peps/pep-3131/. In this case we'll get an
+    # AttributeError.
+    #
+    try:
+        encoding, _ = tokenize.detect_encoding(
+            open(inspect.getsourcefile(f), "rb").readline
+        )
+        source_bytes = source.encode(encoding)
+        source_bytes = source_bytes[lambda_ast.col_offset :].strip()
+        source = source_bytes.decode(encoding)
+    except (AttributeError, OSError, TypeError):
+        source = source[lambda_ast.col_offset :].strip()
+
+    # Note: It is possible for this line to throw a ValueError in Python 3 if:
+    #
+    #  - There's a Unicode character in the line before the Lambda, and
+    #  - For some reason we can't detect the source encoding of the file
+    #
+    # because slicing on `lambda_ast.col_offset` will account for bytes, but
+    # the slice will be on Unicode characters.
+    #
+    # In practice this seems relatively rare, and I'm going to ignore this bug
+    # until I have a compelling reason to add even more complexity here.
+    #
     source = source[source.index("lambda") :]
+
     for i in hrange(len(source), len("lambda"), -1):  # pragma: no branch
         try:
             parsed = ast.parse(source[:i])
diff --git a/hypothesis-python/tests/cover/test_reflection.py b/hypothesis-python/tests/cover/test_reflection.py
index 2da9558189..ee6d02afcc 100644
--- a/hypothesis-python/tests/cover/test_reflection.py
+++ b/hypothesis-python/tests/cover/test_reflection.py
@@ -24,7 +24,7 @@
 import pytest
 from mock import MagicMock, Mock, NonCallableMagicMock, NonCallableMock
 
-from hypothesis.internal.compat import PY3, FullArgSpec, getfullargspec
+from hypothesis.internal.compat import PY2, PY3, FullArgSpec, getfullargspec
 from hypothesis.internal.reflection import (
     arg_string,
     convert_keyword_arguments,
@@ -666,3 +666,29 @@ def method(self, a, b):
 def test_required_args(target, args, kwargs, expected):
     # Mostly checking that `self` (and only self) is correctly excluded
     assert required_args(target, args, kwargs) == expected
+
+
+# fmt: off
+pi = "π"; is_str_pi = lambda x: x == pi  # noqa: E731
+# fmt: on
+
+
+def test_can_handle_unicode_identifier_in_same_line_as_lambda_def():
+    assert get_pretty_function_description(is_str_pi) == "lambda x: x == pi"
+
+
+@pytest.mark.skipif(PY2, reason="detect_encoding does not exist in Python 2")
+def test_can_render_lambda_with_no_encoding():
+    is_positive = lambda x: x > 0
+
+    # Monkey-patching out the `tokenize.detect_encoding` method here means
+    # that our reflection can't detect the encoding of the source file, and
+    # has to fall back to assuming it's ASCII.
+    import tokenize
+
+    old_detect_encoding = tokenize.detect_encoding
+    try:
+        del tokenize.detect_encoding
+        assert get_pretty_function_description(is_positive) == "lambda x: x > 0"
+    finally:
+        tokenize.detect_encoding = old_detect_encoding
diff --git a/hypothesis-python/tests/py3/test_unicode_identifiers.py b/hypothesis-python/tests/py3/test_unicode_identifiers.py
index d0bdfe98ac..9f6a639eaa 100644
--- a/hypothesis-python/tests/py3/test_unicode_identifiers.py
+++ b/hypothesis-python/tests/py3/test_unicode_identifiers.py
@@ -17,7 +17,8 @@
 
 from __future__ import absolute_import, division, print_function
 
-from hypothesis.internal.reflection import proxies
+from hypothesis import given, strategies as st
+from hypothesis.internal.reflection import get_pretty_function_description, proxies
 
 
 def test_can_copy_argspec_of_unicode_args():
@@ -40,3 +41,20 @@ def bar():
         return 2
 
     assert bar() == 2
+
+
+is_approx_π = lambda x: x == 3.1415  # noqa: E731
+
+
+def test_can_handle_unicode_identifier_in_same_line_as_lambda_def():
+    assert get_pretty_function_description(is_approx_π) == "lambda x: x == 3.1415"
+
+
+def test_regression_issue_1700():
+    π = 3.1415
+
+    @given(st.floats(min_value=-π, max_value=π).filter(lambda x: abs(x) > 1e-5))
+    def test_nonzero(x):
+        assert x != 0
+
+    test_nonzero()


Code review comments:
diff_hunk: @@ -347,9 +348,54 @@ def extract_lambda_source(f):
         return if_confused
     lambda_ast = aligned_lambdas[0]
     assert lambda_ast.lineno == 1
-    source = source[lambda_ast.col_offset :].strip()
 
+    # If the source code contains Unicode characters, the bytes of the original
+    # file don't line up with the string indexes, and `col_offset` doesn't match
+    # the string we're using.  We need to convert the source code into bytes
+    # before slicing.
+    #
+    # Under the hood, the inspect module is using `tokenize.detect_encoding` to
+    # detect the encoding of the original source file.  We'll use the same
+    # approach to get the source code as bytes.
+    #
+    # See https://github.com/HypothesisWorks/hypothesis/issues/1700 for an
+    # example of what happens if you don't correct for this.
+    #
+    # Note: if the code doesn't come from a file (but, for example, a doctest),
+    # `getsourcefile` will return `None` and the `open()` call will fail with
+    # an OSError.  Or if `f` is a built-in function, in which case we get a
+    # TypeError.  In both cases, fall back to splitting the Unicode string.
+    # It's not perfect, but it's the best we can do.
+    #
+    # Note 2: You can only detect the encoding with `tokenize.detect_encoding`
+    # in Python 3.2 or later.  But that's okay, because the only version that
+    # affects for us is Python 2.7, and 2.7 doesn't support non-ASCII identifiers:
+    # https://www.python.org/dev/peps/pep-3131/. In this case we'll get an
+    # AttributeError.
+    #
+    try:
+        encoding, _ = tokenize.detect_encoding(
+            open(inspect.getsourcefile(f), "rb").readline
+        )
+        source_bytes = source.encode(encoding)
+        source_bytes = source_bytes[lambda_ast.col_offset :].strip()
+        source = source_bytes.decode(encoding)
+    except (AttributeError, OSError, TypeError):
+        source = source[lambda_ast.col_offset :].strip()
+
+    # Note: It is possible for this line to throw a ValueError in Python 3 if:
+    #
+    #  - There's a Unicode character in the line before the Lambda, and
+    #  - For some reason we can't detect the source encoding of the file
+    #
+    # because slicing on `lambda_ast.col_offset` will account for bytes, but
+    # the slice will be on Unicode characters.
+    #
+    # In practice this seems relatively rare, and I'm going to ignore this bug
+    # until I have a compelling reason to add even more complexity here.
+    #
     source = source[source.index("lambda") :]
path: hypothesis-python/src/hypothesis/internal/reflection.py
text: @user1:
Certainly not saying that you have to *handle* the undetectable encoding problem, but I'd like to avoid the crash.  I think this is relatively simple, with 

```python
if "lambda" not in source:
    return if_confused
source = source[source.index("lambda") :]
```

...and a test that uses a unicode identifier with `detect_encoding` monkeypatched away.