Issue: RecursionError when querying by postgresql_ext BinaryJSONField in pytest
Author: cejohnson
State: closed
Created: 2017-11-03 19:30:34

Hello,

I'm using postgresql_ext with peewee, and am getting a RecursionError when attempting to query by a BinaryJSONField key, i.e.

```python
Model.select().where(Model.json['key'] == 'value')
```

This happens if you attempt to reference the field in at least the WHERE or ORDER BY clauses, and probably elsewhere.

What is really strange is that this only happens when testing with pytest; the same query seems to works fine when you actually run it live, and if you use raw SQL that also works in both cases.

Versions:
```
python: 3.6.2
peewee: 2.10.2
pytest: 3.2.3
```

I'm attaching a minimal test case that for me results in the following:
```
$ pytest -v test_model.py 
============================================================================================================================ test session starts =============================================================================================================================
platform darwin -- Python 3.6.2, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- []/.venv/bin/python3.6
cachedir: .cache
rootdir: [], inifile:
plugins: cov-2.5.1, celery-4.1.0
collected 3 items                                                                                                                                                                                                                                                             

test_model.py::test_select_basic PASSED
test_model.py::test_raw_select_json PASSED
test_model.py::test_select_json FAILED

================================================================================================================================== FAILURES ==================================================================================================================================
______________________________________________________________________________________________________________________________ test_select_json ______________________________________________________________________________________________________________________________

    def test_select_json():
        Model.drop_table(fail_silently=True)
        Model.create_table()
    
        Model.create(json={'key': 'value'})
    
        # RecursionError
        models = Model.select().where(Model.json['key'] == 'value')
>       assert len(models) == 1

test_model.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.venv/lib/python3.6/site-packages/peewee.py:3298: in __len__
    return len(self.execute())
../.venv/lib/python3.6/site-packages/peewee.py:3274: in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
../.venv/lib/python3.6/site-packages/peewee.py:2938: in _execute
    sql, params = self.sql()
../.venv/lib/python3.6/site-packages/peewee.py:3240: in sql
    return self.compiler().generate_select(self)
../.venv/lib/python3.6/site-packages/peewee.py:2043: in generate_select
    return self.build_query(clauses, alias_map)
../.venv/lib/python3.6/site-packages/peewee.py:1929: in build_query
    return self.parse_node(Clause(*clauses), alias_map)
../.venv/lib/python3.6/site-packages/peewee.py:1882: in parse_node
    sql, params, unknown = self._parse(node, alias_map, conv)
../.venv/lib/python3.6/site-packages/peewee.py:1848: in _parse
    sql, params = self._parse_map[node_type](node, alias_map, conv)
../.venv/lib/python3.6/site-packages/peewee.py:1774: in _parse_clause
    node.nodes, alias_map, conv, node.glue)
../.venv/lib/python3.6/site-packages/peewee.py:1907: in parse_node_list
    node_sql, node_params = self.parse_node(node, alias_map, conv)
../.venv/lib/python3.6/site-packages/peewee.py:1882: in parse_node
    sql, params, unknown = self._parse(node, alias_map, conv)
E   RecursionError: maximum recursion depth exceeded while calling a Python object
!!! Recursion detected (same locals & position)
```

I'm not sure if this might somehow be a pytest issue since I've only seen it when testing, but I figured I'd go ahead and start here. Apologies in advance if this is documented somewhere or I'm doing something dumb, and thanks for your time!

[test_model.py.txt](https://github.com/coleifer/peewee/files/1442250/test_model.py.txt)
(This should work basically out of the box, just need to change the postgresql connection string)

I believe this is a PyTest problem. Peewee uses recursive-descent to convert the AST into SQL. Given that it runs fine outside the py.test suite, I'd suggest you file this with the py.test developers.

@coleifer thanks for the quick response. I found a similar open issue there, I'll follow up with the py.test developers.

https://github.com/pytest-dev/pytest/issues/2724

Looks like I jumped the gun a little. I was able to recreate this issue using plain unittest, so I dug into it a little further using pdb. The root issue, as far as I can tell, appears to be that calling `self.interpolation` on a JsonLookup object returns itself as a param, leading to the recursion error at line 1894/1899 of parse_node. Why this would only happen during testing I have no idea; I'm pretty new to the Python ecosystem, so I'm not sure if it's doing anything differently when testing. Same thing happens with 2.7 FWIW.

pdb output:
```
(Pdb) s
--Call--
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1881)parse_node()
-> def parse_node(self, node, alias_map=None, conv=None):
(Pdb) s
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1882)parse_node()
-> sql, params, unknown = self._parse(node, alias_map, conv)
(Pdb) pp node
<playhouse.postgres_ext.JsonLookup object at 0x1070248d0>
(Pdb)

...

(Pdb) s
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1739)_parse_default()
-> return self.interpolation, [node]
(Pdb) s
--Return--
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1739)_parse_default()->('%s', [<playhouse.po...t 0x1070248d0>])
-> return self.interpolation, [node]
(Pdb)

...

--Return--
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1879)_parse()->('%s', [<playhouse.po...t 0x1070248d0>], True)
-> return sql, params, unknown
(Pdb) 

...

(Pdb) s
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1898)parse_node()
-> if isinstance(param, Node):
(Pdb) s
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1899)parse_node()
-> csql, cparams = self.parse_node(param)
(Pdb) pp param
<playhouse.postgres_ext.JsonLookup object at 0x1070248d0>
(Pdb) s
--Call--
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1881)parse_node()
-> def parse_node(self, node, alias_map=None, conv=None):
(Pdb) s
> /Users/chris/Projects/recursion_error/.venv/lib/python3.6/site-packages/peewee.py(1882)parse_node()
-> sql, params, unknown = self._parse(node, alias_map, conv)
(Pdb) pp node
<playhouse.postgres_ext.JsonLookup object at 0x1070248d0>
(Pdb)
```

I'm confused how something so fundamental could be an issue, as the `BinaryJSONField` is tested extensively in peewee's test suite. Furthermore, it's only happening in the tests?

Can you add a test-case to Peewee that replicates the issue? See:

https://github.com/coleifer/peewee/blob/f4ebd78b990efc6f0fc57b9a4d81f0cd6ecdf276/playhouse/tests/test_postgres.py#L606-L948

So, turns out the issue is not with core peewee, but apparently with the playhouse db_url module, or at least how I was using it.

Initializing the database with:
```
_database = peewee.Proxy()
_database.initialize(connect("postgresql://test:test@localhost:5432/test"))
```
results in the RecursionError, while everything is fine when doing:
```
_database = peewee.Proxy()
_database.initialize(postgres_ext.PostgresqlExtDatabase('test', user='test', password='test', host='localhost'))
```

Whether this is an actual problem with db_url or there was something else wrong with my configuration I'm not sure, but definitely an unintuitive error to track down. (Turns out we had been doing it the latter way in the actual project the whole time, which is presumably why the issue only appeared in testing. Whoops.)

Anyway, thanks again for your time!

Ahh, you can use the db_url module, but you need to use `postgresqlext` as the protocol.

https://github.com/coleifer/peewee/blob/master/playhouse/db_url.py#L129

LOL. Yep, that did it. I really appreciate your responsiveness, thanks again.