Repository: horejsek/python-fastjsonschema

Issue:
Unexpected caching when swapping `handlers` on `compile` function
Hello, recently I noticed that I cannot use the `compile` function twice if there is a change in the schemas...
Even when I change the handlers, the schemas are getting globally cached somehow...

The following example illustrates the problem:

```python
>>> from fastjsonschema import compile
>>> repo1 = {
...     "sch://schema1": {"type": "array"}
... }
>>> repo2 = {
...     "sch://schema1": {"type": "string"}
... }
>>> validator1 = compile({"$ref": "sch://schema1"}, handlers={"sch": repo1.__getitem__})
>>> assert validator1([1, 2, 3]) == [1, 2, 3]
>>> validator1([1, 2, 3])
[1, 2, 3]

>>> validator2 = compile({"$ref": "sch://schema1"}, handlers={"sch": repo2.__getitem__})
>>> validator2("hello world")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 4, in validate
  File "<string>", line 9, in validate_sch___schema1
fastjsonschema.exceptions.JsonSchemaValueException: data must be array

data must be array
>>>
```

I think this problem is happening because of the definition of the `RefResolver`:
```python
def __init__(self, base_uri, schema, store={}, cache=True, handlers={}):
    #                                      ^--- This is a mutable default (and I believe the culprit)
    ...
```
(probably related to the usage of a mutable value as default argument, ref: https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/)

Discussion:
horejsek: Thanks for finding the issue and preparing PR! Did you find the root issue? Because I would like to keep {} in the definition if possible, I like it like that so the doc is nicely generated and clear what type it expects.
abravalheri: Hi @horejsek thank you very much for the review.

I am afraid that function calls that do not specify all the arguments but instead rely on a default mutable argument are the root cause of the problem  😅

If you want to keep the empty dict in the function definition,  then the only way around is to always make sure all the calls to the function pass explicitly the handlers and formats arguments.

At that point it would be easier to make them mandatory arguments without a default value, right? If documentation is the problem, there could be a type annotation to cover for the lack of a default value...

There is another, not elegant workaround, that would be checking if the argumebt is empty and assigning a brand new dict to it...
horejsek: The mutable objects can be issue for sure. But this seems like something a bit different. What I meant is that probably there is only one place which is problematic, and not all the others need the change. I will take look into it later. If my feeling is correct, then I would prefer to keep interface as is and fix differently. If not, I will accept your solution.
abravalheri: Thanks @horejsek . Is it OK if I investigate this one better next week? I don't have access to a computer right now 😅
horejsek: Sure, no problem. I thought I will take a look. But if you want to deep more and will be faster than me (that might be true), feel free to do so. :)

Patch:
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index 6f97d76..1b1232d 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -1,11 +1,3 @@
-=== 2.15.2 (2021-12-12)
-
-* Fix nested oneOf and anyOf
-* Fix if-else block in anyOf block
-* Fix uniqueItems including lists or dicts
-* Add option to pass custom formats in generated code
-
-
 === 2.15.1 (2021-05-05)
 
 * Fix parsing date-time with +hhmm format
diff --git a/fastjsonschema/__init__.py b/fastjsonschema/__init__.py
index 572e6c1..7a372b4 100644
--- a/fastjsonschema/__init__.py
+++ b/fastjsonschema/__init__.py
@@ -75,7 +75,6 @@ Note that there are some differences compared to JSON schema standard:
 API
 ***
 """
-from functools import partial, update_wrapper
 
 from .draft04 import CodeGeneratorDraft04
 from .draft06 import CodeGeneratorDraft06
@@ -178,10 +177,7 @@ def compile(definition, handlers={}, formats={}, use_default=True):
     global_state = code_generator.global_state
     # Do not pass local state so it can recursively call itself.
     exec(code_generator.func_code, global_state)
-    func = global_state[resolver.get_scope_name()]
-    if formats:
-        return update_wrapper(partial(func, custom_formats=formats), func)
-    return func
+    return global_state[resolver.get_scope_name()]
 
 
 # pylint: disable=dangerous-default-value
@@ -217,7 +213,7 @@ def compile_to_code(definition, handlers={}, formats={}, use_default=True):
 
 
 def _factory(definition, handlers, formats={}, use_default=True):
-    resolver = RefResolver.from_schema(definition, handlers=handlers)
+    resolver = RefResolver.from_schema(definition, handlers=handlers, store={})
     code_generator = _get_code_generator_class(definition)(
         definition,
         resolver=resolver,
diff --git a/fastjsonschema/draft04.py b/fastjsonschema/draft04.py
index 7dd097e..394f263 100644
--- a/fastjsonschema/draft04.py
+++ b/fastjsonschema/draft04.py
@@ -65,7 +65,6 @@ class CodeGeneratorDraft04(CodeGenerator):
             ('patternProperties', self.generate_pattern_properties),
             ('additionalProperties', self.generate_additional_properties),
         ))
-        self._any_or_one_of_count = 0
 
     @property
     def global_state(self):
@@ -146,18 +145,16 @@ class CodeGeneratorDraft04(CodeGenerator):
 
         Valid values for this definition are 3, 4, 5, 10, 11, ... but not 8 for example.
         """
-        self._any_or_one_of_count += 1
-        count = self._any_or_one_of_count
-        self.l('{variable}_any_of_count{count} = 0', count=count)
+        self.l('{variable}_any_of_count = 0')
         for definition_item in self._definition['anyOf']:
             # When we know it's passing (at least once), we do not need to do another expensive try-except.
-            with self.l('if not {variable}_any_of_count{count}:', count=count, optimize=False):
+            with self.l('if not {variable}_any_of_count:', optimize=False):
                 with self.l('try:', optimize=False):
                     self.generate_func_code_block(definition_item, self._variable, self._variable_name, clear_variables=True)
-                    self.l('{variable}_any_of_count{count} += 1', count=count)
+                    self.l('{variable}_any_of_count += 1')
                 self.l('except JsonSchemaValueException: pass')
 
-        with self.l('if not {variable}_any_of_count{count}:', count=count, optimize=False):
+        with self.l('if not {variable}_any_of_count:', optimize=False):
             self.exc('{name} must be valid by one of anyOf definition', rule='anyOf')
 
     def generate_one_of(self):
@@ -176,18 +173,16 @@ class CodeGeneratorDraft04(CodeGenerator):
 
         Valid values for this definition are 3, 5, 6, ... but not 15 for example.
         """
-        self._any_or_one_of_count += 1
-        count = self._any_or_one_of_count
-        self.l('{variable}_one_of_count{count} = 0', count=count)
+        self.l('{variable}_one_of_count = 0')
         for definition_item in self._definition['oneOf']:
             # When we know it's failing (one of means exactly once), we do not need to do another expensive try-except.
-            with self.l('if {variable}_one_of_count{count} < 2:', count=count, optimize=False):
+            with self.l('if {variable}_one_of_count < 2:', optimize=False):
                 with self.l('try:', optimize=False):
                     self.generate_func_code_block(definition_item, self._variable, self._variable_name, clear_variables=True)
-                    self.l('{variable}_one_of_count{count} += 1', count=count)
+                    self.l('{variable}_one_of_count += 1')
                 self.l('except JsonSchemaValueException: pass')
 
-        with self.l('if {variable}_one_of_count{count} != 1:', count=count):
+        with self.l('if {variable}_one_of_count != 1:'):
             self.exc('{name} must be valid exactly by one of oneOf definition', rule='oneOf')
 
     def generate_not(self):
@@ -355,16 +350,8 @@ class CodeGeneratorDraft04(CodeGenerator):
         """
         self.create_variable_is_list()
         with self.l('if {variable}_is_list:'):
-            self.l(
-                'def fn(var): '
-                'return frozenset(dict((k, fn(v)) '
-                'for k, v in var.items()).items()) '
-                'if hasattr(var, "items") else tuple(fn(v) '
-                'for v in var) '
-                'if isinstance(var, (dict, list)) else str(var) '
-                'if isinstance(var, bool) else var')
             self.create_variable_with_length()
-            with self.l('if {variable}_len > len(set(fn({variable}_x) for {variable}_x in {variable})):'):
+            with self.l('if {variable}_len > len(set(str({variable}_x) for {variable}_x in {variable})):'):
                 self.exc('{name} must contain unique items', rule='uniqueItems')
 
     def generate_items(self):
diff --git a/fastjsonschema/draft07.py b/fastjsonschema/draft07.py
index 470e23b..23f792a 100644
--- a/fastjsonschema/draft07.py
+++ b/fastjsonschema/draft07.py
@@ -46,7 +46,7 @@ class CodeGeneratorDraft07(CodeGeneratorDraft06):
 
         Valid values are any between -10 and 0 or any multiplication of two.
         """
-        with self.l('try:', optimize=False):
+        with self.l('try:'):
             self.generate_func_code_block(
                 self._definition['if'],
                 self._variable,
diff --git a/fastjsonschema/generator.py b/fastjsonschema/generator.py
index 71e9b58..eb3442d 100644
--- a/fastjsonschema/generator.py
+++ b/fastjsonschema/generator.py
@@ -32,7 +32,6 @@ class CodeGenerator:
     def __init__(self, definition, resolver=None):
         self._code = []
         self._compile_regexps = {}
-        self._custom_formats = {}
 
         # Any extra library should be here to be imported only once.
         # Lines are imports to be printed in the file and objects
@@ -55,7 +54,7 @@ class CodeGenerator:
         self._validation_functions_done = set()
 
         if resolver is None:
-            resolver = RefResolver.from_schema(definition)
+            resolver = RefResolver.from_schema(definition, store={})
         self._resolver = resolver
 
         # add main function to `self._needed_validation_functions`
@@ -137,7 +136,7 @@ class CodeGenerator:
         self._validation_functions_done.add(uri)
         self.l('')
         with self._resolver.resolving(uri) as definition:
-            with self.l('def {}(data, custom_formats={{}}):', name):
+            with self.l('def {}(data):', name):
                 self.generate_func_code_block(definition, 'data', 'data', clear_variables=True)
                 self.l('return data')
 
@@ -191,7 +190,7 @@ class CodeGenerator:
             if uri not in self._validation_functions_done:
                 self._needed_validation_functions[uri] = name
             # call validation function
-            self.l('{}({variable}, custom_formats)', name)
+            self.l('{}({variable})', name)
 
 
     # pylint: disable=invalid-name
diff --git a/fastjsonschema/ref_resolver.py b/fastjsonschema/ref_resolver.py
index 38cf7e0..00b00b6 100644
--- a/fastjsonschema/ref_resolver.py
+++ b/fastjsonschema/ref_resolver.py
@@ -77,12 +77,19 @@ class RefResolver:
     def __init__(self, base_uri, schema, store={}, cache=True, handlers={}):
         """
         `base_uri` is URI of the referring document from the `schema`.
+        `store` is an dictionary that will be used to cache the fetched schemas
+        (if `cache=True`).
+
+        Please notice that you can have caching problems when compiling schemas
+        with colliding `$ref`. To force overwriting use `cache=False` or
+        explicitly pass the `store` argument (with a brand new dictionary)
         """
         self.base_uri = base_uri
         self.resolution_scope = base_uri
         self.schema = schema
         self.store = store
         self.cache = cache
+        # ^-- Create a brand new cache if the 
         self.handlers = handlers
         self.walk(schema)
 
diff --git a/fastjsonschema/version.py b/fastjsonschema/version.py
index 388db64..a3c7a0b 100644
--- a/fastjsonschema/version.py
+++ b/fastjsonschema/version.py
@@ -1,1 +1,1 @@
-VERSION = '2.15.2'
+VERSION = '2.15.1'


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 9651199..6238669 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -13,11 +13,6 @@ from fastjsonschema import JsonSchemaValueException, compile
 from fastjsonschema.draft07 import CodeGeneratorDraft07
 
 
-def pytest_configure(config):
-    config.addinivalue_line("markers", "none")
-    config.addinivalue_line("markers", "benchmark")
-
-
 @pytest.fixture
 def asserter():
     def f(definition, value, expected, formats={}):
diff --git a/tests/test_array.py b/tests/test_array.py
index 99c9254..8a04fa1 100644
--- a/tests/test_array.py
+++ b/tests/test_array.py
@@ -51,20 +51,6 @@ def test_min_items(asserter, value, expected):
     ([1], [1]),
     ([1, 1], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
     ([1, 2, 3], [1, 2, 3]),
-    ([True, False], [True, False]),
-    ([True, True], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    (['abc', 'bce', 'hhh'], ['abc', 'bce', 'hhh']),
-    (['abc', 'abc'], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    ([{'a': 'a'}, {'b': 'b'}], [{'a': 'a'}, {'b': 'b'}]),
-    ([{'a': 'a'}, {'a': 'a'}], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    ([{'a': 'a', 'b': 'b'}, {'b': 'b', 'c': 'c'}], [{'a': 'a', 'b': 'b'}, {'b': 'b', 'c': 'c'}]),
-    ([{'a': 'a', 'b': 'b'}, {'b': 'b', 'a': 'a'}], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    ([1, '1'], [1, '1']),
-    ([{'a': 'b'}, "{'a': 'b'}"], [{'a': 'b'}, "{'a': 'b'}"]),
-    ([[1, 2], [2, 1]], [[1, 2], [2, 1]]),
-    ([[1, 2], [1, 2]], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    ([{'a': {'b': {'c': [1, 2]}}}, {'a': {'b': {'c': [1, 2]}}}], JsonSchemaValueException('data must contain unique items', value='{data}', name='data', definition='{definition}', rule='uniqueItems')),
-    ([{'a': {'b': {'c': [2, 1]}}}, {'a': {'b': {'c': [1, 2]}}}], [{'a': {'b': {'c': [2, 1]}}}, {'a': {'b': {'c': [1, 2]}}}]),
 ])
 def test_unique_items(asserter, value, expected):
     asserter({
diff --git a/tests/test_compile_to_code.py b/tests/test_compile_to_code.py
index 84bc981..509e044 100644
--- a/tests/test_compile_to_code.py
+++ b/tests/test_compile_to_code.py
@@ -2,7 +2,6 @@ import os
 import pytest
 import shutil
 
-from fastjsonschema import JsonSchemaValueException
 from fastjsonschema import compile_to_code, compile as compile_spec
 
 @pytest.yield_fixture(autouse=True)
@@ -85,40 +84,3 @@ def test_compile_complex_one_of_all_of():
             }
         ]
     })
-
-
-def test_compile_to_code_custom_format():
-    formats = {'my-format': str.isidentifier}
-    code = compile_to_code({'type': 'string', 'format': 'my-format'}, formats=formats)
-    with open('temp/schema_3.py', 'w') as f:
-        f.write(code)
-    from temp.schema_3 import validate
-    assert validate("valid", formats) == "valid"
-    with pytest.raises(JsonSchemaValueException) as exc:
-        validate("not-valid", formats)
-    assert exc.value.message == "data must be my-format"
-
-
-def test_compile_to_code_custom_format_with_refs():
-    schema = {
-        'type': 'object',
-        'properties': {
-            'a': {'$ref': '#/definitions/a'}
-        },
-        'definitions': {
-            'a': {
-                '$id': '#/definitions/a',
-                'type': 'array',
-                'items': {'type': 'string', 'format': 'my-format'}
-            }
-        }
-    }
-    formats = {'my-format': str.isidentifier}
-    code = compile_to_code(schema, formats=formats)
-    with open('temp/schema_4.py', 'w') as f:
-        f.write(code)
-    from temp.schema_4 import validate
-    assert validate({"a": ["identifier"]}, formats) is not None
-    with pytest.raises(JsonSchemaValueException) as exc:
-        validate({"a": ["identifier", "not-valid"]}, formats)
-    assert exc.value.message == "data[1] must be my-format"
\ No newline at end of file
diff --git a/tests/test_integration.py b/tests/test_integration.py
index 706052c..8145fd6 100644
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1,6 +1,6 @@
 import pytest
 
-from fastjsonschema import JsonSchemaValueException
+from fastjsonschema import JsonSchemaValueException, compile
 
 
 definition = {
@@ -127,3 +127,18 @@ def test_any_of_with_patterns(asserter):
     }, {
         'hash': 'AAAXXX',
     })
+
+
+def test_swap_handlers():
+    # Make sure that by swapping resolvers, the schemas do not get cached
+    repo1 = {
+        "sch://schema": {"type": "array"}
+    }
+    validator1 = compile({"$ref": "sch://schema"}, handlers={"sch": repo1.__getitem__})
+    assert validator1([1, 2, 3]) is not None
+
+    repo2 = {
+        "sch://schema": {"type": "string"}
+    }
+    validator2 = compile({"$ref": "sch://schema"}, handlers={"sch": repo2.__getitem__})
+    assert validator2("hello world") is not None