Repository: tefra/xsdata
Pull request #777 (2023-04-17 06:48:59)

fixes
## 📒 Description

> Write a brief description of your PR.


Resolves #xxxx

## 🔗 What I've Done

> Write a description of the steps taken to resolve the issue


## 💬 Comments

> A place to write any comments to the reviewer.
>

## 🛫 Checklist

- [ ] Updated docs
- [ ] Added unit-tests
- [ ] [Sample tests](https://github.com/tefra/xsdata-samples) pass
- [ ] [W3C tests](https://github.com/tefra/xsdata-w3c-tests) pass


sonarcloud[bot]: SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=tefra_xsdata&pullRequest=777)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=tefra_xsdata&pullRequest=777)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=tefra_xsdata&pullRequest=777)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=tefra_xsdata&pullRequest=777&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=tefra_xsdata&pullRequest=777&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=tefra_xsdata&pullRequest=777&metric=new_duplicated_lines_density&view=list)



Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 5ba43442..f7f22f7a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -11,11 +11,11 @@ repos:
     hooks:
       - id: reorder-python-imports
   - repo: https://github.com/ambv/black
-    rev: 23.1.0
+    rev: 23.3.0
     hooks:
       - id: black
   - repo: https://github.com/PyCQA/autoflake
-    rev: v2.0.1
+    rev: v2.1.0
     hooks:
       - id: autoflake
   - repo: https://github.com/PyCQA/flake8
@@ -41,7 +41,7 @@ repos:
       - id: docformatter
         args: [ "--in-place", "--pre-summary-newline" ]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.0.1
+    rev: v1.2.0
     hooks:
       - id: mypy
         files: ^(xsdata/)
diff --git a/setup.cfg b/setup.cfg
index 7e5af103..e0976271 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -61,7 +61,6 @@ lxml =
 soap =
     requests
 test =
-    codecov
     pre-commit
     pytest
     pytest-benchmark
diff --git a/xsdata/cli.py b/xsdata/cli.py
index 6d2c5b9b..c36dc314 100644
--- a/xsdata/cli.py
+++ b/xsdata/cli.py
@@ -116,6 +116,7 @@ def download(source: str, output: str):
 )
 @click.option("-c", "--config", default=".xsdata.xml", help="Project configuration")
 @click.option("-pp", "--print", is_flag=True, default=False, help="Print output")
+@click.option("--cache", is_flag=True, default=False, help="Cache sources loading")
 @model_options(GeneratorOutput)
 def generate(**kwargs: Any):
     """
@@ -127,6 +128,7 @@ def generate(**kwargs: Any):
     """
     source = kwargs.pop("source")
     stdout = kwargs.pop("print")
+    cache = kwargs.pop("cache")
     recursive = kwargs.pop("recursive")
     config_file = Path(kwargs.pop("config")).resolve()
 
@@ -136,7 +138,7 @@ def generate(**kwargs: Any):
 
     transformer = SchemaTransformer(config=config, print=stdout)
     uris = sorted(resolve_source(source, recursive=recursive))
-    transformer.process(uris)
+    transformer.process(uris, cache=cache)
 
     handler.emit_warnings()
 
diff --git a/xsdata/codegen/container.py b/xsdata/codegen/container.py
index 457c38cd..9e2ec605 100644
--- a/xsdata/codegen/container.py
+++ b/xsdata/codegen/container.py
@@ -38,13 +38,14 @@ class Steps:
 
 
 class ClassContainer(ContainerInterface):
-    __slots__ = ("data", "processors")
+    __slots__ = ("data", "processors", "step")
 
     def __init__(self, config: GeneratorConfig):
         """Initialize a class container instance with its processors based on
         the provided configuration."""
         super().__init__(config)
 
+        self.step: int = 0
         self.data: Dict = {}
 
         self.processors = {
@@ -82,8 +83,8 @@ class ClassContainer(ContainerInterface):
         condition callable."""
         for row in self.data.get(qname, []):
             if condition(row):
-                if row.status == Status.RAW:
-                    self.process_class(row, Steps.FLATTEN)
+                if row.status < self.step:
+                    self.process_class(row, self.step)
                     return self.find(qname, condition)
 
                 return row
@@ -91,8 +92,8 @@ class ClassContainer(ContainerInterface):
 
     def find_inner(self, source: Class, qname: str) -> Class:
         inner = ClassUtils.find_inner(source, qname)
-        if inner.status == Status.RAW:
-            self.process_class(inner, Steps.FLATTEN)
+        if inner.status < self.step:
+            self.process_class(inner, self.step)
 
         return inner
 
@@ -123,6 +124,7 @@ class ClassContainer(ContainerInterface):
         self.designate_classes()
 
     def process_classes(self, step: int) -> None:
+        self.step = step
         for obj in self:
             if obj.status < step:
                 self.process_class(obj, step)
diff --git a/xsdata/codegen/transformer.py b/xsdata/codegen/transformer.py
index 0ecbf951..69721afe 100644
--- a/xsdata/codegen/transformer.py
+++ b/xsdata/codegen/transformer.py
@@ -1,7 +1,11 @@
+import hashlib
 import io
 import json
 import os
+import pickle
+import tempfile
 from collections import defaultdict
+from pathlib import Path
 from typing import Callable
 from typing import Dict
 from typing import List
@@ -99,7 +103,19 @@ class SchemaTransformer:
         self.processed: List[str] = []
         self.preloaded: Dict = {}
 
-    def process(self, uris: List[str]):
+    def process(self, uris: List[str], cache: bool = False):
+        cache_file = self.get_cache_file(uris) if cache else None
+        if cache_file and cache_file.exists():
+            self.classes = pickle.loads(cache_file.read_bytes())
+        else:
+            self.process_sources(uris)
+
+        if cache_file and not cache_file.exists():
+            cache_file.write_bytes(pickle.dumps(self.classes))
+
+        self.process_classes()
+
+    def process_sources(self, uris: List[str]):
         sources = defaultdict(list)
         for uri in uris:
             tp = self.classify_resource(uri)
@@ -110,7 +126,6 @@ class SchemaTransformer:
         self.process_dtds(sources[TYPE_DTD])
         self.process_xml_documents(sources[TYPE_XML])
         self.process_json_documents(sources[TYPE_JSON])
-        self.process_classes()
 
     def process_definitions(self, uris: List[str]):
         """Process a list of wsdl resources."""
@@ -322,3 +337,9 @@ class SchemaTransformer:
             inner += sum(self.count_classes(cls.inner))
 
         return main, inner
+
+    @classmethod
+    def get_cache_file(cls, uris: List[str]) -> Path:
+        key = hashlib.md5("".join(uris).encode()).hexdigest()
+        tempdir = tempfile.gettempdir()
+        return Path(tempdir).joinpath(f"{key}.cache")
diff --git a/xsdata/formats/dataclass/context.py b/xsdata/formats/dataclass/context.py
index 3029f48d..67bd8a3c 100644
--- a/xsdata/formats/dataclass/context.py
+++ b/xsdata/formats/dataclass/context.py
@@ -89,6 +89,10 @@ class XmlContext:
             attribute_name_generator=self.attribute_name_generator,
         )
         for clazz in self.get_subclasses(object):
+            # This is new!
+            if clazz.__module__.startswith("_pytest"):
+                continue
+
             if self.class_type.is_model(clazz):
                 meta = builder.build_class_meta(clazz, None)
 


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index bc96fce6..927bca36 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -34,10 +34,10 @@ jobs:
           sudo apt-get install libxml2-dev libxslt-dev
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip setuptools tox codecov
+          python -m pip install --upgrade pip setuptools tox
       - name: Test
         run: |
-          tox -e py -- --cov=./xsdata --cov-branch --doctest-glob="docs/*.rst"
+          tox -e py -- --cov=./xsdata --cov-report=xml --cov-branch --doctest-glob="docs/*.rst"
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v3
         with:
diff --git a/tests/codegen/test_container.py b/tests/codegen/test_container.py
index ba0e3fe6..3512780b 100644
--- a/tests/codegen/test_container.py
+++ b/tests/codegen/test_container.py
@@ -69,6 +69,7 @@ class ClassContainerTests(FactoryTestCase):
         class_c = ClassFactory.enumeration(2, qname="b", status=Status.FLATTENING)
         mock_process_class.side_effect = process_class
         self.container.extend([class_a, class_b, class_c])
+        self.container.step = Steps.FLATTEN
 
         self.assertIsNone(self.container.find("nope"))
         self.assertEqual(class_a, self.container.find(class_a.qname))
@@ -89,6 +90,7 @@ class ClassContainerTests(FactoryTestCase):
             x.status = Status.FLATTENED
 
         mock_process_class.side_effect = process_class
+        self.container.step = Steps.FLATTEN
 
         self.assertEqual(first, self.container.find_inner(obj, "{a}a"))
         self.assertEqual(second, self.container.find_inner(obj, "{a}b"))
diff --git a/tests/codegen/test_transformer.py b/tests/codegen/test_transformer.py
index 6a4202fd..52cb5d5e 100644
--- a/tests/codegen/test_transformer.py
+++ b/tests/codegen/test_transformer.py
@@ -1,3 +1,4 @@
+import pickle
 import tempfile
 from pathlib import Path
 from unittest import mock
@@ -71,6 +72,45 @@ class SchemaTransformerTests(FactoryTestCase):
         mock_process_dtds.assert_called_once_with(uris[8:])
         mock_process_classes.assert_called_once_with()
 
+    @mock.patch.object(SchemaTransformer, "process_classes")
+    @mock.patch.object(SchemaTransformer, "process_sources")
+    @mock.patch.object(SchemaTransformer, "get_cache_file")
+    def test_process_from_cache(
+        self, mock_get_cache_file, mock_process_sources, mock_process_classes
+    ):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            cache = Path(tmpdir).joinpath("foo.cache")
+            classes = ClassFactory.list(2)
+            cache.write_bytes(pickle.dumps(classes))
+
+            mock_get_cache_file.return_value = cache
+            uris = ["a.xml", "b.xml"]
+
+            self.transformer.process(uris, cache=True)
+
+            self.assertEqual(classes, self.transformer.classes)
+            self.assertEqual(0, mock_process_sources.call_count)
+            mock_process_classes.assert_called_once_with()
+
+    @mock.patch.object(SchemaTransformer, "process_classes")
+    @mock.patch.object(SchemaTransformer, "process_sources")
+    @mock.patch.object(SchemaTransformer, "get_cache_file")
+    def test_process_with_cache(
+        self, mock_get_cache_file, mock_process_sources, mock_process_classes
+    ):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            cache = Path(tmpdir).joinpath("foo.cache")
+            classes = ClassFactory.list(2)
+            self.transformer.classes = classes
+            mock_get_cache_file.return_value = cache
+            uris = ["a.xml", "b.xml"]
+
+            self.transformer.process(uris, cache=True)
+            self.assertEqual(1, mock_process_sources.call_count)
+
+            self.assertEqual(classes, pickle.loads(cache.read_bytes()))
+            mock_process_classes.assert_called_once_with()
+
     @mock.patch.object(SchemaTransformer, "convert_schema")
     @mock.patch.object(SchemaTransformer, "convert_definitions")
     @mock.patch.object(SchemaTransformer, "parse_definitions")
@@ -449,3 +489,11 @@ class SchemaTransformerTests(FactoryTestCase):
         self.assertIsInstance(actual, ClassContainer)
         self.assertEqual(2, len(actual.data))
         self.assertEqual(self.transformer.config, actual.config)
+
+    def test_get_cache_file(self):
+        uris = ["a.xml", "b.json"]
+        actual = self.transformer.get_cache_file(uris)
+        tempdir = Path(tempfile.gettempdir())
+        expected = tempdir.joinpath("ae1bed744d3d3611e698a2d2ef5335d2.cache")
+
+        self.assertEqual(expected, actual)