Repository: tefra/xsdata (python)

Issue:
remove useless classes in the generated models + use booleans instead of True False enum.
I have 2 things that I need to improve to simplify the generated classes from my xsd file. A sample to illustrate the issue:
```
<?xml version="1.0" encoding="utf-8"?>
	<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
		<xs:element name="Parent">
		<xs:complexType>
			<xs:sequence>
					<xs:element ref="Child1" />
					<xs:element ref="Child2" />
			</xs:sequence>
		</xs:complexType>
	</xs:element>
	<xs:element name="Child1">
		<xs:complexType>
			<xs:attribute name="name" type="xs:string" use="required" />
			<xs:attribute name="id" type="xs:string" use="required" />
			<xs:attribute name="isTrue" type="xs:boolean" use="required" />
		</xs:complexType>
	</xs:element>
	<xs:element name="Child2">
		<xs:complexType>
			<xs:sequence>
				<xs:element minOccurs="0" maxOccurs="1" ref="Name" />
			</xs:sequence>
			<xs:attribute name="id" type="xs:string" use="required" />
			<xs:attribute name="isTrue" use="required">
				<xs:simpleType>
					<xs:restriction base="xs:NMTOKEN">
						<xs:enumeration value="Yes" />
						<xs:enumeration value="No" />
					</xs:restriction>
				</xs:simpleType>
			</xs:attribute>
		</xs:complexType>
	</xs:element>
	<xs:element name="Name" type="xs:string" />
</xs:schema>
```
Child1 is the ideal schema representation that will result in the needed output, but Child2 is the one that I have.

Problems:
1. The xsd file represents some strings attributes as a sequence of elements that refers to another element. 
Ex: `Name` in `Child2` element. This will results in an unnecessary `Name` class and I will need to remove it manually.

2. Some attributes are represented as a "Yes" and "No" enum, is there a way to convert such an attribute to boolean? and then parse it from the XML file as True if the value is Yes, and False if the value is No? 
 
 

Pull request:
Introduce codegen filter strategy: referred globals
## 📒 Description

Add a new filter strategy for generated classes in order to filter out global types that are not referenced anywhere.

This new strategy will accommodate certain schemas where simple types are defined as global elements and since xsdata default behavior is to generate all global types these dummy type elements with no usages make it to the final output.


Example

```xml
<?xml version="1.0" encoding="utf-8"?>
    ...
    <xs:element name="Name" type="xs:string" />
</xs:schema>
```

Resolves #691 

## 🔗 What I've Done

- Moved the filtering from the container into its own handler so we can add/update things easier.
- Added the new strategy
- Added the fallback strategy all: if the selected strategy filters out all types.

## 💬 Comments

- The default strategy is the highly suggested solution for most use cases and all wsdl!!!

## 🛫 Checklist

- [ ] Updated docs
- [ ] Added unit-tests
- [ ] [Sample tests](https://github.com/tefra/xsdata-samples) pass
- [ ] [W3C tests](https://github.com/tefra/xsdata-w3c-tests) pass


Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 93b6f028..669a9003 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -32,7 +32,7 @@ repos:
       - id: end-of-file-fixer
       - id: debug-statements
   - repo: https://github.com/myint/docformatter
-    rev: v1.5.0-rc1
+    rev: v1.5.0
     hooks:
       - id: docformatter
         args: ["--in-place", "--pre-summary-newline"]
diff --git a/xsdata/codegen/container.py b/xsdata/codegen/container.py
index 9b3b5e2b..0e14e00f 100644
--- a/xsdata/codegen/container.py
+++ b/xsdata/codegen/container.py
@@ -7,6 +7,7 @@ from typing import Optional
 from xsdata.codegen.handlers import AddAttributeSubstitutions
 from xsdata.codegen.handlers import CreateCompoundFields
 from xsdata.codegen.handlers import DesignateClassPackages
+from xsdata.codegen.handlers import FilterClasses
 from xsdata.codegen.handlers import FlattenAttributeGroups
 from xsdata.codegen.handlers import FlattenClassExtensions
 from xsdata.codegen.handlers import MergeAttributes
@@ -22,7 +23,6 @@ from xsdata.codegen.handlers import VacuumInnerClasses
 from xsdata.codegen.handlers import ValidateAttributesOverrides
 from xsdata.codegen.mixins import ContainerInterface
 from xsdata.codegen.models import Class
-from xsdata.codegen.models import get_qname
 from xsdata.codegen.models import Status
 from xsdata.codegen.utils import ClassUtils
 from xsdata.models.config import GeneratorConfig
@@ -150,25 +150,8 @@ class ClassContainer(ContainerInterface):
             designator.run()
 
     def filter_classes(self):
-        """
-        Filter classes to be generated.
-
-        1. If there are no global elements generate them all
-        2. Generate all global types and the referred types
-        """
-
-        # No global elements, generate them all!!!
-        if not any(obj.is_global_type for obj in self):
-            return
-
-        occurs = {}
-        for obj in self:
-            if obj.is_global_type:
-                occurs[obj.ref] = None
-                occurs.update({ref: None for ref in obj.references})
-
-        candidates = [obj for obj in self if obj.ref in occurs]
-        self.data = collections.group_by(candidates, key=get_qname)
+        """Filter the classes to be generated."""
+        FilterClasses(self).run()
 
     def add(self, item: Class):
         """Add class item to the container."""
@@ -178,6 +161,10 @@ class ClassContainer(ContainerInterface):
         self.data[qname].remove(item)
         self.add(item)
 
+    def set(self, items: List[Class]):
+        self.data.clear()
+        self.extend(items)
+
     def extend(self, items: List[Class]):
         """Add a list of classes the container."""
         collections.apply(items, self.add)
diff --git a/xsdata/codegen/handlers/__init__.py b/xsdata/codegen/handlers/__init__.py
index bb737faf..dd6521c6 100644
--- a/xsdata/codegen/handlers/__init__.py
+++ b/xsdata/codegen/handlers/__init__.py
@@ -1,6 +1,7 @@
 from .add_attribute_substitutions import AddAttributeSubstitutions
 from .create_compound_fields import CreateCompoundFields
 from .designate_class_packages import DesignateClassPackages
+from .filter_classes import FilterClasses
 from .flatten_attribute_groups import FlattenAttributeGroups
 from .flatten_class_extensions import FlattenClassExtensions
 from .merge_attributes import MergeAttributes
@@ -19,6 +20,7 @@ __all__ = [
     "AddAttributeSubstitutions",
     "CreateCompoundFields",
     "DesignateClassPackages",
+    "FilterClasses",
     "FlattenAttributeGroups",
     "FlattenClassExtensions",
     "MergeAttributes",
diff --git a/xsdata/codegen/handlers/filter_classes.py b/xsdata/codegen/handlers/filter_classes.py
new file mode 100644
index 00000000..47bb4df5
--- /dev/null
+++ b/xsdata/codegen/handlers/filter_classes.py
@@ -0,0 +1,52 @@
+from typing import List
+
+from xsdata.codegen.mixins import ContainerHandlerInterface
+from xsdata.codegen.models import Class
+from xsdata.logger import logger
+from xsdata.models.config import ClassFilterStrategy
+
+
+class FilterClasses(ContainerHandlerInterface):
+    """Filter classes for code generation based on the configuration output
+    filter strategy."""
+
+    __slots__ = ()
+
+    def run(self):
+        classes = []
+        filter_strategy = self.container.config.output.filter_strategy
+        if filter_strategy == ClassFilterStrategy.ALL_GLOBALS:
+            classes = self.filter_all_globals()
+        elif filter_strategy == ClassFilterStrategy.REFERRED_GLOBALS:
+            classes = self.filter_referred_globals()
+
+        if classes:
+            self.container.set(classes)
+        elif filter_strategy != ClassFilterStrategy.ALL:
+            logger.warning(
+                "The filter strategy '%s' returned no classes,"
+                " will generate all types.",
+                filter_strategy.value,
+            )
+
+    def filter_all_globals(self) -> List[Class]:
+        """Filter all globals and any referenced types."""
+        occurs = set()
+        for obj in self.container:
+            if obj.is_global_type:
+                occurs.add(obj.ref)
+                occurs.update(obj.references)
+
+        return [obj for obj in self.container if obj.ref in occurs]
+
+    def filter_referred_globals(self) -> List[Class]:
+        """Filter globals with any references."""
+        occurs = set()
+        for obj in self.container:
+            if obj.is_global_type:
+                references = list(obj.references)
+                occurs.update(references)
+                if references:
+                    occurs.add(obj.ref)
+
+        return [obj for obj in self.container if obj.ref in occurs]
diff --git a/xsdata/codegen/mixins.py b/xsdata/codegen/mixins.py
index 340a1c7a..b88bcb60 100644
--- a/xsdata/codegen/mixins.py
+++ b/xsdata/codegen/mixins.py
@@ -44,12 +44,16 @@ class ContainerInterface(abc.ABC):
 
     @abc.abstractmethod
     def extend(self, items: List[Class]):
-        """Add a list of classes the container."""
+        """Add a list of classes to the container."""
 
     @abc.abstractmethod
     def reset(self, item: Class, qname: str):
         """Update the given class qualified name."""
 
+    @abc.abstractmethod
+    def set(self, items: List[Class]):
+        """Set the list of classes to the container."""
+
 
 class HandlerInterface(abc.ABC):
     """Class handler interface."""
@@ -62,7 +66,7 @@ class HandlerInterface(abc.ABC):
 
 
 class RelativeHandlerInterface(HandlerInterface, metaclass=ABCMeta):
-    """Class handler interface with access to the complete classes
+    """Class handler interface with access to the complete classes'
     container."""
 
     __slots__ = "container"
diff --git a/xsdata/models/config.py b/xsdata/models/config.py
index b2b59408..9d8f9ccc 100644
--- a/xsdata/models/config.py
+++ b/xsdata/models/config.py
@@ -134,6 +134,21 @@ class DocstringStyle(Enum):
     BLANK = "Blank"
 
 
+class ClassFilterStrategy(Enum):
+    """
+    Class filter strategy.
+
+    :cvar ALL: all: Generate all types, discouraged!!!
+    :cvar ALL_GLOBALS: allGlobals: Generate all global types
+    :cvar REFERRED_GLOBALS: referredGlobals: Generate all global types
+        with at least one reference.
+    """
+
+    ALL = "all"
+    ALL_GLOBALS = "allGlobals"
+    REFERRED_GLOBALS = "referredGlobals"
+
+
 class ObjectType(Enum):
     """
     Object type enumeration.
@@ -223,6 +238,7 @@ class GeneratorOutput:
     :param format: Output format
     :param structure_style: Output structure style, default: filenames
     :param docstring_style: Docstring style, default: reStructuredText
+    :param filter_strategy: Class filter strategy, default: globals
     :param relative_imports: Use relative imports, default: false
     :param compound_fields: Use compound fields for repeatable elements, default: false
     :param max_line_length: Adjust the maximum line length, default: 79
@@ -237,6 +253,9 @@ class GeneratorOutput:
         default=StructureStyle.FILENAMES, name="Structure"
     )
     docstring_style: DocstringStyle = element(default=DocstringStyle.RST)
+    filter_strategy: ClassFilterStrategy = element(
+        default=ClassFilterStrategy.ALL_GLOBALS
+    )
     relative_imports: bool = element(default=False)
     compound_fields: CompoundFields = element(default_factory=CompoundFields)
     max_line_length: int = attribute(default=79)


Test patch:
diff --git a/tests/codegen/handlers/test_filter_classes.py b/tests/codegen/handlers/test_filter_classes.py
new file mode 100644
index 00000000..c61a277d
--- /dev/null
+++ b/tests/codegen/handlers/test_filter_classes.py
@@ -0,0 +1,73 @@
+from unittest import mock
+
+from xsdata.codegen.container import ClassContainer
+from xsdata.codegen.handlers import FilterClasses
+from xsdata.models.config import ClassFilterStrategy
+from xsdata.models.config import GeneratorConfig
+from xsdata.models.enums import Tag
+from xsdata.utils.testing import AttrFactory
+from xsdata.utils.testing import ClassFactory
+from xsdata.utils.testing import FactoryTestCase
+
+
+class FilterClassesTests(FactoryTestCase):
+    maxDiff = None
+
+    def setUp(self):
+        super().setUp()
+
+        self.container = ClassContainer(config=GeneratorConfig())
+        self.handler = FilterClasses(self.container)
+
+    def test_filter_all_globals(self):
+        complex_type = ClassFactory.elements(1)
+        enum_1 = ClassFactory.enumeration(2)
+        complex_type.attrs[0].types[0].reference = enum_1.ref
+
+        simple_type = ClassFactory.simple_type()
+        enum_2 = ClassFactory.enumeration(3)
+        simple_type.attrs[0].types[0].reference = enum_2.ref
+
+        element = ClassFactory.create(tag=Tag.ELEMENT, abstract=True)
+
+        expected = [complex_type, enum_1]
+        self.container.extend([complex_type, enum_1, simple_type, enum_2, element])
+        self.handler.run()
+        self.assertEqual(expected, list(self.container))
+
+    def test_filter_referred_globals(self):
+        self.container.config.output.filter_strategy = (
+            ClassFilterStrategy.REFERRED_GLOBALS
+        )
+
+        element_1 = ClassFactory.create(tag=Tag.ELEMENT, attrs=AttrFactory.list(2))
+        element_2 = ClassFactory.create(tag=Tag.ELEMENT, attrs=AttrFactory.list(2))
+        enum_1 = ClassFactory.enumeration(2)
+        element_2.attrs[0].types[0].reference = enum_1.ref
+
+        expected = [element_2, enum_1]
+        self.container.extend([element_1, element_2, enum_1])
+        self.handler.run()
+        self.assertEqual(expected, list(self.container))
+
+    def test_filter_all(self):
+        simple_type = ClassFactory.simple_type()
+        enumeration = ClassFactory.enumeration(2)
+        complex_type = ClassFactory.elements(2)
+        self.container.extend([simple_type, enumeration, complex_type])
+        self.container.config.output.filter_strategy = ClassFilterStrategy.ALL
+
+        self.handler.run()
+        self.assertEqual(3, len(list(self.container)))
+
+    @mock.patch("xsdata.codegen.handlers.filter_classes.logger.warning")
+    def test_run_with_strategy_not_all_with_no_classes(self, mock_warning):
+        classes = [ClassFactory.enumeration(2), ClassFactory.simple_type()]
+        self.container.extend(classes)
+        self.handler.run()
+        self.assertEqual(classes, list(self.container))
+
+        mock_warning.assert_called_once_with(
+            "The filter strategy '%s' returned no classes, will generate all types.",
+            "allGlobals",
+        )
diff --git a/tests/codegen/test_container.py b/tests/codegen/test_container.py
index 9d6bde9c..ba0e3fe6 100644
--- a/tests/codegen/test_container.py
+++ b/tests/codegen/test_container.py
@@ -7,7 +7,6 @@ from xsdata.codegen.models import Status
 from xsdata.models.config import GeneratorConfig
 from xsdata.models.enums import Tag
 from xsdata.utils.testing import AttrFactory
-from xsdata.utils.testing import AttrTypeFactory
 from xsdata.utils.testing import ClassFactory
 from xsdata.utils.testing import FactoryTestCase
 
@@ -144,11 +143,3 @@ class ClassContainerTests(FactoryTestCase):
         expected = [complex_type, enum_1]
         container.filter_classes()
         self.assertEqual(expected, list(container))
-
-    def test_filter_classes_with_only_simple_types(self):
-        classes = [ClassFactory.enumeration(2), ClassFactory.simple_type()]
-        container = ClassContainer(config=GeneratorConfig())
-        container.extend(classes)
-        container.filter_classes()
-
-        self.assertEqual(classes, list(container))
diff --git a/tests/models/test_config.py b/tests/models/test_config.py
index 4aea0c4f..2ed0b8a7 100644
--- a/tests/models/test_config.py
+++ b/tests/models/test_config.py
@@ -33,6 +33,7 @@ class GeneratorConfigTests(TestCase):
             '    <Format repr="true" eq="true" order="false" unsafeHash="false" frozen="false" slots="false" kwOnly="false">dataclasses</Format>\n'
             "    <Structure>filenames</Structure>\n"
             "    <DocstringStyle>reStructuredText</DocstringStyle>\n"
+            "    <FilterStrategy>allGlobals</FilterStrategy>\n"
             "    <RelativeImports>false</RelativeImports>\n"
             '    <CompoundFields defaultName="choice" forceDefaultName="false">false</CompoundFields>\n'
             "    <PostponedAnnotations>false</PostponedAnnotations>\n"
@@ -90,6 +91,7 @@ class GeneratorConfigTests(TestCase):
             ' frozen="false" slots="false" kwOnly="false">dataclasses</Format>\n'
             "    <Structure>filenames</Structure>\n"
             "    <DocstringStyle>reStructuredText</DocstringStyle>\n"
+            "    <FilterStrategy>allGlobals</FilterStrategy>\n"
             "    <RelativeImports>false</RelativeImports>\n"
             '    <CompoundFields defaultName="choice" forceDefaultName="false">false</CompoundFields>\n'
             "    <PostponedAnnotations>false</PostponedAnnotations>\n"