Repository: useblocks/sphinx-needs
Pull request #1255 (2024-08-29 08:01:55)

🔧 Improve internal API for needs access
This PR makes it clearer when we are accessing the (mutable) unprocessed needs, during the read/collection phase,
or when we are accessing the (immutable) processed needs, during the write/analysis phase.
It does not affect the user facing API.

It also adds the abstracted `NeedsView` type for the processed needs.
This is not changed from the simple dict (`id -> need dict`), but
it provides the groundwork for potential future changes to make this more performant for large amounts of needs.

----

Note, this is essentially the lifecycle of needs access:

- reading/collection phase
   - retrieve cached (unprocessed) needs, or instantiate
   - add any external needs
   - before document read: remove all needs for that document
   - during document read: add needs from directives
   - after document read: update all needs for that document (with data such as `sections`)
   - cache (unprocessed) needs

- before write/analysis phase
   -  process needs (resolving dynamic values, computing backlinks etc)

- write/analysis phase
   - retrieve views of the processed needs (to be filtered, read, etc) 

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) Report
Attention: Patch coverage is `95.08197%` with `3 lines` in your changes missing coverage. Please review.
> Project coverage is 86.54%. Comparing base [(`4e10030`)](https://app.codecov.io/gh/useblocks/sphinx-needs/commit/4e10030581c03c59764e517ab750d3ddbb2695d8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) to head [(`929b773`)](https://app.codecov.io/gh/useblocks/sphinx-needs/commit/929b773f27fc038f1d821db2049d318a9663653a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks).
> Report is 23 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) | Patch % | Lines |
|---|---|---|
| [sphinx\_needs/data.py](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?src=pr&el=tree&filepath=sphinx_needs%2Fdata.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks#diff-c3BoaW54X25lZWRzL2RhdGEucHk=) | 92.85% | [2 Missing :warning: ](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) |
| [sphinx\_needs/directives/needflow/\_shared.py](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?src=pr&el=tree&filepath=sphinx_needs%2Fdirectives%2Fneedflow%2F_shared.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks#diff-c3BoaW54X25lZWRzL2RpcmVjdGl2ZXMvbmVlZGZsb3cvX3NoYXJlZC5weQ==) | 66.66% | [1 Missing :warning: ](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #1255      +/-   ##
==========================================
- Coverage   86.87%   86.54%   -0.34%     
==========================================
  Files          56       60       +4     
  Lines        6532     6903     +371     
==========================================
+ Hits         5675     5974     +299     
- Misses        857      929      +72     
```

| [Flag](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) | Coverage Δ | |
|---|---|---|
| [pytests](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks) | `86.54% <95.08%> (-0.34%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/useblocks/sphinx-needs/pull/1255?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=useblocks).


Patch:
diff --git a/docs/api.rst b/docs/api.rst
index 42b028b7..91abae59 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -30,3 +30,9 @@ Exceptions
 ----------
 .. automodule:: sphinx_needs.api.exceptions
    :members:
+
+Data
+----
+
+.. automodule:: sphinx_needs.data
+   :members: NeedsInfoType, NeedsView
diff --git a/docs/conf.py b/docs/conf.py
index 485421fc..2913a542 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -49,7 +49,6 @@ nitpick_ignore = [
     ("py:class", "docutils.statemachine.StringList"),
     ("py:class", "T"),
     ("py:class", "sphinx_needs.debug.T"),
-    ("py:class", "sphinx_needs.data.NeedsInfoType"),
 ]
 
 rst_epilog = """
@@ -757,7 +756,7 @@ def create_tutorial_needs(app: Sphinx, _env, _docnames):
 
     We do this dynamically, to avoid having to maintain the JSON file manually.
     """
-    all_data = SphinxNeedsData(app.env).get_or_create_needs()
+    all_data = SphinxNeedsData(app.env).get_needs_view()
     writer = NeedsList(app.config, outdir=app.confdir, confdir=app.confdir)
     for i in range(1, 5):
         test_id = f"T_00{i}"
diff --git a/sphinx_needs/api/need.py b/sphinx_needs/api/need.py
index 9de22f52..54e28f86 100644
--- a/sphinx_needs/api/need.py
+++ b/sphinx_needs/api/need.py
@@ -312,7 +312,7 @@ def add_need(
     # Add need to global need list
     #############################################################################################
 
-    if need_id in SphinxNeedsData(env).get_or_create_needs():
+    if SphinxNeedsData(env).has_need(need_id):
         if id:
             message = f"A need with ID {need_id} already exists, " f"title: {title!r}."
         else:  # this is a generated ID
@@ -457,7 +457,7 @@ def add_need(
     if needs_info["post_template"]:
         needs_info["post_content"] = _prepare_template(app, needs_info, "post_template")
 
-    SphinxNeedsData(env).get_or_create_needs()[need_id] = needs_info
+    SphinxNeedsData(env).add_need(needs_info)
 
     if needs_info["is_external"]:
         return []
@@ -604,12 +604,10 @@ def del_need(app: Sphinx, need_id: str) -> None:
     :param need_id: Sphinx need id.
     """
     data = SphinxNeedsData(app.env)
-    needs = data.get_or_create_needs()
-    if need_id in needs:
-        del needs[need_id]
-        data.remove_need_node(need_id)
-    else:
+    if not data.has_need(need_id):
         log_warning(logger, f"Given need id {need_id} not exists!", None, None)
+    else:
+        data.remove_need(need_id)
 
 
 def add_external_need(
diff --git a/sphinx_needs/builder.py b/sphinx_needs/builder.py
index d7514914..05c93867 100644
--- a/sphinx_needs/builder.py
+++ b/sphinx_needs/builder.py
@@ -85,7 +85,7 @@ class NeedsBuilder(Builder):
 
         filter_string = needs_config.builder_filter
         filtered_needs: list[NeedsInfoType] = filter_needs(
-            data.get_or_create_needs().values(),
+            data.get_needs_view().values(),
             needs_config,
             filter_string,
             append_warning="(from need_builder_filter)",
@@ -178,7 +178,7 @@ class NeedsIdBuilder(Builder):
 
         data = SphinxNeedsData(self.env)
         needs = (
-            data.get_or_create_needs().values()
+            data.get_needs_view().values()
         )  # We need a list of needs for later filter checks
         version = getattr(self.env.config, "version", "unset")
         needs_config = NeedsSphinxConfig(self.env.config)
diff --git a/sphinx_needs/data.py b/sphinx_needs/data.py
index 26824c7c..460bba00 100644
--- a/sphinx_needs/data.py
+++ b/sphinx_needs/data.py
@@ -4,7 +4,7 @@ which is stored in the Sphinx environment.
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Any, Final, Literal, Mapping, TypedDict
+from typing import TYPE_CHECKING, Any, Dict, Final, Literal, Mapping, NewType, TypedDict
 
 from sphinx.util.logging import getLogger
 
@@ -676,23 +676,85 @@ class NeedsUmlType(NeedsBaseDataType):
     content_calculated: str
 
 
+NeedsMutable = NewType("NeedsMutable", Dict[str, NeedsInfoType])
+"""A mutable view of the needs, before resolution
+"""
+
+NeedsView = NewType("NeedsView", Mapping[str, NeedsInfoType])
+"""A read-only view of the needs, after resolution
+(e.g. back links have been computed etc)
+"""
+
+
 class SphinxNeedsData:
     """Centralised access to sphinx-needs data, stored within the Sphinx environment."""
 
     def __init__(self, env: BuildEnvironment) -> None:
         self.env = env
 
-    def get_or_create_needs(self) -> dict[str, NeedsInfoType]:
-        """Get all needs, mapped by ID.
-
-        This is lazily created and cached in the environment.
-        """
+    @property
+    def _env_needs(self) -> dict[str, NeedsInfoType]:
         try:
             return self.env.needs_all_needs
         except AttributeError:
             self.env.needs_all_needs = {}
         return self.env.needs_all_needs
 
+    def has_need(self, need_id: str) -> bool:
+        """Check if a need with the given ID exists."""
+        return need_id in self._env_needs
+
+    def add_need(self, need: NeedsInfoType) -> None:
+        """Add an unprocessed need to the cache.
+
+        This will overwrite any existing need with the same ID.
+
+        .. important:: this should only be called within the read phase,
+            before the needs have been fully collected and resolved.
+        """
+        self._env_needs[need["id"]] = need
+
+    def remove_need(self, need_id: str) -> None:
+        """Remove a single need from the cache, if it exists.
+
+        .. important:: this should only be called within the read phase,
+            before the needs have been fully collected and resolved.
+        """
+        if need_id in self._env_needs:
+            del self._env_needs[need_id]
+        self.remove_need_node(need_id)
+
+    def remove_doc(self, docname: str) -> None:
+        """Remove all data related to a document from the cache.
+
+        .. important:: this should only be called within the read phase,
+            before the needs have been fully collected and resolved.
+        """
+        for need_id in list(self._env_needs):
+            if self._env_needs[need_id]["docname"] == docname:
+                del self._env_needs[need_id]
+                self.remove_need_node(need_id)
+        docs = self.get_or_create_docs()
+        for key, value in docs.items():
+            docs[key] = [doc for doc in value if doc != docname]
+
+    def get_needs_mutable(self) -> NeedsMutable:
+        """Get all needs, mapped by ID.
+
+        .. important:: this should only be called within the read phase,
+            before the needs have been fully collected and resolved.
+        """
+        return self._env_needs  # type: ignore[return-value]
+
+    def get_needs_view(self) -> NeedsView:
+        """Return a read-only view of all needs, after resolution.
+
+        .. important:: this should only be called within the write phase,
+            after the needs have been fully collected
+            and resolved (e.g. back links have been computed etc)
+        """
+        return self._env_needs  # type: ignore[return-value]
+
     @property
     def has_export_filters(self) -> bool:
         """Whether any filters have export IDs."""
@@ -793,7 +855,7 @@ class SphinxNeedsData:
             del self._needs_all_nodes[need_id]
 
     def get_need_node(self, need_id: str) -> Need | None:
-        """Get a need node from the cache, if it exists."""
+        """Get a copy of a need node from the cache, if it exists."""
         if need_id in self._needs_all_nodes:
             # We must create a copy of the node, as it may be reused several time
             # (multiple needextract for the same need) and the Sphinx ImageTransformator add location specific
@@ -822,8 +884,8 @@ def merge_data(
     this_data.get_or_create_filters().update(other_data.get_or_create_filters())
 
     # Update needs
-    needs = this_data.get_or_create_needs()
-    other_needs = other_data.get_or_create_needs()
+    needs = this_data._env_needs
+    other_needs = other_data._env_needs
     for other_id, other_need in other_needs.items():
         if other_id in needs:
             # we only want to warn if the need comes from one of the docs parsed in this worker
diff --git a/sphinx_needs/directives/need.py b/sphinx_needs/directives/need.py
index daeabfd9..7dc0964e 100644
--- a/sphinx_needs/directives/need.py
+++ b/sphinx_needs/directives/need.py
@@ -15,7 +15,7 @@ from sphinx.util.docutils import SphinxDirective
 from sphinx_needs.api import add_need
 from sphinx_needs.api.exceptions import NeedsInvalidException
 from sphinx_needs.config import NEEDS_CONFIG, NeedsSphinxConfig
-from sphinx_needs.data import NeedsInfoType, SphinxNeedsData
+from sphinx_needs.data import NeedsMutable, SphinxNeedsData
 from sphinx_needs.debug import measure_time
 from sphinx_needs.defaults import NEED_DEFAULT_OPTIONS
 from sphinx_needs.directives.needextend import Needextend, extend_needs_data
@@ -284,12 +284,7 @@ def purge_needs(app: Sphinx, env: BuildEnvironment, docname: str) -> None:
     Gets executed, if a doc file needs to be purged/ read in again.
     So this code delete all found needs for the given docname.
     """
-    data = SphinxNeedsData(env)
-    needs = data.get_or_create_needs()
-    for need_id in list(needs):
-        if needs[need_id]["docname"] == docname:
-            del needs[need_id]
-            data.remove_need_node(need_id)
+    SphinxNeedsData(env).remove_doc(docname)
 
 
 def analyse_need_locations(app: Sphinx, doctree: nodes.document) -> None:
@@ -305,7 +300,7 @@ def analyse_need_locations(app: Sphinx, doctree: nodes.document) -> None:
     """
     env = app.env
 
-    needs = SphinxNeedsData(env).get_or_create_needs()
+    needs = SphinxNeedsData(env).get_needs_mutable()
 
     hidden_needs: list[Need] = []
     for need_node in doctree.findall(Need):
@@ -380,7 +375,7 @@ def post_process_needs_data(app: Sphinx) -> None:
     """
     needs_config = NeedsSphinxConfig(app.config)
     needs_data = SphinxNeedsData(app.env)
-    needs = needs_data.get_or_create_needs()
+    needs = needs_data.get_needs_mutable()
     if needs and not needs_data.needs_is_post_processed:
         extend_needs_data(needs, needs_data.get_or_create_extends(), needs_config)
         resolve_dynamic_values(needs, app)
@@ -406,7 +401,7 @@ def process_need_nodes(app: Sphinx, doctree: nodes.document, fromdocname: str) -
     needs_data = SphinxNeedsData(app.env)
 
     # If no needs were defined, we do not need to do anything
-    if not needs_data.get_or_create_needs():
+    if not needs_data.get_needs_view():
         return
 
     post_process_needs_data(app)
@@ -426,7 +421,7 @@ def format_need_nodes(
 ) -> None:
     """Replace need nodes in the document with node trees suitable for output"""
     env = app.env
-    needs = SphinxNeedsData(env).get_or_create_needs()
+    needs = SphinxNeedsData(env).get_needs_view()
 
     # We try to avoid findall as much as possibles. so we reuse the already found need nodes in the current document.
     # for node_need in doctree.findall(Need):
@@ -448,7 +443,7 @@ def format_need_nodes(
         node_need.parent.replace(node_need, rendered_node)
 
 
-def check_links(needs: dict[str, NeedsInfoType], config: NeedsSphinxConfig) -> None:
+def check_links(needs: NeedsMutable, config: NeedsSphinxConfig) -> None:
     """Checks if set links are valid or are dead (referenced need does not exist.)
 
     For needs with dead links, an extra ``has_dead_links`` field is added and,
@@ -493,9 +488,7 @@ def check_links(needs: dict[str, NeedsInfoType], config: NeedsSphinxConfig) -> N
                                 )
 
 
-def create_back_links(
-    needs: dict[str, NeedsInfoType], config: NeedsSphinxConfig
-) -> None:
+def create_back_links(needs: NeedsMutable, config: NeedsSphinxConfig) -> None:
     """Create back-links in all found needs.
 
     These are fields for each link type, ``<link_name>_back``,
diff --git a/sphinx_needs/directives/needbar.py b/sphinx_needs/directives/needbar.py
index d3f16f4b..1d7f212a 100644
--- a/sphinx_needs/directives/needbar.py
+++ b/sphinx_needs/directives/needbar.py
@@ -292,7 +292,7 @@ def process_needbar(
         # 5. process content
         local_data_number = []
         need_list = list(
-            prepare_need_list(needs_data.get_or_create_needs().values())
+            prepare_need_list(needs_data.get_needs_view().values())
         )  # adds parts to need_list
 
         for line in local_data:
diff --git a/sphinx_needs/directives/needextend.py b/sphinx_needs/directives/needextend.py
index a6e84364..622ecae2 100644
--- a/sphinx_needs/directives/needextend.py
+++ b/sphinx_needs/directives/needextend.py
@@ -9,7 +9,11 @@ from sphinx.util.docutils import SphinxDirective
 
 from sphinx_needs.api.exceptions import NeedsInvalidFilter
 from sphinx_needs.config import NeedsSphinxConfig
-from sphinx_needs.data import NeedsExtendType, NeedsInfoType, SphinxNeedsData
+from sphinx_needs.data import (
+    NeedsExtendType,
+    NeedsMutable,
+    SphinxNeedsData,
+)
 from sphinx_needs.filter_common import filter_needs
 from sphinx_needs.logging import get_logger, log_warning
 from sphinx_needs.utils import add_doc
@@ -94,7 +98,7 @@ def _split_value(value: str) -> Sequence[tuple[str, bool]]:
 
 
 def extend_needs_data(
-    all_needs: dict[str, NeedsInfoType],
+    all_needs: NeedsMutable,
     extends: dict[str, NeedsExtendType],
     needs_config: NeedsSphinxConfig,
 ) -> None:
@@ -105,7 +109,7 @@ def extend_needs_data(
 
     for current_needextend in extends.values():
         need_filter = current_needextend["filter"]
-        if need_filter in all_needs:
+        if need_filter and need_filter in all_needs:
             # a single known ID
             found_needs = [all_needs[need_filter]]
         elif need_filter is not None and re.fullmatch(
diff --git a/sphinx_needs/directives/needextract.py b/sphinx_needs/directives/needextract.py
index 5742cf33..ecf36e0a 100644
--- a/sphinx_needs/directives/needextract.py
+++ b/sphinx_needs/directives/needextract.py
@@ -92,7 +92,7 @@ def process_needextract(
             continue
 
         current_needextract: NeedsExtractType = node.attributes
-        all_needs = SphinxNeedsData(env).get_or_create_needs()
+        all_needs = SphinxNeedsData(env).get_needs_view()
         content = nodes.container()
         content.attributes["ids"] = [current_needextract["target_id"]]
 
diff --git a/sphinx_needs/directives/needfilter.py b/sphinx_needs/directives/needfilter.py
index ddbfe6ca..9e13638c 100644
--- a/sphinx_needs/directives/needfilter.py
+++ b/sphinx_needs/directives/needfilter.py
@@ -85,7 +85,7 @@ def process_needfilters(
     builder = app.builder
     env = app.env
     needs_config = NeedsSphinxConfig(env.config)
-    all_needs = SphinxNeedsData(env).get_or_create_needs()
+    all_needs = SphinxNeedsData(env).get_needs_view()
 
     # NEEDFILTER
     # for node in doctree.findall(Needfilter):
diff --git a/sphinx_needs/directives/needflow/_graphviz.py b/sphinx_needs/directives/needflow/_graphviz.py
index 97707466..c84f8322 100644
--- a/sphinx_needs/directives/needflow/_graphviz.py
+++ b/sphinx_needs/directives/needflow/_graphviz.py
@@ -49,7 +49,7 @@ def process_needflow_graphviz(
 ) -> None:
     needs_config = NeedsSphinxConfig(app.config)
     env_data = SphinxNeedsData(app.env)
-    all_needs = env_data.get_or_create_needs()
+    all_needs = env_data.get_needs_view()
 
     link_type_names = [link["option"].upper() for link in needs_config.extra_links]
     allowed_link_types_options = [link.upper() for link in needs_config.flow_link_types]
diff --git a/sphinx_needs/directives/needflow/_plantuml.py b/sphinx_needs/directives/needflow/_plantuml.py
index c94ed97f..94a9d896 100644
--- a/sphinx_needs/directives/needflow/_plantuml.py
+++ b/sphinx_needs/directives/needflow/_plantuml.py
@@ -203,7 +203,7 @@ def process_needflow_plantuml(
     env = app.env
     needs_config = NeedsSphinxConfig(app.config)
     env_data = SphinxNeedsData(env)
-    all_needs = env_data.get_or_create_needs()
+    all_needs = env_data.get_needs_view()
 
     link_type_names = [link["option"].upper() for link in needs_config.extra_links]
     allowed_link_types_options = [link.upper() for link in needs_config.flow_link_types]
diff --git a/sphinx_needs/directives/needflow/_shared.py b/sphinx_needs/directives/needflow/_shared.py
index 15db87da..94ef39c4 100644
--- a/sphinx_needs/directives/needflow/_shared.py
+++ b/sphinx_needs/directives/needflow/_shared.py
@@ -8,6 +8,7 @@ from sphinx_needs.config import LinkOptionsType
 from sphinx_needs.data import (
     NeedsFlowType,
     NeedsInfoType,
+    NeedsView,
 )
 from sphinx_needs.logging import get_logger
 
@@ -15,17 +16,16 @@ logger = get_logger(__name__)
 
 
 def filter_by_tree(
-    all_needs: dict[str, NeedsInfoType],
+    all_needs: NeedsView,
     root_id: str,
     link_types: list[LinkOptionsType],
     direction: Literal["both", "incoming", "outgoing"],
     depth: int | None,
-) -> dict[str, NeedsInfoType]:
+) -> NeedsView:
     """Filter all needs by the given ``root_id``,
     and all needs that are connected to the root need by the given ``link_types``, in the given ``direction``."""
-    need_items: dict[str, NeedsInfoType] = {}
     if root_id not in all_needs:
-        return need_items
+        return NeedsView({})
     roots = {root_id: (0, all_needs[root_id])}
     link_prefixes = (
         ("_back",)
@@ -37,6 +37,7 @@ def filter_by_tree(
     links_to_process = [
         link["option"] + d for link in link_types for d in link_prefixes
     ]
+    need_items: dict[str, NeedsInfoType] = {}
     while roots:
         root_id, (root_depth, root) = roots.popitem()
         if root_id in need_items:
@@ -53,7 +54,7 @@ def filter_by_tree(
                 }
             )
 
-    return need_items
+    return NeedsView(need_items)
 
 
 def get_root_needs(found_needs: list[NeedsInfoType]) -> list[NeedsInfoType]:
diff --git a/sphinx_needs/directives/needgantt.py b/sphinx_needs/directives/needgantt.py
index 50b0367c..897ed65e 100644
--- a/sphinx_needs/directives/needgantt.py
+++ b/sphinx_needs/directives/needgantt.py
@@ -170,7 +170,7 @@ def process_needgantt(
             continue
 
         current_needgantt: NeedsGanttType = node.attributes
-        all_needs_dict = SphinxNeedsData(env).get_or_create_needs()
+        all_needs_dict = SphinxNeedsData(env).get_needs_view()
 
         content = []
         try:
diff --git a/sphinx_needs/directives/needlist.py b/sphinx_needs/directives/needlist.py
index 6f130a75..8d692d31 100644
--- a/sphinx_needs/directives/needlist.py
+++ b/sphinx_needs/directives/needlist.py
@@ -85,7 +85,7 @@ def process_needlist(
 
         current_needfilter: NeedsListType = node.attributes
         content: list[nodes.Node] = []
-        all_needs = list(SphinxNeedsData(env).get_or_create_needs().values())
+        all_needs = list(SphinxNeedsData(env).get_needs_view().values())
         found_needs = process_filters(
             app,
             all_needs,
diff --git a/sphinx_needs/directives/needpie.py b/sphinx_needs/directives/needpie.py
index 366376c5..838d3b66 100644
--- a/sphinx_needs/directives/needpie.py
+++ b/sphinx_needs/directives/needpie.py
@@ -158,7 +158,7 @@ def process_needpie(
 
         sizes = []
         need_list = list(
-            prepare_need_list(needs_data.get_or_create_needs().values())
+            prepare_need_list(needs_data.get_needs_view().values())
         )  # adds parts to need_list
         if content and not current_needpie["filter_func"]:
             for line in content:
diff --git a/sphinx_needs/directives/needsequence.py b/sphinx_needs/directives/needsequence.py
index cc604c28..05a23c91 100644
--- a/sphinx_needs/directives/needsequence.py
+++ b/sphinx_needs/directives/needsequence.py
@@ -12,7 +12,12 @@ from sphinxcontrib.plantuml import (
 )
 
 from sphinx_needs.config import NeedsSphinxConfig
-from sphinx_needs.data import NeedsInfoType, NeedsSequenceType, SphinxNeedsData
+from sphinx_needs.data import (
+    NeedsInfoType,
+    NeedsSequenceType,
+    NeedsView,
+    SphinxNeedsData,
+)
 from sphinx_needs.diagrams_common import (
     DiagramBase,
     add_config,
@@ -86,7 +91,7 @@ def process_needsequence(
     # Replace all needsequence nodes with a list of the collected needs.
     env = app.env
     needs_data = SphinxNeedsData(env)
-    all_needs_dict = needs_data.get_or_create_needs()
+    all_needs_dict = needs_data.get_needs_view()
 
     needs_config = NeedsSphinxConfig(env.config)
     include_needs = needs_config.include_needs
@@ -251,7 +256,7 @@ def get_message_needs(
     app: Sphinx,
     sender: NeedsInfoType,
     link_types: list[str],
-    all_needs_dict: dict[str, NeedsInfoType],
+    all_needs_dict: NeedsView,
     tracked_receivers: list[str] | None = None,
     filter: str | None = None,
 ) -> tuple[dict[str, dict[str, Any]], str, str]:
diff --git a/sphinx_needs/directives/needtable.py b/sphinx_needs/directives/needtable.py
index 7a55f5db..e153f247 100644
--- a/sphinx_needs/directives/needtable.py
+++ b/sphinx_needs/directives/needtable.py
@@ -147,7 +147,7 @@ def process_needtables(
             link_type_list["OUTGOING"] = link_type
             link_type_list["INCOMING"] = link_type
 
-    all_needs = needs_data.get_or_create_needs()
+    all_needs = needs_data.get_needs_view()
 
     # for node in doctree.findall(Needtable):
     for node in found_nodes:
diff --git a/sphinx_needs/directives/needuml.py b/sphinx_needs/directives/needuml.py
index 6f277382..d54babcf 100644
--- a/sphinx_needs/directives/needuml.py
+++ b/sphinx_needs/directives/needuml.py
@@ -271,7 +271,7 @@ class JinjaFunctions:
         parent_need_id: None | str,
         processed_need_ids: ProcessedNeedsType,
     ) -> None:
-        self.needs = SphinxNeedsData(app.env).get_or_create_needs()
+        self.needs = SphinxNeedsData(app.env).get_needs_view()
         self.app = app
         self.fromdocname = fromdocname
         self.parent_need_id = parent_need_id
diff --git a/sphinx_needs/directives/utils.py b/sphinx_needs/directives/utils.py
index 9322e987..2d1e68ce 100644
--- a/sphinx_needs/directives/utils.py
+++ b/sphinx_needs/directives/utils.py
@@ -91,7 +91,7 @@ def analyse_needs_metrics(env: BuildEnvironment) -> dict[str, Any]:
     :param env: Sphinx build environment
     :return: Dictionary consisting of needs metrics.
     """
-    needs = SphinxNeedsData(env).get_or_create_needs()
+    needs = SphinxNeedsData(env).get_needs_view()
     metric_data: dict[str, Any] = {"needs_amount": len(needs)}
     needs_types = {i["directive"]: 0 for i in NeedsSphinxConfig(env.config).types}
 
diff --git a/sphinx_needs/external_needs.py b/sphinx_needs/external_needs.py
index 93bc595c..756c565a 100644
--- a/sphinx_needs/external_needs.py
+++ b/sphinx_needs/external_needs.py
@@ -162,7 +162,7 @@ def load_external_needs(app: Sphinx, env: BuildEnvironment, docname: str) -> Non
             # check if external needs already exist
             ext_need_id = need_params["id"]
 
-            need = SphinxNeedsData(env).get_or_create_needs().get(ext_need_id)
+            need = SphinxNeedsData(env).get_needs_view().get(ext_need_id)
 
             if need is not None:
                 # check need_params for more detail
diff --git a/sphinx_needs/functions/functions.py b/sphinx_needs/functions/functions.py
index 428eb5ff..2825593c 100644
--- a/sphinx_needs/functions/functions.py
+++ b/sphinx_needs/functions/functions.py
@@ -10,7 +10,7 @@ from __future__ import annotations
 
 import ast
 import re
-from typing import Any, Callable, Dict, List, Union
+from typing import Any, Callable, List, Union
 
 from docutils import nodes
 from sphinx.application import Sphinx
@@ -19,7 +19,7 @@ from sphinx.errors import SphinxError
 from sphinx.util.tags import Tags
 
 from sphinx_needs.config import NeedsSphinxConfig
-from sphinx_needs.data import NeedsInfoType, SphinxNeedsData
+from sphinx_needs.data import NeedsInfoType, NeedsMutable, NeedsView, SphinxNeedsData
 from sphinx_needs.debug import measure_time_func
 from sphinx_needs.logging import get_logger
 from sphinx_needs.utils import NEEDS_FUNCTIONS, match_variants
@@ -30,7 +30,7 @@ ast_boolean = ast.NameConstant
 
 # TODO these functions also take optional *args and **kwargs
 DynamicFunction = Callable[
-    [Sphinx, NeedsInfoType, Dict[str, NeedsInfoType]],
+    [Sphinx, NeedsInfoType, NeedsView],
     Union[str, int, float, List[Union[str, int, float]]],
 ]
 
@@ -88,7 +88,7 @@ def execute_func(app: Sphinx, need: NeedsInfoType, func_string: str) -> Any:
     func_return = func(
         app,
         need,
-        SphinxNeedsData(app.env).get_or_create_needs(),
+        SphinxNeedsData(app.env).get_needs_view(),
         *func_args,
         **func_kwargs,
     )
@@ -175,7 +175,7 @@ def find_and_replace_node_content(
     return node
 
 
-def resolve_dynamic_values(needs: dict[str, NeedsInfoType], app: Sphinx) -> None:
+def resolve_dynamic_values(needs: NeedsMutable, app: Sphinx) -> None:
     """
     Resolve dynamic values inside need data.
 
@@ -269,7 +269,7 @@ def resolve_dynamic_values(needs: dict[str, NeedsInfoType], app: Sphinx) -> None
 
 
 def resolve_variants_options(
-    needs: dict[str, NeedsInfoType],
+    needs: NeedsMutable,
     needs_config: NeedsSphinxConfig,
     tags: Tags,
 ) -> None:
diff --git a/sphinx_needs/need_constraints.py b/sphinx_needs/need_constraints.py
index ebcd1bb2..ce3c7651 100644
--- a/sphinx_needs/need_constraints.py
+++ b/sphinx_needs/need_constraints.py
@@ -4,16 +4,14 @@ import jinja2
 
 from sphinx_needs.api.exceptions import NeedsConstraintFailed, NeedsConstraintNotAllowed
 from sphinx_needs.config import NeedsSphinxConfig
-from sphinx_needs.data import NeedsInfoType
+from sphinx_needs.data import NeedsMutable
 from sphinx_needs.filter_common import filter_single_need
 from sphinx_needs.logging import get_logger, log_warning
 
 logger = get_logger(__name__)
 
 
-def process_constraints(
-    needs: dict[str, NeedsInfoType], config: NeedsSphinxConfig
-) -> None:
+def process_constraints(needs: NeedsMutable, config: NeedsSphinxConfig) -> None:
     """Analyse constraints of all needs,
     and set corresponding fields on the need data item:
     ``constraints_passed`` and ``constraints_results``.
diff --git a/sphinx_needs/needs.py b/sphinx_needs/needs.py
index 4e4a0bda..116a5297 100644
--- a/sphinx_needs/needs.py
+++ b/sphinx_needs/needs.py
@@ -506,7 +506,7 @@ def prepare_env(app: Sphinx, env: BuildEnvironment, _docname: str) -> None:
     """
     needs_config = NeedsSphinxConfig(app.config)
     data = SphinxNeedsData(env)
-    data.get_or_create_needs()
+    data.get_needs_view()
     data.get_or_create_filters()
     data.get_or_create_docs()
     services = data.get_or_create_services()
diff --git a/sphinx_needs/roles/need_count.py b/sphinx_needs/roles/need_count.py
index 0d657718..2df0105d 100644
--- a/sphinx_needs/roles/need_count.py
+++ b/sphinx_needs/roles/need_count.py
@@ -30,7 +30,7 @@ def process_need_count(
 ) -> None:
     needs_config = NeedsSphinxConfig(app.config)
     for node_need_count in found_nodes:
-        all_needs = list(SphinxNeedsData(app.env).get_or_create_needs().values())
+        all_needs = list(SphinxNeedsData(app.env).get_needs_view().values())
         filter = node_need_count["reftarget"]
 
         if filter:
diff --git a/sphinx_needs/roles/need_incoming.py b/sphinx_needs/roles/need_incoming.py
index 66dccc75..d3e75083 100644
--- a/sphinx_needs/roles/need_incoming.py
+++ b/sphinx_needs/roles/need_incoming.py
@@ -24,7 +24,7 @@ def process_need_incoming(
     builder = app.builder
     env = app.env
     needs_config = NeedsSphinxConfig(env.config)
-    all_needs = SphinxNeedsData(env).get_or_create_needs()
+    all_needs = SphinxNeedsData(env).get_needs_view()
 
     # for node_need_backref in doctree.findall(NeedIncoming):
     for node_need_backref in found_nodes:
diff --git a/sphinx_needs/roles/need_outgoing.py b/sphinx_needs/roles/need_outgoing.py
index b777c932..0266947a 100644
--- a/sphinx_needs/roles/need_outgoing.py
+++ b/sphinx_needs/roles/need_outgoing.py
@@ -31,7 +31,7 @@ def process_need_outgoing(
     # for node_need_ref in doctree.findall(NeedOutgoing):
     for node_need_ref in found_nodes:
         node_link_container = nodes.inline()
-        needs_all_needs = SphinxNeedsData(env).get_or_create_needs()
+        needs_all_needs = SphinxNeedsData(env).get_needs_view()
         ref_need = needs_all_needs[node_need_ref["reftarget"]]
 
         # Let's check if NeedIncoming shall follow a specific link type
diff --git a/sphinx_needs/roles/need_ref.py b/sphinx_needs/roles/need_ref.py
index e2f7e4b8..00286793 100644
--- a/sphinx_needs/roles/need_ref.py
+++ b/sphinx_needs/roles/need_ref.py
@@ -60,7 +60,7 @@ def process_need_ref(
     builder = app.builder
     env = app.env
     needs_config = NeedsSphinxConfig(env.config)
-    all_needs = SphinxNeedsData(env).get_or_create_needs()
+    all_needs = SphinxNeedsData(env).get_needs_view()
     # for node_need_ref in doctree.findall(NeedRef):
     for node_need_ref in found_nodes:
         # Let's create a dummy node, for the case we will not be able to create a real reference
diff --git a/sphinx_needs/utils.py b/sphinx_needs/utils.py
index b065f802..8f09db23 100644
--- a/sphinx_needs/utils.py
+++ b/sphinx_needs/utils.py
@@ -16,7 +16,7 @@ from sphinx.application import BuildEnvironment, Sphinx
 
 from sphinx_needs.api.exceptions import NeedsInvalidFilter
 from sphinx_needs.config import LinkOptionsType, NeedsSphinxConfig
-from sphinx_needs.data import NeedsInfoType, SphinxNeedsData
+from sphinx_needs.data import NeedsInfoType, NeedsView, SphinxNeedsData
 from sphinx_needs.defaults import NEEDS_PROFILING
 from sphinx_needs.logging import get_logger, log_warning
 
@@ -77,7 +77,7 @@ def split_need_id(need_id_full: str) -> tuple[str, str | None]:
 def row_col_maker(
     app: Sphinx,
     fromdocname: str,
-    all_needs: dict[str, NeedsInfoType],
+    all_needs: NeedsView,
     need_info: NeedsInfoType,
     need_key: str,
     make_ref: bool = False,
diff --git a/sphinx_needs/warnings.py b/sphinx_needs/warnings.py
index 7d010d07..2077a603 100644
--- a/sphinx_needs/warnings.py
+++ b/sphinx_needs/warnings.py
@@ -33,7 +33,7 @@ def process_warnings(app: Sphinx, exception: Exception | None) -> None:
         return
 
     env = app.env
-    needs = SphinxNeedsData(env).get_or_create_needs()
+    needs = SphinxNeedsData(env).get_needs_view()
     # If no needs were defined, we do not need to do anything
     if not needs:
         return


Test patch:
diff --git a/tests/test_import.py b/tests/test_import.py
index 90be273c..b3fad731 100644
--- a/tests/test_import.py
+++ b/tests/test_import.py
@@ -6,6 +6,8 @@ import pytest
 import responses
 from syrupy.filters import props
 
+from sphinx_needs.data import SphinxNeedsData
+
 
 @pytest.mark.parametrize(
     "test_app",
@@ -198,7 +200,7 @@ def test_needimport_needs_json_download(test_app, snapshot):
         m.get("http://my_company.com/docs/v1/remote-needs.json", json=remote_json)
         app.build()
 
-    needs_all_needs = app.env.needs_all_needs
+    needs_all_needs = SphinxNeedsData(app.env).get_needs_view()
     assert needs_all_needs == snapshot()
 
 
diff --git a/tests/test_needuml.py b/tests/test_needuml.py
index c04a4fe3..591433d0 100644
--- a/tests/test_needuml.py
+++ b/tests/test_needuml.py
@@ -3,6 +3,8 @@ from pathlib import Path
 
 import pytest
 
+from sphinx_needs.data import SphinxNeedsData
+
 
 @pytest.mark.parametrize(
     "test_app",
@@ -15,7 +17,7 @@ def test_doc_build_html(test_app, snapshot):
 
     assert Path(app.outdir, "index.html").read_text(encoding="utf8")
 
-    all_needs = app.env.needs_all_needs
+    all_needs = SphinxNeedsData(app.env).get_needs_view()
     assert all_needs == snapshot()
 
     all_needumls = app.env.needs_all_needumls