Repository: aiidateam/aiida-core
Pull request #6866:  Issue #6687: Fix QueryBuilder filtering for AbstractCode
Author: login: danielhollas
Created: 2025-05-07T14:28:33Z | Merged: 2025-05-09T13:11:45Z

Continuing from #6857 

> This PR fixes an issue in QueryBuilder where filtering for AbstractCode was not working correctly. The update ensures that queries involving AbstractCode return expected results.
>Changes Made:
1.Fixed filtering logic in querybuilder.py.
2.Updated pyproject.toml accordingly.
3.Added pytest tests for QueryBuilder functionality, as previously requested.


Closes #6687

Linked issue #6687: `QueryBuilder` can't find codes when looking for `AbstractCode`
### Describe the bug

`orm.QueryBuilder().append(orm.AbstractCode)` will not find any codes, as the resultant query has `{'node_type': {'like': 'data.code.abstract.%'}`, whereas codes carry an entry point of `data.core.code.%`.

### Steps to reproduce

Try finding any code with `orm.QueryBuilder().append(orm.AbstractCode)`. Count should always come up null.

### Expected behavior

Successfully find codes.

### Your environment

- Linux (aiidalab/full-stack image)
- Python 3.9.13
- aiida-core v2.6.3

### Additional context

`orm.QueryBuilder().append(orm.Code)` works, yielding `{'node_type': {'like': 'data.core.code.%'}`. But then you're using the deprecated `orm.Code`.
Linked issue comments: user: login: adityagh006
created_at: 2025-02-15T18:14:49Z
body: Hey, I’m interested in contributing to this issue. I see that orm.QueryBuilder().append(orm.AbstractCode) isn’t returning results due to the node_type mismatch. Would updating the node_type filter in QueryBuilder to match data.core.code.% be a suitable fix, or is there a reason it was originally set to data.code.abstract.%?
author_association: CONTRIBUTOR
user: login: danielhollas
created_at: 2025-02-24T18:58:11Z
body: > Would updating the node_type filter in QueryBuilder to match data.core.code.% be a suitable fix, or is there a reason it was originally set to data.code.abstract.%?

CC @sphuber in case he remembers the design decisions behind AbstractCode. 
FWIW I am not sure if this is a "good first issue" kind of thing.
author_association: COLLABORATOR
user: login: sphuber
created_at: 2025-02-24T21:02:59Z
body: The problem here is that the hierarchy of ORM classes are defined differently in Python compared to how it is represented in the database (which is what the `QueryBuilder` ultimately uses). The ORM classes in Python obviously use Python's class hierarchy system. So since `InstalledCode` and `PortableCode` inherit from `AbstractCode`, instances of the former will also be instances of the latter. However, the class system does not exist in the database. Instead, the querybuilder uses a proxy based on the node type strings. For example, the `Data` class has `data` as type string, and its subclass `StructureData` has `data.structure`. To find "subclasses" of `Data`, the query then includes a filter on the `node_type` that matches `data.%`. 

So for all of this to work, the `node_type` strings need to be defined such that they correctly reflect the Python class hierarchy. But the type strings are not determined based on the class hierarchy but rather on the entry point strings. This is mostly for historical reasons. Originally (and I mean a long time ago when there was just one AiiDA repository without any plugin system and all code plugins were living in the same repo), the node type string was based on the _module_ hierarchy. When we introduced the plugin system allowing external packages to define ORM classes (`Data` plugins mostly) this no longer worked and we based it off the entry point string instead. I have to admit that I am not a 100% sure anymore if we tried to use the actual class hierarchy to derive the node string and that didn't work, or we never considered it. One thing that comes to mind is that the `node_type` is not only used for capturing class hierarchy in querying, but mostly it is necessary to be able to load the correct Python class when retrieving a node from the database. Imagine you get a row from the `DbNode` table, how do you know which Python class to return an instance of? That is done using the `node_type`. It is converted to an entry point string, which is then used to load the corresponding class that is registered at that entry point.

With that all out of the way, we now come to the specific problem of the OP. The `AbstractCode` is a later addition (I think somewhere around AiiDA v2.2 if memory serves me right). Originally there was the `Code` class and the various types (installed and portable, called `remote` and `local` at the time) where represented by this one class. It turned out that this design led to lots of complications and the `remote/local` naming was confusing (because a `remote`, i.e. `installed` code, could also be installed on the localhost, i.e. "locally") so we decided to refactor. But since we needed to keep backwards compatibility, I could not get rid of the `Code` class and had to come up with a new base class, which is `AbstractCode`. But on top of that, I could not reappropriate the `code` entry point string of `Code` either. I just checked and noticed that the `AbstractCode` doesn't actually even seem to be registered as an entry point: 
https://github.com/aiidateam/aiida-core/blob/f4c55f5f78cd7fde9a5b4a4e48cad2159fd666b2/pyproject.toml#L104
This has presumably not yet given rise to any problems because it is never actually instantiated, only its subclasses. But I am a bit suprised as to how the query string is generated then when calling `QueryBuilder().append(AbstractCode)`.

Anyway, I think the solution should be the following:
1. Register `AbstractCode` in `pyproject.toml` as `'core.code.abstract' = 'aiida.orm.nodes.data.code.abstractAbstractCode'`
2. Add a special case/clause in the `QueryBuilder` code that when `AbstractCode` is appended, the `node_type` query string is not `data.core.code.abstract%` but `data.core.code%` instead.
 
I think querying for `AbstractCode` with subclassing enabled should then match subclasses as expected.
author_association: CONTRIBUTOR
user: login: OPMTerra
created_at: 2025-03-01T19:55:35Z
body: Hi @sphuber  @danielhollas  @edan-bainglass , I'm interested in working on #6687. 

Before proceeding:
1. Is this issue still available for contribution?
2. Are there any backward compatibility concerns I should consider?
3. Could you point me to existing code handling similar entry point registration?

Current Understanding:
- Need to register AbstractCode in pyproject.toml
- Modify QueryBuilder logic for node type matching

I'll study entry point documentation while awaiting your response.
author_association: NONE
user: login: danielhollas
created_at: 2025-03-01T21:40:58Z
body: @OPMTerra if you don't mind, let's wait a bit first to see if @adityagh006 is still interested in contributing this issue. 

@sphuber thank you for a very detailed writeup!

author_association: COLLABORATOR
user: login: OPMTerra
created_at: 2025-03-01T21:54:33Z
body: @danielhollas 

No worries at all! 

Just putting this here so you know I'm ready to jump in if needed:
- Can handle the `pyproject.toml` entry point bit
- Tweak the QueryBuilder filters (looks straightforward from @sphuber's notes!)
author_association: NONE
user: login: adityagh006
created_at: 2025-03-02T13:06:39Z
body: Yes, I'm still interested in contributing to this issue. Based on @sphuber's explanation, my plan is:

1. Register AbstractCode in pyproject.toml as 'core.code.abstract' = 'aiida.orm.nodes.data.code.abstract:AbstractCode'.
2.Modify QueryBuilder to ensure that appending AbstractCode results in data.core.code% instead of data.core.code.abstract%.

I'll start looking into the relevant files for these changes. Let me know if there's anything else I should consider before proceeding!
author_association: CONTRIBUTOR
user: login: danielhollas
created_at: 2025-03-02T16:54:38Z
body: Sounds good, thanks! I am not really familiar with query builder part of the code so we'll see once you put up a PR.

CC @unkcpz @agoscinski @GeigerJ2 it might be a good  idea to go through the existing issues and label some of them as "good first issue" so that first contributors have more choice.
author_association: COLLABORATOR
user: login: OPMTerra
created_at: 2025-03-02T17:30:20Z
body: Hey @danielhollas I've been trying to find a good beginner level issues to solve. 

Is there any open issues that I could try contributing to? It's my first time and every good first issues seems to be already assigned to someone.
author_association: NONE
user: login: adityagh006
created_at: 2025-03-10T18:29:32Z
body: I've submitted this PR to fix an issue in QueryBuilder where filtering for AbstractCode was not working correctly. The update ensures that queries involving AbstractCode return expected results.

Changes made:
1. Updated pyproject.toml to register AbstractCode under 'core.code.abstract'.
2. Fixed the filtering logic in querybuilder.py to ensure AbstractCode correctly matches data.core.code% instead of data.core.code.abstract%.
Let me know if any additional changes are needed. Looking forward to your feedback!
author_association: CONTRIBUTOR

Diff:
diff --git a/pyproject.toml b/pyproject.toml
index 4abb8bd387..1bc0949071 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -103,6 +103,7 @@ requires-python = '>=3.9'
 'core.bool' = 'aiida.orm.nodes.data.bool:Bool'
 'core.cif' = 'aiida.orm.nodes.data.cif:CifData'
 'core.code' = 'aiida.orm.nodes.data.code.legacy:Code'
+'core.code.abstract' = 'aiida.orm.nodes.data.code.abstract:AbstractCode'
 'core.code.containerized' = 'aiida.orm.nodes.data.code.containerized:ContainerizedCode'
 'core.code.installed' = 'aiida.orm.nodes.data.code.installed:InstalledCode'
 'core.code.portable' = 'aiida.orm.nodes.data.code.portable:PortableCode'
diff --git a/src/aiida/orm/querybuilder.py b/src/aiida/orm/querybuilder.py
index 1769714494..5ce822690e 100644
--- a/src/aiida/orm/querybuilder.py
+++ b/src/aiida/orm/querybuilder.py
@@ -1338,6 +1338,14 @@ def _get_node_type_filter(classifiers: Classifier, subclassing: bool) -> dict:
 
     value = classifiers.ormclass_type_string
 
+    # Users searching for `AbstractCode` (sub)classes want to get all the codes (Portable, Installed etc.)
+    # Unfortunately, because AbstractCode was introduced later, its entry point is 'data.core.code.abstract',
+    # while the 'core.code' entry point is claimed by the Legacy Code class.
+    # So to get all the code types, including the Legacy Code, we adjust the filter to 'data.core.code'.
+    # Note, this only make sense if `subclassing` parameter is True!
+    if value == 'data.core.code.abstract.AbstractCode.':
+        value = 'data.core.code.'
+
     if not subclassing:
         filters = {'==': value}
     else:
diff --git a/tests/orm/nodes/data/test_data.py b/tests/orm/nodes/data/test_data.py
index 1dc5ed48ca..248832f459 100644
--- a/tests/orm/nodes/data/test_data.py
+++ b/tests/orm/nodes/data/test_data.py
@@ -154,7 +154,11 @@ def _generate_class_instance(data_class):
 
 @pytest.fixture(
     scope='function',
-    params=[entry_point for entry_point in plugins.get_entry_points('aiida.data') if entry_point.name != 'core.code'],
+    params=[
+        entry_point
+        for entry_point in plugins.get_entry_points('aiida.data')
+        if entry_point.name not in ('core.code', 'core.code.abstract')
+    ],
 )
 def data_plugin(request):
     """Fixture that parametrizes over all the registered entry points of the ``aiida.data`` entry point group."""
diff --git a/tests/orm/test_fields/fields_aiida.data.core.code.abstract.AbstractCode.yml b/tests/orm/test_fields/fields_aiida.data.core.code.abstract.AbstractCode.yml
new file mode 100644
index 0000000000..4dc178d9fc
--- /dev/null
+++ b/tests/orm/test_fields/fields_aiida.data.core.code.abstract.AbstractCode.yml
@@ -0,0 +1,25 @@
+append_text: QbStrField('append_text', dtype=<class 'str'>, is_attribute=True)
+attributes: QbDictField('attributes', dtype=typing.Optional[typing.Dict[str, typing.Any]],
+  is_attribute=False, is_subscriptable=True)
+computer: QbNumericField('computer', dtype=typing.Optional[int], is_attribute=False)
+ctime: QbNumericField('ctime', dtype=typing.Optional[datetime.datetime], is_attribute=False)
+default_calc_job_plugin: QbStrField('default_calc_job_plugin', dtype=typing.Optional[str],
+  is_attribute=True)
+description: QbStrField('description', dtype=<class 'str'>, is_attribute=True)
+extras: QbDictField('extras', dtype=typing.Optional[typing.Dict[str, typing.Any]],
+  is_attribute=False, is_subscriptable=True)
+label: QbStrField('label', dtype=<class 'str'>, is_attribute=True)
+mtime: QbNumericField('mtime', dtype=typing.Optional[datetime.datetime], is_attribute=False)
+node_type: QbStrField('node_type', dtype=typing.Optional[str], is_attribute=False)
+pk: QbNumericField('pk', dtype=typing.Optional[int], is_attribute=False)
+prepend_text: QbStrField('prepend_text', dtype=<class 'str'>, is_attribute=True)
+process_type: QbStrField('process_type', dtype=typing.Optional[str], is_attribute=False)
+repository_content: QbDictField('repository_content', dtype=typing.Optional[dict[str,
+  bytes]], is_attribute=False)
+repository_metadata: QbDictField('repository_metadata', dtype=typing.Optional[typing.Dict[str,
+  typing.Any]], is_attribute=False)
+source: QbDictField('source', dtype=typing.Optional[dict], is_attribute=True, is_subscriptable=True)
+use_double_quotes: QbField('use_double_quotes', dtype=<class 'bool'>, is_attribute=True)
+user: QbNumericField('user', dtype=typing.Optional[int], is_attribute=False)
+uuid: QbStrField('uuid', dtype=typing.Optional[str], is_attribute=False)
+with_mpi: QbField('with_mpi', dtype=typing.Optional[bool], is_attribute=True)
diff --git a/tests/orm/test_querybuilder.py b/tests/orm/test_querybuilder.py
index 83f156c168..8754d13d30 100644
--- a/tests/orm/test_querybuilder.py
+++ b/tests/orm/test_querybuilder.py
@@ -852,6 +852,76 @@ def test_empty_filters(self):
         qb = orm.QueryBuilder().append(orm.Data, filters={'or': [{}, {}]})
         assert qb.count() == count
 
+    @pytest.mark.usefixtures('suppress_internal_deprecations')
+    @pytest.mark.usefixtures('aiida_profile_clean')
+    def test_abstract_code_filtering(self, aiida_localhost, aiida_code, tmp_path):
+        """Test that querying for AbstractCode correctly returns all code instances.
+
+        This tests the fix for issue #6687, where QueryBuilder couldn't find codes
+        when looking for AbstractCode due to a node_type mismatch.
+        """
+        installed_code = aiida_code(
+            'core.code.installed',
+            label='installed-code',
+            computer=aiida_localhost,
+            filepath_executable='/bin/bash',
+        )
+        (tmp_path / 'fake_exec').touch()
+        portable_code = aiida_code(
+            'core.code.portable',
+            label='portable-code',
+            filepath_executable='fake_exec',
+            filepath_files=tmp_path,
+        )
+        legacy_code = aiida_code(
+            'core.code',
+            label='legacy-code',
+            remote_computer_exec=(aiida_localhost, '/bin/bash'),
+        )
+
+        qb = orm.QueryBuilder
+
+        # Verify specific code type queries work as expected
+        installed_results = qb().append(orm.InstalledCode).all(flat=True)
+        assert installed_code in installed_results
+        assert len(installed_results) == 1
+
+        portable_results = qb().append(orm.PortableCode).all(flat=True)
+        assert portable_code in portable_results
+        assert len(portable_results) == 1
+
+        # Using orm.Code actually matches all codes.
+        # for backwards compatibility reasons we will not fix this.
+        legacy_results = qb().append(orm.Code).all(flat=True)
+        assert legacy_code in legacy_results
+        assert len(legacy_results) == 3
+
+        # Turning off subclassing should however only match the one legacy Code
+        legacy_results = qb().append(orm.Code, subclassing=False).all(flat=True)
+        assert legacy_code in legacy_results
+        assert len(legacy_results) == 1
+
+        # AbstractCode query should find all code types
+        abstract_results = qb().append(orm.AbstractCode).all(flat=True)
+        assert (
+            installed_code in abstract_results
+        ), f'InstalledCode not found with AbstractCode query. Result: {abstract_results}'
+        assert (
+            portable_code in abstract_results
+        ), f'PortableCode not found with AbstractCode query. Result: {abstract_results}'
+        assert legacy_code in abstract_results, f'Code not found with AbstractCode query. Result: {abstract_results}'
+        assert len(abstract_results) == 3
+
+        # AbstractCode with basic filtering
+        qb_filtered = qb().append(orm.AbstractCode, filters={'label': 'installed-code'})
+        filtered_results = qb_filtered.all(flat=True)
+        assert installed_code in filtered_results
+        assert len(filtered_results) == 1
+
+        # QB should find no codes if subclassing is False
+        subclassing_off_results = qb().append(orm.AbstractCode, subclassing=False).all(flat=True)
+        assert len(subclassing_off_results) == 0
+
 
 class TestAttributes:
     @pytest.mark.requires_psql


Reviews:
user: login: danielhollas
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-07T17:03:55Z
user: login: agoscinski
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-09T11:52:17Z
user: login: edan-bainglass
body: LGTM! @danielhollas thanks for handling this one 🙏
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-09T11:56:30Z
user: login: danielhollas
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T12:57:44Z

Review comments:
diff_hunk: @@ -154,7 +154,11 @@ def _generate_class_instance(data_class):
 
 @pytest.fixture(
     scope='function',
-    params=[entry_point for entry_point in plugins.get_entry_points('aiida.data') if entry_point.name != 'core.code'],
+    params=[
+        entry_point
+        for entry_point in plugins.get_entry_points('aiida.data')
+        if entry_point.name not in ('core.code', 'core.code.abstract')
path: tests/orm/nodes/data/test_data.py
user: login: danielhollas
body: It's somewhat unfortunate that we're adding an entry point for an abstract class that cannot be instantiated. :-( 
created_at: 2025-05-07T17:03:55Z
author_association: COLLABORATOR
diff_hunk: @@ -154,7 +154,11 @@ def _generate_class_instance(data_class):
 
 @pytest.fixture(
     scope='function',
-    params=[entry_point for entry_point in plugins.get_entry_points('aiida.data') if entry_point.name != 'core.code'],
+    params=[
+        entry_point
+        for entry_point in plugins.get_entry_points('aiida.data')
+        if entry_point.name not in ('core.code', 'core.code.abstract')
path: tests/orm/nodes/data/test_data.py
user: login: agoscinski
body: I think it as okay. We also mention it in the RTD so it is an implementation detail we tried to hide from plugin developers so far.
created_at: 2025-05-09T11:44:38Z
author_association: COLLABORATOR
diff_hunk: @@ -852,6 +852,76 @@ def test_empty_filters(self):
         qb = orm.QueryBuilder().append(orm.Data, filters={'or': [{}, {}]})
         assert qb.count() == count
 
+    @pytest.mark.usefixtures('suppress_internal_deprecations')
+    @pytest.mark.usefixtures('aiida_profile_clean')
+    def test_abstract_code_filtering(self, aiida_localhost, aiida_code, tmp_path):
+        """Test that querying for AbstractCode correctly returns all code instances.
+
+        This tests the fix for issue #6687, where QueryBuilder couldn't find codes
+        when looking for AbstractCode due to a node_type mismatch.
+        """
+        installed_code = aiida_code(
+            'core.code.installed',
+            label='installed-code',
+            computer=aiida_localhost,
+            filepath_executable='/bin/bash',
+        )
path: tests/orm/test_querybuilder.py
user: login: agoscinski
body: I like how using the fixture ends up with the same amount of code. I think the check the fixture adds to the creation is actually performed when one invokes `store`. Anyway cleaning up the fixtures is another topic for another PR.
created_at: 2025-05-09T11:49:21Z
author_association: COLLABORATOR
diff_hunk: @@ -852,6 +852,76 @@ def test_empty_filters(self):
         qb = orm.QueryBuilder().append(orm.Data, filters={'or': [{}, {}]})
         assert qb.count() == count
 
+    @pytest.mark.usefixtures('suppress_internal_deprecations')
+    @pytest.mark.usefixtures('aiida_profile_clean')
+    def test_abstract_code_filtering(self, aiida_localhost, aiida_code, tmp_path):
+        """Test that querying for AbstractCode correctly returns all code instances.
+
+        This tests the fix for issue #6687, where QueryBuilder couldn't find codes
+        when looking for AbstractCode due to a node_type mismatch.
+        """
+        installed_code = aiida_code(
+            'core.code.installed',
+            label='installed-code',
+            computer=aiida_localhost,
+            filepath_executable='/bin/bash',
+        )
path: tests/orm/test_querybuilder.py
user: login: danielhollas
body: for the `installed_code` we could you the `aiida_code_installed` fixture, but I thought I'd keep things symmetric for all code types is better. 
created_at: 2025-05-09T12:57:44Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-05-07T16:54:12Z
body: ## [Codecov](https://app.codecov.io/gh/aiidateam/aiida-core/pull/6866?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiidateam) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 78.59%. Comparing base ([`2fd4b89`](https://app.codecov.io/gh/aiidateam/aiida-core/commit/2fd4b8931d4abd7799f6c17aec21612a7f85d1b3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiidateam)) to head ([`bc46b3c`](https://app.codecov.io/gh/aiidateam/aiida-core/commit/bc46b3c363953d407209bac02dd0cbd1fc01c7ec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiidateam)).
:warning: Report is 49 commits behind head on main.

<details><summary>Additional details and impacted files</summary>



```diff
@@            Coverage Diff             @@
##             main    #6866      +/-   ##
==========================================
+ Coverage   78.59%   78.59%   +0.01%     
==========================================
  Files         567      567              
  Lines       43090    43092       +2     
==========================================
+ Hits        33864    33866       +2     
  Misses       9226     9226              
```
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aiidateam/aiida-core/pull/6866?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiidateam).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aiidateam).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: danielhollas
created_at: 2025-05-07T17:05:06Z
body: Ooof, this was more work than I expected but is now ready I think! 
I needed to fix two tests that started failing since we added an entry point for AbstractCode.

@edan-bainglass it would be great if you could have a  look as well as the reported of the original issue. Specifically look whether the behaviour in test cases makes sense now.
author_association: COLLABORATOR
user: login: danielhollas
created_at: 2025-05-09T12:55:38Z
body: @agoscinski leaving the merge up to you, in case you want to include this in the next rc release.

Thanks for the reviews!
author_association: COLLABORATOR