Repository: stfc/PSyclone
Pull request #2994: (Closes #2992) fix bug in LFRicKern.all_updates_are_writes
Author: login: arporter
Labels: name: bug
name: ready to be merged
name: LFRic
description: Issue relates to the LFRic domain
Created: 2025-05-13T10:55:09Z | Merged: 2025-05-23T10:38:02Z


Linked issue #2992: Turning `COMPUTE_ANNEXED_DOFS=false` results in dirty annexed dofs being used in calculations
## Description

Found during the adjoint development - setting `COMPUTE_ANNEXED_DOFS=false` results in annexed dofs becoming dirty. The problem here is that kernels unknowingly use the dirty annexed dofs and this can cause error in calculations (shown through an adjoint test).

## Further information

In particular, it was noticed in a matrix operation between W2/W3 functionspaces. The psy-layer does not check the "dirtiness" of annexed dofs if it does not access the halo (there is no distinction).

```fortran
call invoke( X_plus_Y(w2, some_other_w2_a, some_other_w2_b), &
             ! W2 annexed dofs out-of-date/dirty at this point
             X_plus_Y(w3, some_other_w3_a, some_other_w3_b), &
             ! Dirty annexed dofs used by this kernel. Psy-layer does not check `dirty(depth=1)`
             dg_matrix_vector_kernel(w2, w3, matrix) )
```
Linked issue comments: user: login: arporter
created_at: 2025-05-13T08:33:28Z
body: ```fortran
  type, public, extends(kernel_type) :: dg_matrix_vector_kernel_type
    private
    type(arg_type) :: meta_args(3) = (/                                       &
         arg_type(GH_FIELD,    GH_REAL, GH_WRITE, ANY_DISCONTINUOUS_SPACE_1), &
         arg_type(GH_FIELD,    GH_REAL, GH_READ,  ANY_SPACE_1),               &
         arg_type(GH_OPERATOR, GH_REAL, GH_READ,  ANY_DISCONTINUOUS_SPACE_1,  ANY_SPACE_1)                &
         /)
    integer :: operates_on = CELL_COLUMN
  end type
```
so this kernel writes to a field on a discontinuous space but reads from a field on a continuous space (which therefore needs its annexed dofs to be clean). If I do a `setval_c` (for the field on ANY_SPACE_1) and then a user-defined kernel *without* an operator argument then we get a halo exchange as expected. However, if I then switch to using dg_matrix_vector_kernel_type then we set the halo dirty but do not add a halo exchange:
```fortran
    ! Set halos dirty/clean for fields modified in the above loop(s)
    call f5_proxy%set_dirty()
    do df = loop1_start, loop1_stop, 1
      ! Built-in: setval_c (set a real-valued field to a real scalar value)
      f2_data(df) = 0.0
    enddo

    ! Set halos dirty/clean for fields modified in the above loop(s)
    call f2_proxy%set_dirty()
    do cell = loop2_start, loop2_stop, 1
      call dg_matrix_vector_code(cell, nlayers_f4, f4_data, f2_data, mass_matrix_proxy%ncell_3d, mass_matrix_local_stencil, ndf_adspc1_f4, undf_adspc1_f4, map_adspc1_f4(:,cell), ndf_aspc1_f2, undf_aspc1_f2, map_aspc1_f2(:,cell))
    enddo
```
where
```fortran
    loop1_stop = f2_proxy%vspace%get_last_dof_owned()
    loop2_stop = mesh%get_last_edge_cell()
```
I agree this looks to be a bug but I need to check the subtleties around operators.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-13T08:40:32Z
body: The Developer Guide says:
![Image](https://github.com/user-attachments/assets/d578cf4c-15f8-4063-9df6-10149d6fb769)
so I think this is a bug.
author_association: MEMBER
user: login: mo-joshuacolclough
created_at: 2025-05-13T09:24:52Z
body: Thanks for taking a look, just confirming the test added 106d87e matches what we have 👍
author_association: NONE

Diff:
diff --git a/.github/workflows/lfric_test.yml b/.github/workflows/lfric_test.yml
index a080cf066d..9b9eec324f 100644
--- a/.github/workflows/lfric_test.yml
+++ b/.github/workflows/lfric_test.yml
@@ -75,7 +75,9 @@ jobs:
         # pip install external/fparser
         pip install .[test]
 
-    # PSyclone, compile and run MetOffice LFRic with 6 MPI ranks
+    # PSyclone, compile and run MetOffice LFRic with 6 MPI ranks.
+    # COMPUTE_ANNEXED_DOFS is set to false for this test by using the
+    # appropriate cfg file.
     - name: LFRic passthrough (with DistributedMemory)
       run: |
         # Set up environment
@@ -83,9 +85,14 @@ jobs:
         spack load lfric-build-environment%gcc@14
         source .runner_venv/bin/activate
         export PSYCLONE_LFRIC_DIR=${GITHUB_WORKSPACE}/examples/lfric/scripts
-        export PSYCLONE_CONFIG_FILE=${PSYCLONE_LFRIC_DIR}/KGOs/lfric_psyclone.cfg
+        export PSYCLONE_CONFIG_FILE=${PSYCLONE_LFRIC_DIR}/KGOs/lfric_psyclone_no_annexed.cfg
         export LFRIC_DIR=${HOME}/LFRic/lfric_apps_${LFRIC_APPS_REV}
         cd ${LFRIC_DIR}
+        # Ensure no transformations are applied by supplying an empty
+        # transformation script.
+        export OPT_DIR=${LFRIC_DIR}/applications/gungho_model/optimisation/psyclone-test
+        mkdir -p ${OPT_DIR}
+        cp ${PSYCLONE_LFRIC_DIR}/does_nothing.py ${OPT_DIR}/global.py
         # Clean previous version and compile again
         rm -rf applications/gungho_model/working
         ./build/local_build.py -a gungho_model -v
diff --git a/changelog b/changelog
index 8774accc01..dfc18c33ef 100644
--- a/changelog
+++ b/changelog
@@ -1,3 +1,5 @@
+   28) PR #2994 for #2992. Fix bug with COMPUTE_ANNEXED_DOFS set to False.
+
    27) PR #2979 for #2942. Change 'dynamo' references to 'LFRic' throughout.
 
    26) PR #3002 for #2838. Upgrade CI dependencies.
diff --git a/examples/lfric/scripts/KGOs/lfric_psyclone_no_annexed.cfg b/examples/lfric/scripts/KGOs/lfric_psyclone_no_annexed.cfg
new file mode 100644
index 0000000000..df0dfc7dc9
--- /dev/null
+++ b/examples/lfric/scripts/KGOs/lfric_psyclone_no_annexed.cfg
@@ -0,0 +1,101 @@
+# -----------------------------------------------------------------------------
+# BSD 3-Clause License
+#
+# Copyright (c) 2017-2025, Science and Technology Facilities Council
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+#
+# * Neither the name of the copyright holder nor the names of its
+#   contributors may be used to endorse or promote products derived from
+#   this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+# COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+# Author: A. R. Porter, STFC Daresbury Laboratory
+# Modified by: R. W. Ford, STFC Daresbury Laboratory
+#              S. Siso, STFC Daresbury Laboratory
+#              J. Henrichs, Bureau of Meteorology,
+#              I. Kavcic and O. Brunt, Met Office
+
+# This is the PSyclone configuration file.
+
+# Settings common to all APIs
+[DEFAULT]
+DISTRIBUTED_MEMORY = true
+REPRODUCIBLE_REDUCTIONS = false
+# Amount to pad the local summation array when REPRODUCIBLE_REDUCTIONS is true
+REPROD_PAD_SIZE = 8
+PSYIR_ROOT_NAME = psyir_tmp
+VALID_PSY_DATA_PREFIXES = profile, extract, read_only_verify, nan_test
+
+# Specify number of OpenCL devices per node. When combining OpenCL with MPI,
+# the mpirun/mpiexec ranks_per_node parameter must match this number.
+OCL_DEVICES_PER_NODE = 1
+
+# Symbols imported from the following modules will be ignored when parsing
+# and will not produce a warning message if they cannot be found.
+IGNORE_MODULES = netcdf, mpi
+
+# Settings specific to the LFRic (Dynamo 0.3) API
+# ===============================================
+[lfric]
+access_mapping = gh_read: read, gh_write: write, gh_readwrite: readwrite,
+                 gh_inc: inc, gh_readinc: readinc, gh_sum: sum
+
+# Specify whether we compute annexed dofs when a kernel is written so
+# that it iterates over dofs. This is currently only the case for
+# builtins. If annexed dofs are computed then in certain cases we
+# remove the need for a halo exchange call.
+COMPUTE_ANNEXED_DOFS = false
+
+# Specify supported Fortran datatypes
+supported_fortran_datatypes = real, integer, logical
+
+# Specify default kind (precision) for the supported LFRic data types
+default_kind = real: r_def, integer: i_def, logical: l_def
+
+# Specify number of bytes for the supported LFRic data types.
+# The values for 'r_tran', 'r_solver', 'r_def', 'r_bl' and 'r_phys' are
+# set according to CPP ifdefs. The values given below are the defaults.
+# 'l_def' is included in this dict so that it contains a complete
+# record of the various precision symbols used in LFRic.
+precision_map = i_def: 4,
+                l_def: 1,
+                r_def: 8,
+                r_double: 8,
+                r_ncdf: 8,
+                r_quad: 16,
+                r_second: 8,
+                r_single: 4,
+                r_solver: 4,
+                r_tran: 8,
+                r_bl: 8,
+                r_phys: 8,
+                r_um: 8
+
+# Specify whether we generate code to perform runtime correctness checks
+RUN_TIME_CHECKS = false
+
+# Number of ANY_SPACE and ANY_DISCONTINUOUS_SPACE function spaces
+NUM_ANY_SPACE = 10
+NUM_ANY_DISCONTINUOUS_SPACE = 20
diff --git a/examples/lfric/scripts/does_nothing.py b/examples/lfric/scripts/does_nothing.py
new file mode 100644
index 0000000000..823aca9ed9
--- /dev/null
+++ b/examples/lfric/scripts/does_nothing.py
@@ -0,0 +1,49 @@
+# -----------------------------------------------------------------------------
+# BSD 3-Clause License
+#
+# Copyright (c) 2025, Science and Technology Facilities Council
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+#
+# * Neither the name of the copyright holder nor the names of its
+#   contributors may be used to endorse or promote products derived from
+#   this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+# COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+# Author: A. R. Porter, STFC Daresbury Laboratory
+
+'''PSyclone transformation script that does nothing. This is required
+to force the LFRic build system not to apply any transformations.
+
+'''
+
+
+def trans(psyir):
+    '''
+    Does nothing.
+
+    :param psyir: the PSyIR of the PSy-layer.
+    :type psyir: :py:class:`psyclone.psyir.nodes.FileContainer`
+
+    '''
diff --git a/src/psyclone/domain/lfric/lfric_kern.py b/src/psyclone/domain/lfric/lfric_kern.py
index 115f308e3a..10db6f3a33 100644
--- a/src/psyclone/domain/lfric/lfric_kern.py
+++ b/src/psyclone/domain/lfric/lfric_kern.py
@@ -663,12 +663,10 @@ def mesh(self):
         return self._mesh_properties
 
     @property
-    def all_updates_are_writes(self):
+    def all_updates_are_writes(self) -> bool:
         '''
-        :returns: True if all of the arguments updated by this kernel have \
+        :returns: True if all arguments updated by this kernel have
                   'GH_WRITE' access, False otherwise.
-        :rtype: bool
-
         '''
         accesses = set(arg.access for arg in self.args)
         all_writes = AccessType.all_write_accesses()
diff --git a/src/psyclone/domain/lfric/lfric_loop.py b/src/psyclone/domain/lfric/lfric_loop.py
index 646a71943a..a2fb0d1c63 100644
--- a/src/psyclone/domain/lfric/lfric_loop.py
+++ b/src/psyclone/domain/lfric/lfric_loop.py
@@ -739,7 +739,12 @@ def _halo_read_access(self, arg):
                     # An upper bound that is part of the halo means
                     # that the halo might be accessed.
                     return True
+                # If a kernel updates fields on both continuous and
+                # discontinuous spaces then the iteration space is taken to be
+                # the continuous one.
+                iter_arg = self.kernel.arguments.iteration_space_arg()
                 if (not arg.discontinuous and
+                        not iter_arg.discontinuous and
                         self.kernel.iterates_over == "cell_column" and
                         self.kernel.all_updates_are_writes and
                         self._upper_bound_name == "ncells"):
diff --git a/src/psyclone/lfric.py b/src/psyclone/lfric.py
index af045f1ad2..2abd102e83 100644
--- a/src/psyclone/lfric.py
+++ b/src/psyclone/lfric.py
@@ -5046,7 +5046,8 @@ def _compute_from_field(self, field):
                 pass
             else:  # there is no stencil
                 if (field.discontinuous or call.iterates_over == "dof" or
-                        call.all_updates_are_writes):
+                        (not call.arguments.iteration_space_arg().discontinuous
+                         and call.all_updates_are_writes)):
                     # There are only local accesses or the kernel is of the
                     # special form where any iteration is guaranteed to write
                     # the same value to a given shared entity.
diff --git a/src/psyclone/tests/domain/lfric/lfric_field_mdata_test.py b/src/psyclone/tests/domain/lfric/lfric_field_mdata_test.py
index 7bfaeba1e7..7c2bbcf897 100644
--- a/src/psyclone/tests/domain/lfric/lfric_field_mdata_test.py
+++ b/src/psyclone/tests/domain/lfric/lfric_field_mdata_test.py
@@ -91,10 +91,15 @@
 '''
 
 
+@pytest.fixture(name="disable_fparser_logging", scope="function", autouse=True)
+def disable_fparser_logging_fixture():
+    '''Fixture to automate disabling of fparser logging.'''
+    fparser.logging.disable(fparser.logging.CRITICAL)
+
+
 def test_ad_fld_type_1st_arg():
     ''' Tests that an error is raised when the first argument descriptor
     metadata for a field is invalid. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace(
         "arg_type(gh_field,  gh_real,    gh_inc,     w1)",
         "arg_type(gh_hedge,  gh_real,    gh_inc,     w1)", 1)
@@ -112,7 +117,6 @@ def test_ad_fld_type_1st_arg():
 def test_ad_field_invalid_data_type():
     ''' Tests that an error is raised when the argument descriptor
     metadata for a field has an invalid data type. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     name = "testkern_field_type"
     # Check real field
     code = FIELD_CODE.replace(
@@ -140,7 +144,6 @@ def test_ad_field_invalid_data_type():
 def test_field_gh_sum_invalid():
     ''' Tests that an error is raised when a field is specified with
     access type 'gh_sum'. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace(
         "arg_type(gh_field,  gh_real,    gh_read,    w2)",
         "arg_type(gh_field,  gh_real,    gh_sum,     w2)", 1)
@@ -157,7 +160,6 @@ def test_field_gh_sum_invalid():
 def test_ad_fld_type_too_few_args():
     ''' Tests that an error is raised when the field argument descriptor
     metadata for a field has fewer than 3 args. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace(
         "arg_type(gh_field,  gh_real,    gh_inc,     w1)",
         "arg_type(gh_field,  gh_real,    gh_inc)", 1)
@@ -172,7 +174,6 @@ def test_ad_fld_type_too_few_args():
 def test_ad_fld_type_too_many_args():
     ''' Tests that an error is raised when the field argument descriptor
     metadata has more than 4 args. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace(
         "arg_type(gh_field,  gh_real,    gh_inc,     w1)",
         "arg_type(gh_field,  gh_real,    gh_inc,   w1, w1, w2)", 1)
@@ -239,7 +240,6 @@ def test_ad_field_init_wrong_data_type(monkeypatch):
 def test_arg_descriptor_invalid_fs():
     ''' Tests that an error is raised when an invalid function space
     name is provided as the third argument for a field. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     name = "testkern_field_type"
     # Check real field
     code = FIELD_CODE.replace(
@@ -270,7 +270,6 @@ def test_ad_field_init_wrong_iteration_space():
     LFRicArgDescriptor._init_field() method.
 
     '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     ast = fpapi.parse(FIELD_CODE, ignore_comments=False)
     metadata = LFRicKernMetadata(ast, name="testkern_field_type")
     field_arg = metadata._inits[1]
@@ -288,7 +287,6 @@ def test_ad_field_init_wrong_iteration_space():
 def test_fs_discontinuous_inc_error():
     ''' Test that an error is raised if a discontinuous function space
     and 'gh_inc' are provided for the same field in the metadata. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     const = LFRicConstants()
     for fspace in const.VALID_DISCONTINUOUS_NAMES:
         code = FIELD_CODE.replace(
@@ -310,7 +308,6 @@ def test_fs_continuous_cells_readwrite_error():
     in kernel metadata.
 
     '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     const = LFRicConstants()
     for fspace in const.CONTINUOUS_FUNCTION_SPACES:
         acc = "gh_readwrite"
@@ -333,7 +330,6 @@ def test_fs_anyspace_cells_readwrite_error():
     "'gh_readwrite' access in the metadata.
 
     '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     const = LFRicConstants()
     for fspace in const.VALID_ANY_SPACE_NAMES:
         acc = "gh_readwrite"
@@ -357,7 +353,6 @@ def test_fs_anyspace_dofs_inc_error(access):
     operates on DoFs.
 
     '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     dof_code = FIELD_CODE.replace("integer :: operates_on = cell_column",
                                   "integer :: operates_on = dof", 1)
     # gh_readinc also causes an exception so remove it for this test.
@@ -379,7 +374,6 @@ def test_fs_anyspace_dofs_inc_error(access):
 def test_arg_descriptor_field():
     ''' Test that the LFRicArgDescriptor argument representation works
     as expected for a field argument. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     ast = fpapi.parse(FIELD_CODE, ignore_comments=False)
     metadata = LFRicKernMetadata(ast, name="testkern_field_type")
     field_descriptor = metadata.arg_descriptors[2]
@@ -408,7 +402,6 @@ def test_arg_descriptor_field():
 def test_invalid_vector_operator():
     ''' Tests that an error is raised when a field vector does not
     use "*" as its operator. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace(
         "(gh_field,  gh_real,    gh_inc,     w1)",
         "(gh_field+3,  gh_real,    gh_inc,    w1)", 1)
@@ -422,7 +415,6 @@ def test_invalid_vector_operator():
 def test_invalid_vector_value_type():
     ''' Tests that an error is raised when a vector value is not a valid
     integer. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace("(gh_field,  gh_real,    gh_inc,     w1)",
                               "(gh_field*n,  gh_real,    gh_inc,     w1)", 1)
     ast = fpapi.parse(code, ignore_comments=False)
@@ -437,7 +429,6 @@ def test_invalid_vector_value_type():
 def test_invalid_vector_value_range():
     ''' Tests that an error is raised when a vector value is not a valid
     value (<2). '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace("(gh_field,  gh_real,    gh_inc,     w1)",
                               "(gh_field*1,  gh_real,    gh_inc,     w1)", 1)
     ast = fpapi.parse(code, ignore_comments=False)
@@ -455,7 +446,6 @@ def test_invalid_vector_value_range():
 def test_arg_descriptor_field_vector():
     ''' Test that the LFRicArgDescriptor argument representation works
     as expected when we have a field vector. '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     # Change the meta-data so that the second argument is a vector
     code = FIELD_CODE.replace("(gh_field,  gh_real,    gh_inc,     w1)",
                               "(gh_field*3,  gh_real,    gh_inc,    w1)", 1)
@@ -595,7 +585,6 @@ def test_field_arg_lfricconst_properties(monkeypatch):
 def test_multiple_updated_field_args():
     ''' Check that we successfully parse a kernel that writes to more
     than one of its field arguments '''
-    fparser.logging.disable(fparser.logging.CRITICAL)
     code = FIELD_CODE.replace("arg_type(gh_field,  gh_real,    gh_read,  w2)",
                               "arg_type(gh_field, gh_real, gh_inc, w2)", 1)
     ast = fpapi.parse(code, ignore_comments=False)
diff --git a/src/psyclone/tests/domain/lfric/lfric_kern_test.py b/src/psyclone/tests/domain/lfric/lfric_kern_test.py
index 823a81ef44..bd2762a1fa 100644
--- a/src/psyclone/tests/domain/lfric/lfric_kern_test.py
+++ b/src/psyclone/tests/domain/lfric/lfric_kern_test.py
@@ -449,22 +449,30 @@ def test_kern_last_cell_all_colours_intergrid():
             "last_edge_cell_all_colours_field1")
 
 
-def test_kern_all_updates_are_writes():
-    ''' Tests for the 'all_updates_are_writes' property of LFRicKern. '''
+def test_kern_all_updates_are_writes(monkeypatch):
+    ''' Tests for the 'all_updates_are_writes' property of
+    LFRicKern. '''
     _, invoke_info = parse(os.path.join(BASE_PATH, "1_single_invoke.f90"),
                            api=TEST_API)
     psy = PSyFactory(TEST_API, distributed_memory=True).create(invoke_info)
     sched = psy.invokes.invoke_list[0].schedule
     loop = sched.walk(LFRicLoop)[0]
+    kernel = loop.kernel
     # The only argument updated by this kernel has GH_INC access.
-    assert not loop.kernel.all_updates_are_writes
-    # Patch the kernel so that a different argument has GH_WRITE access.
-    loop.kernel.args[2]._access = AccessType.WRITE
-    # There is still a GH_INC argument.
-    assert not loop.kernel.all_updates_are_writes
-    # Change the GH_INC to be GH_WRITE.
-    loop.kernel.args[1]._access = AccessType.WRITE
-    assert loop.kernel.all_updates_are_writes
+    assert not kernel.all_updates_are_writes
+    # Patch the kernel so that two arguments have GH_WRITE access.
+    kernel.args[2]._access = AccessType.WRITE
+    kernel.args[1]._access = AccessType.WRITE
+    assert kernel.all_updates_are_writes
+    # Patch the kernel so that both updated field arguments appear to be
+    # on a discontinuous space.
+    monkeypatch.setattr(
+        kernel.arguments._args[1]._function_spaces[0],
+        "_orig_name", "w3")
+    monkeypatch.setattr(
+        kernel.arguments._args[2]._function_spaces[0],
+        "_orig_name", "w3")
+    assert kernel.all_updates_are_writes
 
 
 def test_kern_not_coloured_inc(monkeypatch):
diff --git a/src/psyclone/tests/domain/lfric/lfric_loop_test.py b/src/psyclone/tests/domain/lfric/lfric_loop_test.py
index e93a3f4914..a9265c85e9 100644
--- a/src/psyclone/tests/domain/lfric/lfric_loop_test.py
+++ b/src/psyclone/tests/domain/lfric/lfric_loop_test.py
@@ -698,6 +698,34 @@ def test_halo_for_discontinuous_2(tmpdir, monkeypatch, annexed):
     assert LFRicBuild(tmpdir).code_compiles(psy)
 
 
+def test_halo_for_annexed_dofs_read(tmpdir, annexed, monkeypatch):
+    '''
+    Test that a halo exchange is inserted before a kernel that reads from
+    both a continuous field and an operator in order to ensure annexed dofs
+    are clean.
+
+    '''
+    api_config = Config.get().api_conf(TEST_API)
+    monkeypatch.setattr(api_config, "_compute_annexed_dofs", annexed)
+    psy, invoke = get_invoke("15.1.11_builtin_and_op_kernel_invoke.f90",
+                             idx=0, api=TEST_API, dist_mem=True)
+    result = str(psy.gen).lower()
+    if annexed:
+        assert "loop0_stop = f2_proxy%vspace%get_last_dof_annexed" in result
+        assert "loop1_stop = mesh%get_last_edge_cell()" in result
+        # Halo dofs are left dirty, even though the annexed dofs are clean
+        assert "call f2_proxy%set_dirty()" in result
+        # No halo exchange required
+        assert "call f2_proxy%halo_exchange" not in result
+    else:
+        assert "loop0_stop = f2_proxy%vspace%get_last_dof_owned()" in result
+        assert "loop1_stop = mesh%get_last_edge_cell()" in result
+        # f2 must be halo-exchanged to clean its annexed dofs.
+        assert "call f2_proxy%set_dirty()" in result
+        assert "call f2_proxy%halo_exchange(depth=1)" in result
+    assert LFRicBuild(tmpdir).code_compiles(psy)
+
+
 def test_lfricloop_halo_read_access_error1(monkeypatch):
     '''Test that the halo_read_access method in class LFRicLoop raises the
     expected exception when an unsupported field access is found.
diff --git a/src/psyclone/tests/test_files/lfric/15.1.11_builtin_and_op_kernel_invoke.f90 b/src/psyclone/tests/test_files/lfric/15.1.11_builtin_and_op_kernel_invoke.f90
new file mode 100644
index 0000000000..476afcd104
--- /dev/null
+++ b/src/psyclone/tests/test_files/lfric/15.1.11_builtin_and_op_kernel_invoke.f90
@@ -0,0 +1,58 @@
+! -----------------------------------------------------------------------------
+! BSD 3-Clause License
+!
+! Copyright (c) 2025, Science and Technology Facilities Council.
+! All rights reserved.
+!
+! Redistribution and use in source and binary forms, with or without
+! modification, are permitted provided that the following conditions are met:
+!
+! * Redistributions of source code must retain the above copyright notice, this
+!   list of conditions and the following disclaimer.
+!
+! * Redistributions in binary form must reproduce the above copyright notice,
+!   this list of conditions and the following disclaimer in the documentation
+!   and/or other materials provided with the distribution.
+!
+! * Neither the name of the copyright holder nor the names of its
+!   contributors may be used to endorse or promote products derived from
+!   this software without specific prior written permission.
+!
+! THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+! "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+! LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+! FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+! COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+! INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+! BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+! LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+! CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+! LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+! ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+! POSSIBILITY OF SUCH DAMAGE.
+! -----------------------------------------------------------------------------
+! Author: A. R. Porter, STFC Daresbury Lab
+
+program single_invoke_builtin_then_kernel
+
+  ! Description: single invoke call with a builtin followed by a kernel call
+  ! with an operator argument.
+  use constants_mod,        only: r_def
+  use field_mod,            only: field_type
+  use operator_mod,         only: operator_type
+  use dg_matrix_vector_kernel_mod, only: dg_matrix_vector_kernel_type
+
+  implicit none
+
+  type(field_type) :: f2, f4
+  real(r_def)      :: scalar = 0.0
+  type(operator_type), pointer    :: mass_matrix => null()
+
+  call invoke(                               &
+       setval_c(f2, 0.0),                    &
+       ! f4 - discont. space, written
+       ! f2 - continuous space, read - needs halo exchange to get
+       ! clean annexed dofs.
+       dg_matrix_vector_kernel_type(f4, f2, mass_matrix) )
+
+end program single_invoke_builtin_then_kernel
diff --git a/src/psyclone/tests/test_files/lfric/dg_matrix_vector_kernel_mod.F90 b/src/psyclone/tests/test_files/lfric/dg_matrix_vector_kernel_mod.F90
new file mode 100644
index 0000000000..8a539b9d1a
--- /dev/null
+++ b/src/psyclone/tests/test_files/lfric/dg_matrix_vector_kernel_mod.F90
@@ -0,0 +1,146 @@
+!-----------------------------------------------------------------------------
+! Copyright (c) 2017,  Met Office, on behalf of HMSO and Queen's Printer
+! For further details please refer to the file LICENCE.original which you
+! should have received as part of this distribution.
+!-----------------------------------------------------------------------------
+!> @brief This version is for use on discontinous spaces and over-writes the
+!>        output field entirely, hence can only be used for W3 spaces
+module dg_matrix_vector_kernel_mod
+
+  use constants_mod, only : i_def, r_single, r_double
+  use kernel_mod,    only : kernel_type
+  use argument_mod,  only : arg_type,                  &
+                            GH_FIELD, GH_OPERATOR,     &
+                            GH_READ, GH_WRITE,         &
+                            GH_REAL, ANY_SPACE_1,      &
+                            ANY_DISCONTINUOUS_SPACE_1, &
+                            CELL_COLUMN
+
+  implicit none
+
+  private
+
+  !---------------------------------------------------------------------------
+  ! Public types
+  !---------------------------------------------------------------------------
+
+  type, public, extends(kernel_type) :: dg_matrix_vector_kernel_type
+    private
+    type(arg_type) :: meta_args(3) = (/                                       &
+         arg_type(GH_FIELD,    GH_REAL, GH_WRITE, ANY_DISCONTINUOUS_SPACE_1), &
+         arg_type(GH_FIELD,    GH_REAL, GH_READ,  ANY_SPACE_1),               &
+         arg_type(GH_OPERATOR, GH_REAL, GH_READ,  ANY_DISCONTINUOUS_SPACE_1,  &
+                                                  ANY_SPACE_1)                &
+         /)
+    integer :: operates_on = CELL_COLUMN
+  end type
+
+  !---------------------------------------------------------------------------
+  ! Contained functions/subroutines
+  !---------------------------------------------------------------------------
+  public :: dg_matrix_vector_code
+
+  ! Generic interface for real32 and real64 types
+  interface dg_matrix_vector_code
+    module procedure  &
+      dg_matrix_vector_code_r_single, &
+      dg_matrix_vector_code_r_double
+  end interface
+
+contains
+
+!> @brief Computes lhs = matrix*x for discontinuous function spaces
+!> @param[in] cell Horizontal cell index
+!> @param[in] nlayers Number of layers
+!> @param[in,out] lhs Output lhs (A*x)
+!> @param[in] x input data
+!> @param[in] ncell_3d Total number of cells
+!> @param[in] matrix Matrix values in LMA form
+!> @param[in] ndf1 Number of degrees of freedom per cell for the output field
+!> @param[in] undf1 Unique number of degrees of freedom  for the output field
+!> @param[in] map1 Dofmap for the cell at the base of the column for the output field
+!> @param[in] ndf2 Number of degrees of freedom per cell for the input field
+!> @param[in] undf2 Unique number of degrees of freedom for the input field
+!> @param[in] map2 Dofmap for the cell at the base of the column for the input field
+
+! R_SINGLE PRECISION
+! ==================
+subroutine dg_matrix_vector_code_r_single(cell,              &
+                                          nlayers,           &
+                                          lhs, x,            &
+                                          ncell_3d,          &
+                                          matrix,            &
+                                          ndf1, undf1, map1, &
+                                          ndf2, undf2, map2)
+
+  implicit none
+
+  ! Arguments
+  integer(kind=i_def),                   intent(in) :: cell, nlayers, ncell_3d
+  integer(kind=i_def),                   intent(in) :: undf1, ndf1
+  integer(kind=i_def),                   intent(in) :: undf2, ndf2
+  integer(kind=i_def), dimension(ndf1),  intent(in) :: map1
+  integer(kind=i_def), dimension(ndf2),  intent(in) :: map2
+  real(kind=r_single), dimension(undf2),              intent(in)    :: x
+  real(kind=r_single), dimension(undf1),              intent(inout) :: lhs
+  real(kind=r_single), dimension(ndf1,ndf2,ncell_3d), intent(in)    :: matrix
+
+  ! Internal variables
+  integer(kind=i_def)                  :: df, k, ik
+  real(kind=r_single), dimension(ndf2) :: x_e
+  real(kind=r_single), dimension(ndf1) :: lhs_e
+
+  do k = 0, nlayers-1
+    do df = 1, ndf2
+      x_e(df) = x(map2(df)+k)
+    end do
+    ik = (cell-1)*nlayers + k + 1
+    lhs_e = matmul(matrix(:,:,ik),x_e)
+    do df = 1,ndf1
+       lhs(map1(df)+k) = lhs_e(df)
+    end do
+  end do
+
+end subroutine dg_matrix_vector_code_r_single
+
+! R_DOUBLE PRECISION
+! ==================
+subroutine dg_matrix_vector_code_r_double(cell,              &
+                                          nlayers,           &
+                                          lhs, x,            &
+                                          ncell_3d,          &
+                                          matrix,            &
+                                          ndf1, undf1, map1, &
+                                          ndf2, undf2, map2)
+
+  implicit none
+
+  ! Arguments
+  integer(kind=i_def),                   intent(in) :: cell, nlayers, ncell_3d
+  integer(kind=i_def),                   intent(in) :: undf1, ndf1
+  integer(kind=i_def),                   intent(in) :: undf2, ndf2
+  integer(kind=i_def), dimension(ndf1),  intent(in) :: map1
+  integer(kind=i_def), dimension(ndf2),  intent(in) :: map2
+  real(kind=r_double), dimension(undf2),              intent(in)    :: x
+  real(kind=r_double), dimension(undf1),              intent(inout) :: lhs
+  real(kind=r_double), dimension(ndf1,ndf2,ncell_3d), intent(in)    :: matrix
+
+  ! Internal variables
+  integer(kind=i_def)                  :: df, k, ik
+  real(kind=r_double), dimension(ndf2) :: x_e
+  real(kind=r_double), dimension(ndf1) :: lhs_e
+
+  do k = 0, nlayers-1
+    do df = 1, ndf2
+      x_e(df) = x(map2(df)+k)
+    end do
+    ik = (cell-1)*nlayers + k + 1
+    lhs_e = matmul(matrix(:,:,ik),x_e)
+    do df = 1,ndf1
+       lhs(map1(df)+k) = lhs_e(df)
+    end do
+  end do
+
+end subroutine dg_matrix_vector_code_r_double
+
+end module dg_matrix_vector_kernel_mod


Reviews:
user: login: sergisiso
body: @arporter See may comments about the updated method
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T10:16:52Z
user: login: arporter
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T18:34:36Z
user: login: sergisiso
body: @arporter I like this solution more. Thanks for updating it and finding where the problems were.

> However, I think the default transformation script used by LFRic will still be adding redundant computation to setval_? builtins. We could perhaps undo that as well?

I agree our own empty no_op.py could be useful for the passthrough . Could you add it please and then this can go.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T12:26:40Z
user: login: sergisiso
body: All good now @arporter and I see the integration tests that you launched are green. This is ready to merge.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-23T10:09:41Z

Review comments:
diff_hunk: @@ -601,13 +601,20 @@ def mesh(self):
         return self._mesh_properties
 
     @property
-    def all_updates_are_writes(self):
+    def all_continuous_updates_are_writes(self) -> bool:
path: src/psyclone/domain/lfric/lfric_kern.py
user: login: sergisiso
body: I am confused, the name does not match the implementation: it is called 'all_continuous_updates_are_writes' but it returns True if all updates are write-only, including the discontinuous, as far as and the argument we use to dereference the iteration space is not discontinuous.

So my question is why is the argument we use to dereference the iteration space special? Should it be if any argument is discontinuous it returns False? Or is it because if the iteration space is discontinuous, we iterate over the halo which is what we are testing here. If its the second, the name will be something different or even the caller should be responsible for the check. Do the discontinuous check be off if the distributed memory is off?

Also, wouldn't READS need the same halo exchanges, isn't the caller already checking the same conditions?

I have the feeling that this is trying to fix a problem of the caller making this internal functionality very specific.

Also, I do not understand the docstring, again it seems to talk about a very specific condition for a method with a very generic name, and I can't figure out the context.
created_at: 2025-05-16T10:13:17Z
author_association: COLLABORATOR
diff_hunk: @@ -601,13 +601,20 @@ def mesh(self):
         return self._mesh_properties
 
     @property
-    def all_updates_are_writes(self):
+    def all_continuous_updates_are_writes(self) -> bool:
path: src/psyclone/domain/lfric/lfric_kern.py
user: login: arporter
body: I take your point(s). It was poor. I've reverted my changes to this method and have put the additional logic (checking that the field being updated is on a continuous space) in the appropriate places. This has simplified things quite a lot.
created_at: 2025-05-21T18:34:36Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-05-13T11:09:04Z
body: ## [Codecov](https://app.codecov.io/gh/stfc/PSyclone/pull/2994?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stfc) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 99.91%. Comparing base [(`9776bce`)](https://app.codecov.io/gh/stfc/PSyclone/commit/9776bced52cc7c0c7aa446a143d90fa8003608ae?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stfc) to head [(`da7a139`)](https://app.codecov.io/gh/stfc/PSyclone/commit/da7a1393633bf15366f91974d6e73a03e81440d1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stfc).
> Report is 27 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master    #2994   +/-   ##
=======================================
  Coverage   99.91%   99.91%           
=======================================
  Files         365      365           
  Lines       51898    51899    +1     
=======================================
+ Hits        51852    51853    +1     
  Misses         46       46           
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/stfc/PSyclone/pull/2994?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stfc).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stfc).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: arporter
created_at: 2025-05-13T12:44:24Z
body: With this fix, I can run GungHo and get the same answers (checksums) with and without COMPUTE_ANNEXED_DOFS set. @hiker / @christophermaynard does that mean the original problem that we had is fixed or is there some other test I need to do?
author_association: MEMBER
user: login: arporter
created_at: 2025-05-13T12:51:24Z
body: The generated code is definitely different:
```bash
with_annexed_working]$ find . -name \*_psy.f90 -exec wc -l {} \; | awk '{sum += $1}END{print sum}' - 
105800
working]$ find . -name \*_psy.f90 -exec wc -l {} \; | awk '{sum += $1}END{print sum}' - 
106630
```
author_association: MEMBER
user: login: arporter
created_at: 2025-05-14T10:22:03Z
body: If I repeat this with `master` (and `COMPUTE_ANNEXED_DOFS=False`) then GungHo runs but the checksums are wrong.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-14T14:13:21Z
body: This is a small but significant bug fix to the functionality that identifies special LFRic kernels that have GH_WRITE on continuous fields. This impacts the halo-exchange logic and is a cause of problems identified when COMPUTE_ANNEXED_DOFS is set to False.
author_association: MEMBER
user: login: hiker
created_at: 2025-05-15T13:45:16Z
body: > With this fix, I can run GungHo and get the same answers (checksums) with and without COMPUTE_ANNEXED_DOFS set. @hiker / @christophermaynard does that mean the original problem that we had is fixed or is there some other test I need to do?

Not sure what exactly the problem is either, i.e. if it was a crash, invalid results (different checksums). I am not even sure how much it affected gungho, I assume the original test was lfric_atm.
author_association: COLLABORATOR
user: login: mo-joshuacolclough
created_at: 2025-05-16T09:19:57Z
body: Thanks for all the work on this. I'll be able to test this soon with the adjoint - currently asking around about how to do so!

> if it was a crash, invalid results (different checksums). I am not even sure how much it affected gungho, I assume the original test was lfric_atm.

For some context: This was found when developing the LFRic adjoint model. For those tests we use `setval_random` to initialise fields, and in this case with `COMUTE_ANNEXED_DOFS=false`. When also running with a debug build, unset field values are set as `nan` values, which is useful for seeing when `dirty` dofs are being accessed. This is quite useful for the adjoint, as instances of this happening may result in an invalid adjoint & a hard-to-find bug!

In this particular case, dirty annexed dofs are accessed in the linear code call. If I manually extend the `setval_random` calls to populate annexed dofs (i.e, dirty but at least set to a value), then the nan issues go away, but we see an adjoint test failure. And so a halo exchange is required.

> I am not even sure how much it affected gungho

It's possible this has a small impact on results from MPI runs, around the partition boundaries, when kernels operate between functionspaces with annexed dofs in particular... However I have not measured the impact myself; it perhaps should be checked by someone in-the-know.

On the other hand the adjoint model has to be an exact transpose of the linear model, which is why this small discrepancy has been picked up here and must be fixed in this case.
author_association: NONE
user: login: arporter
created_at: 2025-05-19T15:51:04Z
body: I've significantly tightened-up the function but am now getting an integration test failure with the OMP offload version of LFRic. Passthrough with DM works fine. Failure is in the solver:
```
20250519164016.214+0100:P0:S1:INFO : BLOCK_GCR starting ... ||b|| =  0.10021990E
+04: 0.10266128E+04
20250519164017.246+0100:P0:S1:INFO : BLOCK_GCR solver_algorithm: [ 1, 4], iters,
 init=  0.1027E+04 final=            NaN abs=            NaN
```
weirdly, this is with `COMPUTE_ANNEXED_DOFS = true`.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-19T16:09:02Z
body: I can reproduce this failure on CPU with `everything_everywhere_all_at_once.py` and ANNEXED_DOFS set to false. My latest changes have substantially changed the behaviour of the method - it excludes those kernels that update fields on discontinuous spaces and, separately, those that update an operator. It didn't do that previously.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-19T16:15:46Z
body: Original version of the method:
![image](https://github.com/user-attachments/assets/a3d8c6e3-5d54-4a32-a630-2b86fe9a3763)
For debugging purposes, I can put this code back into the method and compare the result with that of my new code - when they differ I can break into pdb.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-20T08:33:22Z
body: Having slept on it, I decided that what I needed to do was effectively revert my changes and just include the check that the field with GH_WRITE access is on a continuous space. I can build and run GungHo with `everything_everywhere_all_at_once.py` with this change and get the correct results (with COMPUTE_ANNEXED_DOFS=false).

However, now that I understand what we're checking for, it seems incredibly fragile. A user could easily make the mistake of specifying GH_WRITE on a continuous field when they really meant GH_INC. As I've found to my cost, debugging such a mistake is hard. I would therefore argue that, for this special case, we should have a different form of access, e.g. `GH_CONTINUOUS_WRITE` to make it clear this is something special. Obviously this would be a non-backwards compatible change but I think it's important. What do @mo-joshuacolclough, @TeranIvy and @tommbendall think?
author_association: MEMBER
user: login: tommbendall
created_at: 2025-05-20T14:58:03Z
body: > Having slept on it, I decided that what I needed to do was effectively revert my changes and just include the check that the field with GH_WRITE access is on a continuous space. I can build and run GungHo with `everything_everywhere_all_at_once.py` with this change and get the correct results (with COMPUTE_ANNEXED_DOFS=false).
> 
> However, now that I understand what we're checking for, it seems incredibly fragile. A user could easily make the mistake of specifying GH_WRITE on a continuous field when they really meant GH_INC. As I've found to my cost, debugging such a mistake is hard. I would therefore argue that, for this special case, we should have a different form of access, e.g. `GH_CONTINUOUS_WRITE` to make it clear this is something special. Obviously this would be a non-backwards compatible change but I think it's important. What do @mo-joshuacolclough, @TeranIvy and @tommbendall think?

Hi Andy, just to check I've understood, are you proposing that we again ban `GH_WRITE` metadata for continuous fields (it was banned before #1604), but instead have a new `GH_CONTINUOUS_WRITE` metadata? What exactly would that do differently?
author_association: COLLABORATOR
user: login: arporter
created_at: 2025-05-20T16:27:17Z
body: > Hi Andy, just to check I've understood, are you proposing that we again ban `GH_WRITE` metadata for continuous fields (it was banned before #1604), but instead have a new `GH_CONTINUOUS_WRITE` metadata? What exactly would that do differently?

Hi Tom, functionally it would be identical. However, it would make it clearer that the kernel does something special. I'm worried about users just cut-n-pasting some metadata and modifying it until it appears to work, without realising that GH_WRITE on a continuous space is a special thing. I'm aware that changing the metadata will have a large cost associated with it so perhaps there are other solutions? Ideally we would extend PSyclone to check whether the kernel implementation is conformant but I suspect that will be very difficult or impossible. Perhaps we could introduce support for PSyKE to generate tests for kernels based on their metadata @hiker ? Actually, it might not be a job for PSyKE - it would be similar to what PSyAD does in generating a test harness for adjointed kernels.
author_association: MEMBER
user: login: tommbendall
created_at: 2025-05-21T08:45:52Z
body: > > Hi Andy, just to check I've understood, are you proposing that we again ban `GH_WRITE` metadata for continuous fields (it was banned before #1604), but instead have a new `GH_CONTINUOUS_WRITE` metadata? What exactly would that do differently?
> 
> Hi Tom, functionally it would be identical. However, it would make it clearer that the kernel does something special. I'm worried about users just cut-n-pasting some metadata and modifying it until it appears to work, without realising that GH_WRITE on a continuous space is a special thing. I'm aware that changing the metadata will have a large cost associated with it so perhaps there are other solutions? Ideally we would extend PSyclone to check whether the kernel implementation is conformant but I suspect that will be very difficult or impossible. Perhaps we could introduce support for PSyKE to generate tests for kernels based on their metadata @hiker ? Actually, it might not be a job for PSyKE - it would be similar to what PSyAD does in generating a test harness for adjointed kernels.

Hi Andy, actually I think this is a good idea. I'm not sure how many instances of `GH_WRITE` on a continuous field that there are in LFRic, but it shouldn't be too much work to change those.

I don't know if this is relevant, but there are also situations in LFRic where there are continuous fields but the metadata is specified as `GH_WRITE` and `ANY_DISCONTINUOUS_SPACE`, because only one column will write to the shared DoFs.
author_association: COLLABORATOR
user: login: arporter
created_at: 2025-05-21T18:36:36Z
body: I've modified the passthrough integration test to have COMPUTE_ANNEXED_DOFS=false. However, I think the default transformation script used by LFRic will still be adding redundant computation to `setval_?` builtins. We could perhaps undo that as well?

If the CI and integration tests are green then this is ready for another look.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-22T17:44:11Z
body: I've added a `does_nothing.py` script and have tested it with `COMPUTE_ANNEXED_DOFS=false` and the answers matched. I've also updated the LFRic 'passthrough' integration test to use this script. I'll fire that off to check it works.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-23T09:24:36Z
body: Integration test was green :-) I'll just put Alistair's big PR on (#2979) and then bring this up-to-date.
author_association: MEMBER
user: login: arporter
created_at: 2025-05-23T09:52:48Z
body: Ready for another look now @sergisiso.
author_association: MEMBER