Repository: crossplane/crossplane

Issue:
`TestUsageComposition` consistently failing with realtime compositions
<!--
Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues
for bug reports and feature requests. Please find us at https://slack.crossplane.io
for questions, support, and discussion.
-->

### What happened?
<!--
Please let us know what behaviour you expected and how Crossplane diverged from
that behaviour.
-->

I'm seeing the `TestUsageComposition` E2E fail almost all of the time when realtime compositions are enabled, especially since I merged https://github.com/crossplane/crossplane/pull/6395, which should make realtime composition work more consistently.

For example:

* https://github.com/crossplane/crossplane/pull/6407
* https://github.com/crossplane/crossplane/pull/6405 (fails 3-4 times in a row)

### How can we reproduce it?
<!--
Help us to reproduce your bug as succinctly and precisely as possible. Artifacts
such as example manifests or a script that triggers the issue are highly
appreciated!
-->

See PRs above.

### What environment did it happen in?
Crossplane version: See PRs above.

<!--
Include at least the version or commit of Crossplane you were running. Consider
also including your:

* Cloud provider or hardware configuration
* Kubernetes version (use `kubectl version`)
* Kubernetes distribution (e.g. Tectonic, GKE, OpenShift)
* OS (e.g. from /etc/os-release)
* Kernel (e.g. `uname -a`)
-->


Discussion:
@negz: When I run the tests locally I can see that:

* The XR reconciler is stuck in a hot loop, constantly reconciling the XR
* The "using" resource's `finalizer.managedresource.crossplane.io` is being removed and re-added

Looks like perhaps two controllers are fighting to add/remove this finalizer, and making realtime compositions reliably work has caused them to enter a tight loop?

I notice the `delay-deletion-of-using-resource` finalizer is always there.@negz: https://github.com/crossplane/crossplane/blob/e7763245b5d81f6a6be6c0b71998acf74875fe0f/test/e2e/manifests/apiextensions/usage/composition/setup/composition.yaml#L35-L36

https://github.com/crossplane/crossplane-runtime/blob/v1.19.0/pkg/resource/api.go#L86

Ah. The test is using native P&T to set a finalizer. Native P&T just uses a dumb JSON merge patch (https://datatracker.ietf.org/doc/html/rfc7386), to apply the base. So that'll replace the entire finalizer array (removing the provider's finalizer) every time it's applied.

So the MR controller is watching the MR and adding its finalizer, while the XR controller is watching the resource and removing its finalizer.

There's actually an identical test that uses functions. I'm just going to delete the native P&T test. Ignoring this somewhat artificial pattern of adding a finalizer explicitly as part of composition, I don't see any meaningful reason the choice of composition engine would affect how Usages work. 

Solution patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5de3a1013..dea3dcdb5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -288,7 +288,7 @@ jobs:
       - name: Run E2E Tests
         run: |
           earthly --strict --allow-privileged --remote-cache ghcr.io/crossplane/earthly-cache:${{ github.job }}-${{ matrix.test-suite}} \
-            +e2e --FLAGS="-test.failfast -fail-fast -prior-crossplane-version=${CROSSPLANE_PRIOR_VERSION} --test-suite ${{ matrix.test-suite }}"
+            +e2e --GOTESTSUM_FORMAT="testname" --FLAGS="-test.failfast -fail-fast -prior-crossplane-version=${CROSSPLANE_PRIOR_VERSION} --test-suite ${{ matrix.test-suite }}"
 
       - name: Publish E2E Test Flakes
         if: '!cancelled()'
diff --git a/Earthfile b/Earthfile
index c35a30fff..d35883250 100644
--- a/Earthfile
+++ b/Earthfile
@@ -51,6 +51,7 @@ e2e:
   ARG GOARCH=${TARGETARCH}
   ARG GOOS=${TARGETOS}
   ARG FLAGS="-test-suite=base"
+  ARG GOTESTSUM_FORMAT="standard-verbose"
   # Using earthly image to allow compatibility with different development environments e.g. WSL
   FROM earthly/dind:alpine-3.20-docker-26.1.5-r0
   RUN wget https://dl.google.com/go/go${GO_VERSION}.${GOOS}-${GOARCH}.tar.gz
@@ -71,7 +72,12 @@ e2e:
     WITH DOCKER --load crossplane-e2e/crossplane:latest=(+image --CROSSPLANE_VERSION=v0.0.0-e2e)
       # TODO(negz:) Set GITHUB_ACTIONS=true and use RUN --raw-output when
       # https://github.com/earthly/earthly/issues/4143 is fixed.
-      RUN gotestsum --no-color=false --format testname --junitfile e2e-tests.xml --raw-command go tool test2json -t -p E2E ./e2e -test.v ${FLAGS}
+      RUN gotestsum \
+        --rerun-fails \
+      	--no-color=false \
+	--format ${GOTESTSUM_FORMAT} \
+	--junitfile e2e-tests.xml \
+	--raw-command go tool test2json -t -p E2E ./e2e -test.v ${FLAGS}
     END
   FINALLY
     SAVE ARTIFACT --if-exists e2e-tests.xml AS LOCAL _output/tests/e2e-tests.xml



Test patch:
diff --git a/test/e2e/README.md b/test/e2e/README.md
index f7bc7cb87..77e3bed99 100644
--- a/test/e2e/README.md
+++ b/test/e2e/README.md
@@ -89,21 +89,6 @@ $ apk add kubectl
 # Now you can use kubectl to access the cluster
 ```
 
-### Test Logging
-
-The E2E tests are run using [`gotestsum`] that helps output the test results in
-a format that integrates easily with CI systems. The downside though is that
-test logging statements, like `t.Logf("foo")`, are not streamed in real time and
-they are only included in the output if there is a failure.
-
-To change this behavior when running the E2E tests locally, you can update the
-`Earthfile` to call `gotestsum` with a format like `--format standard-verbose`
-instead, for example:
-
-```Dockerfile
-RUN gotestsum --no-color=false --format standard-verbose ...
-```
-
 ### Running Tests Directly
 
 The E2E tests are typically run via `earthly` for convenience and consistency.