Repository: tektoncd/pipeline (language: go)

Issue:
ClusterTask creates CustomRun (v0.41.1 -> v0.44.2)
# Expected Behavior

I get an error/warning when I create my `ClusterTasks` or they run as normal `TaskRuns`.

# Actual Behavior

My `ClusterTasks` create `CustomRuns` instead of `TaskRuns` when I update from `v0.41.1` to `v0.44.2`.

# Steps to Reproduce the Problem

1. `minikube start --driver=kvm2 --cpus=4 --memory=6g --disk-size=30GB --embed-certs`
2. `kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.41.1/release.yaml`
3. `kubect apply --filename dummy.yaml` (file below)
4. `kubect get pipelinerun dummy` (this should complete after a while)
5. `kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.44.2/release.yaml`
6. `kubect apply --filename dummy.yaml` (file below)
7. `kubect get pipelinerun dummy` (this should hang)
8. `kubectl get customruns` (this should return a single one, called `dummy-init`)

<details>
<summary>dummy.yaml</summary>

```
# dummy.yaml
---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: dummy-2
  namespace: default
spec:
  tasks:
  - name: init
    params:
    - name: resource
      value: resouce
    taskRef:
      apiVersion: tekton.dev/v1beta1
      kind: ClusterTask
      name: init
    workspaces:
    - name: output
      workspace: shared
  workspaces:
  - description: Main workspace that is shared across each task in the build pipeline
    name: shared
---
apiVersion: tekton.dev/v1beta1
kind: ClusterTask
metadata:
  name: init
spec:
  workspaces:
    - name: output
  params:
  - name: resource
    type: string
  steps:
  - name: hello
    image: registry.access.redhat.com/ubi9-micro:latest
    args: ["$(params.resource)"]
    command: ["echo"]
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: dummy
spec:
  pipelineRef:
    name: dummy-2
  workspaces:
  - name: shared
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Mi

```

</details>

# Additional Info

- Kubernetes version:

  **Output of `kubectl version`:**

```
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.26.3
```

- Tekton Pipeline version:

  **Output of `tkn version` or `kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'`**

```
v0.41.1

v0.44.2
```


<!-- Any other additional information -->

I am trying to understand why this is happening. Thanks!


Discussion / hints:
I think this is the commit is the direct cause of the issue https://github.com/tektoncd/pipeline/pull/5858. Which should be in the release of v0.43.0
If the taskref has apiversion and kind, it is considered as custom task and will create customrun. 
To fix this issue, there are several options
1) if you're using v0.44.2, you could turn off the "enable-custom-tasks" feature flag.
2) remove the apiversion from the taskref
 
This is also related to this issue https://github.com/tektoncd/pipeline/issues/6457
Maybe we should exclude task and clustertask from this function? 
https://github.com/tektoncd/pipeline/blob/cec2422542b5f187e041ef9f96c55f3b2ab5380a/pkg/apis/pipeline/v1beta1/taskref_types.go#L58C1-L61
cc @jerop 
Yeah, this seems like a bug, we should exclude both Tasks and ClusterTasks
@Yongxuanzhang , I set `enable-custom-tasks: "false"` in the `feature-flags` configmap, but there was no change in behavior. In fact, that option wasn't there at all in the configmap.

Is this option still recognized in 0.44.2? 

I'd really prefer to use that to shut it off completely. Otherwise we'll have to make changes in multiple places.
Hi @Yongxuanzhang 

The feature flag was removed on [v0.44.0](https://github.com/tektoncd/pipeline/releases/tag/v0.44.0), and I also tried to make it work on v0.44.2, but it does not.

Removing the apiVersion from the `taskRef` worked properly, now a `TaskRun` is created instead.

Thank you very much.

> @Yongxuanzhang , I set `enable-custom-tasks: "false"` in the `feature-flags` configmap, but there was no change in behavior. In fact, that option wasn't there at all in the configmap.
> 
> Is this option still recognized in 0.44.2?
> 
> I'd really prefer to use that to shut it off completely. Otherwise we'll have to make changes in multiple places.

Oh sorry, the feature flag was removed. If you don't want to make changes, another way is that we can patch a fix to previous versions. 
I will open a fix today and keep you updated. 
@Yongxuanzhang , rather than a patch to add the `enable-custom-tasks`  option back, why not just patch the real fix to exclude `Task`s and `ClusterTask`s?
> @Yongxuanzhang , rather than a patch to add the `enable-custom-tasks` option back, why not just patch the real fix to exclude `Task`s and `ClusterTask`s?

I didn't mean to add that feature flag back, it will be a fix to exclude them as I mentioned above. 
If you could use v0.43, I think the feature flag should be there?
/assign

Fix patch:
diff --git a/docs/pipelines.md b/docs/pipelines.md
index 63e02e8b46a..8f0f99297ea 100644
--- a/docs/pipelines.md
+++ b/docs/pipelines.md
@@ -279,6 +279,8 @@ tasks:
       name: build-push
 ```
 
+**Note:** Using both `apiVersion` and `kind` will create [CustomRun](customruns.md), don't set `apiVersion` if only referring to [`Task`](tasks.md).
+
 or
 
 ```yaml
@@ -1537,7 +1539,8 @@ that is responsible for watching and updating `CustomRun`s which reference their
 ### Specifying the target Custom Task
 
 To specify the custom task type you want to execute, the `taskRef` field
-must include the custom task's `apiVersion` and `kind` as shown below:
+must include the custom task's `apiVersion` and `kind` as shown below.
+Using `apiVersion` will always create a `CustomRun`. If `apiVersion` is set, `kind` is required as well.
 
 ```yaml
 spec:


Test patch:
diff --git a/pkg/reconciler/taskrun/resources/taskref_test.go b/pkg/reconciler/taskrun/resources/taskref_test.go
index a739c20a8f1..444103a5475 100644
--- a/pkg/reconciler/taskrun/resources/taskref_test.go
+++ b/pkg/reconciler/taskrun/resources/taskref_test.go
@@ -396,10 +396,9 @@ func TestGetTaskFunc(t *testing.T) {
 				},
 			},
 			ref: &v1beta1.TaskRef{
-				Name:       "simple",
-				APIVersion: "tekton.dev/v1beta1",
-				Kind:       v1beta1.ClusterTaskKind,
-				Bundle:     u.Host + "/remote-cluster-task",
+				Name:   "simple",
+				Kind:   v1beta1.ClusterTaskKind,
+				Bundle: u.Host + "/remote-cluster-task",
 			},
 			expected: &v1beta1.Task{
 				ObjectMeta: metav1.ObjectMeta{Name: "simple"},