Repository: linkerd/linkerd2
Pull request #11805 (2023-12-20 14:56:19)

Introduce ExternalWorkload CRD
To support mesh expansion, the control plane needs to read configuration associated with an external instance (i.e. a VM) for the purpose of service and inbound authorization policy discovery.

This change introduces a new CRD that supports the required configuration options. The resource supports:

* a list of workload IPs (with a generic format to support ipv4 and ipv6 in the future)
* a set of mesh TLS settings (SNI and identity)
* a set of ports exposed by the workload
* a set of status conditions

---

## Some background on choices

OpenAPI supports validation, where applicable, we add validation rules.

* We validate SNI & Identity are max 253 characters (max allowed for a DNS name, although for identity it might not make sense since we will support URIs. SNI is based on DNS though).
* We validate conditions, but a resource may be created without a status. Condition validation is taken from the httproute resource since it follows a very similar format.
* We do not require meshTls settings, but we require workloadIPs and ports.

Unfortunately, for ips we cannot add any server side validation through the schema since we plan on supporting both ipv4 and ipv6. All IPs will have to be parsed and validated in the controllers that read the resource.

For printer columns, we list the age and the identity. IPs and ports will not render properly since printer columns only support primitive data types (record types can still be printed but in debug format). A choice would be to eliminate the need for an `ip` key and have the IPs printed as `["192.0.2.0", "192.0.3.0"].


Output:

```
NAME              IDENTITY                                           AGE
nginx-vm-test-3   foo.default.serviceaccount.linkerd.cluster.local   2m56s
nginx-vm-test-4   bar.default.serviceaccount.linkerd.cluster.local   2s
```

and with IPs as a printer column:

```
NAME              IPS                    PORTS
nginx-vm-test-3   [{"ip":"192.0.2.2"}]   [{"port":8080,"protocol":"TCP"},{"port":7777,"protocol":"TCP"}]
```

```yaml
apiVersion: workload.linkerd.io/v1alpha1
kind: ExternalWorkload
metadata:
  annotations:
    config.linkerd.io/default-inbound-policy: deny
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"workload.linkerd.io/v1alpha1","kind":"ExternalWorkload","metadata":{"annotations":{"config.linkerd.io/default-inbound-policy":"deny"},"labels":{"app":"echo-service"},"name":"nginx-vm-test-3","namespace":"default"},"spec":{"meshTls":{"identity":"foo.default.serviceaccount.linkerd.cluster.local","serverName":"foo.default.serviceaccount.linkerd.cluster.local"},"ports":[{"port":8080,"protocol":"TCP"},{"port":7777,"protocol":"TCP"}],"workloadIPs":[{"ip":"192.0.2.2"}]},"status":{"conditions":[{"status":"True","type":"Ready"}]}}
  creationTimestamp: "2023-12-20T14:48:22Z"
  generation: 1
  labels:
    app: echo-service
  name: nginx-vm-test-3
  namespace: default
  resourceVersion: "31812"
  uid: 21a22e4e-36c7-4315-973c-90cc39ad83af
spec:
  meshTls:
    identity: foo.default.serviceaccount.linkerd.cluster.local
    serverName: foo.default.serviceaccount.linkerd.cluster.local
  ports:
  - port: 8080
    protocol: TCP
  - port: 7777
    protocol: TCP
  workloadIPs:
  - ip: 192.0.2.2
status:
  conditions:
  - status: "True"
    type: Ready

```
<!--  Thanks for sending a pull request!

If you already have a well-structured git commit message, chances are GitHub
set the title and description of this PR to the git commit message subject and
body, respectively. If so, you may delete these instructions and submit your PR.

If this is your first time, please read our contributor guide:
https://github.com/linkerd/linkerd2/blob/main/CONTRIBUTING.md

The title and description of your Pull Request should match the git commit
subject and body, respectively. Git commit messages are structured as follows:

```
Subject

Problem

Solution

Validation

Fixes #[GitHub issue ID]

DCO Sign off
```

Example git commit message:

```
Introduce Pull Request Template

GitHub's community guidelines recommend a pull request template, the repo was
lacking one.

Introduce a `PULL_REQUEST_TEMPLATE.md` file.

Once merged, the
[Community profile checklist](https://github.com/linkerd/linkerd2/community)
should indicate the repo now provides a pull request template.

Fixes #3321

Signed-off-by: Jane Smith <jane.smith@example.com>
```

Note the git commit message subject becomes the pull request title.

For more details around git commits, see the section on Committing in our
contributor guide:
https://github.com/linkerd/linkerd2/blob/main/CONTRIBUTING.md#committing
-->



Patch:
diff --git a/charts/linkerd-crds/templates/workload/external-workload.yaml b/charts/linkerd-crds/templates/workload/external-workload.yaml
new file mode 100644
index 000000000..056d6a275
--- /dev/null
+++ b/charts/linkerd-crds/templates/workload/external-workload.yaml
@@ -0,0 +1,157 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: externalworkloads.workload.linkerd.io
+  annotations:
+    {{ include "partials.annotations.created-by" . }}
+  labels:
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
+    linkerd.io/control-plane-ns: {{.Release.Namespace}}
+spec:
+  group: workload.linkerd.io
+  names:
+    categories:
+    - external
+    kind: ExternalWorkload
+    listKind: ExternalWorkloadList
+    plural: externalworkloads
+    singular: externalworkload
+    shortNames: []
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.meshTls.identity
+      name: Identity
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    served: true
+    storage: true
+    schema:
+      openAPIV3Schema:
+        description: >-
+          An ExternalWorkload describes a single workload (i.e. a deployable unit) external
+          to the cluster that should be enrolled in the mesh.
+        type: object
+        required: [spec]
+        properties:
+          apiVerson:
+            type: string
+          kind:
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              meshTls:
+                description: meshTls describes TLS settings associated with an
+                  external workload.
+                properties:
+                  identity:
+                    type: string
+                    description: identity of the workload. Corresponds to the
+                      identity used in the workload's certificate. It is used
+                      by peers to perform verification in the mTLS handshake.
+                    minLength: 1
+                    maxLength: 253
+                  serverName:
+                    type: string
+                    description: serverName is the name of the workload in DNS
+                      format. It is used by the workload to terminate TLS using
+                      SNI.
+                    minLength: 1
+                    maxLength: 253
+                type: object
+              ports:
+                type: array
+                description: ports describes a list of ports exposed by the
+                  workload
+                items:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: name must be an IANA_SVC_NAME and unique
+                        within the ports set. Each named port can be referred
+                        to by services.
+                    port:
+                      format: int32
+                      maximum: 65535
+                      minimum: 1
+                      type: integer
+                    protocol:
+                      description: protocol exposed by the port. Must be UDP or
+                        TCP. Defaults to TCP.
+                      type: string
+                      default: "TCP"
+              workloadIPs:
+                type: array
+                description: workloadIPs contains a list of IP addresses that
+                  can be used to send traffic to the workload.
+                items:
+                  type: object
+                  properties:
+                    ip:
+                      type: string
+            type: object
+            required:
+            - ports
+            - workloadIPs
+          status:
+            type: object
+            properties:
+              conditions:
+                type: array
+                items:
+                  type: object
+                  properties:
+                    lastProbeTime:
+                      description: lastProbeTime is the last time the
+                        healthcheck endpoint was probed.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: lastTransitionTime is the last time the
+                        condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    status:
+                      description: status of the condition (one of True, False, Unknown)
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of the condition in CamelCase or in
+                        foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                    reason:
+                      description: reason contains a programmatic identifier
+                        indicating the reason for the condition's last
+                        transition. Producers of specific condition types may
+                        define expected values and meanings for this field, and
+                        whether the values are considered a guaranteed API. The
+                        value should be a CamelCase string. This field may not
+                        be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    message:
+                      description: message is a human readable message
+                        indicating details about the transition. This may be an
+                        empty string.
+                      maxLength: 32768
+                      type: string
+                required:
+                - lastTransitionTime
+                - status
+                - type
+                - reason
+                - message
diff --git a/cli/cmd/install.go b/cli/cmd/install.go
index 0c0e4a881..90e46b636 100644
--- a/cli/cmd/install.go
+++ b/cli/cmd/install.go
@@ -55,6 +55,7 @@ var (
 		"templates/policy/server.yaml",
 		"templates/serviceprofile.yaml",
 		"templates/gateway.networking.k8s.io_httproutes.yaml",
+		"templates/workload/external-workload.yaml",
 	}
 
 	templatesControlPlane = []string{


Test patch:
diff --git a/cli/cmd/testdata/install_crds.golden b/cli/cmd/testdata/install_crds.golden
index 62baf3bbd..abaf6946d 100644
--- a/cli/cmd/testdata/install_crds.golden
+++ b/cli/cmd/testdata/install_crds.golden
@@ -10137,3 +10137,160 @@ status:
     plural: ""
   conditions: null
   storedVersions: null
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: externalworkloads.workload.linkerd.io
+  annotations:
+    linkerd.io/created-by: linkerd/cli dev-undefined
+  labels:
+    helm.sh/chart: linkerd-crds-1.9.1-edge
+    linkerd.io/control-plane-ns: linkerd
+spec:
+  group: workload.linkerd.io
+  names:
+    categories:
+    - external
+    kind: ExternalWorkload
+    listKind: ExternalWorkloadList
+    plural: externalworkloads
+    singular: externalworkload
+    shortNames: []
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.meshTls.identity
+      name: Identity
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    served: true
+    storage: true
+    schema:
+      openAPIV3Schema:
+        description: >-
+          An ExternalWorkload describes a single workload (i.e. a deployable unit) external
+          to the cluster that should be enrolled in the mesh.
+        type: object
+        required: [spec]
+        properties:
+          apiVerson:
+            type: string
+          kind:
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              meshTls:
+                description: meshTls describes TLS settings associated with an
+                  external workload.
+                properties:
+                  identity:
+                    type: string
+                    description: identity of the workload. Corresponds to the
+                      identity used in the workload's certificate. It is used
+                      by peers to perform verification in the mTLS handshake.
+                    minLength: 1
+                    maxLength: 253
+                  serverName:
+                    type: string
+                    description: serverName is the name of the workload in DNS
+                      format. It is used by the workload to terminate TLS using
+                      SNI.
+                    minLength: 1
+                    maxLength: 253
+                type: object
+              ports:
+                type: array
+                description: ports describes a list of ports exposed by the
+                  workload
+                items:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: name must be an IANA_SVC_NAME and unique
+                        within the ports set. Each named port can be referred
+                        to by services.
+                    port:
+                      format: int32
+                      maximum: 65535
+                      minimum: 1
+                      type: integer
+                    protocol:
+                      description: protocol exposed by the port. Must be UDP or
+                        TCP. Defaults to TCP.
+                      type: string
+                      default: "TCP"
+              workloadIPs:
+                type: array
+                description: workloadIPs contains a list of IP addresses that
+                  can be used to send traffic to the workload.
+                items:
+                  type: object
+                  properties:
+                    ip:
+                      type: string
+            type: object
+            required:
+            - ports
+            - workloadIPs
+          status:
+            type: object
+            properties:
+              conditions:
+                type: array
+                items:
+                  type: object
+                  properties:
+                    lastProbeTime:
+                      description: lastProbeTime is the last time the
+                        healthcheck endpoint was probed.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: lastTransitionTime is the last time the
+                        condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    status:
+                      description: status of the condition (one of True, False, Unknown)
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of the condition in CamelCase or in
+                        foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                    reason:
+                      description: reason contains a programmatic identifier
+                        indicating the reason for the condition's last
+                        transition. Producers of specific condition types may
+                        define expected values and meanings for this field, and
+                        whether the values are considered a guaranteed API. The
+                        value should be a CamelCase string. This field may not
+                        be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    message:
+                      description: message is a human readable message
+                        indicating details about the transition. This may be an
+                        empty string.
+                      maxLength: 32768
+                      type: string
+                required:
+                - lastTransitionTime
+                - status
+                - type
+                - reason
+                - message
diff --git a/cli/cmd/testdata/install_helm_crds_output.golden b/cli/cmd/testdata/install_helm_crds_output.golden
index a106d8844..b010d4d93 100644
--- a/cli/cmd/testdata/install_helm_crds_output.golden
+++ b/cli/cmd/testdata/install_helm_crds_output.golden
@@ -10153,3 +10153,162 @@ status:
     plural: ""
   conditions: null
   storedVersions: null
+---
+# Source: linkerd-crds/templates/workload/external-workload.yaml
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: externalworkloads.workload.linkerd.io
+  annotations:
+    linkerd.io/created-by: linkerd/helm linkerd-version
+  labels:
+    helm.sh/chart: linkerd-crds-
+    linkerd.io/control-plane-ns: linkerd-dev
+spec:
+  group: workload.linkerd.io
+  names:
+    categories:
+    - external
+    kind: ExternalWorkload
+    listKind: ExternalWorkloadList
+    plural: externalworkloads
+    singular: externalworkload
+    shortNames: []
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.meshTls.identity
+      name: Identity
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    served: true
+    storage: true
+    schema:
+      openAPIV3Schema:
+        description: >-
+          An ExternalWorkload describes a single workload (i.e. a deployable unit) external
+          to the cluster that should be enrolled in the mesh.
+        type: object
+        required: [spec]
+        properties:
+          apiVerson:
+            type: string
+          kind:
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              meshTls:
+                description: meshTls describes TLS settings associated with an
+                  external workload.
+                properties:
+                  identity:
+                    type: string
+                    description: identity of the workload. Corresponds to the
+                      identity used in the workload's certificate. It is used
+                      by peers to perform verification in the mTLS handshake.
+                    minLength: 1
+                    maxLength: 253
+                  serverName:
+                    type: string
+                    description: serverName is the name of the workload in DNS
+                      format. It is used by the workload to terminate TLS using
+                      SNI.
+                    minLength: 1
+                    maxLength: 253
+                type: object
+              ports:
+                type: array
+                description: ports describes a list of ports exposed by the
+                  workload
+                items:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: name must be an IANA_SVC_NAME and unique
+                        within the ports set. Each named port can be referred
+                        to by services.
+                    port:
+                      format: int32
+                      maximum: 65535
+                      minimum: 1
+                      type: integer
+                    protocol:
+                      description: protocol exposed by the port. Must be UDP or
+                        TCP. Defaults to TCP.
+                      type: string
+                      default: "TCP"
+              workloadIPs:
+                type: array
+                description: workloadIPs contains a list of IP addresses that
+                  can be used to send traffic to the workload.
+                items:
+                  type: object
+                  properties:
+                    ip:
+                      type: string
+            type: object
+            required:
+            - ports
+            - workloadIPs
+          status:
+            type: object
+            properties:
+              conditions:
+                type: array
+                items:
+                  type: object
+                  properties:
+                    lastProbeTime:
+                      description: lastProbeTime is the last time the
+                        healthcheck endpoint was probed.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: lastTransitionTime is the last time the
+                        condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    status:
+                      description: status of the condition (one of True, False, Unknown)
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of the condition in CamelCase or in
+                        foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                    reason:
+                      description: reason contains a programmatic identifier
+                        indicating the reason for the condition's last
+                        transition. Producers of specific condition types may
+                        define expected values and meanings for this field, and
+                        whether the values are considered a guaranteed API. The
+                        value should be a CamelCase string. This field may not
+                        be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    message:
+                      description: message is a human readable message
+                        indicating details about the transition. This may be an
+                        empty string.
+                      maxLength: 32768
+                      type: string
+                required:
+                - lastTransitionTime
+                - status
+                - type
+                - reason
+                - message
diff --git a/cli/cmd/testdata/install_helm_crds_output_ha.golden b/cli/cmd/testdata/install_helm_crds_output_ha.golden
index a106d8844..b010d4d93 100644
--- a/cli/cmd/testdata/install_helm_crds_output_ha.golden
+++ b/cli/cmd/testdata/install_helm_crds_output_ha.golden
@@ -10153,3 +10153,162 @@ status:
     plural: ""
   conditions: null
   storedVersions: null
+---
+# Source: linkerd-crds/templates/workload/external-workload.yaml
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: externalworkloads.workload.linkerd.io
+  annotations:
+    linkerd.io/created-by: linkerd/helm linkerd-version
+  labels:
+    helm.sh/chart: linkerd-crds-
+    linkerd.io/control-plane-ns: linkerd-dev
+spec:
+  group: workload.linkerd.io
+  names:
+    categories:
+    - external
+    kind: ExternalWorkload
+    listKind: ExternalWorkloadList
+    plural: externalworkloads
+    singular: externalworkload
+    shortNames: []
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.meshTls.identity
+      name: Identity
+      type: string
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    served: true
+    storage: true
+    schema:
+      openAPIV3Schema:
+        description: >-
+          An ExternalWorkload describes a single workload (i.e. a deployable unit) external
+          to the cluster that should be enrolled in the mesh.
+        type: object
+        required: [spec]
+        properties:
+          apiVerson:
+            type: string
+          kind:
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              meshTls:
+                description: meshTls describes TLS settings associated with an
+                  external workload.
+                properties:
+                  identity:
+                    type: string
+                    description: identity of the workload. Corresponds to the
+                      identity used in the workload's certificate. It is used
+                      by peers to perform verification in the mTLS handshake.
+                    minLength: 1
+                    maxLength: 253
+                  serverName:
+                    type: string
+                    description: serverName is the name of the workload in DNS
+                      format. It is used by the workload to terminate TLS using
+                      SNI.
+                    minLength: 1
+                    maxLength: 253
+                type: object
+              ports:
+                type: array
+                description: ports describes a list of ports exposed by the
+                  workload
+                items:
+                  type: object
+                  properties:
+                    name:
+                      type: string
+                      description: name must be an IANA_SVC_NAME and unique
+                        within the ports set. Each named port can be referred
+                        to by services.
+                    port:
+                      format: int32
+                      maximum: 65535
+                      minimum: 1
+                      type: integer
+                    protocol:
+                      description: protocol exposed by the port. Must be UDP or
+                        TCP. Defaults to TCP.
+                      type: string
+                      default: "TCP"
+              workloadIPs:
+                type: array
+                description: workloadIPs contains a list of IP addresses that
+                  can be used to send traffic to the workload.
+                items:
+                  type: object
+                  properties:
+                    ip:
+                      type: string
+            type: object
+            required:
+            - ports
+            - workloadIPs
+          status:
+            type: object
+            properties:
+              conditions:
+                type: array
+                items:
+                  type: object
+                  properties:
+                    lastProbeTime:
+                      description: lastProbeTime is the last time the
+                        healthcheck endpoint was probed.
+                      format: date-time
+                      type: string
+                    lastTransitionTime:
+                      description: lastTransitionTime is the last time the
+                        condition transitioned from one status to another.
+                      format: date-time
+                      type: string
+                    status:
+                      description: status of the condition (one of True, False, Unknown)
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of the condition in CamelCase or in
+                        foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                    reason:
+                      description: reason contains a programmatic identifier
+                        indicating the reason for the condition's last
+                        transition. Producers of specific condition types may
+                        define expected values and meanings for this field, and
+                        whether the values are considered a guaranteed API. The
+                        value should be a CamelCase string. This field may not
+                        be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    message:
+                      description: message is a human readable message
+                        indicating details about the transition. This may be an
+                        empty string.
+                      maxLength: 32768
+                      type: string
+                required:
+                - lastTransitionTime
+                - status
+                - type
+                - reason
+                - message