vSphere with TanzuのWorkload Cluster上にHarborを立てる

vSphere with TanzuのWorkload Cluster上にHarborを立てる方法がTKG1.6ベースの方法にupdateされていたので、それを試した時のメモ。

[こちら](https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-00A2BB49-DBDE-4E2B-B9EE-38C36E261185.html?hWord=N4IghgNiBcIA5gMYGswHMCmIC+Q)によると、’22/10/31時点ではTKG1.6のパッケージを使えば良いことになっているので、リンク先の[こちらの手順](
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.6/vmware-tanzu-kubernetes-grid-16/GUID-packages-prep-tkgs-kapp.html)に従う。

## 前提条件
以下が前提条件となっている。
  - vSphere7.0u2以上のvSphere with Tanzuでworkload clusterを構築済み
  - kubectlとtanzu-cli(v1.4以上)をインストール済み

また、今回は`type: LoadBalancer`を使うためにNSX Advanced Load Balancerも構築済みとしている。（MetalLBなどでもOK）

## kapp-controllerのインストール

tanzu cliにManagement Clusterを追加する。
```
export VCENTER_IP=xxxxx
export SUPERVISOR_IP=xxxxx
export KUBECTL_VSPHERE_PASSWORD='xxxxx'
export SUPERVISOR_NAME=xxx
kubectl vsphere login --vsphere-username=administrator@vsphere.local --server=$SUPERVISOR_IP --insecure-skip-tls-verify
kubectl config use-context $VCENTER_IP
tanzu login --kubeconfig ~/.kube/config --context $SUPERVISOR_IP --name $SUPERVISOR_NAME
```

Workload Clusterに切り替える。
```
export WORKLOAD_CLUSTER_NAME=tkgs-cluster-1
export WORKLOAD_CLUSTER_NAMESPACE=ns1
kubectl vsphere login --vsphere-username=administrator@vsphere.local \
--server=$SUPERVISOR_IP --insecure-skip-tls-verify \
--tanzu-kubernetes-cluster-name=$WORKLOAD_CLUSTER_NAME \
--tanzu-kubernetes-cluster-namespace=$WORKLOAD_CLUSTER_NAMESPACE 
kubectl config use-context $WORKLOAD_CLUSTER_NAME
```

Cluster作成直後だとdefaultのStorageClassが設定されていないので、defaultを設定する。
```
export STORAGE_CLASS=xxxxx
kubectl patch storageclass $STORAGE_CLASS -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```

kapp-controllerのPSPを作成する。
```yaml
cat <<EOF > ./tanzu-system-kapp-ctrl-restricted.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: tanzu-system-kapp-ctrl-restricted
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false
EOF
kubectl apply -f tanzu-system-kapp-ctrl-restricted.yaml
```

kapp-controllerをapplyする。
<details><summary>kapp-controllerのmanifest</summary><div>

```yaml
cat << EOF > ./kapp-controller.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tkg-system
---
apiVersion: v1
kind: Namespace
metadata:
  name: tanzu-package-repo-global
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1alpha1.data.packaging.carvel.dev
spec:
  group: data.packaging.carvel.dev
  groupPriorityMinimum: 100
  service:
    name: packaging-api
    namespace: tkg-system
  version: v1alpha1
  versionPriority: 100
---
apiVersion: v1
kind: Service
metadata:
  name: packaging-api
  namespace: tkg-system
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: api
  selector:
    app: kapp-controller
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: internalpackagemetadatas.internal.packaging.carvel.dev
spec:
  group: internal.packaging.carvel.dev
  names:
    kind: InternalPackageMetadata
    listKind: InternalPackageMetadataList
    plural: internalpackagemetadatas
    singular: internalpackagemetadata
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              categories:
                items:
                  type: string
                type: array
              displayName:
                type: string
              iconSVGBase64:
                type: string
              longDescription:
                type: string
              maintainers:
                items:
                  properties:
                    name:
                      type: string
                  type: object
                type: array
              providerName:
                type: string
              shortDescription:
                type: string
              supportDescription:
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: internalpackages.internal.packaging.carvel.dev
spec:
  group: internal.packaging.carvel.dev
  names:
    kind: InternalPackage
    listKind: InternalPackageList
    plural: internalpackages
    singular: internalpackage
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              capacityRequirementsDescription:
                type: string
              licenses:
                items:
                  type: string
                type: array
              refName:
                type: string
              releaseNotes:
                type: string
              releasedAt:
                format: date-time
                nullable: true
                type: string
              template:
                properties:
                  spec:
                    properties:
                      canceled:
                        description: Canceled when set to true will stop all active changes
                        type: boolean
                      cluster:
                        properties:
                          kubeconfigSecretRef:
                            properties:
                              key:
                                type: string
                              name:
                                type: string
                            type: object
                          namespace:
                            type: string
                        type: object
                      deploy:
                        items:
                          properties:
                            kapp:
                              properties:
                                delete:
                                  properties:
                                    rawOptions:
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                inspect:
                                  properties:
                                    rawOptions:
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                intoNs:
                                  type: string
                                mapNs:
                                  items:
                                    type: string
                                  type: array
                                rawOptions:
                                  items:
                                    type: string
                                  type: array
                              type: object
                          type: object
                        type: array
                      fetch:
                        items:
                          properties:
                            git:
                              description: TODO implement git
                              properties:
                                lfsSkipSmudge:
                                  type: boolean
                                ref:
                                  type: string
                                secretRef:
                                  description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
                                  properties:
                                    name:
                                      description: Object is expected to be within same namespace
                                      type: string
                                  type: object
                                subPath:
                                  type: string
                                url:
                                  type: string
                              type: object
                            helmChart:
                              properties:
                                name:
                                  description: 'Example: stable/redis'
                                  type: string
                                repository:
                                  properties:
                                    secretRef:
                                      properties:
                                        name:
                                          description: Object is expected to be within same namespace
                                          type: string
                                      type: object
                                    url:
                                      type: string
                                  type: object
                                version:
                                  type: string
                              type: object
                            http:
                              properties:
                                secretRef:
                                  description: 'Secret may include one or more keys: username, password'
                                  properties:
                                    name:
                                      description: Object is expected to be within same namespace
                                      type: string
                                  type: object
                                sha256:
                                  type: string
                                subPath:
                                  type: string
                                url:
                                  description: 'URL can point to one of following formats: text, tgz, zip'
                                  type: string
                              type: object
                            image:
                              properties:
                                secretRef:
                                  description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                                  properties:
                                    name:
                                      description: Object is expected to be within same namespace
                                      type: string
                                  type: object
                                subPath:
                                  type: string
                                url:
                                  description: 'Example: username/app1-config:v0.1.0'
                                  type: string
                              type: object
                            imgpkgBundle:
                              properties:
                                image:
                                  type: string
                                secretRef:
                                  description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                                  properties:
                                    name:
                                      description: Object is expected to be within same namespace
                                      type: string
                                  type: object
                              type: object
                            inline:
                              properties:
                                paths:
                                  additionalProperties:
                                    type: string
                                  type: object
                                pathsFrom:
                                  items:
                                    properties:
                                      configMapRef:
                                        properties:
                                          directoryPath:
                                            type: string
                                          name:
                                            type: string
                                        type: object
                                      secretRef:
                                        properties:
                                          directoryPath:
                                            type: string
                                          name:
                                            type: string
                                        type: object
                                    type: object
                                  type: array
                              type: object
                          type: object
                        type: array
                      noopDelete:
                        description: When NoopDeletion set to true, App deletion should delete App CR but preserve App's associated resources
                        type: boolean
                      paused:
                        description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
                        type: boolean
                      serviceAccountName:
                        type: string
                      syncPeriod:
                        description: Controls frequency of app reconciliation
                        type: string
                      template:
                        items:
                          properties:
                            helmTemplate:
                              properties:
                                name:
                                  type: string
                                namespace:
                                  type: string
                                path:
                                  type: string
                                valuesFrom:
                                  items:
                                    properties:
                                      configMapRef:
                                        properties:
                                          name:
                                            type: string
                                        type: object
                                      path:
                                        type: string
                                      secretRef:
                                        properties:
                                          name:
                                            type: string
                                        type: object
                                    type: object
                                  type: array
                              type: object
                            jsonnet:
                              description: TODO implement jsonnet
                              type: object
                            kbld:
                              properties:
                                paths:
                                  items:
                                    type: string
                                  type: array
                              type: object
                            kustomize:
                              description: TODO implement kustomize
                              type: object
                            sops:
                              properties:
                                paths:
                                  items:
                                    type: string
                                  type: array
                                pgp:
                                  properties:
                                    privateKeysSecretRef:
                                      properties:
                                        name:
                                          type: string
                                      type: object
                                  type: object
                              type: object
                            ytt:
                              properties:
                                fileMarks:
                                  items:
                                    type: string
                                  type: array
                                ignoreUnknownComments:
                                  type: boolean
                                inline:
                                  properties:
                                    paths:
                                      additionalProperties:
                                        type: string
                                      type: object
                                    pathsFrom:
                                      items:
                                        properties:
                                          configMapRef:
                                            properties:
                                              directoryPath:
                                                type: string
                                              name:
                                                type: string
                                            type: object
                                          secretRef:
                                            properties:
                                              directoryPath:
                                                type: string
                                              name:
                                                type: string
                                            type: object
                                        type: object
                                      type: array
                                  type: object
                                paths:
                                  items:
                                    type: string
                                  type: array
                                strict:
                                  type: boolean
                                valuesFrom:
                                  items:
                                    properties:
                                      configMapRef:
                                        properties:
                                          name:
                                            type: string
                                        type: object
                                      path:
                                        type: string
                                      secretRef:
                                        properties:
                                          name:
                                            type: string
                                        type: object
                                    type: object
                                  type: array
                              type: object
                          type: object
                        type: array
                    type: object
                required:
                - spec
                type: object
              valuesSchema:
                description: valuesSchema can be used to show template values that can be configured by users when a Package is installed in an OpenAPI schema format.
                properties:
                  openAPIv3:
                    nullable: true
                    type: object
                    x-kubernetes-preserve-unknown-fields: true
                type: object
              version:
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: apps.kappctrl.k14s.io
spec:
  group: kappctrl.k14s.io
  names:
    kind: App
    listKind: AppList
    plural: apps
    singular: app
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Friendly description
      jsonPath: .status.friendlyDescription
      name: Description
      type: string
    - description: Last time app started being deployed. Does not mean anything was changed.
      jsonPath: .status.deploy.startedAt
      name: Since-Deploy
      type: date
    - description: Time since creation
      jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              canceled:
                description: Canceled when set to true will stop all active changes
                type: boolean
              cluster:
                properties:
                  kubeconfigSecretRef:
                    properties:
                      key:
                        type: string
                      name:
                        type: string
                    type: object
                  namespace:
                    type: string
                type: object
              deploy:
                items:
                  properties:
                    kapp:
                      properties:
                        delete:
                          properties:
                            rawOptions:
                              items:
                                type: string
                              type: array
                          type: object
                        inspect:
                          properties:
                            rawOptions:
                              items:
                                type: string
                              type: array
                          type: object
                        intoNs:
                          type: string
                        mapNs:
                          items:
                            type: string
                          type: array
                        rawOptions:
                          items:
                            type: string
                          type: array
                      type: object
                  type: object
                type: array
              fetch:
                items:
                  properties:
                    git:
                      description: TODO implement git
                      properties:
                        lfsSkipSmudge:
                          type: boolean
                        ref:
                          type: string
                        secretRef:
                          description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
                          properties:
                            name:
                              description: Object is expected to be within same namespace
                              type: string
                          type: object
                        subPath:
                          type: string
                        url:
                          type: string
                      type: object
                    helmChart:
                      properties:
                        name:
                          description: 'Example: stable/redis'
                          type: string
                        repository:
                          properties:
                            secretRef:
                              properties:
                                name:
                                  description: Object is expected to be within same namespace
                                  type: string
                              type: object
                            url:
                              type: string
                          type: object
                        version:
                          type: string
                      type: object
                    http:
                      properties:
                        secretRef:
                          description: 'Secret may include one or more keys: username, password'
                          properties:
                            name:
                              description: Object is expected to be within same namespace
                              type: string
                          type: object
                        sha256:
                          type: string
                        subPath:
                          type: string
                        url:
                          description: 'URL can point to one of following formats: text, tgz, zip'
                          type: string
                      type: object
                    image:
                      properties:
                        secretRef:
                          description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                          properties:
                            name:
                              description: Object is expected to be within same namespace
                              type: string
                          type: object
                        subPath:
                          type: string
                        url:
                          description: 'Example: username/app1-config:v0.1.0'
                          type: string
                      type: object
                    imgpkgBundle:
                      properties:
                        image:
                          type: string
                        secretRef:
                          description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                          properties:
                            name:
                              description: Object is expected to be within same namespace
                              type: string
                          type: object
                      type: object
                    inline:
                      properties:
                        paths:
                          additionalProperties:
                            type: string
                          type: object
                        pathsFrom:
                          items:
                            properties:
                              configMapRef:
                                properties:
                                  directoryPath:
                                    type: string
                                  name:
                                    type: string
                                type: object
                              secretRef:
                                properties:
                                  directoryPath:
                                    type: string
                                  name:
                                    type: string
                                type: object
                            type: object
                          type: array
                      type: object
                  type: object
                type: array
              noopDelete:
                description: When NoopDeletion set to true, App deletion should delete App CR but preserve App's associated resources
                type: boolean
              paused:
                description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
                type: boolean
              serviceAccountName:
                type: string
              syncPeriod:
                description: Controls frequency of app reconciliation
                type: string
              template:
                items:
                  properties:
                    helmTemplate:
                      properties:
                        name:
                          type: string
                        namespace:
                          type: string
                        path:
                          type: string
                        valuesFrom:
                          items:
                            properties:
                              configMapRef:
                                properties:
                                  name:
                                    type: string
                                type: object
                              path:
                                type: string
                              secretRef:
                                properties:
                                  name:
                                    type: string
                                type: object
                            type: object
                          type: array
                      type: object
                    jsonnet:
                      description: TODO implement jsonnet
                      type: object
                    kbld:
                      properties:
                        paths:
                          items:
                            type: string
                          type: array
                      type: object
                    kustomize:
                      description: TODO implement kustomize
                      type: object
                    sops:
                      properties:
                        paths:
                          items:
                            type: string
                          type: array
                        pgp:
                          properties:
                            privateKeysSecretRef:
                              properties:
                                name:
                                  type: string
                              type: object
                          type: object
                      type: object
                    ytt:
                      properties:
                        fileMarks:
                          items:
                            type: string
                          type: array
                        ignoreUnknownComments:
                          type: boolean
                        inline:
                          properties:
                            paths:
                              additionalProperties:
                                type: string
                              type: object
                            pathsFrom:
                              items:
                                properties:
                                  configMapRef:
                                    properties:
                                      directoryPath:
                                        type: string
                                      name:
                                        type: string
                                    type: object
                                  secretRef:
                                    properties:
                                      directoryPath:
                                        type: string
                                      name:
                                        type: string
                                    type: object
                                type: object
                              type: array
                          type: object
                        paths:
                          items:
                            type: string
                          type: array
                        strict:
                          type: boolean
                        valuesFrom:
                          items:
                            properties:
                              configMapRef:
                                properties:
                                  name:
                                    type: string
                                type: object
                              path:
                                type: string
                              secretRef:
                                properties:
                                  name:
                                    type: string
                                type: object
                            type: object
                          type: array
                      type: object
                  type: object
                type: array
            type: object
          status:
            properties:
              conditions:
                items:
                  description: TODO rename to Condition
                  properties:
                    message:
                      description: Human-readable message indicating details about last transition.
                      type: string
                    reason:
                      description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
                      type: string
                    status:
                      type: string
                    type:
                      type: string
                  required:
                  - status
                  - type
                  type: object
                type: array
              consecutiveReconcileFailures:
                type: integer
              consecutiveReconcileSuccesses:
                type: integer
              deploy:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  finished:
                    type: boolean
                  startedAt:
                    format: date-time
                    type: string
                  stderr:
                    type: string
                  stdout:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              fetch:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  startedAt:
                    format: date-time
                    type: string
                  stderr:
                    type: string
                  stdout:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              friendlyDescription:
                type: string
              inspect:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  stderr:
                    type: string
                  stdout:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              managedAppName:
                type: string
              observedGeneration:
                format: int64
                type: integer
              template:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  stderr:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              usefulErrorMessage:
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: packageinstalls.packaging.carvel.dev
spec:
  group: packaging.carvel.dev
  names:
    kind: PackageInstall
    listKind: PackageInstallList
    plural: packageinstalls
    shortNames:
    - pkgi
    singular: packageinstall
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: PackageMetadata name
      jsonPath: .spec.packageRef.refName
      name: Package name
      type: string
    - description: PackageMetadata version
      jsonPath: .status.version
      name: Package version
      type: string
    - description: Friendly description
      jsonPath: .status.friendlyDescription
      name: Description
      type: string
    - description: Time since creation
      jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              canceled:
                description: Canceled when set to true will stop all active changes
                type: boolean
              cluster:
                properties:
                  kubeconfigSecretRef:
                    properties:
                      key:
                        type: string
                      name:
                        type: string
                    type: object
                  namespace:
                    type: string
                type: object
              noopDelete:
                description: When NoopDelete set to true, PackageInstall deletion should delete PackageInstall/App CR but preserve App's associated resources.
                type: boolean
              packageRef:
                properties:
                  refName:
                    type: string
                  versionSelection:
                    properties:
                      constraints:
                        type: string
                      prereleases:
                        properties:
                          identifiers:
                            items:
                              type: string
                            type: array
                        type: object
                    type: object
                type: object
              paused:
                description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
                type: boolean
              serviceAccountName:
                type: string
              syncPeriod:
                description: Controls frequency of App reconciliation in time + unit format. Always >= 30s. If value below 30s is specified, 30s will be used.
                type: string
              values:
                items:
                  properties:
                    secretRef:
                      properties:
                        key:
                          type: string
                        name:
                          type: string
                      type: object
                  type: object
                type: array
            type: object
          status:
            properties:
              conditions:
                items:
                  description: TODO rename to Condition
                  properties:
                    message:
                      description: Human-readable message indicating details about last transition.
                      type: string
                    reason:
                      description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
                      type: string
                    status:
                      type: string
                    type:
                      type: string
                  required:
                  - status
                  - type
                  type: object
                type: array
              friendlyDescription:
                type: string
              observedGeneration:
                format: int64
                type: integer
              usefulErrorMessage:
                type: string
              version:
                description: TODO this is desired resolved version (not actually deployed)
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    packaging.carvel.dev/global-namespace: tanzu-package-repo-global
  name: packagerepositories.packaging.carvel.dev
spec:
  group: packaging.carvel.dev
  names:
    kind: PackageRepository
    listKind: PackageRepositoryList
    plural: packagerepositories
    shortNames:
    - pkgr
    singular: packagerepository
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Time since creation
      jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    - description: Friendly description
      jsonPath: .status.friendlyDescription
      name: Description
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            properties:
              fetch:
                properties:
                  git:
                    description: TODO implement git
                    properties:
                      lfsSkipSmudge:
                        type: boolean
                      ref:
                        type: string
                      secretRef:
                        description: 'Secret may include one or more keys: ssh-privatekey, ssh-knownhosts'
                        properties:
                          name:
                            description: Object is expected to be within same namespace
                            type: string
                        type: object
                      subPath:
                        type: string
                      url:
                        type: string
                    type: object
                  http:
                    properties:
                      secretRef:
                        description: 'Secret may include one or more keys: username, password'
                        properties:
                          name:
                            description: Object is expected to be within same namespace
                            type: string
                        type: object
                      sha256:
                        type: string
                      subPath:
                        type: string
                      url:
                        description: 'URL can point to one of following formats: text, tgz, zip'
                        type: string
                    type: object
                  image:
                    properties:
                      secretRef:
                        description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                        properties:
                          name:
                            description: Object is expected to be within same namespace
                            type: string
                        type: object
                      subPath:
                        type: string
                      url:
                        description: 'Example: username/app1-config:v0.1.0'
                        type: string
                    type: object
                  imgpkgBundle:
                    properties:
                      image:
                        type: string
                      secretRef:
                        description: 'Secret may include one or more keys: username, password, token. By default anonymous access is used for authentication. TODO support docker config formated secret'
                        properties:
                          name:
                            description: Object is expected to be within same namespace
                            type: string
                        type: object
                    type: object
                type: object
              paused:
                description: Paused when set to true will ignore all pending changes, once it set back to false, pending changes will be applied
                type: boolean
              syncPeriod:
                description: Controls frequency of PackageRepository reconciliation
                type: string
            required:
            - fetch
            type: object
          status:
            properties:
              conditions:
                items:
                  description: TODO rename to Condition
                  properties:
                    message:
                      description: Human-readable message indicating details about last transition.
                      type: string
                    reason:
                      description: Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
                      type: string
                    status:
                      type: string
                    type:
                      type: string
                  required:
                  - status
                  - type
                  type: object
                type: array
              consecutiveReconcileFailures:
                type: integer
              consecutiveReconcileSuccesses:
                type: integer
              deploy:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  finished:
                    type: boolean
                  startedAt:
                    format: date-time
                    type: string
                  stderr:
                    type: string
                  stdout:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              fetch:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  startedAt:
                    format: date-time
                    type: string
                  stderr:
                    type: string
                  stdout:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              friendlyDescription:
                type: string
              observedGeneration:
                format: int64
                type: integer
              template:
                properties:
                  error:
                    type: string
                  exitCode:
                    type: integer
                  stderr:
                    type: string
                  updatedAt:
                    format: date-time
                    type: string
                type: object
              usefulErrorMessage:
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kapp-controller-config
  namespace: tkg-system
  annotations:
    kapp.k14s.io/change-group: apps.kappctrl.k14s.io/kapp-controller-config
data:
  caCerts: ""
  httpProxy: ""
  httpsProxy: ""
  noProxy: ""
  dangerousSkipTLSVerify: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kapp-controller.carvel.dev/version: v0.30.0
    kapp.k14s.io/change-rule: upsert after upserting apps.kappctrl.k14s.io/kapp-controller-config
  name: kapp-controller
  namespace: tkg-system
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: kapp-controller
  template:
    metadata:
      labels:
        app: kapp-controller
    spec:
      containers:
      - args:
        - -packaging-global-namespace=tanzu-package-repo-global
        - -concurrency=4
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        - name: KAPPCTRL_SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KAPPCTRL_API_PORT
          value: "10350"
        image: projects.registry.vmware.com/tkg/kapp-controller:v0.30.0_vmware.1
        name: kapp-controller
        ports:
        - containerPort: 10350
          name: api
          protocol: TCP
        resources:
          requests:
            cpu: 120m
            memory: 100Mi
        securityContext:
          runAsGroup: 2000
          runAsUser: 1000
        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
      securityContext:
        fsGroup: 3000
      serviceAccount: kapp-controller-sa
      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs
      priorityClassName: system-cluster-critical
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        key: node.kubernetes.io/not-ready
      - effect: NoSchedule
        key: node.cloudprovider.kubernetes.io/uninitialized
        value: "true"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kapp-controller-sa
  namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kapp-controller-cluster-role
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
- apiGroups:
  - kappctrl.k14s.io
  resources:
  - apps
  - apps/status
  verbs:
  - '*'
- apiGroups:
  - packaging.carvel.dev
  resources:
  - packageinstalls
  - packageinstalls/status
  verbs:
  - '*'
- apiGroups:
  - packaging.carvel.dev
  resources:
  - packagerepositories
  - packagerepositories/status
  verbs:
  - '*'
- apiGroups:
  - internal.packaging.carvel.dev
  resources:
  - internalpackagemetadatas
  verbs:
  - '*'
- apiGroups:
  - data.packaging.carvel.dev
  resources:
  - packagemetadatas
  - packagemetadatas/status
  verbs:
  - '*'
- apiGroups:
  - internal.packaging.carvel.dev
  resources:
  - internalpackages
  verbs:
  - '*'
- apiGroups:
  - data.packaging.carvel.dev
  resources:
  - packages
  - packages/status
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - '*'
- apiGroups:
  - apiregistration.k8s.io
  resources:
  - apiservices
  verbs:
  - update
  - get
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
  - watch
  - get
  - update
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  resourceNames:
  - tanzu-system-kapp-ctrl-restricted
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kapp-controller-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kapp-controller-cluster-role
subjects:
- kind: ServiceAccount
  name: kapp-controller-sa
  namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pkg-apiserver:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: kapp-controller-sa
  namespace: tkg-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pkgserver-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: kapp-controller-sa
  namespace: tkg-system
EOF
```
</div></details>

```
kubectl apply -f kapp-controller.yaml
```

## PackageRepositoryの導入
Harborは`tanzu package`コマンドでインストールしていくため、packageの配布元となるPackageRepositoryを導入する。名前は適当で問題ない。
```
export REPOSITORY_NAME=tanzu-repo
tanzu package repository add $REPOSITORY_NAME -n tanzu-package-repo-global --url projects.registry.vmware.com/tkg/packages/standard/repo:v1.6.0
```
これでHarborを展開できる状態になっている。
```
$ tanzu package available list
　：（省略）
  harbor.tanzu.vmware.com                       harbor                     OCI Registry                                                                      2.5.3+vmware.1-tkg.1
```

## cert-managerのインストール
次にHarborが利用する証明書の管理のためにcert managerをインストールする。\
最初に利用可能なバージョンを取得する。
```
tanzu package available list cert-manager.tanzu.vmware.com -A
```

利用可能なバージョンを指定してインストールする。
```
export NAMESPACE=my-packages
export CERT_MANAGER_VER='1.7.2+vmware.1-tkg.1'
tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace $NAMESPACE --version $CERT_MANAGER_VER --create-namespace
```

## Contourのインストール
HarborにはIngress経由でアクセスしたいため、Ingress Controllerもインストールする。\
cert-managerと同じようにバージョンを調べる。
```
tanzu package available list contour.tanzu.vmware.com -A
export CONTOUR_VER=1.20.2+vmware.1-tkg.1
```
Contourの設定ファイルを抽出する手順がマニュアルには書かれているが、その下にあるManifestをコピーした方が早いので、こちらをコピーして利用する。ただし、今回はLoadBalancerを用意しているので、typeを`LoadBalancer`に変更する。
```yaml
cat << EOF > ./contour-default-values.yaml
---
infrastructure_provider: vsphere
namespace: tanzu-system-ingress
contour:
 configFileContents: {}
 useProxyProtocol: false
 replicas: 2
 pspNames: "vmware-system-restricted"
 logLevel: info
envoy:
 service:
   type: LoadBalancer
   annotations: {}
   nodePorts:
     http: null
     https: null
   externalTrafficPolicy: Cluster
   disableWait: false
 hostPorts:
   enable: true
   http: 80
   https: 443
 hostNetwork: false
 terminationGracePeriodSeconds: 300
 logLevel: info
 pspNames: null
certificates:
 duration: 8760h
 renewBefore: 360h
EOF
```
hostPortsを将来的に使いそうなので、以下のPSP設定も入れる。
```
kubectl create clusterrolebinding envoy-tkg-admin-privileged-binding --clusterrole=psp:vmware-system-privileged --serviceaccount=tanzu-system-ingress:envoy
```

Contourをインストールする。
```
tanzu package install contour \
--package-name contour.tanzu.vmware.com \
--version $CONTOUR_VER \
--values-file contour-default-values.yaml \
--namespace $NAMESPACE
```

<!-- ```
tanzu package available get contour.tanzu.vmware.com/$CONTOUR_VER --generate-default-values-file
``` -->
## Harborのインストール
いつものごとくバージョンを取得する。
```
tanzu package available list harbor.tanzu.vmware.com -A
export HARBOR_VER=2.5.3+vmware.1-tkg.1
```
設定ファイルに関しては、生成せずに[ドキュメント](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.6/vmware-tanzu-kubernetes-grid-16/GUID-packages-harbor-registry.html#deploy-harbor-into-a-cluster-5)に記載があるものを使う。なお、そのままコピーすると先頭に2つスペースが入っているので、多分取り除いて使った方がいい。（自分はそうした）
<details><summary>Harborの設定ファイル(harbor-default-values.yaml)</summary><div>

```yaml
cat <<'EOF' > ./harbor-default-values.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True

---
#! The namespace to install Harbor
namespace: tanzu-system-registry

#! The FQDN for accessing Harbor admin UI and Registry service.
hostname: harbor.yourdomain.com
#! The network port of the Envoy service in Contour or other Ingress Controller.
port:
  https: 443

#! The log level of core, exporter, jobservice, registry. Its value is debug, info, warning, error or fatal.
logLevel: info

#! [Optional] The certificate for harbor FQDN if you want to use your own TLS certificate.
#! We will issue the certificate by cert-manager when it's empty and the tlsCertificateSecretName is empty.
tlsCertificate:
  #! [Required] the certificate
  tls.crt:
  #! [Required] the private key
  tls.key:
  #! [Optional] the certificate of CA, this enables the download
  #! link on portal to download the certificate of CA
  ca.crt:

#! [Optional] The name of the secret for harbor FQDN if you want to use your own TLS certificate,
#! which contains keys named:
#! "tls.crt" - the certificate
#! "tls.key" - the private key
#! We will issue the certificate by cert-manager when it's empty and the tlsCertificate is empty.
tlsCertificateSecretName:

#! Use contour http proxy instead of the ingress when it's true
enableContourHttpProxy: true

#! [Required] The initial password of Harbor admin.
harborAdminPassword:

#! [Required] The secret key used for encryption. Must be a string of 16 chars.
secretKey:

database:
  #! [Required] The initial password of the postgres database.
  password:
  shmSizeLimit:
  maxIdleConns:
  maxOpenConns:

exporter:
  cacheDuration:

core:
  replicas: 1
  #! [Required] Secret is used when core server communicates with other components.
  secret:
  #! [Required] The XSRF key. Must be a string of 32 chars.
  xsrfKey:
jobservice:
  replicas: 1
  #! [Required] Secret is used when job service communicates with other components.
  secret:
registry:
  replicas: 1
  #! [Required] Secret is used to secure the upload state from client
  #! and registry storage backend.
  #! See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
  secret:
notary:
  #! Whether to install Notary
  enabled: true
trivy:
  #! enabled the flag to enable Trivy scanner
  enabled: true
  replicas: 1
  #! gitHubToken the GitHub access token to download Trivy DB
  gitHubToken: ""
  #! skipUpdate the flag to disable Trivy DB downloads from GitHub
  #
  #! You might want to set the value of this flag to `true` in test or CI/CD environments to avoid GitHub rate limiting issues.
  #! If the value is set to `true` you have to manually download the `trivy.db` file and mount it in the
  #! `/home/scanner/.cache/trivy/db/trivy.db` path.
  skipUpdate: false

#! The persistence is always enabled and a default StorageClass
#! is needed in the k8s cluster to provision volumes dynamically.
#! Specify another StorageClass in the "storageClass" or set "existingClaim"
#! if you have already existing persistent volumes to use
#
#! For storing images and charts, you can also use "azure", "gcs", "s3",
#! "swift" or "oss". Set it in the "imageChartStorage" section
persistence:
  persistentVolumeClaim:
    registry:
      #! Use the existing PVC which must be created manually before bound,
      #! and specify the "subPath" if the PVC is shared with other components
      existingClaim: ""
      #! Specify the "storageClass" used to provision the volume. Or the default
      #! StorageClass will be used(the default).
      #! Set it to "-" to disable dynamic provisioning
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 10Gi
    jobservice:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    database:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    redis:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    trivy:
      existingClaim: ""
      storageClass: ""
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
  #! Define which storage backend is used for registry to store
  #! images and charts. Refer to
  #! https://github.com/docker/distribution/blob/master/docs/configuration.md#storage
  #! for the detail.
  imageChartStorage:
    #! Specify whether to disable `redirect` for images and chart storage, for
    #! backends which not supported it (such as using minio for `s3` storage type), please disable
    #! it. To disable redirects, simply set `disableredirect` to `true` instead.
    #! Refer to
    #! https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect
    #! for the detail.
    disableredirect: false
    #! Specify the "caBundleSecretName" if the storage service uses a self-signed certificate.
    #! The secret must contain keys named "ca.crt" which will be injected into the trust store
    #! of registrys containers.
    #! caBundleSecretName:

    #! Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",
    #! "oss" and fill the information needed in the corresponding section. The type
    #! must be "filesystem" if you want to use persistent volumes for registry
    type: filesystem
    filesystem:
      rootdirectory: /storage
      #maxthreads: 100
    azure:
      accountname: accountname #! required
      accountkey: base64encodedaccountkey #! required
      container: containername #! required
      realm: core.windows.net #! optional
    gcs:
      bucket: bucketname #! required
      #! The base64 encoded json file which contains the key
      encodedkey: base64-encoded-json-key-file #! optional
      rootdirectory: null #! optional
      chunksize: 5242880 #! optional
    s3:
      region: us-west-1 #! required
      bucket: bucketname #! required
      accesskey: null #! eg, awsaccesskey
      secretkey: null #! eg, awssecretkey
      regionendpoint: null #! optional, eg, http://myobjects.local
      encrypt: false #! optional
      keyid: null #! eg, mykeyid
      secure: true #! optional
      skipverify: false #! optional
      v4auth: true #! optional
      chunksize: null #! optional
      rootdirectory: null #! optional
      storageclass: STANDARD #! optional
      multipartcopychunksize: null #! optional
      multipartcopymaxconcurrency: null #! optional
      multipartcopythresholdsize: null #! optional
    swift:
      authurl: https://storage.myprovider.com/v3/auth
      username: username
      password: password
      container: containername
      region: null #! eg, fr
      tenant: null #! eg, tenantname
      tenantid: null #! eg, tenantid
      domain: null #! eg, domainname
      domainid: null #! eg, domainid
      trustid: null #! eg, trustid
      insecureskipverify: null #! bool eg, false
      chunksize: null #! eg, 5M
      prefix: null #! eg
      secretkey: null #! eg, secretkey
      accesskey: null #! eg, accesskey
      authversion: null #! eg, 3
      endpointtype: null #! eg, public
      tempurlcontainerkey: null #! eg, false
      tempurlmethods: null #! eg
    oss:
      accesskeyid: accesskeyid
      accesskeysecret: accesskeysecret
      region: regionname
      bucket: bucketname
      endpoint: null #! eg, endpoint
      internal: null #! eg, false
      encrypt: null #! eg, false
      secure: null #! eg, true
      chunksize: null #! eg, 10M
      rootdirectory: null #! eg, rootdirectory

#! The http/https network proxy for core, exporter, jobservice, trivy
proxy:
  httpProxy:
  httpsProxy:
  noProxy: 127.0.0.1,localhost,.local,.internal

#! The PSP names used by Harbor pods. The names are separated by ','. 'null' means all PSP can be used.
pspNames: null

#! The metrics used by core, registry and exporter
metrics:
  enabled: false
  core:
    path: /metrics
    port: 8001
  registry:
    path: /metrics
    port: 8001
  jobservice:
    path: /metrics
    port: 8001
  exporter:
    path: /metrics
    port: 8001

network:
  ipFamilies: ["IPv4","IPv6"]
EOF
```
</div></details>

Harborのパスワード等を自動生成・設定するスクリプトを実行する。
```
image_url=$(kubectl -n tanzu-package-repo-global get packages harbor.tanzu.vmware.com.$HARBOR_VER -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
imgpkg pull -b $image_url -o /tmp/harbor-package-$HARBOR_VER
bash /tmp/harbor-package-$HARBOR_VER/config/scripts/generate-passwords.sh ./harbor-default-values.yaml
```

ログインパスワードは任意のものにしたいので、そこだけ書き換える。
```
sed -ie 's/harborAdminPassword: .*/harborAdminPassword: xxxxx/g' harbor-default-values.yaml
```

HarborのFQDNも設定する。今回は`harbor.tkg.tanzu`とした。
```
sed -ie 's/hostname: .*/hostname: harbor.tkg.tanzu/g' harbor-default-values.yaml
```

公的な証明書を利用する場合は`tls.crt`、`tls.key`、`ca.crt`も設定するが、今回はオレオレ証明書で済ますため、特に設定しない。

またvSphere with TanzuだとstorageClassとpspNamesも設定する必要がある。
```
sed -ie 's/storageClass: .*/storageClass: xxxxx/g' harbor-default-values.yaml
sed -ie 's/pspNames: .*/pspNames: vmware-system-restricted/g' harbor-default-values.yaml
```

また、StorageClassがReadWriteManyをサポートしている場合、`registry`, `jobservice`, `database`, `redis`のaccessModeを`ReadWriteMany`に変更する必要がある。
今回はまだReadWriteManyを有効化していなかったので、ここは設定しない。


最終的には以下の差分となった。
<details><summary>Harborの設定ファイルの差分</summary><div>

```diff
@@ -4,7 +4,7 @@
 #! The namespace to install Harbor
 namespace: tanzu-system-registry
 #! The FQDN for accessing Harbor admin UI and Registry service.
-hostname: harbor.yourdomain.com
+hostname: harbor.tkg.tanzu
 #! The network port of the Envoy service in Contour or other Ingress Controller.
 port:
   https: 443
@@ -29,12 +29,12 @@
 #! Use contour http proxy instead of the ingress when it's true
 enableContourHttpProxy: true
 #! [Required] The initial password of Harbor admin.
-harborAdminPassword:
+harborAdminPassword: xxxxx
 #! [Required] The secret key used for encryption. Must be a string of 16 chars.
-secretKey:
+secretKey: ndWrBUEHOMHNyot5
 database:
   #! [Required] The initial password of the postgres database.
-  password:
+  password: wUe5bdoIRMot3een
   shmSizeLimit:
   maxIdleConns:
   maxOpenConns:
@@ -43,19 +43,19 @@
 core:
   replicas: 1
   #! [Required] Secret is used when core server communicates with other components.
-  secret:
+  secret: ayaLYKLnj050aGfT
   #! [Required] The XSRF key. Must be a string of 32 chars.
-  xsrfKey:
+  xsrfKey: jXhEa3pn5xZGd4B5NNlHnIsW3KNggSuy
 jobservice:
   replicas: 1
   #! [Required] Secret is used when job service communicates with other components.
-  secret:
+  secret: rccYdGB03hgj1BhH
 registry:
   replicas: 1
   #! [Required] Secret is used to secure the upload state from client
   #! and registry storage backend.
   #! See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http
-  secret:
+  secret: YstfT0CPcmiYU3xK
 notary:
   #! Whether to install Notary
   enabled: true
@@ -87,31 +87,31 @@
       #! Specify the "storageClass" used to provision the volume. Or the default
       #! StorageClass will be used(the default).
       #! Set it to "-" to disable dynamic provisioning
-      storageClass: ""
+      storageClass: xxxxx
       subPath: ""
       accessMode: ReadWriteOnce
       size: 10Gi
     jobservice:
       existingClaim: ""
-      storageClass: ""
+      storageClass: xxxxx
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
     database:
       existingClaim: ""
-      storageClass: ""
+      storageClass: xxxxx
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
     redis:
       existingClaim: ""
-      storageClass: ""
+      storageClass: xxxxx
       subPath: ""
       accessMode: ReadWriteOnce
       size: 1Gi
     trivy:
       existingClaim: ""
-      storageClass: ""
+      storageClass: xxxxx
       subPath: ""
       accessMode: ReadWriteOnce
       size: 5Gi
@@ -204,7 +204,7 @@
   httpsProxy:
   noProxy: 127.0.0.1,localhost,.local,.internal
 #! The PSP names used by Harbor pods. The names are separated by ','. 'null' means all PSP can be used.
-pspNames: null
+pspNames: vmware-system-restricted
 #! The metrics used by core, registry and exporter
 metrics:
   enabled: false
```
</div></details>

最後にコメントを削除する。
```
yq -i eval '... comments=""' harbor-default-values.yaml
```
コメント削除時、消しすぎることがあるので、先頭に`---`がなければ追加する。
```
sed -i '1i ---' harbor-default-values.yaml
```
作成した設定ファイルを利用してHarborをインストールする。
```
tanzu package install harbor \
--package-name harbor.tanzu.vmware.com \
--version $HARBOR_VER \
--values-file harbor-default-values.yaml \
--namespace $NAMESPACE
```

## 動作確認
今回はContourを利用しているため、HTTPProxyリソースが作成されている。
```
$ kubectl get httpproxy -n tanzu-system-registry
NAME                      FQDN                      TLS SECRET   STATUS   STATUS DESCRIPTION
harbor-httpproxy          harbor.tkg.tanzu          harbor-tls   valid    Valid HTTPProxy
harbor-httpproxy-notary   notary.harbor.tkg.tanzu   harbor-tls   valid    Valid HTTPProxy
```
Contourの場合のEndPointはEnvoyとなる。
```
$ kubectl get svc -n tanzu-system-ingress envoy
NAME    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
envoy   LoadBalancer   10.102.115.203   10.220.46.101   80:30763/TCP,443:31909/TCP   15h
```
試しに通信してみると、通信できていることが分かる。
```
$ curl  -H "Host: harbor.tkg.tanzu" 10.220.46.101 --head
HTTP/1.1 301 Moved Permanently
location: https://harbor.tkg.tanzu/
vary: Accept-Encoding
date: Tue, 01 Nov 2022 02:46:35 GMT
server: envoy
transfer-encoding: chunked
```

DNSサーバにIPとFQDNを登録してブラウザでアクセスすると、Harborの画面にもアクセス出来る。
![1667291002786.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/547981/a93d2f33-7594-a88a-770c-674098d16128.png)