OCP 4.10 Cluster Logging 5.4 - Vector for log collector

Red Hat OpenShift Container Platform (OCP) 4.10 では、`Logging 5.4 Technology Previews` の位置付けながら log collector に `Vector` を使用することができます。
  
__Release notes for Logging__
Logging 5.4 Technology Previews
https://docs.openshift.com/container-platform/4.10/logging/cluster-logging-release-notes.html#cluster-logging-technology-previews-5.4
> About Vector
> Vector is a log collector offered as a tech-preview alternative to the current default collector for the logging subsystem.
>
> The following outputs are supported:
>
> * elasticsearch. An external Elasticsearch instance. The elasticsearch output can use a TLS connection.
> * kafka. A Kafka broker. The kafka output can use an unsecured or TLS connection.
> * loki. Loki, a horizontally scalable, highly available, multi-tenant log aggregation system.
  
ここでは、上記の記述に従って、Cluster Logging 5.4 の log collector を `fluentd` から `vector` に変更してみます。
#### Cluster Logging Version の確認
Cluster Logging 5.4 が導入されていることを確認します。
```shell
$ oc version
Client Version: 4.10.13
Server Version: 4.10.13
Kubernetes Version: v1.23.5+b463d71

$ oc get csv -n openshift-logging
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
cluster-logging.5.4.1-24          Red Hat OpenShift Logging          5.4.1-24              Succeeded
elasticsearch-operator.5.4.1-24   OpenShift Elasticsearch Operator   5.4.1-24              Succeeded
```

#### ClusterLogging custom resource (CR) の変更
以下のコマンドで、`ClusterLogging` CR を変更します。
```shell
$ oc -n openshift-logging edit ClusterLogging instance
```
通常は、`.spec.collection` に `fluentd` が設定されています。
```shell
  spec:
    collection:
      logs:
        type: "fluentd"
        fluentd:
          resources: 
            limits:
              memory: 736Mi   
            requests:
              cpu: 200m
              memory: 736Mi
```
これを、以下のように `vector` に変更します。
```shell
  spec:
    collection:
      logs:
        type: "vector"
        vector: {}
```
#### log collector Pod の確認
現在の実装では、log collector を `vector` に変更しても、Pod Name、Container Name は変更されません。
そこで、log collector Pod が再起動された後に、`vector` で稼働していることを `oc logs` コマンドで確認してみます。
```shell
$ oc get pod -l component=collector -n openshift-logging
NAME              READY   STATUS    RESTARTS   AGE
collector-7fq8w   2/2     Running   0          2m
collector-cm5n6   2/2     Running   0          2m
collector-gbwzv   2/2     Running   0          2m
collector-grkg4   2/2     Running   0          2m
collector-h4c8x   2/2     Running   0          2m
collector-rzqsm   2/2     Running   0          3m
collector-sgfhs   2/2     Running   0          2m
collector-zpg25   2/2     Running   0          2m

$ oc get pod -l component=collector -n openshift-logging -o jsonpath='{range .items[*]}{"POD : "}{.metadata.name}{" CONTAINER : "}{.spec.containers[*].name}{"\n"}{end}'
POD : collector-7fq8w CONTAINER : collector logfilesmetricexporter
POD : collector-cm5n6 CONTAINER : collector logfilesmetricexporter
POD : collector-gbwzv CONTAINER : collector logfilesmetricexporter
POD : collector-grkg4 CONTAINER : collector logfilesmetricexporter
POD : collector-h4c8x CONTAINER : collector logfilesmetricexporter
POD : collector-rzqsm CONTAINER : collector logfilesmetricexporter
POD : collector-sgfhs CONTAINER : collector logfilesmetricexporter
POD : collector-zpg25 CONTAINER : collector logfilesmetricexporter

$ oc logs collector-7fq8w -c collector | head -n 31
Jun 13 06:04:16.693  INFO vector::app: Log level is enabled. level="info"
Jun 13 06:04:16.694  INFO vector::app: Loading configs. path=[("/etc/vector/vector.toml", Some(Toml))]
Jun 13 06:04:16.698  INFO vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="infra01"
Jun 13 06:04:16.725  INFO vector::topology: Running healthchecks.
Jun 13 06:04:16.725  INFO vector::topology::builder: Healthcheck: Passed.
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="raw_journal_logs"
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="k8s_audit_logs"
Jun 13 06:04:16.726  INFO source{component_kind="source" component_name=k8s_audit_logs component_type=file}: vector::sources::file: Starting file server. include=["/var/log/kube-apiserver/audit.log"] exclude=[]
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="openshift_audit_logs"
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="host_audit_logs"
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="internal_metrics"
Jun 13 06:04:16.726  INFO vector::topology: Starting source. name="raw_container_logs"
Jun 13 06:04:16.726  INFO source{component_kind="source" component_name=openshift_audit_logs component_type=file}: vector::sources::file: Starting file server. include=["/var/log/oauth-apiserver.audit.log"] exclude=[]
Jun 13 06:04:16.726  INFO source{component_kind="source" component_name=k8s_audit_logs component_type=file}:file_server: file_source::checkpointer: Loaded checkpoint data.
Jun 13 06:04:16.726  INFO source{component_kind="source" component_name=host_audit_logs component_type=file}: vector::sources::file: Starting file server. include=["/var/log/audit/audit.log"] exclude=[]
Jun 13 06:04:16.727  INFO source{component_kind="source" component_name=openshift_audit_logs component_type=file}:file_server: file_source::checkpointer: Loaded checkpoint data.
Jun 13 06:04:16.727  INFO source{component_kind="source" component_name=host_audit_logs component_type=file}:file_server: file_source::checkpointer: Loaded checkpoint data.
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="route_container_logs.app"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="audit"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="default_add_es_id"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="route_container_logs.infra"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="default_dedot_and_flatten"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="journal_logs"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="all-to-default"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="container_logs"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="infrastructure"
Jun 13 06:04:16.727  INFO vector::topology: Starting transform. name="application"
Jun 13 06:04:16.727  INFO vector::topology: Starting sink. name="default"
Jun 13 06:04:16.728  INFO vector::topology: Starting sink. name="prometheus_output"
Jun 13 06:04:16.728  INFO source{component_kind="source" component_name=raw_container_logs component_type=kubernetes_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
Jun 13 06:04:16.728  INFO vector: Vector has started. version="0.14.1" arch="x86_64" build_id="none"
```
最下行に `Vector has started. version="0.14.1" arch="x86_64" build_id="none"` と表示されていることが分かります。

#### Kibana UI の確認
log collector を `vector` に変更した状態で Kibana UI から Elasticsearch の情報が参照できるか確認します。
Discover 画面で `pipeline_metadata.collector.name` が存在（exists）する log を Filter して、`pipeline_metadata.collector.name` と `kubernetes.pod_name` を表示してみます。

![pic1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/54d1c894-12b8-e4c4-5f6d-04518d0745fc.png)

先頭の log の詳細を確認してみます。
![pic2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2128125/1bc158c3-9e37-4239-687f-2f46b008fdf4.png)

`pipeline_metadata.collector.name` が `vector`、`pipeline_metadata.collector.version` が `0.14.1` となっており、`oc logs` コマンドで確認した内容と一致している事が分かります。
  
簡単に確認した限りではありますが、Clutster Logging の log store（Elasticsearch）、visualizer（Kibana）の設定変更を行わなくても、`vector` で実装した log collector で取得した情報が正常に参照できるようです。