Kubernetes 1.18: Metrics Changes と SIG Instrumentation の変更内容

- [Kubernetes 1.16 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/e22c23c2a999ed5f8eaa)
- [Kubernetes 1.17 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/96a11f283ef98e6dcdd8)

## はじめに(Feature の内容含む)
ここでは、Kubernetes 1.18 の [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md) から Metrics の変更と SIG Instrumentation の取り組みについてまとめています。

[前回](https://qiita.com/watawuwu/items/96a11f283ef98e6dcdd8)に引き続き、[Control-Plane Metrics Stability](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20190404-kubernetes-control-plane-metrics-stability.md) のため `show-hidden-metrics-for-version` フラグが次のコンポーネントで[実装](https://github.com/kubernetes/kubernetes/issues/85270)されました。

- kube-apiserver(form v1.17)
- kube-proxy([#85279](https://github.com/kubernetes/kubernetes/pull/85279), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango))
- kube-scheduler([#84913](https://github.com/kubernetes/kubernetes/pull/84913), [@&#xFFF0;serathius](https://github.com/serathius))
- kube-controller-manager([#85281](https://github.com/kubernetes/kubernetes/pull/85281), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango))
- kubelet([#85282](https://github.com/kubernetes/kubernetes/pull/85282), [@&#xFFF0;serathius ](https://github.com/serathius))

それでは [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md) を見ていきます。

## メトリクスの変更（Metrics Changes）

### 追加
- [component-base] `rest_client_rate_limiter_duration_seconds`
  - クライアント側の Rate Limiter のレイテンシを秒単位で記録し、verb と url のラベルを持ちます([#88134](https://github.com/kubernetes/kubernetes/pull/88134), [@&#xFFF0;jennybuckley](https://github.com/jennybuckley))
- [component-base] `rest_client_certificate_expiration_seconds`
  - クライアント証明書の有効期限を 1970 年 1 月 1 日 UTC からの経過を秒単位で記録した Gauge([#84382](https://github.com/kubernetes/kubernetes/pull/84382), [@&#xFFF0;sambdavidson](https://github.com/sambdavidson))
- [component-base] `rest_client_certificate_rotation_age`
    - ローテーション後のクライアント証明書の経過時間を記録した Histgram([#84382](https://github.com/kubernetes/kubernetes/pull/84382), [@&#xFFF0;sambdavidson](https://github.com/sambdavidson))
- [controller-manager] `workqueue_(depth|adds_total|queue_duration_seconds|work_duration_seconds|unfinished_work_seconds|longest_running_processor_seconds|retries_total)`
    - workqueue のメトリクスが追加されました([#87967](https://github.com/kubernetes/kubernetes/pull/87967), [@&#xFFF0;zhan849](https://github.com/zhan849))
    - :memo: もともと提供されていたのですが、どこかでリグレッションしました?
- [api-server] 次のメトリクスの subresource ラベルに、`/healthz`、`/livez`、`/readyz` へのリクエストが記録されるようになりました([#83598](https://github.com/kubernetes/kubernetes/pull/83598), [@&#xFFF0;jktomer](https://github.com/jktomer))
    - `apiserver_request_duration_seconds`
    - `apiserver_request_total`
    - `apiserver_response_sizes`
- [kubelet] `(server|client)_expiration_renew_failure`
    - 証明書のローテーションが失敗した場合に記録します([#84614](https://github.com/kubernetes/kubernetes/pull/84614), [@&#xFFF0;rphillips](https://github.com/rphillips))
- [kubelet] `kubelet_pleg_last_seen_seconds`
    - 正常に PLEG が動作しているかを判断するために、最後にヘルスチェックした時間を記録します([#86251](https://github.com/kubernetes/kubernetes/pull/86251), [@&#xFFF0;bboreham](https://github.com/bboreham))

### 変更/非推奨

- [kubelet] 次のメトリクスの Stability Level が ALPHA となりました([#85446](https://github.com/kubernetes/kubernetes/pull/85446), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango))
    - `process_start_time_seconds`
- [kubelet] 次のメトリクスが非表示となりました([#83841](https://github.com/kubernetes/kubernetes/pull/83841), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango))
  - `kubelet_pod_worker_latency_microseconds`
  - `kubelet_pod_start_latency_microseconds`
  - `kubelet_cgroup_manager_latency_microseconds`
  - `kubelet_pod_worker_start_latency_microseconds`
  - `kubelet_pleg_relist_latency_microseconds`
  - `kubelet_pleg_relist_interval_microseconds`
  - `kubelet_eviction_stats_age_microseconds`
  - `kubelet_runtime_operations`
  - `kubelet_runtime_operations_latency_microseconds`
  - `kubelet_runtime_operations_errors`
  - `kubelet_device_plugin_registration_count`
  - `kubelet_device_plugin_alloc_latency_microseconds`
  - `kubelet_docker_operations`
  - `kubelet_docker_operations_latency_microseconds`
  - `kubelet_docker_operations_errors`
  - `kubelet_docker_operations_timeout`
  - `network_plugin_operations_latency_microseconds`

- 次のメトリクスは非表示となったので、対応するメトリクスに置き換えてください ([#76496](https://github.com/kubernetes/kubernetes/pull/76496), [@&#xFFF0;danielqsj](https://github.com/danielqsj))
  - `rest_client_request_latency_seconds` -> `rest_client_request_duration_seconds`
  - `scheduler_scheduling_latency_seconds` -> `scheduler_scheduling_duration_seconds `
  - `docker_operations` -> `docker_operations_total`
  - `docker_operations_latency_microseconds` -> `docker_operations_duration_seconds`
  - `docker_operations_errors` -> `docker_operations_errors_total`
  - `docker_operations_timeout` -> `docker_operations_timeout_total`
  - `network_plugin_operations_latency_microseconds` -> `network_plugin_operations_duration_seconds`
  - `kubelet_pod_worker_latency_microseconds` -> `kubelet_pod_worker_duration_seconds`
  - `kubelet_pod_start_latency_microseconds` -> `kubelet_pod_start_duration_seconds`
  - `kubelet_cgroup_manager_latency_microseconds` -> `kubelet_cgroup_manager_duration_seconds`
  - `kubelet_pod_worker_start_latency_microseconds` -> `kubelet_pod_worker_start_duration_seconds`
  - `kubelet_pleg_relist_latency_microseconds` -> `kubelet_pleg_relist_duration_seconds`
  - `kubelet_pleg_relist_interval_microseconds` -> `kubelet_pleg_relist_interval_seconds`
  - `kubelet_eviction_stats_age_microseconds` -> `kubelet_eviction_stats_age_seconds`
  - `kubelet_runtime_operations` -> `kubelet_runtime_operations_total`
  - `kubelet_runtime_operations_latency_microseconds` -> `kubelet_runtime_operations_duration_seconds`
  - `kubelet_runtime_operations_errors` -> `kubelet_runtime_operations_errors_total`
  - `kubelet_device_plugin_registration_count` -> `kubelet_device_plugin_registration_total`
  - `kubelet_device_plugin_alloc_latency_microseconds` -> `kubelet_device_plugin_alloc_duration_seconds`
  - `scheduler_e2e_scheduling_latency_microseconds` -> `scheduler_e2e_scheduling_duration_seconds`
  - `scheduler_scheduling_algorithm_latency_microseconds` -> `scheduler_scheduling_algorithm_duration_seconds`
  - `scheduler_scheduling_algorithm_predicate_evaluation` -> `scheduler_scheduling_algorithm_predicate_evaluation_seconds`
  - `scheduler_scheduling_algorithm_priority_evaluation` -> `scheduler_scheduling_algorithm_priority_evaluation_seconds`
  - `scheduler_scheduling_algorithm_preemption_evaluation` -> `scheduler_scheduling_algorithm_preemption_evaluation_seconds`
  - `scheduler_binding_latency_microseconds` -> `scheduler_binding_duration_seconds`
  - `kubeproxy_sync_proxy_rules_latency_microseconds` -> `kubeproxy_sync_proxy_rules_duration_seconds`
  - `apiserver_request_latencies` -> `apiserver_request_duration_seconds`
  - `apiserver_dropped_requests` -> `apiserver_dropped_requests_total`
  - `etcd_request_latencies_summary` -> `etcd_request_duration_seconds`
  - `apiserver_storage_transformation_latencies_microseconds ` -> `apiserver_storage_transformation_duration_seconds`
  - `apiserver_storage_data_key_generation_latencies_microseconds` -> `apiserver_storage_data_key_generation_duration_seconds`
  - `apiserver_request_count` -> `apiserver_request_total`
  - `apiserver_request_latencies_summary`
  - `apiserver_storage_transformation_failures_total` -> `apiserver_storage_transformation_operations_total`

## その他の変更 （Other [Bug, Cleanup or Flake]）
- [kube-apiserver] `--show-hidden-metrics-for-version` フラグで非表示のメトリクスを表示できなかったバグを修正([#85444](https://github.com/kubernetes/kubernetes/pull/85444), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango))

## 所感
v1.16 から始まった [KEP Metrics Stability Migration](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20190605-metrics-stability-migration.md) の作業も進み、次のマイナーバージョンからは [KEP Structured Logging](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20191115-structured-logging.md) の取り組みも入ってきそうです。

JSON ログフォーマットで出力してくれるとログストアの機能によっては検索やフィルターがしやすくなるので個人的には注目しています。


<!--
- Add `rest_client_rate_limiter_duration_seconds` metric to component-base to track client side rate limiter latency in seconds. Broken down by verb and URL. ([#88134](https://github.com/kubernetes/kubernetes/pull/88134), [@&#xFFF0;jennybuckley](https://github.com/jennybuckley)) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
- a gauge reporting the lifetime of the current client certificate. Reports the time of expiry in seconds since January 1, 1970 UTC.
- a histogram reporting the age of a just rotated client certificate in seconds. ([#84382](https://github.com/kubernetes/kubernetes/pull/84382), [@&#xFFF0;sambdavidson](https://github.com/sambdavidson)) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
-->
<!-- Controller manager serve workqueue metrics ([#87967](https://github.com/kubernetes/kubernetes/pull/87967), [@&#xFFF0;zhan849](https://github.com/zhan849)) [SIG API Machinery] -->
<!-- Kube-apiserver metrics will now include request counts, latencies, and response sizes for /healthz, /livez, and /readyz requests. ([#83598](https://github.com/kubernetes/kubernetes/pull/83598), [@&#xFFF0;jktomer](https://github.com/jktomer)) [SIG API Machinery] -->
<!-- Kubelet now exports a `server_expiration_renew_failure` and `client_expiration_renew_failure` metric counter if the certificate rotations cannot be performed. ([#84614](https://github.com/kubernetes/kubernetes/pull/84614), [@&#xFFF0;rphillips](https://github.com/rphillips)) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Release] -->
<!-- New metric `kubelet_pleg_last_seen_seconds` to aid diagnosis of PLEG not healthy issues. ([#86251](https://github.com/kubernetes/kubernetes/pull/86251), [@&#xFFF0;bboreham](https://github.com/bboreham)) [SIG Node] -->
<!-- Following metrics have been turned off: ([#83841](https://github.com/kubernetes/kubernetes/pull/83841), [@&#xFFF0;RainbowMango](https://github.com/RainbowMango)) [SIG Network and Node] -->
<!-- The following deprecated metrics are removed, please convert to the corresponding metrics -->
<!-- kube-apiserver: Fixes a bug that hidden metrics can not be enabled by the command-line option --show-hidden-metrics-for-version. (#85444, @RainbowMango) -->