Kubernetes 1.28 SIG Instrumentation の変更内容

- [Kubernetes 1.26 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/9259782239976825d287)
- [Kubernetes 1.27 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/92e849642f4477b10a20)

## はじめに

ここでは、Kubernetes 1.28 の [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md) からメトリクスの変更と SIG Instrumentation の取り組みについてまとめています。

Kubernetes v1.28 で SIG Instrumentation が扱う主要なテーマはありませんが、SIG Node のテーマに SIG Instrumentation に関連した施策があるため記載します（注意: v1.28 のマイルストーンから外れました）。

## CRI container and pod statistics without cAdvisor

<!-- This encompasses two related pieces of work (changes to the kubelet's /metrics/cadvisor endpoint and improvements to the replacement summary API). -->
<!-- There are two main APIs that consumers use to gather stats about running containers and pods: summary API and /metrics/cadvisor. The Kubelet is responsible for implementing the summary API, and cadvisor is responsible for fulfilling /metrics/cadvisor. -->
<!-- This enhances CRI implementations to be able to fulfill all the stats needs of Kubernetes. At a high level, there are two pieces of this: -->
<!-- It enhances the CRI API with enough metrics to supplement the pod and container fields in the summary API directly from CRI. -->
<!-- It enhances the CRI implementations to broadcast the required metrics to fulfill the pod and container fields in the /metrics/cadvisor endpoint. -->

このテーマは SIG Node のため詳細については割愛しますが、kubelet の `/metrics/cadvisor` 、`/stats/summary` エンドポイントで取得できる統計情報の取得元の整理と改善が進められています。

- [KEP-2371: cAdvisor-less, CRI-full Container and Pod Stats](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2371-cri-pod-container-stats)

エンドポイント自体の廃止などはノンゴールとなっていますが、一部のメトリクスは新しい情報源である CRI API でサポートされない可能性もあるため注視していきたいと思います。
この施策によりメトリクスの変更がある場合も可能な限り `メトリクスの変更` で記載していきます。

ただし、残念ながら v1.28 のマイルストーンからは外れたため Beta ではなく Alpha のままとなっています。  
[cAdvisor-less, CRI-full Container and Pod Stats · Issue #2371 · kubernetes/enhancements](https://github.com/kubernetes/enhancements/issues/2371#issuecomment-1641436551)

それ以外の更新された KEP のマイルストーンと参考リンクを記載しておきます。

- [KEP-3498: Extending Metrics Stability](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3498-extending-stability)
  - v1.28 で GA になりました
  - [こちらの記事で](https://qiita.com/watawuwu/items/9259782239976825d287#metrics-stability-framework-%E3%81%AE%E6%8B%A1%E5%BC%B5)紹介しています
- [KEP-2305: Dynamic Cardinality Enforcement](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2305-metrics-cardinality-enforcement)
  - v1.28 で Beta の予定でしたが  [v1.28 のマイルストーンから現在は削除されています](https://github.com/kubernetes/enhancements/issues/2305#issuecomment-1648133527)

それでは `Metrics Changes` について説明していきます。
`メトリクスの変更` は、私が全ての変更点からメトリクスの変更に関連するものを抜粋した内容となります。

## メトリクスの変更（Metrics Changes）

### component-base

<!-- - Allow to monitor client-go DNS resolver latencies via `rest_client_dns_resolution_duration_seconds` Prometheus metric. ([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@&#xFFF0;mfojtik](https://github.com/mfojtik)) -->
- [ADD] client-go の DNS 解決のレイテンシーを計測するメトリクスが追加されました([#115357](https://github.com/kubernetes/kubernetes/pull/115357), [@&#xFFF0;mfojtik](https://github.com/mfojtik))
  - Histogram: `rest_client_dns_resolution_duration_seconds`
    - Labels: {"host"}
    - Buckets: {0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0}
<!-- - Client-go now exposes two new metrics to monitor the client-go logic that generate http.Transports for the clients. -->
<!--   - `rest_client_transport_cache_entries` is a gauge metric -->
<!--   with the number of existing entries in the internal cache -->
<!--   - `rest_client_transport_create_calls_total` is a counter -->
<!--   that increments each time a new transport is created, storing the result of the operation needed to generate it: hit, miss or uncacheable. ([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@&#xFFF0;aojea](https://github.com/aojea)) -->
- [ADD] client-go の http.Transports の処理で、キャッシュエントリ数とトランスポートの作成数を記録するメトリクスが追加されました([#117295](https://github.com/kubernetes/kubernetes/pull/117295), [@&#xFFF0;aojea](https://github.com/aojea))
  - Gauge: `rest_client_transport_cache_entries`
    - Labels: None
  - Counter: `rest_client_transport_create_calls_total`
    - Labels: {"result"}
<!-- - registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
- [MOD] 次のメトリクスが Beta になりました([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Gauge: `kubernetes_feature_enabled`
- [MOD] 次のメトリクスの名前がリネームされ、Beta になりました([#119264](https://github.com/kubernetes/kubernetes/pull/119264), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Counter: `registered_metric_total` => `registered_metrics_total`
  - Counter: `disabled_metric_total` => `disabled_metrics_total`
  - Counter: `hidden_metric_total` => `hidden_metrics_total`
<!-- - Promoted `kubernetes_healthcheck` and `kubernetes_healthchecks_total` to `BETA` stability level. ([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
- [MOD] 次のメトリクスが Beta になりました([#118986](https://github.com/kubernetes/kubernetes/pull/118986), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Counter: `kubernetes_healthcheck`
  - Counter: `kubernetes_healthchecks_total`

### kube-apiserver

<!-- - Added `apiserver_admission_match_condition_evaluation_seconds` and `apiserver_admission_match_condition_exclusions_total` metrics. ([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@&#xFFF0;ivelichkovich](https://github.com/ivelichkovich)) -->
- [ADD] ValidatingWebhookConfiguration、または MutatingWebhookConfiguration の [matchCondition](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions) の評価時間と除外された数を記録するメトリクスが追加されました([#119311](https://github.com/kubernetes/kubernetes/pull/119311), [@&#xFFF0;ivelichkovich](https://github.com/ivelichkovich))
  - Histogram: `apiserver_admission_match_condition_evaluation_seconds`
    - Labels: {"name", "kind", "type", "operation"}
    - Buckets: {0.001, 0.005, 0.01, 0.025, 0.1, 0.2, 0.25}
  - Counter: `apiserver_admission_match_condition_exclusions_total`
    - Labels: {"name", "kind", "type", "operation"}
<!-- - Apiserver adds two new metrics `etcd_requests_total` and `etcd_request_errors_total` that allow users to monitor requests to etcd storage, split by operation and resource type. ([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@&#xFFF0;iyear](https://github.com/iyear)) -->
- [ADD] apiserver から etcd へのリクエストをモニタリングするためのメトリクスが追加されました。`type` ラベルにはリソースタイプが記録されます([#117222](https://github.com/kubernetes/kubernetes/pull/117222), [@&#xFFF0;iyear](https://github.com/iyear))
  - Counter: `etcd_requests_total`
    - Labels: {"operation", "type"}
  - Counter: `etcd_request_errors_total`
    - Labels: {"operation", "type"}
<!-- - New Metrics Added for Encryption Configuration Controller. This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include: -->
<!--   - `apiserver_encryption_config_controller_automatic_reload_failures_total`: Total number of failed automatic reloads of encryption configuration. -->
<!--   - `apiserver_encryption_config_controller_automatic_reload_success_total`: Total number of successful automatic reloads of encryption configuration. -->
<!--   - `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`: Timestamp of the last successful or failed automatic reload of encryption configuration. -->
<!--   These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@&#xFFF0;nilekhc](https://github.com/nilekhc)) -->
- [ADD] Encryption Configuration Controller の暗号設定の自動再読み込みをモニタリングするメトリクスが追加されました ([#119008](https://github.com/kubernetes/kubernetes/pull/119008), [@&#xFFF0;nilekhc](https://github.com/nilekhc))
  - Counter: `apiserver_encryption_config_controller_automatic_reload_failures_total`
    - Help: Total number of failed automatic reloads of encryption configuration.
    - Labels: None
  - Counter: `apiserver_encryption_config_controller_automatic_reload_success_total`
    - Help: Total number of successful automatic reloads of encryption configuration
    - Labels: None
  - Gauge: `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds`
    - Help: Timestamp of the last successful or failed automatic reload of encryption configuration
    - Labels: None
<!-- - `kube-apiserver` added two new metrics `authorization_attempts_total` and `authorization_duration_seconds` that allow users to monitor requests to authorization webhooks, split by result. ([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@&#xFFF0;HirazawaUi](https://github.com/HirazawaUi)) -->
- [ADD] Authorization Webhook をモニタリングするメトリクスが追加されました([#117211](https://github.com/kubernetes/kubernetes/pull/117211), [@&#xFFF0;HirazawaUi](https://github.com/HirazawaUi))
  - Counter: `authorization_attempts_total`
    - Labels: {"result"}
  - Histgram: `authorization_duration_seconds`
    - Labels: {"result"}
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 15)
<!-- - Kube-apiserver adds two new alpha metrics `conversion_webhook_request_total` and `conversion_webhook_duration_seconds` that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). ([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@&#xFFF0;cchapla](https://github.com/cchapla)) -->
- [ADD] CRD Conversion Webhook のリクエストをモニタリングするメトリクスが追加されました([#118292](https://github.com/kubernetes/kubernetes/pull/118292), [@&#xFFF0;cchapla](https://github.com/cchapla))
  - Counter: `apiserver_webhook_conversion_request_total`
    - Labels: {"result", "failure_type"}
  - Histogram: `apiserver_webhook_conversion_duration_seconds`
    - Labels: {"result", "failure_type"}
    - Bucket: {0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 30, 45, 60}
<!-- - The metric `apiserver_flowcontrol_current_executing_seats` has been introduced as a duplicate of `apiserver_flowcontrol_request_concurrency_in_use` because the latter has a confusing name and will be removed in a later release. ([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer)) -->
- [ADD|DEPRECATED] 次の紛らわしい名前のメトリクスが非推奨となり、代わりのメトリクスが追加されました([#118960](https://github.com/kubernetes/kubernetes/pull/118960), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer))
  - [DEPRECATED] Gauge: `apiserver_flowcontrol_request_concurrency_in_use`
  - [ADD] Gauge: `apiserver_flowcontrol_current_executing_seats`
    - Labels: {"priorityLevel", "flowSchema"}
<!-- - Promoted the following apiserver flowcontrol metrics to Beta: -->
<!--   - apiserver_flowcontrol_request_wait_duration_seconds -->
<!--   - apiserver_flowcontrol_current_executing_seats -->
<!--   - apiserver_flowcontrol_nominal_limit_seats -->
<!--   - apiserver_flowcontrol_rejected_requests_total -->
<!--   - apiserver_flowcontrol_dispatched_requests_total -->
<!--   - apiserver_flowcontrol_current_inqueue_requests -->
<!--   - apiserver_flowcontrol_current_executing_requests ([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim)) -->
- [MOD] 次の Flowcontrol に関するメトリクスが Beta となりました([#119110](https://github.com/kubernetes/kubernetes/pull/119110), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim))
  - Histgram: `apiserver_flowcontrol_request_wait_duration_seconds`
  - Gauge: `apiserver_flowcontrol_current_executing_seats`
  - Gauge: `apiserver_flowcontrol_nominal_limit_seats`
  - Counter: `apiserver_flowcontrol_rejected_requests_total`
  - Counter: `apiserver_flowcontrol_dispatched_requests_total`
  - Gauge: `apiserver_flowcontrol_current_inqueue_requests`
  - Gauge: `apiserver_flowcontrol_current_executing_requests`
- <!-- Replaced `apiserver_storage_db_total_size_in_bytes` with `apiserver_storage_size_bytes` metric. ([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@&#xFFF0;serathius](https://github.com/serathius)) -->
- [MOD] 次のメトリクスがリネームされました([#118812](https://github.com/kubernetes/kubernetes/pull/118812), [@&#xFFF0;serathius](https://github.com/serathius))
    - `apiserver_storage_db_total_size_in_bytes` => `apiserver_storage_size_bytes`
<!-- - Updated apiserver metric `request_filter_duration_seconds` to include a 10s, 15s and 30s bucket. Updated apiserver metric `request_wait_duration_seconds` to include a 15s bucket. ([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim)) -->
- [MOD] 次のメトリクスのバケットを更新しました([#118945](https://github.com/kubernetes/kubernetes/pull/118945), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim))
  - Histogram: `apiserver_request_filter_duration_seconds`
    - Buckets: {0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0, 5.0, **10.0**, **15.0**, **30.0**}
  - Histogram: `apiserver_request_wait_duration_seconds`
    - Buckets: {0, 0.005, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, **15**, 30}
<!-- - The metric `apiserver_flowcontrol_request_concurrency_limit` has been deprecated and will be removed in a future release.  It is a duplicate of `apiserver_flowcontrol_nominal_limit_seats` (introduced in release 1.26) but has an outdated name and had an outdated HELP string. ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer)) -->
- [DEPRECATED] 次のメトリクスが非推奨となりました。合わせてヘルプテキストも修正されています ([#118959](https://github.com/kubernetes/kubernetes/pull/118959), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer))
  - Gauge: `apiserver_flowcontrol_request_concurrency_limit`
    - 代わりに `apiserver_flowcontrol_nominal_limit_seats` の利用を検討してください

### kubelet

<!-- - Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). ([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@&#xFFF0;iholder101](https://github.com/iholder101)) -->
- [ADD] `/stats/summary` と `/metrics/resource` のエンドポイントにスワップのメトリクスが追加されました([#118865](https://github.com/kubernetes/kubernetes/pull/118865), [@&#xFFF0;iholder101](https://github.com/iholder101))
  - Gauge: `node_swap_usage_bytes`
    - Labels: None
  - Gauge: `pod_swap_usage_bytes`
    - Labels: None
  - Gauge: `container_swap_usage_bytes`
    - Labels: None
<!-- - CRI: exposed commit memory bytes in container stats specific to Windows ([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@&#xFFF0;kiashok](https://github.com/kiashok)) -->
- [ADD] CRI: Windows 固有のメモリメトリクスが公開されました([#119238](https://github.com/kubernetes/kubernetes/pull/119238), [@&#xFFF0;kiashok](https://github.com/kiashok))
<!-- - Number of errors reported to the metric `storage_operation_duration_seconds_count` for emptyDir decreased significantly because previously one error was reported for each projected volume created. ([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@&#xFFF0;mpatlasov](https://github.com/mpatlasov)) -->
- [FIX] `volume_plugin="kubernetes.io/empty-dir"` のラベル値でカウントされるエラー数に、Projected Volume が作成されるたびにカウントされている問題が修正されました([#117022](https://github.com/kubernetes/kubernetes/pull/117022), [@&#xFFF0;mpatlasov](https://github.com/mpatlasov))
  - Couner: `storage_operation_duration_seconds_count`
<!-- - Static pods were taking extra time to be restarted after being updated.  Static pods that are waiting to restart were not correctly counted in `kubelet_working_pods`. ([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton)) -->
- [FIX] 再起動待ちの Static Pod で正しくカウントされていない問題が修正されました([#116995](https://github.com/kubernetes/kubernetes/pull/116995), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton))
  - Counter: `kubelet_working_pods`

### kube-controller-manager

<!-- - Added reason to metric `attachdetach_controller_forced_detaches` in the attach detach controller. ([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@&#xFFF0;xing-yang](https://github.com/xing-yang)) -->
- [MOD] Attach detach controller に、以下のメトリクスのラベルが追加されました([#119185](https://github.com/kubernetes/kubernetes/pull/119185), [@&#xFFF0;xing-yang](https://github.com/xing-yang))
  - `attachdetach_controller_forced_detaches`
    - Labels: {"reason"}
<!-- - `force_delete_pods_total` and `force_delete_pod_errors_total` metrics count all pod deletion behaviors. ([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@&#xFFF0;carlory](https://github.com/carlory)) -->
- [FIX] Pod GC による強制削除の全ての挙動で記録されるように修正され、ラベルも追加されました([#118480](https://github.com/kubernetes/kubernetes/pull/118480), [@&#xFFF0;carlory](https://github.com/carlory))
  - Counter: `force_delete_pods_total`
    - Labels: {"namespace", "reason"}
  - Counter: `force_delete_pod_errors_total`
    - Labels: {"namespace", "reason"}

### kube-scheduler

<!-- - Metric `scheduler_scheduler_goroutines` is removed. Use `scheduler_goroutines` instead. ([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@&#xFFF0;kerthcet](https://github.com/kerthcet)) -->
- [DEL] 次のメトリクスが削除されました([#117727](https://github.com/kubernetes/kubernetes/pull/117727), [@&#xFFF0;kerthcet](https://github.com/kerthcet))
  - Gauge: `scheduler_scheduler_goroutines`
    - 代わりに `scheduler_goroutines` を利用してください
<!-- - `plugin_evaluation_total` metric supports prescore/score extension point. The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. ([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@&#xFFF0;AxeZhan](https://github.com/AxeZhan)) -->
- [MOD] `extension_point` ラベルに `Prescore` と `Score` のサポートが追加されました。`Prescore` と `Score` に関係のない Pod はカウントされません。([#118025](https://github.com/kubernetes/kubernetes/pull/118025), [@&#xFFF0;AxeZhan](https://github.com/AxeZhan))
  - Counter: `scheduler_plugin_evaluation_total`
    - Labels: {"extension_point","plugin","profile"}
<!-- - The `pod_scheduling_duration_seconds` metrics won't consider the time when a pod fails `PreEnqueue` (like being gated). ([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@&#xFFF0;helayoty](https://github.com/helayoty)) -->
- [FIX] Pod が `PreEnqueue` に失敗した時間を含まないように修正されました([#118049](https://github.com/kubernetes/kubernetes/pull/118049), [@&#xFFF0;helayoty](https://github.com/helayoty))
  - Histgram: `pod_scheduling_duration_seconds`
    - Labels: {"attempts"}

### kube-proxy

<!-- - `kube-proxy` in iptables mode will now have separate `sync_full_proxy_rules_duration_seconds` and `sync_partial_proxy_rules_duration_seconds` (in addition to the existing\n`sync_proxy_rules_duration_seconds`), giving better information about the duration of each \nsync type, rather than only giving a weighted average of the two sync types together. ([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@&#xFFF0;danwinship](https://github.com/danwinship)) -->
- `kubeproxy_sync_proxy_rules_duration_seconds` メトリクスとは別に、以下のメトリクスが追加されました。([#117787](https://github.com/kubernetes/kubernetes/pull/117787), [@&#xFFF0;danwinship](https://github.com/danwinship))
  - Histgram `kubeproxy_sync_full_proxy_rules_duration_seconds`
    - Labels: None
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 15)
  - Histgram: `kubeproxy_sync_partial_proxy_rules_duration_seconds`
    - Labels: None
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 15)
  - :memo: `kubeproxy_sync_proxy_rules_duration_seconds` は 2 つのメトリクスは分割されましたが、今のところ元のメトリクスも継続して利用できるようです
<!-- - The kube-proxy `sync_proxy_rules_iptables_total` metric has now reverted back to its pre-1.27 behavior of tracking the total number of iptables rules that kube-proxy is responsible for, rather than only counting the number of rules that it re-synced on the last sync. The new `sync_proxy_rules_iptables_last` metric now gives the latter number. ([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@&#xFFF0;danwinship](https://github.com/danwinship)) -->
- [MOD|ADD] `sync_proxy_rules_iptables_total` メトリクスは v1.27 以前の方法である kube-proxy が管理する iptables のルール総数を記録する方法にリバートされました。最後に同期したルール数をカウントするメトリクスは別のメトリクスとして追加されました。([#119140](https://github.com/kubernetes/kubernetes/pull/119140), [@&#xFFF0;danwinship](https://github.com/danwinship))
  - Gauge: `kubeproxy_sync_proxy_rules_iptables_last`
    - Labels: {"table"}

### その他
<!-- - Cloud controller manager's node controller now emits timing metrics for initial `Node` synchronization. These metrics measure the delay between the creation of a new `Node` and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's `--concurrent-node-syncs` flag. ([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@&#xFFF0;cartermckinnon](https://github.com/cartermckinnon)) -->
- [ADD] Cloud Controller Manager の Node Controller が `Node` の初期同期のタイミングでメトリクスを出力するようになりました。このメトリクスは新しい `Node` の作成と、クラウドプロバイダのテイントの削除、Node Controller の初期アクションとの間のレイテンシーを記録します。これらのメトリクスは、Cloud Manager の `--concurrent-node-syncs` フラグを設定する際に参照する必要があります。([#119241](https://github.com/kubernetes/kubernetes/pull/119241), [@&#xFFF0;cartermckinnon](https://github.com/cartermckinnon))
  - Histgram: `node_controller_cloud_provider_taint_removal_delay_seconds`
    - Labels: None
    - Buckets: metrics.ExponentialBuckets(1, 4, 6)
  - Histgram: `node_controller_initial_node_sync_delay_seconds`
    - Labels: None
    - Buckets: metrics.ExponentialBuckets(1, 4, 6)

## メトリクス以外の変更点

<!-- - `component-base/logs` is now stricter about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. ([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@&#xFFF0;pohly](https://github.com/pohly)) -->
- `component-base/logs` は設定を複数回適用するとエラーとして処理するようになりました([#117108](https://github.com/kubernetes/kubernetes/pull/117108), [@&#xFFF0;pohly](https://github.com/pohly))
<!-- - Bumped metrics-server to `v0.6.3`. ([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- metrics-server が `v0.6.3` にアップデートされました([#117120](https://github.com/kubernetes/kubernetes/pull/117120), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))
<!-- - Updated etcd image to `3.5.9-0`. ([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@&#xFFF0;kkkkun](https://github.com/kkkkun)) -->
- etcd が `3.5.9-0` にアップデートされました([#117999](https://github.com/kubernetes/kubernetes/pull/117999), [@&#xFFF0;kkkkun](https://github.com/kkkkun))
<!-- - Updated cAdvisor to `v0.47.2` and fixed metrics in `cri-o` when a container restarts. ([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@&#xFFF0;harche](https://github.com/harche)) -->
- cAdvisor が `v0.47.2` にアップデートされました。このバージョンでは `cri-o` でコンテナがリスタートした場合にメトリクスが欠損する問題が修正されています([#118774](https://github.com/kubernetes/kubernetes/pull/118774), [@&#xFFF0;harche](https://github.com/harche))
<!-- - Bumped cadvisor version to `v0.47.3`. ([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@&#xFFF0;iholder101](https://github.com/iholder101)) -->
- cadvisor `v0.47.3` にアップデートされました([#119225](https://github.com/kubernetes/kubernetes/pull/119225), [@&#xFFF0;iholder101](https://github.com/iholder101))
<!-- - Hashing of KeyID in Logs. This release adds a feature to hash the `KeyID` values in the logs. The `KeyID` values are sensitive information that should not be exposed in plain text in the logs. By hashing the `KeyID` values, we can protect the confidentiality of the data while still being able to log the necessary information. ([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@&#xFFF0;nilekhc](https://github.com/nilekhc)) -->
- [KMSv2] ログ中の `KeyID` 値をハッシュ化する機能が追加されました。`KeyID` 値は機密情報であるため平文で記録しないようにします。ハッシュ化することで、必要な情報を記録しつつ、データの機密性を保護できます([#118988](https://github.com/kubernetes/kubernetes/pull/118988), [@&#xFFF0;nilekhc](https://github.com/nilekhc))
<!-- - The scheduler skips the `InterPodAffinity` Score plugin when nothing to do with the Pod. It will affect some metrics values related to the InterPodAffinity Score plugin. ([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@&#xFFF0;utam0k](https://github.com/utam0k)) -->
- スケジューラーは Pod に関連した設定がない場合、 `InterPodAffinity` Score プラグインの処理をスキップします。この処理によりスケジューラーのメトリクスに影響が生じます([#117794](https://github.com/kubernetes/kubernetes/pull/117794), [@&#xFFF0;utam0k](https://github.com/utam0k))
<!-- - The scheduler skips the `PodTopologySpread` Filter plugin if no spread constraints. It will affect some metrics values related to the `PodTopologySpread` Filter plugin. ([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@&#xFFF0;utam0k](https://github.com/utam0k)) -->
- スケジューラーは Pod Topology Spread Constraint の設定がない場合、`PodTopologySpread` Filter プラグインの処理をスキップします。この処理によりスケジューラーのメトリクスに影響が生じます([#117683](https://github.com/kubernetes/kubernetes/pull/117683), [@&#xFFF0;utam0k](https://github.com/utam0k))
<!-- - The scheduler skips the `PodTopologySpread` Score plugin when nothing to do with the Pod. It will affect some metrics values related to the PodTopologySpread Score plugin. ([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@&#xFFF0;AxeZhanutam0k](https://github.com/utam0k)) -->
- スケジューラーは Pod に関連した設定がない場合 `PodTopologySpread` Score プラグインの処理をスキップします。この処理によりスケジューラーのメトリクスに影響が生じます([#118608](https://github.com/kubernetes/kubernetes/pull/118608), [@&#xFFF0;AxeZhanutam0k](https://github.com/utam0k))
<!-- - Kube-proxy: added `--logging-format` flag to support structured logging. ([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@&#xFFF0;cyclinder](https://github.com/cyclinder)) -->
- kube-proxy に `--logging-format` フラグが追加され構造化ロギングがサポートされました([#117800](https://github.com/kubernetes/kubernetes/pull/117800), [@&#xFFF0;cyclinder](https://github.com/cyclinder))
<!-- - Migrated `pkg/controller/endpoint` to contextual logging. ([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@&#xFFF0;my-git9](https://github.com/my-git9)) -->
<!-- - Migrated `pkg/scheduler/framework/preemption` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated `pod-security-admission` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001)) -->
<!-- - Migrated controller functions to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@&#xFFF0;fatsheep9146](https://github.com/fatsheep9146)) -->
<!-- - Migrated the Job controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@&#xFFF0;fatsheep9146](https://github.com/fatsheep9146)) -->
<!-- - Migrated the `EndpointSlice` and `EndpointSliceMirroring` controllers (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001)) -->
<!-- - Migrated the certificate controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated the noderesources scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated the podtopologyspread scheduler plugins to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated `pkg/scheduler/framework/runtime` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated the disruption controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated the interpodaffinity scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Migrated the podgc controller and some other remaining log calls within `kube-controller-manager` to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). `kube-controller-manager` is now converted completely. ([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@&#xFFF0;pohly](https://github.com/pohly)) -->
<!-- - Migrated the volumezone scheduler plugin to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
<!-- - Updated the scheduler interface and cache methods to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging). ([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
- 次のパッケージ、コントローラー、または処理などは[構造化ロギング](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging)に移行しました
  - `pkg/controller/endpoint`([#116755](https://github.com/kubernetes/kubernetes/pull/116755), [@&#xFFF0;my-git9](https://github.com/my-git9))
  - `pkg/scheduler/framework/preemption`([#116835](https://github.com/kubernetes/kubernetes/pull/116835), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - `pod-security-admission`([#114471](https://github.com/kubernetes/kubernetes/pull/114471), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001))
  - `pkg/controller/volume`, `pkg/controller/util/`, `pkg/controller/replicaset`, `pkg/controller/nodeipam`, `pkg/controller/garbagecollector`, `pkg/controller/endpoint`([#116930](https://github.com/kubernetes/kubernetes/pull/116930), [@&#xFFF0;fatsheep9146](https://github.com/fatsheep9146))
  - `pkg/scheduler/framework/runtime`([#116842](https://github.com/kubernetes/kubernetes/pull/116842), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Job Controller([#116910](https://github.com/kubernetes/kubernetes/pull/116910), [@&#xFFF0;fatsheep9146](https://github.com/fatsheep9146))
  - EndpointSlice Controller、EndpointSliceMirroring Controller([#115295](https://github.com/kubernetes/kubernetes/pull/115295), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001))
  - Certificate Controller([#113994](https://github.com/kubernetes/kubernetes/pull/113994), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Disruption Controller([#119147](https://github.com/kubernetes/kubernetes/pull/119147), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - PodGC Controller/other remaining log calls within `kube-controller-manager`([#119250](https://github.com/kubernetes/kubernetes/pull/119250), [@&#xFFF0;pohly](https://github.com/pohly))
  - NodeResources Scheduler Plugin([#116748](https://github.com/kubernetes/kubernetes/pull/116748), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - PodtopologySpread Scheduler Plugins([#116797](https://github.com/kubernetes/kubernetes/pull/116797), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Interpodaffinity Scheduler Plugin([#116635](https://github.com/kubernetes/kubernetes/pull/116635), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Volumezone Scheduler Plugin([#116829](https://github.com/kubernetes/kubernetes/pull/116829), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Scheduler Interface/Cache Method([#116849](https://github.com/kubernetes/kubernetes/pull/116849), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
<!-- - Set metrics-server's metric-resolution to 15s. ([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- [Addons] metrics-server の metric-resolution が 30s から 15s に変更されました([#117121](https://github.com/kubernetes/kubernetes/pull/117121), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))
<!-- - `klog` text output now uses JSON as encoding for structs, maps and slices. ([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@&#xFFF0;pohly](https://github.com/pohly)) -->
- `klog` が `v2.100.1` にアップデートされました。テキスト出力の構造体、マップ、スライスのエンコーディングに JSON が使用されます([#117687](https://github.com/kubernetes/kubernetes/pull/117687), [@&#xFFF0;pohly](https://github.com/pohly))
<!-- - A `v2-level` info log will be added, which will output the details of the pod being preempted, including victim and preemptor. ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@&#xFFF0;HirazawaUi](https://github.com/HirazawaUi)) -->
- [Scheduler] プリエンプトされた Pod の詳細がログに記録されうようになりました（ログレベル 2 ） ([#117214](https://github.com/kubernetes/kubernetes/pull/117214), [@&#xFFF0;HirazawaUi](https://github.com/HirazawaUi))