Kubernetes 1.27 SIG Instrumentation の変更内容

- [Kubernetes 1.25 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/d4632f750437081265cb)
- [Kubernetes 1.26 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/9259782239976825d287)

## はじめに(What’s New の内容含む)

ここでは、Kubernetes 1.27 の [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md) から Metrics の変更と SIG Instrumentation の取り組みについてまとめています。


Kubernetes v1.27 で SIG Instrumentation が扱う主要なテーマと注目したいテーマはないため、更新された KEP の Milestone と参考リンクを記載しておきます。

- [KEP-2831: Kubelet Tracing](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2831-kubelet-tracing)
  - v1.27 で Beta になりました
  - [Traces For Kubernetes System Components | Kubernetes](https://kubernetes.io/docs/concepts/cluster-administration/system-traces/#kubelet-traces)
- [KEP-647: APIServer Tracing](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/647-apiserver-tracing)
  - v1.27 で Beta になりました
  - [Traces For Kubernetes System Components | Kubernetes](https://kubernetes.io/docs/concepts/cluster-administration/system-traces/#kube-apiserver-traces)
- [KEP-3466: Kubernetes Component SLIs](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3466-kubernetes-component-health-slis)
  - v1.27 で Beta になりました
  - [こちらの記事で](https://qiita.com/watawuwu/items/9259782239976825d287#kubernetes-%E3%82%B3%E3%83%B3%E3%83%9D%E3%83%BC%E3%83%8D%E3%83%B3%E3%83%88%E3%81%AE-service-level-indicator-sli--%E3%83%A1%E3%83%88%E3%83%AA%E3%82%AF%E3%82%B9)紹介しています
  - [Kubernetes Component SLI Metrics | Kubernetes](https://kubernetes.io/docs/reference/instrumentation/slis/)
- [KEP-3498: Extending Metrics Stability](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3498-extending-stability)
  - v1.27 で Beta になりました
  - [こちらの記事で](https://qiita.com/watawuwu/items/9259782239976825d287#metrics-stability-framework-%E3%81%AE%E6%8B%A1%E5%BC%B5)紹介しています
- [KEP-1748: Expose Pod Resource Request Metrics](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1748-pod-resource-metrics)
  - v1.27 で GA となりました

それでは `Metrics Changes` について説明していきます。
`メトリクスの変更` は、私が全ての変更点からメトリクスの変更に関連するものを抜粋した内容となります。

## メトリクスの変更（Metrics Changes）

#### component-base

<!-- - [ADD] A new client side metric `rest_client_request_retries_total` has been added that tracks the number of retries sent to the server, partitioned by status code, verb and host ([#108396](https://github.com/kubernetes/kubernetes/pull/108396), [@&#xFFF0;tkashem](https://github.com/tkashem)) -->
- [ADD] クライアント側でサーバーへの再試行回数を計測したメトリクスが追加されました([#108396](https://github.com/kubernetes/kubernetes/pull/108396), [@&#xFFF0;tkashem](https://github.com/tkashem))
  - Counter: `rest_client_request_retries_total`
    - Labels: {"code", "verb", "host"}

#### kube-apiserver

<!-- - [ADD] Added apiserver_envelope_encryption_invalid_key_id_from_status_total to measure number of times an invalid keyID is returned by the Status RPC call. ([#115846](https://github.com/kubernetes/kubernetes/pull/115846), [@&#xFFF0;ritazh](https://github.com/ritazh)) [SIG API Machinery and Auth] -->
- [ADD] Status RPC から無効な keyID が返された回数を計測するメトリクスが追加されました([#115846](https://github.com/kubernetes/kubernetes/pull/115846), [@&#xFFF0;ritazh](https://github.com/ritazh))
  - Counter: `apiserver_envelope_encryption_invalid_key_id_from_status_total`
    - Labels: {"provider_name", "error"}
<!-- - [ADD] Added apiserver_envelope_encryption_kms_operations_latency_seconds metric to measure the KMSv2 grpc calls latency. ([#115649](https://github.com/kubernetes/kubernetes/pull/115649), [@&#xFFF0;aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing] -->
- [ADD] KMSv2 GRPC サービスのレイテンシーを計測するメトリクスが追加されました([#115649](https://github.com/kubernetes/kubernetes/pull/115649), [@&#xFFF0;aramase](https://github.com/aramase))
  - Histogram: `apiserver_envelope_encryption_kms_operations_latency_seconds`
    - Labels: {"provider_name", "method_name", "grpc_status_code"}
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 18)
<!-- - [ADD] Kube-apiserver: errors decoding objects in etcd are now recorded in an `apiserver_storage_decode_errors_total` counter metric ([#114376](https://github.com/kubernetes/kubernetes/pull/114376), [@&#xFFF0;baomingwang](https://github.com/baomingwang)) [SIG API Machinery and Instrumentation] -->
- [ADD]  etcd でオブジェクトをデコードする際に発生したエラー回数を記録するメトリクスが追加されました([#114376](https://github.com/kubernetes/kubernetes/pull/114376), [@&#xFFF0;baomingwang](https://github.com/baomingwang))
  - Counter: `apiserver_storage_decode_errors_total`
    - Labels: {"resource"}
<!-- - [ADD] Introduced new metrics removing the redundant subsystem in kube-apiserver pod logs metrics and deprecate the original ones: -->
<!--   - kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total becomes kube_apiserver_pod_logs_backend_tls_failure_total -->
<!--   - kube_apiserver_pod_logs_pods_logs_insecure_backend_total becomes kube_apiserver_pod_logs_insecure_backend_total ([#114497](https://github.com/kubernetes/kubernetes/pull/114497), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- [ADD|DEPRECATED] 冗長なサブシステム名がメトリクスに含まれていたので削除し新しいメトリクスが追加されました。元のメトリクスは非推奨となります([#114497](https://github.com/kubernetes/kubernetes/pull/114497), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))
  - `kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total` => `kube_apiserver_pod_logs_backend_tls_failure_total`
  - `kube_apiserver_pod_logs_pods_logs_insecure_backend_total` => `kube_apiserver_pod_logs_insecure_backend_total`
<!-- - [MOD] Apiserver_storage_transformation_operations_total metric has been updated to include labels transformer_prefix and status. ([#115394](https://github.com/kubernetes/kubernetes/pull/115394), [@&#xFFF0;ritazh](https://github.com/ritazh)) [SIG API Machinery, Auth, Instrumentation and Testing] -->
- [ADD] KMSv2 関連のメトリクスが追加されました([#115394](https://github.com/kubernetes/kubernetes/pull/115394), [@&#xFFF0;ritazh](https://github.com/ritazh))
  - Counter: `apiserver_envelope_encryption_key_id_hash_total`
    - Labels: {"transformation_type", "provider_name", "key_id_hash"}
  - Histogram: `apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds`
    - Lables: {"transformation_type", "provider_name", "key_id_hash"}
  - Gauge: `apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds`
    - Labels: {"provider_name", "key_id_hash"}
- [MOD] `transformer_prefix` のラベルが追加されました ([#115394](https://github.com/kubernetes/kubernetes/pull/115394), [@&#xFFF0;ritazh](https://github.com/ritazh))
  - Histogram: `apiserver_storage_transformation_duration_seconds`
    - Labels: {"transformation_type", "transformer_prefix"}
    - Buckets: metrics.ExponentialBuckets(5e-6, 2, 25)
<!-- - [ADD] Changed metrics for aggregated discovery to publish new time series (alpha). ([#115630](https://github.com/kubernetes/kubernetes/pull/115630), [@&#xFFF0;Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing] -->
- [ADD] [Aggregated Discovery](https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/3352-aggregated-discovery/README.md) のメトリクスが追加されました ([#115630](https://github.com/kubernetes/kubernetes/pull/115630), [@&#xFFF0;Jefftree](https://github.com/Jefftree))
  - Counter: `aggregator_discovery_aggregation_count_total`
    - Labels: なし
<!-- - [MOD] Updated `kube-apiserver` SLO/SLI latency metrics to exclude priority & fairness queue wait times ([#116420](https://github.com/kubernetes/kubernetes/pull/116420), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim)) -->
- [MOD] SLO/SLI レイテンシーのメトリクスから APF の待ち時間が除外されました([#116420](https://github.com/kubernetes/kubernetes/pull/116420), [@&#xFFF0;andrewsykim](https://github.com/andrewsykim))
  - Histogram: `apiserver_request_slo_duration_seconds`
  - Histogram: `apiserver_request_sli_duration_seconds`
<!-- - [MOD] `apiserver_admission_webhook_admission_duration_seconds` buckets have been expanded, 25s is now the largest bucket size to match the webhook default timeout. ([#115802](https://github.com/kubernetes/kubernetes/pull/115802), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) [SIG API Machinery and Instrumentation] -->
- [MOD]  Webhook のタイムアウト秒数に合わせて 10 秒と 25 秒のバケットが追加されました([#115802](https://github.com/kubernetes/kubernetes/pull/115802), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Histogram: `apiserver_admission_webhook_admission_duration_seconds`
    - Labels: {"name", "type", "operation", "rejected"}
    - Buckets: {0.005, 0.025, 0.1, 0.5, 1.0, 2.5, 10, 25}

#### kubelet

<!-- - [ADD] Added `kubelet` Topology Manager metrics to track admission requests processed and occured admission errors. ([#115137](https://github.com/kubernetes/kubernetes/pull/115137), [@&#xFFF0;swatisehgal](https://github.com/swatisehgal)) -->
- [ADD] Topology Manager のアドミッションリクエスト数とアドミッションエラー数を計測するメトリクスが追加されました([#115137](https://github.com/kubernetes/kubernetes/pull/115137), [@&#xFFF0;swatisehgal](https://github.com/swatisehgal))
  - Counter: `kubelet_topology_manager_admission_requests_total`
    - Labels: なし
  - Counter: `kubelet_topology_manager_admission_errors_total`
    - Labels: なし
<!-- - [ADD] Added kubelet Topology Manager metric to measure topology manager admission latency. ([#115590](https://github.com/kubernetes/kubernetes/pull/115590), [@&#xFFF0;swatisehgal](https://github.com/swatisehgal)) -->
- [ADD] Topology Manager のアドミッションリクエストのレイテンシーを計測するメトリクスが追加されました([#115590](https://github.com/kubernetes/kubernetes/pull/115590), [@&#xFFF0;swatisehgal](https://github.com/swatisehgal))
  - Counter: `kubelet_topology_manager_admission_duration_ms`
    - Labels: なし
    - Buckets: metrics.ExponentialBuckets(.05, 2, 15)
    - Notes: 単位は混在させてない方針だと思うので、そのうち seconds に修正されるかと思います
<!-- - [ADD] Added metrics for volume reconstruction during kubelet startup. ([#115965](https://github.com/kubernetes/kubernetes/pull/115965), [@&#xFFF0;jsafrane](https://github.com/jsafrane)) [SIG Node and Storage] -->
- [ADD] [kubelet 起動時のボリューム再構築](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3756-volume-reconstruction/README.md)に関するメトリクスが追加されました([#115965](https://github.com/kubernetes/kubernetes/pull/115965), [@&#xFFF0;jsafrane](https://github.com/jsafrane))
  - Counter: `reconstruct_volume_operations_total`
    - Labels: なし
  - Counter: `reconstruct_volume_operations_errors_total`
    - Labels: なし
  - Counter: `force_cleaned_failed_volume_operations_total`
    - Labels: なし
  - Counter: `force_cleaned_failed_volume_operation_errors_total`
    - Labels: なし
<!-- - [ADD] Force deleted pods may fail to terminate until the kubelet is restarted when the container runtime returns an error during termination. We have strengthened testing for runtime failures and now perform a more rigorous reconciliation to ensure static pods (especially those that use fixed UIDs) are restarted.  As a side effect of these changes static pods will be restarted with lower latency than before (2s vs 4s, on average) and rapid updates to pod configuration should take effect sooner. -->
<!--   A new metric `kubelet_known_pods` has been added at ALPHA stability to report the number of pods a Kubelet is tracking in a number of internal states.  Operators may use the metrics to track an excess of pods in the orphaned state that may not be completing. ([#113145](https://github.com/kubernetes/kubernetes/pull/113145), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton)) [SIG API Machinery, Auth, Cloud Provider, Node and Testing] -->
- [ADD] 強制削除された Pod は終了中にコンテナランタイムでエラーが発生すると kubelet が再起動されるまで終了に失敗することがあります。kubelet が追跡している Pod の数を計測する新しいメトリクスにより、オペレーターは完了しない可能性のある Pod などがモニタリングできます([#113145](https://github.com/kubernetes/kubernetes/pull/113145), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton))
  - Counter: `kubelet_desired_pods`
    - Labels: {"static"}
  - Counter: `kubelet_active_pods`
    - Labels: {"static"}
  - Counter: `kubelet_mirror_pods`
    - Labels: なし
  - Counter: `kubelet_working_pods`
    - Labels: {"lifecycle", "config", "static"}
  - Counter: `kubelet_orphaned_runtime_pods_total`
    - Labels: なし
  - Counter: `kubelet_active_pods`
    - Labels: {"static"}
<!-- - [DEL] Removed the following deprecated metrics: -->
<!--   - node_collector_evictions_number replaced by node_collector_evictions_total -->
<!--   - scheduler_e2e_scheduling_duration_seconds replaced by scheduler_scheduling_attempt_duration_seconds ([#115209](https://github.com/kubernetes/kubernetes/pull/115209), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- [DEL] 非推奨のメトリクスが削除されました([#115209](https://github.com/kubernetes/kubernetes/pull/115209), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))
  - `node_collector_evictions_number`
    - 代わりに `node_collector_evictions_total` を利用してください
  - `scheduler_e2e_scheduling_duration_seconds`
    - 代わりに `scheduler_scheduling_attempt_duration_seconds` を利用してください

#### kube-controller-manager

<!-- - [ADD] HPA controller starts to expose metrics from the kube-controller-manager.\n- `reconciliations_total`: Number of reconciliation of HPA controller. \n- `reconciliation_duration_seconds`: The time(seconds) that the HPA controller takes to reconcile once. ([#116010](https://github.com/kubernetes/kubernetes/pull/116010), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
<!-- - [ADD] HPA controller exposes the following metrics from the kube-controller-manager. -->
<!--   - `metric_computation_duration_seconds`: The time(seconds) that the HPA controller takes to calculate one metric. -->
<!--   - `metric_computation_total`: Number of metric computations. ([#116326](https://github.com/kubernetes/kubernetes/pull/116326), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) [SIG Apps, Autoscaling and Instrumentation] -->
- [ADD] HPA コントローラーで以下のメトリクスが追加されました([#116010](https://github.com/kubernetes/kubernetes/pull/116010), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) ([#116326](https://github.com/kubernetes/kubernetes/pull/116326), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
  - Counter: `horizontal_pod_autoscaler_controller_reconciliations_total`
    - Labels: {"action", "error"}
  - Histogram: `horizontal_pod_autoscaler_controller_reconciliation_duration_seconds`
    - Labels: {"action", "error"}
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 15)
  - Counter: `horizontal_pod_autoscaler_controller_metric_computation_total`
    - Labels: {"action", "error", "metric_type"}
  - Histogram: `horizontal_pod_autoscaler_controller_metric_computation_duration_seconds`
    - Labels: {"action", "error", "metric_type"}
    - Buckets: metrics.ExponentialBuckets(0.001, 2, 15)
<!-- - [ADD] Node `ipam` controller now exposes metrics `cidrset_cidrs_max_total` and `multicidrset_cidrs_max_total` with information about the max number of CIDRs that can be allocated. ([#112260](https://github.com/kubernetes/kubernetes/pull/112260), [@&#xFFF0;aryan9600](https://github.com/aryan9600))
-->
- [ADD] Node IPAM コントローラーに割り当て可能な CIDER の最大数を計測するメトリクスが追加されました([#112260](https://github.com/kubernetes/kubernetes/pull/112260), [@&#xFFF0;aryan9600](https://github.com/aryan9600))
  - Gauge: `node_ipam_controller_cirdset_max_cidrs`
    - Labels: {"clusterCIDR"}
  - Gauge: `node_ipam_controller_multicirdset_max_cidrs`
    - Labels: {"clusterCIDR"}

#### kube-scheduler

<!-- - [ADD] New plugin_evaluation_total is added to the scheduler.This metric counts how many times the specific plugin affects the scheduling result. The metric does not get incremented when the plugin has nothing to do with an incoming Pod. ([#115082](https://github.com/kubernetes/kubernetes/pull/115082), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
- [ADD] プラグインがスケジューリング結果に何回影響を与えたかを計測するメトリクスが追加されました。無関係な Pod の場合、この値は増加しません。([#115082](https://github.com/kubernetes/kubernetes/pull/115082), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
  - Counter: `scheduler_plugin_evaluation_total`
    - Labels: {"plugin", "extension_point", "profile"}
<!-- - [MOD] The scheduler's metric `plugin_execution_duration_seconds` now records `PreEnqueue` plugins execution seconds. ([#116201](https://github.com/kubernetes/kubernetes/pull/116201), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
- [MOD] PreEnqueue プラグインの実行時間を含めるようになりました([#116201](https://github.com/kubernetes/kubernetes/pull/116201), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
  - Histogram: `scheduler_plugin_execution_duration_seconds`
<!-- - [MOD] Promoted pod resource `limit/request` metrics to stable. ([#115454](https://github.com/kubernetes/kubernetes/pull/115454), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- [MOD] 以下のメトリクスの Stability Level が Stable になりました([#115454](https://github.com/kubernetes/kubernetes/pull/115454), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))
  - Gauge: `kube_pod_resource_request`
  - Gauge: `kube_pod_resource_limit`

#### kube-proxy

- なし

#### その他

- `cAdvisor` が `v0.47.0` に更新されました([#114883](https://github.com/kubernetes/kubernetes/pull/114883), [@&#xFFF0;bobbypage](https://github.com/bobbypage))
  - [DEL] `container_accelerator_duty_cycle`
  - [DEL] `container_accelerator_memory_total_bytes`
  - [DEL] `container_accelerator_memory_used_bytes`

## メトリクス以外の変更点

<!-- - Graduated `KubeletTracing` to beta, which means that the feature gate is now enabled by default. ([#115750](https://github.com/kubernetes/kubernetes/pull/115750), [@&#xFFF0;saschagrunert](https://github.com/saschagrunert)) -->
- `KubeletTracing` フィーチャーゲートが Beta に移行になりデフォルトで有効になりました([#115750](https://github.com/kubernetes/kubernetes/pull/115750), [@&#xFFF0;saschagrunert](https://github.com/saschagrunert))
<!-- - Graduated the container resource metrics feature on `HPA` to beta. ([#116046](https://github.com/kubernetes/kubernetes/pull/116046), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
- `HPAContainerMetrics` フィーチャーゲートが Beta に移行しました([#116046](https://github.com/kubernetes/kubernetes/pull/116046), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
<!-- - K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message ([#114680](https://github.com/kubernetes/kubernetes/pull/114680), [@&#xFFF0;pohly](https://github.com/pohly)) -->
- go flag set で pflag 使用すると、ヘルプメッセージの出力時にパニックとなる問題が修正されました([#114680](https://github.com/kubernetes/kubernetes/pull/114680), [@&#xFFF0;pohly](https://github.com/pohly))
<!-- - `/metrics/slis` is now available for control plane components allowing you to scrape health check metrics. ([#114997](https://github.com/kubernetes/kubernetes/pull/114997), [@&#xFFF0;Richabanker](https://github.com/Richabanker)) -->
- コントロールプレーンのコンポーネントで `/metrics/slis` エンドポイントが利用可能となりました([#114997](https://github.com/kubernetes/kubernetes/pull/114997), [@&#xFFF0;Richabanker](https://github.com/Richabanker))
<!-- - `APIServerTracing` feature gate is now enabled by default. Tracing in the API Server is still disabled by default, and requires a config file to enable. ([#116144](https://github.com/kubernetes/kubernetes/pull/116144), [@&#xFFF0;dashpole](https://github.com/dashpole)) -->
- `APIServerTracing` フィーチャーゲートが Beta となりデフォルトで有効となりましたが、利用するためには設定で有効化する必要があります([#116144](https://github.com/kubernetes/kubernetes/pull/116144), [@&#xFFF0;dashpole](https://github.com/dashpole))
<!-- - `k8s.io/component-base/logs` now also supports adding command line flags to a `flag.FlagSet`. ([#114731](https://github.com/kubernetes/kubernetes/pull/114731), [@&#xFFF0;pohly](https://github.com/pohly)) -->
- `k8s.io/component-base/logs` で `flag.FlagSet` へのコマンドラインフラグの追加もサポートしました([#114731](https://github.com/kubernetes/kubernetes/pull/114731), [@&#xFFF0;pohly](https://github.com/pohly))
<!-- - Added logging-format option to CCMs based on `k8s.io/cloud-provider` ([#108984](https://github.com/kubernetes/kubernetes/pull/108984), [@&#xFFF0;LittleFox94](https://github.com/LittleFox94)) -->
- Cloud Controller Manager にロギングフォーマットのオプションが追加されました([#108984](https://github.com/kubernetes/kubernetes/pull/108984), [@&#xFFF0;LittleFox94](https://github.com/LittleFox94))
<!-- - Performance improvements in `klog` ([#115277](https://github.com/kubernetes/kubernetes/pull/115277), [@&#xFFF0;pohly](https://github.com/pohly)) -->
- klog のパフォーマンスが改善されました([#115277](https://github.com/kubernetes/kubernetes/pull/115277), [@&#xFFF0;pohly](https://github.com/pohly))
<!-- - Scheduler now skips `InterPodAffinity` Filter plugin when `InterPodAffinity` Filter plugin has nothing to do with a Pod. -->
<!--   It may affect some metrics values related to the `InterPodAffinity` Filter plugin. ([#114889](https://github.com/kubernetes/kubernetes/pull/114889), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
- `InterPodAffinity` フィルタープラグインが Pod と無関係な場合、Scheduler が `InterPodAffinity` フィルタープラグインをスキップするようになりました。 `InterPodAffinity` フィルタープラグインに関連するメトリクスの計測に影響を与える可能性があります([#114889](https://github.com/kubernetes/kubernetes/pull/114889), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
<!-- - From now on, the HPA controller will return an error for the container resource metrics when the feature gate `HPAContainerMetrics` is disabled. As a result, HPA with a container resource metric performs no scale-down and performs only. ([#116043](https://github.com/kubernetes/kubernetes/pull/116043), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) -->
- `HPAContainerMetrics` フィーチャーゲートが無効な場合に HPA コントローラーはコンテナリソースメトリクスでエラーを返すようになりました([#116043](https://github.com/kubernetes/kubernetes/pull/116043), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
<!-- - Improved misleading message, in case of no metrics received for the `HPA` controlled pods. ([#114740](https://github.com/kubernetes/kubernetes/pull/114740), [@&#xFFF0;kushagra98](https://github.com/kushagra98)) -->
- HPA で誤解を招くメッセージを改善しました([#114740](https://github.com/kubernetes/kubernetes/pull/114740), [@&#xFFF0;kushagra98](https://github.com/kushagra98))
<!-- - `pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go, pkg/controller/nodeipam/ipam/multi_cidr_range_allocator.go pkg/controller/nodeipam/ipam/range_allocator.go pkg/controller/nodelifecycle/node_lifecycle_controller.go` to structured logging ([#112670](https://github.com/kubernetes/kubernetes/pull/112670), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085)) -->
- 構造化ロギングに対応([#112670](https://github.com/kubernetes/kubernetes/pull/112670), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085))
- `kube-proxy` は `ContextualLogging`, `LoggingAlphaOptions`, `LoggingBetaOptions` に対応しました([#115233](https://github.com/kubernetes/kubernetes/pull/115233), [@&#xFFF0;pohly](https://github.com/pohly))
- [Contextual Logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) への移行
  - DaemonSet Controller[@&#xFFF0;249043822](https://github.com/249043822))
  - Controller Helper Functions ([#115049](https://github.com/kubernetes/kubernetes/pull/115049), [@&#xFFF0;fatsheep9146](https://github.com/fatsheep9146))
  - ResourceQuota Controller([#113315](https://github.com/kubernetes/kubernetes/pull/113315), [@&#xFFF0;ncdc](https://github.com/ncdc))
  - StatefulSet Controller([#113840](https://github.com/kubernetes/kubernetes/pull/113840), [@&#xFFF0;249043822](https://github.com/249043822))
  - ClusterRole Aggregation Controller ([#113910](https://github.com/kubernetes/kubernetes/pull/113910), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Deployment Controller([#113525](https://github.com/kubernetes/kubernetes/pull/113525), [@&#xFFF0;249043822](https://github.com/249043822))
  - ReplicaSet Controller([#114871](https://github.com/kubernetes/kubernetes/pull/114871), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001))
  - Bootstrap Signer Controller/Token Cleaner Controller([#113464](https://github.com/kubernetes/kubernetes/pull/113464), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Defaultbinder Scheduler Plugin([#116571](https://github.com/kubernetes/kubernetes/pull/116571), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - Kube-controller-manager Binary([#116529](https://github.com/kubernetes/kubernetes/pull/116529), [@&#xFFF0;pohly](https://github.com/pohly))
  - Namespace Controller([#113443](https://github.com/kubernetes/kubernetes/pull/113443), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085))
  - Service Account Controller([#114918](https://github.com/kubernetes/kubernetes/pull/114918), [@&#xFFF0;Namanl2001](https://github.com/Namanl2001))
  - Volume Attach/Detach Controller/PersistentVolumeClaim Protection Controller/PersistentVolume Protection Controller([#113584](https://github.com/kubernetes/kubernetes/pull/113584), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085))
  - TTL after finished Controller([#113916](https://github.com/kubernetes/kubernetes/pull/113916), [@&#xFFF0;songxiao-wang87](https://github.com/songxiao-wang87))
  - Cronjob Controller([#113428](https://github.com/kubernetes/kubernetes/pull/113428), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
  - GC Controller([#113986](https://github.com/kubernetes/kubernetes/pull/113986), [@&#xFFF0;songxiao-wang87](https://github.com/songxiao-wang87))
  - Kubernetes Object Garbage Collector([#113471](https://github.com/kubernetes/kubernetes/pull/113471), [@&#xFFF0;ncdc](https://github.com/ncdc))
  - TTlafterfinished Controller([#115332](https://github.com/kubernetes/kubernetes/pull/115332), [@&#xFFF0;obaranov1](https://github.com/obaranov1))
  - Sample-controller Controller([#113879](https://github.com/kubernetes/kubernetes/pull/113879), [@&#xFFF0;pchan](https://github.com/pchan))