Kubernetes 1.26 SIG Instrumentation の変更内容

- [Kubernetes 1.24 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/fe9b8e7f8d3089813020)
- [Kubernetes 1.25 SIG Instrumentation の変更内容](https://qiita.com/watawuwu/items/d4632f750437081265cb)

## はじめに

この記事では、Kubernetes 1.26 の [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md) から Metrics の変更と SIG Instrumentation の取り組みについてまとめています。

Kubernetes v1.26 で SIG Instrumentation が扱う主要なテーマに `Kubernetes メトリクスの改善(Improvements to Kubernetes metrics)` があり、注目すべきいくつかの改善が入っています。それではその内容を確認していきます。

### Metrics Stability Framework の拡張

Metrics Stability Framework の拡張が [導入（Alpha）](https://github.com/kubernetes/enhancements/issues/3498) されました。

- [KEP 3498-extending-stability](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3498-extending-stability)

v1.25 までの Stability Level は `ALPHA` と `STABLE` のみでしたが、v1.26 では `INTERNAL` と `BETA` が追加されました。

https://github.com/kubernetes/kubernetes/blob/v1.26.0/staging/src/k8s.io/component-base/metrics/opts.go#L65-L81

```go
// StabilityLevel represents the API guarantees for a given defined metric.
type StabilityLevel string

const (
	// INTERNAL metrics have no stability guarantees, as such, labels may
	// be arbitrarily added/removed and the metric may be deleted at any time.
	INTERNAL StabilityLevel = "INTERNAL"
	// ALPHA metrics have no stability guarantees, as such, labels may
	// be arbitrarily added/removed and the metric may be deleted at any time.
	ALPHA StabilityLevel = "ALPHA"
	// BETA metrics are governed by the deprecation policy outlined in by
	// the control plane metrics stability KEP.
	BETA StabilityLevel = "BETA"
	// STABLE metrics are guaranteed not be mutated and removal is governed by
	// the deprecation policy outlined in by the control plane metrics stability KEP.
	STABLE StabilityLevel = "STABLE"
)
```

Stability Level は Kubernetes 独自のメトリクスフレームで定義されたものです。Prometheus のメトリクスのヘルプテキストに `# HELP xxxxx [ALPHA] xxxx` のようなフォーマットで記録されます。Metrics Stability Framework 自体は Kubernetes v1.17 で Beta、v1.21 で Stable となっています。

```txt
メトリクスのヘルプテキストの例

# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
```

KEP を確認すると `INTERNAL` は内部開発用であり、どのバージョンでも自由に消せるようなので Kubernetes 利用者(Kubernetes Administrator/Kubernetes Application Developer)が利用することは避けた方が良さそうです。

`BETA` は `ALPHA` と `STABLE` の間の安定性です。モチベーションを見ると `ALPHA` をより初期段回のメトリクスである点を強調させ、今までの `ALPHA` 相当の安定性を `BETA` として扱う方向のようです。

`ALPHA` と `BETA` の Stability Level は Feature Gate のステージと同様に周期を決めて強制的にアップグレード、または非推奨となるような仕組みが検討されています。Alpha では未実装となり Beta で実装予定になります。

また今回のリリースでは Stability Level や Label が記載された[メトリクスのドキュメントも公開されました](https://kubernetes.io/docs/reference/instrumentation/metrics/) :tada:  
このドキュメントはコードから自動で生成されています。Kubernetes の Changelog だけではどうしても抜け漏れが発生するので、私も次のバージョンからは自動生成されたものを比較しメトリクスの記事を記載する予定です。

### Kubernetes コンポーネントの Service Level Indicator (SLI)  メトリクス

Kubernetes コンポーネントの Service Level Indicator (SLI)  メトリクスが導入(Alpha)されました。
`ComponentSLIs` フィーチャーゲートを有効にすると、ヘルスチェックのデータからメトリクスに変換された Service Level Objectives(SLO) を算出できるメトリクスのエンドポイント `/metrics/slis` が利用できるようになります。

- [KEP 3466-kubernetes-component-health-slis](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3466-kubernetes-component-health-slis)
- [Kubernetes Component SLI Metrics](https://kubernetes.io/docs/reference/instrumentation/slis/)

実際に `ComponentSLIs` フィーチャーゲートを有効にして `/metrics/slis` エンドポイントにアクセスすると `Cauge: kubernetes_healthcheck` と `Counter: kubernetes_healthchecks_total` の 2 つのメトリクスが取得できます。

<details>

<summary>kube-controller-manager の SLI メトリクス</summary>

```txt
# HELP kubernetes_healthcheck [ALPHA] This metric records the result of a single healthcheck.
# TYPE kubernetes_healthcheck gauge
kubernetes_healthcheck{name="attachdetach",type="healthz"} 1
kubernetes_healthcheck{name="bootstrapsigner",type="healthz"} 1
kubernetes_healthcheck{name="clusterrole-aggregation",type="healthz"} 1
kubernetes_healthcheck{name="cronjob",type="healthz"} 1
kubernetes_healthcheck{name="csrapproving",type="healthz"} 1
kubernetes_healthcheck{name="csrcleaner",type="healthz"} 1
kubernetes_healthcheck{name="csrsigning",type="healthz"} 1
kubernetes_healthcheck{name="daemonset",type="healthz"} 1
kubernetes_healthcheck{name="deployment",type="healthz"} 1
kubernetes_healthcheck{name="disruption",type="healthz"} 1
kubernetes_healthcheck{name="endpoint",type="healthz"} 1
kubernetes_healthcheck{name="endpointslice",type="healthz"} 1
kubernetes_healthcheck{name="endpointslicemirroring",type="healthz"} 1
kubernetes_healthcheck{name="ephemeral-volume",type="healthz"} 1
kubernetes_healthcheck{name="garbagecollector",type="healthz"} 1
kubernetes_healthcheck{name="horizontalpodautoscaling",type="healthz"} 1
kubernetes_healthcheck{name="job",type="healthz"} 1
kubernetes_healthcheck{name="leaderElection",type="healthz"} 1
kubernetes_healthcheck{name="namespace",type="healthz"} 1
kubernetes_healthcheck{name="nodeipam",type="healthz"} 1
kubernetes_healthcheck{name="nodelifecycle",type="healthz"} 1
kubernetes_healthcheck{name="persistentvolume-binder",type="healthz"} 1
kubernetes_healthcheck{name="persistentvolume-expander",type="healthz"} 1
kubernetes_healthcheck{name="podgc",type="healthz"} 1
kubernetes_healthcheck{name="pv-protection",type="healthz"} 1
kubernetes_healthcheck{name="pvc-protection",type="healthz"} 1
kubernetes_healthcheck{name="replicaset",type="healthz"} 1
kubernetes_healthcheck{name="replicationcontroller",type="healthz"} 1
kubernetes_healthcheck{name="resourcequota",type="healthz"} 1
kubernetes_healthcheck{name="root-ca-cert-publisher",type="healthz"} 1
kubernetes_healthcheck{name="serviceaccount",type="healthz"} 1
kubernetes_healthcheck{name="statefulset",type="healthz"} 1
kubernetes_healthcheck{name="tokencleaner",type="healthz"} 1
kubernetes_healthcheck{name="ttl",type="healthz"} 1
kubernetes_healthcheck{name="ttl-after-finished",type="healthz"} 1
# HELP kubernetes_healthchecks_total [ALPHA] This metric records the results of all healthcheck.
# TYPE kubernetes_healthchecks_total counter
kubernetes_healthchecks_total{name="attachdetach",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="bootstrapsigner",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="clusterrole-aggregation",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="cronjob",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="csrapproving",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="csrcleaner",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="csrsigning",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="daemonset",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="deployment",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="disruption",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="endpoint",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="endpointslice",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="endpointslicemirroring",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="ephemeral-volume",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="garbagecollector",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="horizontalpodautoscaling",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="job",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="leaderElection",status="success",type="healthz"} 138
kubernetes_healthchecks_total{name="namespace",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="nodeipam",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="nodelifecycle",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="persistentvolume-binder",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="persistentvolume-expander",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="podgc",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="pv-protection",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="pvc-protection",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="replicaset",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="replicationcontroller",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="resourcequota",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="root-ca-cert-publisher",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="serviceaccount",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="statefulset",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="tokencleaner",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="ttl",status="success",type="healthz"} 137
kubernetes_healthchecks_total{name="ttl-after-finished",status="success",type="healthz"} 137
```

</details>

### フィーチャーゲートメトリクスの提供

Kubernetes コンポーネントごとにフィーチャーゲートの有効無効が確認できる `kubernetes_feature_enabled` メトリクスが利用できるようになりました。

<details>

https://github.com/kubernetes/kubernetes/pull/112690

```txt
# HELP kubernetes_feature_enabled [ALPHA] This metric records the data about the stage and enablement of a k8s feature.
# TYPE kubernetes_feature_enabled gauge
kubernetes_feature_enabled{name="APIListChunking",stage="BETA"} 1
kubernetes_feature_enabled{name="APIPriorityAndFairness",stage="BETA"} 1
kubernetes_feature_enabled{name="APIResponseCompression",stage="BETA"} 1
kubernetes_feature_enabled{name="APISelfSubjectReview",stage="ALPHA"} 0
kubernetes_feature_enabled{name="APIServerIdentity",stage="ALPHA"} 0
kubernetes_feature_enabled{name="APIServerTracing",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AdvancedAuditing",stage=""} 1
kubernetes_feature_enabled{name="AggregatedDiscoveryEndpoint",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AllAlpha",stage="ALPHA"} 0
kubernetes_feature_enabled{name="AllBeta",stage="BETA"} 0
kubernetes_feature_enabled{name="AnyVolumeDataSource",stage="BETA"} 1
kubernetes_feature_enabled{name="AppArmor",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManager",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManagerPolicyAlphaOptions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CPUManagerPolicyBetaOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="CPUManagerPolicyOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="CSIInlineVolume",stage=""} 1
kubernetes_feature_enabled{name="CSIMigration",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAWS",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAzureDisk",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationAzureFile",stage="BETA"} 1
kubernetes_feature_enabled{name="CSIMigrationGCE",stage=""} 1
kubernetes_feature_enabled{name="CSIMigrationPortworx",stage="BETA"} 0
kubernetes_feature_enabled{name="CSIMigrationRBD",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CSIMigrationvSphere",stage="BETA"} 1
kubernetes_feature_enabled{name="CSINodeExpandSecret",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CSIStorageCapacity",stage=""} 1
kubernetes_feature_enabled{name="CSIVolumeHealth",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ContainerCheckpoint",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ContextualLogging",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ControllerManagerLeaderMigration",stage=""} 1
kubernetes_feature_enabled{name="CronJobTimeZone",stage="BETA"} 1
kubernetes_feature_enabled{name="CustomCPUCFSQuotaPeriod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="CustomResourceValidationExpressions",stage="BETA"} 1
kubernetes_feature_enabled{name="DaemonSetUpdateSurge",stage=""} 1
kubernetes_feature_enabled{name="DelegateFSGroupToCSIDriver",stage="BETA"} 1
kubernetes_feature_enabled{name="DevicePlugins",stage="BETA"} 1
kubernetes_feature_enabled{name="DisableAcceleratorUsageMetrics",stage=""} 1
kubernetes_feature_enabled{name="DisableCloudProviders",stage="ALPHA"} 0
kubernetes_feature_enabled{name="DisableKubeletCloudCredentialProviders",stage="ALPHA"} 0
kubernetes_feature_enabled{name="DownwardAPIHugePages",stage="BETA"} 1
kubernetes_feature_enabled{name="DryRun",stage=""} 1
kubernetes_feature_enabled{name="DynamicKubeletConfig",stage="DEPRECATED"} 0
kubernetes_feature_enabled{name="EfficientWatchResumption",stage=""} 1
kubernetes_feature_enabled{name="EndpointSliceTerminatingCondition",stage="BETA"} 1
kubernetes_feature_enabled{name="EphemeralContainers",stage=""} 1
kubernetes_feature_enabled{name="ExecProbeTimeout",stage=""} 1
kubernetes_feature_enabled{name="ExpandCSIVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandInUsePersistentVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandPersistentVolumes",stage=""} 1
kubernetes_feature_enabled{name="ExpandedDNSConfig",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ExperimentalHostUserNamespaceDefaulting",stage="BETA"} 0
kubernetes_feature_enabled{name="GRPCContainerProbe",stage="BETA"} 1
kubernetes_feature_enabled{name="GracefulNodeShutdown",stage="BETA"} 1
kubernetes_feature_enabled{name="GracefulNodeShutdownBasedOnPodPriority",stage="BETA"} 1
kubernetes_feature_enabled{name="HPAContainerMetrics",stage="ALPHA"} 0
kubernetes_feature_enabled{name="HPAScaleToZero",stage="ALPHA"} 0
kubernetes_feature_enabled{name="HonorPVReclaimPolicy",stage="ALPHA"} 0
kubernetes_feature_enabled{name="IPTablesOwnershipCleanup",stage="ALPHA"} 0
kubernetes_feature_enabled{name="IdentifyPodOS",stage=""} 1
kubernetes_feature_enabled{name="InTreePluginAWSUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginAzureDiskUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginAzureFileUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginGCEUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginPortworxUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginRBDUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="InTreePluginvSphereUnregister",stage="ALPHA"} 0
kubernetes_feature_enabled{name="JobMutableNodeSchedulingDirectives",stage="BETA"} 1
kubernetes_feature_enabled{name="JobPodFailurePolicy",stage="ALPHA"} 0
kubernetes_feature_enabled{name="JobReadyPods",stage="BETA"} 1
kubernetes_feature_enabled{name="JobTrackingWithFinalizers",stage="BETA"} 1
kubernetes_feature_enabled{name="KMSv2",stage="ALPHA"} 0
kubernetes_feature_enabled{name="KubeletCredentialProviders",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletInUserNamespace",stage="ALPHA"} 0
kubernetes_feature_enabled{name="KubeletPodResources",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletPodResourcesGetAllocatable",stage="BETA"} 1
kubernetes_feature_enabled{name="KubeletTracing",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LegacyServiceAccountTokenNoAutoGeneration",stage="BETA"} 1
kubernetes_feature_enabled{name="LocalStorageCapacityIsolation",stage=""} 1
kubernetes_feature_enabled{name="LocalStorageCapacityIsolationFSQuotaMonitoring",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LogarithmicScaleDown",stage="BETA"} 1
kubernetes_feature_enabled{name="LoggingAlphaOptions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="LoggingBetaOptions",stage="BETA"} 1
kubernetes_feature_enabled{name="MatchLabelKeysInPodTopologySpread",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MaxUnavailableStatefulSet",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MemoryManager",stage="BETA"} 1
kubernetes_feature_enabled{name="MemoryQoS",stage="ALPHA"} 0
kubernetes_feature_enabled{name="MinDomainsInPodTopologySpread",stage="BETA"} 0
kubernetes_feature_enabled{name="MixedProtocolLBService",stage="BETA"} 1
kubernetes_feature_enabled{name="MultiCIDRRangeAllocator",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NetworkPolicyEndPort",stage=""} 1
kubernetes_feature_enabled{name="NetworkPolicyStatus",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeInclusionPolicyInPodTopologySpread",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeOutOfServiceVolumeDetach",stage="ALPHA"} 0
kubernetes_feature_enabled{name="NodeSwap",stage="ALPHA"} 0
kubernetes_feature_enabled{name="OpenAPIEnums",stage="BETA"} 1
kubernetes_feature_enabled{name="OpenAPIV3",stage="BETA"} 1
kubernetes_feature_enabled{name="PodAndContainerStatsFromCRI",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodDeletionCost",stage="BETA"} 1
kubernetes_feature_enabled{name="PodDisruptionConditions",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodHasNetworkCondition",stage="ALPHA"} 0
kubernetes_feature_enabled{name="PodSecurity",stage=""} 1
kubernetes_feature_enabled{name="ProbeTerminationGracePeriod",stage="BETA"} 1
kubernetes_feature_enabled{name="ProcMountType",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ProxyTerminatingEndpoints",stage="ALPHA"} 0
kubernetes_feature_enabled{name="QOSReserved",stage="ALPHA"} 0
kubernetes_feature_enabled{name="ReadWriteOncePod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RecoverVolumeExpansionFailure",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RemainingItemCount",stage="BETA"} 1
kubernetes_feature_enabled{name="RemoveSelfLink",stage=""} 1
kubernetes_feature_enabled{name="RetroactiveDefaultStorageClass",stage="ALPHA"} 0
kubernetes_feature_enabled{name="RotateKubeletServerCertificate",stage="BETA"} 1
kubernetes_feature_enabled{name="SELinuxMountReadWriteOncePod",stage="ALPHA"} 0
kubernetes_feature_enabled{name="SeccompDefault",stage="BETA"} 1
kubernetes_feature_enabled{name="ServerSideApply",stage=""} 1
kubernetes_feature_enabled{name="ServerSideFieldValidation",stage="BETA"} 1
kubernetes_feature_enabled{name="ServiceIPStaticSubrange",stage="BETA"} 1
kubernetes_feature_enabled{name="ServiceInternalTrafficPolicy",stage="BETA"} 1
kubernetes_feature_enabled{name="SizeMemoryBackedVolumes",stage="BETA"} 1
kubernetes_feature_enabled{name="StatefulSetAutoDeletePVC",stage="ALPHA"} 0
kubernetes_feature_enabled{name="StatefulSetMinReadySeconds",stage=""} 1
kubernetes_feature_enabled{name="StorageVersionAPI",stage="ALPHA"} 0
kubernetes_feature_enabled{name="StorageVersionHash",stage="BETA"} 1
kubernetes_feature_enabled{name="TopologyAwareHints",stage="BETA"} 1
kubernetes_feature_enabled{name="TopologyManager",stage="BETA"} 1
kubernetes_feature_enabled{name="UserNamespacesStatelessPodsSupport",stage="ALPHA"} 0
kubernetes_feature_enabled{name="VolumeCapacityPriority",stage="ALPHA"} 0
kubernetes_feature_enabled{name="WatchBookmark",stage=""} 1
kubernetes_feature_enabled{name="WinDSR",stage="ALPHA"} 0
kubernetes_feature_enabled{name="WinOverlay",stage="BETA"} 1
kubernetes_feature_enabled{name="WindowsHostProcessContainers",stage="BETA"} 1
```

</details>

<br>
<br>
以上が v1.26 で SIG Instrumentation が扱う主要なテーマとなります。次は SIG Instrumentation に関連する変更点についてピックアップしたものを紹介します。

## メトリクスの追加

### kube-apiserver

<!-- - 'Priority and Fairness has introduced a new feature called _borrowing_ that allows an API priority level to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing for a certain priority level configuration object via the two newly introduced fields `lendablePercent`, and `borrowingLimitPercent` located under the `.spec.limited` field of the designated priority level. -->
<!--   This change added the following metrics: -->
<!--     - `apiserver_flowcontrol_nominal_limit_seats`: Nominal number of execution seats configured for each priority level -->
<!--     - `apiserver_flowcontrol_lower_limit_seats`: Configured lower bound on number of execution seats available to each priority level -->
<!--     - `apiserver_flowcontrol_upper_limit_seats`: Configured upper bound on number of execution seats available to each priority level -->
<!--     - `apiserver_flowcontrol_demand_seats`: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level) -->
<!--     - `apiserver_flowcontrol_demand_seats_high_watermark`: High watermark, over last adjustment period, of demand_seats -->
<!--     - `apiserver_flowcontrol_demand_seats_average`: Time-weighted average, over last adjustment period, of demand_seats -->
<!--     - `apiserver_flowcontrol_demand_seats_stdev`: Time-weighted standard deviation, over last adjustment period, of demand_seats -->
<!--     - `apiserver_flowcontrol_demand_seats_smoothed`: Smoothed seat demands -->
<!--     - `apiserver_flowcontrol_target_seats`: Seat allocation targets -->
<!--     - `apiserver_flowcontrol_seat_fair_frac`: Fair fraction of server's concurrency to allocate to each priority level that can use it -->
<!--     - `apiserver_flowcontrol_current_limit_seats`: current derived number of execution seats available to each priority level -->
<!--   The possibility of borrowing means that the old metric `apiserver_flowcontrol_request_concurrency_limit` can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit.' ([#113485](https://github.com/kubernetes/kubernetes/pull/113485), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer)) -->
- APF(Priority and Fairness) に `_borrowing_` という新しい機能が導入され、API Priority Level が他の Priority Level からシート数を借用できるようになりました。クラスタのオペレーターは、指定された Priority Level の `.spec.limited` フィールド配下に新しく導入された 2 つのフィールド `lendablePercent` と `borrowingLimitPercent` を利用して、特定の PriorityLevelConfiguration オブジェクトを借用できます。この機能の導入により以下のメトリクスが追加されました。また借用の可能性があるため、既存メトリクスである `apiserver_flowcontrol_request_concurrency_limit` は、設定された同時実行数の制限と強制された同時実行数の制限の両方ではなく、設定された同時実行数のみを記録します([#113485](https://github.com/kubernetes/kubernetes/pull/113485), [@&#xFFF0;MikeSpreitzer](https://github.com/MikeSpreitzer))。
    - `apiserver_flowcontrol_nominal_limit_seats`: Nominal number of execution seats configured for each priority level
    - `apiserver_flowcontrol_lower_limit_seats`: Configured lower bound on number of execution seats available to each priority level
    - `apiserver_flowcontrol_upper_limit_seats`: Configured upper bound on number of execution seats available to each priority level
    - `apiserver_flowcontrol_demand_seats`: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
    - `apiserver_flowcontrol_demand_seats_high_watermark`: High watermark, over last adjustment period, of demand_seats
    - `apiserver_flowcontrol_demand_seats_average`: Time-weighted average, over last adjustment period, of demand_seats
    - `apiserver_flowcontrol_demand_seats_stdev`: Time-weighted standard deviation, over last adjustment period, of demand_seats
    - `apiserver_flowcontrol_demand_seats_smoothed`: Smoothed seat demands
    - `apiserver_flowcontrol_target_seats`: Seat allocation targets
    - `apiserver_flowcontrol_seat_fair_frac`: Fair fraction of server's concurrency to allocate to each priority level that can use it
    - `apiserver_flowcontrol_current_limit_seats`: current derived number of execution seats available to each priority level



### kubelet
<!-- - A new `pod_status_sync_duration_seconds` histogram is reported at alpha metrics stability that estimates how long the Kubelet takes to write a pod status change once it is detected. ([#107896](https://github.com/kubernetes/kubernetes/pull/107896), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton)) [SIG Apps, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing] -->
- Pod ステータスの変更が検出されてから、その Pod が正常に更新されるまでの時間を記録するメトリクスが追加されました([#107896](https://github.com/kubernetes/kubernetes/pull/107896), [@&#xFFF0;smarterclayton](https://github.com/smarterclayton))
  - Histogram: `pod_status_sync_duration_seconds`
    - Labels: None
<!-- - Added kubelet metrics to track the cpumanager cpu allocation and pinning ([#112855](https://github.com/kubernetes/kubernetes/pull/112855), [@&#xFFF0;fromanirh](https://github.com/fromanirh)) -->
- CPU コアの専有(ピンニング) 割当数と失敗数を記録するメトリクスが追加されました([#112855](https://github.com/kubernetes/kubernetes/pull/112855), [@&#xFFF0;fromanirh](https://github.com/fromanirh))
  - Counter: `kubelet_cpu_manager_pinning_requests_total`
    - Labels: None
  - Counter: `kubelet_cpu_manager_pinning_errors_total`
    - Labels: None
<!-- - Added the metric `pod_start_sli_duration_seconds` to kubelet. ([#111930](https://github.com/kubernetes/kubernetes/pull/111930), [@&#xFFF0;azylinski](https://github.com/azylinski)) -->
- 全てのコンテナが開始され watch で観測できるまでの、イメージ取得と Init コンテナの実行時間を除いた Pod 開始までの時間（秒単位）を記録するメトリクスが追加されました([#111930](https://github.com/kubernetes/kubernetes/pull/111930), [@&#xFFF0;azylinski](https://github.com/azylinski))
  - Histogram: `kubelet_pod_start_sli_duration_seconds`
    - Labels: None

### kube-controller-manager
<!-- - Added new metric `job_controller_terminated_pods_tracking_finalizer` which can be used to monitor whether the job controller is removing Pod finalizers from terminated Pods after accounting them in Job status. ([#113176](https://github.com/kubernetes/kubernetes/pull/113176), [@&#xFFF0;alculquicondor](https://github.com/alculquicondor)) -->
- Finalizer に `batch.kubernetes.io/job-tracking` を持つ、フェーズが Failed、または Succeeded の Pod 数を記録します。ジョブステータスに計上した後に Pod の Finalizer が削除されているかどうかを監視するために使用できます([#113176](https://github.com/kubernetes/kubernetes/pull/113176), [@&#xFFF0;alculquicondor](https://github.com/alculquicondor))
  - Counter: `job_controller_terminated_pods_tracking_finalizer_total`
    - Labels: {"event"}
      - "add" or "delete"
<!-- - Adds metrics `force_delete_pods_total` and `force_delete_pod_errors_total` in the Pod GC Controller. ([#113519](https://github.com/kubernetes/kubernetes/pull/113519), [@&#xFFF0;xing-yang](https://github.com/xing-yang)) [SIG Apps] -->
- Pod GC コントローラーに、強制的に削除された Pod 数と、強制的に削除する際に発生したエラー数を記録するメトリクスが追加されました([#113519](https://github.com/kubernetes/kubernetes/pull/113519), [@&#xFFF0;xing-yang](https://github.com/xing-yang))
  - Counter: `pod_gc_collector_force_delete_pods_total`
    - Labels: None
  - Counter: `pod_gc_collector_force_delete_pod_errors_total`
    - Labels: None
<!-- - Extend the job `job_finished_total` metric by new `reason` label and introduce a new job metric to count pod failures  handled by pod failure policy with respect to the action applied. ([#113324](https://github.com/kubernetes/kubernetes/pull/113324), [@&#xFFF0;mimowo](https://github.com/mimowo)) [SIG Apps and Testing] -->
- podFailurePolicy で適用したアクション(FailJob|Ignore|Count)ごとに Pod Failure をカウントするメトリクスが追加されました([#113324](https://github.com/kubernetes/kubernetes/pull/113324), [@&#xFFF0;mimowo](https://github.com/mimowo))
  - Counter: `job_controller_pod_failures_handled_by_failure_policy_total`
    - Labels: {"action"}
<!-- - Metrics for `RetroactiveDefaultStorageClass` feature are now available. To see an attempt count for updating PVC retroactively with a default StorageClass see `retroactive_storageclass_total` metric and for total numer of errors see `retroactive_storageclass_errors_total`. ([#113323](https://github.com/kubernetes/kubernetes/pull/113323), [@&#xFFF0;RomanBednar](https://github.com/RomanBednar)) -->
- `RetroactiveDefaultStorageClass` フィーチャーゲートのメトリクスが利用できるようになりました ([#113323](https://github.com/kubernetes/kubernetes/pull/113323), [@&#xFFF0;RomanBednar](https://github.com/RomanBednar))
  - Counter: `retroactive_storageclass_total`
    - Labels: None
  - Counter: `retroactive_storageclass_errors_total`
    - Labels: None

### kube-scheduler
<!-- - The `goroutines` metric is newly added in the scheduler. It replaces `scheduler_goroutines` metric and it counts the number of goroutine in more places than `scheduler_goroutine` does. ([#112003](https://github.com/kubernetes/kubernetes/pull/112003), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) [SIG Instrumentation and Scheduling] -->
- kube-scheduler に goroutine を記録する以下のメトリクスが追加されました。既存の `scheduler_scheduler_goroutines` メトリクスより多くの場所で goroutine をカウントします([#112003](https://github.com/kubernetes/kubernetes/pull/112003), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))
  - Gauge: `scheduler_goroutines`
    - Labels: {"operation"}

### その他
<!-- - Added `kubernetes_feature_enabled` metric series to track whether each active feature gate is enabled. ([#112690](https://github.com/kubernetes/kubernetes/pull/112690), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
<!-- - Added a `kubernetes_feature_enabled` metric which will tell you if a feature is enabled. ([#112652](https://github.com/kubernetes/kubernetes/pull/112652), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
- Kubernetes コンポーネントごとにアクティブなフィーチャーゲートの有効無効を記録するメトリクスが追加されました([#112690](https://github.com/kubernetes/kubernetes/pull/112690), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) ([#112652](https://github.com/kubernetes/kubernetes/pull/112652), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Gauge: `kubernetes_feature_enabled`
    - Labels: {"name", "stage"}
<!-- - `registered_metric_total` will now report the number of metrics broken down bystability level and deprecated version. ([#112907](https://github.com/kubernetes/kubernetes/pull/112907), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
- Kubernetes メトリクスフレームワークメトリクスの Deprecated Version と Stability Level ごとにカウントするメトリクスが追加されました([#112907](https://github.com/kubernetes/kubernetes/pull/112907), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - Counter: `registered_metric_total`
    - Labels: {"stability_level", "deprecated_version"}
<!-- - Added new Golang runtime-related metrics to Kubernetes components: -->
<!--   - go_gc_cycles_automatic_gc_cycles_total -->
<!--   - go_gc_cycles_forced_gc_cycles_total -->
<!--   - go_gc_cycles_total_gc_cycles_total -->
<!--   - go_gc_heap_allocs_by_size_bytes -->
<!--   - go_gc_heap_allocs_bytes_total -->
<!--   - go_gc_heap_allocs_objects_total -->
<!--   - go_gc_heap_frees_by_size_bytes -->
<!--   - go_gc_heap_frees_bytes_total -->
<!--   - go_gc_heap_frees_objects_total -->
<!--   - go_gc_heap_goal_bytes -->
<!--   - go_gc_heap_objects_objects -->
<!--   - go_gc_heap_tiny_allocs_objects_total -->
<!--   - go_gc_pauses_seconds -->
<!--   - go_memory_classes_heap_free_bytes -->
<!--   - go_memory_classes_heap_objects_bytes -->
<!--   - go_memory_classes_heap_released_bytes -->
<!--   - go_memory_classes_heap_stacks_bytes -->
<!--   - go_memory_classes_heap_unused_bytes -->
<!--   - go_memory_classes_metadata_mcache_free_bytes -->
<!--   - go_memory_classes_metadata_mcache_inuse_bytes -->
<!--   - go_memory_classes_metadata_mspan_free_bytes -->
<!--   - go_memory_classes_metadata_mspan_inuse_bytes -->
<!--   - go_memory_classes_metadata_other_bytes -->
<!--   - go_memory_classes_os_stacks_bytes -->
<!--   - go_memory_classes_other_bytes -->
<!--   - go_memory_classes_profiling_buckets_bytes -->
<!--   - go_memory_classes_total_bytes -->
<!--   - go_sched_goroutines_goroutines -->
<!--   - go_sched_latencies_seconds ([#111910](https://github.com/kubernetes/kubernetes/pull/111910), [@&#xFFF0;tosi3k](https://github.com/tosi3k)) -->
- Go の ランタイムに関連するメトリクスが追加されました([#111910](https://github.com/kubernetes/kubernetes/pull/111910), [@&#xFFF0;tosi3k](https://github.com/tosi3k))
  - `go_gc_cycles_automatic_gc_cycles_total`
  - `go_gc_cycles_forced_gc_cycles_total`
  - `go_gc_cycles_total_gc_cycles_total`
  - `go_gc_heap_allocs_by_size_bytes`
  - `go_gc_heap_allocs_bytes_total`
  - `go_gc_heap_allocs_objects_total`
  - `go_gc_heap_frees_by_size_bytes`
  - `go_gc_heap_frees_bytes_total`
  - `go_gc_heap_frees_objects_total`
  - `go_gc_heap_goal_bytes`
  - `go_gc_heap_objects_objects`
  - `go_gc_heap_tiny_allocs_objects_total`
  - `go_gc_pauses_seconds`
  - `go_memory_classes_heap_free_bytes`
  - `go_memory_classes_heap_objects_bytes`
  - `go_memory_classes_heap_released_bytes`
  - `go_memory_classes_heap_stacks_bytes`
  - `go_memory_classes_heap_unused_bytes`
  - `go_memory_classes_metadata_mcache_free_bytes`
  - `go_memory_classes_metadata_mcache_inuse_bytes`
  - `go_memory_classes_metadata_mspan_free_bytes`
  - `go_memory_classes_metadata_mspan_inuse_bytes`
  - `go_memory_classes_metadata_other_bytes`
  - `go_memory_classes_os_stacks_bytes`
  - `go_memory_classes_other_bytes`
  - `go_memory_classes_profiling_buckets_bytes`
  - `go_memory_classes_total_bytes`
  - `go_sched_goroutines_goroutines`
  - `go_sched_latencies_seconds`

## メトリクスの変更

### kube-apiserver
<!-- - The metric `etcd_db_total_size_in_bytes` is renamed to `apiserver_storage_db_total_size_in_bytes`. ([#113310](https://github.com/kubernetes/kubernetes/pull/113310), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) [SIG API Machinery] -->
- 以下のメトリクスがリネームされました([#113310](https://github.com/kubernetes/kubernetes/pull/113310), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - `etcd_db_total_size_in_bytes` => `apiserver_storage_db_total_size_in_bytes`

### kube-controller-manager
<!-- - Extend the job `job_finished_total` metric by new `reason` label and introduce a new job metric to count pod failures  handled by pod failure policy with respect to the action applied. ([#113324](https://github.com/kubernetes/kubernetes/pull/113324), [@&#xFFF0;mimowo](https://github.com/mimowo)) [SIG Apps and Testing] -->
- 以下のメトリクスに `reason` ラベルが追加されました([#113324](https://github.com/kubernetes/kubernetes/pull/113324), [@&#xFFF0;mimowo](https://github.com/mimowo))
  - Counter: `job_controller_job_finished_total`
    - Labels: {"completion_mode", "reason", "result"}
<!-- - Promoted `cronjob_job_creation_skew` metric to stable to follow the cronjob v2 controller, the following metrics had their name updated to match metrics API guidelines: `cronjob_job_creation_skew_duration_seconds` -> `job_creation_skew_duration_seconds`.' ([#113008](https://github.com/kubernetes/kubernetes/pull/113008), [@&#xFFF0;soltysh](https://github.com/soltysh)) -->
- 以下のメトリクスがリネームされ、Stability Level が Stable になりました([#113008](https://github.com/kubernetes/kubernetes/pull/113008), [@&#xFFF0;soltysh](https://github.com/soltysh))
  - `cronjob_controller_cronjob_job_creation_skew` => `cronjob_controller_cronjob_job_creation_skew_duration_seconds` 
<!-- - Promoted job-related metrics to stable to follow IndexedJobs GA. The following metrics have their name updated to match metrics API guidelines: -->
<!--   - `job_sync_total` -> `job_syncs_total` -->
<!--   - `job_finished_total` -> `jobs_finished_total`' ([#113010](https://github.com/kubernetes/kubernetes/pull/113010), [@&#xFFF0;soltysh](https://github.com/soltysh)) -->
- 以下のメトリクスがリネームされ、Stability Level が Stable になりました([#113010](https://github.com/kubernetes/kubernetes/pull/113010), [@&#xFFF0;soltysh](https://github.com/soltysh))
  - `job_controller_job_sync_total`     -> `job_controller_job_syncs_total`
  - `job_controller_job_finished_total` -> `job_controller_jobs_finished_total`

### kubelet
<!-- - `kubelet_kubelet_credential_provider_plugin_duration` was renamed to `kubelet_credential_provider_plugin_duration` and `kubelet_kubelet_credential_provider_plugin_errors` was renamed to `kubelet_credential_provider_plugin_errors` ([#113754](https://github.com/kubernetes/kubernetes/pull/113754), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
- 以下のメトリクスがリネームされました([#113754](https://github.com/kubernetes/kubernetes/pull/113754), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - `kubelet_kubelet_credential_provider_plugin_duration` => `kubelet_credential_provider_plugin_duration`
  - `kubelet_kubelet_credential_provider_plugin_errors` => `kubelet_credential_provider_plugin_errors`

### kube-scheduler
<!-- - Changed `preemption_victims` metric bucket from `LinearBuckets` to `ExponentialBuckets`. ([#112939](https://github.com/kubernetes/kubernetes/pull/112939), [@&#xFFF0;lengrongfu](https://github.com/lengrongfu)) -->
- `scheduler_preemption_victims` メトリクスのバケットが `LinearBuckets` から `ExponentialBuckets` に変更となりました([#112939](https://github.com/kubernetes/kubernetes/pull/112939), [@&#xFFF0;lengrongfu](https://github.com/lengrongfu))

## メトリクスの非推奨

### kube-apiserver

<!-- - Deprecated the `apiserver_request_slo_duration_seconds` metric for v1.27 in favor of `apiserver_request_sli_duration_seconds` for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. ([#112679](https://github.com/kubernetes/kubernetes/pull/112679), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet)) -->
- `apiserver_request_slo_duration_seconds` メトリクスは v1.27 で非推奨となる予定です。一定の命名規則に従うため追加された `apiserver_request_sli_duration_seconds` メトリクスを利用してください([#112679](https://github.com/kubernetes/kubernetes/pull/112679), [@&#xFFF0;dgrisonnet](https://github.com/dgrisonnet))

### kube-scheduler
<!-- - The `goroutines` metric is newly added in the scheduler. It replaces `scheduler_goroutines` metric and it counts the number of goroutine in more places than `scheduler_goroutine` does. ([#112003](https://github.com/kubernetes/kubernetes/pull/112003), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho)) [SIG Instrumentation and Scheduling] -->
- `scheduler_scheduler_goroutine` は非推奨となりました。代わりに `scheduler_goroutine` メトリクスを利用してください([#112003](https://github.com/kubernetes/kubernetes/pull/112003), [@&#xFFF0;sanposhiho](https://github.com/sanposhiho))

## メトリクスの削除

なし。

## メトリクスの修正

### kube-apiserver
<!-- - Kube-apiserver: `DELETECOLLECTION API` requests are now recorded in metrics with the correct verb. ([#113133](https://github.com/kubernetes/kubernetes/pull/113133), [@&#xFFF0;sxllwx](https://github.com/sxllwx)) -->
- DeleteCollection 時に正しい verb(delete_collection) で記録されるように修正されました([#113133](https://github.com/kubernetes/kubernetes/pull/113133), [@&#xFFF0;sxllwx](https://github.com/sxllwx))
<!-- - The metrics `etcd_request_duration_seconds` and `etcd_bookmark_counts` now differentiate by group resource instead of object type, allowing unique entries per `CustomResourceDefinition`, instead of grouping them all under `*unstructured.Unstructured`. ([#112042](https://github.com/kubernetes/kubernetes/pull/112042), [@&#xFFF0;ncdc](https://github.com/ncdc)) -->
- `etcd_request_duration_seconds` と `etcd_bookmark_counts` メトリクスは、オブジェクトタイプではなく GroupResource で区別するようになりました。これまで `*unstructured.Unstructured` ですべてグループ化していましたが、`CustomResourceDefinition` ごとに記録されるようになりました([#112042](https://github.com/kubernetes/kubernetes/pull/112042), [@&#xFFF0;ncdc](https://github.com/ncdc))

### kube-controller-manager
<!-- - Fixed the occasional double-counting of the `job_finished_total` metric. ([#112948](https://github.com/kubernetes/kubernetes/pull/112948), [@&#xFFF0;mimowo](https://github.com/mimowo)) -->
- `job_finished_total` メトリクスが二重にカウントされていた不具合を修正しました([#112948](https://github.com/kubernetes/kubernetes/pull/112948), [@&#xFFF0;mimowo](https://github.com/mimowo))

### kube-scheduler
<!-- - The time duration of a failed or unschedulable scheduling attempt will be longer, it now includes the time duration of the unreserve operation. ([#113113](https://github.com/kubernetes/kubernetes/pull/113113), [@&#xFFF0;kerthcet](https://github.com/kerthcet)) -->
- `scheduler_scheduling_attempt_duration_seconds` メトリクスが正しく記録されるように修正されました([#113113](https://github.com/kubernetes/kubernetes/pull/113113), [@&#xFFF0;kerthcet](https://github.com/kerthcet))

## そのほか
<!-- - If `ComponentSLIs` feature gate is enabled, then `/metrics/slis` becomes available on kubelet, allowing you to scrape health check metrics. ([#113030](https://github.com/kubernetes/kubernetes/pull/113030), [@&#xFFF0;Richabanker](https://github.com/Richabanker)) [SIG Node] -->
<!-- - Introduce `ComponentSLIs` alpha feature-gate for component SLIs metrics endpoint. ([#112884](https://github.com/kubernetes/kubernetes/pull/112884), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) [SIG API Machinery] -->
<!-- - Exposed health check SLI metrics on `metrics/slis` for apiserver. ([#112741](https://github.com/kubernetes/kubernetes/pull/112741), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
<!-- - When `ComponentSLIs` feature gate is enabled, then `/metrics/slis` becomes available on kube-controller-manager, allowing you to scrape health check metrics. ([#112978](https://github.com/kubernetes/kubernetes/pull/112978), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) -->
<!-- - If `ComponentSLIs` feature gate is enabled, then `/metrics/slis` now becomes available on cloud-controller-manager allowing you to scrape health check metrics. ([#113340](https://github.com/kubernetes/kubernetes/pull/113340), [@&#xFFF0;Richabanker](https://github.com/Richabanker)) -->
<!-- - When `ComponentSLIs` feature gate is enabled, then `/metrics/slis` becomes available on `kube-proxy` allowing you to scrape health check metrics. ([#113057](https://github.com/kubernetes/kubernetes/pull/113057), [@&#xFFF0;Richabanker](https://github.com/Richabanker)) -->
- `ComponentSLIs` フィーチャーゲートの導入
  - kubelet([#113030](https://github.com/kubernetes/kubernetes/pull/113030), [@&#xFFF0;Richabanker](https://github.com/Richabanker))
  - kube-apiserver([#112884](https://github.com/kubernetes/kubernetes/pull/112884), [@&#xFFF0;logicalhan](https://github.com/logicalhan)) ([#112741](https://github.com/kubernetes/kubernetes/pull/112741), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - kube-controller-manager([#112978](https://github.com/kubernetes/kubernetes/pull/112978), [@&#xFFF0;logicalhan](https://github.com/logicalhan))
  - cloud-controller-manager([#113340](https://github.com/kubernetes/kubernetes/pull/113340), [@&#xFFF0;Richabanker](https://github.com/Richabanker))
  - kube-proxy([#113057](https://github.com/kubernetes/kubernetes/pull/113057), [@&#xFFF0;Richabanker](https://github.com/Richabanker))
<!-- - Reverted regression that prevented `client-go` latency metrics to be reported with a template URL to avoid label cardinality. ([#111752](https://github.com/kubernetes/kubernetes/pull/111752), [@&#xFFF0;aanm](https://github.com/aanm)) -->
- ラベルのカーディナリティが高くなることを避けるために、`client-go` のレイテンシーメトリクスが記録されないようにするリグレッションを元に戻しました([#111752](https://github.com/kubernetes/kubernetes/pull/111752), [@&#xFFF0;aanm](https://github.com/aanm))
<!-- - API Server Tracing now includes a variety of new spans and span events. ([#113172](https://github.com/kubernetes/kubernetes/pull/113172), [@&#xFFF0;dashpole](https://github.com/dashpole)) [SIG API Machinery, Architecture, Auth, Instrumentation, Network, Node and Scheduling] -->
- API Server Tracing に新しい Span および Span Event が追加されました([#113172](https://github.com/kubernetes/kubernetes/pull/113172), [@&#xFFF0;dashpole](https://github.com/dashpole))
<!-- - API Server tracing now includes the latency of authorization, priorityandfairness, impersonation, audit, and authentication filters. ([#113217](https://github.com/kubernetes/kubernetes/pull/113217), [@&#xFFF0;dashpole](https://github.com/dashpole)) -->
- API Server Tracing で認可の待ち時間、APF、なりすまし、監査、認証フィルタのレイテンシーがトラックされるようになりました([#113217](https://github.com/kubernetes/kubernetes/pull/113217), [@&#xFFF0;dashpole](https://github.com/dashpole))
<!-- - API Server tracing root span name for opentelemetry is changed from `KubernetesAPI` to `HTTP GET`. ([#112545](https://github.com/kubernetes/kubernetes/pull/112545), [@&#xFFF0;dims](https://github.com/dims)) -->
- API Server Tracing で Root Span 名(opentelemetry) を `KubernetesAPI` から `HTTP GET` に変更しました([#112545](https://github.com/kubernetes/kubernetes/pull/112545), [@&#xFFF0;dims](https://github.com/dims))
<!-- - Added alpha support for returning container and pod metrics from CRI, instead of cAdvsior. ([#113609](https://github.com/kubernetes/kubernetes/pull/113609), [@&#xFFF0;haircommander](https://github.com/haircommander)) -->
- cAdvisor の代わりに CRI で コンテナと Pod のメトリクスを生成する Alpha サポートを追加しました([#113609](https://github.com/kubernetes/kubernetes/pull/113609), [@&#xFFF0;haircommander](https://github.com/haircommander))
  - [KEP sig-node/2371-cri-pod-container-stats](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2371-cri-pod-container-stats)
<!-- - Fixed the `PodAndContainerStatsFromCRI` feature, instead of supplementing with stats from cAdvisor. ([#113291](https://github.com/kubernetes/kubernetes/pull/113291), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu)) -->
- `PodAndContainerStatsFromCRI` フィーチャーゲートが有効で CRI で Stats を実装していればその Stats を利用するように修正しました([#113291](https://github.com/kubernetes/kubernetes/pull/113291), [@&#xFFF0;mengjiao-liu](https://github.com/mengjiao-liu))
<!-- - Updated cAdvisor to v0.46.0. ([#113769](https://github.com/kubernetes/kubernetes/pull/113769), [@&#xFFF0;bobbypage](https://github.com/bobbypage)) -->
- cAdvisor を v0.46.0 に更新しました([#113769](https://github.com/kubernetes/kubernetes/pull/113769), [@&#xFFF0;bobbypage](https://github.com/bobbypage))
<!-- - Log messages and metrics for the watch cache are now keyed by `<resource>.<group>` instead of `go` struct type. This means e.g. that `*v1.Pod` becomes `pods`. Additionally, resources that come from `CustomResourceDefinitions` are displayed as the correct resource and group, instead of `*unstructured.Unstructured`. ([#111807](https://github.com/kubernetes/kubernetes/pull/111807), [@&#xFFF0;ncdc](https://github.com/ncdc)) -->
- Watch のキャッシュのログメッセージとメトリクスは、Go の構造体型ではなく `<resource>.<group>` をキーにするようになりました。例えば `*v1.Pod` の場合、`pods` のようになります。さらに `CustomResourceDefinitions` の場合、 `*unstructured.Unstructured` ではなく、正しいリソースとグループとして記録されます([#111807](https://github.com/kubernetes/kubernetes/pull/111807), [@&#xFFF0;ncdc](https://github.com/ncdc))
<!-- - Release-note ([#111708](https://github.com/kubernetes/kubernetes/pull/111708), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085)) [SIG Apps, Instrumentation and Network] -->
- klog.V(0).InfoS の代わりに klog.InfoS を使用するように修正しました。[kubernetes-sigs/logtools#2](https://github.com/kubernetes-sigs/logtools/issues/2) で述べられているように、パラメータとしてゼロを指定して V を呼び出すと無駄なオーバーヘッドが生じます([#111708](https://github.com/kubernetes/kubernetes/pull/111708), [@&#xFFF0;yangjunmyfm192085](https://github.com/yangjunmyfm192085))