Project: Katalyst (Orchestration & Management / Scheduling & Orchestration)
File: 20221202-katalyst-custom-metrics-apiserver.md

title: Katalyst Custom Metrics APIServer authors: "waynepeking348" reviewers: "luomingmeng" "xuchen-xiaoying" creation-date: 2022-12-02 last-updated: 2023-02-23 status: implemented <!-- toc --> - - - - <!-- /toc --> Katalyst Custom Metrics APIServer (KCMS for short) is responsible for collecting and storing customized metrics in the production environment, and it implements the standard custom-metrics-apiserver interface for inquiring. It is mainly used by elastic resource recommendation and re-scheduling in the katalyst system. Provide a customized solution to obtain real-time metrics for systems or modules in katalyst. Replace the native custom metrics server. Works as a general solution for observability. In katalyst, many centralized components may depend on customized real-time metrics as critical inputs to generate controlling strategies. Resource autoscaling (both horizontal and vertical) is a core functionality for cloud-native services. It is usually deployed as a centralized component, and it will dynamically adjust the resource requests for pod or workload level based on real-time running states. To achieve this, the autoscaling module needs to obtain metrics, both for general ones like cpu usage and customized ones like requests qps. Those metrics must be as fresh as possible, and it must be easier enough if we want to add new metrics for more intelligent recommendation strategies. Rescheulder or Descheduler is a must-to-have ability in a large cluster, since the running states may change frequently, especially when compared with the time when pods are scheduled. So katalyst will try to rebalance the global scheduling states based on real-time metrics periodically. And the requirements for time-efficiency, stability, reliability, and convenience for new metrics are almost the same as autoscaling. <div align="center"> <picture> <img src="/docs/imgs/custom-metrics-overview.jpg" width=80% title="Katalyst Overview" loading="eager" /> </picture> </div> There are four main components in KCMS. KCMS Agent is embedded as a module in SysAdvisor, and it's responsible for collecting metrics from multiple sources, including raw metrics obtained from malachite, encapsulated metrics generated by SysAdvisor and so on. Those metrics will be exported through standard Prometheus interface and opentelemetry framework. Each metric item will be kept for 5 minutes before GC. KCMS Collector scrapes metric items from KCMS Agent periodically (default for 5 seconds). It uses label selectors to match KCMS Agent, and always uses Node IP in KCMS Agent Pod as the quering endpoint. After scraping metric, KCMS Collector will send them to KCMS Store. KCMS Store is an in-memory cache for metrics. It receives metric items from KCMS Collector, and builds indexes in local cache for KCMS Server to query. Each metric item will be kept for 10 minutes before GC. KCMS Server is actually an extended APIServer, and implements the native custom-metrics-apiserver interface. Native APIServer passes through metric requests to KCMS Server according to the registered APIService, and then KCMS Server proxies the requests to KCMS Store, and returns the responded data list back to Native APIServer. KCMS implements the standard custom-metrics-apiserver interface, so users can use metricsClient to refer to those metrics without any extra efforts. For more information about custom-metrics-apiserver (including the basic concepts and common use cases), please refer to . ``` type CustomMetricsProvider interface { // GetMetricByName fetches a particular metric for a particular object. // The namespace will be empty if the metric is root-scoped. GetMetricByName(ctx context.Context, name

info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error) // GetMetricBySelector fetches a particular metric for a set of objects matching // the given label selector. The namespace will be empty if the metric is root-scoped. GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error) // ListAllMetrics provides a list of all available metrics at // the current time. Note that this is not allowed to return // an error, so it is reccomended that implementors cache and // periodically update this list, instead of querying every time. ListAllMetrics() []CustomMetricInfo } // ExternalMetricsProvider is a source of external metrics. // Metric is normally identified by a name and a set of labels/tags. It is up to a specific // implementation how to translate metricSelector to a filter for metric values. // Namespace can be used by the implemetation for metric identification, access control or ignored. type ExternalMetricsProvider interface { GetExternalMetric(ctx context.Context, namespace string, metricSelector labels.Selector, info ExternalMetricInfo) (*external_metrics.ExternalMetricValueList, error) ListAllExternalMetrics() []ExternalMetricInfo } type MetricsProvider interface { CustomMetricsProvider ExternalMetricsProvider } ``` To get information about the detailed custom and external metrics that katalyst supports, please refer to , and we plan to provide more metrics along with the implementation of autoscaling and recheduler. In this section, we will try to introduce the deployment modes that KCMS supports. In StandAlone mode, all centralized components (KCMS Collector, KCMS Store, and KCMS Server) all run in a monolith container. This means that all data dependency is achieved by function calls rather RPC calls. The benefit is performance is strong and deploying is convenient, but reliability is lacking. So it is usually used in testing environments, or in small clusters where it's acceptable to lose some metric. <div align="center"> <picture> <img src="/docs/imgs/custom-metrics-monolith-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> </picture> </div> In Multi Store mode, centralized components run in separated containers, and each component has multiple instances. For KCMS Collector, only one instance will be the leader while others are cold standbies. Since Collector is stateless, cold standby is enough in case the leader instance crashes and fails to restart. For KCMS Store, all instances share the same metric contents. KCMS Collector must perform Quorum write and KCMS Server must perform Quorum read to ensure reliability. For KCMS Serve, all instances are equal to handling querying requests since they are only proxies. Multi Store mode sacrifices some performance to win the most reliability, and is often used in production environments. <div align="center"> <picture> <img src="/docs/imgs/custom-metrics-multi-store-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> </picture> </div> Sharing mode is the future work in case metric items are too large to be stored in one instance. In this mode, both KCMS Collector and KCMS Store will be shared, and each instance will be responsible for an independent piece of items. To ensure reliability, each metric shard will still contain duplicated replicas. <div align="center"> <picture> <img src="/docs/imgs/custom-metrics-sharding-mode.jpg" width=80% title="Katalyst Overview" loading="eager" /> </picture> </div> Expand metrics in the native custom metrics server, but it doesn't support external metrics. Besides, it only contains several metrics in cadvisor, and the cost of adding new metrics is heavy, let along adding labels or something like that. And it doesn't consider reliability, time-efficiency, stability as its main functionalities.