Change:
Merging has broken e2e tests: https://k8s- Based on the logs it seems that HPA can no longer retrieve custom metrics provided by adapter (it still works for external metrics though). Quickly looking through the code it looks like updated to use Custom Metrics v1beta2 API. Since v1beta2 is not backward compatible with v1beta1 and there is no fallback to old API, I suspect HPA will no longer work with any pre-existing adapter implementing v1beta1 API (like the stackdriver adapter we use in tests). If my suspicion is correct I think this would be a huge regression - all existing adapters will stop working in 1.12. As per we can't just drop support for v1beta1 - we need to support it for 9 months / 3 releases after officially deprecating it. /kind bug /priority critical-urgent /sig autoscaling cc:
Yeah, I realized that yesterday. Let's get fallback logic merged.
and I will get a fix up today
Correct, I believe
Actually, we have conversions in place. We just need to switch back to v1beta1. No fallback need.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen It seems this issue is not fixed. and I were debugging issue with HPA scaling based on custom-metrics, where metrics are exposed via v1beta1 api. It looks that conversion is not working properly. Here is what we are seeing. Pod (custom-metrics-adapter) is exposing metrics using v1beta1 API. Exposed metrics value is 200. HPA controller reads metrics and performs conversion but value is 0 after conversion. Therefore no scaling up occurs While debugging we augmented method with logging: Here is the logging output: Note that other fields are also set to default values for given type after conversion. Even though we debugged using slightly different setup we are pretty sure that the same issue is reason for following test failures from https://k8s- [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with Custom Metric of type Pod from Stackdriver [Feature:CustomMetricsAutoscaling] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with Custom Metric of type Pod from Stackdriver with Prometheus [Feature:CustomMetricsAutoscaling] [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale up with two metrics of type Pod from Stackdriver [Feature:CustomMetricsAutoscaling]
Reopening this issue. <detailsIn response to : /reopen and I were debugging issue with HPA scaling based on custom-metrics, where metrics are exposed via v1beta1 api. It looks that conversion is not working properly. Here is what we are seeing. [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale up with two metrics of type Pod from Stackdriver [Feature:CustomMetricsAutoscaling] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Could this be causing it? Looks like there were a number fields missed in the conversion functions:
Looks promising. The was converted correctly (at least , is nil so hard to say). Other fields were not. I don't know what is the proper way to mix auto-generated conversion code with hand written one. But maybe if you provide hand-written conversion function for you have to handle all the fields.
Just a side note. If you will be fixing that could you add more verbose test coverage to ? It would be nice to have different objects tested (v1beta1 <-v1beta2) directly calling function.
I one testcase which shows failing scenario:
The first 2 pass. But the last one fails with being converted to
in your code it looks like the first 2 are converting while the 3rd is converting . Would you mind modifying the first 2 to still check but with a ?
I pasted it just for sake of showing the issue. It is not in commitable shape at all. As I mentioned earlier I think that we need more thorough coverage. And for sure more complete test objects. Not sure if we should test against internal API as it is somewhat implementation detail. But feel free to prove me wrong.
I've these conversions in In testing locally using an HPA and custom adapter running v1beta1, I am able to see the metric value correctly parsed: (opposed to the current version, which parses to 0): While I am limited in my access to run the Stackdriver tests, we do have improvements to the custom metrics tests in the works (see ), which should allow blocking e2e tests using a generic custom adapter but is ultimately dependent on working out the bugs in HPA we've introduced first. Please give my fix pr a review!

Discussion:
package v1beta1 import ( <ins> "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/conversion" "k8s.io/apimachinery/pkg/runtime" "k8s.io/metrics/pkg/apis/custom_metrics"

} func Convert_v1beta1_MetricValue_To_custom_metrics_MetricValue(in *MetricValue, out *custom_metrics.MetricValue, s conversion.Scope) error { <ins> out.TypeMeta = in.TypeMeta out.DescribedObject = custom_metrics.ObjectReference{ Kind:            in.DescribedObject.Kind, Namespace:       in.DescribedObject.Namespace, Name:            in.DescribedObject.Name, UID:             in.DescribedObject.UID, APIVersion:      in.DescribedObject.APIVersion, ResourceVersion: in.DescribedObject.ResourceVersion, FieldPath:       in.DescribedObject.FieldPath, } out.Timestamp = in.Timestamp out.WindowSeconds = in.WindowSeconds out.Value = in.Value </ins> out.Metric.Name = in.MetricName out.Metric.Selector = in.Selector return nil } func Convert_custom_metrics_MetricValue_To_v1beta1_MetricValue(in *custom_metrics.MetricValue, out *MetricValue, s conversion.Scope) error { <ins> out.TypeMeta = in.TypeMeta out.DescribedObject = v1.ObjectReference{ Kind:            in.DescribedObject.Kind, Namespace:       in.DescribedObject.Namespace, Name:            in.DescribedObject.Name, UID:             in.DescribedObject.UID, APIVersion:      in.DescribedObject.APIVersion, ResourceVersion: in.DescribedObject.ResourceVersion, FieldPath:       in.DescribedObject.FieldPath, } out.Timestamp = in.Timestamp out.WindowSeconds = in.WindowSeconds out.Value = in.Value </ins> out.MetricName = in.Metric.Name out.Selector = in.Metric.Selector return nil