Repository: kedacore/keda

Issue:
Connection error from keda-operator-metrics-apiserver logs
### Report

After updating KEDA to version 2.9.2 (via chart version 2.9.3) from version 2.8.1 the metrics apiserver logs shows connection issue from the keda operator service at port 9666 

### Expected Behavior

The keda metrics apiserver to be able to use gRPC connection to successfully get metrics from operator

### Actual Behavior

When I deployed the latest KEDA version 2.9.2 started seeing the below error in the metrics apiserver logs,

I0201 19:41:19.967541       1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.mynamespace.svc.cluster.local:9666"

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 502 Bad Gateway\\r\\n

### Steps to Reproduce the Problem

Update KEDA version from 2.8.1 to 2.9.2


### Logs from KEDA operator

```
I0201 19:41:19.967541       1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.<MYNAMESPACE>.svc.cluster.local:9666"

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 502 Bad Gateway\\r\\n
```


### KEDA Version

2.9.2

### Kubernetes Version

1.23

### Platform

Amazon Web Services

### Scaler Details

_No response_

### Anything else?

The keda service (13-keda-service.yaml) was introduced with the latest version referring to this issue https://github.com/kedacore/keda/issues/3920 as this service wasn't there in the older version

I can try KEDA_USE_METRICS_SERVICE_GRPC=false but as it will be deprecated, wanted to see how to resolve the issue

KEDA runs behind enterprise proxy and the values are set via HTTP_PROXY HTTPS_PROXY and NO_PROXY

Discussion:
@JorTurFer: Hello,
I can see a typo in the url, the message says `keda-operator..svc.cluster.local:9666` but it should be `keda-operator.KEDA_NAMESPACE.svc.cluster.local:9666`. 
The value is set here https://github.com/kedacore/charts/blob/v2.9.3/keda/templates/22-metrics-deployment.yaml#L109 
Could you check if that value is present? I think that the message you see is because there are missing configs after the update. If that line isn't provided, KEDA tries to generate it in fallback but KEDA namespace is required, and that variables is also missing @prashathsenthil: > Hello, I can see a typo in the url, the message says `keda-operator..svc.cluster.local:9666` but it should be `keda-operator.KEDA_NAMESPACE.svc.cluster.local:9666`. The value is set here https://github.com/kedacore/charts/blob/v2.9.3/keda/templates/22-metrics-deployment.yaml#L109 Could you check if that value is present? I think that the message you see is because there are missing configs after the update. If that line isn't provided, KEDA tries to generate it in fallback but KEDA namespace is required, and that variables is also missing

@JorTurFer its just a typo while I wrote it, it actually has the namespace name there ""address"="keda-operator.mynamespace.svc.cluster.local:9666""@JorTurFer: @zroubalik , Maybe we should exclude this communication from the proxied traffic, as it's internal traffic. I guess that we could explicitly set the values without the proxy to skip env vars configs
WDYT?@prashathsenthil: After adding cluster.local to NO_PROXY list, the 502 bad gateway error is resolved, however still seeing the below error in the metric server logs,

grpc: addrConn.createTransport failed to connect to "Addr": "keda-operator.mynamespace.svc.cluster.local:9666"@JorTurFer: > After adding cluster.local to NO_PROXY list, the 502 bad gateway error is resolved, however still seeing the below error in the metric server logs,
> 
> grpc: addrConn.createTransport failed to connect to "Addr": "keda-operator.mynamespace.svc.cluster.local:9666"

Sorry, I missed your reply 😢 

Is the error persistent? I ask because we print the failure message but we don't print the established connection message, if you see the error only once or twice, it's not a problem and it's normal.@jstefankowski: Has this been resolved ?  

After upgrading keda from 2.6.1 to 2.10.1 autoscaling is not working and seeing this error in apiserver log:

Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup keda-sa.keda.svc.cluster.local on 10.106.0.2:53: no such host"
1 client.go:93] keda_metrics_adapter/provider "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" 
1 provider.go:110] keda_metrics_adapter/provider "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "server"="keda-sa.keda.svc.cluster.local:9666"@JorTurFer: > Has this been resolved ?
 It is, yep.

Seeing that error during the startup could be normal if the operator is down, but it should be transitory.
Is your operator working?@jstefankowski: I'm new to Keda.  How can I confirm the operator is working correctly ? @zroubalik: Also, did you change anything in the configuration? ` keda-sa` in the part with the url doesn't seem right to me.@jstefankowski: No configuration changes whatsoever.

I think the url configured in metrics apiserver deployment should be the operator internal endpoint:  keda-sa.keda:9666 instead of keda-sa.keda.svc.cluster.local:9666.  

keda-sa is the operator service name. @jstefankowski: After changing the apiserver deployment metrics-service-address image arg sill getting the same error:
  lookup keda-sa.keda.svc on 10.108.0.2:53: no such host

Wondering if the --metrics-service-address argument format is wrong.   It is:  --metrics-service-address=keda-sa.keda.svc:9666'
which translates to:  <operator name>.<namespace>.svc:9666@jstefankowski: From operator logs and from testing keda by sending SQS messages keda is scaling out successfully.

2023-06-15T17:10:16Z INFO scaleexecutor Successfully updated ScaleTarget {"scaledobject.Name": "aws-sqs-queue-scaledobject-ds", "scaledObject.Namespace": "test-namespace", "scaleTarget.Name": "ds", "Original Replicas Count": 0, "New Replicas Count": 1}

Just trying to understand the root cause of the timeout errors and its potential impact on production deployments. @zroubalik: Imho there should be `keda-operator` instead of `keda-sa` - basically the name of K8s Service that fronts the operator Deployment.@zroubalik: > keda-sa is the operator service name

I missed this note.

So you did some changes because the original name is `keda-operator`@zroubalik: > From operator logs and from testing keda by sending SQS messages keda is scaling out successfully.
> 
> 2023-06-15T17:10:16Z INFO scaleexecutor Successfully updated ScaleTarget {"scaledobject.Name": "aws-sqs-queue-scaledobject-ds", "scaledObject.Namespace": "test-namespace", "scaleTarget.Name": "ds", "Original Replicas Count": 0, "New Replicas Count": 1}
> 
> Just trying to understand the root cause of the timeout errors and its potential impact on production deployments.

It won't scale to more than 1 replica@jstefankowski: @zroubalik  
Spot on.  I just realized, that's exactly the behavior, keda won't scale to more than 1 replica. 
Why is that ?@jstefankowski: > > keda-sa is the operator service name
> 
> I missed this note.
> 
> So you did some changes because the original name is `keda-operator`

I tried patching metrics-apiserver deployment by changing metric-service-address to the operator service endpoint which is 
keda-sa.keda:9666 but to no avail. @zroubalik: Well, in first place we should ask why is the operator service name changed? @jstefankowski: That I'm not sure (inherited cluster).  The operator name is set to equal service account hence keda-sa.  
--set operator.name=${service_account_name} \

Do you think non default operator service name is the culprit here ?@jstefankowski: @zroubalik Also, can you expand on why keda "won't scale to more than 1 replica" ?@Gershon-A: Does someone have a solution?
I facing the same issue ....@JorTurFer: @Gershon-A , this issue was closed 5 months ago, I suggest opening another issue to track your case. There are multiple reasons that can produce this issue@kekjoshi: does somone has the solution for it , facing the same issue 
address is - --metrics-service-address=keda-operator.keda.svc.cluster.local:9666
I0325 15:16:20.694598       1 main.go:111] "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.keda.svc.cluster.local:9666" "logger"="keda_metrics_adapter"
I0325 15:20:05.194649       1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"@WitoldSlawko: Facing same issue

```
E0528 14:31:51.217655       1 provider.go:90] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
```

any help ?@rafaelsoutto: @WitoldSlawko did you find any solutions? Facing the same error.

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d55292f61..937e5213f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -78,6 +78,7 @@ Here is an overview of all new **experimental** features:
 
 - **General**: Fix regression in fallback mechanism ([#4249](https://github.com/kedacore/keda/issues/4249))
 - **General**: Prevent a panic that might occur while refreshing a scaler cache ([#4092](https://github.com/kedacore/keda/issues/4092))
+- **Azure Pipelines Scaler**: Add new parameter to limit the jobs returned ([#4324](https://github.com/kedacore/keda/issues/4324))
 - **Azure Queue Scaler**: Fix azure queue length ([#4002](https://github.com/kedacore/keda/issues/4002))
 - **Azure Service Bus Scaler:** Use correct auth flows with pod identity ([#4026](https://github.com/kedacore/keda/issues/4026)|[#4123](https://github.com/kedacore/keda/issues/4123))
 - **Azure Service Bus Scaler:**: Improve way clients are created to reduce amount of ARM requests ([#4262](https://github.com/kedacore/keda/issues/4262))
@@ -106,6 +107,7 @@ New deprecation(s):
 - **General**: Compare error with `errors.Is` ([#4004](https://github.com/kedacore/keda/pull/4004))
 - **General**: Consolidate `GetMetrics` and `IsActive` to `GetMetricsAndActivity` for Azure Event Hub, Cron and External scalers ([#4015](https://github.com/kedacore/keda/issues/4015))
 - **General**: Improve test coverage in `pkg/util` ([#3871](https://github.com/kedacore/keda/issues/3871))
+- **General**: Metrics Server: print a message on successful connection to gRPC server ([#4190](https://github.com/kedacore/keda/issues/4190))
 - **General**: Pass deep copy object to scalers cache from the ScaledObject controller ([#4207](https://github.com/kedacore/keda/issues/4207))
 - **General**: Review CodeQL rules and enable it on PRs ([#4032](https://github.com/kedacore/keda/pull/4032))
 - **RabbitMQ Scaler:** Move from `streadway/amqp` to `rabbitmq/amqp091-go` ([#4004](https://github.com/kedacore/keda/pull/4039))
diff --git a/cmd/adapter/main.go b/cmd/adapter/main.go
index 93c17ad76..e650bd266 100644
--- a/cmd/adapter/main.go
+++ b/cmd/adapter/main.go
@@ -156,6 +156,11 @@ func (a *Adapter) makeProvider(ctx context.Context, globalHTTPTimeout time.Durat
 	prometheusServer := &prommetrics.PrometheusMetricServer{}
 	go func() { prometheusServer.NewServer(fmt.Sprintf(":%v", prometheusMetricsPort), prometheusMetricsPath) }()
 
+	stopCh := make(chan struct{})
+	if err := runScaledObjectController(ctx, mgr, handler, logger, externalMetricsInfo, externalMetricsInfoLock, maxConcurrentReconciles, stopCh, secretInformer.Informer().HasSynced); err != nil {
+		return nil, nil, err
+	}
+
 	logger.Info("Connecting Metrics Service gRPC client to the server", "address", metricsServiceAddr)
 	grpcClient, err := metricsservice.NewGrpcClient(metricsServiceAddr, a.SecureServing.ServerCert.CertDirectory)
 	if err != nil {
@@ -163,10 +168,6 @@ func (a *Adapter) makeProvider(ctx context.Context, globalHTTPTimeout time.Durat
 		return nil, nil, err
 	}
 
-	stopCh := make(chan struct{})
-	if err := runScaledObjectController(ctx, mgr, handler, logger, externalMetricsInfo, externalMetricsInfoLock, maxConcurrentReconciles, stopCh, secretInformer.Informer().HasSynced); err != nil {
-		return nil, nil, err
-	}
 	return kedaprovider.NewProvider(ctx, logger, handler, mgr.GetClient(), *grpcClient, useMetricsServiceGrpc, namespace, externalMetricsInfo, externalMetricsInfoLock), stopCh, nil
 }
 
@@ -210,6 +211,23 @@ func getWatchNamespace() (string, error) {
 	return ns, nil
 }
 
+// printWelcomeMsg prints welcome message during the start of the adater
+func printWelcomeMsg(cmd *Adapter) error {
+	clientset, err := cmd.DiscoveryClient()
+	if err != nil {
+		logger.Error(err, "not able to get Kubernetes version")
+		return err
+	}
+	version, err := clientset.ServerVersion()
+	if err != nil {
+		logger.Error(err, "not able to get Kubernetes version")
+		return err
+	}
+	kedautil.PrintWelcome(logger, kedautil.NewK8sVersion(version), "metrics server")
+
+	return nil
+}
+
 func main() {
 	ctx := ctrl.SetupSignalHandler()
 	var err error
@@ -254,26 +272,17 @@ func main() {
 		return
 	}
 
-	kedaProvider, stopCh, err := cmd.makeProvider(ctx, time.Duration(globalHTTPTimeoutMS)*time.Millisecond, controllerMaxReconciles)
+	err = printWelcomeMsg(cmd)
 	if err != nil {
-		logger.Error(err, "making provider")
 		return
 	}
-	cmd.WithExternalMetrics(kedaProvider)
 
-	clientset, err := cmd.DiscoveryClient()
-	if err != nil {
-		logger.Error(err, "not able to get Kubernetes version")
-		return
-	}
-	version, err := clientset.ServerVersion()
+	kedaProvider, stopCh, err := cmd.makeProvider(ctx, time.Duration(globalHTTPTimeoutMS)*time.Millisecond, controllerMaxReconciles)
 	if err != nil {
-		logger.Error(err, "not able to get Kubernetes version")
+		logger.Error(err, "making provider")
 		return
 	}
-	kubeVersion := kedautil.NewK8sVersion(version)
-
-	kedautil.PrintWelcome(logger, kubeVersion, "metrics server")
+	cmd.WithExternalMetrics(kedaProvider)
 
 	logger.Info(cmd.Message)
 	if err = cmd.Run(stopCh); err != nil {
diff --git a/pkg/metricsservice/client.go b/pkg/metricsservice/client.go
index 40d1d4179..ba6692b4f 100644
--- a/pkg/metricsservice/client.go
+++ b/pkg/metricsservice/client.go
@@ -107,3 +107,8 @@ func (c *GrpcClient) WaitForConnectionReady(ctx context.Context, logger logr.Log
 	}
 	return true
 }
+
+// GetServerURL returns url of the gRPC server this client is connected to
+func (c *GrpcClient) GetServerURL() string {
+	return c.connection.Target()
+}
diff --git a/pkg/provider/provider.go b/pkg/provider/provider.go
index 7f429a512..c6a575270 100644
--- a/pkg/provider/provider.go
+++ b/pkg/provider/provider.go
@@ -56,6 +56,8 @@ type KedaProvider struct {
 var (
 	logger            logr.Logger
 	promMetricsServer prommetrics.PrometheusMetricServer
+
+	grpcClientConnected bool
 )
 
 // NewProvider returns an instance of KedaProvider
@@ -72,6 +74,17 @@ func NewProvider(ctx context.Context, adapterLogger logr.Logger, scaleHandler sc
 	}
 	logger = adapterLogger.WithName("provider")
 	logger.Info("starting")
+
+	go func() {
+		if !grpcClient.WaitForConnectionReady(ctx, logger) {
+			grpcClientConnected = false
+			logger.Error(fmt.Errorf("timeout while waiting to establish gRPC connection to KEDA Metrics Service server"), "timeout", "server", grpcClient.GetServerURL())
+		} else if !grpcClientConnected {
+			grpcClientConnected = true
+			logger.Info("Connection to KEDA Metrics Service gRPC server has been successfully established", "server", grpcClient.GetServerURL())
+		}
+	}()
+
 	return provider
 }
 
@@ -93,9 +106,14 @@ func (p *KedaProvider) GetExternalMetric(ctx context.Context, namespace string,
 	// Get Metrics from Metrics Service gRPC Server
 	if p.useMetricsServiceGrpc {
 		if !p.grpcClient.WaitForConnectionReady(ctx, logger) {
-			logger.Error(fmt.Errorf("timeout while waiting to establish gRPC connection to KEDA Metrics Server"), "timeout")
+			grpcClientConnected = false
+			logger.Error(fmt.Errorf("timeout while waiting to establish gRPC connection to KEDA Metrics Service server"), "timeout", "server", p.grpcClient.GetServerURL())
 			return nil, err
 		}
+		if !grpcClientConnected {
+			grpcClientConnected = true
+			logger.Info("Connection to KEDA Metrics Service gRPC server has been successfully established", "server", p.grpcClient.GetServerURL())
+		}
 
 		// selector is in form: `scaledobject.keda.sh/name: scaledobject-name`
 		scaledObjectName := selector.Get(kedav1alpha1.ScaledObjectOwnerAnnotation)
diff --git a/pkg/scalers/azure_pipelines_scaler.go b/pkg/scalers/azure_pipelines_scaler.go
index 8d03e7e0d..ed9c3af7c 100644
--- a/pkg/scalers/azure_pipelines_scaler.go
+++ b/pkg/scalers/azure_pipelines_scaler.go
@@ -133,6 +133,7 @@ type azurePipelinesMetadata struct {
 	poolID                               int
 	targetPipelinesQueueLength           int64
 	activationTargetPipelinesQueueLength int64
+	jobsToFetch                          int64
 	scalerIndex                          int
 	requireAllDemands                    bool
 }
@@ -218,6 +219,15 @@ func parseAzurePipelinesMetadata(ctx context.Context, config *ScalerConfig, http
 		meta.demands = ""
 	}
 
+	meta.jobsToFetch = 250
+	if val, ok := config.TriggerMetadata["jobsToFetch"]; ok && val != "" {
+		jobsToFetch, err := strconv.ParseInt(val, 10, 64)
+		if err != nil {
+			return nil, fmt.Errorf("error parsing jobsToFetch: %w", err)
+		}
+		meta.jobsToFetch = jobsToFetch
+	}
+
 	meta.requireAllDemands = false
 	if val, ok := config.TriggerMetadata["requireAllDemands"]; ok && val != "" {
 		requireAllDemands, err := strconv.ParseBool(val)
@@ -322,7 +332,7 @@ func getAzurePipelineRequest(ctx context.Context, url string, metadata *azurePip
 }
 
 func (s *azurePipelinesScaler) GetAzurePipelinesQueueLength(ctx context.Context) (int64, error) {
-	url := fmt.Sprintf("%s/_apis/distributedtask/pools/%d/jobrequests", s.metadata.organizationURL, s.metadata.poolID)
+	url := fmt.Sprintf("%s/_apis/distributedtask/pools/%d/jobrequests?$top=%d", s.metadata.organizationURL, s.metadata.poolID, s.metadata.jobsToFetch)
 	body, err := getAzurePipelineRequest(ctx, url, s.metadata, s.httpClient)
 	if err != nil {
 		return -1, err



Test patch:
diff --git a/pkg/scalers/azure_pipelines_scaler_test.go b/pkg/scalers/azure_pipelines_scaler_test.go
index 872791a4c..058063d4b 100644
--- a/pkg/scalers/azure_pipelines_scaler_test.go
+++ b/pkg/scalers/azure_pipelines_scaler_test.go
@@ -27,7 +27,7 @@ var testAzurePipelinesMetadata = []parseAzurePipelinesMetadataTestData{
 	// empty
 	{"empty", map[string]string{}, true, testAzurePipelinesResolvedEnv, map[string]string{}},
 	// all properly formed
-	{"all properly formed", map[string]string{"organizationURLFromEnv": "AZP_URL", "personalAccessTokenFromEnv": "AZP_TOKEN", "poolID": "1", "targetPipelinesQueueLength": "1"}, false, testAzurePipelinesResolvedEnv, map[string]string{}},
+	{"all properly formed", map[string]string{"organizationURLFromEnv": "AZP_URL", "personalAccessTokenFromEnv": "AZP_TOKEN", "poolID": "1", "targetPipelinesQueueLength": "1", "jobsToFetch": "300"}, false, testAzurePipelinesResolvedEnv, map[string]string{}},
 	// using triggerAuthentication
 	{"using triggerAuthentication", map[string]string{"poolID": "1", "targetPipelinesQueueLength": "1"}, false, testAzurePipelinesResolvedEnv, map[string]string{"organizationURL": "https://dev.azure.com/sample", "personalAccessToken": "sample"}},
 	// using triggerAuthentication with personalAccessToken terminating in newline
@@ -40,6 +40,8 @@ var testAzurePipelinesMetadata = []parseAzurePipelinesMetadataTestData{
 	{"missing poolID", map[string]string{"organizationURLFromEnv": "AZP_URL", "personalAccessTokenFromEnv": "AZP_TOKEN", "poolID": "", "targetPipelinesQueueLength": "1"}, true, testAzurePipelinesResolvedEnv, map[string]string{}},
 	// activationTargetPipelinesQueueLength malformed
 	{"all properly formed", map[string]string{"organizationURLFromEnv": "AZP_URL", "personalAccessTokenFromEnv": "AZP_TOKEN", "poolID": "1", "targetPipelinesQueueLength": "1", "activationTargetPipelinesQueueLength": "A"}, true, testAzurePipelinesResolvedEnv, map[string]string{}},
+	// jobsToFetch malformed
+	{"jobsToFetch malformed", map[string]string{"organizationURLFromEnv": "AZP_URL", "personalAccessTokenFromEnv": "AZP_TOKEN", "poolID": "1", "targetPipelinesQueueLength": "1", "jobsToFetch": "test"}, true, testAzurePipelinesResolvedEnv, map[string]string{}},
 }
 
 var testJobRequestResponse = `{"count":2,"value":[{"requestId":890659,"queueTime":"2022-09-28T11:19:49.89Z","assignTime":"2022-09-28T11:20:29.5033333Z","receiveTime":"2022-09-28T11:20:32.0530499Z","lockedUntil":"2022-09-28T11:30:32.07Z","serviceOwner":"xxx","hostId":"xxx","scopeId":"xxx","planType":"Build","planId":"xxx","jobId":"xxx","demands":["kubectl","Agent.Version -gtVersion 2.182.1"],"reservedAgent":{"_links":{"self":{"href":"https://dev.azure.com/FOO/_apis/distributedtask/pools/44/agents/11735"},"web":{"href":"https://dev.azure.com/FOO/_settings/agentpools?view=jobs&poolId=44&agentId=11735"}},"id":11735,"name":"kube-scaledjob-5nlph-kzpgf","version":"2.210.1","osDescription":"Linux 5.4.0-1089-azure #94~18.04.1-Ubuntu SMP Fri Aug 5 12:34:50 UTC 2022","enabled":true,"status":"online","provisioningState":"Provisioned","accessPoint":"CodexAccessMapping"},"definition":{"_links":{"web":{"href":"https://dev.azure.com/FOO/1858395a-257e-4efd-bbc5-eb618128452b/_build/definition?definitionId=4869"},"self":{"href":"https://dev.azure.com/FOO/1858395a-257e-4efd-bbc5-eb618128452b/_apis/build/Definitions/4869"}},"id":4869,"name":"base - main"},"owner":{"_links":{"web":{"href":"https://dev.azure.com/FOO/1858395a-257e-4efd-bbc5-eb618128452b/_build/results?buildId=673584"},"self":{"href":"https://dev.azure.com/FOO/1858395a-257e-4efd-bbc5-eb618128452b/_apis/build/Builds/673584"}},"id":673584,"name":"20220928.2"},"data":{"ParallelismTag":"Private","IsScheduledKey":"False"},"poolId":44,"orchestrationId":"5c5c8ec9-786f-4e97-99d4-a29279befba3.build.__default","priority":0},{"requestId":890663,"queueTime":"2022-09-28T11:20:22.4633333Z","serviceOwner":"00025394-6065-48ca-87d9-7f5672854ef7","hostId":"41a18c7d-df5e-4032-a4df-d533b56bd2de","scopeId":"02696e26-a35b-424c-86b8-1f54e1b0b4b7","planType":"Build","planId":"b718cfed-493c-46be-a650-88fe762f75aa","jobId":"15b95994-59ec-5502-695d-0b93722883bd","demands":["dotnet60","java","Agent.Version -gtVersion 2.182.1"],"matchedAgents":[{"_links":{"self":{"href":"https://dev.azure.com/FOO/_apis/distributedtask/pools/44/agents/1755"},"web":{"href":"https://dev.azure.com/FOO/_settings/agentpools?view=jobs&poolId=44&agentId=1755"}},"id":1755,"name":"dotnet60-keda-template","version":"2.210.1","enabled":true,"status":"offline","provisioningState":"Provisioned"},{"_links":{"self":{"href":"https://dev.azure.com/FOO/_apis/distributedtask/pools/44/agents/11732"},"web":{"href":"https://dev.azure.com/FOO/_settings/agentpools?view=jobs&poolId=44&agentId=11732"}},"id":11732,"name":"dotnet60-scaledjob-5dsgc-pkqvm","version":"2.210.1","enabled":true,"status":"online","provisioningState":"Provisioned"},{"_links":{"self":{"href":"https://dev.azure.com/FOO/_apis/distributedtask/pools/44/agents/11733"},"web":{"href":"https://dev.azure.com/FOO/_settings/agentpools?view=jobs&poolId=44&agentId=11733"}},"id":11733,"name":"dotnet60-scaledjob-zgqnp-8h4z4","version":"2.210.1","enabled":true,"status":"online","provisioningState":"Provisioned"},{"_links":{"self":{"href":"https://dev.azure.com/FOO/_apis/distributedtask/pools/44/agents/11734"},"web":{"href":"https://dev.azure.com/FOO/_settings/agentpools?view=jobs&poolId=44&agentId=11734"}},"id":11734,"name":"dotnet60-scaledjob-wr65c-ff2cv","version":"2.210.1","enabled":true,"status":"online","provisioningState":"Provisioned"}],"definition":{"_links":{"web":{"href":"https://FOO.visualstudio.com/02696e26-a35b-424c-86b8-1f54e1b0b4b7/_build/definition?definitionId=3129"},"self":{"href":"https://FOO.visualstudio.com/02696e26-a35b-424c-86b8-1f54e1b0b4b7/_apis/build/Definitions/3129"}},"id":3129,"name":"Other Build CI"},"owner":{"_links":{"web":{"href":"https://FOO.visualstudio.com/02696e26-a35b-424c-86b8-1f54e1b0b4b7/_build/results?buildId=673585"},"self":{"href":"https://FOO.visualstudio.com/02696e26-a35b-424c-86b8-1f54e1b0b4b7/_apis/build/Builds/673585"}},"id":673585,"name":"20220928.11"},"data":{"ParallelismTag":"Private","IsScheduledKey":"False"},"poolId":44,"orchestrationId":"b718cfed-493c-46be-a650-88fe762f75aa.buildtest.build_and_test.__default","priority":0}]}`