Repository: cloud-custodian/cloud-custodian

Issue:
AWS Workspaces 'c7n:ConnectionStatus' Failing
### Describe the bug

```bash-4.2# custodian run -s ~/log -r eu-west-1 workspaces.yaml
2022-05-10 16:22:59,328: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 531, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.7/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.7/site-packages/c7n/resources/workspaces.py", line 89, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.7/site-packages/c7n/resources/workspaces.py", line 93, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-05-10 16:22:59,329: custodian.commands:ERROR Error while executing policy workspaces, continuing
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 1210, in __call__
    resources = mode.run()
  File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 531, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.7/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.7/site-packages/c7n/resources/workspaces.py", line 89, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.7/site-packages/c7n/resources/workspaces.py", line 93, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-05-10 16:22:59,331: custodian.commands:ERROR The following policies had errors while executing
```
```
bash-4.2# cat workspaces.yaml
policies:
  - name: workspaces
    resource: workspaces
    filters:
      - type: connection-status
        value_type: age
        key: LastKnownUserConnectionTimestamp
        op: ge
        value: 30
```

### What did you expect to happen?

When using the c7n:ConnectionStatus its failing with the error in the bug.

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

```shell
bash-4.2# custodian version --debug

Please copy/paste the following info along with any bug reports:

Custodian:   0.9.15
Python:      3.7.10 (default, Jun  3 2021, 00:01:07)
             [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
Platform:    posix.uname_result(sysname='Linux', nodename='f408cde285fa', release='4.14.231-173.361.amzn2.aarch64', version='#1 SMP Mon Apr 26 20:57:10 UTC 2021', machine='aarch64')
Using venv:  False
Docker: True
Installed:

argcomplete==2.0.0
attrs==21.4.0
boto3==1.22.10
botocore==1.25.10
docutils==0.17.1
importlib-metadata==4.11.3
importlib-resources==5.7.1
jmespath==0.10.0
jsonschema==4.5.1
pyrsistent==0.18.1
python-dateutil==2.8.2
pyyaml==6.0
s3transfer==0.5.2
six==1.16.0
tabulate==0.8.9
typing-extensions==4.2.0
urllib3==1.26.9
zipp==3.8.0
```


### Policy

```shell
bash-4.2# cat workspaces.yaml
policies:
  - name: workspaces
    resource: workspaces
    filters:
      - type: connection-status
        value_type: age
        key: LastKnownUserConnectionTimestamp
        op: ge
        value: 30
```


### Relevant log/traceback output

_No response_

### Extra information or context

_No response_


Discussion:
Hmm that's pretty fishy :thinking: . Are you seeing this consistently across multiple accounts and regions, or does it only show up in specific situations? Some steps that might help tease out some useful troubleshooting info:

* After having your policy fail, see if `c7n:ConnectionStatus` appears anywhere in the `resources.json` output.
* Run `aws workspaces describe-workspaces-connection-status` using custodian's credentials in an account/region where you saw this issue, and see if you run into any errors. 
Yah nothing for the account that fails - no resources.json.  I worked around it by using the cli in c7n-org run-script.  It’s been a week or so but I think I had issues with an older aws cli and had to upgrade it to use on this account too so maybe something in the aws api?  I think there were 450 worksspaces in the account/region. 


> On May 17, 2022, at 20:27, AJ Kerrigan ***@***.***> wrote:
> 
> ﻿
> Hmm that's pretty fishy 🤔 . Are you seeing this consistently across multiple accounts and regions, or does it only show up in specific situations? Some steps that might help tease out some useful troubleshooting info:
> 
> After having your policy fail, see if c7n:ConnectionStatus appears anywhere in the resources.json output.
> Run aws workspaces describe-workspaces-connection-status using custodian's credentials in an account/region where you saw this issue, and see if you run into any errors.
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

Possible. You can only request connection status for 25 resources at a time, so I can imagine more room for error with that many calls firing close together. I would expect the policy to bomb earlier in the case of exhausted retries due to throttling, or other errors. If you see the error again I'd be curious to get a full verbose log of the failing job, if you have it and are willing to share!
```18:51 $ cat custodian-run.log
2022-05-24 18:50:26,289 - custodian.policy - DEBUG - Running policy:workspaces-withstatus resource:workspaces region:eu-west-1 c7n:0.9.16
2022-05-24 18:50:34,782 - custodian.filters - DEBUG - Querying connection status for 434 workspaces
2022-05-24 18:50:42,909 - custodian.output - DEBUG - metric:PolicyException Count:1 policy:workspaces-withstatus restype:workspaces
2022-05-24 18:50:42,909 - custodian.output - DEBUG - metric:ApiCalls Count:41 policy:workspaces-withstatus restype:workspaces
2022-05-24 18:50:42,909 - custodian.output - ERROR - Error while executing policy
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.9/site-packages/c7n/query.py", line 532, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 337, in process
    return self.process_set(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 353, in process_set
    r[rtype_id] for r in f.process(resources, event)])
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 89, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 93, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'

2022-05-24 18:50:42,909: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.9/site-packages/c7n/query.py", line 532, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 337, in process
    return self.process_set(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 353, in process_set
    r[rtype_id] for r in f.process(resources, event)])
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 89, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 93, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-05-24 18:50:42,910: custodian.commands:ERROR Error while executing policy workspaces-withstatus, continuing
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 1242, in __call__
    resources = mode.run()
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 290, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.9/site-packages/c7n/query.py", line 532, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 337, in process
    return self.process_set(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 353, in process_set
    r[rtype_id] for r in f.process(resources, event)])
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 89, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 93, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-05-24 18:50:42,912: custodian.commands:ERROR The following policies had errors while executing
 - workspaces-withstatus
I'm seeing seemingly the same failure in one region (ap-southeast-2) for a job that seems to run fine in others. `aws workspaces describe-workspaces-connection-status` seems to run fine.

I'm attempting to run via lambda so I don't think I can play funny games with a cli exec.

Is there any data I can pull to help out?
I'm trying to find a way to reproduce this behavior. Once you see this error in a specific account/region, do you continue seeing it if you re-run the policy using `--cache-period 0`? Or is it an intermittent thing? I'm trying to sort out of this comes from some transient state or if it's something regional.

One naive "fix" might be deciding that if the `c7n:ConnectionStatus` key doesn't exist, there's no status info available. I'm not comfortable with that assumption yet though, and worry about having the filter be wrong more quietly.
Just ran again - it's every time on certain accounts.  I'm fully updated now too python 3.9 and c7n 0.9.18

I've got around this problem by running cli commands and joining the data (c7n-org run-script...)

```
16:08 $ custodian run --dryrun -s ~/log/custodian --region us-west-2 c7n_issue.yaml
2022-08-16 16:09:28,104: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 293, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.9/site-packages/c7n/query.py", line 532, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 97, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 101, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-08-16 16:09:28,105: custodian.commands:ERROR Error while executing policy workspaces, continuing
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/c7n/commands.py", line 301, in run
    policy()
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 1280, in __call__
    resources = PullMode(self).run()
  File "/usr/local/lib/python3.9/site-packages/c7n/policy.py", line 293, in run
    resources = self.policy.resource_manager.resources()
  File "/usr/local/lib/python3.9/site-packages/c7n/query.py", line 532, in resources
    resources = self.filter_resources(resources)
  File "/usr/local/lib/python3.9/site-packages/c7n/manager.py", line 111, in filter_resources
    resources = f.process(resources, event)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 97, in process
    return list(filter(self, resources))
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 540, in __call__
    matched = self.match(i)
  File "/usr/local/lib/python3.9/site-packages/c7n/filters/core.py", line 576, in match
    r = self.get_resource_value(self.k, i)
  File "/usr/local/lib/python3.9/site-packages/c7n/resources/workspaces.py", line 101, in get_resource_value
    k, i[self.annotation_key])
KeyError: 'c7n:ConnectionStatus'
2022-08-16 16:09:28,106: custodian.commands:ERROR The following policies had errors while executing
 - workspaces
✘-2 ~/Projects/finops/Workspaces [master|✚ 1…3]
16:09 $ custodian version
0.9.18
```

does it every time.


Resolving patch:
diff --git a/c7n/resources/workspaces.py b/c7n/resources/workspaces.py
--- a/c7n/resources/workspaces.py
+++ b/c7n/resources/workspaces.py
@@ -85,15 +85,23 @@ def get_connection_status(self, client, workspace_ids):
 
     def process(self, resources, event=None):
         client = local_session(self.manager.session_factory).client('workspaces')
-        annotate_map = {r['WorkspaceId']: r for r in resources if self.annotation_key not in r}
+        unannotated = {r['WorkspaceId']: r for r in resources if self.annotation_key not in r}
+        status_map = {}
         with self.executor_factory(max_workers=2) as w:
             self.log.debug(
-                'Querying connection status for %d workspaces' % len(annotate_map))
+                'Querying connection status for %d workspaces' % len(unannotated))
             for status in itertools.chain(*w.map(
                 functools.partial(self.get_connection_status, client),
-                chunks(annotate_map.keys(), 25)
+                chunks(unannotated.keys(), 25)
             )):
-                annotate_map[status['WorkspaceId']][self.annotation_key] = status
+                status_map[status['WorkspaceId']] = status
+
+        # Note: In some cases (e.g. workspaces that just launched or reached ERROR state during
+        # initialization) there will be no connection status information. Here we'll make
+        # sure that every workspace gets _some_ status annotation, even if it's empty.
+        for ws_id, r in unannotated.items():
+            r[self.annotation_key] = status_map.get(ws_id, {})
+
         return list(filter(self, resources))
 
     def get_resource_value(self, k, i):


Test patch:
diff --git a/tests/data/placebo/test_workspaces_connection_status/workspaces.DescribeWorkspaces_1.json b/tests/data/placebo/test_workspaces_connection_status/workspaces.DescribeWorkspaces_1.json
--- a/tests/data/placebo/test_workspaces_connection_status/workspaces.DescribeWorkspaces_1.json
+++ b/tests/data/placebo/test_workspaces_connection_status/workspaces.DescribeWorkspaces_1.json
@@ -93,6 +93,26 @@
                     "ComputeTypeName": "POWER"
                 },
                 "ModificationStates": []
+            },
+            {
+                "WorkspaceId": "ws-5fm1ykjjj",
+                "DirectoryId": "d-5183055e94",
+                "UserName": "rburnham",
+                "State": "ERROR",
+                "BundleId": "wsb-c4dc00cc1",
+                "ErrorMessage": "There was an issue joining the WorkSpace to your domain.  Verify that your service account is allowed to complete domain join operations.  If you continue to see an issue, contact AWS Support.",
+                "ErrorCode": "ConfigureWorkspace.DomainJoin",
+                "VolumeEncryptionKey": "arn:aws:kms:us-east-1:644160558196:key/14a03569-d26b-4496-92e5-dfe8cb1855fe",
+                "UserVolumeEncryptionEnabled": true,
+                "RootVolumeEncryptionEnabled": true,
+                "WorkspaceProperties": {
+                    "RunningMode": "AUTO_STOP",
+                    "RunningModeAutoStopTimeoutInMinutes": 120,
+                    "RootVolumeSizeGib": 80,
+                    "UserVolumeSizeGib": 100,
+                    "ComputeTypeName": "POWER"
+                },
+                "ModificationStates": []
             }
         ],
         "NextToken": "",