Repository: Parsl/parsl

Issue:
AWS provider (and maybe GCS provider) does not report job state
**Describe the bug**
When using the AWS provider, this error happens with every job status poller loop:

```
2020-06-01 13:40:16.122 parsl.dataflow.task_status_poller:62 [DEBUG]  Polling
2020-06-01 13:40:16.126 parsl.dataflow.flow_control:114 [ERROR]  Flow control callback threw an exception - logging and proceeding anyway
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/dataflow/flow_control.py", line 112, in make_callback
    self.callback(tasks=self._event_buffer, kind=kind)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/task_status_poller.py", line 64, in poll
    self._strategy.strategize(self._poll_items, tasks)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/strategy.py", line 198, in _strategy_simple
    running = sum([1 for x in status.values() if x.state == JobState.RUNNING])
  File "/home/benc/parsl/src/parsl/parsl/dataflow/strategy.py", line 198, in <listcomp>
    running = sum([1 for x in status.values() if x.state == JobState.RUNNING])
AttributeError: 'JobState' object has no attribute 'state'
```

It looks like this arises from the AWS provider `.status` method returning incorrect values. This type annotation reveals that with `make mypy`:

```
 -    def status(self, job_ids):
 +    def status(self, job_ids) -> List[JobStatus]:
```

I think this was probably introduced by the work around PR #1493 which rearranged job status representation.

**To Reproduce**
parsl master 7b1c4594582773cf701f7c0096956a87375acdf5
set some arbitrary config for ec2 testing in parsl/tests/configs/local_user_opts.py:
```
   'ec2': {
        'image_id': "ami-0e63f5af5130c9681",
        'region': "eu-west-2",
        'key_name': "aws-cqx-eu-west-2"
    }
```

> pytest parsl --config parsl/tests/configs/ec2_single_node.py

This will hang, but the runinfo_*/000/parsl.log file should now be showing the above exception stack trace.

It looks likely that this problem will also happen with the `googlecloud` provider: add the same status type annotation gives the same mypy error.

**Environment**
my laptop, python 3.7



Discussion:

Resolving patch:
diff --git a/parsl/providers/aws/aws.py b/parsl/providers/aws/aws.py
--- a/parsl/providers/aws/aws.py
+++ b/parsl/providers/aws/aws.py
@@ -558,8 +558,9 @@ def status(self, job_ids):
             for i in r['Instances']:
                 instance_id = i['InstanceId']
                 instance_state = translate_table.get(i['State']['Name'], JobState.UNKNOWN)
-                self.resources[instance_id]['status'] = JobStatus(instance_state)
-                all_states.extend([instance_state])
+                instance_status = JobStatus(instance_state)
+                self.resources[instance_id]['status'] = instance_status
+                all_states.extend([instance_status])
 
         return all_states
 
diff --git a/parsl/providers/googlecloud/googlecloud.py b/parsl/providers/googlecloud/googlecloud.py
--- a/parsl/providers/googlecloud/googlecloud.py
+++ b/parsl/providers/googlecloud/googlecloud.py
@@ -151,8 +151,9 @@ def status(self, job_ids):
         statuses = []
         for job_id in job_ids:
             instance = self.client.instances().get(instance=job_id, project=self.project_id, zone=self.zone).execute()
-            self.resources[job_id]['status'] = JobStatus(translate_table[instance['status']])
-            statuses.append(translate_table[instance['status']])
+            job_status = JobStatus(translate_table[instance['status']])
+            self.resources[job_id]['status'] = job_status
+            statuses.append(job_status)
         return statuses
 
     def cancel(self, job_ids):
diff --git a/parsl/providers/grid_engine/grid_engine.py b/parsl/providers/grid_engine/grid_engine.py
--- a/parsl/providers/grid_engine/grid_engine.py
+++ b/parsl/providers/grid_engine/grid_engine.py
@@ -181,9 +181,9 @@ def _status(self):
             if parts and parts[0].lower().lower() != 'job-id' \
                     and not parts[0].startswith('----'):
                 job_id = parts[0]
-                status = translate_table.get(parts[4].lower(), JobState.UNKNOWN)
+                state = translate_table.get(parts[4].lower(), JobState.UNKNOWN)
                 if job_id in self.resources:
-                    self.resources[job_id]['status'] = status
+                    self.resources[job_id]['status'] = JobStatus(state)
                     jobs_missing.remove(job_id)
 
         # Filling in missing blanks for jobs that might have gone missing


Test patch: