Repository: Parsl/parsl

Issue:
parsl-visualize creates invalid monitoring.db SQL schema
**Describe the bug**
Hi there, and thanks for your support. I'm using HighThroughputExecutor+SlurmProvider to manage my workflow, and I sometimes encounter an unexpected OperationalError when including a MonitoringHub in my Config, which then breaks the monitoring information and also keeps the main python script hanging. The hanging behavior should be related to https://github.com/Parsl/parsl/pull/1917#discussion_r647487589. More generally, I guess that this issue is related to https://github.com/Parsl/parsl/issues/2123 (and indeed I do sometimes have to stop my scripts with ctrl-c, which may have triggered this behavior).

**To Reproduce**
Unfortunately I don't have a minimal working example to show, since (1) this behavior is not reproducible (sometimes the same script does work correctly, sometimes it doesn't), and (2) parsl only enters in a small part of a more complex script that would be useless to fully describe here. Hopefully some parts of the configuration and the logs (below) can be useful to understand what is going wrong.


To configure parsl, I have some trivial auxiliary functions that define a MonitoringHub, a HighThroughputExecutor and a SlurmProvider, for instance like this one:
```
def define_MonitoringHub(workflow_name=None):
    kwargs = dict(
        hub_address=address_by_hostname(),
        monitoring_debug=True,
        resource_monitoring_interval=30,
    )
    if workflow_name is not None:
        kwargs["workflow_name"] = workflow_name
    return MonitoringHub(**kwargs)
```
In the main script, these are called as
```
    provider = define_SlurmProvider(...)
    htex = define_HighThroughputExecutor(provider=provider)
    monitoring = define_MonitoringHub(workflow_name="test")
    config = Config(executors=[htex], monitoring=monitoring)
    parsl.clear()
    parsl.load(config)
```

I let my main script run, and it correctly executes a set of `python_app`'s through SLURM (I see them running with `squeue`, and I verify that the output is correct). After all apps executed, the main script remains hanging, and if I check the logs I see in `runinfo/000/database_manager.log` that the script got stuck in an OperationalError loop:
```
$ tail -n 8 runinfo/000/database_manager.log 
2022-05-30 10:34:02.599 database_manager:597 [WARNING] [MainThread 140641839281920] Got a database OperationalError. Ignoring and retrying on the assumption that it is recoverable: (sqlite3.OperationalError) table task has no column named task_time_invoked
[SQL: INSERT INTO task (task_id, run_id, task_depends, task_func_name, task_memoize, task_inputs, task_outputs, task_time_invoked, task_fail_count, task_fail_cost) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (0, '903cd6fa-cda7-416d-9ada-bdeb09a030d2', '', 'app_create_zarr_structure', False, 'None', 'None', '2022-05-30 10:28:28.462702', 0, 0.0)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
2022-05-30 10:34:03.604 database_manager:597 [WARNING] [MainThread 140641839281920] Got a database OperationalError. Ignoring and retrying on the assumption that it is recoverable: (sqlite3.OperationalError) table task has no column named task_time_invoked
[SQL: INSERT INTO task (task_id, run_id, task_depends, task_func_name, task_memoize, task_inputs, task_outputs, task_time_invoked, task_fail_count, task_fail_cost) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (0, '903cd6fa-cda7-416d-9ada-bdeb09a030d2', '', 'app_create_zarr_structure', False, 'None', 'None', '2022-05-30 10:28:28.462702', 0, 0.0)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
```
where it just prints this error message once per second. I attach the full `database_manager.log`, `monitoring_router.log` and `parsl.log`. You see that `parsl.log` is missing the final "DFK cleanup complete" cleanup line (which is present for cases that run smoothly).

[database_manager.log](https://github.com/Parsl/parsl/files/8798140/database_manager.log)
[monitoring_router.log](https://github.com/Parsl/parsl/files/8798141/monitoring_router.log)
[parsl.log](https://github.com/Parsl/parsl/files/8798142/parsl.log)



After this situation takes place, re-running the same script typically leads to the same behavior, even if I remove `monitoring.db` and the `runinfo` folder (which is puzzling.. is there a state of the DFK which somehow remains active after I kill the python script?), although sometimes I recover the expected correct behavior.
Meanwhile, these "wrong" runs cannot be accessed through `parsl-visualize` (when I select a workflow in the web interface, I either get "500 Internal server error" or "Workflow 903cd6fa-cda7-416d-9ada-bdeb09a030d2 does not have any resource usage records. ") , but this is not surprising given that the database is probably corrupted somewhere.


**Expected behavior**
I expected no OperationalError, as the `task_time_invoked` column should be present.

**Environment**
 - OS: OS: Ubuntu 16
 - Python version: 3.8.13
 - Parsl version: 1.2.0

**Distributed Environment**
- Where are you running the Parsl script from ? Login node
- Where do you need the workers to run ? Compute nodes

(apologies for the non-reproducible issue description, I'll keep trying to make the unexpected behavior more robust)


Discussion:
Can you look at a database when this is happening and paste the schema for the task table?

eg like this:

```
$ sqlite3 runinfo/monitoring.db 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> .schema task
CREATE TABLE task (
	task_id INTEGER NOT NULL, 
	run_id TEXT NOT NULL, 
	task_depends TEXT, 
	task_func_name TEXT NOT NULL, 
	task_memoize TEXT NOT NULL, 
	task_hashsum TEXT, 
	task_inputs TEXT, 
	task_outputs TEXT, 
	task_stdin TEXT, 
	task_stdout TEXT, 
	task_stderr TEXT, 
	task_time_invoked DATETIME, 
	task_time_returned DATETIME, 
	task_fail_count INTEGER NOT NULL, 
	task_fail_cost FLOAT NOT NULL, 
	PRIMARY KEY (task_id, run_id)
);
CREATE INDEX ix_task_task_hashsum ON task (task_hashsum);
sqlite> 
```
I cannot open `monitoring.db` for a broken case, because I typically remove it often while testing (exactly to try to avoid the current issue), and "unfortunately" I cannot reproduce the issue right now (even with a few ctrl-c here and there). I will keep trying.

Meanwhile, I noted that some other issue with `task_time_invoked` also appears when the run correctly ends (that is, the script is not hanging, and there is no OperationalError in the logs). In this case, the table reads
```
$ sqlite3 monitoring.db 
SQLite version 3.38.3 2022-04-27 12:03:15
Enter ".help" for usage hints.
sqlite> .schema task
CREATE TABLE task (
	task_id INTEGER NOT NULL, 
	run_id TEXT NOT NULL, 
	task_depends TEXT, 
	task_func_name TEXT NOT NULL, 
	task_memoize TEXT NOT NULL, 
	task_hashsum TEXT, 
	task_inputs TEXT, 
	task_outputs TEXT, 
	task_stdin TEXT, 
	task_stdout TEXT, 
	task_stderr TEXT, 
	task_time_invoked DATETIME, 
	task_time_returned DATETIME, 
	task_fail_count INTEGER NOT NULL, 
	task_fail_cost FLOAT NOT NULL, 
	PRIMARY KEY (task_id, run_id)
);
CREATE INDEX ix_task_task_hashsum ON task (task_hashsum);
sqlite> 
```
which seems correct. However, the web interface for any specific task shows an error:

![Screenshot from 2022-05-31 14-28-09](https://user-images.githubusercontent.com/3862206/171173958-c535691d-0dd1-4661-9f58-701a1b05e901.png)


Let me add, since we're at the monitoring web interface, that I also consistently receive the message `The tasks per app plot cannot be generated because of exception 'task_time_returned'.` on the page of each workflow, below the (correctly-shown) Gantt chart. I don't know whether this is related, but it's to give you as much information as possible.




Finally I could reproduce the bug once again (and there was indeed a ctrl-c shutdown just prior to the run going wrong).

The schema is indeed missing `task_time_returned`:
```
$ sqlite3 monitoring.db 
SQLite version 3.38.3 2022-04-27 12:03:15
Enter ".help" for usage hints.
sqlite> .schema task
CREATE TABLE task (
	task_id INTEGER NOT NULL, 
	run_id TEXT NOT NULL, 
	task_func_name TEXT NOT NULL, 
	task_depends TEXT, 
	task_time_returned DATETIME, 
	task_memoize TEXT NOT NULL, 
	task_inputs TEXT, 
	task_outputs TEXT, 
	task_stdin TEXT, 
	task_stdout TEXT, 
	task_stderr TEXT, 
	PRIMARY KEY (task_id, run_id)
);
sqlite> 
```

And here are the relevant logs (with the OperationalError clearly visible in `database_manager.log`):
[parsl.log](https://github.com/Parsl/parsl/files/8844710/parsl.log)
[monitoring_router.log](https://github.com/Parsl/parsl/files/8844711/monitoring_router.log)
[database_manager.log](https://github.com/Parsl/parsl/files/8844713/database_manager.log)

Also notice that `parsl.log` is missing the final line with DFK cleanup.

My main suspicion is that this is a problem at the *start* of a run - i.e. the schema being created incorrectly, with the symptoms only showing at the end (as the monitoring code desperately resists shutting down until it can write the data that it cannot write).

Are you using any other code to access the monitoring db that is not the v1.2 parsl runtime or the v1.2 parsl visualize code?
The schema you pasted *does* have `task_time_returned DATETIME` -  however I notice that that there are *other* fields missing. Let me investigate that a little bit. (for example, task_fail_cost)
I have recreated this: when the database is created by parsl-visualize, then it is created with an incorrect schema. when the database is created by a parsl workflow run, then it is created with the correct schema. This is likely going to look something approximately like: whichever you run first gets to create the database.

I'm surprised at the error message in this case, but unsurprised that having non-synchronised copies of the database schema in multiple places of the codebase results in this error.

There are two issues, then, I think:

i) it is fragile for parsl-visualize to contain a copy of the schema that it wants, distinct from the schema that the parsl workflow runtime wants.

ii) `OperationalError` doesn't seem to give enough resolution of error: operations are retried on that error because it is what happens when someone else has a read lock on the DB (for example, if you are using a visualization tool on the DB while a workflow is running); but in the case that the schema is permanently incorrect, that retry is the wrong thing to do.

Probably not needed any more, but:

> Are you using any other code to access the monitoring db that is not the v1.2 parsl runtime or the v1.2 parsl visualize code?

No, just `parsl-visualize`.

> The schema you pasted does have task_time_returned DATETIME

Of course, sorry. I meant `task_time_invoked` (the one triggering the OperationalError).

Meanwhile, thanks for digging into this. It is true that I often run `parsl-visualize` first, and then run the parsl scripts (so that I can immediately see new workflows appearing on the monitoring interface). As a temporary workaround, I will try to always start the parsl scripts first, before running `pars-visualize`.
renamed from original title: Hanging script and broken monitoring due to OperationalError in monitoring (HTEX+SlurmProvider)
Quick remark: unless I'm missing something trivial, `task_time_invoked` is not present in  https://github.com/Parsl/parsl/blob/a1e4d6b0d680f6a0e9f5784b994f88941a92bdb6/parsl/monitoring/visualization/models.py#L55-L71

Resolving patch:
diff --git a/parsl/monitoring/visualization/models.py b/parsl/monitoring/visualization/models.py
--- a/parsl/monitoring/visualization/models.py
+++ b/parsl/monitoring/visualization/models.py
@@ -58,6 +58,8 @@ class Task(db.Model):
     run_id = db.Column('run_id', db.Text, nullable=False)
     task_func_name = db.Column('task_func_name', db.Text, nullable=False)
     task_depends = db.Column('task_depends', db.Text, nullable=True)
+    task_time_invoked = db.Column(
+        'task_time_invoked', db.DateTime, nullable=True)
     task_time_returned = db.Column(
         'task_time_returned', db.DateTime, nullable=True)
     task_memoize = db.Column('task_memoize', db.Text, nullable=False)


Test patch: