Repository: team-ocean/veros

Issue:
Job resubmission with job scheduler doesn't work 
I was not able to find out the reason behind resubmission issue with job scheduler, such as:
`veros-resubmit -i acc.lowres -n 50 -l 62208000 -c "python acc.py -b bohrium -v debug" --callback "/usr/bin/sbatch /groups/ocean/nutrik/veros_cases/paper/acc/veros_batch.sh"`
Although jobs with run length of up to 29 days are resubmitted fine, those with longer run length are not resubmitted and no errors or messages are reported.

In fact, jobs are successfully resubmitted without scheduler (`--callback "./veros_batch.sh"`) for any run length.


Discussion:
What’s the content of `veros_batch.sh`?
```
#!/bin/bash -l
#
#SBATCH -p aegir
#SBATCH -A ocean
#SBATCH --job-name=acc_lr
#SBATCH --time=23:59:59
#SBATCH --constraint=v2
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --exclusive

export BH_STACK=openmp
module load bohrium/05102018 veros/05102018
veros-resubmit -i acc.lowres -n 50 -l 62208000 -c "python acc.py -b bohrium -v debug" --callback "/usr/bin/sbatch /groups/ocean/nutrik/veros_cases/paper/acc/veros_batch.sh"
```
How long (in real time) do the 29 day runs take?
11 minutes. 
The issue is not about wall clock time limit exceedance. Every model cycle is successfully completed with "Timing summary" information but the case is not resubmitted.
And how do you execute the first run?
`sbatch veros_batch.sh`
Ok. So the thing is that resubmission has an inherent race condition. The job reschedules itself, waits for a short while, then exits, but if the wait time is too short it might be that the job is killed before the new job has gone through. Unless there is a reliable way to determine whether a new job has been scheduled there's not much we can do about that.

One thing you can try is to add a `time.sleep(5)` after the following line:

https://github.com/dionhaefner/veros/blob/0079cde8dbf39f9a3944902792292807b20f443c/veros/cli/veros_resubmit.py#L81

If that doesn't help, we'll have to dig deeper.

On a side note, the reason why it works without the `--callback` argument is that the default behavior for the script is to call itself. In this case, it just runs again on the same node, and is never really rescheduled. It will thus time out after 24h.
Yeah, adding `time.sleep(5)` has solved the issue! Thanks!
But, I think the master branch must be fixed for other users.
Let’s keep this open for now. I’ll probably introduce some waiting time, but I doubt we can make the problem go away in every case.

Resolving patch:
diff --git a/veros/cli/veros_resubmit.py b/veros/cli/veros_resubmit.py
--- a/veros/cli/veros_resubmit.py
+++ b/veros/cli/veros_resubmit.py
@@ -5,10 +5,13 @@
 import shlex
 import sys
 import os
+import time
 
 import click
 
 LAST_N_FILENAME = "{identifier}.current_run"
+CHILD_TIMEOUT = 10
+POLL_DELAY = 0.1
 
 
 class ShellCommand(click.ParamType):
@@ -60,7 +63,21 @@ def resubmit(identifier, num_runs, length_per_run, veros_cmd, callback):
 
     call_veros(veros_cmd, identifier, current_n, length_per_run)
     write_next_n(current_n + 1, last_n_filename)
-    subprocess.Popen(callback)
+    next_proc = subprocess.Popen(callback)
+
+    # catch immediately crashing processes
+    timeout = CHILD_TIMEOUT
+
+    while timeout > 0:
+        retcode = next_proc.poll()
+        if retcode is not None:
+            if retcode > 0:
+                # process crashed
+                raise RuntimeError("Callback exited with {}".format(retcode))
+            else:
+                break
+        time.sleep(POLL_DELAY)
+        timeout -= POLL_DELAY
 
 
 @click.command("veros-resubmit", short_help="Re-run a Veros setup several times")
@@ -78,6 +95,7 @@ def resubmit(identifier, num_runs, length_per_run, veros_cmd, callback):
 def cli(*args, **kwargs):
     if kwargs["callback"] is None:
         kwargs["callback"] = sys.argv
+
     resubmit(*args, **kwargs)
 
 


Test patch: