Repository: kestra-io/kestra
Issue: More soft control on the execution
Labels: enhancement

When you have a major outage, the data inside the store can be corrupted. Example: 
- execution mark as running but not 
- execution mark as failed 
Right now, we have a lot of control that prevent exceptional case. 

We need to have softer check to allow for example: 
- to kill an execution that is not running
- ... 

Issue comments:
Matrix of state (from this [spreadsheet](https://docs.google.com/spreadsheets/d/1yKCFT0yoc3kVTrtXpgSAG2Y3TKBtfLxgRtr81HlFddw/edit?usp=sharing))

| CURRENT_STATE | WANTED_STATE | WHY ?                                                                                                                                                                                                                                                                                             |
|---------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CREATED       | RUNNING      |                                                                                                                                                                                                                                                                                                   |
|               | PAUSED       | No way to force it. Only done through Pause task. Related to https://github.com/kestra-io/kestra/issues/1449 which wants to be able to tell execution to pause itself at a specific task id at execution creation                                                                                 |
|               | RESTARTED    | UI (gray button cause not running / not failed) + Backend (only failed or paused taskruns can be restarted)                                                                                                                                                                                       |
|               | KILLING      |                                                                                                                                                                                                                                                                                                   |
|               | SUCCESS      | UI (No taskrun displayed in Gantt / Logs tab which prevent us from accessing change status popup) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                        |
|               | WARNING      | UI (No taskrun displayed in Gantt / Logs tab which prevent us from accessing change status popup) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                        |
|               | FAILED       | UI (No taskrun displayed in Gantt / Logs tab which prevent us from accessing change status popup) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                        |
|               | KILLED       | No force-kill                                                                                                                                                                                                                                                                                     |
|               |              |                                                                                                                                                                                                                                                                                                   |
| RUNNING       | PAUSED       | No way to force it. Only done through Pause task. Related to https://github.com/kestra-io/kestra/issues/1449 which wants to be able to tell execution to pause itself at a specific task id at execution creation                                                                                 |
|               | RESTARTED    | Useless ?                                                                                                                                                                                                                                                                                         |
|               | KILLING      |                                                                                                                                                                                                                                                                                                   |
|               | SUCCESS      | UI (Gray button) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                                                                                                         |
|               | WARNING      | UI (Gray button) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                                                                                                         |
|               | FAILED       | UI (Gray button) + Backend (Won't let a task change for non-terminated / non-paused tasks, service guard)                                                                                                                                                                                         |
|               | KILLED       | Only automatically by backend once we receive a KILLED confirmation from underlying taskruns. Currently no way to force it                                                                                                                                                                        |
|               |              |                                                                                                                                                                                                                                                                                                   |
| PAUSED        | RUNNING      | Resume button                                                                                                                                                                                                                                                                                     |
|               | RESTARTED    | UI (gray button). Backend OK but strange behaviour cause it pushes the paused task as RESTARTED which then goes to running, bypassing the paused state without going through resume call                                                                                                          |
|               | KILLING      | When killing a PAUSED execution, it goes to running then back to paused. It executes tasks that are after the PAUSE task but then stays in PAUSE (only in KAFKA)                                                                                                                                  |
|               | SUCCESS      | UI (no change-state in Gantt chart cause no log) + Mark-as service throws if execution is paused. Need to resume which will success the paused task. However there is no way to force the whole execution to being in a success state without executing after-paused tasks                        |
|               | WARNING      | ''                                                                                                                                                                                                                                                                                                |
|               | FAILED       | ''                                                                                                                                                                                                                                                                                                |
|               | KILLED       | Only automatically by backend once we receive a KILLED confirmation from underlying taskruns. Currently no way to force it                                                                                                                                                                        |
|               |              |                                                                                                                                                                                                                                                                                                   |
| RESTARTED     | RUNNING      | Automatically by backend, but it's a standard behaviour                                                                                                                                                                                                                                           |
|               | PAUSED       | No way to force it. Only done through Pause task. Related to https://github.com/kestra-io/kestra/issues/1449 which wants to be able to tell execution to pause itself at a specific task id at execution creation                                                                                 |
|               | KILLING      |                                                                                                                                                                                                                                                                                                   |
|               | SUCCESS      | UI is OK but backend requires a terminated state to change state (and restarted isn't one)                                                                                                                                                                                                        |
|               | WARNING      | ''                                                                                                                                                                                                                                                                                                |
|               | FAILED       | ''                                                                                                                                                                                                                                                                                                |
|               | KILLED       |                                                                                                                                                                                                                                                                                                   |
|               |              |                                                                                                                                                                                                                                                                                                   |
| KILLING       | RUNNING      | No UI button to do so + Taskruns are put as killed as soon as an execution is marked as KILLING                                                                                                                                                                                                   |
|               | PAUSED       | No interaction possible                                                                                                                                                                                                                                                                           |
|               | RESTARTED    | ''                                                                                                                                                                                                                                                                                                |
|               | SUCCESS      | ''                                                                                                                                                                                                                                                                                                |
|               | WARNING      | ''                                                                                                                                                                                                                                                                                                |
|               | FAILED       | ''                                                                                                                                                                                                                                                                                                |
|               | KILLED       | Only automatically by backend once we receive a KILLED confirmation from underlying taskruns. Currently no way to force it                                                                                                                                                                        |
|               |              |                                                                                                                                                                                                                                                                                                   |
| SUCCESS       | RUNNING      | No UI button -> Must go through a replay which creates a new execution -> No restart possible (both UI & Backend makes it impossible: backend searches for failed or paused taskruns to restart from)                                                                                             |
|               | PAUSED       | No way to force it. Only done through Pause task. Related to https://github.com/kestra-io/kestra/issues/1449 which wants to be able to tell execution to pause itself at a specific task id at execution creation                                                                                 |
|               | RESTARTED    | UI (gray button cause not running / not failed) + Backend (only failed or paused taskruns can be restarted)                                                                                                                                                                                       |
|               | KILLING      | No UI button + backend will not accept a terminated execution                                                                                                                                                                                                                                     |
|               | WARNING      |                                                                                                                                                                                                                                                                                                   |
|               | FAILED       |                                                                                                                                                                                                                                                                                                   |
|               | KILLED       |                                                                                                                                                                                                                                                                                                   |
|               |              |                                                                                                                                                                                                                                                                                                   |
|               |              |                                                                                                                                                                                                                                                                                                   |
| WARNING       | RUNNING      | No UI button -> Must go through a replay which creates a new execution -> No restart possible (both UI & Backend makes it impossible: backend searches for failed or paused taskruns to restart from)                                                                                             |
|               | PAUSED       |                                                                                                                                                                                                                                                                                                   |
|               | RESTARTED    | UI (gray button cause not running / not failed) + Backend (only failed or paused taskruns can be restarted)                                                                                                                                                                                       |
|               | KILLING      | No UI button + backend will not accept a terminated execution                                                                                                                                                                                                                                     |
|               | SUCCESS      |                                                                                                                                                                                                                                                                                                   |
|               | FAILED       |                                                                                                                                                                                                                                                                                                   |
|               | KILLED       |                                                                                                                                                                                                                                                                                                   |
|               |              |                                                                                                                                                                                                                                                                                                   |
| FAILED        | RUNNING      | Indirect state because you need to restart the execution in overview menu which changes the execution to RESTARTED then RUNNING by the executor                                                                                                                                                   |
|               | PAUSED       |                                                                                                                                                                                                                                                                                                   |
|               | RESTARTED    | Restart from overview menu                                                                                                                                                                                                                                                                        |
|               | KILLING      | No UI button + Backend checks and refuses terminated states                                                                                                                                                                                                                                       |
|               | SUCCESS      |                                                                                                                                                                                                                                                                                                   |
|               | WARNING      |                                                                                                                                                                                                                                                                                                   |
|               | KILLED       |                                                                                                                                                                                                                                                                                                   |
|               |              |                                                                                                                                                                                                                                                                                                   |
| KILLED        | RUNNING      | You cannot restart a killed execution (UI button is not present + backend will search for a failed or paused taskrun to start from).  Change status doesn't display running as an available switch-to state. Meaning you are forced to either replay in another execution or skip the killed task |
|               | PAUSED       |                                                                                                                                                                                                                                                                                                   |
|               | RESTARTED    | You cannot restart a killed execution (UI button is not present + backend will search for a failed or paused taskrun to start from).                                                                                                                                                              |
|               | KILLING      | No usage                                                                                                                                                                                                                                                                                          |
|               | SUCCESS      | Change status button                                                                                                                                                                                                                                                                              |
|               | WARNING      |                                                                                                                                                                                                                                                                                                   |
|               | FAILED       |                                                                                                                                                                                                                                                                                                   |
We just had a case creating a lot of execution (several thousand) in "RESTARTED" state. This was a mistake.
Due to Kestra check, we have not been able to stop all these executions until they change to "RUNNING" state.
This could be usefull to be able to stop executions before they are running.
@aurelienWls killing `CREATED` execution is already possible on Kestra 0.10, but we will check if we need to handle more use case.
UI currently prevents changing the Execution state when some state transition is not allowed. However, often this is required to unblock some applications. More input is needed here to decide about next steps

It seems that the only state changes we should allow that are not currently allowed are
CREATED & RESTARTED -> terminated state (SUCCESS, WARNING, FAIL).
It opens to potential concurrency issues but even if it starts after the status change it should still work :thinking: 
Decission is to allow transitionning from CREATED & RESTARTED to a terminal state (SUCCESS, WARNING, FAILED).
Allow transitionning from RESTARTED to a terminal state (SUCCESS, WARNING, FAILED) was already implemented so only CREATED needs to be allowed.

Pull request review comments:

Patch that resolved the issue:
diff --git a/core/src/main/java/io/kestra/core/services/ExecutionService.java b/core/src/main/java/io/kestra/core/services/ExecutionService.java
index b838edc9af..ee89cd2b9d 100644
--- a/core/src/main/java/io/kestra/core/services/ExecutionService.java
+++ b/core/src/main/java/io/kestra/core/services/ExecutionService.java
@@ -193,12 +193,6 @@ public Execution replay(final Execution execution, @Nullable String taskRunId, @
     }
 
     public Execution markAs(final Execution execution, String taskRunId, State.Type newState) throws Exception {
-        if (!(execution.getState().isTerminated() || execution.getState().isPaused())) {
-            throw new IllegalStateException("Execution must be terminated to be restarted, " +
-                "current state is '" + execution.getState().getCurrent() + "' !"
-            );
-        }
-
         final Flow flow = flowRepositoryInterface.findByExecution(execution);
 
         Set<String> taskRunToRestart = this.taskRunToRestart(
diff --git a/ui/src/components/executions/ChangeStatus.vue b/ui/src/components/executions/ChangeStatus.vue
index 6cbb823ed9..665dc1189f 100644
--- a/ui/src/components/executions/ChangeStatus.vue
+++ b/ui/src/components/executions/ChangeStatus.vue
@@ -168,7 +168,7 @@
                     return false;
                 }
 
-                if (this.taskRun.state.current === "PAUSED") {
+                if (this.taskRun.state.current === "PAUSED" || this.taskRun.state.current === "CREATED") {
                     return true;
                 }
 


Test patch: