Repository: apache/dolphinscheduler
Issue: After the soft kill task, the process still exists (parent process child process)(soft kill task 后 进程依旧存在(父进程 子进程))


In the shell script, the task is executed in a multi-threaded manner. The timeout of the task is set to fail. After the timeout, the process of starting the task is executed, but the parent process is not yet killed.

Worker LOG:
[INFO] 2019-07-18 02:03:01.513 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_33_552_739] task run command:
Sudo sudo -u rundeck sh /tmp/escheduler/exec/process/5/33/552/739/33_552_739.command
[INFO] 2019-07-18 02:03:01.516 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_33_552_739] process start, process id is: 185464
[INFO] 2019-07-18 02:04:18.082 cn.escheduler.server.worker.log.TaskLogger:[188] - [taskAppId=TASK_33_552_739] process has exited, work dir:/tmp/escheduler/exec/process/ 5/33/552/739, pid:185464, exitStatusCode: 0
[INFO] 2019-07-18 02:04:18.083 cn.escheduler.server.worker.runner.TaskScheduleThread:[198] - task : 33_552_739 exit status code : 0
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_24_548_737] -> ============== ==========
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_24_548_737] cancel process: 170216
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[188] - [taskAppId=TASK_24_548_737] soft kill task:24_548_737, process id:170216, cmd:sudo kill 170216
[WARN] 2019-07-18 02:17:15.533 cn.escheduler.server.worker.log.TaskLogger:[243] - [taskAppId=TASK_24_548_737] process timeout, work dir:/tmp/escheduler/exec/process/3 /24/548/737, pid:170216
[INFO] 2019-07-18 02:17:15.533 cn.escheduler.server.worker.runner.TaskScheduleThread:[198] - task : 24_548_737 exit status code : -1
[INFO] 2019-07-18 02:18:18.758 cn.escheduler.common.queue.TaskQueueZkImpl:[181] - consume task /escheduler/tasks_queue/0_548_2_740
[INFO] 2019-07-18 02:18:18.758 cn.escheduler.common.queue.TaskQueueZkImpl:[188] - consume task: 740,there still have 0 tasks need to be executed

---
shell脚本中使用多线程的方式执行任务, 对任务设定超时失败, 超时后启动任务的进程被执行 kill, 但父进程 并没有被killed 子进程也依旧存在.

Worker LOG:
[INFO] 2019-07-18 02:03:01.513 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_33_552_739] task run command:
sudo sudo -u rundeck sh /tmp/escheduler/exec/process/5/33/552/739/33_552_739.command
[INFO] 2019-07-18 02:03:01.516 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_33_552_739] process start, process id is: 185464
[INFO] 2019-07-18 02:04:18.082 cn.escheduler.server.worker.log.TaskLogger:[188] - [taskAppId=TASK_33_552_739] process has exited, work dir:/tmp/escheduler/exec/process/5/33/552/739, pid:185464 ,exitStatusCode:0
[INFO] 2019-07-18 02:04:18.083 cn.escheduler.server.worker.runner.TaskScheduleThread:[198] - task : 33_552_739 exit status code : 0
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_24_548_737]  -> ========================
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_24_548_737] cancel process: 170216
[INFO] 2019-07-18 02:17:15.531 cn.escheduler.server.worker.log.TaskLogger:[188] - [taskAppId=TASK_24_548_737] soft kill task:24_548_737, process id:170216, cmd:sudo kill 170216
[WARN] 2019-07-18 02:17:15.533 cn.escheduler.server.worker.log.TaskLogger:[243] - [taskAppId=TASK_24_548_737] process timeout, work dir:/tmp/escheduler/exec/process/3/24/548/737, pid:170216
[INFO] 2019-07-18 02:17:15.533 cn.escheduler.server.worker.runner.TaskScheduleThread:[198] - task : 24_548_737 exit status code : -1
[INFO] 2019-07-18 02:18:18.758 cn.escheduler.common.queue.TaskQueueZkImpl:[181] - consume task /escheduler/tasks_queue/0_548_2_740
[INFO] 2019-07-18 02:18:18.758 cn.escheduler.common.queue.TaskQueueZkImpl:[188] - consume task: 740,there still have 0 tasks need to be executed

Changed files:
escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractCommandExecutor.java

diff --git a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractCommandExecutor.java b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractCommandExecutor.java
index e3bd40139..82c9be9f0 100644
--- a/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractCommandExecutor.java
+++ b/escheduler-server/src/main/java/cn/escheduler/server/worker/task/AbstractCommandExecutor.java
@@ -162,7 +162,12 @@ public abstract class AbstractCommandExecutor {
                 exitStatusCode = updateState(processDao, exitStatusCode, pid, taskInstId);
 
             } else {
-                cancelApplication();
+                TaskInstance taskInstance = processDao.findTaskInstanceById(taskInstId);
+                if (taskInstance == null) {
+                    logger.error("task instance id:{} not exist", taskInstId);
+                } else {
+                    ProcessUtils.kill(taskInstance);
+                }
                 exitStatusCode = -1;
                 logger.warn("process timeout, work dir:{}, pid:{}", taskDir, pid);
             }