Project: Vitess (App Definition and Development / Database)
File: LongRunningJobs.md

Currently, in Vitess, long running tasks are used in a couple places: `vtworker` runs tasks that can take a long time, as they can deal with a lot of data. `vttablet` runs backups and restores that can also take a long time. In both cases, we have a streaming RPC API that starts the job, and streams some status back while it is running. If the streaming RPC is interrupted, the behavior is different: `vtworker` and `vttablet backup` will interrupt their jobs, whereas `vttablet restore` will try to keep going if it is past a point of no return for restores. This document proposes a different, but common design for both use cases. We introduce a model with three different RPCs: Start the job will be a synchronous non-streaming RPC, and just starts the job. It returns an identifier for the job. Getting the job status is a streaming RPC. It takes the job identifier, and streams the current status back. This job status can contain log entries (regular messages) or progress (percentage, N/M completion, ...). Canceling a running job is its own synchronous non-streaming RPC, and takes the job identifier. For simplicity, let's try to make these the same API for 'vtworker' and 'vttablet'. Usually, a single destination can only run a single job, but let's not assume that in the API. If a destination process cannot run a job, it should return the usual `RESOURCE_EXHAUSTED` canonical error code. These RPCs should be grouped in a new API service. Let's describe it as usual in `jobdata.proto` and `jobservice.proto`. The current `vtworkerdata.proto` and `vtworkerservice.proto` will eventually be removed and replaced by the new service. Let's use the usual `repeated string args` to describe the job. `vtworker` already uses that. So the proposed proto definitions: ``` proto message StartRequest { repeated string args = 1; } message StartResponse { string uid = 1; } message StatusRequest { string uid = 1; } // Progress describes the current progress of the task. // Note the fields here match the Progress and ProgressMessage from the Node // display of

message Progress { // percentage can be 0-100 if known, or -1 if unknown. int8 percentage = 1; // message can be empty if percentage is set. string message = 2; } // FinalStatus describes the end result of a job. message FinalStatus { // error is empty if the job was successful. string error = 1; } // StatusResponse can have any of its fields set. message StatusResponse { // event is optional, used for logging. logutil.Event event = 1; // progress is optional, used to indicate progress. Progress progress = 2; // If final_status is set, this is the last StatusResponse for this job, // it is terminated. FinalStatus final_status = 3; } message CancelRequest { string uid = 1; } message CancelResponse { } service Job { rpc Start (StartRequest) returns (StartResponse) {}; rpc Status (StatusRequest) returns (stream StatusResponse) {}; rpc Cancel (CancelRequest) returns (CancelResponse) {}; } ``` This design is very simple to implement within vtworker. At first, we don't need to link the progress in, just the logging part. vtworker will only support running one job like this at a time. This is also somewhat easy to implement within vttablet. Only `Backup` and `Restore` will be changed to use this. vttablet will only support running one job like this at a time. It will also take the ActionLock, so no other tablet actions can run at the same time (as we do now). The link here is also very straightforward: When successfully starting a remote job, the address of the remote worker and the UID of the job can be checkpointed. After that, the workflow can just connect and update its status and logs when receiving an update. If the workflow is aborted and reloaded somewhere else (vtctld restart), it can reconnect to the running job easily. Canceling the job is also easy, just call the RPC. Both vtworker and vttablet could remember the last N jobs they ran, and their status. So when a workflow tries to reconnect to a finished job, they just stream a single `StatusResponse` with a `final_status` field.