Repository: tus/tusd (Go)

Issue:
Is there a way to avoid races involving `tusd` deleting files from storage?
The [Zulip](https://github.com/zulip/zulip) implementation of `tusd` involves using a `pre-finish` hook to create some data structures in our database (used to manage quota, track the messages referencing the file, etc.), verify that image files are thumbnailable, and kick off a worker job to begin thumbnailing images.

The problem I'm seeing is that the `tusd` protocol allows the client to delete the upload via a `DELETE` request at any time, including **after an upload has completed**, and `tusd` implements that by first removing the file from the backing storage and then notifying the application via the `post-terminate` hook. A specific case where this occurred involved `nginx` deciding the client had gone away when `tusd` was trying to write the response in the `finish` stage, serving a `499`, which appears to have caused `uppy` to ask to delete/terminate the upload? At least, that's what we can piece together from logs. The end result is that our worker job for thumbnailing images sees a corrupt state where the application database thinks the file exists, but it's been deleted from the backing store by `tusd`.

I don't see a good way to implement this aspect of the protocol in a way that is safe from races where the application database is in a corrupt state. Options that we've considered include:
- Enabling the `--disable-terminate` option on the `tusd` command line. This is not ideal, in that I think this will prevent all terminations, including of partially uploaded files? I'm also not sure after some source-diving whether clients like `uppy` have a way to be configured to support that setting.
- Having the `post-terminate` hook clean up our database structures associated with the upload after tusd has deleted the file. The problem is that this happens after the file is removed from the backing storage, so there is a fundamental race where the database thinks the file exists but it is missing when other application code is trying to process it (for thumbnailing or whatever). This results in file-not-found exceptions that normally would be very scary being a routine/normal thing.
- Deferring visible side effects, like queuing the file to be processed for thumbnailing, to the `post-finish` hook. This might limit our exposure to the race, but it doesn't offer a way to solve the problem that `tusd` might at any time delete already-uploaded files from the backing storage before adjusting our data structures.

I see the following options for how `tusd` could support a race-free application implementation:
- **Recommended**: Add a `pre-terminate` hook that can be used by the application to either reject the deletion request (say, if the user doesn't have permission to delete the file for whatever reason) or indicate to `tusd` that as part of its processing, it has deleted the file from the backing store, and `tusd` does not need to do that itself. This would allow us to make that hook use our existing application-layer code to delete uploaded files safely (using 2-phase commit or other standard techniques to avoid invariant violations).
- Add an option to disallow `tusd` deleting/terminated an upload after the `pre-finish` hook has completed successfully.

But maybe I'm misunderstanding something about the protocol -- is there a way that we can close this race?



Discussion: That's a valid point! Access to the uploaded files during the pre-finish hook is protected through tusd's internal locking system. The pre-finish hook is part of the PATCH request handling and the lock is only released once the request handling is completed. Deleting uploads requires acquiring the corresponding locks, which avoids prevents races between pre-finish and termination requestss.

However, post-finish hooks or worker jobs that were started by pre-finish are not protected through these locks, of course. This leads to the situation you observe where a user can delete an upload while your application is trying to process it. Uppy does send a termination request in a few situations. For example, when the user cancels the upload, Uppy terminates the tus upload, as far as I know. You might want to double check with the Uppy project to see if that's correct and whether it can be configured. How a 499 response from Nginx can trigger this though, I don't know.

With all of that in mind, we still need a solution. A pre-terminate hook for giving the user control over file termination is a useful tool, but also defers responsibility of preventing data races to the user, which is not ideal. A more general solution might be better to help in situations where the application is accessing the uploaded files and needs to prevent concurrent access from tusd. I can think of three typical situations where this happens:

1. Application processes uploaded file and tusd is instructed to terminate the upload (what you are mentioning here)
2. Application modifies or deleted uploaded file and tusd attempts to read the upload for a concatenation operation
3. Application wants to cleanup unfinished upload, but user wants to upload at the same time

With a pre-terminate hook (and custom logic) or an option to disable termination of completed uploads, you can prevent 1), but not the others.

I wonder if it's sensible to allow application to interact with tusd's locks. For example, before an application access an uploaded file, it asks tusd to acquire the lock/lease for the corresponding upload resource. This prevents concurrent access from tusd, so no data races are possible. Once the application is done processing, the lock/lease can be released again. This might be a bit more involved, but should cover all cases. What do you think?
I think a pre-terminate hook would be safe enough if `tusd` guaranteed that it was holding its own locks, and accepted a parameter in the hook response for whether the server deleted the file itself or not... if one could rely on the application to not try to interact with unfinished uploads (which seems likely?), those second two problems seem like they'd mostly not be important. 

I guess the structural question is "who owns the uploaded file?" I feel like there's a reasonable mode where `tusd` does and the application isn't allowed to mess with the file, or needs to grab `tusd` locks to do it. As an application author, though, I don't want to have to think about `tusd`'s locking scheme; I'd prefer to abstract that away.

It seems like a lot of applications will want a model where once `tusd` has uploaded a file, the upload process is done, and the application has full ownership over the file. (Zulip is OK with clients deleting previously uploaded files as long as our data structures aren't corrupted in so doing, but one can certainly imagine applications that aren't!).

So conceptually, that'd be a mode where there is a handoff once the upload is complete from `tusd` owning the file to the application owning it, and after that handoff, `tusd` is unable to mutate/delete the file after upload completion, and needs to do something to verify whether the file is still there and unmodified before using it.

I've not had a chance to think this through fully, but that's my initial reaction.
> With a pre-terminate hook (and custom logic) or an option to disable termination of completed uploads, you can prevent 1), but not the others.

I think that this is the easiest way to implement applications claiming ownership of files once they have finished uploading, so I agree with Tim that this is the path that we'd prefer.

Thoughts?
We can add a pre-terminate hook to tusd. That would also allow users to have better control over the termination process and disallow terminations depending on their applications need. For example, they might not want to allows termination of completed uploads.

Would you be open to working on a PR for this?
Work on a `pre-terminate` hook has started in https://github.com/tus/tusd/pull/1248.



Fix (patch):
diff --git a/docs/_advanced-topics/hooks.md b/docs/_advanced-topics/hooks.md
index dac8b94d6..9da5e4917 100644
--- a/docs/_advanced-topics/hooks.md
+++ b/docs/_advanced-topics/hooks.md
@@ -33,6 +33,7 @@ The table below provides an overview of all available hooks.
 | post-receive   | No        | regularly while data is being transmitted.                             | logging upload progress, stopping running uploads                               | Yes                 |
 | pre-finish     | Yes       | after all upload data has been received but before a response is sent. | sending custom data when an upload is finished                                  | No                  |
 | post-finish    | No        | after all upload data has been received and after a response is sent.  | post-processing of upload, logging of upload end                                | Yes                 |
+| pre-terminate  | Yes       | before an upload will be terminated.                                   | checking if an upload should be deleted                                         | No                  |
 | post-terminate | No        | after an upload has been terminated.                                   | clean up of allocated resources                                                 | Yes                 |
 
 Users should be aware of following things:
@@ -161,6 +162,13 @@ Below you can find an annotated, JSON-ish encoded example of a hook response:
     // to the client.
     "RejectUpload": false,
 
+    // RejectTermination will cause upload terminations via DELETE requests to be rejected,
+    // allowing the hook to control whether associated resources are deleted.
+    // This value is only respected for pre-terminate hooks. For other hooks,
+    // it is ignored. Use the HTTPResponse field to send details about the rejection
+    // to the client.
+    "RejectTermination": false,
+
     // ChangeFileInfo can be set to change selected properties of an upload before
     // it has been created.
     // Changes are applied on a per-property basis, meaning that specifying just
diff --git a/pkg/handler/config.go b/pkg/handler/config.go
index e5c670263..480fa243c 100644
--- a/pkg/handler/config.go
+++ b/pkg/handler/config.go
@@ -75,6 +75,13 @@ type Config struct {
 	// If the error is non-nil, the error will be forwarded to the client. Furthermore,
 	// HTTPResponse will be ignored and the error value can contain values for the HTTP response.
 	PreFinishResponseCallback func(hook HookEvent) (HTTPResponse, error)
+	// PreUploadTerminateCallback will be invoked on DELETE requests before an upload is terminated,
+	// giving the application the opportunity to reject the termination. For example, to ensure resources
+	// used by other services are not deleted.
+	// If the callback returns no error, optional values from HTTPResponse will be contained in the HTTP response.
+	// If the error is non-nil, the error will be forwarded to the client. Furthermore,
+	// HTTPResponse will be ignored and the error value can contain values for the HTTP response.
+	PreUploadTerminateCallback func(hook HookEvent) (HTTPResponse, error)
 	// GracefulRequestCompletionTimeout is the timeout for operations to complete after an HTTP
 	// request has ended (successfully or by error). For example, if an HTTP request is interrupted,
 	// instead of stopping immediately, the handler and data store will be given some additional
diff --git a/pkg/handler/unrouted_handler.go b/pkg/handler/unrouted_handler.go
index b56542203..1e1998a79 100644
--- a/pkg/handler/unrouted_handler.go
+++ b/pkg/handler/unrouted_handler.go
@@ -60,6 +60,7 @@ var (
 	ErrInvalidUploadDeferLength         = NewError("ERR_INVALID_UPLOAD_LENGTH_DEFER", "invalid Upload-Defer-Length header", http.StatusBadRequest)
 	ErrUploadStoppedByServer            = NewError("ERR_UPLOAD_STOPPED", "upload has been stopped by server", http.StatusBadRequest)
 	ErrUploadRejectedByServer           = NewError("ERR_UPLOAD_REJECTED", "upload creation has been rejected by server", http.StatusBadRequest)
+	ErrUploadTerminationRejected        = NewError("ERR_UPLOAD_TERMINATION_REJECTED", "upload termination has been rejected by server", http.StatusBadRequest)
 	ErrUploadInterrupted                = NewError("ERR_UPLOAD_INTERRUPTED", "upload has been interrupted by another request for this upload resource", http.StatusBadRequest)
 	ErrServerShutdown                   = NewError("ERR_SERVER_SHUTDOWN", "request has been interrupted because the server is shutting down", http.StatusServiceUnavailable)
 	ErrOriginNotAllowed                 = NewError("ERR_ORIGIN_NOT_ALLOWED", "request origin is not allowed", http.StatusForbidden)
@@ -1203,7 +1204,7 @@ func (handler *UnroutedHandler) DelFile(w http.ResponseWriter, r *http.Request)
 	}
 
 	var info FileInfo
-	if handler.config.NotifyTerminatedUploads {
+	if handler.config.NotifyTerminatedUploads || handler.config.PreUploadTerminateCallback != nil {
 		info, err = upload.GetInfo(c)
 		if err != nil {
 			handler.sendError(c, err)
@@ -1211,15 +1212,26 @@ func (handler *UnroutedHandler) DelFile(w http.ResponseWriter, r *http.Request)
 		}
 	}
 
+	resp := HTTPResponse{
+		StatusCode: http.StatusNoContent,
+	}
+
+	if handler.config.PreUploadTerminateCallback != nil {
+		resp2, err := handler.config.PreUploadTerminateCallback(newHookEvent(c, info))
+		if err != nil {
+			handler.sendError(c, err)
+			return
+		}
+		resp = resp.MergeWith(resp2)
+	}
+
 	err = handler.terminateUpload(c, upload, info)
 	if err != nil {
 		handler.sendError(c, err)
 		return
 	}
 
-	handler.sendResp(c, HTTPResponse{
-		StatusCode: http.StatusNoContent,
-	})
+	handler.sendResp(c, resp)
 }
 
 // terminateUpload passes a given upload to the DataStore's Terminater,
diff --git a/pkg/hooks/hooks.go b/pkg/hooks/hooks.go
index e808d5620..71b3c1598 100644
--- a/pkg/hooks/hooks.go
+++ b/pkg/hooks/hooks.go
@@ -68,6 +68,12 @@ type HookResponse struct {
 	// to the client.
 	RejectUpload bool
 
+	// RejectTermination will cause the termination of the upload to be rejected, keeping the upload.
+	// This value is only respected for pre-terminate hooks. For other hooks,
+	// it is ignored. Use the HTTPResponse field to send details about the rejection
+	// to the client.
+	RejectTermination bool
+
 	// ChangeFileInfo can be set to change selected properties of an upload before
 	// it has been created. See the handler.FileInfoChanges type for more details.
 	// Changes are applied on a per-property basis, meaning that specifying just
@@ -91,10 +97,11 @@ const (
 	HookPostCreate    HookType = "post-create"
 	HookPreCreate     HookType = "pre-create"
 	HookPreFinish     HookType = "pre-finish"
+	HookPreTerminate  HookType = "pre-terminate"
 )
 
 // AvailableHooks is a slice of all hooks that are implemented by tusd.
-var AvailableHooks []HookType = []HookType{HookPreCreate, HookPostCreate, HookPostReceive, HookPostTerminate, HookPostFinish, HookPreFinish}
+var AvailableHooks []HookType = []HookType{HookPreCreate, HookPostCreate, HookPostReceive, HookPreTerminate, HookPostTerminate, HookPostFinish, HookPreFinish}
 
 func preCreateCallback(event handler.HookEvent, hookHandler HookHandler) (handler.HTTPResponse, handler.FileInfoChanges, error) {
 	ok, hookRes, err := invokeHookSync(HookPreCreate, event, hookHandler)
@@ -128,6 +135,26 @@ func preFinishCallback(event handler.HookEvent, hookHandler HookHandler) (handle
 	return httpRes, nil
 }
 
+func preTerminateCallback(event handler.HookEvent, hookHandler HookHandler) (handler.HTTPResponse, error) {
+	ok, hookRes, err := invokeHookSync(HookPreTerminate, event, hookHandler)
+	if !ok || err != nil {
+		return handler.HTTPResponse{}, err
+	}
+
+	httpRes := hookRes.HTTPResponse
+
+	// If the hook response includes the instruction to reject the termination, reuse the error code
+	// and message from ErrUploadTerminationRejected, but also include custom HTTP response values.
+	if hookRes.RejectTermination {
+		err := handler.ErrUploadTerminationRejected
+		err.HTTPResponse = err.HTTPResponse.MergeWith(httpRes)
+
+		return handler.HTTPResponse{}, err
+	}
+
+	return httpRes, nil
+}
+
 func postReceiveCallback(event handler.HookEvent, hookHandler HookHandler) {
 	ok, hookRes, _ := invokeHookSync(HookPostReceive, event, hookHandler)
 	// invokeHookSync already logs the error, if any occurs. So by checking `ok`, we can ensure
@@ -166,12 +193,14 @@ func SetupHookMetrics() {
 	MetricsHookErrorsTotal.WithLabelValues(string(HookPostCreate)).Add(0)
 	MetricsHookErrorsTotal.WithLabelValues(string(HookPreCreate)).Add(0)
 	MetricsHookErrorsTotal.WithLabelValues(string(HookPreFinish)).Add(0)
+	MetricsHookErrorsTotal.WithLabelValues(string(HookPreTerminate)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPostFinish)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPostTerminate)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPostReceive)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPostCreate)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPreCreate)).Add(0)
 	MetricsHookInvocationsTotal.WithLabelValues(string(HookPreFinish)).Add(0)
+	MetricsHookInvocationsTotal.WithLabelValues(string(HookPreTerminate)).Add(0)
 }
 
 func invokeHookAsync(typ HookType, event handler.HookEvent, hookHandler HookHandler) {
@@ -248,6 +277,12 @@ func NewHandlerWithHooks(config *handler.Config, hookHandler HookHandler, enable
 		}
 	}
 
+	if slices.Contains(enabledHooks, HookPreTerminate) {
+		config.PreUploadTerminateCallback = func(event handler.HookEvent) (handler.HTTPResponse, error) {
+			return preTerminateCallback(event, hookHandler)
+		}
+	}
+
 	// Create handler
 	handler, err := handler.NewHandler(*config)
 	if err != nil {