An excellent question about using the celery retry.  The reason is that we may right now be in a retry, trying to update the status recorded in the InstructorTask to indicate that this subtask is being retried.  It is problematic to try to put this update_subtask_status logic into a retry when the code assumes that update_subtask_status has completed successfully before a retry.

I don't think that "the release has to happen before we make the recursive call".  We want to keep the subtask locked from other subtasks until it is complete, if we were to keep trying with updating the InstructorTask here.  The lock on the subtask is actually grabbed in check_subtask_is_valid(), not in _update_subtask_status().  (If it were in the latter, I could use the ContextManager approach to set the lock.)

I guess it might be nice to have a delay, if we were able to use celery retries and make use of a separate process checking whether a particular task should be picked up.  But I don't think that there is a particular advantage in this case to wait between retries that would be worth sleeping for.  It took too long for a select_for_update to get a row lock, yes, but it seems pretty random when this would occur, and retrying as soon as possible seems a reasonable strategy to me.