Partially fix use after free/timing issues when worker still executes after client has been closed
Extra protection to make sure we're not freeing clients while reads are occuring