Revert "Include OAuth token with worker payload"

This reverts commit a0ee9d6b9a935a1bc0f92f34836d49f35eeee83e.

It's most possibly causing performance problems, reverting to see if
it's going to help things.