Make worker non-copyable and non-movable

Use workers via `unique_ptr` instead of using them as values. This gets rid of
the weird worker semantics regarding construction and assignment.