Change it to simply take a pointer with the explicit requirement that the caller is responsible for the lifetime of the object?  For most C++ programmers that's the principle of least surprise and that way you can, for example, embed the worker object in another object.