inject worker buffer in the worker when destroying an actor

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>