Split out refcounting code that's about to be shared.

#386 will introduce a shared `struct task_group`.