Only initialze sweepables on new threads

Closes #3999