Cleans up background threads related to speech activity more aggressively.