time-selector: ref builder objects before destroying it

Fixes the annoying bug