Remove a useless collection
Revert "Don't recreate the set all the time"

This reverts commit 753b48cce0dcca21f7d0a4516ba07df820df2155.
Fix possible overcalculation of constraints

When two tasks ended at the same time and they were followed by idle,
the idle costs would've been incurred multiple times. This is not a
score corruption, but it may have lead the solver astray.