Don't try to remove affected building twice.