task-list: try even harder to not expire removed tasks

probably actually fixed https://bugzilla.gnome.org/show_bug.cgi?id=724931