task-list: ensure we don't expire tasks that have been removed

may or may not fix https://bugzilla.gnome.org/show_bug.cgi?id=724931