Prevent KeyErrors for changeset alerts

When the changeset was old we would delete it before trying to check for alerted_already, causing a KeyError.