Only add/remove album if it hasn't been already

I think the owners/activity update and library adding should be done
atomically, maybe...