Reduce reliance on delivery_class

There are a bunch of place we rely on a conversation's `delivery_class` field. Conversations are no longer guaranteed to have this.