Change m_pending_variants to a vector

Fixes #358
Access shared_ptrs by reference here instead of by copy in order to avoid unnecessary reference count increments.