Fix groups super slowness

Spenser Shumaker <sshumake@redhat.com>