Remove unneeded check for when queue group has 1 queue.

There's no reason to have this extra check since it will be
handled without issue.