Hide user list when unusable for filtering.

The user list is hidden when it is empty or contains just one user. In
both cases it is of no use to the currently logged in user.