Found and fixed existing bug.
Remove groups from roster when asked for ones without them.