I prefer having them separated. Then authorization looks prettier. 
profile/leave 
  -> I can leave any group if im not the last owner
user_groups/destroy 
  -> I can manage group membership only if I have manager access