Set the Team permissions to override Groups, and only set the default to 'All' for Groups.