You can specify the grouping on a smartplaylist including grouping into sets (mixed or not) but we don't have the functionality to make the grouping conditional on the number of children in a group.