Avoid the cast in toSet and toBuffer