Skip work when modifier set is empty.

This both avoids an exception from EnumSet.copyOf when empty and avoids wasting an iterator on looping over an empty set.
Merge pull request #55 from square/jw/empty-non-enum-sets

Skip work when modifier set is empty.