No need to use unordered_set of a small set.

This also avoids TR1 compatibility issues between clang
and g++.