Actually, make ConfigReferenceSet an abstract class.

If this were java 8 I'd make it an interface with a default method.
Different config reference gatherers might store things differently.