Moved set parsing to parseFactor

Basically reverted all code back and then never handle sets until
parseFactor.