Well apparently fontspec wasn't pointless.