Don't overwrite X when not using a Gram matrix with no intercept