Fix buildModelMatrix to be robust to single factor

Bug reported by @ncalverley2