the training set and testing set creation was incorrect, we do not need
the testing set.
added regularization to always obtain positive definite estimates
only use 3 worker pools