combined processing for training and test set, new dummy functions to handle categorical features