added a bunch of small changes to improve data handling