tidied up, need to do perf tests and add unit tests