optimised for speed - transformation now takes 10 minutes