> So for a data table of len ~ 40k this still took a second, and has the problems identified above. Not ideal!

How does this compare to the version currently in the code?  Is that much more than a second for this size file?