processing code for converting text files to intermediate data