made two versions of tokenization of directory of text files. one uses boost filesystem, which needs a compiled library. the other uses the added dirent.h. filesystem stuff is a pain and needs to be tested on Windows...