change MultiThreadFileProducer to an abstract class, allow its subclass
to implement the getNextFile method to specify the logic of getting
files.
aslo fix some bugs