things seem to be working and are running fast enough for this to work now. Now set up to run fewer files at once for distributing amongst nodes on hafen