allow user to throttle # of directories pulled at a time.  Will "help" allow multiple processes to work on the same directory.
this is the user var for the change I just committed
method shouldn't be clearing errors
cause all processing to stop if it sees a flag file