Fixup no config runs

@catlee I've been playing with runner to get a feel for the project. These changes make it possible to run it against a directory full of executable files without needing to pass in a config. It also makes the 'get_task_name' logic more flexible.
@rail I decided to harden this a bit. r?