Update automation script to handle new application-based folder structure and add support for metro testruns (#89)

Basically a transplant from #97