[1.5] Update automation script to also handle the new application-based structure (#89)

Update for #91

The code mentioned in Dave's last comment was indeed not needed.
All the checks and fallback mechanism is already done as part of get_tests_folder()

I've checked that we use the correct folder to run tests on both OSX and Windows across all testruns