Testruns are failing on OS X if real binary or new signed app bundle get specified (#183)

This will fix issue #183, and allows us to run a real binary and the new signed builds on OS X. @andreeamatei please review ASAP. Also can you please test this patch on Linux and Windows? I would kinda appreciate that.