Testruns are failing on OS X if real binary or new signed app bundle get specified (#183)

(cherry picked from commit 8ce43df489f813a7dc77f9bd25b2bd1051f8fa1d)