Fix test to match new behaviour

[#53073143]
Merge 53073143-staged-play-app-fails-to-start to master

[Completes #53073143]