Turns out it can be done.

By using navigation view controllers which store the hierachy of
viewcontrollers in an array it can return to the previous view
controller