Fail if patch in "patches" does not exist

Rebased #4175 on staging. cc @bjornfor