Only wait and close fenceFds if there's factually one

Signed-off-by: Carsten Munk <carsten.munk@jollamobile.com>