test-hwloc-diffpatch: exit the temp directory before removing it

Otherwise rmdir fails on AIX

(cherry picked from commit c05fc46fa58d792b13b6b4aec30d7a81877185f4)