test-hwloc-diffpatch: exit the temp directory before removing it

Otherwise rmdir fails on AIX