buffer: Run destructors before unmapping buffer

This way we avoid unmapping the buffer twice if a destructor calls
wld_unmap (as in wayland-shm).