FFI: fix map insert/remove algorithm

Double-remove is not an error due to possible cleanup calls. Cleanup will clean
the map