mm: vm: Fix thread safety and fix segment removal

Signed-off-by: Bart Kuivenhoven <bemkuivenhoven@gmail.com>