Properly remove callback entries when removing nodes or edges.