Improved undoing of remove to support restoring observers/observees.

Signed-off-by: Jakob Brandsborg Ehmsen <jakobehmsen.github@gmail.com>