Backend: Fix the fix: Keep in mind that the frontend may keep a reference on some objects

Signed-off-by: Jerome Flesch <jflesch@gmail.com>