Fix issue when removing QObject counterpart of model elements

Change-Id: I9c3a3ae63d8f8e738a8bdb903c02b47a48627b70
Reviewed-by: Sandro S. Andrade <sandroandrade@kde.org>