rtspconnection: No remove child if destroyed.

Fixes https://bugzilla.gnome.org/show_bug.cgi?id=740730