Item views: do not track the geometry of items being removed

Task-number: QTBUG-31873
Change-Id: I4230893ccb2925ed9c2429d26b411264bf7c1c65
Reviewed-by: Andrew den Exter <andrew.den.exter@qinetic.com.au>