More robust check before eliminating observers

Check to ensure all rows have really been removed before eliminating
observers. Also provide better tolerance of missing rows.