If an element is not part of a row, don't do all the logic to remove it.
Catch mouse events that have have invalid rows.