Fix logic error in sparse diff

Fixes 5853