Fix logic error in sparse diff

Fixes #5853