Fix cases where owner warning is shown

In b0431a5 I added the check in the wrong place which could cause the warning to be shown because of key changes rather than owner changes like the warning suggests.