Fix cases where owner warning is shown

In e78d580 I added the check in the wrong place which could cause the warning to be shown because of key changes rather than owner changes like the warning suggests.