FIx leaky listeners

This is for #9418 
So comment should be: "we don't need to refresh if the item was not moved or opened"