Fix #781 in which clicks were being lost
Fix removal of hoverables and change pointerEvents to 'none' only when necessary