Clear the cached handle if any attributes changes, not just timeout
Many r/w attributes are taken into account only at connection time, so
if the user sets them, we should clear the cached handle.