More careful layout updates to fix mouse dragged events in OS X