Correctly update buffer location when setting from outside