Make sure to set the dirty flag when we changed something