Ok, So that means there's a bug in dirty marking/halo exchanging.  I can't remember if our semantics are if when we do f.dat.data we force a halo exchange or if we only guarantee that local bits are up to date.