Simplified refinement neighbor passing