Thread reference through to the raster block
Keep track of reference frames and copy (with zero MV) for inter prediction