Got this working finally
Added a optical flow calculation that controls that the flow is the same in both directions, not used