Removed LerpToTarget and left just interpolation. This has fixed the jitter when replicated objects start moving.