Funnily, repaired the problem without actually finding out what it was. So, I changed the direction calculation and it seems to work better now.