Found issue, math was fine.
I just didn't know that the destination for the bullet had to be the destination according to the map offset.