You will always go to the site that you choose, as long as the other player could travel there. If you rule the Shrouded Wood, you choose their destination, which can be different from the site that you go to. If they rule the Shrouded Wood, they will travel to the same site as you. If a third party rules the Shrouded Wood, the third party will choose where the player travels.