The problem is that we're measuring distance based on the 'natural' key of the track when we're constructing everything, but then in the end we basically disregard that in order to join the tracks together. We need to reinvision the approach we take in deciding how far we've warped a song in its final version, not the best possible warp between it and the other track. The track playing before it could be severely detuned itself, meaning we have to adjust our track even more to get it to match. This implies we can't have a constant track distance -- the distance must adjust based on the warp applied to the track before it.