calculate similarity for each pair of trees