## 4 Discussion  
By looking at the results in Figure 4, it can be easily noticed that three distinct models produce a maximum mAP = 0.64 for different settings by using cosine similarity between pairs of embedding vectors and by considering the aggregation of the best chunks' scores instead of using the virtual document vector. In this last case, the obtained mAP is sensibly lower. The most stable model with different input settings of the chunks' definitions, both window and paragraph, and when aggregating the first two best chunk's scores, either by computing their average or their sum, is **(b) all-MiniLM-L6-v2.**  
This model also achieves the same mAP = 0.64 by considering the aggregation of the first best four chunks' scores when using the paragraph as the chunks' definition. Models **(a)** and **(e)** also achieve a maximum mAP = 0.64 when using sentence as the chunks' definition and when aggregating three and six best chunk scores using both sum and average, respectively. Nevertheless, these two models, when aggregating all the chunk scores, produce a lower value of mAP with respect to the model **(b) all-MiniLM-L6-v2.**  
Figure 5 reports the mAP values using the dot product as the matching function. In this case, the best performing model is **(e) msmarco-distilbert-base-tas-b** that, when fed with chunks defined by sentences, reaches mAP = 0.65, taking into account from four to six best chunk scores using either their sum or average. The fact that this model produces the best results when using sentences as the definition of the chunks, whose number in the documents of the collection has the maximum variability (as can be seen in Figure 3) with respect to the other definitions (both paragraphs and windows with different sizes), makes this model the most stable.  
Finally, a significance test was performed to exclude the null hypothesis. To this end, we considered as baseline the model _ch_per_doc_None_avg_0_ that computes the RSV of documents by considering the dot product as the matching function, and by determining the final score as the maximum of the chunks' relevance scores.To evaluate the significance, we implemented the permutation test defined in [35].  
Figure 6 reports 73 out of 345 tested cases in which the significance test is positive when the tested model yields results with respect to the baseline that are both improved and worsened. What can be noticed is that there are more cases of significantly worse results (59 out of 73 significant cases) with respect to the baseline, with a high decrease in mAP (down to \(-\)56%), with respect to significant cases of improved results (14 case out of 73), which also exhibit a small improvement (the maximum improvement reaches 6%). By analysing the results more closely, in the case of both the models **msmarco-distilbert-cos-v5** and **msmarco-roberta-base-ance-firstp**, the decrease in mAP in significant cases is proportional to the number of the chunk relevance scores considered for determining the document relevance; in such cases, the greater the number of considered chunks, the greater the decrease in mAP. This finding suggests that the users' relevance judgments were determined by considering the relevance of one or a few documents, up to four sentences, and not many chunks.  
This finding can also be confirmed for both the models **all-MiniLM-L6-v2** and **msmarco-bert-base-dot-v5**, even if for these two models we do not have many significant cases when the number of considered chunks is high. A different behaviour can be observed for the last model **msmarco-distilbert-base-tas-b**; in this case, improved results are obtained by aggregating four or five sentences' relevance scores, while for the other types of chunks, the results are worse.