# Leveraging Large Language Models for Sensor Data Retrieval
## 5 Evaluation
### Results and Discussion
#### 5.3.1 Retrieval Performance  
This section presents the evaluation of the performance of the different LLMs. In order to automatically evaluate the retrieval performance of the models, for each CSV file in the dataset gathered, 10% of the rows were randomly extracted and used to configure a new table to be used as a query in the evaluation. The remaining 90% of the content of each table was indexed and used as the target of the data retrieval system. Thus, when a query table with 10% of the information was issued to the system, the goal was to retrieve the table with the 90% of information that complements it. If this is the case, there is a match, and it is considered that the system satisfactorily retrieved a dataset that can augment the query data. The system was evaluated in terms of precision at one (P@1) and MMR, in the task of retrieving relevant sensor data to augment a query table. The P@1 measure indicates the model's accuracy in retrieving the most relevant dataset as the top result, while MRR provides an average of the reciprocal ranks of the models' results, giving a sense of the overall quality of the retrieved datasets across the top-ranked positions. Other popular metrics in information retrieval, such as nDCG or recall, cannot be applied in this context since they require more than one relevant document to be implemented. Table 2 provides the performance achieved by each model. The results show that the current cutting-edge model _UAE-Large-v1_ is not yielding the expected outcomes, showing the poorest scores in P@1 and MMR. This behaviour might be attributed to its backbone model, the pre-trained uncased BERT base model (with 110M parameters), fine-tuned on a \begin{table}
\begin{tabular}{l l} \hline \hline
**Characteristic** & **Value** \\ \hline Number of tables & 1299 \\ Total number of rows & 163,981,050 \\ Total number of columns & 10,487 \\ Total number of numerical columns & 4746 \\ Avg. number of rows & 126,236.37 \\ Avg. number of columns & 8.07 \\ Avg. number of numerical columns & 3.65 \\ Max. number of rows & 324,000 \\ Max. number of columns & 26 \\ Max. number of numerical columns & 20 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Characteristics of the sensors dataset gathered. wide range of data primarily focusing on other specific topics. This indicates that the model may struggle to provide representative embeddings for the experimental data without prior fine-tuning. The _bge-base-en-v1.5_ model exhibited exceptional accuracy, achieving 0.90 P@1 and 0.94 MRR, indicating that it is highly accurate at not only placing the most relevant dataset at the top of its retrieval list but also maintaining high relevance across its top-ranked results. This suggests that _bge-base-en-v1.5_ has a robust understanding of tabular data representation. Interestingly, the results of its large version did not surpass its smaller model. The _bge-small-en-v1.5_ model, utilising only 384 dimensions, exhibited remarkable performance, achieving the second-best results and emerging as a promising option for real-time systems due to its speed and lightweight nature. Ember's results, despite holding a top position on the MTEB leaderboard, did not reflect similarly in these experiments. This discrepancy might be related to their training data not aligning with the data used in this experiment, resulting in non-representative word embeddings. On the other hand, the GTE models achieved moderately good results, with the large model showing slightly higher precision compared to the base models. Conversely, the _all-MiniLM-L6-v2_, one of the smallest and oldest models, delivered scores slightly higher than _ember-v1_, maintaining competitiveness but being outperformed by more recent models like the BGE variants. The results suggest that base versions of models (_bge-base-en-v1.5_ and _gte-base_) provide a good balance between size and performance.