# Is Open Source the Future of AI? A Data-Driven Approach
## 3 Methodology  
To address our research questions, we first needed to collect data. Given our focus on impact, general data from repositories was not especially beneficial. Instead, we collected data from the Open LLM Leaderboard on Hugging Face [11], where we obtained information on repositories of models that are currently on the leaderboard and models that are awaiting evaluation for the leaderboard through scraping. A Python (version 3.10.12) pipeline was developed to clean and enrich the available data on GitHub ([[https://github.com/VakeDomen/HF_analysis](https://github.com/VakeDomen/HF_analysis)]([https://github.com/VakeDomen/HF_analysis](https://github.com/VakeDomen/HF_analysis)), accessed on 1 March 2025). The leaderboard data include model architecture and precision as well as the model type and performance on the following benchmarks: ARC [12], HellaSwag [13], MMLU [14], TruthfulQA [15], Winograde [16], and GSM8K [17] (see Table 1 for sample of data). In addition to the data provided on the leaderboard, we received supplementary information about the given models using the HF API client as can be seen in Tables 2-4. This included details on repository contributors, tags, base models, used datasets, and repository activity. It is important to note that Hugging Face does not enforce the reporting of this information. Due to the self-reported and optional nature of the data, many models lack this information. The leaderboard also includes duplicates, since the developers can replace models in the repository with different models under the same name. As a result, these duplicates share identical repository data but have distinct performances. Since it is not possible to programmatically determine the current model within the repository, we selected the best-performing model to represent the repository when removing duplicates. Thus, all datasets were prepared for further use. The following analysis was conducted using the R programming language, focusing primarily on obvious trends. The data were categorised using several criteria, such as model type, model architecture, and parameter count. The data were initially selected and aggregated to ensure that all crucial components were easily accessible. Any models that were flagged were excluded from the dataset. In addition, we collected and analysed data on the authors' activities. To fill in the missing components, we extracted the information from tags. To ensure that the analysis reflects genuine human contributions, we manually reviewed the top contributors in the dataset. Automated bot accounts were identified and removed using a two-step process. First filtering usernames that end with 'bot' using string-pattern detection, and then manually verifying suspicious accounts in the top 100 contributors based on activity patterns and naming conventions.  
\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Model** & **Type** & **Params (B)** & **ARC** & **MMLU** & **GSM8K** & **Avg Score** \\ \hline Le Triemphant-ECE-TW3 & Base Merge & 72 & 78.5 & 77.81 & 79.83 & 81.31 \\ Ein-70B-v2 & Fine-tuned & 72 & 79.86 & 78.05 & 75.44 & 81.29 \\ Free-evo-qwen72b-v0.8 & Fine-tuned & 72 & 79.86 & 78.0 & 75.89 & 81.28 \\ Rhea-72b-v0.5 & Fine-tuned & 72 & 79.78 & 77.95 & 76.12 & 81.22 \\ MultiVerse-70B & Chat Model & 72 & 78.67 & 78.22 & 76.65 & 81.00 \\ Ein-72B-v0.1 & Adapter & 72 & 76.45 & 77.14 & 80.06 & 80.99 \\ \hline \hline \end{tabular}  
* Full dataset available on GitHub.  
\end{table}
Table 1: Summary of open-source LLMs’ performance data. Sample of models selected based on Avg Score.