# OSS PESTO: An Open Source Software Project Evaluation and Selection TOol

Xiaozhou Li, Sergio Moreschini

► To cite this version:

Xiaozhou Li, Sergio Moreschini. OSS PESTO: An Open Source Software Project Evaluation and Selection TOol. 17th IFIP International Conference on Open Source Systems (OSS), May 2021, Lathi/virtual event, Finland. pp.42-50, 10.1007/978-3-030-75251-4_4. hal-03254058

HAL Id: hal-03254058

https://inria.hal.science/hal-03254058

Submitted on 9 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# OSS PESTO: An Open Source Software Project Evaluation and Selection Tool

Xiaozhou Li and Sergio Moreschini

Tampere University
Kalevantie 4, 33100, Tampere, Finland
{xiaozhou.li, sergio.moreschini}@tuni.fi

**Abstract.** Open source software (OSS), playing an increasingly critical role nowadays, has been commonly adopted and integrated in various software products. For many practitioners, selecting and adopting suitable OSS can help them greatly. Though many studies have been conducted on proposing OSS evaluation and selection models, a limited number are followed and used in the industry. Meanwhile, many existing OSS evaluation tools, though providing valuable details, fall short on offering intuitive suggestions in terms of framework-supported evaluation factors. Towards filling the gap, we propose an Open Source Software Project Evaluation and Selection Tool (OSS PESTO). Targeting OSS on Github, the largest OSS source code host, it facilitates the evaluation practice by enabling practitioners to compare candidates therein in terms of selected OSS evaluation models. It also allows in-time Github data collection and customized evaluation that enriches its effectiveness and ease of use.

**Keywords:** Open source Software · Open Source Evaluation · Github Mining

## 1 Introduction

During the last two decades, open source software (OSS) has been flourishing with such trend continuing [10] and nowadays, OSS is adopted by the vast majority of IT companies [13][12]. On Github, the largest OSS source code host, more than 60 million users<fnref n="1" /> have participated in over 100 million open source projects<fnref n="2" />, among which many have been widely adopted by users and companies. However, due to such a large number of candidates, for many practitioners, selecting a suitable OSS product or library is difficult, especially when the relevant information are not explicitly provided [12].

To support the OSS evaluation and selection practice, many studies provide models and frameworks as guidance [7, 18, 21, 24, 23]. During the last two decades, 35 models are proposed with checklists, measures or both provided [12].

<footnote n="1">¹ https://github.com/search?q=type: user&type=Users</footnote>

<footnote n="2">² https://github.com/search?q= is: public</footnote>
They all work similarly with a process of “candidate software identification - factor evaluation - scoring”. In addition, many tools are designed and proposed to facilitate such practice [18, 21, 4, 23]. However, many tools are rigidly designed and allow limited customization. Meanwhile, among those proposed, only two are properly maintained with the majority being not available any more [12].

As the largest OSS host and platform, Github is a valuable channel restoring and presenting OSS related information. Retrospective analysis of Github repositories, based on the abundant information on code, developers, organizations and activities within, can yield valuable insights into the evolution and growth of OSS and facilitate decision making processes [17]. Many studies have used such data source and conduct research on various OSS related perspectives [9, 15, 22]. However, the approaches towards using Github data for OSS evaluation are limited, let alone tools to support such practice.

Herein, we propose OSS PESTO, an open source tool facilitating OSS evaluation and selection. Comparatively, besides being fully open sourced and free to use, OSS PESTO has the following advantages: 1) it allows users to update in-time Github data; 2) it allows users to customize evaluation with models or preference; 3) it allows users to save data locally and to use it without network connection; 4) it is always accessible as maintained in Github repository. It shall largely help the practitioners to compare and evaluate OSS candidates freely, timely and efficiently.

The remainder of this paper is organized as follows. Section 2 introduces the related work on OSS evaluation tools. Section 3 presents OSS PESTO with details. Section 4 presents an experiment validating its applicability. Section 5 concludes the article.

## 2 Related Work

The Open Source Maturity Model (OSMM) is the first proposed model and open standard that aims for such purpose [7]. Guided by OSMM, the practitioners will evaluate OSS by its maturity of each aspect, weight each aspect with importance, and compute its overall maturity by the weighted sum. Compared to OSMM, the Open Business Readiness Rating (OpenBRR) is an OSS evaluation method with more indicators, the idea of target uses and the customized evaluation [24]. The method provides an index applicable to all OSS development initiatives. Its main limits are related to the incompatibility of the requirements between different targets and to the difficulty of choosing the proper reference application for some projects. Similarly, a number of evaluation models are proposed, for example, Qualification and Selection of Open Source Software (QSOS) [18], OpenBQR [21], OSSPAL [23], which provide enhanced guidance and methodological support.

QSOS tool is designed to support the QSOS model which aims to qualify, select and compare OSS products [18]. However, the rigidness of compulsory Identity Card setting and all criteria inclusion is commonly seen as its limitation. OpenQBR [21] requires specification on factor importance before the assessment
of the project. Compared to the QSOS tool, OpenQBR is more elastics as not require to evaluate factors which are not relevant to the specific project.

OSS-PAL [23], though similar to QSOS, aims to partially automate the evaluation of the projects. Despite the appealing goal of the project, it fails to provide the automated data collection function. Other works investigated the availability of the information on online portals [20][14], but they did not provide tools for collecting or aggregating data.

In addition, many other tools are available over time but have been discontinued, including real-time OpenSSL execution monitoring system (ROSEN) [4], RAP TOOL [8], SQO-OSS [19], OMM Tool [5], T-Doc Tool [16], QualiPSo Trustworthiness Checklist [3], MOSST [6] and OP2A Checklist [1] and other checklist included in marketing models for OSS [11][2].

With OSS PESTO, we aim to overcome some of the most common drawbacks of all of these tools, such as, the focus on specific factors, the evaluation of factors before adding a weight function or the lack of control for both internal and external product quality.

## 3 OSS PESTO

We implement OSS PESTO<fnref n="3" /> by following the commonly acknowledged OSS evaluation process summarized from previous studies [12]. It shall contain the following main activities: 1) identify the OSS candidates; 2) elicit a list of factors that need to be evaluated and the according metrics that measure such factors; 3) provide scores or selection recommendation as evaluation output.

In addition, in order to use the latest Github data to evaluate OSS, we integrate a data crawler module in OSS PESTO. It enables the users to crawl the required repository and activities information of any existing OSS projects. Additionally, it also allows them to crawl the data of a list of projects based on the selected range of stars. Furthermore, OSS PESTO allows users to customize evaluation factors based on the selection of models and/or their personal preferences.

<figure><img src="image_2.png" /><figcaption>Fig. 1. OSS PESTO Framework</figcaption></figure>

Shown in Fig. 1, OSS PESTO contains three individual modules as follows.

<footnote n="3">³ Source code: https://github.com/clowee/OSS-PESTO</footnote>
- **Data Crawler**: The data crawler module contains a set of Python scripts that extract Github repository data via Github APIs<fnref n="4" />. It enables the users to select the candidate OSS and extract the according data.

- **Server**: The server side is implemented by ReactJS<fnref n="5" /> while database with MongoDB<fnref n="6" />. The evaluation model is described with the config.json file, which can be altered with users' preference of evaluation factors.

- **Client**: The client side is also implemented by ReactJS. It mainly displays the candidate OSS projects with the selected attributes/factors shown. It also shows the results of candidate comparison which facilitates OSS evaluation and selection.

Fig. 1 also shows the activities of utilizing OSS PESTO to evaluate candidate OSS projects as follows.

- **Step 1.** identifies the OSS candidates by running the data crawler module to extract the according dataset.

- **Step 2.** select the evaluation model, configure evaluation preference, and run the server module.

- **Step 3.** run the client module and compare the OSS candidates by the selected factors.

The crawled data is saved locally in A comma-separated values (CSV) file with each row containing the values of an individual OSS candidate. To be noted, the required data can be selectively crawled according to the users, who determine which metrics are the important ones when evaluating particular aspects of OSS. Such selection of data can be guided by the evaluation model chosen by the evaluator. For example, when selecting only the most popular OSS, the numbers of stars, watches, and download are the ones to be crawled.

<figure><img src="image_3.png" /><figcaption>Fig. 2. An example of Configuration File</figcaption></figure>

Furthermore, the configuration file is a Javascript file mapping the category tabs displayed by the client and the data features/metrics that are selected to evaluate the according categories. Shown in 2 is an example of how a configuration file works. By editing the configuration file, the users can customize their

<footnote n="4">⁴ https://docs.github.com/en/graphql; https://docs.github.com/en/rest</footnote>

<footnote n="5">⁵ https://reactjs.org/</footnote>

<footnote n="6">⁶ https://www.mongodb.com/</footnote>
selection of metrics, the evaluation categories and the links in between. For example, if the user chooses to focus on the popularity of OSS and uses the number of watches as the metric for it, the according piece of code `{ Header: "#Watch", accessor: "watcher_count" }` shall be added to the "Popularity" tab block.

## 4 Experiment Showcase

In order to validate the applicability of OSS PESTO, we conduct a series of experiments, including the testing of all three modules. The testing scenario is to evaluate and compare three JavaScript frameworks, i.e., Angular<sup>7</sup>, Redux<sup>8</sup> and Vue<sup>9</sup> using the OSSPAL model [23]. The evaluation categories include “Community”, “Support”, “Operational Software Characteristics”, “Documentation”, “Software Technology Attributes”, “Functionality” and “Development Process”. Herein, we focus on the “Community”, “Support” and “Software Technology Attributes” aspects, which can be well demonstrated by the obtained data.

To start crawling the Github data, given the user’s Github personal token and the target OSS candidates as input, the data crawler module can be ran individually and continuously. Towards the stated objective, the crawling process takes within two minutes. When the data is ready, we prepare the config.json by selecting the target metrics that are valuable towards evaluating each factors of the candidates. For each of the selected factors, the according metrics are as follows.

- Community: number of watches, number of stars, age, average issue active time, average issue comments, number of pull requests, and number of issue raiser.

- Support: average issue closed time, number of contributor, organization issue raiser.

- Software Technology Attributes: number of open issues, number of dependence.

Thereafter, when running both the server and the client, the comparison result is shown in Fig. 3.

Based on such comparison, we can easily observe that despite not being the oldest community, Vue is more popular than the other two candidates in terms of watches and stars. However, these three communities are active in different ways, as Angular has more comments on issues, pull requests, and different issue raisers while the others are more responsive to issues (shown in Fig.3 (a)). Regarding support, Angular has a much larger contributor group and organizational issue raiser for support, while on software technology attribute aspect, Redux has much less dependence and open issues (shown in Fig.3 (b) and (c)).

When adopting a different evaluation model, it is possible that by taking into account particular overseen metrics, the user obtains new insights regarding the

<footnote n="7">⁷ https://angular.io/</footnote>

<footnote n="8">⁸ https:// redux.js.org/</footnote>

<footnote n="9">⁹ https://vuejs.org/</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Experiment Results Demonstration</figcaption></figure>

selected candidates. For example, the SQO-OSS model [19] sees “Growth in active developers” as a metric to evaluate the “Developer base” category, when OSSPAL has not such category. However, due to the fact that same dataset is used for all potential models, it is hardly possible to have opposite comparative evaluation result for the same category from different models.

## 5 Conclusion

This paper presents OSS PESTO, an open source software project evaluation and selection tool, to support the practitioners' need towards OSS evaluation and selection. This tool provides a Github-repository-data-oriented, easy-to-maintain, customization-friendly solution. It shall benefit the practitioners in both industry and academia in terms of the different focuses on either the OSS projects or the OSS evaluation models respectively.

However, the current version of this tool can certainly be improved in the following ways. Firstly, OSS PESTO has not yet supported the practitioners' selection of OSS candidates at the identification phase in terms of their target functionalities. As the accessible data obtained from Github does not provide explicit information regarding the main features of the OSS, such candidate selection cannot be automated via direct identification. A potential solution is to apply natural language processing (NLP) techniques to identify and summarize such main features from the description and Readme text of the projects. Such a feature shall be implemented in our future work.

Furthermore, the current version only utilizes limited amount of the attributes provided by the Github API. For many such attributes, the explicit mappings towards particular OSS evaluation categories are not verified. For example, the number of OSS downloads can be seen as a metric for its popularity. However, unless a particular user insists it being a critical evaluation criterion for his/her customized evaluation model, such value can be ignored when it does not contribute to any pre-defined evaluation categories. Nonetheless, the inclusion of more data features shall be taken into account in the future work. However, it should be noted such work can result in the exhaustion of Github API query limit, as some values (e.g., issues) can only be obtained via looping enumerated results.
In addition, more features, in terms of the ease of use perspective of the tool, shall be also considered. For example, a graphic user interface is needed for the data crawler module which can also be integrated to the server side. Furthermore, the data from Github has its limitation on reflecting certain aspects of OSS. For example, the development process of the projects cannot be easily accessed externally, except for the number of releases and the release pace. Thus, in order to improve the potential scope of this tool, more data sources are required with more techniques required to process possibly unstructured data as well. Meanwhile, more practical features, such as, exporting the evaluation results, adding weight to different factors, editor of configure files, and model customization interface, are also required.

Our future work shall focus on integrating the modules and enhancing the overall quality of the tool according to the above mentioned limitation. It is also important to investigate the ways of evaluating individual OSS by providing unified quantified results. In addition, we shall systematically investigate the availability of data from multiple sources that could be used to support OSS evaluation.

## References

1. Benlian, A., Hess, T.: Comparing the relative importance of evaluation criteria in proprietary and open-source enterprise application software selection - a conjoint study of erp and office systems. Information Systems Journal **21**(6), 503-525 (2011)

2. del Bianco, V., Lavazza, L., Lenarduzzi, V., Morasca, S., Taibi, D., Tosi, D.: A study on oss marketing and communication strategies. In: Open Source Systems: Long-Term Sustainability. pp. 338-343. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)

3. del Bianco, V., Lavazza, L., Morasca, S., Taibi, D., Tosi, D.: The qualispo approach to oss product quality evaluation. In: Proceedings of the 3rd International Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development. pp. 23-28 (2010)

4. Choi, S.j., Kang, Y.h., Lee, G.s.: A security evaluation and testing methodology for open source software embedded information security system. In: International Conference on Computational Science and Its Applications. pp. 215-224. Springer (2005)

5. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D.: Quality of open source software: the qualipso trustworthiness model. In: IFIP International Conference on Open Source Systems. pp. 199-212. Springer (2009)

6. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D., Tosi, D.: A survey on the importance of some economic factors in the adoption of open source software. In: Software Engineering Research, Management and Applications 2010, pp. 151-162. Springer (2010)

7. Duijnhouwer, F.W., Widdows, C.: Capgemini expert letter open source maturity model. Capgemini pp. 1-18 (2003)

8. Immonen, A., Palviainen, M.: Trustworthiness evaluation and testing of open source components. In: Seventh International Conference on Quality Software (QSIC 2007). pp. 316-321. IEEE (2007)
9. Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: The promises and perils of mining github. In: Proceedings of the 11th working conference on mining software repositories. pp. 92–101 (2014)

10. Kilamo, T., Lenarduzzi, V., Ahoniemi, T., Jaaksi, A., Rahikkala, J., Mikkonen, T.: How the cathedral embraced the bazaar, and the bazaar became a cathedral. In: Ivanov, V., Kruglov, A., Masyagin, S., Sillitti, A., Succi, G. (eds.) Open Source Systems. pp. 141–147. Springer International Publishing, Cham (2020)

11. Lenarduzzi, V.: Towards a marketing strategy for open source software. In: Proceedings of the 12th International Conference on Product Focused Software Development and Process Improvement. p. 31–33. Profes ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2181101.2181109

12. Lenarduzzi, V., Taibi, D., Tosi, D., Lavazza, L., Morasca, S.: Open source software evaluation, selection, and adoption: a systematic literature review. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). pp. 437–444 (2020). https://doi.org/10.1109/SEAA51224.2020.00076

13. Lenarduzzi, V., Tosi, D., Lavazza, L., Morasca, S.: Why do developers adopt open source software? past, present and future. In: Open Source Systems. pp. 104–115. Springer International Publishing, Cham (2019)

14. Li, X., Moreschini, S., Zhang, Z., Taibi, D.: Exploring factors and measures to select open source software. In: Arxiv (2021)

15. Lima, A., Rossi, L., Musolesi, M.: Coding together at scale: Github as a collaborative social network. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 8 (2014)

16. Morasca, S., Taibi, D., Tosi, D.: T-doc: A tool for the automatic generation of testing documentation for oss products. In: IFIP International Conference on Open Source Systems. pp. 200–213. Springer (2010)

17. Munaiah, N., Kroh, S., Cabrey, C., Nagappan, M.: Curating github for engineered software projects. Empirical Software Engineering **22**(6), 3219–3253 (2017)

18. Origin, A.: Method for qualification and selection of open source software (qsos). http://www.qsos.org (Accessed: 2021-01-22)

19. Samoladas, I., Gousios, G., Spinellis, D., Stamelos, I.: The sqo-oss quality model: measurement based open source software evaluation. In: IFIP international conference on open source systems. pp. 237–248. Springer (2008)

20. Sbai, N., Lenarduzzi, V., Taibi, D., Sassi, S.B., Ghezala, H.H.B.: Exploring information from oss repositories and platforms to support oss selection decisions. Information and Software Technology **104**, 104–108 (2018). https://doi.org/10.1016/j.infsof.2018.07.009; https://www.sciencedirect.com/science/article/pii/S0950584918301526

21. Taibi, D., Lavazza, L., Morasca, S.: Openbqr: a framework for the assessment of oss. In: Feller, J., Fitzgerald, B., Scacchi, W., Sillitti, A. (eds.) Open Source Development, Adoption and Innovation. pp. 173–186. Springer US, Boston, MA (2007)

22. Tsay, J., Dabbish, L., Herbsleb, J.: Influence of social and technical factors for evaluating contribution in github. In: Proceedings of the 36th international conference on Software engineering. pp. 356–366 (2014)

23. Wasserman, A.I., Guo, X., McMillian, B., Qian, K., Wei, M.Y., Xu, Q.: Osspal: finding and evaluating open source software. In: IFIP International Conference on Open Source Systems. pp. 193–203. Springer, Cham (2017)

24. Wasserman, A.I., Pal, M., Chan, C.: The business readiness rating: a framework for evaluating open source technical report (2006)