# Towards Improving OSS Products Selection – Matching Selectors and OSS Communities Perspectives

Claudia Ayala, Daniela S. Cruzes, Xavier Franch, Reidar Conradi

► To cite this version:

Claudia Ayala, Daniela S. Cruzes, Xavier Franch, Reidar Conradi. Towards Improving OSS Products Selection – Matching Selectors and OSS Communities Perspectives. 9th Open Source Software (OSS), Oct 2011, Salvador, Brazil. pp.244-258, 10.1007/978-3-642-24418-6_17. hal-01570750

HAL Id: hal-01570750

https://inria.hal.science/hal-01570750

Submitted on 31 Jul 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Improving OSS Products Selection– Matching Selectors and OSS Communities Perspectives

Claudia Ayala¹, Daniela Cruzes², Xavier Franch¹, Reidar Conradi²

¹ Technical University of Catalunya
UPC-Campus Nord (Omega), 08034 Barcelona, Spain
{cayala, franch}@essi.upc.edu

² Norwegian University of Science and Technology
NTNU-Gløshaugen, Trondheim, Norway.
{dcruzes, conradi}@idi.ntnu.no

**Abstract.** Adopting third-party software is becoming an economical and strategic need for today organizations. A fundamental part of its successful adoption is the informed selection of products that best fit the organization needs. One of the main current problems hampering selection, specially of OSS products is the vast amount of unstructured, incomplete, evolvable and widespread information about products that highly increases the risks of taking a wrong decision. In this paper, we aim to inform and provide evidence to OSS communities that help them to envisage improvements on their information rendering strategies to satisfy industrial OSS selectors' needs. Our results are from the matching between the informational needs of 23 OSS selectors from diverse software-intensive organizations, and the in-depth study of 9 OSS communities of different sizes and domains. The results evidenced specific areas of improvement that might help to enhance the industrial OSS selection practice.

**Keywords:** Open Source Software, selection, information rendering strategy, empirical study.

## 1 Introduction

Nowadays, the use of Open Source Software (OSS) provided by OSS communities is revolutionizing the software industry [1]. The fact that OSS products are freely available has influenced not only their significant adoption, but also the way that software is developed and commercialized [12]. Thus, fostering OSS adoption has been recognized as a crucial task for progressing towards improvements in a great variety of application areas [26].

The potential advantages of adopting OSS greatly depend on the ability to select the most suitable product for the task at hand [4]. Improper selection of an OSS product may result in wrong strategic decisions with subsequent economic loss and adverse effects on the business processes of the organizations [16].
In recent years there has been a plethora of proposals aimed to support software products selection, usually suggesting sets of evaluation criteria to evaluate and decide the most suitable alternative(s) (see [16], [18], [21], for comprehensive surveys). However, a recent survey about industrial OSS selection practices [2] shows that these proposals have not been greatly adopted in the industrial practice. Instead, in order to face time-to-market demands and reducing the potential risks, selectors (i.e., the person(s) in charge of the selection process) just base most of their decisions on their experience and tend to limit the use of OSS products to those that are already known and used by the development team. While the value of experience is important, the fact that it is currently considered as the most influential factor for selecting components is at the same time hampering the adoption and fully exploitation of the potential benefits of the high variety of OSS products in the marketplace. Furthermore, such study evidenced that one of the key problems is that selectors are struggling not only with the current diversity of OSS products available in the marketplace, but also with the great deal of widespread, incomplete, heterogeneous, and unstructured information describing each of them (e.g., formal/informal documentation, tutorials, comments in forums, internal experiences) that makes difficult to face a suitable selection process under time-to-market pressures [3], [4], [17]. In addition, the study emphasized that the main source for gathering OSS product information is the OSS community website.

In this context, in order to contribute to enable suitable OSS selection processes we need to envisage more pragmatic approaches than suggesting sets of evaluation criteria (especially when the evidence shows that the data to fill in these criteria is not usually available). Therefore, the goal of this study is to explore the following research questions:

- RQ1. How much of the information required by OSS selectors for performing a suitable selection process is actually provided by OSS communities?

- RQ2: Are there OSS community characteristics that seem to influence its level of readiness for supporting OSS selection?

With RQ1, we want to investigate the gap between the information “provided” by OSS communities on their OSS community websites and the information required by OSS selectors to perform an informed selection. RQ2 aims to explore whether some OSS community characteristic(s) seem to affect its readiness (i.e., the degree that the community covers the needs of OSS selectors). By answering these research questions, this paper aims to inform and provide evidence to OSS communities that help them to envisage improvements on their information rendering strategies. It is a first step to raise their awareness on areas that are required to improve the OSS selection industrial practice. This paper reports our results from a study about the matching of 9 OSS communities vs. the needs of 23 industrial OSS selectors.

The rest of the paper is organized as follows: Section 2 provides a brief background of the OSS literature and marketing principles that emphasize the importance of dealing with information rendering aspects and their influence on OSS products selection. Section 3 describes the methodological approach followed to perform the study. Section 4 presents the results obtained from the study, while
Section 5 provides a discussion of main findings. Threats to validity are presented in Section 6. Section 7 summarizes the conclusions and future work.

## 2 Background

OSS research has largely ignored one interesting aspect that is becoming crucial for OSS projects: first-impression management [7]. Impression management theory refers to the process by which individuals or organizations try to control or manage the impressions that others form of them [28]. Due to the ever-increasing amount of information available on the Internet and the need to make quick choices among competing alternatives, first-impression management has been adopted as one of the main theoretical lenses in marketing literature. Choi et al [7] demonstrated that the OSS community website plays a critical role in attracting developers and users to the community. The mature status of well-known OSS projects likely attracts users given their greater activity and vitality. Furthermore some OSS products have become de facto standards. However this pathway is unavailable for most of the OSS projects and those newly initiated projects that struggle to attract users and contributors [6].

In the context of OSS selection if an OSS project is poorly presented and potential selectors feel that the community does not invest much care in providing the needed information for selection, they might formulate negative opinions about the project and fail to consider it as a candidate even if it might represent a promising alternative. Thus, poor first impressions not only impact on the rate of potential users in the short run, they can also produce negative externalities for the project in the long run. For instance, they might lose the synergies derived from collaborating with companies, such as greater project activity, higher user's base and popularity [7].

Several initiatives exist to develop a framework for the assessment of OSS products. Most of these initiatives suggest different kinds of criteria such as functionality, maturity, and the strategy of the organization around OSS. Relevant examples are: OpenBRR (Open Business Readiness Rating) [22], QSOS (Qualification and Selection of Open Source software) [25], or OSMM (Navica Open Source Maturity Model) [14]. The evaluation criteria are further explored by, for instance, Cruz et al. [9], the QualOSS Model Framework [8], and the QualiPSo model of OSS trustworthiness [10]. In addition, the factors that might attract developers to participate in OSS projects in order to sustain the vitality of the community have been also studied [5], [7], [11], [15]. However, as far as we know, there are not empirical studies that consider the needs of industrial OSS selectors as a way to improve first impression management.

Consolidated works from the marketing research have developed relevant tactics to help to influence first impression in a positive way [29]. We think that dealing with first impression management is important for OSS research for two main reasons: 1) OSS community contributors typically join OSS projects by first becoming selectors, subsequently as users and then evolving into contributors [7], therefore first impression management is a potential way to attract OSS potential users. 2) Marketing strategies are becoming crucial to pose OSS products into the marketplace given that nowadays the OSS phenomenon has evolved into a more commercially
viable form, where both volunteers and commercial organizations collaborate in its ongoing development [1].

One of the initial grounds of first-impression management is to explore the needs of the potential users and to envisage the improvement tactics. Therefore, this study was designed to explore how the needs of OSS selectors are covered by OSS communities in order to raise observations that may serve to envisage suitable improvement tactics.

# 3 The Study

The study performed in this work was exploratory and aimed to investigate the research questions introduced above. Our main research strategy consisted of the in-depth study of 9 OSS community-based projects and exploring how these communities covered the informational needs of 23 industrial OSS selectors.

## 3.1 Sampling

The target population of the study was OSS community projects. Since the variety of OSS projects is quite wide not only regarding domain and size, but also regarding activity and popularity, we approached a stratified random sampling for improving its representativeness as well as the analysis of the results. We used the Ohloh.net directory as the reference directory for selecting OSS projects. We choose Ohloh as it is one of the largest and up-to-date OSS directories available, and has been widely used to create historical reports about the changing demographics of OSS projects.

From the 437,982 OSS projects referenced in Ohloh by February 2010, we ordered them with respect to their number of users and downloads. Then, we divided such a list into three equal parts (that were considered as stratums). Subsequently, we randomly selected 3 projects from each stratum. Table 1 summarizes the projects that fall into each stratum and provides a brief description of each of them.

## 3.2 Data Collection Instrument

In order to assess the OSS projects in a homogeneous way, we developed a data collection instrument based on the survey reported in [2]. This survey provides data about the information that is required in order to perform an informed OSS selection. 23 OSS selectors from 20 small and medium organizations in Spain, Norway and Luxembourg participated in the survey. It consisted of semi-structured interviews that were recorded in audio and then transcribed to text. We had access to the raw data from the respondents of such study (audio and text documents). From the responses of the selectors that participated in the study we elicited a total of 85 informational needs (i.e., specific information that they referred as needed for making informed decisions). We arranged similar answers using content analysis [19]. This process resulted in 21 informational needs. It is important to remark that similar efforts for establishing important evaluation criteria for selecting OSS products have
been done in the literature as stated in section 2. Our primary goal was to assess if the information required by the selectors was provided by the OSS communities.

<table><caption>**Table 1.** Stratified random sampling</caption><thead><tr><th>Stratum</th><th>Name</th><th>Description</th></tr></thead><tbody><tr><td></td><td>Agilo for Scrum</td><td>It is one of the most widely used Scrum tools, offering many features to support Scrum and software development teams.</td></tr><tr><td rowspan="2">1</td><td>Joomla</td><td>It is an award-winning content management system (CMS), which enables to build Web sites and powerful online applications.</td></tr><tr><td>Subclipse</td><td>It is an Eclipse Team Provider plug-in providing support for Subversion within the Eclipse IDE.</td></tr><tr><td></td><td>Gimp</td><td>GIMP is an acronym for GNU Image Manipulation Program. It is a freely distributed program for such tasks as photo retouching, image composition and image authoring.</td></tr><tr><td rowspan="2">2</td><td>GNU Grub</td><td>GRUB is the GRand Unified Bootloader for GNU</td></tr><tr><td>IpTables</td><td>iptables is the user space command line program used to configure the Linux 2.4.x and 2.6.x IPv4 packet filtering ruleset. It is targeted towards system administrators.</td></tr><tr><td></td><td>Fluent NHibernate</td><td>A fluent API for simplifying the entity mapping of NHibernate. Add compile time safety, testability, and improved readability to NHibernate projects.</td></tr><tr><td rowspan="2">3</td><td>MediaCoder</td><td>It is a free universal batch media transcoder, which integrates most popular audio/video codecs and tools into an all-in-one solution. New codecs and tools are added in constantly as well as support for new devices.</td></tr><tr><td>StatusNet</td><td>StatusNet (formerly Laconica) is a microblogging service. Users post short (140 character) notices that are broadcast to their friends and fans using the Web, RSS, or instant messages.</td></tr></tbody></table>

In order to improve the quality of the data collection instrument, it was pre-tested with three researchers. Consequently, we decided to arrange the 21 informational needs in categories and subcategories that provided a more understandable, structured and informative way of collecting them. For instance, the informational need *Time of the product in the market* was grouped in the sub-category *History of the Product* which at its time was grouped in the category *To ensure technological stability and evolution of the OSS product and its provider*. This arrangement demonstrated to provide researchers with a better understanding of the informational needs and their contexts; it therefore enhanced the information gathering process. As a result, the 21 informational needs originally gathered were grouped into 8 categories and 3 sub-categories. The data collection instrument also gather information as: whether the informational need was provided or not by the OSS community; where the information was found, the time required to skim the OSS community website to find the information, and further comments from the researchers that performed and/or reviewed the OSS community. As it can be noted, our intention was not only to explore if the informational needs were available but also to have a first impression about how it was advertised, and how difficult it was to extract it. The resulting data collection instrument is shown in Table 2 and it also provides an example to illustrate the kind of information that was gathered.
<table><caption>**Table 2.** Data collection instrument. Example of the assessment of the Agilo for Scrum project</caption><thead><tr><th>Category/ Subcategory/ Informational Need</th><th>Status</th><th>Where</th><th>Results of the exploration</th><th>Further comments</th></tr></thead><tbody><tr><td>1 Compliance with client's functional requirements</td><td></td><td></td><td></td><td></td></tr><tr><td>List of main functional requirements of the OSS project.</td><td>√</td><td>(1)</td><td>The list of features provided seems quite comprehensive. If further information is required, there is an email available. We asked for further information about the product and our request was quickly processed.</td><td></td></tr><tr><td>2 To ensure technological stability and evolution of the OSS product and its provider</td><td></td><td></td><td></td><td></td></tr><tr><td>Is it a commercial firm leading the community?</td><td>√</td><td></td><td>The community is lead by a single company called Agile42</td><td></td></tr><tr><td>List of companies/organizations collaborating in the community.</td><td>x</td><td></td><td>There is no information about any other company participating in the community.</td><td></td></tr><tr><td>History of the product.</td><td></td><td></td><td></td><td></td></tr><tr><td>Time of the product in the market.</td><td>x</td><td></td><td>Navigated through the wiki and could not find this information</td><td></td></tr><tr><td>Versions of the product available.</td><td>x</td><td></td><td></td><td></td></tr><tr><td>3 Evidence of successful OSS product usage</td><td></td><td></td><td></td><td></td></tr><tr><td>Number of registered users</td><td>x</td><td></td><td></td><td></td></tr><tr><td>List of companies using the product</td><td>√</td><td>(2)</td><td>There is documentation about success stories in companies as: ASDIS, eBuddy, be2, Ericsson, DHD24, Hypoport, Princeton Financial systems and RES software among others.</td><td></td></tr><tr><td>Number of downloads</td><td>x</td><td></td><td></td><td></td></tr><tr><td>Ratings and comments from users</td><td></td><td>√</td><td>Some textual comments from users as: "The Agile project approach allowed be2 the ability to monitor the project's performance every two weeks, and to evaluate the performance and quality. agile42 did a perfect job in training and coaching distributed Scrum teams based in both Germany and Armenia for this project. agile42 has been an indispensable link for a successful company transformation." Dave Sharrock, Director IT, be2 S.a.r.l....</td><td></td></tr><tr><td>4 Ease of OSS product integration</td><td></td><td></td><td></td><td></td></tr><tr><td>Interoperability issues</td><td></td><td></td><td></td><td></td></tr><tr><td>List of software system and subsystems required to ensure the correct functioning of the product.</td><td>√</td><td></td><td>Windows, MAC/Linux, and Trac 0.11. The information was widespread and it was not easy to gather it.</td><td></td></tr><tr><td>Hardware requirements</td><td>x</td><td></td><td></td><td></td></tr><tr><td>Suitability of Code</td><td></td><td></td><td></td><td></td></tr><tr><td>Well-commented code</td><td>√</td><td></td><td>Very well commented</td><td></td></tr><tr><td>Programming language</td><td>√</td><td></td><td>Python</td><td></td></tr><tr><td>5 Availability of support</td><td></td><td></td><td></td><td></td></tr><tr><td>Free services</td><td>√</td><td>(3)</td><td>Very basic ones. Most of their services are not free. They have a commercial license where they provide further professional services. They also offer a blog, Google groups for commenting things about the product, and a Wiki for free.</td><td></td></tr><tr><td>Non-Free services</td><td>√</td><td>(4)</td><td>Personalized services. In addition there is an improved version of the product that is not OSS.</td><td></td></tr><tr><td>6 Availability of Tests Results</td><td></td><td></td><td></td><td></td></tr><tr><td>Tests done by the OSS community</td><td>x</td><td></td><td></td><td></td></tr><tr><td>Tests done by an external party</td><td>x</td><td></td><td></td><td></td></tr><tr><td>7 Licensing terms</td><td></td><td></td><td></td><td></td></tr><tr><td>Availability of detailed information about the licensing terms and explicitly state if they are listed by the OSS initiative</td><td>√</td><td>(5)</td><td>There are 2 licensing schemas. One that complies with the Apache Software License, and a non OSS (offering an improved version of the product)</td><td></td></tr><tr><td>8 Availability of documentation</td><td></td><td></td><td></td><td></td></tr><tr><td>Documentation for final users</td><td>√</td><td>(6)</td><td>The documentation quality seems acceptable</td><td></td></tr><tr><td>Documentation for developers</td><td>√</td><td>(7)</td><td>The documentation for integrators is very basic and scarce</td><td></td></tr><tr><td>Available languages of the documentation</td><td>☑</td><td></td><td>Only English</td><td></td></tr></tbody><tfoot><tr><td colspan="5">General comments: The page is more oriented to business (the community is lead by a company called Agile42). A wiki is provided to report bugs and to inform about possible further involvement with the community as contributors.</td></tr><tr><td colspan="5">Name of the researchers: TP + CA<br/>Date of the assessment: 11/2010 - reviewed 02/2011<br/>Mean time required for skimming the webpage: 4 hours</td></tr></tfoot></table>

<footnote>(1) http://www.agile42.com/cms/pages/features/; (2) http://www.agile42.com/cms/pages/references/; (3) https://dev.agile42.com/wiki</footnote>

<footnote>http://groups.google.com/group/agilo/topics http://agile 42.com/cms/blog/; (4) http://agile42.com/cms/pages/support/; (5)</footnote>

<footnote>http://agile42.com/cms/pages/agilo/; (6) http://agile42.com/cms/pages/agilo-documentation/; (7) https://dev.agile42.com/wiki/agilo/dev</footnote>
## 3.3 Study Procedures and Data Analysis

Each OSS project was assessed using the data collection instrument introduced above. Two different researchers were in charge of assessing each OSS community website. Subsequently, they discussed and agreed the observations. Once all OSS projects were explored and reviewed, the whole research team held discussion meetings to analyze the data and consolidate the results.

# 4 Results of the Study

Results are grouped in two subsections according to the research questions introduced above.

Table 3 shows a summary of the results from the analysis of the 9 OSS communities. In order to provide insights of the coverage of each OSS community to the selectors' needs, we assigned relative weights to each category of the data collection instrument as shown at the right side of each category in Table 3. Such assignment was based on the number of selectors' responses grouped into the category. For instance, the category *Compliance with client's functional requirements* had 18 similar responses; therefore its relative weight with respect to the 85 selector's responses resulted in 21.18%. The category *Availability and quality of the documentation* grouped 3 responses; therefore its weight was 3.53%. Based on such weights we calculated the percentage of coverage of each community to the categories of the data collection instrument. The last row of Table 3 shows the final coverage of each OSS community to the OSS selectors needs. These weights allow us to summarize our findings and provide useful insights to the reader to identify and understand the categories where there is a higher need of improvements.

### 4.1 How much of the information required by selectors is provided by OSS communities?

We found that the most important informational need belonging to the category *Compliance with client's functional requirements* was covered by all the analyzed projects. All of them show (with diverse levels of detail) a list of features of the OSS product.

The information required *To ensure technological stability and evolution of the OSS product and its provider* was poorly covered by most of the studied OSS communities. In addition, the coverage of informational needs belonging to this category was very diverse (see Standard Deviation in last column of Table 3). Most of the analyzed communities failed to clarify the kind of involvement of commercial firms. While in some cases it was clear that the leader of the community was a commercial firm and that several companies were also collaborating in the community under diverse schemas (coding, sponsoring, donating, etc.), in some other cases this information was not clear. The case of MediaCoder was outstanding as the project has radically changed its OSS nature by a purely commercial approach. At this respect, we found controversial comments in Ohloh claiming that MediaCoder should
not be listed therein anymore mainly because the source code is not actually available
and this violates one of the principles of OSS [23]. Other projects that did not offer
clear information about the involvement of companies were FluentNHibernate and
StatusNet, our observations regarding these projects led us to realize that such lack of
clarity might come from the fact that these communities are currently in the process
of defining a new business strategy by establishing commercial entities for making
business around the products (e.g., selling expert support). Furthermore, basic
informational needs as *Time of the product in the market* and *Versions of the product
available* were not provided by several communities, especially those with a
commercial orientation.

The informational needs grouped in the category Evidence of Successful OSS product usage were the ones that most communities failed to cover. None of the studied projects covered all the informational needs belonging to this category. Any of the OSS projects offered information about the number of downloads. Only two communities stated the number of registered users. Five projects stated a list of companies that have successfully used the product, and just one project offered comments from users of the product.

The informational needs belonging to the category *Evidence of Successful OSS product usage* were mostly covered by the studied projects. The only informational need that was not successfully covered by most projects was related to the *Hardware requirements* needed to ensure the correct functioning of the OSS product. MediaCoder also failed to provide *well-commented code and programming language* (this is again due to the fact that it does not provide the source code of the product).

Regarding the category *Availability of support*, in all projects it was explicitly stated whether they provide non-free support, while free support was commonly characterized by wikis, email lists, IRC channels, and forums.

Most projects, excepting two (MediaCoder and StatusNet) offered clear information regarding *Licensing terms*. As mentioned above, these two projects were facing a business model change and therefore their licensing schemas were not clearly stated. Finally, regarding the *Availability of documentation*, almost all projects offered documentation for final users and for developers and most of them offered a variety of languages.

Summarizing, we found that the analyzed OSS projects cover the selectors' needs in a diverse degree. Such coverage ranges from 44.96% to 80.89%. Further discussions are provided in section 5.

## 4.2 Are there OSS project characteristics that influence its level of readiness for supporting selection?

The assessment of the 9 OSS community projects leads us to state some observations regarding characteristics that might affect the information rendering aspects of OSS communities and therefore their readiness for supporting selection. The most relevant ones suggest that two interrelated characteristics seem to affect the information rendering aspects of OSS projects: the involvement of commercial firms and the strata that the projects belong to.
We observed substantial differences among the 3 Stratums. Surprisingly, all OSS projects from Stratum 1 have a close involvement of commercial firms in the community. This finding is in line with the results from the study reported in [5] that evidenced that firms coordinate, develop code for, or provide libraries to one third of the 300 most active OSS projects in SourceForge. Projects from Stratum 2 did not have commercial firms leading the projects, instead they referred to volunteer-based communities that fully adhered to the Free Software Foundation (FSF) and two of them (Gimp and Grub) were part of the GNU project that advocate for the “free software” philosophy. Projects from the Stratum 3 also show a high involvement of commercial firms. While MediaCoder is currently a purely commercial project, Fluent NHibernate and StatusNet are facing a transition stage for becoming business-oriented OSS communities. These facts are in line with the “commercialization” of OSS predicted by [12].

The involvement of firms in the OSS communities seems to influence their aesthetic appearance and information rendering aspects. For instance, Agilo for Scrum is an OSS project entirely governed and led by the company Agile42, and so its website is more oriented to business (i.e., selling services around the product) than to promote the involvement of potential contributors to the community. In the case of the Joomla!, the involvement of firms seems to be quite different as even if firms are quite involved in the project, the project is governed by the community. Thus, its website reflects a strong interest to promote resources for consolidating the community and attract contributors. It also offers several schemas for companies and organizations to participate in the community (i.e., donations, selling services around the product, merchandizing). The website of Subclipse is led by the company CollabNet and the provided resources are more oriented to final users (i.e., instructions on how to install the plugging) than to contribute to the community. Other examples are FluentNHibernate and StatusNet that are currently approaching business oriented models and are also improving the aesthetic appearance of their portals. Therefore, we suggest that: as higher the involvement of commercial firms is, the lesser seems to be the attention paid to promote the involvement of potential contributors in the community.

Communities without commercial firms involved shared several commonalities. It seems that they are mostly aimed to provide technical resources to strengthen the developers' community than aesthetic and attractive resources. This coincides with some studies that emphasize that some OSS projects mostly leaded by community programmers often value substance over form and some exhibit an antipathy for marketing and public relations work [13]. In all cases, these OSS projects provide mailing lists, forums and wikis aimed to enable the collaboration among the members.

Thus, our results might suggest that it would be useful to distinguish among: Commercial OSS, Foundation-based OSS, and Community-based OSS in order to better understand and assess the implications of selecting each kind of OSS product.
<table><caption>Table 3. Summary of results</caption><thead><tr><th>Category/Subcategory/ Informational Need</th><th>Agilo for Scrum</th><th>Joomla</th><th>Subclipse</th><th>Gimp</th><th>GNU Grub</th><th>IpTables</th><th>Fluent NHiberate</th><th>Media Coder</th><th>StatusNet</th><th>Std Dev.</th></tr></thead><tbody><tr><td>Compliance with client's functional requirements (21.18%)</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>21.18</td><td>0</td></tr><tr><td>List of main functional requirements of the OSS project.</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>To ensure technological stability and evolution of the OSS product and its provider (21.18%)</td><td>5.30</td><td>15.89</td><td>10.59</td><td>21.18</td><td>21.18</td><td>21.18</td><td>10.59</td><td>5.30</td><td>0</td><td>7.28</td></tr><tr><td>Is the project governed by a commercial firm or by the community?</td><td>✓ commercial</td><td>✓ community</td><td>✓ commercial</td><td>✓ GNU</td><td>✓ GNU</td><td>✓ community</td><td>Not clear</td><td>✓ commercial</td><td>Not clear</td><td></td></tr><tr><td>List of companies/organizations collaborating in the community (others than the leader)</td><td>×</td><td>✓</td><td>×</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>History of the product.</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Time of the product in the market.</td><td>×</td><td>✓</td><td>×</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td></td></tr><tr><td>Versions of the product available.</td><td>×</td><td>×</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td></td></tr><tr><td>Evidence of successful OSS product usage (12.94%)</td><td>6.47</td><td>6.47</td><td>3.24</td><td>3.24</td><td>3.24</td><td>0</td><td>0</td><td>0</td><td>3.24</td><td>2.53</td></tr><tr><td>Number of registered users</td><td>×</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>List of companies using the product</td><td>✓</td><td>✓</td><td>×</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>Number of downloads</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>Ratings and comments from users</td><td>✓</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>Ease of OSS product integration (12.94%)</td><td>9.71</td><td>9.71</td><td>12.94</td><td>12.94</td><td>9.71</td><td>9.71</td><td>6.47</td><td>9.71</td><td>9.71</td><td>1.94</td></tr><tr><td>Interoperability issues</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>List of software system and subsystems required to ensure the correct functioning of the product.</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Hardware requirements</td><td>×</td><td>×</td><td>×</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td>✓</td><td>×</td><td></td></tr><tr><td>Suitability of Code</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Well-commented code</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>✓</td><td></td></tr><tr><td>Programming language</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>✓</td><td></td></tr><tr><td>Availability of support (11.76%)</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>11.76</td><td>0</td></tr><tr><td>Free services</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Non-Free services</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>×</td><td>×</td><td>✓</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Availability of test results (9.41%)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Tests done by the OSS community</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>Tests done by an external party</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td></td></tr><tr><td>Licensing terms (7.06%)</td><td>7.06</td><td>7.06</td><td>7.06</td><td>7.06</td><td>7.06</td><td>7.06</td><td>7.06</td><td>0</td><td>0</td><td>2.77</td></tr><tr><td>Availability of detailed information about the licensing terms and explicitly state if they are listed by the OSS initiative</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>Not clear</td><td></td></tr><tr><td>Availability of documentation (3.53%)</td><td>2.6</td><td>3.53</td><td>2.60</td><td>3.53</td><td>3.53</td><td>3.53</td><td>2.60</td><td>0.25</td><td>3.53</td><td>1.06</td></tr><tr><td>Documentation for final users</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>✓</td><td></td></tr><tr><td>Documentation for integrators</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>×</td><td>✓</td><td></td></tr><tr><td>Available languages of the documentation</td><td>Only English</td><td>Several</td><td>Only English</td><td>Several</td><td>Several</td><td>Several</td><td>Only English</td><td>Only English</td><td>Several</td><td></td></tr><tr><td>Mean time spent by the two researchers for skimming the portal (hrs):</td><td>4:00</td><td>4:50</td><td>3:05</td><td>3:02</td><td>3:03</td><td>3:50</td><td>2:25</td><td>3:00</td><td>2:43</td><td></td></tr><tr><td>Resulting percentage of coverage of the portal to the needs of selectors:</td><td>64.07</td><td>75.59</td><td>66.13</td><td>80.89</td><td>80.89</td><td>74.42</td><td>62.90</td><td>44.96</td><td>49.41</td><td>12.13</td></tr></tbody></table>
# 5 Discussion of Results

The previous section aimed to present a comprehensive view of the results. This section aims at emphasizing and discussing the most important findings and observations.

One of the main difficulties we faced in our OSS projects assessment was that the information—even if it was sometimes available—was not directly accessible. We had to browse the project website and explore among help files, manuals, or even demos. This fact increased the time spent on skimming the portal to find the information and definitely rules out any possibility of trying to automate the search for the information as previously stated by [3].

Even if we found that the involvement of commercial firms and the strata seem to have a significant influence on information rendering aspects, the coverage of selectors' needs varies from project to project.

OSS communities are not aware of the importance of making some information available. We observed that some informational needs are actually known by OSS communities but are not explicitly provided by them. For instance, most of the analyzed projects did not explicitly offer information about the *Number or registered users* in the community, *Time of the product in the market*, or *List of companies using the product* (if any). So, we hope that the results provided here help to raise the awareness of the importance of providing such information. In addition, there are categories that seem to be almost always provided (i.e., *Compliance with client's functional requirements and Availability of support*) while there are others that are not fully covered (e.g., *To ensure technological stability and evolution of the OSS product and its provider*) or are not covered by any of the studied project as *Availability of test results*.

Providing such evidence is important to envisage the corresponding improvement strategies and increase the competitive advantage of the OSS products.

# 6 Threats to Validity

This section discusses the threats to validity of our study in terms of construct, internal, and external validity, as suggested by [24] and [27]. It furthermore emphasizes the corresponding strategies used to deal with these threats.

## 6.1 Construct Validity

Regarding construct validity, our study was supported by 2 main principles: rigorous planning of the study, and the establishment of protocols and instruments for data collection and data analysis. The data collection instrument was carefully designed taking into account the informational needs of selectors elicited from a semi-structured interview further reported in [2], as detailed in section 3.2. This allows us to focus the study on the information that is really needed by the industrial OSS selection practice. In addition, the data collection instrument was pre-tested and
enhanced by creating categories and subcategories for grouping informational needs (as detailed in section 3.2). This allows us to improve its understandability and therefore to improve the data gathering process.

## 6.2 Internal Validity

Regarding internal validity, we tried hard to envisage and harmonize the data gathering and the subsequent data analysis strategies. With respect to the data gathering strategy, we took relevant decisions for approaching a better understanding of the availability of the information for covering the selectors' needs. One of the most relevant decisions was to avoid the non-deterministic factors inherent to the OSS selection processes. These non-deterministic factors refer to contextual issues that greatly affect the OSS selection decision. For instance, even if an OSS project provides a list of functional characteristics of the product, it might happen that such a list is not detailed enough for the context of the selection project and the selector have to face such a lack of detail by testing the product himself or by looking for further information in forums, email lists, etc. The strategies for facing (or not) such lack of information depend on the amount of time and resources that a company is willing and able to invest in the selection process [9]. Therefore, to avoid such potential issues we decided to focus our observations just on whether the informational need was covered or not and on how the information was provided.

In addition, we decided that two different researchers independently faced the assessment of each OSS community projects using the data collection instrument. Subsequently they discussed their results in order to agree and merge them. This helps us to deal with the potential subjectivity of the assessment of each researcher. Furthermore, it is important to mention that the researchers participating in the study are impartial parties and do not have any kind of involvement with any of the OSS communities analyzed. In this sense, we consider that there is no any intentional bias regarding the data gathered.

## 6.3 External Validity

Regarding external validity, it is important to highlight that the character of our study is exploratory, and hence we did not aim to make universal generalizations beyond the studied setting, but also provide some observations that might serve as a departing point for further investigations and improvements. Having this in mind, we discuss some mitigation strategies used in the study.

One of the main threats of external validity of the study is that we approached a small set of OSS community projects and these projects might not represent the whole variety of OSS projects. We tried to mitigate any possible bias related to this by having a stratified random sampling so that the studied OSS communities were diverse regarding size, application domain, popularity and success.

The informational needs used as a base to decide the informational coverage of OSS projects were elicited from industrial OSS selectors. While extracting such needs from the industrial practice is a good point to strengthen the external validity of our
observations, we are aware that eliciting such needs from 23 selectors might not represent all real needs. However, we think that such results are useful to have a first approximation to the problem and might serve as a basis to envisage future studies.

Finally, other issues that might affect the presented results (especially the time spent skimming the OSS projects for finding and understanding the information) are:

a) As mentioned above, the assessment of each OSS project was performed using a strategy that avoids the non-deterministic nature of the OSS selection processes (i.e., just capturing whether the informational need was covered or not and further observations about how it was provided). At this respect, we are aware that in the industrial OSS selection practice the complexity and time for gathering the OSS products information is actually higher. Thus, we would like to stand out that the metrics (weights and time for assessing each OSS project) provided in Table 3 are just intending to offer insights of the coverage of each OSS community to the selectors' needs. In any case these metrics are aimed to be representative of the assessment of OSS projects for any specific OSS selection process.

b) The researchers in charge of analyzing the OSS communities were not experts in any of the approached domains. So, we may say that the performance of researchers that performed the data gathered process would be more similar to “novice selectors” than experienced selectors that might perform better in finding and analyzing the OSS communities.

# 7 Conclusions and Future Work

This study presents our results of exploring the current gap between the “required” information needed by 23 industrial OSS selectors for making informed decisions and the information “provided” by 9 OSS communities. The obtained results would contribute to research and practice: a) by informing OSS communities about information rendering aspects that could be improved to attract industrial users. b) by informing OSS selection researchers about informational limitations that might help them to calibrate their OSS selection proposals.

Our future work focus on complementing the results from the study reported here with further information that allows OSS communities to elaborate their tactics of improvement based on the selectors' feedback [29].

# References

1. Ayala, C.P., Cruzes, D., Hauge, Ø.; Conradi, R. Five Facts on the Adoption of Open Source Software. IEEE Software. March-April 2011, pp. 95-99.

2. Ayala, C., Hauge, Ø., Conradi, R., Franch, X., Li, J. Selection of Third Party Software in Off-The-Shelf-Based Software Development - An Interview Study with Industrial Practitioners. The Journal of Systems & Software 84 (2011), pp. 620-637.

3. Bertoa, M., Troya, J.M., Vallecillo, A., 2006. A Survey on the Quality Information Provided by Software Component Vendors. Journal of Systems and Software 79, 427-439.

4. Boeg, J., 2006. Certifying Software Component Attributes. IEEE Software, 23(3), 74-81.
5. Bonaccorsi, A., Lorenzi, D., Merito, M., Rossi, C. Business firms' engagement in community projects. Empirical evidence and further developments of the research, in Proceedings of the First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS 2007), May 21th, Minneapolis, USA, IEEE Computer Society, 2007 pp. 1-5, doi:10.1109/floss.2007.3

6. Capiluppi, A., Lago, P., Morisio, M. "Evidences in the Evolution of OS Projects through Changelog Analyses," Proc. 3rd IEEE Workshop Open Source Software Eng. (WOSSE 03), ICSE, 2003, pp. 10-24.

7. Choi, N., Chengalur-Smith, I., Whitmore, A. Managing First Impressions of New Open Source Software Projects. IEEE Software. November-December 2010. pp., 73-77.

8. Ciokolwski, M., Soto, M., 2008. Towards a Comprehensive Approach for Assessing Open Source Projects. Proceedings of the International Conference on Software Process and Product Measurement. Springer, pp. 316-330.

9. Cruz, D., Wieland, T., Ziegler, A., 2006. Evaluation Criteria for Free/Open Source Software Products Based on Project Analysis. Software Process: Improvement and Practice, 11(2), 107-122.

10. Del Bianco, V., Lavazza, L., Morasca, S., Taibi, D., 2009. Quality of Open Source Software: The QualiPSo Trustworthiness Model, Proceedings of the 5th IFIP Working Group 2.13 International Conference on Open Source Systems (OSS2009) - Open Source Ecosystems: Diverse Communities. Springer. Skövde, Sweden, pp. 199-212.

11. Denner, C.S., Pearson, J., Kon, F., Attractiveness of Free and Open Source Projects. The European Conference on Information Systems (ECIS) 2010. ISBN: 978-0-620-47172-5

12. Fitzgerald, B. The Transformation of Open Source Software. MIS Quarterly, 30(3), 2006.

13. Fogel, K. Producing Open Source Software: How to Run a Successful Free Software Project, O'Reilly, 2006.

14. Golden, B., 2004. Succeeding with Open Source, Addison-Wesley Professional.

15. Hauge, Ø., Ayala, C.P., Conradi, R.: Adoption of open source software in software-intensive organizations - A systematic literature review. Information & Software Technology 52(11): 1133-1154 (2010).

16. Jadhav, A.S., Sonar, R.M., 2009. Evaluating and Selecting Software Packages: A review. Information and Software Technology, 51(3), 555-563.

17. Li, J., Conradi, R., Bunse, C., Torchiano, M., Slyngstad, O.P.N., Morisio, M., 2009. Development with Off-The-Shelf Components: 10 Facts. IEEE Software. 26(2), 80-87.

18. Li, J., Conradi, R., Slyngstad, O.P.N., Torchiano, M., Morisio, M., Bunse, C., 2008. A State-of-the-Practice Survey of Risk Management in Development with Off-the-Shelf Software Components. IEEE Transactions on Software Engineering, 34(2), 271-286.

19. Krippendorff, A., 1980. Content Analysis. Sage Publications, London.

20. Mahmood, S., Lai, R., Kim, Y.S., 2007. Survey of Component-Based Software Development. IET Software, 1 (2), 57-66.

21. Merilinna, J., Matinlassi, M., 2006. State of the Art and Practice of Open-Source Component Integration. Proceedings of the 32nd EUROMICRO Conference on Software Engineering and Advanced Applications. IEEE Computer Society, pp. 170-177.

22. Openbrr, 2005. Business Readiness Rating for Open Source A Proposed Open Standard to Facilitate Assessment and Adoption of Open Source Software. http://www.openbrr.org/wiki/images/d/da/BRR_whitepaper_2005RFC1.pdf, Request For Comments, 2005.

23. Open Source Initiative http://www.opensource.org/
24. Robson, C., 2002. Real World Research: A Resource for Social Scientists and Practitioner-researchers. Second Edition. Blackwell Publishers Inc.

25. Semeteys, R., Pilot, O., Baudrillard, L., Le Bouder. G., Pinkhardt, W., 2006. Method for Qualification and Selection of Open Source software (QSOS) version 1.6, Technical report, Atos Origin.

26. Simmons, G.L., Dillon, T.S., 2006. Towards an Ontology for Open Source Software Development”. In IFIP International Federation for Information Processing, Volume 203, Open Source Systems. Damiani, E., Fitzgerald, B., Scacchi, W., Scotto, M., Succi, G., (Eds). Boston:Springer, pp 65-75.

27. Wohlin, C., Runeson, P., Host, M., Ohlsson, M.C., Regnell, B., Wesslen, A., 2000. Experimentation in Software Engineering - An Introduction. Kluwer Academic Publishers.

28. Winter, S.J., Saunders, C., Hart, P. "Electronic Window Dressing: Impression Management with Websites," European J. Information Systems, vol. 12, 2003, pp. 309–322.

29. Wolfe, E., Bies, R. Impression Management in the Feedback-Seeking Process: A Literature Review and Research Agenda. *Academy of Management Review*. 1991. Vol 15, No. 3, pp. 522-541.