Software development methodologies and practices in start‐ups

IET SoftwareVolume 13, Issue 6 p. 497-509 Review ArticleOpen Access Software development methodologies and practices in start-ups Esubalew Workineh Tegegne, Esubalew Workineh Tegegne M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, FinlandSearch for more papers by this authorPertti Seppänen, Pertti Seppänen M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, FinlandSearch for more papers by this authorMuhammad Ovais Ahmad, Corresponding Author Muhammad Ovais Ahmad Ovais.Ahmad@oulu.fi orcid.org/0000-0002-7885-0369 M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, Finland Department of Mathematics and Computer Science, Karlstad University, Karlstad, Sweden Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this author Esubalew Workineh Tegegne, Esubalew Workineh Tegegne M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, FinlandSearch for more papers by this authorPertti Seppänen, Pertti Seppänen M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, FinlandSearch for more papers by this authorMuhammad Ovais Ahmad, Corresponding Author Muhammad Ovais Ahmad Ovais.Ahmad@oulu.fi orcid.org/0000-0002-7885-0369 M3S Research Unit, University of Oulu, P.O. Box 90570, Oulu, Finland Department of Mathematics and Computer Science, Karlstad University, Karlstad, Sweden Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this author First published: 01 December 2019 https://doi.org/10.1049/iet-sen.2018.5270Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Software start-ups are aiming to develop cutting-edge software products under highly uncertain conditions, overcoming fast-growing markets under multiple influences. This study aims to identify and analyse the existing scientific literature regarding software development methodologies and practices in software start-ups published between January 2006 and December 2017 using the systematic mapping study. The results identified 37 relevant primary studies out of 1982 papers. To validate the results from the mapping study, an empirical study was based on the research data collected from 14 real-life software start-ups located in Finland, Italy and Norway. The result shows that Agile and Lean start-up methodologies are the most commonly used in software start-ups due to the flexible nature and easy tailoring. A total of 144 software development work practices are extracted from the primary studies. This study contributes to the research in several ways: (i) provides state of the art regarding software development methods and practices in software start-up contexts, (ii) reports commonly used methods along with its benefits identified in primary studies and (iii) identifies opportunities for future software start-up research. 1 Introduction In recent years, there is a significant increase in the number of software start-ups around the globe. Start-ups are new phenomena in the technology market, creating new businesses and jobs [1]. Technological advancements have brought an increased demand for new software and services. According to Klotins et al. [2], such advancement makes to establish new software companies and exploit the technology. Software start-ups are playing an important role in addressing the increasing demand for new software products and services [2]. Many success stories surrounding the software start-ups (i.e. Facebook, Instagram, Uber, Airbnb etc.) contribute to the popularity of start-up phenomenon. These success stories are used as a driving fuel in the software start-up environment. However, despite these success stories, not all start-up companies succeed [3-5]. According to Giardino et al. [6], 60% of the total start-ups in the world do not survive the first 5 years from their creation. This has led researchers to study various factors contributing to succeeding or failing of software start-ups [4, 5, 7]. For example, a high failure rate among start-ups is due to lack of resources, immaturity, multiple influences and dynamic technologies. On the other hand, there are assumptions that inadequate software development practices might be a significant contributing factor for the high failure rates in start-ups [2]. Start-up companies develop the software in a situation where resources are very limited and practices are not specified by a prescriptive methodology [7]. Despite the popularity of the start-up phenomenon, the existing literature on software development activities in start-ups is scarce. Few studies have identified, characterised and mapped software development methods and practices in software start-ups [2, 7-9]. According to Klotins et al. [2], the existing knowledge about software engineering practices is limited among the start-ups and the existing body of knowledge is insufficient to support engineering aspects in start-ups. The existing published studies [2, 7] on start-ups have limitations, as shown in Table 1. The mapping study conducted by Paternoster et al. [7] has identified, categorised and analysed 213 software development work practices used in software start-up companies. Klotins et al. [2] identified 54 software engineering practices in start-up companies, which are mapped to the Software Engineering Body of Knowledge (SWEBOK) knowledge areas [10]. However, both studies [2, 7] have focused on development practices but did not make a deeper analysis of development methodologies. Table 1. Comparison of last 6 years (2014–2019) literature reviews on software start-ups Comparison context Paternoster et al. [7] Klotins et al. [2] Berg et al. [11] Klotins et al. [12] This study (2019) purpose extract and analyse software development practices used in start-ups identify and categorise software engineering practices utilised in start-ups and mapped to SWEBOK identifying change in focus of research area and thematic concepts operating start-up research identify key knowledge areas and opportunities for further research identify and analyse software development methods and practices used in start-ups years included 1994–2013 1994–2014 1994–2017 2001–2015 2006–2017 no. of primary studies 43 14 74 88 37 sources scientific literature scientific literature scientific literature experience reports scientific literature More recently, two literature reviews, Berg et al. [11] and Klotins et al. [12], are published in the same vein. The Berg et al. [11] mapping study observes how the software start-up research has evolved compared to the software engineering knowledge areas presented in SWEBOK. Berg et al. [11] primary studies were 74, obtained from three databases naming Scopus, ISI Web of Science and Engineering Village Compendex. The study focus was to map the software start-up literature in the light of SWEBOK, the context of start-ups and inferring applicability of empirical findings as well as to identify the research gaps. Whereas Klotins et al. [12] investigated on how the software engineering is applied in a start-up context with a focus to identify key knowledge areas and opportunities for further research. The data source for Klotins et al. [12] was an experience report from CB Insights website, which is 'global network of executives and startups…' [13, 14]. A total of 88 experienced company reports were analysed to identify various software engineering knowledge areas. Both reviews [11, 12] provide a high level of categorisation of the start-up process and practices dominantly in the light of SWEBOK knowledge areas. Further, 5 years ago, two studies [2, 7] were conducted; however, the start-up environment is fast changing and is dynamic, which needs up-to-date research results. This study aims at filling the gap in the research with a clear focus on development methods, processes and practices in start-ups. Different from the previous mapping and systematic literature reviews [2, 7, 11, 12], this study will be performed as a complementing study conducted by Paternoster et al. [7]. This study investigates the software development methods and practices used in start-ups. The study provides state of art regarding software start-ups, synthesis of adopted development methods, its use benefits and provides direction for future research. The paper is structured as follows: Section 2 describes the background and related work with this study; Section 3 discusses the systematic mapping study (SMS) process (e.g. planning, conducting, reporting) followed to perform the review and the empirical study. The results are presented in Section 4. Section 5 discusses the answers to the research questions; and Section 6 provides a conclusion to the study and future directions. 2 Background Start-ups are described as a human institution created to build a new product under conditions of extreme uncertainty [15]. Blank [3] defines a start-up as a temporary organisation that creates high-tech innovative products and has no accumulated experience in the development activities. A start-up is a company that is challenged by immaturity, severe resource limitations, multiple influences dynamic technologies and markets [8]. The term 'Software Startup' or 'software package startups' was first introduced by Carmel [16]. Software start-ups are young package firms that create new niche products under a short period of time [16]. Since 1994 various other definitions for the term 'Software Startups' are emerged in the field [8, 9, 17]. Coleman and O'Connor [9] describe software start-ups as unique companies that develop software through various processes and without a prescribed methodology. On the other hand, Sutton [8] characterised the software start-ups by the challenge they face as: little or no operating history – software start-ups have little-accumulated experience in development processes and organisation management; young or immature compared to more established and mature companies; limited resources: scarcity in finance, time, human resources and market; typically focus on delivering and promoting the product, and building up customers; multiple influences: under pressure from investors, customers, partners and competitors in decision-making process; dynamic technologies and markets: newness of software companies often require to develop or operate with disruptive technologies to enter into a high-potential target market. Therefore, many researchers shared the above characterisation of software start-ups [7]. From a business model perspective, there are two kinds of software start-ups: product-oriented and project-oriented start-ups. The product-oriented software start-ups develop market and sell their own product, whereas the project-oriented software start-ups do not develop their own product, instead they serve their customers by building products for them [18]. In earlier times, the development of software as a service or a product started as a chaotic activity often referred to as 'code and fix'. Software was written with a little planning, and the design of the product was set by many short-term decisions [19]. As an alternative to this approach, a methodology was introduced. A development methodology imposes a disciplined process upon software development with the aim of making software development more predictable and more efficient [19]. A software development methodology is a set of activities, practices or processes that an organisation uses to develop and maintain the software and the associated products (e.g. project plans, design documents, code, test cases and user manuals) [20]. Chapman [21] defined the software development methodology as a documented collection of policies, process disciplines and procedures exercised by a development team or an organisation to practice software engineering activities and software development practices that a methodology defines to be used in the process [22]. It is important to note that software development methodology does not simply imply the development of software code, it covers the entire process involved in developing and maintaining a software product [21]. However, the process is not a rigid instruction on how to develop software. Rather, it is a flexible approach that enables the software team or companies to do the work by choosing the appropriate set of work actions and tasks [20]. As a result, software companies are able to deliver a software product in a timely manner and with sufficient product quality to satisfy their customers. In the past, there had been a number of different methodologies designed to help companies manage their software development activities. However, no single approach has achieved a generalised acceptance, as there is a number of other contextual and situational factors that influence the choice of methodologies and processes [23]. The start-ups develop innovative software and services without strictly following any specific development methodology or practices [7]. However, the existing studies highlighted that various development methods and practices are partially used in software start-ups, such as Agile methods, Lean start-up [2, 7, 11, 15, 22]. According to Abrahamsson [22] the Agile methods emphasise on the relationship and commonality of software developers and the human role reflected in the contracts, as opposed to institutionalised processes and development tools. Agile methods also manifest itself in close team and working environment relationships, as well as other procedures boosting the team spirit [22]. Scrum is a common example of the Agile methodology. The Agile method is used as a project management approach in the context of this publication. Lean start-up method is a product development process, which combines the business-driven hypothesis experimentation and iterative product releases [15]. According to Ries [15], the Lean start-up method enables to build a product iteratively based on the needs of early customers that could lead to reduced market risks such as expensive product launches and failures. In general, development methods in start-ups have been neglected in the studies or are less familiar [6-8]. In 2015, Klotins et al. [2] conducted a study to identify and categorise software engineering knowledge areas utilised in start-ups to map out the state of art. Klotins et al. [2] identified 54 software development practices from 14 primary studies and categorised them into 11 main knowledge areas from SWEBOK. The study highlighted that existing research does not provide reliable support for software engineering in any phase of a start-up life cycle. Klotins et al. [12] extend their previous study [2] by conducting a multi-vocal exploratory study of 88 start-up experience reports. The study shows that most engineering challenges in start-ups stem from inadequacies in requirements engineering and stress that more research on adaptation of established practices, and validation of new start-up specific practices is needed [12]. Recently, Berg et al. [11] conducted an extensive mapping study on software start-up engineering covering the literature from three databases (i.e. Scopus, ISI Web of Science and Engineering Village Compendex) from 1994 to 2017. The study provides a comprehensive view of software start-ups for software engineering researchers and proposes that future research should focus on start-up evolution models and human aspects. Paternoster et al. [7] provide the state of the art and evidence on software development practices used in start-ups. The study claims that software development work practices are chosen opportunistically, practiced in a way to provide value under constrains imposed by the startup context [7]. However, the study reported that there is lack of highlight quality literature and proposed investigation on identifying the software development strategies engineered by practitioners. The existing studies cover a wide range of areas related to development practices in start-ups [2, 6-8, 11, 12]. Additionally, there is limited fresh knowledge regarding development methodologies in the start-up world. However, little coverage is given in investigating and identifying software development methodologies used by start-ups. This research gap is clearly highlighted by Paternoster et al. [7]. This knowledge gap motivated this study. We conducted systematic mapping study along with empirical research with a focus on software development methodologies and practices in the start-up context. 3 Research methodology 3.1 Systematic mapping study We followed the established SMS guidelines and procedures of Kitchenham and Charters [24]. The SMS process is illustrated in Fig. 1, which consists of three main phases: planning (four steps), conducting (five steps) and documenting (three steps). In the following section, we will discuss these phases in detail. Fig. 1Open in figure viewerPowerPoint Systematic mapping process 3.1.1 Planning the review The planning phase comprises four steps. First, motivation for conducting this SMS is to provide the start of the art related to software development methods and practices used in the context of start-ups. Further, to collect such methods that benefit in the software start-up context. The following research questions drive this SMS: RQ1. What is currently known about the software development methodologies and practices in software start-ups? RQ2. What are the claimed benefits of software development methods in the software start-up literature? RQ3. What are the software development methods and practices used in software start-ups? RQ1 is designed with the intent to identify empirical evidence regarding software development methodologies in start-ups in existing scientific literature. This will help researchers and practitioners to shed light on the use of development methodologies in software context and open an avenue for future research. Then, explore on what research methods are used in software start-up papers and their quality. Further, RQ1 focused to identify software development practices in software start-ups. Those studies were eligible for inclusion, which present empirical data on software development practices in software start-ups. The inclusion criteria used were: The study should be written in English and published between January 2006 and December 2017. The study directly answers one or more of the research questions of this study. The study should have a clear focus on software start-ups. The study should be available in full text. The study can be in the form of systematic mapping study, experience report, applied engineering practices, development models or lessons learned. Studies were excluded from this SMS if their focus was specifically not on development methods and practices in software start-ups or if they did not provide academic rigour or industry relevance: duplicate articles; studies not clearly focused on software start-ups nor development methods and practices in software start-ups (non-software companies: biotech, manufacturing etc.); studies related to technicalities of start-ups (funding, algorithms, programming languages etc.); not peer-reviewed scientific papers (i.e. books, blog posts, presentations etc.). 3.1.2 Conduct the review Pilot search is an important part of conducting SMS [24], which provides an overall idea about available literature. We use Google Scholar for pilot search because it yields diverse and unbiased literature. The initial keywords, startups, software, development, are used in various combinations. The pilot helped to identify that the existing studies use various synonyms for software start-ups i.e. software start-up, high-tech venture, high-tech start-up, IT startup, Lean startup etc. The SMS search strings are structured according to guidelines proposed by Kitchenham and Charters [24] (see Table 2). Table 2. Search strings Criteria Core concept Related words population software start-up start-up, software start-up, software start-up, high-tech venture, high-tech start-up, IT start-up, lean start-up intervention software development software development, engineering, implementation, build comparison practice practice, methodology, method, process, strategy, approach, model Search strings are applied to all the selected data sources. The example of search strings used is as follows: ('early-stage firm' OR 'early-stage company' OR 'high-tech venture' OR 'high-tech ventures' OR 'start-up company' OR 'start-up companies' OR 'high-tech start-up' OR 'high-tech start-ups' OR 'high-tech startups' OR 'high-tech startup' OR 'startup company' OR 'startup companies' OR 'software startup' OR 'lean startup' OR 'lean start-up' OR 'lean startups' OR 'software startups' OR 'IT start-ups' OR 'IT start-up' OR 'IT startup' OR 'IT startups' OR 'software start-up' OR 'software start-ups' OR 'software product startup' OR 'software product startups' OR 'software start up' OR 'web startup' OR 'web start-up' OR 'web startups' OR 'web start-ups' OR 'internet startup' OR 'internet start-up' OR 'internet startups' OR 'internet start-ups' OR 'mobile startup' OR 'mobile start-up' OR 'mobile startups' OR 'mobile start-ups') AND (develop* OR engineer* OR implement* OR build*) AND (software OR process OR model OR methodolog* OR method OR practice OR strategy). 3.1.3 Identification of primary studies Five databases were selected to collect relevant papers: IEEE, Scopus, Web of Science, ACM digital library and ProQuest. These major electronic databases cover software engineering domain and the ability to handle advanced queries [7]. Each database required specific search string to retrieve an initial list of studies. The advanced functionality of each database helps to focus on specific fields such as software engineering and computer science. The records are imported into Google Spreadsheet. The basic input includes meta-data such as (i) title, (ii) author, (iii) year, (iv) publication type and (v) abstract. To screen the retrieved papers, we follow the best practices proposed by Kitchenham and Charters [24] which is illustrated in Fig. 2. Fig. 2Open in figure viewerPowerPoint Identification of primary studies The search strings in each database were used to search each paper title, keywords and abstracts. At the database level, the papers were screened with the advanced features of each database (i.e. screening by year, language and scope of software engineering domain). This step results in a total of 1982 studies. All the studies were imported to Refworks (http://refworks.com) with the aim to remove duplicate elements. In total, 234 studies were removed as duplicates from the list. Then the results were transferred to Google spreadsheet to study inclusion and exclusion criteria. This step indicated that many publications focused on areas that were outside the scope of this study. Our focus is on software start-ups and their software development methods and practices. There were many start-up studies which focused on a variety of different fields such as agricultural, hardware, electronics etc. This screening yielded 71 studies for further analysis. The remaining papers were checked in detail for whether they qualify the inclusion criteria. The papers that met the inclusion criteria were transferred for quality assessment. By reading the full-text of the paper, 41 primary studies were selected. The two researchers have disagreements on the quality of four papers. These papers were jointly discussed and decided to exclude owing to the quality problem. Finally, 37 primary studies were obtained. 3.1.4 Quality assessment The 37 primary studies were assessed on the ten-factor quality criteria proposed by Dybå and Dingsøyr [25]. A binary grade was used for all the primary studies quality assessment, '1' represents 'yes' and '0' represents 'no'. Two researchers independently checked the quality assessment of primary studies: Was this a research paper? (or was it merely 'lessons learned' report based on expert opinion?) Was there a clear statement of the aims of the research? Was there an adequate description of the context in which the research was carried out? Was the research design appropriate to address the aims of the research? Was the recruitment strategy appropriate to the aims of the research? Was there a control group with which to compare treatments? Was the data collected in a way that addressed the research issue? Was the data analysis sufficiently rigorous? Had the relationship between researcher and participants been considered to be an adequate degree? Was there a clear statement of findings? Was the study of value for research or practice? 3.1.5 Data extraction and analysis The data extraction was started by reading full-text analysis of primary papers. One author extracted the data and other authors facilitated and improved the data extraction process. In the data extraction process, the relevant information related to software development methodologies and practices in start-ups was identified from each primary study, using the original author's terms. Then recording the results in a tabular form to enable comparisons and analysis based on different groupings was made. In summary, the data synthesis was achieved by: (i) identification of software development methodologies used by start-ups from each study and (ii) documentation of a set of reported development practices in start-ups from each primary study. 3.1.6 Data extraction and analysis The data extraction was started by reading a full-text analysis of primary papers. One author extracted the data and other authors facilitated and improved the data extraction process. In the data extraction process, the relevant information related to software development methodologies and practices in start-ups was identified from each primary study, using the original author's terms. Then recording the results in a tabular form to enable comparisons and analysis based on different groupings was made. In summary, the data synthesis was achieved by: identification of software development methodologies used by start-ups from each study; documentation of a set of reported development practices in start-ups from each primary study. 3.2 Empirical study An empirical study was conducted as a comparison to the findings of the literature study. The empirical study followed the guidelines presented by Runeson and Höst [26] for conducting the case study research in software engineering. Interviews were opted for the research data gathering due to practical difficulties tied to other data gathering methods proposed by Runeson and Höst [26]. The empirical study was based on the research data collected from 14 real-life software start-ups located in Finland, Italy and Norway. The data was gathered using interviews with a broad interview schema to cover different viewpoints on software development in start-ups (see Table 3). Table 3. Interview schema ID Information Topic Note initial phase IQ01 Do you remember the time (month-year) when you had the initial idea? pivot — IQ02 What was your initial idea? pivot — IQ03 Who were your targeted customers? start-up model, pivot — IQ04 How did you find out about these customers? start-up model, pivot — IQ05 What was the initial team composition? competency — IQ06 How did you come up with the list of features for the first prototype? start-up model, pivot, testing — IQ07 When did you have the first prototype? competency, pivot, testing — IQ08 How did you make the first prototype? start-up model, competency, pivot, testing — IQ09 How did you test it? testing, competency — IQ10 How much effort was spent on testing? testing, competency — IQ11 What are the important quality attributes? testing — IQ12 What were the special competencies needed to make this prototype? competency — significant pivots IQ13 How many changes were made up till now? pivot — IQ14 Is the initial idea and the current product same? (use product, marketing, financial, team dimensions) pivot — IQ15 When did you make the last changes in business, market, and product features? pivot — IQ16 What do you think is the significant pivot? Explain. pivot — evolution IQ19 Time when you had the first launching pivot, start-up model — IQ20 What is the current size of your customer base? pivot — IQ21 What is the current team composition? competency — IQ22 How had the company grown since it was established? (organic or buying subcontracting) competency, start-up model — IQ23 What employees are currently working in the company? competency — IQ24 Why are these people preferred? competency — IQ25 How do you perform testing now? testing — IQ26 How much effort is spent on testing? testing — IQ27 What is an important quality attribute now? testing — future plan IQ28 Assuming growth, which kind of people you wish to reach? competency — IQ29 How will the situation change in the next 3–5 years? general — 3.2.1 Study design overview The interviews were semi-structure