## 3 Methodology
### Wisdom of Crowds  
The technique called Wisdom of Crowds which is used in our case study for technical debt identification, was already proven to be reliable and accurate in data collection and predictions in other fields. It was proposed by [PERSON] (2018), and was also adopted by one of the authors in one of the previous experiments [6]. Here, we will recall the crucial facts about that. The main advantage of this approach lies in the understanding of sociological aspects of human interaction. Each engineer has a different and unique technical background and understanding of the product: its many hidden characteristics, interactions of its modules, etc. This is the reason why everyone may provide meaningful insight into technical debt presence in the codebase and its consequences, and why better data may be collected from a more diverse and bigger team. In our case, the applied Wisdom of Crowds technique strives for answering how the engineers' continually growing knowledge of the system and understanding of its internal mechanisms and dependencies could be more effectively used in order to improve its quality. [PERSON] analysed social situations in which a group of people, i.e., crowd, was unable to produce good judgement, because one of several critical factors for good decision making was missing. He called the group of the people, where all necessary aspects of sound judgement are in place, a _Wise Crowd_, and claimed that there are four elements required to form such an entity:  
1. Diversity of opinion: each person has private information, which may be just an interpretation of the known facts. 2. Independence: opinions of the people are not determined by the opinions of those surrounding them. 3. Decentralisation: people are able to specialise and make use of local knowledge. 4. Aggregation: some mechanism exists to convert private judgments into a group decision. Therefore, if the decision making environment is not ready to accept the crowd, failing to represent the majority of opinions, views, or perspectives, the benefits of individual judgments and private information vanish. [PERSON] also discussed common deficiencies, causing crowd to turn into an irrational one:  
1. Homogeneity: the lack of diversity within a crowd, reducing variability in viewpoints to be taken into account and resulting in limited amounts of private information to be heard. 2. Centralization: limiting the data collection process to ask only those who are supposed to be a wise representation of the team. Usually if it comes to strategic decisions, the knowledge of lower-level engineers, sometimes very unique when it comes to complex systems or situations, is omitted. 3. Division: the failure to aggregate all pieces of information; for example when all necessary information is available but information held by one organisational department is not accessible to another. 4. Imitation: a sequence of activities may result in an information cascade, where only the first people are adding their knowledge to the common pool used by others. When the first decision seems rational, those who came next are willing to mimic that behaviour. 5. Emotionality: describes even a broader group of effects: e.g., a feeling of belonging leading to peer pressure, and herd behaviour. Hence, in order to create a wise crowd during the technical debt identification phase, the following aspects need to be preserved:  
1. Independent data collection. Each engineer provides his/her view on technical debt, identifying technical debt items and their potential consequences independently. No discussions, or alignment happens at this stage to avoid cascading of assessment. It can be done via online collaboration tools like Jamboard [27] or MURAL [28]. 2. In the next step, the list of technical debt items is reviewed to remove duplicates, and identified technical debt items are stored as records in Atlassian Jira [29]. 3. No input is considered better than the other, no prioritisation and weighing factors are applied.