Requirements Quality Defect Detection with the Qualicen Requirements Scout

Our group worked on both redefining quality in RE, as well as simple methods for quality defect detection, their potential and limitations. This report summarizes the main challenges from our industrial perspective, which are: Precision, relevance, process, and summarization. 1 A Brief Team History Our team was founded by four PhDs and a Professor from the department of informatics at the Technical University Munich (TUM). At the chair for software & systems engineering of Manfred Broy, we conducted bilateral research motivated by the needs of our industrial partners, i.a. Munich Re, Daimler AG, TechDivision, and Wacker Chemie AG. At that time, we experimented with combining the existing source code quality analysis toolkit ConQAT with NLP techniques in order to detect quality issues in system tests [HJE13] and requirements [FMJ14]. As our experiments lead to promising results, the companies started asking for productive systems instead of experimental academic prototypes. At this point we founded the HEJF GBR, which was shortly afterwards succeeded by the Qualicen GmbH. Qualicen is now a quickly growing company with currently eleven employees, located in Garching near Munich, Germany. Qualicen does test and requirements engineering coaching, consulting and tooling. Our customers come from various domains, including automotive, aerospace, healthcare, and insurance. 2 Technical Outcome: The Qualicen Requirements Scout The main technical outcome of our group is the Qualicen Scout. Qualicen Scout searches for quality findings in requirements and system tests written in natural language. It is based on the continuous source code analysis tool Teamscale. To detect quality findings, the Scout is attached to a requirements data source, such as a PTC Integrity or DOORS NG requirements database, an SVN or a GIT repository. The Scout then continuously pulls for new versions of the requirements and thus creates a full history of all automatically detectable quality defects. Whenever an author updates the requirements, the system immediately analyzes whether this update introduced new findings or fixed existing findings and reports this back to the team. With this information, Qualicen Scout users serves three basic use cases: Rapid Feedback: The largest possible benefit you can get from automatic tools is, when the engineers creating artifacts are notified straight after they made a mistake. Not only is fixing the defect the cheapest, but the learning effect is also much stronger. Therefore, we support various RE tools, such as PTC Integrity, IBM DOORS NG or Microsoft Word with plugins for rapid feedback (see Fig. 1). Copyright c © 2018 by the paper’s authors. Copying permitted for private and academic purposes. 1http://www.qualicen. de 2https://cqse.eu/en/products/teamscale/landing/ Figure 1: Author plugins (e.g., in Microsoft Word) provide rapid feedback to requirements engineers. Tool-supported audits & reviews: One of the most common situations where we apply our tools is for reviews or tool-supported audits. These can be in one of two situations: Either this is the first time we look at the requirements, or we want to understand how things changed since the last review. For the former, we use similar perspectives as the ones described in the first use case. For the latter, we have a specific delta perspective that describes which files, sections, metrics, and findings have been changed between two point in time. In addition, the delta perspective provides the reviewer with an analysis whether the engineer has worked in a specific section but has ignored a certain defect. In tool-supported audits we found that it is of utmost importance to combine the findings found by the tool with the findings found by a manual review of an expert. Trend analysis: Lastly, there are multiple roles in the RE process that do not so much care about individual quality defects, but more about the trend. Especially in situations with heavy reuse (e.g., in the automotive industry) it is unrealistic to expect people to iterate through all findings for all existing requirements. Instead, quality engineers and project leads assume that over time the quality improves. For this, one can use metrics such as number of findings or findings density and visualize their trend over time. These roles are interested in trends and have a more dashboard-like viewpoint onto the system (see Fig. 2). Of course, the scout offers various necessary features, such as creating custom dashboards for a team, notifying users about changes, customizing the analyzed criteria or hiding (blacklisting) false positive findings. In the following, we explain our past research, which includes the types of defects that Qualicen Scout detects. 3 Academic Outcome: Past Research In the past, we worked specifically on RE and system test quality. For a summary on the works on system test quality, please refer to the work by Hauptmann [Hau16]. In the following, we summarize the RE specific outcomes. They are separated into three main questions: First, what is RE artifact quality as a concept? Second, to which extent can we automatically detect quality defects? And third, where are the (theoretical and practical) limitations of automatic defect detection? 3.1 What is RE Artifact Quality? Regarding the first question, we base our view onto the fact that RE artifacts are just a means and not an end [FMM15, FV18]. As such, the definition of a high quality artifact depends on its purpose. The purpose of RE artifacts can be of different types (see [Fem17, p.12ff] for details), but it usually breaks down into the simple (and simplified) question: Which quality factors (properties of the artifact) can make the artifact more efficient or effective to use? We call this paradigm Activity-based RE artifact quality models or ABRE-QM. The ABRE-QM paradigm allows us to operationalize quality through impact on effectiveness and efficiency in usage. To understand the impact of certain quality factors, we analyzed the impact of passive voice on understanding requirements [FKV14] and creating good test cases based on requirements [MFME15, BJFF17]. 3Probability that a random word is subject to a finding 4As a consequence, the quality meta-model model then is a slight modification of the Quamoco quality model [WLH+12]. Figure 2: The dashboard shows trends to quality engineers and team or department leads. 3.2 Which quality factors can we automatically detect and how? In [FMWE17] and [FUG17], we gained a first understanding to which extent we can and cannot automatically detect quality defects. Based on the notion of code smells [FB99], we refer to these automatically detectable types as requirements smells. Types of Smells. We differentiate the types of defects, depending on the scope of information that is accessed. The defect can relate to a word (lexical smells), the grammar of a text (grammatical smells), the structure of the text (structural), or the semantics (semantical smells). Of these, the last category is a bit imprecise, since most smells come with a semantic problem that they address and a syntactic (for an automated system analyzable) mechanism for detection. Accordingly, smells in the semantic category have to be broken down to lexical, grammatical or structural aspects in order to be automatically detectable. Methods. To detect these smells, we apply the common types of NLP mechanisms: Wordand Sentence Splitting, Morphologic analysis, Lemmatization (and sometimes stemming), POS tagging, and syntactic parsing. We also experimented with shallow semantic parsing and dependency analysis. Our tool chain relies on the framework DKPro [dCG14], which enables to switch NLP tools without intensive effort. Interestingly, as we found in [FUG17], the three most useful mechanisms are not the strong weapons of NLP, but simple mechanisms, such as dictionaries, regular expressions, and formatting information. However, as we found in our work: Our main effort is not spent in detecting defects, but in improving the precision. For that, we make heavy use of context-based filtering mechanisms, similar to the ones proposed by Krisch and Houdek [KH15]. 3.3 Which quality factors can we not automatically detect and why? We found the following reasons, why certain quality factors cannot be automatically detected: The quality factor refers to stakeholder or domain knowledge, to the semantic understanding of natural language, to the scope or goal of the system, to the development process, or that the quality factor itself is vague or subjectively defined. When we analyzed a large guideline by an industrial partner, we found that, surprisingly, the main challenge was not the technical limitations of the NLP. Instead >80% of undetectable rules were actually undetectable due to the vague or imprecise rules [FUG17]. 4 Challenges from an Industrial Perspective In the following, we want to summarize the main challenges in the area of automatic detection of quality defects in RE from our industry-focussed perspective. The Precision Challenge: From our perspective and in contrast to a popular opinion in academia, the main challenge for creating user acceptance is precision. We see two reasons for this: First, in our experience, if users receive a certain percentage of incorrect findings, they less and less tend to respect even the correct findings. So, if users receive more and more false positives they will stop using the system altogether. This is different with recall. In our experience, users are more willing to invest into manual effort for finding additional defects than for ignoring suggested defects. Second, these automatic detection mechanisms quickly turn into metrics and assessments for teams (as in the trend analysis use case described above). For an assessment, however, teams tend to more openly accept a tool, when the tool misses half of the defects, but each finding is correct (the metric is more optimistic