The Case for Dumb Requirements Engineering Tools

Context and Motivation This talk notes the advanced state of the natural language (NL) processing art and considers four broad categories of tools for processing NL requirements documents. These tools are used in a variety of scenarios. The strength of a tool for a NL processing task is measured by its recall and precision.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 2 Question/Problem In some scenarios, for some tasks, any tool with less than 100% recall is not helpful and the user may be better off doing the task entirely manually.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 3 Principal Ideas/Results The talk suggests that perhaps a dumb tool doing an identifiable part of such a task may be better than an intelligent tool trying but failing in unidentifiable ways to do the entire task. Contribution Perhaps a new direction is needed in research for RE tools.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 4 Natural Language in RE A large majority of requirements specifications (RSs) are written in natural language (NL).  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 5 Tools to Help with NL in RE There has been much interest in developing tools to help analysts overcome the shortcomings of NL for producing precise, concise, and unambiguous RSs. Many of these tools draw on research results in NL processing (NLP) and information retrieval (IR) (which we lump together under “NLP”).  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 6 NLP-Based Tools and RE NLP research has yielded excellent results, including search engines! This talk argues that characteristics of RE and some of its tasks impose requirements on NLP-based tools for them and force us to question whether ... for any particular RE task, is an NLP-based tool appropriate for the task?  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 7 Categories of NL RE Tools Most NL RE tools fall into one of 4 broad categories (a–d): a. tools g to find defects and deviations from good practice in NL RSs, e.g., ARM and QuARS, and g to detect ambiguous requirement statements, e.g., SREE and Chantree’s nocuous ambiguity finder.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 8 Categories Cont’d b. tools to generate models from NL descriptions, e.g., Scenario and Dowser. c. tools to discover trace links among NL requirements statements or between NL requirements statements and other artifacts, e.g., Poirot and RETRO. d. tools to identify the key abstractions in NL pre-RS documents, e.g. AbstFinder and RAI.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 9 Key Needed Capability of Tools Except for an occasional tool of category (a), part of whose task may include format and syntax checking ... each RE task supported by the tools requires understanding the contents of the analyzed documents.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 10 Can Tools Deliver Capability? However, understanding NL text is still way beyond computational capabilities. Only a very limited form of semantic-level processing is possible [Ryan1993].  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 11 “I Know I’ve Been Fakin’ It” a e h q  ¿ Consequently, most NLP RE tools ... use mature techniques for identifying lexical or syntactic properties, and ... then infer semantic properties from these. That is, they fake understanding.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 12 Lexing in Category c E.g., in a category (c) tracing tool, ... lexical similarity between two utterances in two artifacts leads to proposing links between the pairs of utterances and the pairs of artifacts.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 13 Drawbacks of This Lexing If the tool’s human user (a requirements analyst) sees no domain relevance in the lexical similarity, then he or she rejects the proposal (imprecision). Moreover, lexical similarity fails to find all relevant links (imperfect recall).  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 14 Recall and Precision Recall is the percentage of the right stuff that is found. Precision is the percentage of the found stuff that is right.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 15 Validation and Interaction Consequently, a human user always has to check and validate the results of any application of the tool, and NL RE tools are nearly always designed for interactive use.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 16 Using an Interactive Tool In interactively using any tool, e.g., a tracing tool, that attempts to simulate understanding with lexical or syntactic properties, ... the user has to know that the output probably will g include some false positives (impresision) and g not include some true positives (imperfect recall).  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 17 Using an Interactive Tool, Cont’d The action the user takes depends on the cost of failing to have the correct output, i.e., the links that show the full impact of a proposed change, vs. ... the costs of g finding the true positives and g eliminating false positives manually.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 18 In General, Though Finding the true positives ... is usually both harder and more critical... than eliminating false positives for the tool’s purpose. (Hence the point size difference on the previous slide!)  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 19 Scenarios of Tool Use Consider an analyst responsible for formulating a RS for a system (S ). The paper describes two scenarios: 1. S does not have high-dependability (HD) requirements. 2. S has HD requirements.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 20 Scenarios of Tool Use, Cont’d A system with HD requirements is one that is safety-, security-, or mission-critical. We ignore Scenario 1 in this talk and focus on Scenario 2 (the more controversial and discussion provoking one )  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 21 Second Scenario The analyst is responsible for formulating a RS for S with HD requirements.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 22 Second Scenario, Cont’d In Scenario 2, ... A complete analysis of all documents about S is essential ... to find all g defects, g abstractions, g traces or modeling elements, and g relationships that are present or implicit in the documents.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 23 Normal Behavior of Analyst Normally, the analyst would do the entire analysis manually. The analyst has the uniquely human ability to g extract semantics from text and g to cope with context, poor spelling, poor grammar, and implicit information (all too hard for NLP techniques).  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 24 Analyst’s Human Potential Thus, with appropriate knowledge, training, and experience, ... the analyst has the potential to achieve g 100% recall and g 100% precision.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 25 A Human is Human, Nu? Of course, g a human suffers fatigue, g and his or her attention wavers, resulting in g slips, g lapses, and g mistakes. In short, humans are fallible [DekhtyarEtAl]. Gasp!!!! ... Oy, Gevalt!  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 26 Even worse! The development of a HD S usually requires copious documentation, ... making fatigue and distraction so likely that ... tool support looks really inviting!  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 27 Second Scenario with Tools Consider Scenario 2 vs. the 4 tool categories: a. tools to find defects and deviations from good practice in NL RSs, b. tools to generate models from NL descriptions, c. tools to discover trace links among NL requirements statements or between NL requirements statements and other artifacts, and d. tools to identify the key abstractions from NL documents.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 28 Categories (a) & (b) Tools in these categories can be useful despite the imprecision and imperfect recall. See the paper. Basically, we expect less than perfection from these tools; so we naturally work with and around them.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 29 Category (a) The paper shows how a tool of category (a) with less than 100% recall overall could have 100% recall on an identifiable subset of the defects, and thus could be useful in Scenario 2. See the paper.  2012 D.M. Berry, R. Gacitua, P. Sawyer, & S.F. Tjong Requirements Engineering RD is Unstoppable Pg. 30 Category (b) The paper shows how a tool of category (b), which is for sure less than perfect, is nevertheless useful for what i