– A Verification Cross -Reference Matrix (V CRM ) is a table that depicts the verification method s for requirement s in a specification . Usually requirement labels are rows, available test method s are columns, and an “X” in a cell indicates usage of a verification method for that requirement. Verification m ethods include Demonstration, Inspection, Analysis and Test , and sometimes C ertification, Similarity and/ or Analogy . V CRM s enable acquirers and stakeholders to quickly understand how a product ’s requirements will be tested. Maintaining consistency of very large VCRMs can be challenging, and inconsistent verification methods can result in a large set of uncoordinated “spaghetti tests”. Natural language processing algorithms that can identify similarities between requirements of fer promise in addressing t his challenge . This paper applies and compares compare s four natural language processing algorithms to the problem of automatically populating V CRMs from natural language requirements: Naïve Bayesian inference, (b) Nearest Neighb or by weighted Dice similarity, (c) Nearest Neighbor with Latent Semantic Analysis similarity , and (d) an ensemble method combining the first three approaches. The VCRMs used for this study are for slot machine technical requirements derived from gaming re gulations f rom the countries of Australia and New Zealand, the province of Nova Scotia (Canada), the state of Michigan (United States) and recommendations from the International Association of Gaming Regulators (IAGR). Keywords —requirements analysis, natur al language processing, verification matrices, verification methods, slot machines. I. INTRODUCTION Stakeholder requirements often form the basis of product acceptance testing , and a verification cross reference matrix (VCRM) table can be used to document plans and agreements on how each requirement should be tested . VCRM s are table s with requirements labels as rows and verification methods (such as (Demonstration, Inspection, Analysi s, and Test) as columns. The t est method for a particular requirement is designated by placing an “X” in the cell of the corresponding row and column. The VCRM links requirements engineering and test planning. Verification method s include (summariz ing from [1]): • Inspection – Visual o r dimensional examination of an element, relying on human senses or simple m easurements , • Analysis – Use of analytical evidence obtained from mathematical or logical reasoning and/or simulations without any in tervention on the submitted element , • Demonstration – Show of correct operation of the examined element against operational and observable characteristics without physical measurement, and • Test – Quantitative verification of m easurable characteristics, operability, supportability or performance capability when subjected to controlled conditions. Other methods that are sometimes included are Analogy or Similarity (form s of analysis comparing one product to a similar product) and Certifica tion (verification by a third party or previous event). Requirements can often be verified by any of several means. For example, correctly computed quantities can often be verified by either Demonstration on a display , or Test with instrumentation to sh ow the inputs and outputs of a calculation. On very large programs, a VCRM may include tens of thousands of entries. Verification method consistency is important, b ecause an internally inconsistent VCRM can result in a large collection of uncoordinated “sp aghetti tests”. Also, for programs that intend to leverage product reuse, an inconsistent VCRM can impair tes t program reuse efforts. Natural language processing (NLP) algorithms offer promise in addressing th e challenge of maintaining VCRM consistency by grouping requirements that are by some measure similar to each other. This study compares four different NLP approaches for automatically populating VCRMs from natural language requirements . Section 2 describes prior work in this area and the signifi cance of this contribution. Section 3 discusses the data sets and their preparation. Section 3.1 describes the data used in this study. T echnical requirements were extracted from four sets of slot machine gaming regulations, one from Australia and New Zealand [2], one from Nova Scotia (Canada) [3], one from the International Association of Gaming Regulators [4] and one from Michigan (United States) [5]. Section 3.2 describes the data pre - processing performed on this data . Pre-processing includes requirements transformation and extraction from the regulations, removing stop words and punctuation, and stemming the remaining keywords. It also includes assigning “truth data” verification methods (VMs) using a rules model and domain knowledge. The res ult is a “truth VCRM ” matrix used to assess model accuracy. Section 4 describes the models analyzed. Four models were compared: Naïve Bayes, Nearest Neighbor (NN) by weighted Dice similarity, NN with Latent Semantic Analysis (LSA) weights, and an e nsembl e model. Analysis of model results is found in Section 5 . The three standalone models showed VM accuracy of 77% -83%, and NN with LSA weights performed the best. Better yet was the ensemble model , which had a VM accuracy of 85% . Modeling and human erro r sources that reduced model accuracy are also analyzed in this section. Conclusions and next steps are in Section 6. Given the accuracy of these models, they can be used as analysis tools for VCRM consistency checking. None of the four algorithms evaluated were robust enough to replace the engineer with domain expertise, however. II. PRIOR WORK AND THIS CONTRIBUTION Many papers have applied Natural Language Processing (NLP) to requirements engineering, and [6] lists over 400 of them. A high -level summary of recent thrusts to apply NLP to problems in requirements engineering can also be found in [7]. One of these thrusts is similarity analysis. Similarity analys es within and be tween engineering documents are found in [8, 9, 10]. Cluster analysis to identify requirement similarities across a product line is found in [11]. Similarity analysis to identify functional and non -functional requirements from user app reviews is found in [12, 13]. A similar effort using requirements derived from mathematical software is found in [14]. NLP was used to identify non - functional requirements from natural language documents in [15]. A similar approach for slot machine development, th e technica l domain of this paper, is found in [16]. On the test side, [17, 18, 19, 20] have all proposed NLP techniques for auto -generation of test cases . These approaches have had varying degrees of success, as the test problem has the potential for exponential ex plosion in test case counts . An effort to optimize test cases by clustering is described in [21]. To date, however, similarity analysis has not been used to automatically generate a VCRM. That is the focus of this paper. III. DATA SETS 3.1. Casino Regulation Source Data Slot machine technical requirements were selected for this study because of the plethora of freely available technical requirements for their construction from regulatory bodies. This study used natural language technical requirement s derived from compliance regulations from Australia and New Zealand [2], Nova Scotia [3], the International Association of Gaming Regulators (IAGR) [4], and the state of Michigan [5] . These regulations are d escribed below. 3.1.1. Australia/New Zealand The Australia/New Zealand Gaming Machine National Standard 2016 [2] is the national standard of both Australian and New Zealand for electronic gaming machines. It contains both exposition and definition information as well as numerous technical requirements for electronic gaming machine s. This includes requirements for payouts , progressive awards, displays (“artwork”), cyber security and auditability. The standard also contains process instructions for certifying new electronic gaming machines. This document is one of the most comprehensive gaming machine regulations available, 82 pages long and containing 1,191 paragraphs and sub-paragraphs. 3.1.2. Nova Scotia Nova Scotia’s Casino Regulations made under Section 127 of the Gaming Cont rol Act [3] contains both regulations for casino operations in Nova Scotia as well as detailed technical requirements for slot machines. The requirements include requirements for hardware, software, error handling (including tilt conditions), and odds calc ulations and progressive payouts. The document contains 201 paragraphs and subparagraphs. 3.1.3. International Association of Gaming Regulators In an effort to consolidate electronic gaming requirements into an international standard, the I nternational A ssociation of Gaming Regulators (I AGR ) developed the International Technical Standards for Electronic Gaming Machines [4]. This 42-page, 353 paragraph proposed standard includes requirements for hardware, interference effects, software, memory, electronic ledgers (“meters”) , communications and artwork. Though it contains a few definitions and some background information, this document is almost entirely comprised of technical requirements. 3.1.4. Michigan Michigan Gaming Control Board Casino Gaming [5], like the Nova Scotia regulations, serves as both a set of regulations for casino operations as well as a set of detailed technical requirements for slot machines. Almost all of the slot machine technical requirements are found in Part 8, Conduct of Gaming/Gaming Equipment , which contains 294 paragraphs and sub -paragraphs. Like the other standards, these requirements include security, error handling, software, hardware, communications and artwork. 3.2. Data Preprocessing Data preprocessing consisted of three activities. First , the technical requirements had to be extracted from the regulations. Then, VMs were tagged to each requirement. These “truth data” VMs formed the basis by which the accurac y of the four NLP models would be judged . Finally, the requirement text was prepared for analysis by removing stop words and punctuation, stemming the remaining words, and discarding one - and two - letter words. 3.2.1. Requirements Extraction Gaming regulations are written in natural languag e by administrat ors, lawyers or regulators, and often lack the word “shall”. Requirements extraction from the regulations w as performed in two passes. First, a rule -based heuristic was used to extract candidate requirements. Then, recommendations from the rule -based heur istic w ere manually reviewed and updated by a systems engineering requirements expert. A 12 -rule heuristic algorithm that had over 99% accuracy in extracting technical slot machine product requirements from Nevada and South Dakota gaming regulations is d escribed in [16]. Th at ruleset was adopted for this study but extended to 14 rules so as to take into account phrasing variations across the different English -speaking countr ies (Canada, New Zealand, Australia and United States) . The 14-rule heuristic employed to transform the text into requirement “shall” statements and conditional “may” statements for this study was : 1) “must”, “should”, “may” in “may not” or “may only”, “will”→ “shall”, 2) “[is, are] required to”, “[is, are] to” → “shall”, 3) “can” → “may”, 4) “[is, are] (not) permitted”, “[is, are] (not) allowed” → “shall (not) be allowed”, 5) “cannot” → “shall not” 6) “[is, are] required” → “shall be required”, 7) “[is, are] only to” → “shall on ly” 8) “[is, are] (not) to be” → “shall (not) be” 9) “requires a” → “shall require a”, 10) “[is, are] [responsible, determined]” →“shall be [responsible, determined]”, 11) “if … are” → “if … shall be”, 12) “responsibility … lies” → “responsibility … shall lie”, 13) “no … may” → “no … shall” 14) Each bullet in a requirement is itself a requirement. To illustrate this last rule, consider the requirement: Each machine shall contain: a) a hopper, and b) a payout meter. This requirement is transformed into the following two requirements: Each machine shall contain a coin hopper . Each machine shall contain a payout meter . These rules were able to extract oper ational and technical requirements from the regulations with 98% accuracy. Rule failures occurred when the ruleset tried to create requirements out of definitions and for certain edge -case negative requirements. This simple 14 -rule heuristic cannot handle complex requirement rewording, and hence the human -in-the- loop was critical to this process . Consider for example the almost nonsensical regulation from Michigan: “Direct collection of the random number data from the actual submitted (game) is required unless it is not possible.” On first blush, this requirement may seem to have no meaning at all, equivalent to “do X unless you cannot do X”. Though poorly worded, t he intent of this regulation is to levy a conditional requirement, with the condition tha t the requirement applies if the vendor selects a random number generator that can be monitored without interfering with the random number generat ion process itself. For example, a r andom number generator relying on quantum processes might not be able to b e monitored without affecting its stat e. A rewording of this requirement to make it clearer might read: “If the slot machine random number generator permits direct collection of random data without affecting the generator’s state, then that data shall be collected, stored and made available for post -test analysis.” This type of complex transformation was beyond the scope of our simple ruleset, and so required a human -in-the-loop. 3.2.3. Setting Verification Method Truth Data VM selection is somewhat subjective, because a requirement may be verifiable by any of several means. T est engineers may have differ ent p