(General Information, Eligibility, and Entitlement Manual)

 Actual Results for the run of each test case in which the actual  results did not match the expected  results; and    Documented proof of each test run, i.e., screen shots, scheduler job logs, etc.  2. The SSM, STC, and MAC shall:    Maintain all test documentation for the four quarterly releases prior to the current release under test.  The documentation must be available for review by CMS (or its agent).    Document all software defects (problems) within the CMS specified repository such as INFOMAN.   3. The SSM shall communicate all confirmed software defects (problems) and fixes directly to CMS in writing through their CMS Maintenance Lead/(GTL) or other designee as specified by the CMS Project Officer.   4. A/B, DME MACs shall provide any testing documentation to their CMS regional office upon request.   Additional requirements for selected shared system maintainers, STC, and CWF hosts may be contained in these organizations' individual contracts.  Electronic screen shots may be incorporated/attached into the results of ALM has proof of online results.  Test Log requirements may be fulfilled by correctly using the ALM run feature as outlined in the Quarterly Release Test Management User Guide.   40.3.9 - Definitions (Rev.  105, Issued: 06 -02-17, Effective: 07- 03-17, Implementation: 07- 03-17) These definitions are provided to ensure common understanding.  Base Shared System  - The FISS, MCS, VMS, or CWF system, which is routinely released by the Shared System Maintainers to their respective user community prior to any user customization.  This includes all components released by the Shared System Maintainer, including but not  limited to the claim adjudication subsystem, the financial subsystems, and other integrated components (i.e., Pricer, OCE, MCE, Grouper).  Functional Testing   Testing to ensure that the functional requirements have been met. Functional testing is performe d by the SSMs and the STC. Integration Testing   Testing combinations of interacting software components that make up parts of a system. Integration testing is performed by the SSMs, and the STC. Interface Testing   Testing conducted to evaluate whether su bsystems or systems pass data. Interface testing is performed by the STC.   Local Components   A Local Component as referenced in section 40.3 is any component or module that supports Medicare claims processing, but is not part of the Base System and is under the control and maintenance of the MAC.  Maintainer   The Maintainer is an entity to which CMS directly contracts to maintain a Medicare claim processing shared system (FISS, MCS, VMS, or the Common Working File (CWF) system).  The Maintainer, as referenced in section 40.3, does not refer to an entity to which a A/B or DME MAC subcontracts to operate their data center or perform other claim processing support activities.  Operational Testing   Testing conducted to evaluate a system in its operational envi ronment. Testing to ensure that the aggregate operational systems and their interfaces can be operated securely with the instructions provided. Operational Testing is performed by the SSMs and the A/B and DME MACs.  Performance Testing   Testing that applie s heavy transaction and processing loads to the system to ensure that response times, throughput rates, and processing windows remain acceptable and can accommodate production workloads. Performance Testing is performed by the STC.  Regression Testing   Tes ting conducted on a system or components to verify that modifications have not caused unintended effects and that the system or components still complies with its requirements. Regression testing is performed by the SSMs and the STC.  Regression Test Set   A set of selectable test cases designed to exercise a system over its functional capabilities and assure that it still works properly after changes have been applied.  Requirement Identifier   A unique number assigned to each requirement comprised of the Shared System Maintainer CR Number, the CMS CR Number, and an alphanumeric element to uniquely qualify each requirement.  For testing purposes CMS requires that each Test Case Identifier incorporate the Requirement Identifier to which it is traced.  Stress Te sting   Testing that applies a steadily increasing load to the system until it reaches the point where performance degrades to unacceptable levels.  System Testing   Testing to discover any incorrect implementation of the requirements or incompatibilities in the software/hardware environment.  System testing includes functional testing, performance testing, and operational testing. System testing is performed by the SSMs and the STC. Test Case Specification   A description of an input situation and of the r equired results associated with a specific test objective or purpose. Test Case Identifier   A unique identifier assigned to each test case.   Test Log   A chronological record of relevant detail about the execution of tests.  Relevant details include run date, run time, test status, and actual results.  Test Requirement  - A specific requirement that is under test and to which one or more test cases are traced.  Test requirements may be derived from various types of requirements i.e., business functional requirements, performance requirements etc.  Note: Any well -written requirement that is testable may be considered a Test Requirement.  Any requirement contained in the Business Requirements section of a CR or transmittal, also constitutes a test requirement. Test Set   A collection of test cases that have a common usage.  Unit Testing   The testing of individual units (i.e., software components, modules) or groups of related units.  It is the lowest level of testing and is usually performed by programmers.  U nit testing may be both functional (requirements oriented) and structural (i.e. logic oriented, code coverage oriented). Unit testing is performed by the SSMs.   40.3.10 - Test Case Specification Standard  (Rev. 97, Issued: 01- 15-16, Effective: 09- 21-15, Im plementation: 09 -21-15)  Purpose:  This standard establishes a controlled outline for the contents and presentation of a Test Case Specification used by the shared system maintainers and the STC.   Applicability:  This standard is applicable to all Test Case  Specifications developed by the shared system maintainers and the STC.   Data Element  Description  Allowable Values or Format  Comments  Test Case Specification Identifier  Multi -part indicator that uniquely identifies the test case specification.  See Test Case Specification Identifier Standard.   Test Purpose  A free form field that captures the intent of the test and identifies any key components of the test, e.g., specific codes.  See attached example.   Input Specification  A free form field that captures critical information used to exercise the system functionality.  Information could be grouped into the following topics:  Claim Data Requirements  Claims History  Beneficiary Information  Provider Information  See attached example.   Intercase Dependencies (Predecessor Transaction Identifier)  The test case specification identifier of the transaction that must be entered into and processed by the system prior to processing the transaction described by the test case specification.  See Test Case Specification I dentifier Standard.  Output Specification  A free form declarative statement that identifies the expected results from performing all the steps, as a collection, within the test.    Test Type  A one -character indicator to identify whether the test is positive or negative.  P = Positive Test  N = Negative Test   ALM  Plan Tab (Required User Defined Fields)  Originator  A one -character indicator to identify the originating entity (designer) of the test case.  B = Beta  C = CMS/QRTM  M = Maintainer  Test Status  Summary indicator for a test case.  PS = Passed  FA= Failed  NR = Not Run  IN = Incomplete  ID = Invalid Data  IC = Invalid Case   Required Test  Execution (Run) Elements   Test Results  Free form declarative statement of actual results for a test case when the actual results do not match the expected results.    Optional Information: Industry best practices demonstrate that additional granularity may be necessary to document discrete key test actions that should be executed and documented.  These items are referred to as test steps.  A test case specification may have one or more test steps.  When documenting test steps, the following standard applies:   Step Number  Unique identifier for each test step.  Step n  Where n is a sequential counter for each ste p starting at 1.  There is at least one test step in each test case specification, but usually contains multiple test steps.       Optional  Test Case  Elements  Step Description  A free form declarative statement that identifies the action taken to perform the  test.  The step description statement usually begins with a verb.    Expected Step Results  A free form declarative statement that identifies the expected results from performing the associated step description.    Example #1   Test Case Identifier  4419 -2825 -5.2-001  Test Purpose  To confirm that the A/B MAC (A) claims processing systems accept, process, and assign reason code 30 (Payment adjusted because the patient has not met the required eligibility, spend down, waiting or residency requirements)  to Inpatient Hospital claims submitted on Type of bill (TOB) 111 (Hospital Inpatient Part A; admit through discharge) with Dates of Service (DOS) on 01/01/2004 when a beneficiary is not lawfully present in the United States.  Input Specification  Claims History  None  Beneficiary Information  Beneficiary must be unlawfully present in United States.  Beneficiary elected English as primary language  Provider Information  Provider Number Range = XX0001 -XX0999  Claim Data Requirements  TOB = 111  DOS = 01/01/2004   Intercase Dependencies  None   Output Specification  Claim will be assigned reason code 30 indicating beneficiary is not lawfully present in the United States, generating MSN message 5.7 (Medicare payment may not be made for the item or service because on the date of service, you were not lawfully present in the United States).   Test Type  P  Originator  C  Test Status  PS  Test Results  Claim was assigned appropriate reason code    40.3.11 - Single Testing Contractor (STC) Non -Testable Conditions and Potential Testing Impacts  (Rev. 125; Issued: 07- 05-19, Effective: 08- 05-19, Implementation: 08- 05-19)