## **I-680 SMART CARPOOL LANE PROJECT** {#i-680-smart-carpool-lane-project .unnumbered}

## **SYSTEM ENGINEERING MANAGEMENT PLAN** {#system-engineering-management-plan .unnumbered}

##  {#section .unnumbered}

## **VERIFICATION (TEST) PLAN** {#verification-test-plan .unnumbered}

**PLAN SECTIONS:**

1.  **GENERAL**

2.  **ROLES AND RESPONSIBILITIES**

3.  **TEST PLAN**

4.  **EQUIPMENT ENVIRONMENTAL TESTING**

5.  **FACTORY ACCEPTANCE TEST**

6.  **ON-SITE INTEGRATION AND COMMISSIONING TEST**

7.  **OPERATIONAL PERFORMANCE TEST**

# GENERAL

> This document will present the I-680 Smart Lane electronic toll system
> (ETS) equipment and system verification testing process. It will
> outline the different types of tests that shall be conducted and
> identify the roles and responsibilities of each group that will be
> working on this project.

# Roles and Responsibilities

> Based upon the recommended distributed responsibility for the
> successful implementation of test procedures to ensure that the
> delivered tolling system operates according to the Request for
> Proposal (RFP) and Contract requirements, the I-680 Smart Lane project
> engineering organization will have various roles and responsibilities.
> Listed below are recommended starting points for defining
> organizational responsibilities pertaining to the verification test
> activities that are required to ensure program success.

## Joint Powers Agency (JPA) Personnel

1.  The JPA Executive Director (ED) shall have full contractual
    responsibility for all of I-680 Smart Lane equipment and system
    verification testing activities and will work closely with the
    Consultant staff to ensure that the testing is accomplished
    properly.

## Project Consultant Staff

> The JPA's I-680 Smart Lane Program and tolling system consultant staff
> shall have the following roles and responsibilities:
>
> 1\. Create and maintain all system verification and testing
> engineering related processes, policies and operating procedures for
> the Smart Lane project.

2.  Participate in all facets of testing activities and provide
    recommendations to the JPA for verification testing related aspects
    of the project.

3.  Provide technical assistance to the Integrator and JPA staff during
    the testing process.

4.  Provide regulatory guidance for security-related requirements in
    conjunction with the JPA management staff.

5.  Maintain a liaison with JPA and Integrator staff to incorporate
    additional testing procedures and ad hoc tests into the I-680 Smart
    Lane verification and testing process.

##  Integrator Systems Engineering Staff

> The Integrator systems engineering personnel for the I-680 Smart Lane
> Project shall have the following roles and responsibilities:

1.  Develop the various test scripts and procedures that are required to
    effectively support the I-680 Smart Lane verification and testing
    process.

2.  Perform the various tests that are required and detailing to the
    JPA/Consultant group each procedure that is being conducted and why.

3.  Ensure that the various system and equipment testing requirements
    and processes are properly flowed down from the RFP and the Contract
    documents by utilizing comprehensive requirements trace matrix.

4.  Develop a detailed testing report that identifies all of the tests
    that were run, any problems that were discovered and how those
    problems would be corrected.

5.  Perform any required re-testing activities that are identified by
    JPA and Consultant staff.

# TEST PLAN

The Integrator shall prepare a detailed plan for testing all hardware,
software and the full integration of the I-680 Smart Lane System. The
test plan shall be developed to confirm that the various functional
requirements that are presented in the RFP, the system design
documentation and the other Contract documents are met by the equipment
and/or system operations. The Test Plan and subsequent testing
activities shall be developed and executed for three (3) distinct
phases. Each test phase shall commence only upon the successful
completion of the previous phase. The three (3) test phases that will be
conducted by the Integrator, in the following order, are presented
below:

1.  Factory Acceptance Test (FAT);

2.  On-site Integration and Commissioning Test; and

3.  Operational Performance Test.

The Integrator shall be responsible to develop comprehensive test
scripts and test plans to ensure that the system development meets all
of the system requirements that are presented in the I-680 RFP and the
other contract documents. The test scripts and plans will be reviewed
and approved by Joint Powers Agency staff or its designated
representatives. The test scripts and plans shall be closely adhered to
during each phase of equipment and system testing. At the completion of
each test the Integrator shall submit final test results to the JPA for
final approval. The JPA ED, or designated staff/Consultant personnel,
will approve the test results.

Presented below in Figure 1 is a schematic that shows the three (3)
distinct I-680 Smart Lane Program testing phases and how each phase
inter-relates with the others.

Figure 1 -- Testing Process

The JPA and their representatives will be permitted to participate in or
otherwise observe any and all of these tests at the JPA's sole
discretion. Tentative dates for conducting the various tests shall be
included in the Test Plan document that shall be submitted by the
Integrator during the system design phase of the Contract. Reasonable
modifications to these dates may be permitted during the course of the
work by the JPA provided a written request for such change is made at
least two (2) weeks prior to the revised test date. The actual change
approval must be granted, in writing, by the JPA.

# EQUIPMENT ENVIRONMENTAL TESTING

As a preliminary effort prior to factory acceptance testing, the
Integrator shall also provide test results, certified by a testing
laboratory approved by the JPA, confirming that all proposed system
equipment that is to be installed is suitable to operate within its
respective environment. If certified test results are not available or
the results are not satisfactory to the JPA, the Integrator shall
arrange for such tests to be conducted or re-conducted at no additional
cost to the JPA. The Integrator shall provide the JPA with certified
test results for all equipment that is to be installed outside, or in
any other non-environmentally controlled location. For equipment that is
to be installed inside a building, cut sheets showing environmental
operating requirements will be acceptable.

# FACTORY ACCEPTANCE TEST (fat)

The intent of the FAT, which will be held at the Integrator's system
development office, is to allow the Integrator to conclusively represent
that the I-680 Smart Lane equipment, subsystems and overall system
complies with the system functional requirements. Representatives from
the JPA, the Consultant group, and possibly the Bay Area Toll Authority
(BATA) and/or Caltrans are expected to be present at the FAT. The FAT
shall be successfully completed, and accepted by the JPA, prior to
commencement of on-site equipment installation, system integration and
field testing. Equipment and/or system failures that are encountered
during performance of the FAT shall be resolved, retested and
acknowledged as being resolved by the JPA prior to issuance of FAT
approval to the Integrator.

It is possible that certain components that comprise the overall I-680
Smart Lane System may not be available during factory testing. The Test
Plan shall indicate those portions or components that will not be able
to be verified during performance of the FAT. Where applicable, the
Integrator shall attempt to simulate the missing components to represent
a fully functioning system.

Components used in the FAT shall be production models, which would
otherwise be suitable for installation in the I-680 Smart Lane System.
Testing and careful evaluation of samples and prototypes shall be
completed prior to the commencement of the FAT.

The FAT is the culmination of the design, development, fabrication and
pre-test of the Smart Lane System equipment, subsystems and overall
system. The FAT shall be performed by the Integrator under the
supervision of, and with the participation of, the JPA. Performance of
the FAT shall be witnessed by the JPA, and their representatives, with
acknowledgment of scenario success and/or failure by the JPA.

The Integrator shall prepare detailed test scripts that will be used as
the basis for the FAT. FAT Scripts shall cover test set-up, step-by-step
procedures and pre-determined expected results. FAT scripts shall be
submitted and approved by the JPA prior to the commencement of the FAT.
FAT scripts shall be submitted for review and approval no less than 60
days prior to scheduling of the factory test.

The following are recommended features of the Smart Lane System that
are, at a minimum, to be demonstrated during the FAT:

-   Equipment power up tests;

-   Verify initialization;

-   Verify data integrity (no loss of data);

-   Verify diagnostic messages;

-   Introduce failures and threats;

-   Degraded mode performance;

-   Verify diagnostic messages;

-   Normal transaction tests;

-   Tolling zone operational tests;

-   Proper association of vehicle and transponder;

-   Tolling zone stand-alone tests;

-   Operation of the Toll Data Center (TDC), including the dynamic
    pricing algorithms and the trip generation process;

-   Interface to the Caltrans Traffic Management Center (TMC);

-   Integration between the TDC and the BATA Regional Customer Service
    Center (RCSC);

-   Communication link between the tolling zones and the TDC;

-   Generation of traffic, toll revenue and reconciliation reports;

-   Data analysis;

-   Integrity checking;

-   System audit; and

-   Others yet to be identified.

# ON-SITE INTEGRATION and commissioning TEST

The On-site Integration and Commissioning Test (OICT) shall be comprised
of equipment, subsystem and system-wide testing of the Smart Lane
System. The purpose of the OICT is to provide both the Integrator and
the JPA a mechanism for verifying and documenting successful system
performance throughout the installation process up to the point of
approval to Commission the lane. Testing procedures and scenarios, which
will be developed by the Integrator and subject to JPA approval, shall
be built upon the previously utilized FAT scripts and test steps.

At the subsystem and component level(s), the OICT shall cover
installation check-out and performance verification at each applicable
location throughout the I-680 Smart Lane System. At the system wide
level, the OICT shall also cover end-to-end testing that represents a
fully integrated and functional Smart Lane System with all subsystems
and components successfully integrated on-site. The intent of end-to-end
testing during the OICT phase is to ensure readiness for the subsequent
Commissioning Test.

The Integrator shall prepare detailed test scripts for the OICT. OICT
scripts shall be designed to verify the equipment installation and
confirm that the subsystem and/or component(s) are ready for operation
on the Smart Lane System and the Commissioning Test can then commence.

The tests to be performed shall be defined in the Integrator's test plan
and test scripts. The detailed test scripts and scenarios of the Test
Plan shall be submitted and approved by the JPA prior to commencement of
any of the specific tests. Performance of the various tests shall be
witnessed by the JPA with acknowledgment of scenario success, failures
or potential system or equipment threats.

The commissioning portion of the OICT will be conducted in order to
represent the operational readiness of the system prior to deployment.
Testing activities and scenarios during these tests shall include
complete end-to-end testing of all functions and operations of the Smart
Lane System. This testing will involve live traffic in the Smart Lane,
both real traffic and test vehicles. To ensure that all of the stated
RFP and Contract operating requirements are effectively met, the
Integrator shall develop a comprehensive requirements trace matrix and
confirm, in writing to the JPA, that the tolling system has met each and
every stated requirement.

Any failures that are encountered during the Commissioning Test must be
resolved, retested and acknowledged by the JPA before opening of the
I-680 Smart Lane and commencement of the Operational Performance Test
(OPT). The tests to be performed shall be similar to the tests in the
FAT and shall be conducted under the supervision of, and with the
participation of the JPA, and their representatives, in accordance with
the test plan and test scripts that were previously approved. Testing
results and corresponding documentation regarding the completion of the
OICT shall be submitted to the JPA for approval. Approval of the OICT
must be granted prior to commencement of the Commissioning Test.

# OPERATIONAL PERFORMANCE TEST

The final phase of testing of the I-680 Smart Lane System is the OPT.
This test shall serve to closely monitor the performance of the Smart
Lane under live traffic operating conditions once the Smart Lane is open
to toll-paying vehicles. Activities during this period of testing will
include all necessary scripted test documentation, unscripted ad-hoc
tests as well as monitoring of day-to-day functions of the Smart Lane
System, including the operation of the equipment at the three (3)
tolling zones, the operation of the TDC, trip building functionality at
the TDC, the interface to the BATA RCSC, etc.

The OPT shall be conducted over a 30-day continuous period without
degradation in performance or failure in compliance with contract system
requirements. Throughout the 30-day test period any system problems,
errors, failures or malfunctions that are not in compliance with the
contract requirements shall be categorized based on its level of
severity. The typical four (4) levels of severity are:

-   Severity 1 -- Hardware or Software component or process that is
    critical to the operation of the Smart Lane that does not function
    and there is the possibility of loss of revenue and/or loss of data.

-   Severity 2 -- Hardware or software component or process that does
    not function. There is no risk of loss of revenue or data, however
    there is the possibility of negative impact to patron usage.

-   Severity 3 -- Hardware or software component or process that does
    not meet the design functionality and/or impedes the operation of
    the system but does not effect the collection of revenue or
    negatively impact the patron usage.

-   Severity 4 -- Hardware or software component or process that does
    not meet the design functionality and/or is "cosmetic" in nature.
    Failure is transparent in nature to both the patron and the JPA.

The success or failure of the 30-day OPT shall be determined by the
number and type of severity levels encountered during the test. The
following represents the impact of each of the severity levels on the
eventual outcome of the test.

**[Severity 1]{.underline}**

-   Indicates a failure of the 30-day test.

-   The 30-day test is stopped. Once the problem is resolved, the 30-day
    test is restarted from Day 1.

-   Milestone payments regarding the 30-day test are withheld.

-   Warranty period will not begin until successful conclusion of the
    OPT.

**[Severity 2]{.underline}**

-   Does not indicate a failure of the 30-day test.

-   The 30-day test is stopped. Once the problem is resolved, the 30-day
    test will resume from the date in which it left off. Once the
    problem is resolved the implemented fix shall operate without flaw
    for seven (7) consecutive days, which may extend the 30-day test
    period.

-   Milestone payments regarding 30-day test will be withheld.

-   Warranty period will not begin.

-   Reoccurrence of the same problem that should have been resolved
    might raise the level of the issue to Severity 1.

**[Severity 3]{.underline}**

-   Does not indicate a failure of the 30-day test.

-   The 30-day test is stopped. Once the problem is resolved, the 30-day
    test will resume from the point in which it left off. Once the
    problem is resolved the fix shall operate without failure for a
    minimum of seven (7) consecutive days, which may extend the 30-day
    test period.

-   Milestone payments will continue to be paid to the Integrator.

-   The Warranty Period will not begin.

**[Severity 4]{.underline}**

-   Does not indicate a failure of the 30-day test.

-   The 30-day test is not stopped. The problem is resolved in a timely
    fashion.

-   Milestone payments will continue to be paid.

-   The Warranty Period will not begin.

All I-680 Smart Lane hardware and software will be carefully tested.
Verification that all reported problems have been resolved will be
obtained using several methods, including event logs and service call
logs and other additional information that can be gleaned from the
Maintenance On-Line Management System maintained by the Integrator or
other data sources approved by the JPA. Multiple meetings per week will
be scheduled during the 30-day test to ensure that the JPA is fully
aware of all system and equipment failures and the meetings will provide
a forum in which JPA and Integrator staff can review system/equipment
failures and classify the severity levels.

  ------------------------------------------------------------------------------
  I-680                                                                
  Smart Lane                                                           
  Project                                                              
  Document                                                             
  Revision                                                             
  History                                                              
  ---------- ---------------- --------------------------- ------------ ---------
  Rev        Revision                                                  Date
             Description                                               

  1          First Release --                                          4-14-06
             Verification                                              
             (Test) Plan                                               

  2          Agency Review                                             4-17-06
             version                                                   

                                                                       

                                                                       

                                                                       

  Approval                    Signature                   Date         
  Person                                                               

                                                                       
  ------------------------------------------------------------------------------