Quality Improvement Organization Manual 
Chapter 15 - Performance Evaluation 
 
Table Of Contents  
(Rev. 31, 07-21-17) 
 
Transmittals for Chapter 15 
 
FEDERAL REGISTER NOTICE OF EVALUATION CRITERIA 
 
15100 - Authorities 
15110 - Provisions of the Notice 
15120 - Uses of Evaluation Criteria 
 
CONTRACT EVALUATION PROCESS 
 
15200 - Timing 
15210 - Methods of Evaluation 
15220 -  Renewal/Non-renewal 
15230 - Methods of Evaluation 
 
PERFORMANCE IMPROVEMENT PLANS (PIPs) 
 
15400 - Performance Improvement Plans 
15420 - Performance Plan Expectations 
 
TERMINATION PROCEDURES 
 
15500 - Statutory Basis 
15510 - Grounds for Termination 
15520 - Recommendation to Initiate Termination 
15530 - Notice of Intent to Terminate Contract 
15540 - Termination Panel 
15550 - Termination Decision 
 
RENEWAL/NON-RENEWAL 
 
15600 - Renewal/Non-renewal 
 
CLOSE-DOWN ACTIVITIES 
 
15700 - Introduction 
15710 - Boxing of Records - General 
15720 - Boxing of Review Records 
15730 - Boxing of Data  15740 - Boxing of Miscellaneous Records 
15750 - Retention of Financial Records 
  15100 - Authorities  
(Rev.31, Issued: 07-21-17, Effective: 07-21-17, Implementation: 07-21-17) 
 
Section 1153 of the Social Security Act (the Act) generally provides authority for CMS to 
enter into contracts with QIOs for the performance of QIO functions described in section 
1154.  CMS may modify, renew, or discontinue all or part of one or more tasks for one or 
more states or territories in the QIO service area or any other part of the contract. 
 
Sections 1153(c)(2) and (4) of the Act provide for CMS to assess the quality and 
effectiveness of each QIO in carrying out the functions specified in its contract when 
evaluating QIO performance. 
 
Section 1153(h)(1) of the Act requires CMS to publish notices in the Federal Register 
whenever it makes significant policy and/or procedural changes that substantially affect 
performance of QIO contract obligations 30-days before such change is to take effect. 
 
Section 1153 (h)(2) of the Act requires the Secretary to publish in the Federal Register 
the general criteria and standards used for evaluating performance of QIO contract 
obligations, and provide an opportunity for public comment.  
 
Section 1153 (h)(3) of the Act requires CMS to regularly provide each QIO an 
assessment of its performance in relation to the performance of other QIOs. 
 
 
15200 - Timing 
(Rev.31, Issued: 07-21-17, Effective: 07-21-17, Implementation: 07-21-17) 
 
CMS publishes Federal Register notices for each type of QIO contract that it plans to 
award for each QIO contract award cycle. These notices provide the opportunity for 
public comment on the general criteria and standards CMS will use to evaluate QIO 
performance based on the tasks or subtasks in the statement of work and the evaluation 
and performance measures included in QIO contracts as required by Section 1153 (h)(2) 
of the Act. 
 
After QIO contracts are awarded, CMS may revise contract performance criteria to 
adjust evaluation measures, adjust the expected minimum thresholds for satisfactory 
performance, or add or remove criteria from a contractual task for any reason, 
including, but not limited to, experience with the amount of improvement achieved during 
the contract cycle, information gathered through evaluation of the performance overall, 
or any unforeseen circumstances through negotiation of contract modifications or change 
orders. CMS negotiations with QIOs of any contract change (modification) or non-
competitive renewal are administered by the cognizant CMS contracting authority in 
accordance with authorities of 48 CFR Federal Acquisition Regulations and notices are 
not posted in the Federal Register unless required by section 1153(h)(1).  
  15210 - Methods of Evaluation  
(Rev.31, Issued: 07-21-17, Effective: 07-21-17, Implementation: 07-21-17) 
 
Substantial elements of QIO contracts are performance-based, such that a QIO is 
assessed relative to specific performance measures established in the contract. 
Performance on these measures can be used to objectively make determinations about 
non-competitive renewals and/or be used as objective criteria in competitive 
procurements. Additionally, CMS may also assess QIO performance based on annual 
routine assessment, regularly scheduled teleconferences, data analysis, off-site reviews of 
QIO project activities, contract deliverables, reporting, data reports, interviews with QIO 
staff, customer satisfaction and/or an examination of other pertinent records. 
 
CMS follows federal acquisition regulations when assessing QIO contractor performance 
on the contract. CMS conducts contract performance monitoring activities throughout 
the contract term and will act upon findings as necessary. 
 
CMS performs regular formal assessment of each QIOs overall contract performance 
based on the general evaluation criteria and standards identified in each QIO contract. 
The assessment is provided to the QIO through the CMS Contracting Officer using the 
Contractor Performance Assessment Reporting System (CPARS) criteria used in 
performance evaluations: Quality, Schedule/Timeliness, Cost/Price Control, Business 
Relations, Management, and Small Business. CMS considers the published general 
evaluation criteria and standards within the Quality sub-factor within the CPARS 
assessment of overall contractor performance.  
 
CMS performs regular program impact assessments and may contract with an 
independent entity to perform assessment of overall QIO Program performance. 
Program evaluations are conducted separately from individual QIO contractual 
performance evaluations. CMS makes QIO Program performance assessment results 
available to each QIO as part of regular QIO program evaluations. 
 
15220 - Renewal/Non-renewal  
(Rev.31, Issued: 07-21-17, Effective: 07-21-17, Implementation: 07-21-17) 
One of the purposes of the evaluation process is to make determinations on whether QIOs 
are eligible for non-competitive contract renewals. Details of the renewal/non-renewal 
process can be found in the QIO contract. 
 
15230 - Methods Of Evaluation - (Rev. 1, 05-23-03) 
 
Substantial elements of QIO contracts are performance-based, where a QIO is assessed 
relative to specific performance measures established in the contract.  Performance on 
these measures can be used to objectively make determinations about non-competitive 
renewals and/or be used as objective criteria in competitive procurements.  Additionally, 
ROs assess QIO performance based on on-site visits, regularly scheduled teleconferences, 
data analysis, and off-site reviews of QIOs.  Specifically, the on-site visit permits the RO 
POs and/or Scientific Officers (SOs) to have a face-to-face meeting and allow CMS an  opportunity to understand all of the QIO activities within the context of their state.  This 
process entails the evaluation of QIO project activities and data reports, interviews with 
QIO staff, and an examination of other pertinent records. 
 
Ongoing monitoring will be accomplished by telephone and videoconferencing between 
the PO and the QIO, on-site visits and analysis of information reported in project plans, 
routine written reports, and review of contract deliverables. 
 
The findings that result from on-site visits are compiled into formal summaries that 
describe the QIOs progress and performance.  If the PO identifies major deficiencies, a 
Performance Improvement Plan (PIP) will be requested.  In the event a QIO is 
unsuccessful at correcting the deficiencies contained in the PIP, CMS will consider 
terminating the contract.  RO POs would continue to perform ongoing monitoring of 
other aspects of the contract.  These include: 
 
  QIO proposed information collection activities (e.g., surveys), abstracts or articles 
submitted to non-CMS sponsored peer reviewed publications or meetings; 
 
  Subcontract arrangements; 
 
  QIOs' internal quality control process; 
 
  QIOs case review requirements; 
 
  Communication requirements (Web site, physician/provider outreach, annual 
reports, etc.); 
 
  Requests related to quality improvement projects (including ad hoc data requests, 
CDAC requests, and approval of locally developed projects); 
 
  Confidentiality requirements; 
 
  Board membership and structure; and 
 
  QIO resource allocation/utilization and management (and other financial 
information). 
 
15400 - Performance Improvement Plans 
(Rev.31, Issued: 07-21-17, Effective: 07-21-17, Implementation: 07-21-17) 
 
CMS may request a QIO to provide a corrective action (Performance Improvement Plan) 
to address noted deficiencies.  
 
 
 
   
 
 
15550 - Termination Decision - (Rev. 1, 05-23-03) 
 
The CMS may or may not accept the findings of the panel.  After the panel has submitted 
its report, CMS may, with concurrence of the QIO, amend the Statement of Work (SOW) 
to modify the QIOs functions or otherwise change the contract.  Also, CMS may elect to 
terminate the QIOs contract upon 90 days notice after submission of the panels report or 
earlier if the QIO agrees.  In accordance with 1153(f) of the Act, any determination by 
the Secretary to terminate a contract shall not be subject to judicial review. 
 
From the time CMS receives the panels report and gives the notice of intent to terminate, 
CMS may transfer review responsibilities of the QIO under the contract being terminated 
to another Utilization and Quality Control QIO, or to an intermediary or carrier having an 
agreement under 1816 of the Act or a contract under 1842 of the Act until CMS enters 
into a contract with another QIO. 
 
15600 - Renewal/Non-renewal Procedures - (Rev. 1, 05-23-03) 
 
One of the purposes of the evaluation process is to make determinations on whether QIOs 
are eligible for non-competitive contract renewals.  Details of the renewal/non-renewal 
process can be found in the QIO contract. 
 
 
 
 
 
  Transmittals Issued for this Chapter 
 
Rev #  Issue Date  Subject  Impl Date  CR# 
R15QIO  07/21/2017  QIO Manual Chapter 15  Performance  07/21/17  N/A 
Evaluation 
R1QIO  05/23/2003  Initial Release of Chapters 2,8,13 and 15  N/A  N/A 
Back to top of chapter