'''''C''. ( t DEPARTMENT OF HEALTH &. HUMAN SERVICES Office of Inspector General 

Washington , D. C. 20201 

MAR 1 6 2006 

TO:	 Abby Block, Director


Center for Beneficiary Choices


Centers for Medicare & Medicaid Services


FROM: Stuar Wright 


Deputy Inspector General


for Evaluation and Inspections


SUBJECT:	 Early Implementation of MM: Status Report of the State Health Insurance 

Assistace Programs (OEI-05-05-00190) 

This memorandum contain information regarding the Centers for Medicare & Medicaid 

Services' (CMS) ability to assess the State Health Insurance Assistace Program (SHIP) as 

SHIPs begin to assist beneficiaries interested in enrollng in the new Medicare drg benefit. Ths 

early implementation review provides a description of the curent status ofCMS' s oversight 

efforts regarding this program and contains implications for the futue, as well as some 

suggestions for fuer improvements. This work also lays a baseline for futue Office of 

Inspector General work in the area. 

Background 

SHIP provides a "health insurance advisory service to assist Medicare eligible individuals with 

the receipt of services under the Medicare and Medicaid programs and other health insurance 

programs." 1 SHIs curently operate in all 50 States, the District of Columbia, the Virgin 

Islands, Puerto Rico, and Guam. SHIPs are generally composed of volunteer networks in local 

communities, with over 1 300 local SHI offces around the countr supported in large par by 

over 10 000 volunteers. The SHIP counselors have approximately 1.8 milion beneficiar 

encounters a year through individualized couneling. Additional beneficiares are served though 

public and media events. 

The Medicare Drug, Improvement, and Modernation Act of 2003 (MMA) made comprehensive 

prescription drug coverage under Medicare Part D available to all 42 milion Medicare 

beneficiaries. Beneficiaries generally have the option of enrollng either in a stad-alone 

prescription drg plan (PDP) and receiving all other Medicare benefits through fee-for-service or 

enrollng in a Medicare Advantage plan (MA plan) and receiving all Medicare benefits (including 

drg coverage) through managed care. To receive this benefit, Medicare beneficiaries select a 

prescription drg benefit package that best meets their needs based on differences in 

I Public Law 101-508. 
Page 2 – Abby Block 

premiums, copayments, formularies, and pharmacy networks.2 Beneficiaries must select a plan 

by May 15, 2006, or face penalties in the form of higher premiums. 

Prior to the passage of the MMA, SHIP was a relatively small program with limited outlays of 

Federal funding. SHIPs provided information and counseling regarding Medicare and other 

insurance services, but did not assist beneficiaries in enrolling for benefits. In response to the 

MMA, SHIPs have taken on a prominent role in conducting outreach and counseling 

beneficiaries on the new drug benefit. In addition to providing information regarding the new 

benefit, CMS has instructed SHIPs to help Medicare beneficiaries enroll in the drug benefit.3 A 

CMS press release characterized SHIPs as the “key foot soldiers in providing one-on-one 

assistance for Medicare beneficiaries to help them get the best health care.”

4 SHIPs are of 

particular importance because they are able to counsel the beneficiary, sometimes in face-to-face 

situations, about drug plans that will best meet their needs. They are able to provide 

individualized assistance and have access to local information. 

To support the increased expectations to counsel and enroll beneficiaries in the new drug benefit, 

CMS increased SHIP funding. In the past, CMS annually allocated $10 million in grants to 

SHIPs based on a formula stipulated by regulation.5 This allocation was increased to 

approximately $21 million in fiscal year (FY) 2004 and approximately $32 million in FY 2005.6 

Given the importance of SHIPs to the successful implementation of the new drug benefit, it is 

imperative that CMS has the tools available to assess the SHIP performance. It is only by 

monitoring and assessing their performance that CMS will be able to ensure that they are 

effectively assisting beneficiaries and using the enhanced funding in an effective and efficient 

manner. 

Methodology 

To ascertain the current ability of CMS to measure the SHIPs’ performance during the early 

stages of MMA implementation, we reviewed CMS’s mechanisms to collect and assess 

performance data and the performance data SHIPs submitted. 

We reviewed the FY 2005 SHIP grant announcement, the notice of grant award, and data 

collection forms to determine what performance expectations exist, the associated performance 

measures, and the data sources used to measure whether expectations are being met. In addition, 

2

 Beneficiaries dually eligible for Medicaid and Medicare and Medicare beneficiaries enrolled in managed 

care plans will have plans automatically assigned to them. However, they have the option of selecting a 

different plan that better suits their needs. 3

 FY 2005 grant announcement, p. 9. 4

 CMS press release, April 2005. 5

 42 CFR § 403.504. 6

 The overall appropriation to SHIPs for FY 2005 was $31.7 million, of which approximately $1.4 million

was applied to the SHIP Resource Center, SHIPtalk.org, SHIP performance assessment, and the SHIP

National Conference, and approximately $1 million was used for SHIP Competitive Leadership Grants. 

This leaves the first $10 million in annual grants and the approximately $19.3 million under the new 

formula. 
Page 3 – Abby Block 

we conducted structured interviews with the CMS staff and their contractors to understand the 

performance expectations, measures, and data collected, as well as to determine how they 

analyze and validate these data. 

We also analyzed the timeliness, completeness, and reliability of the performance data SHIPs 

submit. To do this, we collected data submissions for the SHIP activities from April 2003 

through September 2004 (three reporting periods), the latest complete data available at the time 

of our request. We also compared the SHIP data submissions to the validated, final performance 

report, called the National Performance Report (NPR), to determine if flawed data were being 

included in the final NPR that CMS uses to assess SHIP performance. 

Current Status of CMS’s Ability to Assess SHIP Performance 

CMS’s Performance Assessment Mechanisms 

Performance Assessment Approach. CMS’s authority to assess the SHIP performance is 

outlined in 42 CFR § 403.502,7 which states “CMS awards grants to States subject to the 

availability of funds and, if applicable, subject to the satisfactory progress in the State’s project 

during the preceding grant period.” The performance expectations for determining whether 

satisfactory progress is made are specified in the grant announcement as are the terms and 

conditions included in the notice of grant award sent to each State. Further, SHIPs have the 

following reporting requirements contained in statute:8 (1) SHIPs must demonstrate to the 

Secretary via CMS the ability to provide counseling and assistance,9 and (2) States receiving the 

SHIP grants must submit at least one annual report to CMS and any additional reports as CMS 

may prescribe in the notice of grant award.10 

Based on these authorities, in the FY 2005 grant announcement, CMS established performance 

expectations for the first time. Prior to the establishment of these expectations, there was no 

formal process to assess the SHIPs’ performance. The four performance expectations outlined in 

the grant announcement are: 

(1) Provide counseling to a greater number of individual beneficiaries unable to access other 

channels of information or needing and preferring locally based individual counseling 

services. 

(2) Increase targeted outreach in order to provide access to counseling to low-income, dualeligible, and hard-to-reach populations. 

(3) Increase and enhance the counselor workforce and equip it to be proficient. 

(4) Increase local and State participation in the CMS education and communication 

activities, thus enhancing communication between CMS and SHIPs to assure that the 

SHIP counselors are equipped to respond to a rapidly changing counseling environment 

7

 Complete SHIP requirements mandated by law and required in regulation can be found in Public Law 

101-508 §§ 4359 through 4360 and 42 CFR parts 403.500 through 403.512. 8

 Omnibus Budget Reconciliation Act of 1990. 9 Required by section 4360 of Public Law 101-508. 10 Required by 42 CFR § 403.510. 
Page 4 – Abby Block 

and to provide CMS with information about the support and resources that SHIPs need to 

provide accurate and reliable counseling services. 11 

For each of the four performance expectations, CMS lists between 4 to 10 performance 

measures, totaling 26, that it “may consider”12 in analyzing the SHIP progress. Eight of the 

twenty-six performance measures require SHIPs to provide descriptive information. The 

remaining 18 measures are linked to performance data. See the attachment for a complete list of 

the SHIP performance measures. 

Of the 18 performance measures linked to performance data, only 5 incorporate specific 

performance targets. For example, to meet the performance expectation of enhancing the 

counselor workforce, one performance measure stipulates that all the SHIP counselors were to 

have been trained on the new benefits created by the MMA by July 31, 2005. CMS has not 

similarly delineated specific performance targets in the remaining 13 performance measures. For 

example, other measures that may be used to assess whether SHIPs meet the expectation of 

enhancing the counselor workforce include the number of training hours and the number of 

counseling hours and sessions per counselor. There are no specific numerical targets for either 

of these measures. CMS has not set a target amount of training hours for SHIPs to strive 

towards. Nor has CMS provided the target amount of counseling hours or sessions they expect 

from each counselor. Instead, it is up to SHIPs to infer, from the wording of the performance 

expectation, that the goal is to demonstrate an increase in these activities. 

In addition to these established performance measures, CMS is currently working collaboratively 

with the SHIP Performance Assessment Workgroup and the SHIP Directors to focus more on 

performance outcomes and benchmarking. At the SHIP National Directors Conference in May 

2005, the CMS staff indicated that the ultimate goal is to design and implement a comprehensive 

SHIP performance assessment system that provides an overall assessment for each SHIP, 

including a process to identify performance improvement needs and best practices. To do this, 

CMS has stated plans to produce a range of performance measurement reports for itself and 

SHIPs. For example, the CMS staff advised us of and presented information on their plans to 

group similar SHIPs and rank and compare their performance to established benchmarks. 

Further, CMS reports that the CMS staff are conducting regular telephone surveys of the SHIP 

Directors to assess their ability to recruit, train, and manage volunteers; to identify resources to 

provide outreach, education, and enrollment assistance; and to respond to the increased need for 

the SHIP services. As part of this effort, CMS is sharing best practice information among SHIPs 

to help their directors learn from their colleagues. 

Collection of Performance Data. In 2002, CMS developed three standardized forms for the 

collection of performance data: (1) Client Contact, (2) Public and Media (PAM), and 

(3) Resource Report. Data from these forms are compiled into the NPR and used to monitor 

SHIP performance. Prior to this, the SHIP data submissions were not standardized, data 

11 FY 2005 grant announcement, pp. 8-11. 12 FY 2005 grant announcement, p. 9.
Page 5 – Abby Block 

collection definitions differed among the SHIPs, and data were only collected in the aggregate. 

CMS began receiving standardized data from these three new forms in 2004. 

The Client Contact form collects data about individual counselor-to-client encounters. The PAM 

form collects information about the SHIP outreach activities. The Client Contact and the PAM 

data are submitted quarterly and can be submitted by the encounter (specific information 

regarding an individual contact with the beneficiary or individual presentations) or in the 

aggregate (total number of encounters or presentations). The Resource Report form collects data 

about the SHIP volunteers, staff, and training. These submissions are received in the aggregate 

and are submitted semiannually. The data contained in these three reports can be submitted 

through SHIPtalk.org, a Web site for SHIP NPR reporting. SHIPs can also submit data through 

other means. SHIPs may use State-developed, proprietary systems for the collection and 

submission of data, or they can use a Microsoft Access-based data entry tool to submit Client 

Contact data. 

In July 2005, the three data collection forms were revised. The increased focus on the SHIP 

performance under the MMA necessitated changes to these forms to enable CMS to capture 

more data relevant to the new drug benefit. Specifically, the forms now capture data on 

educating, counseling, and enrolling beneficiaries in drug plans. 

Additionally, CMS began requiring complete reporting by including data reporting as part of the 

terms and conditions of the FY 2005 grant award and by instructing SHIPs to report data at the 

encounter-level through SHIPtalk.org. In the grant award, CMS defined full reporting as timely 

submission of each of the forms. However, CMS did not address the completeness of all data 

fields or the accuracy of data submitted on the forms. CMS also required more frequent 

reporting. CMS now requires SHIPs to submit Client Contact and PAM submissions quarterly 

instead of semiannually and Resource Reports, previously submitted annually, to be submitted 

semiannually. 

SHIP Submissions of Performance Data 

CMS reported that it has taken several steps to assist SHIPs in improving their reporting of data 

used to assess their performance. CMS reports that it provided technical assistance to SHIP 

directors and counselors, including training them on the use of the NPR forms, how to input data 

into SHIPtalk.org, and uploading data from proprietary State systems. Further, CMS encouraged 

full reporting in the FY 2005 grant award, as previously stated. 

Our analysis of the SHIP data submissions indicates that SHIP reporting has increased over the 

three reporting periods analyzed (April 2003 to September 2004). Specifically, SHIPs improved 

from the first reporting period to the third on each of the three forms: Client Contact 

submissions improved from 74 percent to 96 percent, and PAM submissions improved from 

74 percent to 83 percent. However, some of these submissions were received after the deadline 

and the totals were not included in the overall final NPR. Resource Report submissions 

improved from 70 percent to 72 percent. 
Page 6 – Abby Block 

Even with the increased reporting, SHIPs did not submit all reports at the encounter level in the 

reporting period ending September 2004. Two SHIPs did not submit Client Contact reports and 

an additional seven reported Client Contacts only at the aggregate level. The seven aggregate 

submissions accounted for 32 percent of all Client Contacts. Further, 9 of 54 SHIPs did not 

submit PAM reports detailing their public and media activities for this same period. These nine 

SHIPs accounted for almost 30 percent of Client Contacts nationwide during this reporting 

period. Lastly, during this reporting period, 15 SHIPs did not submit Resource Reports detailing 

the number of volunteer and paid counselors at each SHIP and the training they received. CMS 

reports that achieving full reporting is challenging due to the fact that SHIPs rely on a primarily 

volunteer staff who are motivated by providing services and less motivated to report data. 

The SHIP performance data submitted to CMS are not comprehensively validated for accuracy. 

Data that are submitted through SHIPtalk.org are subject to automatic computer edits to prevent 

nonsense entries and ensure the completeness of key fields. However, data that are not submitted 

through SHIPtalk.org do not receive this type of systematic data cleansing; CMS has exempted 3 

States, accounting for 22 percent of all Client Contacts, from using SHIPtalk.org. CMS’s data 

contractor conducts additional cursory reviews for completeness and accuracy of Client Contact 

data submitted via SHIPtalk.org, but not on PAM or Resource Report data. Finally, neither CMS 

nor its contractor validate any of the data onsite by reviewing source documents or assessing the 

procedures used to input data. 

Implications Regarding CMS’s Ability to Assess the SHIP Performance 

Because CMS has yet to establish specific performance benchmarks or targets, it will be 

difficult to hold SHIPs accountable for their use of increased funding through the MMA. 

Given their additional funding and responsibilities in relation to the new drug benefit, it is 

increasingly important that the SHIP performance be adequately monitored. Providing SHIPs 

with known, specific performance targets will be essential to ensure that they are effectively able 

to assist beneficiaries enrolling in the new drug benefit. Without specific performance targets, it 

will be difficult for CMS to assess the extent to which SHIPs are meeting performance 

expectations and to hold them accountable for making improvements commensurate with their 

increased funding. It will also be difficult for SHIPs to plan their activities and the use of their 

resources to achieve the established expectations. 

For example, CMS requests that SHIPs report the total number of beneficiaries served by 

individual counseling contacts in order to assess whether they are meeting the expectation to 

provide counseling to a greater number of individual beneficiaries. It also requests that SHIPs 

report the number of beneficiaries reached at public events in order to determine their success in 

increasing targeted outreach to low-income, dual-eligible and hard-to-reach populations. In both 

cases, the implied performance target for SHIPs is merely to “increase” their activities in these 

areas. Thus, any increase, regardless of magnitude, over the previous year’s performance would 

be considered successfully meeting the performance expectation. 
Page 7 – Abby Block 

CMS publicly announced a target of reaching as many as 5 million Medicare beneficiaries 

through SHIPs in FY 2005.13 This translates into reaching approximately one in eight Medicare 

beneficiaries. However, even in establishing this informal performance target, CMS did not 

define which activities are included, or designate what percentage of the people to be reached 

should be served through personalized counseling. Additionally, SHIPs have not been given 

individual targets or the percentage increases in beneficiaries served needed to achieve this goal. 

Incomplete and inaccurate data may hinder CMS in developing and implementing a 

meaningful performance assessment system. 

Complete, accurate, encounter-level SHIP data submissions are necessary to determine if SHIPs 

are meeting the established performance expectations. Incomplete or inaccurate data in the final 

NPR will result in an inaccurate assessment of the SHIP performance. For example, the final 

data submissions we reviewed for the reporting period ending September 2004 incorporated data 

that indicated one State had experienced a 6,269 percent increase in attendance at interactive 

presentations between FY 2003 and FY 2004. Data reported at the aggregate level rather than 

the encounter level will not provide the specificity needed to provide meaningful, performance 

feedback to individual SHIP sites. Providing SHIPs with meaningful, accurate feedback on their 

performance is essential to continuous quality improvement in SHIPs’ abilities to assist 

beneficiaries applying for the new drug benefit. 

CMS’s lack of complete and accurate data could also hinder its ability to establish the 

benchmarks CMS staff have indicated they would like to construct. It will also hinder them from 

establishing more specific performance targets. 

Suggestions for Improving CMS’s Ability to Assess the SHIP Performance 

CMS has made important strides in improving its ability to assess the SHIP performance and in 

working with SHIPs to improve reporting into this system. Our review indicates, however, that 

there are additional steps CMS should consider to further enhance its ability to accurately assess 

the SHIP performance. We make the following suggestions to augment CMS’s already existing 

plans to improve SHIP oversight. 

CMS should continue to pursue benchmarking and consider using it to establish specific 

performance targets. 

To improve its ability to meaningfully assess the SHIP performance, CMS could further pursue 

its stated intention to establish benchmarks. Once established, these benchmarks could be used 

to create specific performance targets for SHIPs. If specific targets are too difficult to establish, 

given data problems and the substantial flux the program is in due to implementation of the new 

drug benefit, targets could be expressed as an acceptable range of improvement. 

13 This target has been established through an official press release and public presentations by the CMS 

staff. 
Page 8 – Abby Block 

CMS could work with SHIPs to ensure complete and accurate data submission and could 

validate all submitted data. 

CMS could further clarify for SHIPs that full reporting means that all forms are submitted in a 

timely fashion, all fields are completed, and the data are accurate. Further, CMS could more 

systematically validate all performance data. CMS may want to consider incorporating spot 

checks of source documentation or conducting targeted onsite visits at SHIPs that are having 

problems meeting reporting requirements or not using SHIPtalk.org to validate data. 

We hope you find this information useful to CMS in its ongoing efforts to improve the SHIP 

performance. If you have any questions, please do not hesitate to call me, or one of your staff 

may contact Tricia Davis, Director, Medicare and Medicaid Branch, at 410-786-3143, or through 

e-mail (Tricia.Davis@oig.hhs.gov). 

Attachment 
Page 9 – Abby Block 

Attachment: Complete List of SHIP Performance Measures in FY 05 SHIP Grant 

Announcement 

Expectation Measurement Target 

Increase local Compliance with the NPR reporting standards 100% 

and State SHIP 

participation in 

CMS education 

State and local program registration on SHIPtalk.org 100% 

Attendance at trainings, conference calls, and the National Directors’ Conference Attendance at 3 

and communication 

activities 

Inventory of the SHIP publications Description 

Orders of the CMS publications Increase 

The percentage of counselors participating in training and fully trained on the MMA by 

July 31, 2005 100% 

The availability of Internet access in all local counseling centers and the percentage of 

volunteers who are trained on using Internet-based counseling and enrollment tools 

Full Internet access to materials and 

tools, no target for training 

volunteers 

The number and type of counseling hours and sessions per counselor Increase 

Increase and 

enhance the 

counselor workforce and equip it 

to be proficient 

The number of counselor hours per 1,000 beneficiaries in State Increase 

The use of quality assurance measures, including tools such as counselor recertification 

exams Description 

The number of training hours provided to counselors Increase 

A description of counselor workforce and support systems (including the use of 

volunteers and paid staff, staff and local program supervision and support systems, role 

of local/regional coordinators) 

Description 

A description of volunteer roles Description 

A description of targeted counselor recruitment efforts Description 

The demographic makeup of counseling staff as compared with the demographic 

makeup of the community served by the program Increase 

The time counselors spend counseling beneficiaries Increase 

The total number of beneficiaries served by individual counseling contacts Increase 

Provide counseling 

to a greater 

number of 

individual 

beneficiaries 

The percentage of Medicare beneficiaries in the State that are counseled by SHIPs Increase 

The percentage of one-on-one contacts as compared to other types of beneficiary 

contacts and services Increase 

Coverage of the SHIP services in the State by zip code Increase 

Number of beneficiaries enrolled in coverage programs and the savings those 

beneficiaries would realize through their enrollment Increase 

The budget and implementation strategy related to the amount of support for locally 

accessible, in-depth counseling services Description 

Increase targeted 

outreach in order 

Target audiences of public events Description 

Number and reach of public events conducted by the SHIP program Increase 

to provide access 

to counseling to 

low-income, 

dual-eligible, and 

hard-to-reach 

Demographics of beneficiaries served through counseling services, including an 

emphasis on low-income and a representative client base reflective of the ethnicity/race 

and disability status of the beneficiaries in the State served 

Increase efforts to be 

demographically representative of 

the Medicare population 

populations The SHIP program’s partnership development strategy and identification of the partners 

and targeted populations reached through partnerships Description 

Source: Office of Inspector General analysis of the FY 2005 SHIP grant announcement, 2005.