"- 
( t 
'''''C''.
DEPARTMENT OF HEALTH &. HUMAN SERVICES 
Office of Inspector General 
Washington , D. C. 20201 
MAR 1 6 2006 
TO:	 Abby Block, Director

Center for Beneficiary Choices

Centers for Medicare & Medicaid Services

FROM: 
Stuar Wright 

Deputy Inspector General

for Evaluation and Inspections

SUBJECT:	 Early Implementation of MM: Status Report of the State Health Insurance 
Assistace Programs (OEI-05-05-00190) 
This memorandum contain information regarding the Centers for Medicare & Medicaid 
Services' (CMS) ability to assess the State Health Insurance Assistace Program (SHIP) as 
SHIPs begin to assist beneficiaries interested in enrollng in the new Medicare drg benefit. Ths 
early implementation review provides a description of the curent status ofCMS' s oversight 
efforts regarding this program and contains implications for the futue, as well as some 
fuer 
suggestions for  improvements. This work also lays a baseline for futue Office of 
Inspector General work in the area. 
Background 
SHIP provides a "health insurance advisory service to assist Medicare eligible individuals with 
the receipt of services under the Medicare and Medicaid programs and other health insurance 
" 1 SHIs curently 
programs. operate in all 50 States, the District of Columbia, the Virgin 
Islands, Puerto Rico, and Guam. SHIPs are generally composed of volunteer networks in local 
communities, with over 1 300 local SHI offces around the countr supported in large par by 
over 10 000 volunteers. The SHIP counselors have approximately 1.8 milion beneficiar 
encounters a year through individualized couneling. Additional beneficiares are served though 
public and media events. 
The Medicare Drug, Improvement, and Modernation Act of 2003 (MMA) made comprehensive 
prescription drug coverage under Medicare Part D available to all 42 
milion Medicare 
beneficiaries. Beneficiaries generally have the option of enrollng either in a stad-alone 
prescription drg plan (PDP) and receiving all other Medicare benefits through fee-for-service or 
enrollng in a Medicare Advantage plan (MA plan) and receiving all Medicare benefits (including 
drg 
coverage) through managed care. To receive this benefit, Medicare beneficiaries select a 
prescription drg benefit package that best meets their needs based on differences in 
I Public Law 101-
508.  Page 2  Abby Block 
premiums, copayments, formularies, and pharmacy networks.2  Beneficiaries must select a plan 
by May 15, 2006, or face penalties in the form of higher premiums.   
Prior to the passage of the MMA, SHIP was a relatively small program with limited outlays of 
Federal funding. SHIPs provided information and counseling regarding Medicare and other 
insurance services, but did not assist beneficiaries in enrolling for benefits.  In response to the 
MMA, SHIPs have taken on a prominent role in conducting outreach and counseling 
beneficiaries on the new drug benefit.  In addition to providing information regarding the new 
benefit, CMS has instructed SHIPs to help Medicare beneficiaries enroll in the drug benefit.3  A 
CMS press release characterized SHIPs as the key foot soldiers in providing one-on-one 
assistance for Medicare beneficiaries to help them get the best health care.4  SHIPs are of 
particular importance because they are able to counsel the beneficiary, sometimes in face-to-face 
situations, about drug plans that will best meet their needs.  They are able to provide 
individualized assistance and have access to local information. 
To support the increased expectations to counsel and enroll beneficiaries in the new drug benefit, 
CMS increased SHIP funding.  In the past, CMS annually allocated $10 million in grants to 
SHIPs based on a formula stipulated by regulation.5  This allocation was increased to 
approximately $21 million in fiscal year (FY) 2004 and approximately $32 million in FY 2005.6 
Given the importance of SHIPs to the successful implementation of the new drug benefit, it is 
imperative that CMS has the tools available to assess the SHIP performance.  It is only by 
monitoring and assessing their performance that CMS will be able to ensure that they are 
effectively assisting beneficiaries and using the enhanced funding in an effective and efficient 
manner.   
Methodology 
To ascertain the current ability of CMS to measure the SHIPs performance during the early 
stages of MMA implementation, we reviewed CMSs mechanisms to collect and assess 
performance data and the performance data SHIPs submitted.  
We reviewed the FY 2005 SHIP grant announcement, the notice of grant award, and data 
collection forms to determine what performance expectations exist, the associated performance 
measures, and the data sources used to measure whether expectations are being met.  In addition, 
2 Beneficiaries dually eligible for Medicaid and Medicare and Medicare beneficiaries enrolled in managed 

care plans will have plans automatically assigned to them.  However, they have the option of selecting a 

different plan that better suits their needs.

3 FY 2005 grant announcement, p. 9.

4 CMS press release, April 2005. 

5 42 CFR  403.504.

6 The overall appropriation to SHIPs for FY 2005 was $31.7 million, of which approximately $1.4 million

was applied to the SHIP Resource Center, SHIPtalk.org, SHIP performance assessment, and the SHIP

National Conference, and approximately $1 million was used for SHIP Competitive Leadership Grants.  

This leaves the first $10 million in annual grants and the approximately $19.3 million under the new 

formula. 
 Page 3  Abby Block 
we conducted structured interviews with the CMS staff and their contractors to understand the 
performance expectations, measures, and data collected, as well as to determine how they 
analyze and validate these data.   
We also analyzed the timeliness, completeness, and reliability of the performance data SHIPs 
submit.  To do this, we collected data submissions for the SHIP activities from April 2003 
through September 2004 (three reporting periods), the latest complete data available at the time 
of our request. We also compared the SHIP data submissions to the validated, final performance 
report, called the National Performance Report (NPR), to determine if flawed data were being 
included in the final NPR that CMS uses to assess SHIP performance. 
Current Status of CMSs Ability to Assess SHIP Performance 
CMSs Performance Assessment Mechanisms 
Performance Assessment Approach. CMSs authority to assess the SHIP performance is 
outlined in 42 CFR  403.502,7 which states CMS awards grants to States subject to the 
availability of funds and, if applicable, subject to the satisfactory progress in the States project 
during the preceding grant period.  The performance expectations for determining whether 
satisfactory progress is made are specified in the grant announcement as are the terms and 
conditions included in the notice of grant award sent to each State.  Further, SHIPs have the 
following reporting requirements contained in statute:8  (1) SHIPs must demonstrate to the 
Secretary via CMS the ability to provide counseling and assistance,9 and (2) States receiving the 
SHIP grants must submit at least one annual report to CMS and any additional reports as CMS 
may prescribe in the notice of grant award.10 
Based on these authorities, in the FY 2005 grant announcement, CMS established performance 
expectations for the first time.  Prior to the establishment of these expectations, there was no 
formal process to assess the SHIPs performance.  The four performance expectations outlined in 
the grant announcement are:  
(1) Provide counseling to a greater number of individual beneficiaries unable to access other 
channels of information or needing and preferring locally based individual counseling 
services. 
(2) Increase targeted outreach in order to provide access to counseling to low-income, dual-
eligible, and hard-to-reach populations. 
(3) Increase and enhance the counselor workforce and equip it to be proficient. 
(4) Increase local and State participation in the CMS education and communication 
activities, thus enhancing communication between CMS and SHIPs to assure that the 
SHIP counselors are equipped to respond to a rapidly changing counseling environment 
7 Complete SHIP requirements mandated by law and required in regulation can be found in Public Law 

101-508  4359 through 4360 and 42 CFR parts 403.500 through 403.512. 

8 Omnibus Budget Reconciliation Act of 1990. 

9 Required by section 4360 of Public Law 101-508. 

10 Required by 42 CFR  403.510. 
 Page 4  Abby Block 
and to provide CMS with information about the support and resources that SHIPs need to 
provide accurate and reliable counseling services. 11 
For each of the four performance expectations, CMS lists between 4 to 10 performance 
measures, totaling 26, that it may consider12 in analyzing the SHIP progress. Eight of the 
twenty-six performance measures require SHIPs to provide descriptive information.  The 
remaining 18 measures are linked to performance data.  See the attachment for a complete list of 
the SHIP performance measures.   
Of the 18 performance measures linked to performance data, only 5 incorporate specific 
performance targets.  For example, to meet the performance expectation of enhancing the 
counselor workforce, one performance measure stipulates that all the SHIP counselors were to 
have been trained on the new benefits created by the MMA by July 31, 2005.  CMS has not 
similarly delineated specific performance targets in the remaining 13 performance measures.  For 
example, other measures that may be used to assess whether SHIPs meet the expectation of 
enhancing the counselor workforce include the number of training hours and the number of 
counseling hours and sessions per counselor.  There are no specific numerical targets for either 
of these measures.  CMS has not set a target amount of training hours for SHIPs to strive 
towards. Nor has CMS provided the target amount of counseling hours or sessions they expect 
from each counselor.  Instead, it is up to SHIPs to infer, from the wording of the performance 
expectation, that the goal is to demonstrate an increase in these activities.  
In addition to these established performance measures, CMS is currently working collaboratively 
with the SHIP Performance Assessment Workgroup and the SHIP Directors to focus more on 
performance outcomes and benchmarking.  At the SHIP National Directors Conference in May 
2005, the CMS staff indicated that the ultimate goal is to design and implement a comprehensive 
SHIP performance assessment system that provides an overall assessment for each SHIP, 
including a process to identify performance improvement needs and best practices.  To do this, 
CMS has stated plans to produce a range of performance measurement reports for itself and 
SHIPs. For example, the CMS staff advised us of and presented information on their plans to 
group similar SHIPs and rank and compare their performance to established benchmarks. 
Further, CMS reports that the CMS staff are conducting regular telephone surveys of the SHIP 
Directors to assess their ability to recruit, train, and manage volunteers; to identify resources to 
provide outreach, education, and enrollment assistance; and to respond to the increased need for 
the SHIP services. As part of this effort, CMS is sharing best practice information among SHIPs 
to help their directors learn from their colleagues.   
Collection of Performance Data. In 2002, CMS developed three standardized forms for the 
collection of performance data: (1) Client Contact, (2) Public and Media (PAM), and 
(3) Resource Report. Data from these forms are compiled into the NPR and used to monitor 
SHIP performance.  Prior to this, the SHIP data submissions were not standardized, data 
11 FY 2005 grant announcement, pp. 8-11. 

12 FY 2005 grant announcement, p. 9.
 Page 5  Abby Block 
collection definitions differed among the SHIPs, and data were only collected in the aggregate.  
CMS began receiving standardized data from these three new forms in 2004.    
The Client Contact form collects data about individual counselor-to-client encounters.  The PAM 
form collects information about the SHIP outreach activities.  The Client Contact and the PAM 
data are submitted quarterly and can be submitted by the encounter (specific information 
regarding an individual contact with the beneficiary or individual presentations) or in the 
aggregate (total number of encounters or presentations).  The Resource Report form collects data 
about the SHIP volunteers, staff, and training.  These submissions are received in the aggregate 
and are submitted semiannually.  The data contained in these three reports can be submitted 
through SHIPtalk.org, a Web site for SHIP NPR reporting.  SHIPs can also submit data through 
other means.  SHIPs may use State-developed, proprietary systems for the collection and 
submission of data, or they can use a Microsoft Access-based data entry tool to submit Client 
Contact data. 
In July 2005, the three data collection forms were revised.  The increased focus on the SHIP 
performance under the MMA necessitated changes to these forms to enable CMS to capture 
more data relevant to the new drug benefit.  Specifically, the forms now capture data on 
educating, counseling, and enrolling beneficiaries in drug plans.   
Additionally, CMS began requiring complete reporting by including data reporting as part of the 
terms and conditions of the FY 2005 grant award and by instructing SHIPs to report data at the 
encounter-level through SHIPtalk.org.  In the grant award, CMS defined full reporting as timely 
submission of each of the forms.  However, CMS did not address the completeness of all data 
fields or the accuracy of data submitted on the forms.  CMS also required more frequent 
reporting. CMS now requires SHIPs to submit Client Contact and PAM submissions quarterly 
instead of semiannually and Resource Reports, previously submitted annually, to be submitted 
semiannually.    
SHIP Submissions of Performance Data 
CMS reported that it has taken several steps to assist SHIPs in improving their reporting of data 
used to assess their performance.  CMS reports that it provided technical assistance to SHIP 
directors and counselors, including training them on the use of the NPR forms, how to input data 
into SHIPtalk.org, and uploading data from proprietary State systems.  Further, CMS encouraged 
full reporting in the FY 2005 grant award, as previously stated.   
Our analysis of the SHIP data submissions indicates that SHIP reporting has increased over the 
three reporting periods analyzed (April 2003 to September 2004).  Specifically, SHIPs improved 
from the first reporting period to the third on each of the three forms:  Client Contact 
submissions improved from 74 percent to 96 percent, and PAM submissions improved from 
74 percent to 83 percent. However, some of these submissions were received after the deadline 
and the totals were not included in the overall final NPR.  Resource Report submissions 
improved from 70 percent to 72 percent.      Page 6  Abby Block 
Even with the increased reporting, SHIPs did not submit all reports at the encounter level in the 
reporting period ending September 2004.  Two SHIPs did not submit Client Contact reports and 
an additional seven reported Client Contacts only at the aggregate level.  The seven aggregate 
submissions accounted for 32 percent of all Client Contacts.  Further, 9 of 54 SHIPs did not 
submit PAM reports detailing their public and media activities for this same period.  These nine 
SHIPs accounted for almost 30 percent of Client Contacts nationwide during this reporting 
period. Lastly, during this reporting period, 15 SHIPs did not submit Resource Reports detailing 
the number of volunteer and paid counselors at each SHIP and the training they received.  CMS 
reports that achieving full reporting is challenging due to the fact that SHIPs rely on a primarily 
volunteer staff who are motivated by providing services and less motivated to report data.    
The SHIP performance data submitted to CMS are not comprehensively validated for accuracy.  
Data that are submitted through SHIPtalk.org are subject to automatic computer edits to prevent 
nonsense entries and ensure the completeness of key fields.  However, data that are not submitted 
through SHIPtalk.org do not receive this type of systematic data cleansing; CMS has exempted 3 
States, accounting for 22 percent of all Client Contacts, from using SHIPtalk.org.  CMSs data 
contractor conducts additional cursory reviews for completeness and accuracy of Client Contact 
data submitted via SHIPtalk.org, but not on PAM or Resource Report data.  Finally, neither CMS 
nor its contractor validate any of the data onsite by reviewing source documents or assessing the 
procedures used to input data. 
Implications Regarding CMSs Ability to Assess the SHIP Performance 
Because CMS has yet to establish specific performance benchmarks or targets, it will be 
difficult to hold SHIPs accountable for their use of increased funding through the MMA. 
Given their additional funding and responsibilities in relation to the new drug benefit, it is 
increasingly important that the SHIP performance be adequately monitored.  Providing SHIPs 
with known, specific performance targets will be essential to ensure that they are effectively able 
to assist beneficiaries enrolling in the new drug benefit.  Without specific performance targets, it 
will be difficult for CMS to assess the extent to which SHIPs are meeting performance 
expectations and to hold them accountable for making improvements commensurate with their 
increased funding. It will also be difficult for SHIPs to plan their activities and the use of their 
resources to achieve the established expectations. 
For example, CMS requests that SHIPs report the total number of beneficiaries served by 
individual counseling contacts in order to assess whether they are meeting the expectation to 
provide counseling to a greater number of individual beneficiaries. It also requests that SHIPs 
report the number of beneficiaries reached at public events in order to determine their success in 
increasing targeted outreach to low-income, dual-eligible and hard-to-reach populations.  In both 
cases, the implied performance target for SHIPs is merely to increase their activities in these 
areas.  Thus, any increase, regardless of magnitude, over the previous years performance would 
be considered successfully meeting the performance expectation.  Page 7  Abby Block 
CMS publicly announced a target of reaching as many as 5 million Medicare beneficiaries 
through SHIPs in FY 2005.13  This translates into reaching approximately one in eight Medicare 
beneficiaries. However, even in establishing this informal performance target, CMS did not 
define which activities are included, or designate what percentage of the people to be reached 
should be served through personalized counseling.  Additionally, SHIPs have not been given 
individual targets or the percentage increases in beneficiaries served needed to achieve this goal.  
Incomplete and inaccurate data may hinder CMS in developing and implementing a 
meaningful performance assessment system. 
Complete, accurate, encounter-level SHIP data submissions are necessary to determine if SHIPs 
are meeting the established performance expectations.  Incomplete or inaccurate data in the final 
NPR will result in an inaccurate assessment of the SHIP performance.  For example, the final 
data submissions we reviewed for the reporting period ending September 2004 incorporated data 
that indicated one State had experienced a 6,269 percent increase in attendance at interactive 
presentations between FY 2003 and FY 2004.  Data reported at the aggregate level rather than 
the encounter level will not provide the specificity needed to provide meaningful, performance 
feedback to individual SHIP sites.  Providing SHIPs with meaningful, accurate feedback on their 
performance is essential to continuous quality improvement in SHIPs abilities to assist 
beneficiaries applying for the new drug benefit. 
CMSs lack of complete and accurate data could also hinder its ability to establish the 
benchmarks CMS staff have indicated they would like to construct.  It will also hinder them from 
establishing more specific performance targets.   
Suggestions for Improving CMSs Ability to Assess the SHIP Performance 
CMS has made important strides in improving its ability to assess the SHIP performance and in 
working with SHIPs to improve reporting into this system.  Our review indicates, however, that 
there are additional steps CMS should consider to further enhance its ability to accurately assess 
the SHIP performance.  We make the following suggestions to augment CMSs already existing 
plans to improve SHIP oversight. 
CMS should continue to pursue benchmarking and consider using it to establish specific 
performance targets. 
To improve its ability to meaningfully assess the SHIP performance, CMS could further pursue 
its stated intention to establish benchmarks.  Once established, these benchmarks could be used 
to create specific performance targets for SHIPs.  If specific targets are too difficult to establish, 
given data problems and the substantial flux the program is in due to implementation of the new 
drug benefit, targets could be expressed as an acceptable range of improvement.  
13 This target has been established through an official press release and public presentations by the CMS 
staff.  Page 8  Abby Block 
CMS could work with SHIPs to ensure complete and accurate data submission and could 
validate all submitted data. 
CMS could further clarify for SHIPs that full reporting means that all forms are submitted in a 
timely fashion, all fields are completed, and the data are accurate.  Further, CMS could more 
systematically validate all performance data.  CMS may want to consider incorporating spot 
checks of source documentation or conducting targeted onsite visits at SHIPs that are having 
problems meeting reporting requirements or not using SHIPtalk.org to validate data.   
We hope you find this information useful to CMS in its ongoing efforts to improve the SHIP 
performance.  If you have any questions, please do not hesitate to call me, or one of your staff 
may contact Tricia Davis, Director, Medicare and Medicaid Branch, at 410-786-3143, or through 
e-mail (Tricia.Davis@oig.hhs.gov).   
Attachment  Page 9  Abby Block 
Attachment:  Complete List of SHIP Performance Measures in FY 05 SHIP Grant 
Announcement 
Expectation  Measurement  Target 
Compliance with the NPR reporting standards  100% 
Increase local 
and State SHIP   State and local program registration on SHIPtalk.org  100% 
participation in 
CMS education   Attendance at trainings, conference calls, and the National Directors Conference  Attendance at 3 
and communication Inventory of the SHIP publications  Description 
activities 
Orders of the CMS publications  Increase 
The percentage of counselors participating in training and fully trained on the MMA by 
100% 
July 31, 2005 
Full Internet access to materials and 
The availability of Internet access in all local counseling centers and the percentage of 
tools, no target for training 
volunteers who are trained on using Internet-based counseling and enrollment tools 
volunteers  
The number and type of counseling hours and sessions per counselor  Increase 
The number of counselor hours per 1,000 beneficiaries in State  Increase 
Increase and 
enhance the  The use of quality assurance measures, including tools such as counselor recertification  Description 
counselor work- exams 
force and equip it  
The number of training hours provided to counselors  Increase 
to be proficient 
A description of counselor workforce and support systems (including the use of 
volunteers and paid staff, staff and local program supervision and support systems, role  Description 
of local/regional coordinators) 
A description of volunteer roles  Description 
A description of targeted counselor recruitment efforts  Description 
The demographic makeup of counseling staff as compared with the demographic 
Increase 
makeup of the community served by the program 
The time counselors spend counseling beneficiaries  Increase 
The total number of beneficiaries served by individual counseling contacts  Increase 
The percentage of Medicare beneficiaries in the State that are counseled by SHIPs  Increase 
Provide counseling  
to a greater  The percentage of one-on-one contacts as compared to other types of beneficiary 
Increase 
number of   contacts and services 
individual  
beneficiaries  Coverage of the SHIP services in the State by zip code  Increase 
Number of beneficiaries enrolled in coverage programs and the savings those 
Increase 
beneficiaries would realize through their enrollment 
The budget and implementation strategy related to the amount of support for locally 
Description 
accessible, in-depth counseling services 
Target audiences of public events  Description 
Increase targeted 
Number and reach of public events conducted by the SHIP program  Increase 
outreach in order 
to provide access 
to counseling to  Demographics of beneficiaries served through counseling services, including an  Increase efforts to be  
low-income,   emphasis on low-income and a representative client base reflective of the ethnicity/race  demographically representative of 
dual-eligible, and  and disability status of the beneficiaries in the State served  the Medicare population 
hard-to-reach 
populations  The SHIP programs partnership development strategy and identification of the partners 
Description 
and targeted populations reached through partnerships 
Source:  Office of Inspector General analysis of the FY 2005 SHIP grant announcement, 2005.