Measuring the Effects of Reminder Postcards in the Occupational 

Employment Statistics Survey 

Carrie K. Jones 

US Bureau of Labor Statistics 

Division of Occupational Employment Statistics 

Jones.carrie@bls.gov 

 

Abstract 

The Occupational Employment Statistics (OES) Survey is a voluntary establishment survey that produces occupational 

employment and wage estimates for the U.S., each State, and select U. S. Territories and metropolitan areas within States. 

The survey is conducted primarily via mail in cooperation with State Employment Security Agencies by the Bureau of Labor 

Statistics. The OES sample is 1.2 million establishments collected over three years in semi-annual panels. Each panel has an 

initial mailing followed by three follow-up mailings at four-week intervals. In May of 2007, the United States Postal Service 

(USPS) initiated a substantial postage increase and was also granted authority to automatically increase postage rates 

annually. OES absorbed a 30 percent postage increase. Concurrently, examination of data files indicated that many 

respondents received follow-up solicitation packages after they had sent in their completed survey. Faced with increasing 

postage costs, many unnecessary “overlap” mailings, and a shrinking budget OES decided to test using reminder postcards 

during the first follow-up mailings in Fall 2007 and again in Spring 2008. The Fall test featured substitution of a postcard for 

a survey packet. The Spring test featured a postcard in addition to a survey packet with the idea that immediate increases in 

response would reduce the number of follow-up mailings. Six States were randomly selected for a two-panel test and the 

samples for each State were split into two groups: a test group that received the postcard and a control group that received a 

regular mailing. Chi-square tests are used to evaluate the effect of the postcard experiment on response rates, and a cost 

analysis is presented, as well as results from a qualitative assessment questionnaire completed by each test State. 

Background 

The OES survey collects data from business respondents by mailing out survey packets once a month over a four-month 

period. OES is collected on a semi-annual schedule in the fall and spring of each year, and mailouts are conducted in 

November, December, January, and February for the fall panel and in May, June, July, and August for the spring panel. OES 

State personnel have proposed that BLS reconsider this mailout schedule. States have reported that with the current schedule, 

the second mailing (particularly the December mailout) conflicts with respondent work and holiday schedules and is not 

productive. In addition, examination of the return mail patterns shows evidence of considerable overlap between the first and 

second mailings resulting in numerous unnecessary mailings which are costly and bothersome to the respondents that have 

already mailed their survey form back. OES is interested in eliminating this overlap to the greatest extent possible. OES is 

also interested in finding efficiencies that will help off-set rising postage increases1

. Therefore, OES tested the feasibility 

and success of sending a reminder postcard for the second mailing rather than sending a survey packet. 

 

BLS asked for selected States2

 to agree to test the use of postcard at the time of the second mailing by participating in a 

randomized split-panel test where half of each test State’s sample would receive the postcard. The other half of the sample 

would be the control group and receive a regular survey packet. Results from the two groups are compared. The test took 

1

 In 2007, the United States Postal Service implemented a comprehensive overhaul of their postage rates, including the flat-rate envelopes 

OES uses. This resulted in a postage increase of over $300,000 for the year for the OES survey, an increase of over 30 percent. 

Legislation allows the USPS to continue postage increases based on change in the Consumer Price Index. 

2

 Due to State concerns that there are behavioral and operation differences among the different sized States, the States were 

grouped into three categories, small, medium, and large, based on the number of establishments operating in their 

jurisdiction. From these groups the States were randomly chosen to participate. If a State declined to participate an alternate 

was randomly selected. The samples within each State were randomly split between control and test groups. 
place over two panels: Fall 2007 and Spring 2008. During the Fall 2007 panel the test group received a postcard in place of 

the second mailing. In the Spring 2008 panel the test group received a postcard in addition to a complete second mailing. 

 

The OES postcard test was primarily developed as a way to reduce the survey’s postage costs without causing a reduction in 

response rates. The idea of sending it during the second mailout in December was based on State input that the respondents 

probably still have the first survey packet and that many respondents are on vacation during December. Therefore a simple 

reminder would be more effective and economical than sending an entire survey packet. 

Six States were randomly chosen with backups should any State not want to participate. Several States were approached to 

participate but declined due to concerns about a possible negative impact on their response rates. Concerns that were voiced 

included: 

 Too many phone calls from establishments 

 Fear that establishments had thrown away the first survey form and would overwhelm the State with calls 

requesting new ones (not having enough forms available, added work of sending out new forms, and postage 

costs) 

 Overall increased workload impact 

Methodology 

Part 1 – Fall 2007 Panel 

The Fall 2007 panel began in September 2007 with the first mailing commencing in November 2007. There were mailings in 

December, January, and February. The final group of test States were Alaska, Colorado, Idaho, Maryland, Nebraska, and 

Pennsylvania.3

For part 1 of the postcard test, there were four mailings but the second mailing for the test group was replaced with a 

postcard. The test States sent their second mailing address files to the National Office to be split into two groups: those to

receive the regular mailout package (control group) and those to be sent only a reminder postcard (postcard group). The 

States were not told which establishments were selected for each group until after all mailings, one through four, were 

complete. 

The States sent in their survey databases to BLS in April 2008, May 2008, and July 2008. These databases provide a 

snapshot of how the survey is progressing over time and are used along with the mail address files to evaluate the experiment. 

The May and July databases are normal deliverables called interim and final files respectively. The April database was an

extra deliverable that we called a “pre-interim” deliverable. 

Part 2 – Spring 2008 Panel 

The Spring 2008 panel began in March 2008 with the first mailing commencing in May 2008. There were mailings in May, 

June, July, and August. The test States were again Alaska, Colorado, Idaho, Maryland, Nebraska, and Pennsylvania. 

For part 2 of the postcard test, there were four regular mailings for both the control and test groups with an additional 

postcard mailing for the test group. The postcards were sent between the first and second packets mailings. The National 

Office split the second mailing address files into two groups: those to receive the regular mailout packages (control group) 

and those to be sent an additional reminder postcard (postcard group). The States were not told which establishments were 

selected for each group until after all mailings were complete. The States sent in interim and final survey databases in 

November 2008 and January 2009 respectively. 

 

3

 Colorado was monitored closely due to erroneous contact information on their postcards and a system error that truncated 

some of the addresses printed on the postcard. 
Results 

 

 

Establishment Response Rates 

Table 1 shows the progress of the unadjusted early response rates for units during the Fall 2007 panel. The rates are based on

mailout files and the first survey file sent in from the test States. These rates are not final response rates; they have not been 

adjusted for units that were later determined to be out of scope or out of business, were later inactivated, or were not present 

in the later mailing files. 

The response rate for the postcard group was 7.3 percent lower than the response rate for the control group following the 

second mailout. By the first survey database deliverable(Deliverable 1) in April, the gap closes to 3.1 percent. 

Table 1. Unadjusted Response Rates Based on Mailout Files – November Panel 

Record Counts/Non-Response (Unadjusted) Percent Difference 

State Group Mailing 

1 

Mailing 

2 

Mailing 

3 Mailing 4 Deliverable 1 

M2–

M3 

M2-

M4 

M2- 

Del 1 Cumulative 

Alaska Ctl 247 179 141 93 27.5% 42.9% 62.3% 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Test 281 218 172 115 22.4% 38.8% 59.1% 

Total 728 528 397 313 208 24.8% 40.7% 60.6% 71.4% 

 

 

 

 

 

 

 

 

 

 

 

 

Differences -5.1% -4.1% -3.3% 

Colorado Ctl 1324 1005 778 725 24.1% 41.2% 45.2% 

Test 1210 1052 882 750 13.1% 27.1% 38.0% 

Total 3384 2534 2057 1660 1475 18.8% 34.5% 41.8% 56.4% 

Differences -

11.0% 

-

14.1% -7.2% 

Idaho Ctl 462 349 324 154 24.5% 29.9% 66.7% 

Test 465 409 372 168 12.0% 20.0% 63.9% 

Total 1239 927 758 696 322 18.2% 24.9% 65.3% 74.0% 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Differences -

12.4% -9.9% -2.8% 

Maryland Ctl 1138 915 707 495 19.6% 37.9% 56.5% 

Test 1376 1232 945 631 10.5% 31.3% 54.1% 

Total 3103 2514 2147 1652 1126 14.6% 34.3% 55.2% 63.7% 

Differences -9.1% -6.6% -2.4% 

Nebraska Ctl 529 328 140 71 38.0% 73.5% 86.6% 

 

 

Test 516 326 129 73 36.8% 75.0% 85.9% 

Total 1440 1045 654 269 144 37.4% 74.3% 86.2% 90.0% 

Differences -1.2% 1.5% -0.7% 

Pennsylvania Ctl 3043 2496 2082 1861 18.0% 31.6% 38.8% 

 

 

 

 

 

 

 

 

 

 

 

 

Test 3125 2721 2315 1992 12.9% 25.9% 36.3% 

Total 8239 6168 5217 4397 3853 15.4% 28.7% 37.5% 53.2% 

Differences 

 

 

 

 

 

-5.0% -5.7% -2.6% 

All Test 

States Ctl 6743 5272 4172 3399 21.8% 38.1% 49.6% 

Test 6973 5958 4815 3729 14.6% 30.9% 46.5% 

Total 18133 13716 11230 8987 7128 18.1% 34.5% 48.0% 60.7% 

Differences -7.3% -7.2% -3.1% 
Tables 2 and 3 show the adjusted response rates for units for the control and test groups. These rates are adjusted for out of

scope, out of business, and inactivated schedules.4

 Any units that were not part of the December base were also removed. 

As of the interim file, delivered in May 2008, the adjusted response rates for the control group were 59.14 percent while the 

test group had 58.15 percent, with a difference of 1.0 percent. The difference in response had narrowed to 0.17 percent by 

July when the final deliverable files were transmitted to BLS. 

 

 

 

 

 

 

 

 

 

 

 

 

Table 2. Adjusted Units Response Rates Based on the Interim File

Deliverable – Fall Panel 

State Group Count Total Percent Difference 

AK ctrl 144 233 61.80% -3.53%

AK test 148 254 58.27% 

CO ctrl 744 1208 61.59% 2.81%

CO test 738 1146 64.40% 

ID ctrl 251 366 68.58% 0.87%

ID test 275 396 69.44% 

MD ctrl 685 1077 63.60% -1.04%

MD test 814 1301 62.57% 

NE ctrl 452 509 88.80% -1.67%

NE test 440 505 87.13% 

PA ctrl 1366 2765 49.40% -2.61%

PA test 1322 2825 46.80% 

all ctrl 3642 6158 59.14% -1.00%

all test 3737 6427 58.15% 

Table 3. Adjusted Units Response Rates Based on the Final 

Deliverable – Fall Panel 

State Group Count Total Percent Difference 

AK ctrl 149 233 63.95% -4.50%

AK test 151 254 59.45% 

CO ctrl 879 1203 73.07% 2.64%

CO test 860 1136 75.70% 

ID ctrl 274 359 76.32% 0.52%

ID test 302 393 76.84% 

MD ctrl 773 1073 72.04% -0.20%

MD test 931 1296 71.84% 

NE ctrl 452 510 88.63% -1.50%

NE test 440 505 87.13% 

PA ctrl 1895 2749 68.93% -0.61%

PA test 1915 2803 68.32% 

all ctrl 4422 6127 72.17% -0.17%

all test 4599 6387 72.01% 

Part 2 – Spring 2008 Panel 

Table 4 shows the progress of the unadjusted early response rates for units for the Spring 2008 panel. The rates are based on 

central printer mailout files only. These rates are not final response rates; they have not been adjusted for units that were later 

determined to be out of scope or out of business, were later inactivated, or were not present in the second mailing files. 

 

4

 Any units that were not part of the December base were removed from the analysis. 
The response rate for the postcard group was 1.5 percent higher than the response rate for the control group following the 

second mailing. By the fourth mailing, the gap widens to 2.9 percent.

 

 

Table 4. Unadjusted Response Rates Based on Mailout Files – Spring Panel 

Record Counts/Non-Response (Unadjusted) Percent Difference 

State Group Mailing1 Mailing2 Mailing3 Mailing4 M2-M3 M3-M4 M2-M4 Cumulative 

Alaska Ctl 230 169 138 26.5% 18.3% 40.0% 

 

 

 

 

 

Test 250 160 125 36.0% 21.9% 50.0% 

Total 665 480 329 263 31.5% 20.1% 45.2% 60.5% 

Differences 9.5% 3.5% 10.0% 

Colorado Ctl 1013 755 631 25.5% 16.4% 37.7% 

 

 

 

 

 

 

Test 1061 777 632 26.8% 18.7% 40.4% 

Total 2817 2074 1532 1263 26.1% 17.6% 39.1% 55.2% 

Differences 1.3% 2.2% 2.7% 

Idaho Ctl 385 316 258 17.9% 18.4% 33.0% 

Test 397 313 248 21.2% 20.8% 37.5% 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Total 1236 782 629 506 19.6% 19.6% 35.3% 59.1% 

Differences 3.2% 2.4% 4.5% 

Maryland Ctl 1068 796 641 25.5% 19.5% 40.0% 

Test 1296 976 751 24.7% 23.1% 42.1% 

Total 3104 2364 1772 1392 25.0% 21.4% 41.1% 55.2% 

Differences -0.8% 3.6% 2.1% 

Nebraska Ctl 430 247 110 42.6% 55.5% 74.4% 

Test 388 216 75 44.3% 65.3% 80.7% 

Total 1413 818 463 185 43.4% 60.0% 77.4% 86.9% 

Differences 1.8% 9.8% 6.3% 

Pennsylvania Ctl 2323 1839 1646 20.8% 10.5% 29.1% 

 

 

 

 

 

 

 

 

 

Test 2273 1758 1562 22.7% 11.1% 31.3% 

Total 6409 4596 3597 3208 21.7% 10.8% 30.2% 49.9% 

Differences 1.8% 0.7% 2.1% 

All Test States Ctl 5449 4122 3424 24.4% 16.9% 37.2% 

Test 5665 4200 3393 25.9% 19.2% 40.1% 

Total 15644 11114 8322 6817 25.1% 18.1% 38.7% 56.4% 

Differences 1.5% 2.3% 2.9% 

Tables 5 and 6 show the Spring panel adjusted response rates for units for the control and test groups. These rates are 

adjusted for out of scope, out of business, and inactivated schedules. As with the Fall panel, any units that were not part of

the June base were also removed. As of the interim file the adjusted response rates for the control group were 62.45 percent 

while the test group had 62.36 percent, with a difference of 0.09 percent. The difference in response widened to 0.5 percent, 

this time with the test group being higher, by the final file deliverable in January 2009. 
Table 5. Adjusted Units Response Rates Based on the Interim File

Deliverable – Spring Panel 

State Group Count Total Percent Difference 

AK ctrl 131 196 66.84% 3.16%

AK test 161 230 70.00% 

 

 

 

 

 

 

 

 

 

 

CO ctrl 574 963 59.61% 1.03%

CO test 607 1001 60.64% 

ID ctrl 223 349 63.90% 2.96%

ID test 232 347 66.86% 

MD ctrl 609 994 61.27% -1.53%

MD test 721 1207 59.73% 

NE ctrl 357 411 86.86% 3.32%

NE test 340 377 90.19% 

PA ctrl 1304 2208 59.06% -0.83%

PA test 1253 2152 58.22% 

all ctrl 3198 5121 62.45% -0.09%

all test 3314 5314 62.36% 

Table 6. Adjusted Units Response Rates Based on the Final 

Deliverable – Spring Panel 

State Group Count Total Percent Difference 

AK ctrl 131 196 66.84% 3.60%

AK test 162 230 70.43% 

CO ctrl 676 962 70.27% 1.09%

CO test 710 995 71.36% 

ID ctrl 252 343 73.47% 2.03%

ID test 262 347 75.50% 

MD ctrl 712 982 72.51% 0.19%

MD test 860 1183 72.70% 

NE ctrl 357 411 86.86% 3.59%

NE test 341 377 90.45% 

PA ctrl 1520 2195 69.25% -0.55%

PA test 1462 2128 68.70% 

all ctrl 3648 5089 71.68% 0.50%

all test 3797 5260 72.19% 

Employment Response Rates 

In addition to calculating the response rate based on the number of establishments that respond, OES also calculates response 

rates based the number of employees found in the establishment. Tables 7 and 8 show the adjusted employment response 

rates for the interim and final files. As of the interim deliverable, the Fall 2007 employment response rate for the postcard 

group was 3.5 percent less than the response rate for the control group. By early July, that gap had closed to 1.7 percent. The 

large difference seen in Alaska is probably due to their smaller sample size. That the overall difference is much smaller 

supports this. 

The Spring 2008 panel has similar results. As of the interim deliverable, the employment response rate for the postcard 

group was 0.62 percent more than the response rate for the control group. By the final deliverable in early January, that gap 

had widened to 1.86 percent. 
Table 7. Interim and Final Employment Response 

Rates (Adjusted) – Nov Panel

FIPS Group Interim Response Interim Difference Final Response Final Difference 

02 ctrl 25.1% 

 

 

 

 

 

 

87.7% 

02 test 47.9% 22.8% 50.4% -37.3% 

08 ctrl 37.9% 70.5% 

08 test 35.0% -2.9% 58.4% -12.0% 

16 ctrl 65.7% 72.5% 

16 test 60.7% -5.0% 67.0% -5.5% 

24 ctrl 61.1% 66.5% 

24 test 45.5% -15.6% 51.1% -15.4% 

31 ctrl 85.5% 85.4% 

31 test 80.3% -5.2% 80.3% -5.2% 

42 ctrl 25.4% 42.8% 

42 test 25.6% 0.1% 57.0% 14.2% 

All ctrl 40.1% 59.2% 

All test 36.6% -3.5% 57.4% -1.7% 

 

 

 

 

 

 

 

Table 8. Interim and Final Employment Response 

Rates (Adjusted) – Spring Panel 

State Group Interim Response Interim Difference Final Response Final Difference 

AK ctrl 55.53% 45.05% 

AK test 59.36% 3.82% 49.01% 3.96% 

CO ctrl 53.93% 52.50% 

CO test 52.65% -1.28% 55.94% 3.44% 

ID ctrl 51.24% 65.91% 

ID test 50.66% -0.58% 68.79% 2.88% 

MD ctrl 56.79% 55.05% 

MD test 53.73% -3.06% 61.66% 6.61% 

NE ctrl 45.87% 84.15% 

NE test 49.88% 4.01% 86.43% 2.28% 

PA ctrl 44.65% 56.73% 

PA test 47.71% 3.06% 55.57% -1.17% 

All ctrl 49.93% 57.31% 

All test 50.55% 0.62% 59.17% 1.86% 
Responses by Size of the Establishment 

Chart 1 shows the adjusted Fall panel pre-interim response rates by establishment size class graphed out in a line. The size 

class categories are based on the number of employees in the establishment. While the pattern is quite similar to the usual 

size class response pattern seen in OES data5

, there does appear to be a difference between the control and test groups. There 

is a gap starting with size class 7, units with more than 250 employees. The response rates are higher for the control group 

for size classes 7 and 8, units with 250 – 499 employees and 500 – 999 employees, by about 8 and 5 percentage points. 

However, the test group is noticeably higher for the largest establishments, those with more than 999 employees6

. These 

differences become less pronounced by the end of the panel, presumably due to an intentional effort on the part of the State to

acquire data from those units. However, there remains a substantial gap in response for size class 7, units with 250 – 499 

employees. Charts 2 and 3 show the response rates for the interim and final deliverables. 

The data for the Spring 2008 panel (Charts 4 and 5) show a different trend. There is still a gap in response rates, starting with 

size class 8, units with 500 – 999 employees. However, for this panel the test group has the higher response rates. Up until 

units with more than 499 employees the response rates are identical. 

Given that the size class response rates for both panels are virtually identical up to size class 7, it can be argued that for 

smaller units the type of contact does not matter. Also, the overall difference in response rates is due to non-response in the 

larger units. These differences also suggest that for the larger units the postcard alone is not effective and that perhaps a 

mixed approach is warranted. Postcards could be sent to size classes 1 through 6, units with less than 250 employees while a 

full packet is sent to larger units. There are a few hypotheses as to why larger establishments respond differently. One 

hypothesis is that larger establishments treat the mail differently than smaller establishments and the postcard is not reaching 

the person responsible for responding to the survey. Another theory is that larger firms are busier and it takes them longer to 

respond. 

Chart 1. Pre-Interim Response Rates by Size Class (Adjusted) – Fall Panel 

Pre-Interim Response Rates by Size Class (Adjusted Nov Panel)

34.3%

23.2%

15.6%

34.6%

31.1%

41.9%

46.8%

55.2%

60.2%

18.8%

25.9%

34.9% 30.2%

30.1%

40.4%

44.9%

54.5%

58.4%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

123456789

1-4 5-9 10-19 20-49 50-99 100-249 250-499 500-999 1000 +

Employment Size Class

Control

Test

 

5

 In OES the response rates decline as the size of the establishment increases until we reach the largest establishments. For 

the largest units, those that employ more that 999 workers, there is a slight improvement in response rates. 

6

 The number of size class 9 units in this sample is quite small and therefore more volatile. Also, in terms of collection, such 

large units often garner special attention from State OES personnel making it harder to tell if the postcard is the driving factor 

behind the difference in response rates. 
Chart 2. Interim Response Rates by Size Class (Adjusted) – Fall Panel 

Interim Response Rates by Size Class (Adjusted Nov Panel)

50.8%

21.5%

30.4% 30.3%

40.1%

37.9%

43.0%

58.5%

68.9%

80.4%

39.5%

31.4%

37.8%

44.1%

57.1%

49.9%

67.2%

81.5%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

123456789

1-4 5-9 10-19 20-49 50-99 100-249 250-499 500-999 1000 +

Employment Size Class

Control

Test

Chart 3. Final Response Rates by Size Class (Adjusted) – Fall Panel 

Final Response Rates by Size Class (Adjusted November Panel)

89.7% 84.6%

72.0%

64.1%

53.8%

60.8%

52.4%

39.1%

66.7%

91.6%

83.5%

71.1%

62.9% 57.8%

49.0%

53.5%

40.5%

69.8%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

123456789

1-4 5-9 10-19 20-49 50-99 100-249 250-499 500-999 1000 +

Employment Size Class

Control

Test
Chart 4. Interim Response Rates by Size Class (Adjusted) – Spring Panel 

Interim Response Rates by Size Class (Adjusted May Panel)

40.3%

36.0%

26.0%

18.5%

57.1%

48.5%

64.7%

72.2%

83.8%

25.9%

39.8%

35.4%

38.9%

56.9%

49.1%

63.8%

71.3%

82.5%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

123456789

1-4 5-9 10-19 20-49 50-99 100-249 250-499 500-999 1000 +

Size Class

Control

Test

Chart 5. Final Response Rates by Size Class (Adjusted) – Spring Panel 

Final Response Rates by Size Class (Adjusted May Panel)

59.3%

48.0%

54.3%

53.0%

58.3%

66.7%

73.9%

79.5%

89.1%

63.0%

57.7%

51.4%

53.5%

60.5%

67.3%

74.1%

79.7%

87.8%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

123456789

1-4 5-9 10-19 20-49 50-99 100-249 250-499 500-999 1000 +

Size Class

Control

Test
Date Stamp Analysis 

OES examined the date stamp (the date the state changed the establishment record from a nonresponse to a response) in the 

State data files. Allowing for transit times, we were able to identify units that responded during the overlap period - the time 

after the State has generated the first follow-up mailing file but before the establishment would have received the follow-up 

survey packet or postcard. In the Fall panel, the test group received 451 units during the overlap while the control group 

received 581. In the Spring panel 536 control group units sent in responses and 514 test units sent in responses. These 

numbers simply demonstrate that there are a significant number of responses still in transit when the production for the first

follow-up mailing begins and that there is a savings in not mailing a full survey packet to these establishments. 

Table 9. Date Stamp Analysis – Adjusted Fall Panel 

Group 

During 

Overlap 

After 

Postcard 

After 

Packet 

After the 3rd 

Mailing 

After the 4thd

Mailing 

After All 

Mailings 

Control 581 0 1196 768 604 1275 

Test 415 812 0 1197 679 1499 

Table 10. Date Stamp Analysis – Adjusted Spring Panel 

Group 

During 

Overlap 

After 

Postcard 

After 

Packet 

After the 2nd

Follow-up 

After the 3rd

Follow-up 

After All 

Mailings 

Control 536 0 972 590 356 1196 

Test 514 1195 0 569 313 1197 

Cost Savings 

Standardized Approach 

Because the groups are different sizes, it is worthwhile to look at per-unit costs. The per-unit cost of the second mailing for 

the test group was $0.30. The per-unit cost for the control group was between $1.43 and $2.88 depending on whether the unit 

received an unstructured form or a structured form (and the number of pages in the structured form). The second mailing 

typically has about 140,000 units. Projecting the potential cost saving of $1.13 to $2.58 per unit, we see a potential savings

of $158,200 to $361,200 per panel. 

Given that there are differences in the response rates for the two groups, we also looked at the printing and postage costs 

stretched over the panel to see if mailing out extra forms to non-respondents in subsequent mailings would increase the costs 

for the test group. The average cost per unit7

 for the test group was $3.26 while the average cost per unit for the control 

group was $4.98. Even with mailing out more survey packets in subsequent mailings the postcard group has significantly 

lower costs. The difference of $1.72 per unit projected across the panel, using the second mailing base of 140,000 units, also

shows a potential savings of approximately $240,000 per panel. 

 

 

 

Table 11. Cost Analysis - Postcard Treatment for All Units 

M2 - Ctl M2 - Test Mailing3 Mailing4 Total Per-unit 

Test units 5763 4906 3933 

Control units 5419 4267 3394 

postage $1.31 $0.26 $1.31 $0.97 

printing $0.94 $0.04 $0.94 $0.56 

 

 

 

 

 

 

Combined $2.25 $0.30 $2.25 $1.53 

Test costs $1,728.90 $11,038.50 $6,017.49 $18,784.89 $3.26 

Control costs $12,192.75 $9,600.75 $5,192.82 $26,986.32 $4.98 

Difference $1.72 

M2 base 140000 

Projected Savings $240,852.74 

 

7

 The average cost per unit does not take into account the first mailings. The first mailing occurred prior to designating the 

test and control groups. Therefore the groups were not distinguished from each other and received the same treatment in 

terms of mailout; both groups received regular survey packets. 
Mixed Approach 1 - Postcard for Units 1-6 / Regular Packet for Units 7-9 

Cost projections for a mixed treatment approach, where size classes 1 through 6 receive postcard reminders while the larger 

sized firms receive a regular survey packet, also shows a significant savings of about $238,000 per panel. Approximately 

5,500 units of the first follow up mailing are size class 7, 8, and 9 units. Applying the observed higher response rate to that 

number as well as the appropriate costs over the course of the panel shows a cost of $27,690. The total costs of size class 1-6 

are $438,887 for a combined cost of $466,577. This figure is substantially lower than our normal operating costs for a panel 

of $704,8448

. Again, even with a mixed approach we see projected saving of around $238,000. This approach does not cost 

anything in terms of response rates; the size class analysis shows no significant decrease in response rates for the small and 

medium size units receiving the postcard. 

 

 

 

 

Table 12. Cost Analysis – Mixed Treatment Approach 1 

Groups 

Mailing 2 - 

Sizes 7-9 

Mailing 2 - 

Sizes 1-6 Mailing 3 Mailing 4 Subtotals Total 

Postcards 134,500 111,635 89,308 

full packet 5,500 4,290 3,432 

Combined $2.25 $0.30 $2.25 $1.65 

size class 1-6 $40,350.00 $251,178.75 $147,358.20 $438,886.95 

size class 7-9 $12,375.00 $9,652.50 $5,662.80 $27,690.30 $466,577.25 

 

 

 

 

 

Unaltered 140,000 109,200 87,360 

$315,000.00 $245,700.00 $144,144.00 $704,844.00 $704,844.00 

Projected Savings $238,266.75 

Mixed Approach 2 - Postcard for Units 1-6 / Postcard + Follow-up Packet for Units 7-9 

Table 13 shows the cost projections for a mixed treatment approach, where size classes 1 through 6 receive postcard 

reminders while the larger sized firms receive a postcard and a follow-up survey packet. This treatment also shows a 

significant savings of over $200,000 per panel. There are approximately 140,000 units in thesecond mailing. About 5500 

units of those units are in size classes 7, 8, and 9. Applying the observed response rate to the different groups as well as the 

appropriate charges over the course of the panel shows a cost of $474,142. This figure is substantially lower than the normal 

operating costs for a panel of $690,2509

. Again, even with this mixed approach we see projected saving of around $240,000. 

This approach also does not cost anything in terms of response rates but may have a small positive impact on the response 

rates of the largest units. 

Table 13. Cost Analysis – Mixed Treatment Approach 2 

M2 - Pkt M2 – PC Mailing3 Mailing4 Total Per-unit 

Different 5500 134500 121769 96266 

based on test 

resp rates 

 

 

 

 

 

 

 

 

Normal 140000 115195 92565 

based on ave 

resp rates 

postage $1.31 $0.26 $1.31 $0.97 

printing $0.94 $0.04 $0.94 $0.56 

combined $2.25 $0.30 $2.25 $1.53 

Diff costs $12,375.00 $40,350.00 $273,979.23 $147,287.27 $473,991.50 $3.52 

Norm costs $315,000.00 $259,189.75 $141,625.02 $715,814.77 $5.11 

difference $1.59 

Projected Savings $241,823.27 

 

8

 These figures do not include costs associated with the initial mailings. We are not advocating a change in that mailing so it

remains a constant and does not affect this analysis. 

9

 These figures do not include costs associated with the initial mailings. We are not advocating a change in that mailing so it

remains a constant and does not affect this analysis. 
Additive Approach – Adding the Postcard for All Units 

Cost projections (found in Table 14) for adding a postcard mailing to all units in the follow-up mailing base is approximately 

$31,000 per panel. There are approximately 140,000 units in the first follow-up mailing. If all units receive the postcard we

can expect an immediate bump in response rates of about 2-3 percent over a standard mailing and about 8-9 percent over a 

postcard mailing alone. However, the increase in response rates waivers as the panel progresses. By the end of the panel the 

response rates are within 0.5 percent. Is the extra cost of adding the postcard worth this small increase in response? 

Considering the original intent was to save money, we have to question the effectiveness of a simple addition of postcards to 

the mailing regimen. 

Table 14. Cost Analysis – Adding the Postcard 

 

 

M2 - Ctl M2 - Test Mailing3 Mailing4 Total Per-unit 

Test units 5665 4200 3393 

Control units 5449 

 

 

4122 3424 

postage $1.31 $0.26 $1.31 $0.97 

printing $0.94 $0.04 $0.94 $0.56 

combined $2.25 $0.30 $2.25 $1.53 

 

 

 

 

 

Test costs $14,445.75 $9,450.00 $5,191.29 $29,087.04 $5.13 

Control costs $12,260.25 $9,274.50 $5,238.72 $26,773.47 $4.91 

difference -$0.22 

M2 base 140000 

Projected Savings -$30,947.33 

State Feedback 

All test States were sent a feedback questionnaire to help evaluate the results of the postcard test. The States were asked the 

following questions: 

1. Do you think the postcard was effective? 

2. What suggestions do you have for improving the procedure of using a reminder postcard? (Please list each 

suggestion on a separate line below.) 

3. Did you receive requests from any establishments for additional survey forms during the time between the 

postcard mailing and the next centralized mailing of survey forms? 

4. Did you make follow-up telephone calls during the time between the postcard mailing and the next 

centralized mailing of survey forms? 

5. What, if any, feedback did you receive from respondents regarding the postcards? (Please list types of 

feedback on separate lines below.) 

6. Do you have any additional feedback or recommendations: 

Four of the five final test States believed that the postcards were effective. Some States reported that they received calls for 

new survey forms but that they were able to fulfill those requests. Three States began telephone follow-up calls to both 

groups (since the States were not informed of which establishments were in each group). Respondent feedback was minimal, 

and was limited to claims that they had either not received the original form or that they had already sent in the form. 

States reported receiving more email and phone responses from the postcard group than the control group. However an 

analysis of the collection mode in their data files revealed that there were no significant differences between the control and

test groups. We feel that the report of receiving more emails and phone calls is a reflection of the general increase in 

respondents using alternative modes of reporting data. 
Most States participating in the postcard test were positive about the feasibility of using it successfully as an alternative to 

mailing a survey packet, although two States feel that we should still adjust the timing and allow more time between

mailouts. Some also identified the postcard as a valuable tool for accomplishing address refinement. Two States stated on 

their questionnaires that the postcard is an effective cost-saving measure. 

 

 

 

 

Table 15. State Questionnaire Results 

Question Yes No Don't Know 

1. Was postcard effective? 4 1 

2. Did you receive requests for survey forms 3 1 1 

 If yes, were you able to fill the requests 3 

 If yes, how many requests were received 1, 25, 75 

 

 

 

 

 If yes, were requests from the postcard group 3 

3. Did you make telephone follow-up calls 3 2 

 If yes, which groups did you call (postcard, control, both, other) both groups 

If yes, did the postcard group need more calls, same, fewer calls About the same - 2 

Fewer - 1 

4. Respondent Feedback 

 Did not receive the original survey form 3 States - multiple calls 

 

 

 

 Already sent in the filled form 1 State, several calls 

 Can I do this on-line? 1 

5. State Feedback 

 More respondents reported data over the phone 3 

 

 

 

 More respondents sent in data via email 1 

 Postcards generated postal returns for address correction 1 

 This is a good cost-saving measure 2 

 Would like to use the postcard later in the mailout process for 

small units 1 

 

 There is little difference between a follow-up mailing and a 

follow-up postcard 1 

 Would like to continue the use of the postcard 4 

 

 

 

 

 

 

6. Suggestions to make the postcard effective

 Put a tracking feature in SPAM to track who receives a postcard 1 

 Enlarge the title of the survey form in the picture on the postcard 1 

 Identify to the States which establishments receive the postcard 2 

 Use postcards for address refinement 2 

 Change the postcard mailing to before the final packet mailing 1 

Statistical Analysis 

A statistical analysis of the overall response rates from the fall panel of the test did not show any significance at the 95 

percent confidence interval10 using a chi-square test, meaning that the postcard did not have an effect on response rates. 

While most of the establishment size class data also did not meet the criteria for significance, there does appear to be a 

stronger association between postcard units not responding and those units being in a larger size class. Only chi-square 

values for size class 7 met the threshold for significance at the 95-percent confidence interval. For these units, being in the 

test group increased the likelihood of non-response. Lower response rates for size class 7 are seen across all States in the test. 

This is not the case for size class 8 and 9 units. It could be that the common State practice of concentrating on collecting data 

 

10 With such a large sample size, it is not recommended to use a 90-percent confidence interval. 
from the largest units prevented size classes 8 and 9 from also showing increased non-response for the test group. With 

regard to test group response rates in individual States, none of the States had chi-square values showing significance. Using

a logistic regression analysis, controlling for the experiment, we only see the typical OES response patterns; smaller States 

and smaller establishments have higher response rates. Considering the findings in the size class analysis, a conservative 

course for implementing the reminder postcard would be a mixed approach; send postcards to size class 1 through 6 units, 

while continuing to send full survey packets and a postcard as well to size class 7 through 9 units. 

 

Analysis of the Spring panel data showed no significance at the 95 percent confidence interval using chi-square tests. Chisquare tests were run on response rates by size class and State. The only trend revealed is already known to OES; response is 

more likely in smaller establishments. None of the test groups met the criteria for significance. This is not unexpected since 

the methodology for the Spring panel test was simply adding the postcard to the regular mailings as opposed to substituting it 

for a regular survey packet. Given this analysis, there is no advantage to adding a postcard mailing to supplement the regular

survey packet mailings. However, since substituting a postcard has a negative impact on response for at least one size class 

among the larger units, adding a postcard mailing for the larger units would not only maximize response but it could also

generate more savings (according to our cost models) due to increased immediate response. 

Conclusions and Recommendations 

Based on the State data from the combined phases of the test, incorporating the use of a reminder postcard for the second 

mailing is a viable cost-saving measure that has little measurable impact on response rates. The impact of the reminder 

postcard on response rates is minimal for the small and medium sized firms. Larger units see a decrease in response. A 

mixed approach where small and medium sized firms receive a reminder postcard while the larger firms receive a full survey 

packet appears to be an efficient and cost-effective option. Similarly, sending postcards to small and medium sized firms 

while sending both a postcard and a regular survey packet to large-sized firms also achieves significant cost-savings while 

minimizing the impact on response rates. Table 16 summarizes the response rates for each treatment group. Table 17 shows 

the cost savings per panel associated with each treatment. 

Table 16. Combined Response Rates 

Response Rates by Size Class Group and Treatment 

After 2nd

Follow-up 

After 3rd

Follow-up Interim Final 

size 1- 6 packet* 21.5% 37.2% 61.9% 72.8% 

size 7 - 9 packet* 14.7% 26.2% 34.4% 54.8% 

size 1- 6 postcard 12.9% 31.4% 59.6% 73.2% 

size 7 - 9 postcard 9.4% 19.9% 30.1% 49.7% 

size 1- 6 both 26.2% 40.5% 63.6% 73.0% 

size 7 - 9 both 16.0% 28.1% 35.1% 54.3% 

*Combined control groups from both panels 

Table 17. Summary Cost Savings Potential per Panel

Replacement Scenarios Potential Savings per panel 

Standard Replacement $240,853 

Mixed 1 - Postcards 1-6/Packets 7-9 $238,267 

Mixed 2 - Postcards 1-6/Both 7-9 $241,823 

Add Postcards -$30,947 

Sending reminder postcards to small and medium sized firms while sending both a postcard and a regular survey packet to 

large-sized firms achieves the most cost-savings. The response rates associated with this treatment are most similar to the 

response rates using our current procedures. This treatment is therefore our recommendation for final implementation. We 

also suggest that OES reconsider the mailout schedule in order to better incorporate the postcard. 
Acknowledgements 

The author would like to thank Polly Phipps of the Bureau of Labor Statistics, Office of Survey Methods Research for her 

assistance in running and interpreting chi-square values for the statistical portion of this paper. 

Disclaimer 

Any opinions expressed in this paper are those of the author and do not constitute policy of the Bureau of Labor Statistics. 

References 

Dillman, Don A. Mail and Internet Surveys – The Tailored Design Method, Second Edition. New York: John Wiley & Sons, 

Inc., 2000.