2014 Organic Survey –

Promotional Materials 

Usability Testing

Heather Ridolfo, PhD

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-15-05

November 2015

This report has been prepared for limited distribution to the research community outside the U.S. 

Department of Agriculture (USDA). The views expressed herein are not necessarily those of NASS or 

USDA.
i

EXECUTIVE SUMMARY

The 2014 Organic Survey is a data collection project conducted by the United States Department 

of Agriculture’s (USDA) National Agricultural Statistics Service (NASS) for the USDA’s Risk 

Management Agency. The survey collects information on organic production and practices. The 

population of interest is any operation that produces organic products according to USDA’s 

National Organic Program (NOP) standards and is certified or exempt from certification. The 

population also contains operations that reported that they were transitioning to organic in the 2012 

Census of Agriculture.

Prior and during data collection, NASS utilizes promotional materials to notify participants of their 

selection into surveys and to convey information about the survey (e.g., the purpose, potential 

benefits, due dates). For the 2014 Organic Survey, NASS is planning to send participants a 

postcard prior to the start of data collection notifying participants of the upcoming survey and the 

potential benefits of the survey data to the organic community. NASS will also send a cover letter 

with the survey at the start of data collection to inform participants of the purpose of the survey, 

potential benefits, how they can respond and when their responses are due. Near the end of data 

collection, NASS will then send participants a reminder letter asking them to respond if they have 

not already. 

The purpose of the study was to examine whether participants:

1) Understood the information presented in these materials

2) Whether they could easily locate important information, and 

3) How effective this information was in recruiting participants to participate in the survey. 

Usability interviews were conducted with six organic operations. Participants were presented with 

the materials individually, in the order in which they would normally receive them. Interviewers 

were instructed to record participants’ initial reaction to the materials and note how the participants 

went about reviewing them. Participants were then asked a series of questions to assess their 

reactions to the information provided.

In terms of the postcard, participants appreciated having information on why it is important to 

respond to the survey; however, they felt more salient reasons needed to be given. In particular, 

participants did not feel that informing crop insurance programs was a motivating factor for 

responding. Instead, participants would prefer to see findings from the previous Organic Survey.

Participants generally understood and could locate important information presented on the cover 

letter (i.e., purpose of the survey, agency conducting the survey, and how to respond). Some 

participants had difficulty locating the due date of the survey, and some participants had difficulty 

comprehending the confidentiality statement. Participants felt the most important information to 

include on this letter was the purpose of the survey, the agency conducting the survey, reasons 

why they should participate, the due date and that they can respond online. Participants did not 

feel the statement, which said responding online saved taxpayer money, was persuasive. 
ii

Finally, participants felt the most important information presented on the reminder letter was the 

reasons why it is important to respond, that they can respond online, and that an enumerator would 

follow up with them if their response is not received. Participants again had difficulty 

comprehending the confidentiality statement. Participants also felt it was best to only send this 

letter if their data have not been received by the due date. 
iii

RECOMMENDATIONS

1) Remove Reason #2 from the list of the top 3 reasons to respond to the survey on the 

backside of the postcard. Respondents did not feel that improving crop insurance programs 

was a motivating factor for responding to the survey. [This recommendation was accepted.] 

2) Remove Reason #3 from the list of the top 3 reasons to respond to the survey on the 

backside of the postcard. Respondents did not feel that the statement indicating that their 

responses would be kept confidential was a motivating factor for responding to the survey. 

[This recommendation was accepted.]

3) Add findings from the previous Organic Survey that demonstrate why the survey data is 

beneficial to a broad range of organic growers. [This recommendation was accepted. The 

reasons now read: 1) Total organic sales by farms in the U.S. increased by 83 percent 

between 2007 and 2012, 2) Sales from farms with certified or exempt organic product sales 

totaled more than $3.1 billion in 2012, 3) Organic agriculture produces reported direct-toconsumers sales more often than conventional producers, with 42 percent of organic 

farmers selling directly to consumers in 2012.]

4) The following statement in the cover and reminder letters: “This survey is part of the 

Census of Agriculture Program which is required and protected by law (Title 7, U.S. Code 

and the Confidential Information Protection and Statistical Efficiency Act). These federal 

laws require that NASS keeps your identity and your answers confidential” needs to be 

revised for clarity. Consider simplifying this statement by breaking it down into two more 

easily understood statements. “As part of the Census of Agriculture Program, this survey 

and your response is required by law (Title 7, U.S. Code). These federal laws also require 

that NASS keeps your identity and your answers confidential.” [This recommendation was 

accepted]. 

5) Remove the second paragraph in the cover letter, which reads, “The 2014 Organic Survey 

is important because the results will help USDA’s Risk Management Agency shape 

policy decisions, including re-evaluating Federal crop insurance programs to ensure 

organic producers received equitable insurance prices.” Respondents did not feel that 

improving crop insurance programs was a motivating factor for responding to the survey. 

Replace with text similar to the text used in the reminder letter, which respondents did 

find motivating (“Responses to the most recent Census of Agriculture let us know that 

organic agriculture is growing in the United States, with sales increasing 82 percent 

between 2007 and 2012. Your participation in this survey is important because every 

response is a step closer to Federal programs that better meet the needs of organic 

producers like you.”) [This recommendation was accepted. The text in the cover letter 

now reads, “According to the most recent Census of Agriculture, organic agriculture 

sales from farms in the United States increased 83 percent between 2007 and 2012. 

These data show the demand for, value of and potential for continued growth in certified 

organic products. Now the U.S. Department of Agriculture’s (USDA) National 

Agricultural Statistics Service (NASS) is conducting the 2014 Organic Survey to gather 

more detailed production, sales and marketing information from organic farms in the 

United States. Please complete and return your form by February 13, 2015. Your 

response is critical to help support the continued future success of organic agriculture 

production.”]
iv

6) Respondents did not feel the statement in the cover letter which said, “Responding online 

is the easiest, fastest, and safest way to complete your questionnaire and it saves taxpayer 

dollars too.” Consider removing. [This recommendation was accepted.]

7) Only send the reminder letter if we have not received participants’ data by the desired due 

date. [This recommendation was accepted.]
v

TABLE OF CONTENTS

1. INTRODUCTION .................................................................................................................. 1

2. METHODS ............................................................................................................................. 2

3. RESULTS ............................................................................................................................... 3

3.1 Postcard........................................................................................................................... 3

3.2 Cover Leter ..................................................................................................................... 4

3.3 Reminder Letter .............................................................................................................. 6

4. CONCLUSION AND RECOMMENDATIONS ................................................................... 8

APPENDIX A: Interview Protocol and Guide ............................................................................. 10

APPENDIX B: Promotional Materials Used During Testing....................................................... 20

APPENDIX C: Revised Promotional Materials........................................................................... 24
1

2014 Organic Survey – Promotional Materials Usability Testing

Heather Ridolfo1

Abstract

The 2014 Organic Survey is a data collection project conducted by the National Agricultural 

Statistics Service (NASS) for the Risk Management Agency. The survey collects information on 

organic production and practices. NASS utilizes promotional materials to notify participants of

their selection into surveys and to convey information about the survey (e.g., the purpose, potential 

benefits, due dates). Promotional materials for the 2014 Organic Survey will include 1) a postcard 

mailed prior to data collection, 2) a cover letter mailed with the survey, and 3) a reminder letter 

mailed near the end of data collection. The study examines the effectiveness of the messages 

included in these materials. Specifically, we examined 1) whether participants understood the 

information presented in these materials, 2) whether they could easily locate important 

information, and 3) how effective this information was in recruiting participants to participate in 

the survey. 

Key Words: Organic Survey, Promotional Materials, Usability Testing

1. INTRODUCTION 

The 2014 Organic Survey is a data collection project conducted by the United States Department 

of Agriculture’s (USDA) National Agricultural Statistics Service (NASS) for the USDA’s Risk 

Management Agency (RMA). Census funds will also be used as directed by the Agricultural Act 

of 2014. The survey is an organic production and practices survey, parts of which have been 

previously administered in the 2008 and 2011 Organic Survey. The population of interest is any 

operation that produces organic products according to USDA’s National Organic Program (NOP) 

standards and is certified or exempt from certification. The population also contains operations 

that reported that they were transitioning to organic in the 2012 Census of Agriculture. The 

questionnaire collects information on organic production of field crops, vegetables, fruits, tree 

nuts, berries, livestock and poultry, production practices, production expenses, marketing practices 

and value-added production and processing. 

Prior to receiving the survey, participants will all receive a postcard indicating the survey will be 

arriving soon and reasons it is important to respond. At the start of the data collection period, 

participants will receive the survey questionnaire with a cover letter that details 1) who is 

conducting the survey, 2) the purpose of the survey, 3) how to respond, and 4) whom to contact 

for more information. Participants are also informed that the survey is required by law and their 

responses will be kept confidential. Near the end of the data collection period, participants will 

 1 Heather Ridolfo is a Survey Statistician with the National Agricultural Statistics Service, Research and Development 

Division, 3251 Old Lee Highway, Room 305, Fairfax, VA 22030. The author would like to thank Kathy Ott, Dale 

Hawks, Cheryl Turner, Bruce Eklund, and Gary Keough for their assistance on this research project.
2

receive a reminder letter that will encourage them to respond to the survey if they have not already 

done so. The reminder letter will also stress 1) the importance of responding, 2) how they can 

complete the survey, 3) whom to contact for more information, 4) that the survey is required by 

law, and 5) that their responses will be kept confidential. Participants are also informed in this 

letter that a NASS representative will follow up with them if we do not receive their data.

Before sending these materials out to the public, NASS wanted to ensure that effective messaging 

is being used in these materials. That is, that the information presented in these materials is 

understandable, accessible, valuable, and credible. The following report presents findings from six 

usability interviews conducted to assess these factors. 

2. METHODS

Six interviews were conducted in the following states: Virginia, Maryland, Ohio, New Jersey, and 

New Hampshire. The sample included operations that grow organic crops and are certified or 

exempt from certification. Interviewers in each state were responsible for recruiting participants.

Participants were informed of the intent of the testing during the recruitment process and that their 

participation was voluntary.

All interviews were conducted in October 2014. Interviewers were provided an interview protocol 

and interview guide prior to data collection (see Appendix A). Copies of the postcard, cover letter 

and reminder letter used during testing can be found in Appendix B. 

At the start of the interview, participants were informed of the task. Participants were told NASS 

was testing the promotional materials that will be used for the 2014 Organic Survey and NASS

wanted to make sure the materials were informative and that everyone understood the information 

presented. Participants were then handed the postcard and instructed to pretend they had just 

received it in the mail. Interviewers were instructed to record any initial reactions the participants

had to the postcard and note how the participants went about reviewing the material on the 

postcard. After participants had reviewed the postcard, they were asked how much of the postcard 

they read. Participants were then asked a series of questions to assess their reactions to the 

information provided. Participants were asked such things as: What information stands out? Is 

there any information you think is missing? Is it helpful to receive a postcard like this? (See the 

interview protocol and guide in Appendix A for a complete list of questions). 

After providing feedback on the postcard, participants were then asked to review the cover letter. 

Participants were handed the cover letter and a copy of the survey questionnaire and asked to 

review it as if they had just received it in the mail. Interviewers were instructed to record any initial 

reactions the participants had to the cover letter and note how the participants went about reviewing 

the information on the cover letter. After participants reviewed the cover letter, the interviewers 

took the cover letter back and asked the participants how much of the cover letter they read. 

Participants were then asked a series of questions to test their recall of important information 

presented in the letter. Participants were asked if they could recall such things as: who was 

conducting the survey; what was the purpose of the survey; how could they respond to the survey; 
3

when was the survey due? The cover letter was then returned to the participants. If participants

were not able to answer a recall question, they were instructed to try and find the information on 

the cover letter. The interviewers noted participants’ ability to do so. 

Next participants were asked a series of questions to assess their reactions to the cover letter. 

Participants were first asked about interpretation and reactions to two messages in the letter. The 

first message stated: “This survey is part of the Census of Agriculture Program which is required 

and protected by law (Title 7, U.S. Code and the Confidential Information Protection and 

Statistical Efficiency Act). These federal laws require that NASS keeps your identity and your 

answers confidential.” The second message stated: “Responding online is the easiest, fastest, and 

safest way to complete your questionnaire and it saves taxpayer dollars too.” Participants then 

answered questions regarding their general reactions to the letter, such as: what information was 

most important to them; what information was missing; what information was unnecessary? 

Finally, participants were handed the reminder letter. Once again, participants were instructed to 

review the letter as if they had just received it in the mail and interviewers noted participants’ 

initial reactions to the letter and how they went about reviewing the information in the letter. When 

participants finished reading the letter, they were asked how much of the cover letter they read. 

Participants were then asked about their reactions to the letter. They were asked probing questions 

such as: what information stood out; what information was most important to you; what 

information was missing or unnecessary? Participants were also probed on their interpretation and 

reaction to the following statement in the letter: “This survey is part of the Census of Agriculture 

Program which is required by and protected by law (Title 7, U.S. Code and the Confidential 

Information Protection and Statistical Efficiency Act). These federal laws require that NASS keep 

your identity and your answers confidential.”

3. RESULTS

3.1 Postcard

All six participants looked over the front and back of the postcard. When first handed the postcard, 

some participants only skimmed over the information, while others read more thoroughly.

A couple of participants commented that the postcard was attractive. One participant commented 

that the postcard was attractive and the word “organic” and the USDA logo grabbed his attention. 

Another participant thought the picture on the front should be changed to an image of a tractor. He 

asked, “Do you people think all organic growers are hippies with a hoe?” 

For the most part, all of the participants felt the information provided on the postcard was easy to 

understand; however, one participant questioned why it said “Your 2014 Organic Survey,” since 

it wasn’t his survey. Another participant also commented that it was hard to make out the NASS 

logo. He had to look at it very closely to make out what it was. This same participant thought it 

would be better to identify NASS using its acronym rather than writing out the agency name. He 

was also expecting to see the USDA organic logo on the postcard. 
4

Participants felt that providing information about the survey and reasons why they should 

participate was helpful but they didn’t agree with the information that is currently being provided. 

In particular, two participants did not think the second reason given was effective. Reason #2 

stated: “Your input will help re-evaluate Federal crop insurance programs to ensure organic 

producers receive equitable insurance prices.” These participants said a lot of organic growers do 

not use crop insurance. Therefore, the price of crop insurance was not important to them. On a 

related topic, in cognitive testing of Section 9 (Organic Acres under Crop Insurance) in the Organic 

Survey, participants often commented that they did not know crop insurance was available for the 

types of crops they grew or that it was too expensive for them to purchase. Crop insurance might 

not be a good motivator to convince operators to participate in the survey. One participant didn’t 

feel reason #3 was a reason to cooperate. Reason #3 stated: “Your responses are completely 

confidential, as required by federal law.” This participant said that this statement did not positively 

reinforce responding to the survey. Rather it indicated one would be penalized by law if they did 

not respond. 

Several participants commented that more effective reasons needed to be given. Two participants

provided suggestions. One participant suggested stating that more federal dollars would be 

provided to organic programs for things such as certification reimbursements and research as a 

reason to participate in the survey. Another participant suggested including results from past 

surveys on the postcard. 

When asked if they felt it was helpful to receive a postcard like this prior to the survey, three 

participants indicated that it was helpful and three participants said it was not helpful. However, 

one of the participants, who said it was not helpful, indicated he reads all of his mail and responds 

to all legitimate requests. Two other participants, who did not think it was helpful to receive a 

postcard, would rather receive an email about the upcoming survey instead of a postcard.

3.2 Cover Letter

Of the six participants, four skimmed the contents of the cover letter, one read it in full and one 

participant read “most” of the content. After participants initially read the cover letter, we probed 

them to see if they could recall important pieces of information included in the letter, including 

who was conducting the survey, the purpose of the survey, how they could respond, and the due 

date. 

All participants could easily recall who was conducting the survey. Three participants recalled that 

NASS was conducting the survey; two participants recalled that the USDA was conducting the 

survey and another participant recalled “USDA, Ag Statistics.” 

When asked if they recalled the purpose of the survey, four participants indicated that the survey 

was being conducted to gather information/statistics on organic farming. One participant indicated 

the survey was being conducted to evaluate the crop insurance policy and risk management. 

Another participant also commented that the survey data would support RMA. One participant
5

could not recall reading information on the cover letter regarding the purpose of the survey but 

was able to locate this information after looking at the cover letter a second time. 

All participants could recall reading that the survey could be completed online or by mail. One 

participant commented that the information about responding online should be made more visible 

on the letter. Three participants indicated that they preferred to respond online and three 

participants indicated that they would prefer to respond to the survey by mail. One of these 

participants indicated that he prefers to respond by mail because he can use a pencil and fix his 

mistakes. Another participant who prefers to respond by mail commented that he likes to flip back 

and forth through the pages of the survey. 

Three participants recalled that the due date was March 1. Two participants could not recall the 

due date. Another participant also could not recall the due date but stated, “They always call me 

before they are due, so I don’t need to know when they are due.” 

Interviewers also probed participants on their interpretations and reactions to two important 

statements in the cover letter. First, participants were probed on their understanding of the 

following statement: “This survey is part of the Census of Agriculture Program which is required 

and protected by law (Title 7, U.S. Code and the Confidential Information Protection and 

Statistical Efficiency Act). These federal laws require that NASS keeps your identity and your 

answers confidential.” Three participants interpreted this statement as saying that NASS was 

required to keep their information confidential. Two participants interpreted this statement as 

saying they were required by law to complete the survey and both questioned whether anyone had 

ever been fined for not completing the survey. One of these participants then questioned whether 

this statement meant that NASS was required by law to conduct the survey or if it meant that he 

as a participant was required by law to participate in the survey. 

Participants generally trusted that NASS would keep their responses confidential. However, one 

participant commented that although he was confident NASS would keep his data confidential, he 

was still not happy that we publish the total numbers. This participant felt that our survey data only 

benefited big corporations. Another participant commented that he was confident his data would 

be kept secure but he thought other growers might have concerns. 

Participants were also probed on the effectiveness of the following statement encouraging them to 

respond online: “Responding online is the easiest, fastest, and safest way to complete your 

questionnaire and it saves taxpayer dollars too.” Two participants indicated that they would 

respond online regardless of this statement. After reading this statement, another participant stated, 

“It sounds like they really want me to do it online, so I guess I will give it a whirl.” However, this 

participant preferred to respond on paper. Two other participants indicated that this statement 

would have no effect on their decision to respond online. One participant commented, “This does 

not convince me to respond online... it makes me wonder when did the government ever worry 

about saving the taxpayer money?” The other participant, who was not persuaded, stated that 

computers put people out of work and therefore he would prefer not to respond online. 
6

All participants reported that the information presented in the letter was generally easy to 

understand. When asked what information they felt most important in the cover letter participants

reported the following: the due date, who is conducting the survey, the purpose of the survey, why 

they should participate, and the web address for responding online. Three participants felt there 

was unnecessary information included in the letter. One participant felt it was not necessary to 

include the name of Title 7 (Title 7, U.S. Code and the Confidential Information Protection and 

Statistical Efficiency Act). Two other participants felt the second paragraph, which states that the 

results of the survey will help RMA shape policy, was not necessary. These same participants

indicated that most small organic farmers do not qualify for crop insurance. 

Participants generally did not feel that anything was missing from the letter. However, one 

participant would like a NASS address where he could contact the NASS administrator. Another 

participant noted that the code/password for replying online was missing, although he presumed it 

would be in the official letter. 

After reading the letter, all six participants reported they would be willing to participate in the 

survey but two indicated this with some reluctance. One of these participants indicated he would 

participate so he wouldn’t get “pestered” about it. Another participant indicated he would only 

complete the survey because he is required to, but we would have to catch him at a time that “works 

for me.” 

3.3 Reminder Letter

Five of the six participants, reported that they skimmed the contents of this letter and one 

participant read it in full. All of the participants thought the information presented in the letter was 

generally easy to understand. After reading the letter, participants were probed to see what 

information in the letter stood out to them initially. Participants commented on different 

information presented in the letter. One participant noted that the letterhead and the USDA logo 

made the letter appear legitimate. Two participants noted that the letter was reminding them of the 

Organic Survey and that they had not completed it yet. One participant noted that the letter stresses 

the importance of responding to the survey. Another participant noticed the results from the 2012 

Census were included in this letter, although he felt they could be more pronounced. One 

participant felt there was a “subliminal message” in the letter, which indicated that his participation 

in the survey is mandatory. Finally, one participant commented that it seemed like the reminder 

letter was written by a different person than the cover letter because the messages were different. 

The cover letter seemed to focus on how the survey benefited the RMA and the reminder letter 

stressed Federal programs. 

Participants were probed to see what information in the letter was most important to them. Three 

participants felt the information that stated someone would follow up with them if they did not 

complete the survey was most important to them. However, one participant commented that this 

statement was not definitive because of the word “may.” He asked if this meant someone will

follow up with him or someone may follow up with him. Two participants felt the second 

paragraph in the letter, which stated, “Organic agriculture is growing in the United States, with 

sales increasing 82 percent between 2007-2012. Your participation in this survey is important 
7

because every response is a step closer to Federal programs that better meet the needs of organic 

producers like you,” was most important. One of these participants felt this information should be 

in the first paragraph of the letter. He felt this was a better justification for participating in the 

survey than the messages provided in the postcard and the cover letter. Another participant also 

commented that the information stating he could respond online was important to him.

Interviewers also probed participants on their interpretations of and reactions to the following 

statement, “This survey is part of the Census of Agriculture Program which is required by and 

protected by law (Title 7, U.S. Code and the Confidential Information Protection and Statistical 

Efficiency Act).” Participants interpreted this statement in different ways. Two participants

interpreted this statement as saying that NASS is required by law to keep their answers 

confidential. One participant interpreted this statement as saying he is required by law to complete 

the survey and questioned whether anyone was ever fined for not completing the survey. Once 

again, one participant questioned whether it meant that NASS was required by law to conduct the 

survey or if it meant that he as a participant was required by law to participate in the survey. 

Finally, one participant interpreted this statement as saying he is required to do the survey and that 

his responses will be kept confidential. In general, participants felt this statement had no influence 

on their decision to participate in the survey. One participant commented that this was just “high 

powered legal talk.” One participant felt this statement was not necessary because he knows he 

will get a phone call if he doesn’t send in his data. The statement does not convince him at this 

point to participate. Another participant said it would have no impact on his decision because he 

would respond to the survey anyway. 

Participants were probed to see if there was any information in the reminder letter that they felt 

was unnecessary. As with the cover letter, one participant felt the name of Title 7 (Title 7, U.S. 

Code and the Confidential Information Protection and Statistical Efficiency Act) was not 

necessary. Finally, one participant commented that there was a lot of text on the letter but he was

not sure what could be removed. 

Participants were also probed to see if there was any information that they felt was missing from 

the reminder letter. Once again, one participant would like an address for NASS to be included in 

the letter. Two participants commented that there was no due date on this letter and one of these 

participants also pointed out that there were no repercussions stated for not completing the survey. 

This participant stated that he could look up Title 7 to see if there are any repercussions for not 

completing the survey; however, he doesn’t think most people would do this. Finally, one 

participant felt more information could be included that explained how the results of this survey 

would benefit the growers. 

Participants were then asked for their general comments on the letter. Two participants felt it was 

unnecessary to send a reminder letter. One participant felt it was better to just call him if we had 

not received his survey data. Another participant felt it was better to send a reminder postcard or 

an email. Finally, one participant commented that he hopes he would only be sent this letter if he 

hadn’t sent the survey in yet. He said he does not like receiving these letters when he has already 

completed his survey. 
8

4. CONCLUSION AND RECOMMENDATIONS

Most participants understood and could locate important information on the promotional materials. 

However, some participants had difficulty comprehending the confidentiality statement on the 

cover and reminder letters and a few participants could not locate the due date on the cover letter. 

The information presented on these materials was in line with the types of information participants

found important. That is, participants felt it was important to include the purpose of the survey, 

the agency conducting the survey, reasons why they should participate, the due date and that they 

can respond online. However, participants felt the promotional materials could provide more 

salient reasons for why it is important to participate in the survey. Based on the findings from this 

study, the following changes are recommended to increase the effectiveness of the promotional 

material: 

1) Remove Reason #2 from the list of the top 3 reasons to respond to the survey on the 

backside of the postcard. Respondents did not feel that improving crop insurance programs 

was a motivating factor for responding to the survey. [This recommendation was accepted.] 

2) Remove Reason #3 from the list of the top 3 reasons to respond to the survey on the 

backside of the postcard. Respondents did not feel that the statement indicating that their 

responses would be kept confidential was a motivating factor for responding to the survey. 

[This recommendation was accepted.]

3) Add findings from the previous Organic Survey that demonstrate why the survey data is 

beneficial to a broad range of organic growers. [This recommendation was accepted. The 

reasons now read: 1) Total organic sales by farms in the U.S. increased by 83 percent 

between 2007 and 2012, 2) Sales from farms with certified or exempt organic product sales 

totaled more than $3.1 billion in 2012, 3) Organic agriculture produces reported direct-toconsumers sales more often than conventional producers, with 42 percent of organic 

farmers selling directly to consumers in 2012.]

4) The following statement in the cover and reminder letters: “This survey is part of the 

Census of Agriculture Program which is required and protected by law (Title 7, U.S. Code 

and the Confidential Information Protection and Statistical Efficiency Act). These federal 

laws require that NASS keeps your identity and your answers confidential” needs to be 

revised for clarity. Consider simplifying this statement by breaking it down into two more 

easily understood statements. “As part of the Census of Agriculture Program, this survey 

and your response is required by law (Title 7, U.S. Code). These federal laws also require 

that NASS keeps your identity and your answers confidential.” [This recommendation was 

accepted]. 

5) Remove the second paragraph in the cover letter, which reads, “The 2014 Organic Survey 

is important because the results will help USDA’s Risk Management Agency shape 

policy decisions, including re-evaluating Federal crop insurance programs to ensure 

organic producers received equitable insurance prices.” Respondents did not feel that 

improving crop insurance programs was a motivating factor for responding to the survey. 

Replace with text similar to the text used in the reminder letter, which respondents did 

find motivating (“Responses to the most recent Census of Agriculture let us know that 

organic agriculture is growing in the United States, with sales increasing 82 percent 

between 2007 and 2012. Your participation in this survey is important because every 
9

response is a step closer to Federal programs that better meet the needs of organic 

producers like you.”) [This recommendation was accepted. The text in the cover letter 

now reads, “According to the most recent Census of Agriculture, organic agriculture 

sales from farms in the United States increased 83 percent between 2007 and 2012. 

These data show the demand for, value of and potential for continued growth in certified 

organic products. Now the U.S. Department of Agriculture’s (USDA) National 

Agricultural Statistics Service (NASS) is conducting the 2014 Organic Survey to gather 

more detailed production, sales and marketing information from organic farms in the 

United States. Please complete and return your form by February 13, 2015. Your 

response is critical to help support the continued future success of organic agriculture 

production.”]

6) Respondents did not feel the statement in the cover letter which said, “Responding online 

is the easiest, fastest, and safest way to complete your questionnaire and it saves taxpayer 

dollars too.” Consider removing. [This recommendation was accepted.]

7) Only send the reminder letter if we have not received participants’ data by the desired due 

date. [This recommendation was accepted.]

The findings from this report were reviewed by the Organic Team. The promotional materials were 

then revised based on the recommendations made in the report. The revised materials can be found 

in Appendix C. 
10

APPENDIX A: INTERVIEW PROTOCOL AND GUIDE

2014 Organic Survey – Promotional Materials

Usability Testing Protocol

September 26, 2014

Please read this entire document before conducting any interviews to become familiar with the 

data collection procedures and promotional materials.

Background

The 2014 Organic Survey is a data collection project conducted for the Risk Management Agency 

(RMA); Census funds will also be used as directed by the 2014 Farm Appropriations bill. The 

survey is an organic production and practices survey, parts of which have been previously 

conducted in 2008 and 2011. The population of interest is any operation that produces organic 

products according to USDA’s National Organic Program (NOP) standards and is certified or 

exempt from certification. The population also contains operations that reported in the 2012 

Census of Agriculture that they were transitioning to organic. The questionnaire collects 

information on organic production of field crops, vegetables, fruits, tree nuts, berries, livestock 

and poultry, production practices, production expenses, marketing practices and value-added 

production and processing. 

Research Goals

The purpose of this research study is to assess participants’ comprehension of and reactions to the 

Organic Survey promotional materials. In this study, we will assess the postcard, cover letter and 

reminder letter. 

Sample

Since OMB clearance was not obtained for this testing, we will only be conducting a total of 9 

interviews. Interviews will be conducted in the following states: CA, MD, OH, NH, and NJ.

The sample will include operations that grow organic crops and are certified or exempt from 

certification. 

Recruitment

Interviewers in each state are responsible for recruiting participants. Interviewers may recruit 

known organic operations in their area that are certified or exempt from certification. We also 

encourage interviewers to search local CSAs for organic producers. See screener questionnaire at 

the end of this document.
11

Participants should be informed of the intent of the interviews during the recruitment process and 

that their participation is voluntary. Participants should also be told the interviews will be 

conducted in person and are expected to last 1 hour.

Interview Procedures

Each interviewer is expected to conduct 1 interview. Interviewers should bring a copy of the 

interview guide, postcard, cover letter, blank questionnaire, and reminder letter to the interview. 

The following procedures should be followed during the interview. 

Interviewers should explain the task to the participant at the start of the interview. Tell the 

participant we are interested in their impressions of the letters we will send out for this survey. We 

want to know how to make these letters more informative and useful to participants (see script at 

top of interview guide.)

Postcard

After explaining the task to the participant, hand them the postcard and ask them to look it over. 

Tell participant: Pretend you just received this postcard in the mail. I want you to look it over 

however you normally would. If the participant doesn’t look at the postcard at all and says 

something like “I’d throw it away” make a note of this and then ask them to look over the postcard.

Note any comments they make while looking it over (positive or negative). After they have 

finished looking at it ask the following questions: 

If you received this postcard in the mail what would be your initial reaction? 

Is there any information that stands out to you? 

Is there any information on this postcard that you find confusing or have a question about? 

If there any information you think is missing? 

Do you feel it is helpful to receive a postcard like this? 

Organic Cover Letter

Next, hand the participant a copy of the cover letter and the survey instrument. We will not be 

testing the survey instrument. The reason we are giving the survey instrument along with the cover 

letter is we want to mimic the procedures in the field (participants receive these two items 

together). The instrument will provide additional context to help participants understand the 

information presented in the letter. 

Have participants look over the cover letter. Tell participant: Pretend you just received this letter 

and survey in the mail. I want you to look the letter over however you normally would. If the 

participant doesn’t look at the letter at all and says something like “I’d throw it away” make a note 

of this and then ask them to look over the letter.
12

Note any comments they make while reading the letter. When they are finished take the cover 

letter and survey instrument away and ask the following questions: 

(Recall/Comprehension)

How much of this letter did you read? Did you read this letter in full, skim, didn’t read anything?

Do you remember who is conducting the survey? Who?

In your own words, what is the purpose of the survey?

How can you respond to this survey?

When are responses to the survey due?

Would you participate in this survey if you were selected to do so? Why or why not?

If so, How would you respond to survey (i.e., mail or web)? Why? If not, how would you normally 

prefer to participate in surveys?

After the participants have answered the recall questions, hand the cover letter back to them. If 

participants could not recall the information asked about above, ask if they can locate the 

information after reviewing the information a second time. Note their ability/inability to do so. 

Then ask the following questions:

(Reactions)

Is there anything on this letter than you don’t understand or are confused about?

When this statement reads, “This survey is part of the Census of Agriculture Program which is 

required and protected by law (Title 7, U.S. Code and the Confidential Information Protection and 

Statistical Efficiency Act). These federal laws require that NASS keeps your identity and your 

answers confidential?” What does this mean to you??

How confident are you that your data will be kept confidential? Can you tell me more about that? 

(note positive and negative reactions)

What is your initial reaction to the following “Responding online is the easiest, fastest, and safest 

way to complete your questionnaire and it saves taxpayer dollars too”? Does this persuade you in 

any way to respond to the survey online? How so? (note positive and negative reactions)

What information on this letter do you feel is the most important to you? 

Is there any information that you feel is unnecessary? 

Is there any information that you feel is missing? 

Organic Reminder Letter

Finally, hand the participant a copy of the reminder letter and the survey instrument. 

Have participants look over the reminder letter. Tell participant: Pretend you just received this 

letter and survey in the mail. I want you to look the letter over however you normally would. If the 

participant doesn’t look at the letter at all and says something like “I’d throw it away” make a note 

of this and then ask them to look over the letter.

Note any comments they make while reading the letter. When they are finished take the reminder

letter away and ask the following questions:
13

(Reactions)

How much of this letter did you read? Did you read this letter in full, skim, didn’t read anything? 

What information stood out to you when reading this letter?

What information on this letter do you feel is the most important to you? 

Please look at the following statement: This survey is part of the Census of Agriculture Program 

which is required and protected by law (Title 7, U.S. Code and the Confidential Information 

Protection and Statistical Efficiency Act).Can you tell me in your own words what this statement 

is saying? Does this persuade you in any way to respond to the survey? Why or why not? 

If the participant does not discuss the part of the statement that says the survey is required by law, 

ask them about it. What is your initial reaction to this? Does this influence your decision to 

participate? 

Is there any information that you feel is unnecessary? 

Is there any information that you feel is missing? 

Is there any information in this letter that you find confusing or have a question about?

If participant indicated they would not complete this survey during testing of the cover letter, ask: 

Would this letter persuade you to complete the survey? Why or why not?

Do you have any other general comments about these materials?

End the interview and thank the participant for their time. 

Interview Notes

Interviewers are expected to spend up to one hour per interview writing and summarizing their 

notes from that interview. Please provide detailed notes from the interviews. Notes should be 

emailed to Heather Ridolfo (heather.ridolfo@nass.usda.gov) no later than October 3, 2014. 
14

Screener Questionnaire

Hi, my name is <name> and I am calling from the United States Department of Agriculture, 

National Agricultural Statistics Service. We are contacting operations to ask for help in testing 

the 2014 Organic Survey. This survey will provide important information on organic production 

across the United States. Is this something that you’d be willing to speak to us about? 

In order to improve the information we distribute on this survey, we would like to get your 

feedback on the promotional materials we will be sending out. This would take about an hour of 

your time and I could schedule a time that is convenient for you. During this time I will ask you 

to look over our promotional materials and provide your feedback on how we can improve them. 

Could I schedule a time to meet with you and get your input on this?

One of the criteria for this testing is you must have organic crops according to USDA’s National 

Organic Program standards and are either certified or exempt from certification (under $5,000 in 

annual sales). 

Does your operation have any organic production according to NOP standards and is certified or 

exempt from certification? 

YES – [Continue]

NO - [Thank participant for their time and hang up] 

This means you are eligible to help us out. I have the following days and timeframes open. 

<dates, times>

Which of these days and times works best for you? 

I will call you to confirm the appointment a day or two before. If you need to change the 

appointment time, please call me back at <phone number>.
15

2014 Organic Survey – Promotional Materials

Usability Testing Interview Guide

September 26, 2014

Interviewer:

Participant:

Date: 

Before we begin, I want to tell you a little more about the project and what we will be doing today.

I have here the promotional materials that will be used for the 2014 Organic Survey. We want to 

make sure that these materials are informative and that everyone understands the information 

presented in these materials. I’m going to have you look them over and then I will ask you some 

follow up questions. I’m interested in your general reactions to these materials – good and bad. 

Again, everything we discuss today will be kept completely confidential. Do you have any questions 

before we get started?

Postcard

[Hand the participant the postcard and tell them to take a look at it. Tell participant: Pretend 

you just received this postcard in the mail. I want you to look it over however you normally 

would.If the participant doesn’t look at the postcard at all and says something like “I’d throw 

it away” make a note of this and then ask them to look over the postcard.

Note any comments they make while looking it over. After they have finished looking at it 

ask the following questions.]

Comments: 

If you received this postcard in the mail what would be your initial reaction? 

Is there any information that stands out to you? 

Is there any information on this postcard that you find confusing or have a question about?

If there any information you think is missing? 
16

Do you feel it is helpful to receive a postcard like this? 

Organic Cover Letter

[Hand the participant a copy of the cover letter and the survey instrument. Have participants

read the cover letter. Tell participant: Now I want you to pretend you received this letter and 

survey in the mail. I want you to look it over however you normally would. If the participant

doesn’t look at the letter at all and says something like “I’d throw it away” make a note of 

this and then ask them to look over the letter. 

Note any comments they make while reading the letter. When they are finished take the cover 

letter away and ask the following questions.]

Comments: 

How much of this letter did you read? Did you read this letter in full, skim, didn’t read anything?

Do you remember who is conducting the survey? Who?

In your own words, what is the purpose of the survey?

How can you respond to this survey?
17

When are responses to the survey due?

Would you participate in this survey if you were selected to do so? Why or why not?

If so, How would you respond to survey (i.e., mail or web)? Why? If not, how would you normally 

prefer to participate in surveys?

[Hand the cover letter back to participants. If participants could not recall the information 

asked about above, ask if they can locate the information after reviewing the information a 

second time. Note their ability/inability to do so. Then ask the following questions.]

Notes:

Is there anything on this letter that you don’t understand or are confused about?

When this statement reads, “This survey is part of the Census of Agriculture Program which is 

required and protected by law (Title 7, U.S. Code and the Confidential Information Protection and 

Statistical Efficiency Act). These federal laws require that NASS keeps your identity and your answers 

confidential?” What does this mean to you?? 
18

How confident are you that your data will be kept confidential? Can you tell me more about that? 

(note positive and negative reactions)

What is your initial reaction to the following “Responding online is the easiest, fastest, and safest 

way to complete your questionnaire and it saves taxpayer dollars too”? Does this persuade you in 

any way to respond to the survey online? How so? (note positive and negative reactions)

What information on this letter do you feel is the most important to you? 

Is there any information that you feel is unnecessary? 

Is there any information that you feel is missing? 

Organic Reminder Letter

[Hand the participant a copy of the reminder letter and survey instrument. Have participant

look over the reminder letter. Tell participant: Pretend you just received this letter and survey 

in the mail. I want you to look the letter over however you normally would. If the participant

doesn’t look at the letter at all and says something like “I’d throw it away” make a note of 

this and then ask them to look over the letter.

Note any comments they make while reading the letter. When they are finished take the 

reminder letter and instrument away and ask the following questions.]

Comments:

How much of this letter did you read? Did you read this letter in full, skim, didn’t read anything? 
19

What information stood out to you when reading this letter?

What information on this letter do you feel is the most important to you? 

Please look at the following statement: This survey is part of the Census of Agriculture Program 

which is required and protected by law (Title 7, U.S. Code and the Confidential Information 

Protection and Statistical Efficiency Act).Can you tell me in your own words what this statement 

is saying? Does this persuade you in any way to respond to the survey? Why or why not? 

If the participant does not discuss the part of the statement that says the survey is required by law, 

ask them about it. What is your initial reaction to this? Does this influence your decision to 

participate? 

Is there any information that you feel is unnecessary? 

Is there any information that you feel is missing? 

Is there any information in this letter that you find confusing or have a question about?

If participant indicated they would not complete this survey during testing of the cover letter, ask: 

Would this letter persuade you to complete the survey? Why or why not?

Do you have any other general comments about these materials?
 

O

S

A

United States Department of Agriculture National Agricultural Statistics Service Risk Management Agency

 APPENDIX B: PROMOTIONAL MATERIALS USED DURING 

20
Top 3 Reasons to Respond to 

the 2014 Organic Survey:1. Your operation and your

products are a valuable part of the ag industry and should be counted – organic products were a $35 billion industry in the United States in 2013.2. Your input will help

re-evaluate Federal crop insurance programs to ensure organic producers receive equitable insurance prices.3. Your responses arecompletely con

fi dential, as 

required by Federal law.Please respond! Learn more at www.nass.usda.gov. 

21
United States Department of Agriculture

National Agricultural Statistics Service

In Cooperation with Risk Management Agency

January 02, 2015 

Dear Agricultural Producer, 

The U.S. Department of Agriculture’s (USDA) National Agricultural Statistics Service (NASS) is 

conducting the 2014 Organic Survey. This survey gathers detailed production, sales and marketing 

information on organic farming in the United States. Please take the time to fill out your survey form.

The 2014 Organic Survey is important because the results will help USDA’s Risk Management 

Agency shape policy decisions, including re-evaluating Federal crop insurance programs to ensure 

organic producers receive equitable insurance prices. 

Your response is critical to help protect your investments and support your future success. Please 

respond promptly in either of the following ways: 

• Online at www.agcounts.usda.gov. Responding online is the easiest, fastest and safest way to

complete your questionnaire and it saves taxpayer dollars too. All you need is the identification

number on the enclosed form to begin.

• By mail. Complete the enclosed form and mail it back in the return envelope provided.

This survey is part of the Census of Agriculture Program which is required and protected by law 

(Title 7, U.S. Code and the Confidential Information Protection and Statistical Efficiency Act). These 

federal laws require that NASS keeps your identity and your answers confidential. 

If you have questions, please call (888) 424-7828. Thank you in advance for your participation. 

Sincerely,

Joseph Reilly 

Administrator, National Agricultural Statistics Service

United States Department of Agriculture 

Enclosures 

USDA is an equal opportunity provider and employer.

22
United States Department of Agriculture

National Agricultural Statistics Service

In Cooperation with Risk Management Agency

February 17, 2015 

Dear Agricultural Producer, 

According to our records, the U.S. Department of Agriculture’s National Agricultural Statistics 

Service (NASS) has not received your completed 2014 Organic Survey. Please complete and return 

your survey today. If you have recently responded, thank you. 

Responses to the most recent Census of Agriculture let us know that organic agriculture is growing in 

the United States, with sales increasing 82 percent between 2007 and 2012. Your participation in this 

survey is important because every response is a step closer to Federal programs that better meet the 

needs of organic producers like you. 

If you have not yet responded, you can do so in either of the following ways: 

• Online at www.agcounts.usda.gov. Responding online is the easiest, fastest and safest way to

complete your questionnaire and it saves taxpayer dollars too. All you need is the identification

number on the form to begin.

• By mail. Complete the form and mail it back in the return envelope provided.

If a response is not received a NASS representative may personally follow up with you to gather this 

information. This survey is part of the Census of Agriculture Program which is required and 

protected by law (Title 7, U.S. Code and the Confidential Information Protection and Statistical 

Efficiency Act). These federal laws require that NASS keeps your identity and your answers 

confidential. 

To learn more about this survey visit www.nass.usda.gov. If you have questions or need assistance, 

call (888) 424-7828. 

Sincerely,

Joseph T. Reilly

Administrator, National Agricultural Statistics Service

United States Department of Agriculture 

Enclosures 

USDA is an equal opportunity provider and employer.

23
United States Department of Agriculture

National Agricultural Statistics ServiceRisk Management Agency

2

Org

Survey 

arrive s

APPENDIX C: REVISED PROMOTIONAL MATERIALS

2

4
Responding to the 2014 

Organic Survey is important!Your agriculture operation and products are a valuable part of the ag industry and should be counted.1. Total organic sales by farms in Total organic sales by farms inthe U.S. increased by 83 percent between 2007 and 2012. een 2007 and 2012.2. Sales from farms with certi

fi ed

or exempt organic product sales totaled more than $3.1 billion in 2012.3. Organic agriculture producersreported direct-to-consumers sales more often than conventional producers, with 42 percent of organic farmers selling percent of organic farmers selling directly to consumers in 2012.Please respond! Learn more at www.nass.usda.gov. 

25
United States Department of Agriculture 

National Agricultural Statistics Service 

In Cooperation with Risk Management Agency 

USDA is an equal opportunity provider and employer. 

January 02, 2015 

Dear Agricultural Producer, 

According to the most recent Census of Agriculture, organic agriculture sales from farms in the 

United States increased 83 percent between 2007 and 2012. These data show the demand for, value 

of and potential for continued growth in certified organic products. 

Now the U.S. Department of Agriculture’s (USDA) National Agricultural Statistics Service (NASS) 

is conducting the 2014 Organic Survey to gather more detailed production, sales and marketing 

information from organic farms in the United States. Please complete and return your form by 

February 13, 2015.

Your response is critical to help support the continued future success of organic agriculture 

production. Please respond promptly in either of the following ways:

 Online at www.agcounts.usda.gov. All you need is the identification number on the enclosed

form to begin.

 By mail. Complete the enclosed form and mail it back in the return envelope provided.

As part of the Census of Agriculture Program, this survey and your response is required by law (Title 

7, U.S. Code). These federal laws also require that NASS keeps your identity and your answers 

confidential. 

If you have questions, please call (888) 424-7828 or email nass@nass.usda.gov. Thank you in 

advance for your participation. 

Sincerely,

Joseph T. Reilly

Administrator, National Agricultural Statistics Service 

United States Department of Agriculture 

Enclosures 

26
United States Department of Agriculture 

National Agricultural Statistics Service 

In Cooperation with Risk Management Agency 

USDA is an equal opportunity provider and employer. 

January 15, 2015 

Dear Agricultural Producer, 

According to the most recent Census of Agriculture, organic agriculture sales from farms in the 

United States increased 83 percent between 2007 and 2012. These data show the demand for, value 

of and potential for continued growth in certified organic products. 

Now the U.S. Department of Agriculture’s (USDA) National Agricultural Statistics Service (NASS) 

is conducting the 2014 Organic Survey to gather more detailed production, sales and marketing 

information from organic farms in the United States. Please complete and return your form by 

February 13, 2015.

Your response is critical to help support the continued future success of organic agriculture 

production. Please respond promptly in either of the following ways:

 Online at www.agcounts.usda.gov. All you need is the identification number on the enclosed

form to begin.

 By mail. Complete the enclosed form and mail it back in the return envelope provided.

As part of the Census of Agriculture Program, this survey and your response is required by law (Title 

7, U.S. Code). These federal laws also require that NASS keeps your identity and your answers 

confidential. 

If you have questions, please call (888) 424-7828 or email nass@nass.usda.gov. Thank you in 

advance for your participation. 

Sincerely,

Joseph T. Reilly

Administrator, National Agricultural Statistics Service 

United States Department of Agriculture 

Enclosures 

27
United States Department of Agriculture 

National Agricultural Statistics Service 

In Cooperation with Risk Management Agency 

USDA is an equal opportunity provider and employer. 

February 17, 2015 

Dear Agricultural Producer, 

In 2012, there were 14,326 farms with certified or exempt organic product sales totaling more than 

$3.1 billion. The latest Census of Agriculture data are used to help determine federal and state 

programs, research, and support to better meet the needs of organic agriculture producers like you. 

Now, the results of the 2014 Organic Survey will provide even more in-depth knowledge of your 

significant and growing part of the agriculture industry. 

According to our records, the U.S. Department of Agriculture’s National Agricultural Statistics 

Service (NASS) has not received your completed response. Please complete and return the 2014 

Organic Survey by February 27, 2015. If you have recently responded, thank you. 

Please respond in either of the following ways: 

 Online at www.agcounts.usda.gov. All you need is the identification number on the form to

begin.

 By mail. Complete the form and mail it back in the return envelope provided.

If a response is not received a NASS representative will follow up with you to gather this 

information. As part of the Census of Agriculture program, this survey and your response is required 

by law (Title 7, U.S. Code). These federal laws also require that NASS keeps your identity and your 

answers confidential.

To learn more about this survey visit www.agcensus.usda.gov. If you have questions or need 

assistance, call (888) 424-7828 or email nass@nass.usda.gov. 

Sincerely, 

Joseph T. Reilly 

Administrator, National Agricultural Statistics Service 

United States Department of Agriculture 

Enclosures 

28