Confidential - Internal Use Only

 

Evaluating the June Area Survey’s Field

Enumerator Training

by

Michael Gerling & Tyler Wilson

Research & Development Division

Justin Van Wart, Ph.D

Northern Plains Regional Field Office

Joseph Rodhouse

Research & Development Division

Andy Higgins

National Operations Division

United States

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-19-01

March 2019
i

Acknowledgements

Executive Sponsors

Linda J. Young, Ph.D

Barbara Rater

Response Rate Research Team Manager

Gerald Tillman

NASDA Response Rate Research Sub-Team

Rebecca Dubbs – Sub-Team Lead

Michael Gerling

Andy Higgins

Beckie McCracken

Marcella Simmons

Shareefah Williams

Research and Development Division

Tyler Wilson

Joseph Rodhouse

Heather Ridolfo, Ph.D

Shane Ball, Ph.D

Northern Plains Regional Field Offices

Justin VanWart, Ph.D

Census Survey Division

Suzanne Avilla

Dawn Grahm

Methodology Division

Kathy Ott

National Operations Division

Chris Gotschall
ii

Executive Summary

The National Agricultural Statistics Service (NASS) conducts over 400 agricultural surveys

annually to make estimates on crops and livestock, explore production practices, and identify

economic trends.

In 2015, NASS created the Response Rate Research Team (RRRT) to identify ways to improve

response rates. The National Association of State Departments of Agriculture (NASDA) Training

sub-team was formed to focus on telephone and field enumerator training. One of the sub-team’s

tasks was to look at improving current field enumerator training. In early 2017, the sub-team

created pre- and post-field enumerator workshop evaluation forms for NASS’s 12 regional field

offices (RFOs) to use at their June Area Survey (JAS) training workshops.

The JAS is an area-frame-based annual survey that provides information on U.S. crops, livestock,

grain storage capacity as well as number, type, and size of farms. The JAS sample is comprised of

approximately 10,000 designated land areas (segments). A typical segment is about one square

mile--equivalent to 640 acres. NASS Field interviewers are provided a paper aerial photo showing

the sampled segment area and must account for all land inside the segment boundary. They divide

each segment into tracts of land that represent unique operating arrangements. Field enumerators

visit the segments, locate and interview the operator(s) of any land found to have agricultural

activity, and record all agricultural activity associated with the operator on a paper questionnaire.

Prior to the start of the screening and date collection stages, field enumerators attend a training

workshop.

The study showed that overall, field enumerators found that the JAS training workshops increased

their knowledge about the survey and increased their perceived confidence in explaining the

purpose of the survey. This increase was noted across all field enumerators regardless of their level

of experience with the JAS (from new hires to those having over 10 years of service).

The study also revealed other areas of the JAS training needing improvement. Field interviewers

indicated that the trainers should spend more time conducting group exercises and instructing on

how to properly complete Section D (Crops and Land Use on Tract) of the questionnaire and to

draw out tracts and fields on the aerial photos.
iii

Recommendations

 Continue conducting in-person field enumerator training workshops.

o Spend more time on how to complete Section D (Crops and Land Use on Tract) of

the questionnaire

o Spend more time on how to draw out tracts and fields on the aerial photos.

o Spend more time conducting group exercises.

 Explore the resources necessary for establishing a survey monkey license and additional

staffing resources to maintain the data collection instruments.

 Compare the effectiveness of three training options: 1) In-person training 2) Online selfpaced training and 3) Instructor-led online training.
iv.

Table of Contents

 

 

Abstract 1

1.0 Introduction 2

2.0 Goals 2

3.0 June Area Survey 3 

4.0 NASS’s Current Training of Field Enumerators 3

5.0 NASS’s Past Research on Field Enumerator Training 3

6.0 External Research on Field Interviewer Training 3

7.0 The Study 4

8.0 Methods, Analyses & Findings 5

8.1 The Screening Process and Screening Form by Years of Service 5

8.2 The Purpose of Section D and How to Use It 6

8.3 The Purpose of the Other Parts of the Questionnaire 7

8.4 How Confident are you in Your Ability to Explain the Purpose of this Survey 7

8.5 Text Analysis of Opened Ended Question 8

8.5.1 Text Analysis of Pre-Evaluation Form 8

8.5.2 Text Analysis of Post-Evaluation Form 8

9.0 Discussion and Lessons Learned 8

10.0 Recommendations 10

11.0 Conclusion 10

12.0 References 11

Appendices

A Appendix A - JAS Definitions A-1

C Appendix B - Pre-Training Evaluation (On-line version) B-1

D Appendix C - Post-Training Evaluation (On-line version) C-1

1

Evaluating the June Area Survey’s Field Enumerator Training

Michael Gerling, Tyler Wilson, Justin Van Wart,

Joseph Rodhouse and Andy Higgins1/

Abstract

The National Agricultural Statistics Service (NASS) conducts over 400 agricultural

surveys annually to make estimates on crops and livestock, explore production

practices, and identify economic trends. In 2015, NASS created the Response Rate

Research Team (RRRT) to research how to improve response rates. A sub-team

was formed to focus on telephone and field enumerator (interviewer) training. One

of the sub-team’s tasks was to look at improving the current field enumerator

training. In 2018, the sub-team studied the effectiveness of its June Area Survey

(JAS) field enumerator training workshops using pre- and post-training evaluation

forms.

The study showed that overall, field enumerators feel that the JAS training

workshops are increasing their knowledge about the survey and thus their perceived

confidence in explaining the purpose of the survey. This increase is seen across all

field enumerators from new hires to those having over 10 years of service.

The study also revealed other areas of the JAS needing improvement. Field

interviewers indicated that trainers should spend more time conducting group

exercises and instructing on (1) how to properly complete Section D (Crops and

Land Use on Tract) of the questionnaire and (2) how to draw out tracts and fields

on the aerial photos.

Key Words: training, agriculture, workshops, field enumerators, surveys

__________________ 

1/ Michael W. Gerling - Mathematical Statistician & Tyler Wilson - Survey Methodologist of the National Agricultural

Statistics Service’s Research & Development Division located at 1400 Independence Avenue, SW.,

Washington, DC 20250-2001. Justin VanWart – Agricultural Statistician of NASS’ Northern Plains Regional

Field Office located at 100 Centennial Mall N, Suite 263, Lincoln, Nebraska, 68508. Joseph Rodhouse –

Research Statistician from NASS’s Research and Development Division. Andy Higgins – Agricultural

Statistician from NASS’ National Operations Division located in 9700 Page Avenue, Suite 400 St. Louis, MO

63132.
2

1. Introduction

The National Agricultural Statistics Service’s (NASS) primary purpose is to provide timely,

accurate and usefulstatistics on United States’ and Puerto Rico’s agriculture. NASS conducts more

than 400 agricultural surveys annually for the purpose of producing estimates on crops and

livestock, exploring production practices, and identifying economic trends. The data collection

modes employed are mail, fax, telephone, personal interview, and the internet.

In 2015, NASS formed the Response Rate Research Team (RRRT) to determine how to improve

response rates. Multiple sub-teams were established to examine various aspects of the data

collection process and to obtain feedback from NASS customers (farmers, data users, agricultural

associations, NASS staff, and National Association of State Department of Agriculture (NASDA)

enumerators). NASDA employs the telephone and field enumerators for all of NASS’s data

collection activities.

The NASDA Training sub-team was formed to focus on telephone and field enumerator training.

One of the sub-team’s tasks was to look at how to improve field enumerator training.

In early 2017, the sub-team developed and disseminated pre- and post-field enumerator training

evaluations to NASS’s 12 regional field offices (RFOs) to measure the effectiveness of their June

Area Survey (JAS) training workshops. A formal study was never intended. These evaluations

were provided to the RFOs to use as desired. Some, but not all, regions used the forms. After the

training workshops, the sub-team received copies of the evaluation forms. However, after

reviewing them, the sub-team determined that these evaluations would not be practical to draw

statistically valid generalizable conclusions. These findings were not surprising since the forms

were not originally intended to be used in a study. NASS’s Northern Plains RFO (NPRO)

converted the paper evaluation forms to web questionnaires utilizing Survey Monkey. As a result,

the information collected was in an electronic format, making the output readily available for

statistical analysis.

In late 2017, the sub-team decided to conduct a formal study taking the lessons learned from their

earlier efforts. The sub-team originally focused on the Agricultural Resource Management Survey

Phase III (ARMS III). However, resources to design, test, and deploy survey evaluations were

unavailable. In addition, it wastoo late to make changes to the ARMS program. The nextscheduled

field enumerator workshop was for the June Area Survey (JAS). Thus, working jointly with

NASS’s Training Group, the JAS was selected.

2. Goals

The goals of this research were: 1) determine whether field enumerator training is effective; 2)

describe NASS’s current field enumerator training; 3) research NASS’s past research in evaluating

field enumerator training; 4) analyze the pre- and post-training evaluations; and 5) provide

recommendations to improve field enumerator training.
3

3. June Area Survey

The JAS is an annual survey that provides information on U.S. crops, livestock, grain storage

capacity and the number, type, and size of farms. The JAS sample is comprised of approximately

10,000 designated land areas (segments). A typical segment is about one square mile--equivalent

to 640 acres. Each segment is outlined on an aerial photo and provided to NASS field interviewers

(enumerators). Field enumerators visit these segments, locate and interview the operator(s) of any

land found to have agricultural activity, and record all agricultural activity associated with the

operator on a paper questionnaire. A separate paper questionnaire is completed for each

agricultural operation operating any land within the segment. Appendix A provides definitions of

segments, tracts, agricultural tracts, etc.

4. NASS’s Current Training of Field Enumerators

Before the JAS workshop, field enumerators are sent an Interviewer’s Manual, a copy of the JAS

questionnaire, and practice exercises. The training workshop generally involves a review of the

JAS questionnaire including the most difficult sections and a review and handout of JAS

assignments, aerial photos, and questionnaires for each field enumerator. During the workshop,

role playing exercises as well as various practice exercises are conducted. The workshop also

covers how to handle refusals and inaccesibles and how to handle other surveys that are

overlapping with the JAS. Time is also devoted to updating field enumerators’ iPads. Although

currently not used for JAS, the iPADs are being used for those surveys running concurrently with

JAS. In 2018, the training budget was reduced causing some RFOs to limit their workshops to

only the enumerators’ supervisors. The supervisors would afterwards train their enumerator staff

as needed. This was a major change from previous years.

5. NASS’s Past Research on Field Enumerator Training

Typically, at the conclusion of a training workshop, attendees are provided with a post-workshop

evaluation with the typical questions pertaining to what went well and what areas need

improvement. These evaluations are reviewed and used to improve the following year’s training.

For multiple-day workshops, some of NASS’s 12 RFOs conduct daily evaluations to understand

what topics need to be review the next day. In addition, the evaluation forms are not standardized

across each RFO. None of these efforts have led to a formal research report.

6. External Research on Field Interviewer Training

A number of studies point to the importance of evaluating training. The following are some of the

highlights.

In Evaluating Workshops and Institutes Practical Assessment, Ayers (1989) stated that evaluating

training workshops show 1) the real worth of a training program, 2) how to improve future

workshops, and 3) justification for current funding and become the basis for future funding.
4

Beth Kanter (author, speaker, and master trainer) recommends including an evaluation survey for

workshops. The survey should have questions to measure participants’ views on whether the

instructor has been an effective teacher and whether the workshop has been effective in advancing

the participants’ learning. Stiegler & Biedinger (2016) stated that the optimization of data quality

begins with the selection of the interviewers, continues in interviewer training, and ends with

interviewer monitoring.

In 1959, Donald Kirkpatrick created the Four-Level Training Evaluation Model to help trainers

measure the effectiveness of their training in an objective way. This model has evolved over time

and has become the gold standard. Today, the four-levels are as follows: 1) Reaction, 2) Learning,

3) Behavior and 4) Results. Kirkpatrick, (1996) also mentions the importance of keeping in mind

the practicality and expenses when evaluating training.

The Kirkpatrick Model has been modified by companies over time to the following: 

 Pre-training skills assessment: Measure the learner’s level of knowledge or skill prelearning and again post learning.

 Application in the workplace: Assess whether the newly learned skills or knowledge are

being applied in the workplace.

 Individual behavior assessment: Assess whether the training improved the individual’s

embracement of the corporate culture and goals.

 Team behavioral assessment: Measure whether teamwork is being more coherent and

effective.

 Meeting goals or targets: Measure the individual/team performance against goals or

targets every few months to assess the impact of the learning.

7. The Study

After a general review of the 2017 pre- and post-field enumerator training evaluation forms and

receiving technical guidance from NASS’s survey methodologists, Kathy Ott and Heather Ridolfo,

the 2018 field enumerator training evaluations were re-designed. The questionnaire was

comprised of a series of 10-point scale questions. This was based on research by Wittink and

Bayer, (1994). Additionally, the sub-team decided to add several open-ended questions.

Justin VanWart of the NPRO updated the Survey Monkey web survey versions accordingly.

Screen shots of the pre- and post-workshop evaluations are provided in Appendices B and C.

The 12 regional offices were given the option to participate in the research study. The Delta,

Eastern Mountain, Great Lakes, Heartland, Mountain, Northern Plains, Northwest, Pacific,

Southern Plains, and Upper Midwest agreed to participate.
5

8. Methods, Analyses & Findings

The regions participating in the study provided their feedback on using and receiving the pre- and

post-workshop evaluations. The training evaluation data were downloaded from Survey Monkey,

processed, and analyzed. Basic statistics from Survey Monkey were disseminated to the respective

RFOs.

There were 1,108 partially completed/completed pre-evaluation forms and 819 completed postevaluation forms. Duplication caused by multiple submissions by field enumerators were removed

from both data sets. Records with missing or erroneous enumerator IDs and those having only one

of the two evaluations forms completed were also removed from the data sets. Thus, the total

number of good matches was 534 or 48 percent. In the future, the number of missing/erroneous

IDs will be reduced by adding a “hard” edit check in the enumerator ID field and perhaps a more

sophisticated edit can be programmed to determine whether the field enumerator has completed a

pre-evaluation before providing the post evaluation.

The Wilcoxon Signed Rank Test was utilized to assess whether significant differences in perceived

knowledge were found after completing the training. The Wilcoxon signed-rank test is a nonparametric statistical hypothesis test used to compare two related samples, matched samples, or

repeated measurements on a single sample to assess whether their population mean ranks differ

(i.e. it is a paired difference test) (Wilcoxon, Katti & Wilcox, 1963). Each question was evaluated

pre and post training.

8.1 The Screening Process and Screening Form by Years of Service

Overall, trainees experienced a significant increase in knowledge about the screening form and

process. However, according to the values in the actual estimate column (Table 1), new employees

(0-3 years) experienced the greatest increase in perceived knowledge with an average increase of

2.28 points after training. The training had the smallest effect (yet still significant, p < 0.01) on

more experienced employees. The evidence shows that the training was a success in increasing the

enumerators’ perceived knowledge of the survey.

Table 1: Screening Process and Screening Form by Years of Service

Years of Service

Wilcoxon

Signed Rank PValue

Paired T-Test PValueActual Estimate

Absolute Value N

0-3 Years <0.001 <0.001 2.28 237

4-9 Years <0.001 <0.001 0.76 135

10+ Years <0.001 <0.001 0.30 158

All <0.001 <0.001 1.3 534
6

8.2 The Purpose of Section D and How to Use It?

Field enumerators consistently stated that Section D was a difficult section of the questionnaire

due to its complexity. Overall, trainees experienced a significant increase in knowledge in how to

complete Section D. According to the values in the actual absolute estimate column (Table 2), new

employees (0-3 years) experienced the greatest increase of perceived knowledge (an average 2.31

point increase after training). Older employees (10+ years) experienced the least amount (yet still

significant) of knowledge gained. These findings suggest that the majority of enumerators

increased their knowledge of Section D, especially those with little experience.

Table 2: Purpose of Section D and How to Use It?

Years of Service

Wilcoxon

Signed Rank PValue

Paired T-Test PValueActual Estimate

Absolute Value N

0-3 Years <0.001 <0.001 2.31 235

4-9 Years <0.001 <0.001 0.88 133

10+ Years 0.003 0.001 0.29 154

All <0.001 <0.001 1.35 526

8.3 The Purpose of the Other Parts of the Questionnaire?

Field enumerators were asked for their thoughts pertaining to the rest of the questionnaire. Overall,

trainees perceived a significant increase in knowledge about the purpose of other parts of the

questionnaire. According to the values in the actual estimate absolute column (Table 3), new

employees (0-3 years) experienced the greatest increase (an average 2.39 point increase after

training). Older employees (10+ years) experienced the least amount (yet still significant) of

knowledge gained.

Table 3: The Purpose of the Other Parts of the Questionnaire

Years of Service

Wilcoxon

Signed Rank PValue

Paired T-Test PValueActual Estimate

Absolute Value N

0-3 Years <0.001 <0.001 2.39 237

4-9 Years <0.001 <0.001 0.84 133

10+ Years <0.001 <0.001 0.35 150

All <0.001 <0.001 1.39 521
7

8.4 How Confident are you in Your Ability to Explain the Purpose of this Survey?

Overall, trainees experienced a significant increase in their perceived confidence on how to explain

the purpose of the survey. According to the values in the actual estimate column, new employees

(0-3 years) experienced the greatest increase (an average2.01 point increase after training). Older

employees (10+ years) experienced the least amount (yet still significant, p < 0.01) of knowledge

gained. Knowing the purpose of the survey and having self-confidence are key aspects in having

a successful data collection effort and improved data quality.

Table 4: How Confident are you in Your Ability to Explain the Purpose of this Survey?

Years of Service

Wilcoxon

Signed Rank PValue

Paired T-Test PValueActual Estimate

Absolute Value N

0-3 Years <0.001 <0.001 2.01 235

4-9 Years <0.001 <0.001 0.96 133

10+ Years <0.001 0.002 0.31 153

All <0.001 <0.001 1.24 525

8.5 Text Analysis of Opened-ended Questions

A text analysis using SAS 9.2 and Text Explorer in SAS JMP 13 Pro was conducted on the openended questions 6-9 of the pre-training evaluation and questions 6-11 of the post-training

evaluations. See Appendices C and D for copies of the evaluations.

8.5.1 Text Analysis of Pre-Evaluation Form

Questions 6 and 7 pertain to the field enumerators’ perception of the most difficult sections of the

questionnaire and what they would like to have covered more in the training workshop. 

According to term and phrase frequency lists, Section M (Land Values) and Section D were

mentioned over 100 times as being the most difficult for JAS’s respondents to complete. In

addition, Section D was mentioned approximately 100 times when enumerators were asked what

additional information they wanted covered at the training workshop.

Questions 8 and 9 pertained to concerns about enumeration and reasons for non-response. The

number one concern that field enumerators mentioned was time (or lack thereof) of the

farmers/ranches sampled. The overarching theme in these responses, according to term and phrase

lists, is that there are too many surveys and not enough time for the farmers/ranchers to complete
8

them. In addition to lack of time, issues with government trust and busy respondents were the most

mentioned reasons for nonresponse as perceived by field enumerators in Question 9.

8.5.2 Text Analysis of Post-Evaluation Form

Question 6 pertains to the field enumerators’ perception of whether the survey was explained

adequately. According to the post survey response, 97 percent (517) said “yes” when asked if all

aspects of the survey were covered during the training.

Questions 7 and 8 ask for the field enumerators’ perception on what they found most and least

useful in the training. According to Question 7 and Figure 1, the field enumerators thought that

the Section D training was the most useful. Based on the responses in Question 8, a majority of

people indicated that all the information was useful in varying degrees.

Figure 1: Word Cloud of Most Useful Training Items

Question 9 pertains to areas of the survey that field enumerators would prefer more training time

on. According to term and phrase frequency, three themes emerged: (1) Section D, (2) maps, and

(3) group practice.

Enumerators were also asked to indicate where in the training less time could be devoted to going

forward (Question 10). Responses appeared fairly random, except that many (37) mentioned

“nothing” in their answers, indicating that they perceived their time was well spent overall.

Questions 11 and 12 focused on what field enumerators would like to have done differently in the

workshop. Additional comments appear quite specific to each individual. However, there were

high frequencies of positive terms and phrases such as “nothing,” “good,” and “great.” These are

indicative that the information conveyed during the workshop was well-received.

9. Discussion & Lessons Learned

NASS continues to search for ways to improve the efficiency and overall quality of the field

enumerator training. Evaluating field enumerator training workshops provides the agency with a
9

baseline to determine whether resources allocated to training are effective. Future research could

be done to compare the effectiveness of in-person instructor led training, online self-paced training,

and instructor-led online training.

Having the training evaluations available online has several benefits: 1) edit checks to

improve data quality 2) eliminate the need for paper forms, and 3) receive results faster.

Also, the data can be reviewed once the evaluation has been submitted to make sure the

trainees are completing the form as intended. In the future, NASS may want to consider the

excerpt from John Eades’s 2014 blog on “3 Ways to Measure Training Effectiveness”. John

Eades is an author, speaker on leadership development and organizational alignment, and

CEO of an e-learning company. He states that:

“Creating a visual assessment of an employee’s skill set and performance

before and after a training moment. These snapshots, or skylines, of a learner’s

abilities can give a clear picture of performance and skill improvements you

can directly tie to training. A simple example would be, testing a sales person’s

current sales skills prior to training, then retesting the individual after the

event to see the delta.” 

At NASS, this could be a dashboard displaying each enumerator’s performance, including

skills attained, goal-oriented behaviors, teamwork skills and targets met. NASS could better

monitor interviewer performance with this type of visual assessment of employee skills pre

and post training, and apply adaptive or tailored training curriculum where the data show

an area of needed improvement.

Training evaluations and online displays could also be expanded to include the ARMS Phase III

and Objective Yield to measure the effectiveness of these field enumerator training workshops.

The regional field offices’ staff were also greatly appreciative with the insight that the training

evaluations provided.

If formal evaluation of training workshops continues, we suggest the following strategies to

improve future studies:

 Add a “hard” edit check to the enumerator ID question on the pre and post-evaluation forms

where it is not only required but also the length of the input is correct. This will reduce the

number of non-matching completed pre- and post-training evaluations.

 Program Survey Monkey to reduce the number of repeated submissions.

 Define reasons why field enumerators are not able to complete both evaluations.

 Review what the two non-participating regions are doing and try to standardize/add

questions so these regions could be added into future studies.
10

10. Recommendations

 Continue conducting in-person field enumerator training workshops.

o Spend more time on how to complete Section D (Crops and Land Use on Tract) of

the questionnaire and how to draw out tracts and fields on the aerial photos.

o Spend more time conducting group exercises.

 Explore the resources necessary for establishing a Survey Monkey license and the

additional staffing resources required to maintain the data collection instruments.

 Compare the effectiveness of three training options: 1) In-person training, 2) Online selfpaced training, and 3) Instructor led online training.

11. Conclusion

Overall, field enumerators indicated that the JAS training workshops are increasing their

knowledge about the survey and increasing their perceived confidence in explaining the purpose

of the survey. This increase is across all enumerators regardless of the level of experience but

greatest for less experienced field enumerators.

Trainers need to spend more instruction time on Sections D of the questionnaire and on drawing

tracts and fields on the aerial photos. In addition, more time should be dedicated to conducting

group exercises.

As training budgets tighten, other training options, such as online self-paced training and

instructor-led online training, should be explored. This research is a start to defining a baseline to

compare these other training options with the current in-person instructor-led training.

In conclusion, the research shows that NASS should continue to hold in-person instructor-led field

enumerator workshops for the JAS.
11

12. References

Ayers, J. (1989) Evaluating Workshops and Institutes Practical Assessment, Research &

Evaluation, Volume 1, Number 8, ISSN=1531-7714. Retrieved from

http://pareonline.net/getvn.asp?v=1&n=8, accessed on October 11, 2018.)

Berken, S. (Jan 28, 2013.) How to Run a Good Workshop, Scott Berken’s Blog. Retrieved from

http://scottberkun.com/2013/run-a-good-workshop/, accessed October 11, 2018.

DePalma, A. (March 2011) E-Training vs. In-Person Training, American Society of

Mechanical Engineers (ASME). Retrieved from https://www.asme.org/careereducation/articles/certification/e-training-vs-in-person-training, accessed on October 11, 2018.

Dobransky, M., Vanry, N. (estimated 2017) Instructor-led Training vs. eLearning, Edgepoint

Learning’s Blog. Retrieved from https://www.edgepointlearning.com/blog/instructor-ledtraining-vs-elearning/, accessed October 10, 2018.

Eades, J. (September 30, 2014) 3 Ways to Measure Training Effectiveness, eLearning Industry

website. Retrieved from https://elearningindustry.com/3-ways-measure-training-effectiveness,

accessed October 11, 2018.

Flesher, S. (May 23, 2018) How to Measure the Performance of Training Programs. Skill builder

LMS – Base Corp Learning System’s website. Retrieved from

https://www.skillbuilderlms.com/how-to-measure-the-performance-of-training-programs/,

accessed on October 11, 2018.

French Institute for Demographic Studies (INED), (estimated 2017) Interviewer Training, INED’s

website.

Retrieved from https://www.ined.fr/en/resources-methods/survey-methodology/methodologicalchoices/interviewer-training-and-data-collection-management/, accessed October 11, 2018.

Kanter, B. (February 18, 2014) Six Tips for Evaluating Your Nonprofit Training Session,

Beth’s Blog. Retrieved from http://www.bethkanter.org/training-after/, accessed on October 11,

2018.

Kirkpatrick, D. and Kirkpatrick J. (1996) Kirkpatrick’s Four-Level Training Evaluation Model -

Analyzing Training Effectiveness, Mindtools’ website. Retrieved from

https://www.mindtools.com/pages/article/kirkpatrick.htm, accessed on May 11, 2018.

Lavrakas, P. (2008) Interviewer Training, Encyclopedia of Survey Research Methods. Retrieved

from http://methods.sagepub.com/reference/encyclopedia-of-survey-research-methods/n248.xml,

accessed October 11, 2018.

NASDA (2018) About the NASDA-NASS Partnership, National Association of State
13

Departments of Agriculture’s website. Retrieved from http://www.nasda.org/nass/about,

accessed on May 11, 2018.

Neelam-Bhuyan, S. (April 23, 2016) How to Measure the Impact of Your training Program,

MindTickle’s website. Retrieved from https://www.mindtickle.com/blog/measure-impacttraining-program/, accessed on October 10, 2018.

Optimus Learning Services (January, 15, 2017) Metrics to Measure for Effective Learning &

Development Management, Optimus Learning Services’ website. Retrieved from

http://www.optimuslearningservices.com/blog/ld-strategy/5-metrics-to-measure-for-effectivelearning-and-development-management/, accessed October 11, 2018.

Rogers, M., Surrency, A. (March 2002) Child Needs Assessment Tool Kit Training Manual, A

Guide for Training Field Interviewers and Supervisors, The World Bank, Retrieved from

http://siteresources.worldbank.org/INTECD/Resources/CNAToolkitTrainingManual.pdf,

accessed October 11, 2018.

Shepherd R., Turbett P. (December 2006) Post-School Outcomes Data Collection Guide:

Training Interviewers, State University, New York at Potsdam and National Post-School

Outcomes Center University of Oregon. Retrieved from

https://transitionta.org/sites/default/files/dataanalysis/NPSO_TrainingInterviewers.pdf, accessed

October 11, 2018.

Stiegler, A., Biedinger, N. (2016) Interviewer Skills and Training, GESIS Survey Guidelines,

Mannheim, Germany: GSIS-Leibinz Institute for the Social Sciences, doi: 10.15465/gesissg_en_013. Retrieved from https://www.gesis.org/en/gesis-surveyguidelines/statistics/interviewer-training/, accessed October 11, 2018.

Survey Research Center (2016) Guidelines for Best Practice in Cross-Cultural Surveys. Ann

Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan.

Retrieved from http://ccsg.isr.umich.edu/index.php/chapters/interviewer-recruitment-selectionand-training-chapter, accessed on August 28, 2018.

United States Office of Personnel Management Employee Services Executive Resources &

Employee Development (January 2011) Training Evaluation Field Guide Demonstrating the

Value of Training at Every Level, US-OPM, 1900 E Street, NW Washington, DC 20415.

Retrieved from: https://www.opm.gov/policy-data-oversight/training-anddevelopment/reference-materials/training_evaluation.pdf

Wikipedia (2018) Definition of Wilcoxon Signed-Rank Test. Retrieved from

https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, (accessed October 11, 2018).

Wilcoxon, F., Katti, S. K., & Wilcox, R. A. (1963) Critical Values and Probability Levels for the

Wilcoxon Rank Sum Test and the Wilcoxon Signed Rank Test. American Cyanamid Company.
13

Wittink, D., Leonard, R. (1994, The Measurement Imperative, Marketing Research, Vol 6, No 4,

pp14-22.

Yupangco, J. (June 18, 2017) Learning Metrics That Matter: Data Points You Should Be

Measuring eLearning Industry’s website. Retrieved from

https://elearningindustry.com/learning-metrics-that-matter-data-points-measuring, accessed

October 11, 2018.
A-1

Appendix A

JAS Definitions

Segments: Land areas with identifiable boundaries such as ditches, roads, railroads, streams,

etc. that serve as sampling units in the June Area Survey. Segments are assigned a

permanent number and outlined in red on aerial photos. Segments generally range

in size from one-half square mile to three square miles.

Tract: An area of land inside a segment under one type of land operating arrangement.

There are two types of tracts:

1.) Ag Tract: Consists of agricultural land.

 

2.) Non-Ag Tract: Consists of residential, shopping centers, lakes, woods, and

any land not considered agricultural.

Usable: Completed reports for agricultural tracts - questionnaires containing usable data.
B-1

Appendix B

Pre-Workshop Evaluation Form
B-2

Appendix B

Pre-Workshop Evaluation Form – Continued
C-1

Appendix C

Post-Workshop Evaluation Form
C-2

Appendix C

Post-Workshop Evaluation Form - Continued