Development and Validation of an Aviation Research Survey

Surveys are one of the more commonly utilized data collection instruments in aviation research. Unfortunately, there often is little information provided as to how the surveys utilized in such studies were developed. This is problematic because the validity and construction of surveys are critical to the successful collection of data. Further, without the proper distribution methodologies in place, survey response rates can be poor. This qualitative study provides a literature-based process to assist aviation researchers to design and validate a survey instrument. First a literature-based survey development protocol was developed. Next, a validation process utilizing a panel of experts was created. Panel members were sought with experience and qualifications specific to the subject areas covered in the survey including its (a) construction, (b) implementation, (c) graduate level research, (d) higher education, (e) aviation higher education, (f) the aviation industry, (g) flight instruction and certification, and (h) demographic measures. Interviews were conducted each of the five expert panel members using a semi-structured protocol. The results of the interviews were coded using NVivo qualitative analysis software. The panel members determined that the example survey was a valid instrument to use in an applicable study. A summary of panel member feedback that could be generalized to a wide range of aviation surveys is provided. In summary, through a careful review of available literature and through the conduct of a panel of experts review, an explicit, structured process was fashioned for aviation researchers to use in order to successfully create and validate a survey instrument for use in a variety of studies. Surveys have been commonly utilized in recent aviation-related research. A review of the two most recent issues of the International Journal of Applied Aviation Research (Volume 10, Issue 1; Volume 9, Issue 2), there were 19 articles of which six (3 1.5%) included the use of a survey. Four of the articles that included a survey (66.7%) had at least one instrument that was designed by the researcher yet no information was provided as to how the instrument was created or what efforts were made to validate the survey. A similar review of the Collegiate Aviation Review (Volume 28, Issues 1 and 2) revealed 14 articles of which eight (57.1%) utilized at least one survey. Five of the articles that included a survey (62.5%) had at least one survey that was designed by the researcher, yet no information was provided as to how the instrument was created or what efforts were made to validate the survey. This review of the most recent, available aviation literature was clearly provides evidence that not only does aviation research regularly utilize surveys, little information is provided as to how the surveys are developed or if they have been validated. Purpose statement The objective ofthis qualitative research study was to determine a literature-based procedure to create an aviation-related survey. An additional objective was to identify a means to validate such a survey. Similar procedural studies have been conducted in a variety of other fields to assist fellow researchers design and implement successful, well-designed studies (Dimitrov & Rumrill, 2003; Onwuegbuzie & Leech, 2007; Peng, Lee, & Ingersoll, 2002). The findings provided here are meant to serve similar purposes for the aviation research community. Method The first step of this study was to demonstrate a JAAER, Fall 201 1 Page 45 45 Ison: Development and Validation of an Aviation Research Survey Published by ERAU Scholarly Commons, 2011 Aviation Research Survey survey development and editing procedure through an exhaustive literature review and an analysis of existing instruments designed to study aviation faculty. In addition, a thorough investigation of aviation-specific surveys, forms, and applications was conducted. Once the instrument was refined into a draft form, a qualitative analysis of the validity of the survey was conducted using an interviewbased inquiry posed to a panel of experts. The principle source of data for this component was extracted 6-om the transcriptions of expert panel interviews. Suwey Development Process The step-by-step instrument development process presented by Prochaska-Cue (1 988) was utilized to assist in the development of the survey. This process calls for the researcher to: 1. define objectives 2. defme the target population 3. review related measures 4. develop an item pool 5. prepare and pilot a prototype (Prochaska-Cue, 1988, pp. 50-5 1). Dolezal (1991) provided a similar model for the development of an instrument, though in this case, the author specifically mentions the use of a panel of experts to evaluate the prototype version. Creswell (2003) offers a checklist for the development of a survey instrument which was quite similar to those procedures presented by Prochaska-Cue (1 988) and Dolezal (1 991). These three processes acted as a model for the development of the survey instrument for this study. Definition of Objectives. First and foremost, the objectives of the current study were defined. The goal of this study was to develop a survey instrument to evaluate the career pathways, educational pathways, and demographic attributes of postsecondary professional pilot program faculty members. Before moving forward, however, Creswell(2003) argued that justification should be made to support the choice ofthe survey method as the most appropriate form of data collection. Thus, a review of the appropriateness of the application of survey methods was necessary. According to Colorado State University (2008), the survey method is an apposite method when trying to gather attributes of large populations. Also, this method was cited as being suitable for administration to persons scattered geographically. Further, surveys allow for "standardized questions [that] make measurement more precise [. . . and] ensures that similar data can be collected from groups then interpreted comparatively7' (Colorado State University, 2008). In addition, "high reliability is easy to obtain" through the apt use of surveys (Colorado State University, 2008). Ball State University (2008) stated that "surveys are used to: [...I gather information from and about large populations; make comparisons among subgroups of the population [. . . and] gather statistically representative data." These indications supported the use of survey methods in this study. Yet, as with all research methods, there are disadvantages associated with the use of surveys. It was therefore appropriate to be cognizant of these issues so that they could be mitigated as best as possible. Colorado State University (2008) identified five primary faults of the survey method: Standardization issues Inflexibility Low response rates Recall issues Context issues. Standardization issues stem 6-om the fact that surveys "force the researcher to develop questions general enough to be minimally appropriate for all respondents" (Colorado State University, 2008). In this study, standardization issues were determined not to be a factor as the group was relatively homogeneous in experience and education. Therefore, the survey was found to be suitable for all respondents. The inflexibility of surveys, i.e. that they contain all of the same questions throughout the data collection process, was also deemed not be a factor as consistency. The intention was that the survey would provide a one-time data collection which should not require adjustment during the actual collection process. This was viewed as a desirable attribute of the survey so data was standardized and thus easily comparable. It was noted that response rates could be boosted through the multiple contact methods, e.g. making four to five contacts through different media channels (email, phone, regular mail), as outlined by Dillman (2007). It was found that much attention must be made by researchers to insure high response rates. This portion of research design should receive large amounts of attention by researchers. Recall issues refers to the ability of applicants to remember details. This survey aimed to collect job-critical and demographic data which were unlikely to be difficult to recall. Lastly, although a survey does not take into account contextual factors like a direct observation would, because of the nature of the data collected, this did not appear to be an issue of concern (e.g. job history and demographic answers would be no different if collected in person versus