Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time -resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed. In this paper, we present two promising new methods for calculating IRR based on visual repr esentations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR , finding excellent agreement between our new methods and existing statistical formulae . This methods development is exemplified using data for our ongoing research , in which we are working to analyze time -resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze non - traditional data, such as those collected during eye -track ing, screen capture, or observational studies. Introduction This research paper presents two novel image -based methods for calculating interrater reliability (IRR) and compares them with statistical methods for calculating IRR. Across fields, establishi ng quality in the qualitative data analysis process involves calculating a measure of agreement between the human researchers interpreting the data: If researchers cannot agree to an acceptable level, then a coding schema cannot be considered sound and res ults cannot be considered meaningful, transferrable, or conclusive. The extent to which the classification patterns of two or more coders coincide represents the inter rater reliability , sometimes known as interrater agreement. M ethods for calculating IRR have been establis hed across the social sciences, such as those documented by Eckes [1], Zhao [2], Krippendorff [3], and Carletta [4], typical ly calc ulated for nominal data (i.e., data that can be sorted into categories that are not in any meaningful order. ) As part of our group’s ongoing work, we are interested in capturing and studying the time - resolved processes of engineering writers using screen -capture data collected over hours of authentic writing practice. The overarching motivation for the project is to capture similarities and differences in the enacted writing patterns of engineering writers to elicit heuristics and useful writing strategi es that can augment engineering students’ writing strategies in overcoming procrastination, writer’s block, and writing anxiety, which are known to plague engineering students [5]. Data for this project were collected in one semester from three graduate engineering students at a Research -Intensive University as they were applying for the National Science Foundation (NSF) Graduate Research Fellowship Program (GRFP), a competitive fellowship that requires, among other metrics of academic suc cess, a two -page research proposal and a three -page personal statement. During the three participants’ writing process, we instructed the participants to enable screen capture software on their computers before starting the writing process, which operated “in the background” to collect movie files of the screen capture. As noted in our past literature, this also captured all the realistic factors that surround writing in the real world, such as checking email, answering instant messages from friends, and changing music types —tasks not affiliated directly with composition, but that are part of an authentic writing process [5]. To date, we have outlined the choices and justifications involved in characterizing and coding “messy” data through theory -driven coding schemas; defining units of analysis; and introduced strategies for visually representing of hours of time -resolved data in easy -to-interpret figures [5]. In our past work, we simply used two raters working together to analyze the data and code to ag reement, we feel that to advance this project, it is essential to develop methods by which to calculate IRR for non-traditional, messy, and time -resolved data such that other researchers using interesting methods to capture data might find our work transfe rrable and applicable. To code our time - resolved video data, we have established methods for raters to code data in real -time using a web interface [5]; however, with that step forward, the next stem and the focus of this paper is to determine a method to calculate inter -rater reliability on complex data. Our data has some intricacies which traditional calculations for IRR become burdensome to perform . For example, our data are time-resolved and our codes are not mutually exclusive, such that we have overl apping data. Further, traditional measures of IRR have no thresholds to justify how closely two raters’ codes must temporally match: Is a lag of a half a second on the same code still “agreement?” For the purposes of this paper, we are interested in estab lishing interrater reliability only for two raters for nominal data, and will not present methods for IRR that account for more than two raters or for ordinal, ratio, or interval methods of data categorization, because these methods are not related to our context. The remainder of the paper is outlined in the following way. First, we introduce interrater reliability as a methodological requirement for qualitative research, reviewing the development of IRR and attributes and limitations of various approach es as a cohesive review for readers new to IRR. Then, we present the methods for two novel calculations of IRR employing image p rocessing techniques that avoid needlessly -complex statistics, and compare the calculations of IRR from these new techniques wit h five existing methods for statistically calculating IRR. Review of Statistical Methods determining Interrater Reliability for Nominal Data Interrater reliability can be conceptualized as a percentage agreement between two raters . Here we present a sim ple example to calculate agreement between two raters (A and B) , who are tasked to classify the same n pieces of data into either of two categories (1 and 2) . In this example, Raters A and B categorized n 11 subjects in category 1 and n 22 subjects in category 2. However, Rater A categorized n 12 subjects in category 1 , but the same n 12 subjects are been categorized in category 2 by Rater B. Similarly , Rater B categorized n 21 subjects in category 1 , but the same n 21 subjects are been categori zed in category 2 by Rater B . The n ext step is to make an agreement matrix, such as the one it Table 1 . Table 1: Agreement matrix defined for interrater estimation adapted from [6] Rater B Rater A Category 1 2 Total 1 n11 n12 a1 2 n21 n22 a2 b1 b2 n The percent agreement 𝑝𝑎, is defined as the probability of both raters coming to the same conclusion , represented by the mathematical expression, 𝑝𝑎=(𝑛11+𝑛22)/𝑛 (1) where n 11 and n 22 are the categories of agreement and n is the total number of data pieces categorized. The problem with this equation for percent agreement is it doesn’t consider the “chance agreement ,” or agreement that would happen if Rater A rando mly assigned the data into categories . To overcome this limitation, several researchers have proposed modifications to the percent agreement calculation , especially Scott, Cohen, Gwet, Krippendorff, and Brenan and Prediger , the contributions of whom we introduce here. In 1955, William Scott [7] proposed a solution for the percent chance agreement compensation ( 𝑝𝑒), developed for text categorization , a measure of the ratio between the difference between the observed and expected (chance) agreement of raters over the maximum possi ble agreement accounting for expected (chance) agreement. Scott’s Pi coefficient ( 𝜋), is 𝜋= 𝑝𝑎−𝑝𝑒 1−𝑝𝑒 (2) where 𝑝𝑎 is the observed ag reement between two raters on two categories using the percent observed agreement calculation in Equation ( 1) and 𝑝𝑒 is the expected agreement due to chance . The percent chance agreement (𝑝𝑒) calculated using joint proportions and assuming raters have the same response distribution, given by the following expression 𝑝𝑒=∑(𝜋𝑎𝑘+ 𝜋𝑏𝑘) 42𝑞 𝑘=1=(𝑎1 2𝑛+𝑏1 2𝑛)2+(𝑎2 2𝑛+𝑏2 2𝑛)2 (3) Where q is the number of categories , a corresponds to Rater A and b to Rater B, the subscripts 1 and 2 correspond to categories and 𝜋𝑥𝑘 is the probability of Rater x categorizing a subject to the kth category defined as the ratio of number of subject s in category k and total number of subject s. However, this method assumes that the chances of raters randomly assigning an item to same category is based on rater’s average distribution for each category which is not accurate representation of the experiment . In 1960, Cohen [8] estimated the expected percent chance agreement and used it to adjust the percent agreement ( 𝑝𝑎). Cohen defined the Kappa coefficient (𝜅𝑐) as: 𝜅𝑐=𝑝𝑎− 𝑝𝑒 1− 𝑝𝑒 (4) Where the chance agreement ( 𝑝𝑒) is defined as 𝑝𝑒=∑𝜋𝑎𝑘𝜋𝑏𝑘𝑞 𝑘=1=(𝑎1𝑏1+𝑎2𝑏2) 𝑛2 (5) As noted by Feinstein & Cicchetti [9], [1 0], there are two paradoxes of the kappa coefficient where the kappa coefficient is highly inaccurate. First, if the chance agreement is very large, it can convert a relatively larg e percent agreement into a very small 𝑘𝑐. For example, if rater A doesn’t categoriz e any subject into category 2 and rater B categorize s the subject such that n11 = 100 and n12 = 25, the kappa coefficient can be determined using equation s 1, 4, and 5 to achieve the following values: 𝑝𝑎=100 +0 125=0.8, 𝑝𝑒= 125 ×100 +0×20 1252=0.8 𝜅𝑐=0.8− 0.8 1− 0.8=0 In this example, the p ercent chance agreement is very high but the kappa coefficient is zero , suggesting (inaccurately) a total disagreement between raters. The s econd paradox is that a very high disagreement in distribution of subjects between raters can artificially produce high values of 𝜅𝑐. For example, if rater A and B categorized the subject such that n11 = 20, n12 = 15, n21 = 5, n22 = 20 , then t he percentage agreement ( 𝑝𝑎) based on the data in example is 0.67 and the chance agreement ( 𝑝𝑒) is 0.49 , resulting in a kappa coefficient of 0.35 which is very l ow compared to the percentage agreement. The methods disc ussed until now cannot be described as a general method for measuring IRR since they are limited by severe paradoxes. First proposed by Holley and Guilford in 1964 [12] and generalized later by Brenan and Prediger in 1981 [13] for more than two categories, the G-Index is perhaps the simplest expression for percent chance agreement compensation. It is defined as 𝑘𝑏=𝑝𝑎− 1/𝑞 1− 1/𝑞 (6) Where q is the number of categories. Since 𝑘𝑏 depen ds only on the number of categories (q) and hence, it is independent of subject rating distribution by raters making it a paradox resistant method. However, the chance agreement coefficient is not constant and depends upon the rating distribution. In 1970, Krippendorff [11] proposed an agreement coefficient named α (alpha) , in which percent chance agreement is identi cal to Scott ’s Pi coefficient , but the percentage agreement is the weighted average of the observed percent agreement ( 𝑝𝑎′). 𝑝𝑎′=(1− 𝜀𝑛)𝑝𝑎 + 𝜀𝑛 (7) Where the weight parameter ( 𝜀𝑛) is defined as 𝜀𝑛=1/2𝑛, where n is the number of subjects rated by the both raters. The weighted parameter acts as a small sample correction for percentage agreement coefficient , eliminating all the subjects rated by single rater. However, the correction in percent agreement is insignificant for larger number of subjects , yielding in the same result as Scott Pi. This method is preferred over Scott Pi coefficient when all subjects are rated by two or more raters and the number of subject is less than 10. Most recently , in 2008, Gwet [14] proposed an agreement coefficient called AC 1 to overcome the limitations associated with Cohen’ s Kappa and Scott Pi coefficient. The expression of coefficient is identical to Cohen’s Kappa and the only difference is the expression for (𝑝𝑒) 𝑝𝑒= 1 (𝑞−1)∑𝑟𝑘(1−𝑟𝑘) 𝑞 𝑘=1 (8) Where q is the number of categories, and the term rk term is the mean probability of categorizing a subject into kth category and can be calculated using following expression, 𝑟𝑘= (𝜋𝑎𝑘+ 𝜋𝑏𝑘) 2 (9) This method estimates the chance agreement by using a hybrid of average (Pi) and categorical (Kappa) distribution. This approach has been proven to be more stable in a varying marginal probability compared to Cohen’s kappa [15] making it paradox - resistant. One of the limitations of all these methods for calculating IRR is the assumption of an ideal data set with no missing ratings. In practice, though, it is reasonable to assume that a data set will not be ideal and will have missing ratings. This happens when one or both raters are not able to classify the data, a common limitation as researchers analyze data in real -time or collect observational data. In case of both rater s missing classifications , the percent agreement does not change ; however, if one rater codes a behavior and the other misses the occurrence, there will be an effect on the agreement score . To incorporate the missing ratings in agreement matrix, a new row (𝑛𝐵𝑥) and column (𝑛𝐴𝑥) is introduced to the agreement matrix . The new row and column contains q -1 elements. Column 𝑛𝐴𝑥 contains the number of subjects that only rater A has classified and similarly, Row 𝑛𝐵𝑥 contains the number of subjects classified by only rater B (and not rater A). The final generaliz ed agreement matrix is given by 𝐴= [𝑛11 𝑛𝑞1 𝑛𝑏𝑥1 𝑛1𝑞 𝑛𝑞𝑞 𝑛𝑏𝑥𝑞 𝑛𝑎𝑥1 𝑛𝑎𝑥𝑞 0] If 𝑛𝑎𝑥 and 𝑛𝑏𝑥 are the sum s of column X and row X respectively , then to compensate for the missing rating in the percent agreement calculation , the total number of subjects (n) is adjusted by the following expression, 𝑛`= 𝑛−(𝑛𝑎𝑥+ 𝑛𝑏𝑥) (10) Where n is the total number of subjects in the experiment and n` is the adjusted total number of subjects. These five methods summarize some existing statistical methods for calculating interrater reliability for two raters and nominal data. However, these measures are useful for data that is simple and easily categoriz ed, and most of the measures were developed to account for data that was te xt-based in nature. More recently, w ith advances in data collection opportuni ties, technology, and ability to collect and store large quantities of visual and otherwise qualitative data, researchers have needed to begin to apply IRR techniques in modified ways to conduct research across disciplines. As examples of some of this research, Masufumi and Gribble [16] demonstrated a way of estimating intra-rater and inter-rater reliability in image analysis of five image based criteria to investigate foot posture using visually observable criteria. Mishima et al. [17] implemented inter-class correlation methods to determine IRR to investigate images during a posed smile using a video -based motion anal ysis system. These examples show that determining how to calculate IRR in image -based and observational data is an issue that is applicable across disciplines and further motivates our methodological research. Development of Methods to Estimat e IRR for Time -Resolved Writing Data The codebook presented in Table 2 is used to code the screen -recorded data captured from our participants . This codebook is slightly modified from our previous work [5] to reduce complexity . These codes were developed out of cognitive writing theory and have been presented and justified previously [5,18]. For the purposes of calculating IRR, we assigned each code a numerical value so we can manipulate data in MATLAB. The numerical values have no bearing on importance or or der; they are simply for computational bookkeeping. Consistent with the methods we have reported in previous literature [5, 18], raters code on a web -interface that transfers the codebook into an interactive GUI through the free online tool called the Gene ralized Observation and Reflection Platform (GORP), hosted by UC Davis (https://cee.ucdavis.edu/GORP ). While there are limitations to the GORP tool, the advantage of being free, intuitive, and able to be run on a touch screen laptop far outweigh limitations. The data are captured in real time and outputs as a spreadsheet file, which reads the categories as a function of time points. The resulting data file can be manipulated in MATLAB or other programs. Table 2: Codebook and Numerical Values Assigned for Data Processing Level Definition of Level Code Numerical Value Monitoring of process overall Planning (Sense making) 1 Process Level Internal and external processes involved in the process of actually writing text Composing 2 Editing 3 Revision 4 Collaborators & Critics 5 Technology 6 Task Requirements and Materials 7 Resource Level Includes internal memories and general - purpose processes that processes at the other levels can call on Attention (Away from Writing Task) 8 No Apparent Activity 9 Knowledge gathering 10 Reading Text to Date 11 Elimination of Time Overlap and Lag for S tatistical Calculation of IRR The most pressing issue in handling time-dependent observational data, such as observations, screen -capture, and eye -tracking data is that events are continuous —occurring consistently over a period of time, not at a single time point —and often overlap with each other . The overlapping condition is dependent on theoretica l methodological choices: For our time - resolved writing project, cognitive writing theory governs that a writer is often doing multiple things at a time; thus, our data overlaps frequently . A generalized set of time-resolved data here is defined as shown in Table 3. Table 3: General time -dependent observational data table Time Start Time End Code S1k E1k C1k . . . . . . . . . Sik Eik Cik . . . . . . . . . Snk Enk Cnk Here S i and E i are the start time and end time of the event when the rater k coded C i. Here, an assumption of no data missing is taken which can be mathematically express ed as 𝐸𝑖−1≥𝑆𝑖 (11) If 𝐸𝑖−1>𝑆𝑖, there is an overlap between events , meaning that in order to calculate IRR , the overlapping time event will have to be split into distinct non -overlapped events. A four-step process is implemented to manipulate data (Figure 1). Figure 1. (a) Four -step process to clean data to eliminate overlapping data categories (b) Time discretization process To illustrate the process at each step, an example is presented in Figure 2. The first step is to split the time intervals into non -overlapping time intervals for each coder. A generalized process to achieve non -overlapping time intervals is shown in Figure 1(b). The time data ( start time and end time) is stored in a different matr ix and reshaped into a linear matrix. The time data values are sorted in ascending order and the repeating time values are eliminated from the matrix. The sorted matrix is reshaped again with start time and end time. Step 2 assign s codes for the non -overla pping time intervals. A search method is implemented to find codes in a given the time interval. However, the process becomes more complex as it might be possible that the writer is doing multiple things Time discretization for coder ACode assigner for coder A Time discretization for the common matrixAssign codes for common matrixPost processing the matrix and determining the argreement matrix (i) •Reshape the time matrix into linear(ii) •Sort the time data into ascending order(iii) •Delete the repeating time values(iv) •Reshaping the matrix into n*2Time discretization for coder B Code assigner for coder B Step 1 Step 2 Step 4 Step 3 a b at once resulting in multiple code for a single time interval. If categories occur simultaneously, the codes are sorted in ascending order to make the data organized. We implemented an overlapping procedure approach presented in [18] to incorporate multiple codes. We selected a data threshold such that short time intervals (< ~3 s) were ignored. After manipulating the data, an agreement matrix is formed similar to shown in Table 1, on which any statistical IRR calculation can be executed (Cohen’s Kappa, Krippendorff’s Alpha, Sc ott’s Pi, etc.). Figure 2. An example of eliminating the time overlapping issue The limitations of a purely statistical approach to calculating IRR is that it is difficult to implement in practice, with a deep computational and statistical skillset req uired. Because our data can be analyzed visually, representing time -dependent data in images, we present here two novel additional methods for calculating IRR that have not been proposed in literature to our knowledge . Novel Visual IRR Method 1: Image Comparison Approach The Image Comparison Method we propose here leverages the ability to plot observational data in two dimensions simply characterized by codes occurring over a duration of time . These plots can be easily generated for both raters A and B in data processing software and exported as PNG files, as shown in Figure 3. These images are imported to an image processing software ( we employ MATLAB software) where the images showed in Figure 3 are preprocessed for the sake of simplicity. The preproce ssing step involves cleaning the axes and legends and converting the images into grayscale. Then, the images are compared and percent of similarity is found based on the structural similarity index (SSIM) [19]. Both the data visualization and the structural similarity index can be performed using image analysis software that is readily found in analysis software packages such as MATLAB. SSIM is generally used for measuring the image quality by comparing two im ages based on the computation of three parameters namely: Luminance, Contrast and Structure. As we are interested in finding the similarity between structures of the image, the luminance and contrast terms are ignored. Both images are imported in MATLAB as PNG files with a white background. However, the white space of the images will interfere with the results; therefore, we must manipulate the white background before image comparison analysis. This manipulation is achieved using a dynamic control feature o f SSIM where we define the range of Figure 3. Two -dimensional data visualization of observances by Rater A and B colors that are compared. SSIM requires one of the images to be categorized as a reference image from which the similarity between structures is analyzed. The image with maximum number of data points is selected as reference. The modified SSIM result is th en taken to be the p a value that can be supplied to the Brenan -Prediger correction for the G -index calculation (Eq.6) that accounts for the expected (chance) error based solely on number of categories. Novel Visual IRR Method 2: Scattered Points Approac h with Chance Agreement Compensation The scattered points approach leverages the ability of our data to be visualized in a three - dimensional space, with time start point and the code on the x - and y - axes, respectively, and the percentage duration of tota l time on the z-axis, as shown in Figure 4. As such, each occurrence of the observational data can be represented by a point in a three - dimensional space. The scattered point approach is described by the flowchart in Figure 5. First, the coded data from both Rater A and Rater B are converted into three –dimensional data to envision a three - dimensional plot of each Rater’s data. The dataset with maximum number of points (we can call it Data 1) is selected to be the reference data set, to correct for missing data. Systematically, a point is selected from Data 1 (called point t) and the Data 2 is searched for the points falling in the time vicinity of point t selected. The vicinity of the poin t is defined as per time start and percentage total time tolerance value defined by the user. This allows researchers to set thresholds to account for lag time between researchers. If a suitable matching point is found in Data 2, a proxy variable (k) is i ncreased by one, which serves as a counting method to keep track of the points found in common. The process is repeated for each point in Data 1 until all points from Data 1 have been searched in Data 2. The percentage agreement is defined as the ratio of number of points found common (k) and total number of points in Data 1 (n). (a) (b) Figure 4. Three -Dimensional Visualization of Data Figure 5. Algorithm Flow Chart for 3D Scattered Points IRR Approach To account for the expected ( chance ) agreement, Brenan and Prediger ’s method [13] is implemented as it is independent of the data type and depends only the number of categories. IRR then can easily be calculated from the percent agreement calculated by the algorithm using the following equat ion 𝐼𝑅𝑅 =𝑘/𝑛− 1/𝑞 1− 1/𝑞 (11) Where k is the number of codes in agreement within the time threshold defined by the researcher, n is the total number of codes in Data 1 (Rater 1’s 3 -D scatterplot data), and q is the number of categories by which we can account for the expected (chance) agreement. Results and Discussion: Comparison of Statistical and Proposed Visual Methods for IRR We analyzed the five statistical IRR calculations in comparison to the two visual methods proposed in addition to the statistical methods for our data set (Table 4) . The five statistical methods give approximately the same results for the given data, as expected, with m ost variation coming from the methods in which the expected (chance) agreement was corrected . The visual methods for calculating IRR (3D Scatter and Image Comparison) are very close to the statistical methods. The results suggest that the visual IRR method s can be implemented instead of spending significant time cleaning overlapping data to create an appropriate agreement matrix for statistical methods. The scatterplot method has an additional benefit in that the researcher can set appropriate thresholds fo r how much lag time between rater coding can be considered still to be in agreement. Table 4: Comparison of Calculated IRR Based on Statistical and Visual Methods Method Percent agreement (p a) Expected ( chance ) agreement c orrection (pe) IRR Statistical IRR Methods Scott’s Pi 0.7003 0.1446 0.6496 Cohen’s Kappa 0.7003 0.1428 0.6504 Krippendorff alpha 0.7003 0.1446 0.6946 Brenan -Prediger 0.7003 0.0714 0.7190 Gwet AC 0.7003 0.0855 0.6772 Proposed Visual IRR Methods 3D Scatter ed Points Approach 0.7808 0.0714 0.7643 Image Comparison Approach 0.83 0.0714 0.81469 After calculating interrater reliability, researchers must then use judgement to determine if the interrater reliability is high enough to signify that the coding scheme is reliable across rat ers, or if further honing of the coding schema is required. This “goodness” is somewhat flexible, and is context -dependent with regard to how critical perfect agreement is to the quality of research. In general, literature suggests that an interrater rel iability (with respect to Cohens Kappa) of 0.41 -0.6 as moderate, 0.61 -0.80 as substantial, and 0.81 -1 as almost perfect (with 1 being the maximum) [20]. Other scholars [21] instead posit that Kappa values of 0.40 -0.75 as fair to good, and above 0.75 as excellent. Since our data fall within the range of acceptability, we take the IRR calculations for both our proposed methods and our data as a whole to be satisfactory. Conclusion and Future Work In this paper, we presented two new image -based IRR calculation procedures, validating them on five statistical measures of interrater reliability using our data as a context . Our analyses confirm high interrater reliability in our data, and achieved high agreement between our visual methods and existing numerical IRR values. These data are promising, yet preliminary because they are applied to only our data sets. Future work includes applying these visual IRR methods to other time -resolved observational data to validate their transferability, both in the context of writing and in the research group’s other research initiatives. It is easy to envision these methods for visual ly calculating IRR being applied to eye -tracking data, observational data, or other methods that are time -resolved or have overlapping qualitative categories .