ترجم ما يلى الى اللغة العربية:starting from a tentative theory that might stem from teaching experiences, for example. It is 
recommended to start with some theory, however, even if it is a weak theory that will be 
significantly revised over several iterations (Cobb et al., 2003). However, we believe that the 
whole LP validation cycle actually constitutes a design experiment, because the LP is used 
provisionally as it is being tested. A feature of design experiments is that they may operate from 
weak theory—so we added a note to this effect in the theory development stage of the proposed 
LP validation cycle (see box 1 in Figure 1). 
Another suggestion was that the theories of learning behind the LP (such as those 
discussed in the Compare to Competing Models step) be introduced earlier, also at the Theory 
Development stage. In other words, it was suggested that this first step would be an appropriate 
place to review candidate theories for how students transition from novice understanding to the 
target understanding. This suggestion seems consistent with Embretson’s (1998) cognitive 
design systems approach, in which the first step includes reviewing theories. We agree this is a 
more logical place to compare competing theories, which now is included in box 1 of Figure 1. 
Suggested Revisions: Assessment Development 
It was also proposed that steps that are part of most assessment design frameworks, 
including task design, selection of psychometric models, scoring, and reporting all be added to 
the LP validation cycle, between Theory Development (box 1, Figure 1) and Examine Empirical 
Recovery (box 10, Figure 1). 
Although it was mentioned earlier that empirical recovery might include qualitative 
analysis of interview data intended to provide early information to verify the specification of the 
LP, the term certainly suggests quantitative analysis of data collected on a large scale. Therefore, 
it was suggested that qualitative analysis of cognitive interviews be located somewhere between 
12 
Theory Development and Examine Empirical Recovery. We agree with this suggestion; 
however, we think this step might be generalized to “collect/synthesize evidence of student 
thinking,” because there are several methods for collecting information about student thinking 
(e.g., cognitive interviews, focus groups, and classroom discussions). See box 4 of Figure 1. 
In the revised cycle, we assume that both a pilot test and a field test are conducted (see 
boxes 5 and 7 of Figure 1), at least in the first iteration, and that responses to tasks administered 
during both are scored using levels of the LP. It is assumed here that the primary purpose of the 
pilot is to evaluate how the items are performing, before person classification (however tentative) 
is pursued. The pilot is followed by an item and response analysis (box 6 of Figure 1), which can 
provide valuable information on which to base item revisions. Depending on the purpose and the 
scope of the application however, it may not always be the case that both a pilot and field test are 
conducted. We have also added “Fit and Compare Psychometric Models” (box 9 of Figure 1) to 
the cycle. Note that selection of the psychometric models could come earlier, in the Theory 
Development stage. 
We did not include reporting as an explicit step in the cycle. Nevertheless, person 
classification, which is a precursor to reporting, is now included. The components from the Attali 
and Arieli-Attali (2019) framework have all been added to the cycle. An “Examine 
Item/Response Agreement Among Raters” (box 8 of Figure 1) has been added following the 
“Conduct Field Test & Score Responses” step (box 7 of Figure 1). Note that rater agreement 
might also be examined following the pilot, but we do not assume that pilot data collections 
necessarily include scores from multiple raters. We assume that the other cells in the Attali and 
Arieli-Attali framework (represented in the figure as “Examine Empirical Recovery,” (box 10 of 
Figure 1) “Examine Student Classification Consistency,” (box 11 of Figure 1) and “Compare 
13 
Student Classifications to other Measures” (box 12 of Figure 1) follow from fitting and 
comparing psychometric models. Again, each of these steps might follow from a pilot 
administration, but we do not assume it here, both because pilot administrations have smaller 
sample sizes and because their primary purpose is to test items. 
Suggested Revisions: Teacher Interpretation and Use
A final main area of critique of the first version of the cycle focused on the relative lack 
of attention to teacher interpretation and use of the LP, though work on learning trajectory-based 
instruction has emerged (e.g., Sztajn, Confrey, Wilson, & Edgington, 2012). Although the 
Evaluate for Instructional Efficacy step assumes that teachers are using the LP to guide 
instruction, the cycle does not include anything specific either about teachers’ interpretations of 
LPs or about examinations of whether teachers are using LPs to guide instruction. 
In order for an evaluation of instructional efficacy to provide meaningful information 
about the validity of an LP, we must first examine whether teachers are interpreting an LP as 
intended by its designers, and whether instruction is in accordance with the theory of learning 
that supports transitions to higher understanding. Recent work on the interpretation of an LP for 
linear functions indicated fair to substantial agreement among teachers on assigning LP levels to 
items and substantial to almost perfect agreement between the modal LP levels of teachers and 
that of test developers, even though there was limited time for training and the sample was small 
(van Rijn, Graf, Arieli-Attali, & Song, 2018). 
As another example, Wilson, Mojica, and Confrey (2013) found that as teachers gained 
experience with an LT for equipartitioning, their models of student thinking evolved. More work 
like this is needed on specific LPs as they are developed. 
14 
Proposed LP Validation Cycle: Version 2.0 
So far, we have described the rationale for the first version of the LP validation cycle and 
discussed some areas in which it could benefit from additions or changes, as suggested by 
experts who have used LPs formatively in their work, and as suggested by the Attali and ArieliAttali (2019) framework. Assessment is certainly a large part of the LP validation cycle, since 
empirical recovery of an LP necessarily requires selection of a psychometric model, task 
development, and rubric development. These components were not explicitly addressed by the 
Graf and van Rijn (2016) LP validation cycle, but we decided they are important to include in the 
revised cycle. Validating an LP also extends into teaching, and as was pointed out during the 
meeting, there is nothing in the cycle that explicitly addresses teachers’ interpretations of LPs or 
whether LP-based instruction is implemented as intended, though both are assumed by the 
Evaluate for Instructional Efficacy step. It is important to note that when we say “implemented 
as intended” we do not mean to imply that there is only one correct approach to instruction based 
on an LP. Teaching is a creative enterprise, and it is expected that teachers might find many 
ways to use an LP to guide instruction. LP-based interpretation of student work may initially be 
challenging; however, it may also develop with support (e.g., Wilson et al., 2013). Although LPs 
are domain specific, there are general teaching strategies that are in keeping with the intent of 
LP-based instruction. For example, an approach that recognizes that students might show 
evidence of different levels of thinking depending on the context, and therefore need to be 
assessed in several different contexts, is consistent with the intent of LP-based instruction. 
We conclude this section by presenting version 2.0 of the validation cycle (see Figure 1), 
which includes many of the discussed additions