January 16, 2011

Assessment_RFI_Response_Symposium01162011_Final 1

Response to the Assessment Technology Standards Request for Information (RFI) 

Posted in the Federal Register / Vol. 75, No. 243 / Monday, December 20, 2010 / 

Sue Bechard, Measured Progress Phoebe Winter, Pacific Metrics 

Edynn Sato, WestEd Renee Cameto, SRI International 

Patricia Almond, University of Oregon Mike Russell, Nimble Innovation Lab

Introduction 

These recommendations are related to accessibility questions under section 3.2.28, and are based 

on discussions that occurred at the Invitational Research Symposium on Technology-Enabled 

and Universally Designed Assessments, held in Arlington, Virginia, on July 23, 2009. The 

symposium was inspired by the interest of researchers at Measured Progress and SRI International, 

the event sponsors, in the emerging and dynamic field of technology-enabled assessments (TEA) and 

the principles of universal design for assessment as they relate to students with disabilities. Two 

publications were produced as a result of the symposium, one regarding cognition and one 

regarding access as they relate to TEA and students with disabilities: Almond, P., Winter, P., 

Cameto, R., Russell, M., Sato, E., Clarke-Midura, J., Torres, C., Haertel, G., Dolan, R., Beddow, 

P., & Lazarus, S. (2010), and Bechard, S., Sheinker, J., Abell, R., Barton, K., Burling, K., 

Camacho, C., Cameto, R., Haertel, G., Hansen, E., Johnstone, C., Kingston, N., Murray, E., 

Parker, C.E., Redfield, D., and Tucker, B. (2010). In addition to insights on the design, 

development, and validation of technology-enabled and universally designed assessment design 

that includes students with disabilities, these articles recommend a thoughtful research agenda to 

investigate critical questions for the next generation of assessment. 

Main points of discussion 

1. Accessibility1 needs to be considered from the beginning of assessment design and 

development, especially as the nation moves to produce “Next Generation Balanced 

Assessments”. Therefore, it is not only appropriate but essential that interoperability 

standards for assessment contain clear standards for accessibility. Accessibility standards 

need to be purposefully incorporated throughout the interoperability standards and 

conformance testing. Almond, Winter, Cameto, Russell, Sato, Clarke-Midura, Torres, 

Haertel, Dolan, Beddow, & Lazarus (2010) present a model for accessibility that incorporates 

three principles into assessment design to guide accessibility standards, i.e., universal design, 

construct-centered design, and scaffolding. 

                                                            

1

   Accessibility is a desired characteristic of testing by which students with physical, cognitive, sensory, or

linguistic barriers are provided the opportunity to demonstrate the knowledge, skills, and abilities intended for

measurement. It is a prerequisite to validity, the degree to which a test score interpretation is justifiable for a

particular purpose and supported by evidence and theory.  
January 16, 2011

Assessment_RFI_Response_Symposium01162011_Final 2

2. Accessibility standards, including how accommodations will be addressed, need to be 

incorporated into the interoperability standards at the outset.2 Dolan, Strain-Seymour, 

Deokar, & Ostler (2010) address accessibility in relation to separating item content and 

format, increasing interactivity, and providing accessible presentation and response options. 

Currently, draft standards, the Accessible Portable Item Profile (APIP) Standard, are under 

review on the APIP public forum on the IMS website3. APIP represents an eight-state effort 

to develop a test item interoperability standard that includes accessibility. These accessibility 

standards should be considered at the outset of development of interoperability standards. 

3. The system should be continuously - improving: this requires building into the system a 

process for monitoring how the interoperability standards are working so that the system can 

evolve. We recommend establishing a review process for considering relevant emerging 

research and vetting changes. The standards need to be designed to be flexible to 

accommodate, for example, new item types that haven’t been envisioned yet, empirical 

support for effective accessibility of content via different presentation, interaction, and/or 

response conditions, and various semiotic representations of targeted constructs–solutions 

need to be able to evolve. 

4. In addition to how the assessment will be developed and presented to provide students with 

access to targeted content, systems must collect data for a number of purposes beyond 

linking student information, performance, reporting, and curriculum. As next generation 

assessments evolve, technical questions regarding the validity of inferences will arise. The 

data system needs to be able to address these additional measurement and research purposes, 

in particular those regarding accessibility features and accommodations. Technology 

standards need to address and allow for the addition of purposes and uses of assessment and 

data management systems beyond current uses and purposes. 

5. We recommend using a modular approach to the development of the interoperability 

standards to maximize the ability to add and/or change portions of the standards. The 

approach needs to begin with basic ideas related to the assessment system itself as the first 

priority. The RFI seems to ask for solutions covering an entire system, from soup to nuts: 

from student data, through assessment items, to reporting. A steady, incremental approach 

may prevent the time-consuming and complex demands for implementing an entire system of 

interoperability standards from threatening the success of the project and overwhelming the 

effort to make needed changes. 

 

                                                            

2

   Both accessibility and the use of accommodations should be considered upfront. However, accessibility

standards should be incorporated from the beginning of the development of the interoperability standards,

whereas considerations regarding accommodations should be considered upfront and revisited toward the end of

the development of the standards vis‐à‐vis the degree to which accessibility has been addressed/incorporated in

the standards. 3

   http://www.imsglobal.org/community/forum/categories.cfm?catid=110  
January 16, 2011

Assessment_RFI_Response_Symposium01162011_Final 3

References 

Almond, P., Winter, P., Cameto, R., Russell, M., Sato, E., Clarke-Midura, J., Torres, C., Haertel, 

G., Dolan, R., Beddow, P., & Lazarus, S. (2010). Technology-Enabled and Universally 

Designed Assessment: Considering Access in Measuring the Achievement of Students with 

Disabilities—A Foundation for Research. Journal of Technology, Learning, and Assessment, 

10(5). Retrieved 12/26/2010 from http://www.jtla.org . 

Bechard, S., Sheinker, J., Abell, R., Barton, K., Burling, K., Camacho, C., Cameto, R., Haertel, 

G., Hansen, E., Johnstone, C., Kingston, N., Murray, E., Parker, C.E., Redfield, D., and 

Tucker, B. (2010). Measuring Cognition of Students with Disabilities Using TechnologyEnabled Assessments: Recommendations for a Research Agenda. Journal of Technology, 

Learning, and Assessment, 10(4). Retrieved 12/26/2010 from http://www.jtla.org. 

Dolan, B., Strain-Seymour, E., Deokar, A., Ostler, W. (November 2010). Next-Generation 

Assessment Interoperability Standards: A white paper from Pearson. Retrieved from 

http://www.pearsonassessments.com/pai/ai/research/researchandresources.htm on December 

30, 2010.