----

##  Technology-Delivered Assessment: Diamonds or Rocks? ERIC/CASS Digest.        

There can be no doubt that technology has had a major impact on our             daily lives. We read the news online, we shop online, we communicate            with each other online, we are entertained on line, and we learn                online. Education has been a prime beneficiary of technology's                  power. More than 95 percent of all public schools are equipped with             computers, with a majority having Internet access (Market Data                  Retrieval, 1999). Distance learning is becoming more and more                   commonplace, not only in laboratories with computers connected to               the Internet, but also on our desktops at home and work. We can find            information on nearly everything at the stroke of the keyboard and              listen and see events of historical proportion that take place                  thousands of miles from our home. 
Educators have long understood              the potential of technology in supplementing the instructional                  process, including the use of technology in the area of assessment.             Technology-delivered assessment is being implemented in a variety of            settings and grade levels for a variety of uses in education,                   business, government, and the private sector (Sistek-Chandler,                  2000). Under the right conditions and with proper use, a marriage of            technology and assessment can shine like a diamond.  Used without               proper care or forethought, however, it can be a rock. DIAMONDS OF            ASSESSMENT VIA TECHNOLOGYTechnology used with good testing                  practices offers some capabilities that can add value to the                    assessment process.  This includes: 
* accessibility: Individuals              can take various tests by computer or on-line for many purposes                 including college entrance, course placement, certifications and                licensure, career decision-making, academic preparation, military               selection and classification, and personality assessment. As a                  result, individuals can have access to information that they may not            have had before. Bill Gates, in his book The Road Ahead (1995),                 suggested that individuals who are "wired" have access to the same              information, resulting in a virtual equity more achievable than                 real-world equity. 
* immediate feedback: The potential for                    immediate test scoring and feedback is a key advantage of                       technology-delivered assessment which can be a significant motivator            for persons taking assessment instruments. Individuals can learn                their status on assessments quickly and use that information to make            decision or take immediate action. 
* more efficient testing: The              use of computer-adaptive testing, as opposed to                                 computer-administered testing, allows persons to take tests that are            targeted more accurately to their ability levels (Heubert & Hauser,             1999). Use of technology in combination with the increasingly                   popular item response theory can determine an individual's                      performance level using fewer questions. Under certain conditions,              significant time and money can be saved by using computer-adaptive              testing, particularly in large-scale assessment                                 situations. 
*judging portfolio information:  The capability of                placing ones work, experiences and/or educational history on a                  floppy disk or on a website and making that information available               for review and judgment is empowered by technology. Writing samples,            art work, letters of recommendation, journals, test results,                    transcripts, certificates and certifications, descriptions of                   community service, listing of club memberships and activities, and              special project work can be saved electronically, transported                   easily (either physically or electronically), and made readily                  available for evaluation by others. 
* ability to assess higher                order skills: Use of technology permits test developers to use                  techniques and create situations that are difficult or impractical              to construct in regular paper/pencil assessments. Consequently, and             with some creativity through the use multi-media, interactivity,                simulations, etc., assessments can reflect more authentic                       conditions. Crafted carefully, the assessments may provide the                  capability of tapping into higher order cognitive skills than cannot            be easily tested with a standard paper/pencil instrument (Chung &               Baker, 1997). Test developers can construct situations that simulate            the real world. For instance, test items on computer can be designed            to simulate events in biology, economics, architectural design,                 structural engineering, etc., asking students to exhibit skills                 under real or more realistic conditions. Language assessments can               use audio and video to simulate various situations that could be                encountered in a foreign country. 
* helping persons with                      disabilities: Use of assistive technology in assessment can be                  significant in helping persons with disabilities. Text readers can              help persons with visual impairment gain access to testing                      situations. Physically challenged persons can take advantage of                 voice recognition technology to answer test items and even to                   dictate long responses to essay questions. Those with difficulty                controlling fine motor skills can use a touch screen or smart board             to respond to assessment items. Persons who are housebound or not               able to travel to a test site can take a test over the Internet from            their home. Technologies are being created that can respond to                  slight movements of the head or eyes. Not only does technology                  provide testing accommodations, but access to testing opportunities             that were not available before. ROCKS OF ASSESSMENT VIA                       TECHNOLOGYDespite the positive elements described above, there              are potential problems that accompany the use of technology in                  assessment.  These include: 
* lack of accessibility: The                      Department of Commerce has shown in its most recent study that                  access to computers and the Internet is highly dependent on income,             racial and ethnic group, and urbanicity (U.S. Department of                     Commerce, 1999). Other studies indicate that access varies greatly              by income, age, education, and technology optimism. (Forrester                  Research, 2000). In addition to basic computer access, persons may              not be able to afford the cost of the tests themselves. 
* test                security problems: Not unlike paper/pencil instruments, those                   available in electronic formats are vulnerable to compromise.                   Compromised tests, can result in unfair advantage to test takers who            might obtain the questions prior to taking an exam or be able to                find answers to questions on the Internet. Solutions range from                 securing removable hard drives to tracking usage, and blocking                  access to certain Internet addresses. 
* concerns about test taker             identity: Users of test results need to be sure that the person                 taking the assessment has accurately represented his or her                     identity. Special care must be taken to ensure that the person                  completing a licensure test for credentialing, for example, is the              person that is seeking this certification. Solutions can range from             desktop video teleconferencing to fingerprint and retinal                       recognition systems. 
issues of privacy/confidentiality: As with               paper/pencil assessments, information about an individual's                     technology-delivered scores and results should be kept confidential             and be made accessible only to those individuals who have a rightful            need to know. This is particularly critical for sensitive                       assessments that are answered over the Internet 
* lack of                     information on the quality of the instrument: It can be mistakenly              assumed that any test made available on computer or via the Internet            meets professional testing standards. This can be a risky                       assumption. It is quite possible for the instrument to be deficient             in the requisite technical information necessary to show the degree             of quality and suitability of the test. In fact, many assessments               have been posted for easy access via the Internet where little or no            technical information has been made available to the consumer. 
*              problems with test comparability: If an instrument is available in              both paper/pencil and computer-delivered format, it can be                      mistakenly assumed that test developers have produced parallel forms            that provide the same scores regardless of administration format. It            is not unusual for some high-quality tests, offered in the                      paper/pencil format, to result in different scores when those same              items are administered via computer or Internet. Without assurance              of comparability, a test taker may have an advantage by using one               format rather than the other. Disparate results from noncomparable              assessments can transpire for any number of reasons including                   speededness, point size of the words, monitor resolution, use of                color, physical arrangement of the items, inability to change                   completed items, comfort with the equipment, and response mode. 
*             gender, racial, and ethnic disparities: That females, persons of                color, or individuals of different ethnic backgrounds may be                    disadvantaged in certain testing situations has been a long-standing            concern in paper/pencil testing. This can be exacerbated with tests             delivered via computer or the Internet. If a particular group has               disproportionate access to computers and technology, disparity could            be created by the medium alone. 
* reporting and interpretation:               Immediate feedback on tests is clearly desirable. Without                       appropriate interpretation, however, there is danger that the test              taker will take actions that are not warranted by the test results.             The potential exists for interpretations to appear to be so                     definitive that test takers fail to understand the need for caution             due to the lack of accuracy of the scores. Also, feedback can be so             extensive that the test results can paralyze a person's actions. 
*            lack of human contact: With technology-delivered assessments,                   meaningful human contact and intervention to assist with test score             interpretation may be lacking or unavailable. A skilled counselor               can help a test taker to sort out his or her results and use them in            the context of other knowledge and experiences. 
*issues of                    familiarity with technology: There are some studies that show that              students perform better on tests that use the same medium in which              they were taught. For example, if students use the computer to write            essays and papers, they tend to score higher on writing assessments             that allow them to use that technology to take writing assessments              and more poorly if that technology is not available for testing                 purposes. Therefore, students who learn on computers may have an                advantage on computerized assessments. FOLLOWING ASSESSMENT                   STANDARDS AND PRACTICESHow do counselors and educators sort the             diamonds from the rocks? They need to be aware of the various issues            that relate to the construction, production, administration, and                interpretation of tests delivered via the computer or Internet. No              compromise should be made on the quality  of a test administered to             a client or student with either traditional or technology-delivered             assessments. Various agencies and organizations have produced policy            statements and standards for testing that are applicable to both                paper/pencil and technology-delivered assessment including the                  following: 
* American Educational Research Association, American              Psychological Association, and National Council on Measurements in              Education. (1999). Standards for Educational and Psychological                  Testing. Washington, DC: American Educational Research                          Association. 
* American Association for Counseling and                        Development. (1989). Responsibilities of Users of Standardized                  Tests. Alexandria, VA: Author. 
* American School Counselors                   Association and Association for Assessment in Counseling. (1998).               Competencies in Assessment and Evaluation for School Counselors.                Alexandria, VA: Author. 
* Committee on Professional Standards and             Committee on Psychological Tests and Assessment. (1985). Guidelines             for Computer-Based Tests and Interpretations. Washington, DC:                   American Psychological Association. 
* Dahir, C. A., Shelton, C.               B., and Valiga, M. J. (1998). Vision Into Action: Implementing the              National Standards for School Counseling Programs. Alexandria, VA:              American School Counselor Association. 
* Joint Committee on                   Testing Practices. (1988). Code of Fair Testing Practices in                    Education. Washington DC: National Council on Measurement in                    Education. This document is being reviewed and should be revised for            publication in 2001. 
* Joint Committee on Testing                             Practices. (2000). Rights and Responsibilities of Test Takers:                  Guidelines and Expectations. 
* National Board for Certified                   Counselors. (1998). Standards for the Ethical Practice of Web                   Counseling.  Greensboro, NC: Author. 
* National Career Development            Association. (1997). NCDA Guidelines for the Use of the Internet for            the Provision of Career Information and Planning Services.                      Alexandria, VA: Author. 
* U.S. Department of Labor Employment and             Training Administration. (1999). Testing and Assessment: An                     Employer's Guide to Good Practices. Washington, DC:                             Author. REFERENCESChung, G.K.W.K & Baker, E.L. (1997). Year 1             Technology Studies: Implications for Technology in Assessment, CSE              Technical Report 459, National Center for Research on Evaluation,               Standards, and Student Testing (CRESST). University of California,              Los Angeles. 
Forrester Research. (2000) The Truth About the                   Digital Divide. Forrester.com. 
Gates, W. (1995). The Road Ahead.              New York: Penguin. 
Heubert, J. P., & Hauser, R. M. (Eds.). (1999).            High Stakes: Testing for Tracking, Promotion, and Graduation.                   Washington, DC: National Academy Press. 
Market Data                           Retrieval. (1999). Technology in Education, 1999. Shelton, CT:                  Author. 
Sistek-Chandler, C., Online Assessment: Changing the Way              You Test, Converge, November, 2000. 
U.S. Department of                        Commerce. (1999). Falling Through the Net: Defining the Digital                 Divide. Washington, ED: Author. 
----- 
Janet E. Wall is Assistant            Vice President of ACT-Educational Technology Center in Hunt Valley,             Maryland. 
ERIC Digests are in the public domain and may be freely             reproduced and disseminated.  This publication was funded by the                U.S. Department of Education, Office of Educational Research and                Improvement, Contract No. ED-99-CO-0014. Opinions expressed in this             report do not necessarily reflect the positions of the U.S.                     Department of Education, OERI, or ERIC/CASS.                                 **Title:** Technology-Delivered Assessment: Diamonds or Rocks? ERIC/CASS Digest.        **Document Type:** Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073); **Available From:** ERIC Counseling and Student Services Clearinghouse, University of               North Carolina at Greensboro, 201 Ferguson Building, P.O. Box 26171,            Greensboro, NC 27402-6171. Tel: 336-334-4114; Tel:                              800-414-9769 (Toll Free); Fax: 336-334-4116; E-mail:                            [email protected]; Web site: http://ericcass.uncg.edu.                       **Descriptors:** Access to Information, Bias, Confidentiality, Disabilities,                      Evaluation,  Internet, Measures (Individuals), Portfolio                       Assessment, Standards,  Technology, Theories, Thinking Skills                **Identifiers:**
###
[Return to ERIC Digest Search Page]