# Appendix E

Technology and Engineering Literacy Reports

1. Student report

2. School Report

3. Final Adjudication Decisions

# Technology and Engineering Literacy Reports

1. Student report

## Listening

. Learning

. Leading

. 

# NAEP Item Development (ID)
Technology and Engineering Literacy ( TEL )
2014 Probe Survey Questionnaire

Recommendations to NCES

Grade 8 Student Questionnaire

Deliverable in response to ID Task 3.1.2

Submitted: March 12, 20131 

## Table of Contents

2014 Technology and Engineering Literacy (TEL) Grade 8 Student Questionnaire:

Post

- Pilot Analysis and Recommendations

Background

................................ ................................ ................................ ................................ ............. 2

Criteria for item review

................................ ................................ ................................ ........................... 3

Recommendations for the 2014 TEL Probe assessment

– Overview

................................ ..................... 7

Item Review and Recommendations for the 2014 TEL Student Grade 8 Probe assessment

............... 10

References

................................ ................................ ................................ ................................ ............ 432 

### 2014 Technology and Engineering Literacy (TEL ) Grade 8 Student Questionnaires :
Post

- Pilot Analysis and Recommendations

This document provides a post

- pilot review of the grade 8 student and school Technology and Engineering Literacy (TEL) questionnaires using data collected in the 2013 pilot administration. The goal of this report is to evaluate the performance of the items and propose a set of questions that will be included in the 2014 TEL probe assessment. As such, this review serves the research objective to develop items that provide reportable survey results based on measures of contextual factors that might explain di fferences in student performance (e.g., more proficient students have access to more instructional or extracurricular content related to engineering design).

Unlike previous post

- pilot recommendations, a much stricter item evaluation and selection

was req uired for this recommendation given the spiraled design in the 2013 p ilot administration (see more details below). The questionnaire material had to be reduced from approximately 24 minutes of assessment time in the pilot (based on actual pilot timing data) to 10 minutes of assessment time in the probe assessment. Recommendations are based on a combination of criteria including an analysis of frequency data and theoretical considerations.



### Background

The NAEP TEL a ssessment measures three core areas of interest : Technology and Society (T&S), Design and Systems (D&S), and Information and Communication Technologies (ICT). Technology and Society addresses the effects that technology has on society and on the natural wo rld. Design and Systems covers the nature of technology, the engineering design process, and basic principles of dealing with everyday technologies. Information and Communication Technology includes computers and software learning tools, networking systems and protocols, handheld digital devices, and other technologies for accessing, creating, and communicating information and for facilitating creative expression (WestEd, 2010). The TEL issues paper identified four broad issues that inform ed and guided the survey questionnaire development administered in the pilot assessment . These are: a vailability of school resources ; o rganization of technology and engineering instruction ; t eacher

p reparation ; and s tudent e ngagement . E ach of these issues compris es several sub

- issues

. Two of these four issues were covered in both the student and school questionnaire s . Teacher preparation was covered in the school questionnaire only and student engagement was covered in the student questionnaire only

.

The 2013 TEL p ilot used a spiraled design where not every student receive d all questionnaire
items

. In order to maximize the number of questions included certain items we re spiraled across different, partially overlapping, questionnaire forms. This design was chosen to make3



### better use of the limited questionnaire time

, while covering a maximum number of topics and constructs, particularly in the pilot assessment. This ensured that a sufficient number of items pertaining to each topic and each i ssue could be piloted. The available questionnaire response
time was 15 minutes per student, of which approximately 4

– 5 minutes were devoted to a student core section (e.g., demographic items) which is required to be administered as part of every NAEP surv ey questionnaire. Questions in the remaining 10

– 11 minutes were TEL specific questions. At the end of the questionnaire, a few debrief ing items were administered, as was the case in other NAEP questionnaires.

In the spiraling design

, questions were divi d ed into separate block s ( or sections ) that were configured into 10 different booklets (see details below) that were administered to students

. Spiraling provided randomly equivalent samples of students receiving each of the blocks. Using timing data collected from the tryout, an appropriate division of items and a spiraling approach was determined that would adequately pilot all of the items without reducing data quality or increasing respondent burden. Students were allotted four minutes to complete each of five different block s

. There were ten different combinations of the five bloc k s resulting in 10 different booklets. This procedure ensured that 1) every BQ spi raled item was paired with other spiraled BQ items; 2) the position effect of the blocklets in the combination were removed (i.e., each of the five blocklets appeared an equal number of times in the each of two positions), and 3) each of the BQ combination s were paired with each of the cognitive blocks.

Table 1 (on page 6) shows how the items were distributed across the ten booklets with each category represented by a different color.



### Criteria for I tem R eview

Analysis of F requency D istributions 

### Following the same procedure as for other background questionnaire item reviews , a set of frequency

- based flagging criteria were c onsidered in evaluating whether items were applicable to the targeted population. It is important to keep in mind that flags a re indications that a particular item should be thoroughly evaluated. Flags are not absolute criteria for making decisions regarding the use or quality of items. Instead, the flagging criteria should be viewed collectively, along with other criteria and pr ofessional judgment, in recommending keeping, revising, or dropping items from the 2014 TEL Probe survey questionnaires.

For this analysis, we examined item response frequencies for response options. We also assessed item non

- response patterns to determine whether problematic items or response options warrant revising items, expanding or collapsing response categories, or dropping an item. The flagging criteria on response patterns and item non

- response (i.e., missing response) rates for reviewing the data are:4



###  A high percentage of item non

- response (relative to other adjacent items) may indicate that the item content might have been problematic (e.g., ambiguous, burdensome, overly complex, offensive) or that the format might have caused respondents to

over look the item. Note that, this criterion does not apply to multiple selection multiple

- choice items, because the missing rate for “select all that apply” items contains both missing responses and “not apply” responses.

 Low single

- category response rates ( e.g., <10%) may indicate that a category does not apply to this population and possibly that different categories may be more informative.

 High single

- category response rates (e.g., >80%) may indicate that almost all respondents in the population fall int o one category and that a limited range of demographics or behavior indicators are collected.

 In addition to these flagging criteria, we investigated differences in relative frequencies for response categories across the different booklets in which an item had been administered. High variation in relative frequencies across booklets might be an indicator for instability of item performance.

Implications of flags on any of these criteria for a given item will be discussed in more detail.
Note again that

, flagging an item on one of these criteria does not, by itself, warrant the necessity for revising or dropping the item. Whether an item needs to be modified or revised also depends on whether response categories are unique to an item versus representing o ne of the standard formats used across many questionnaire items, and whether certain response categories are needed from a theoretical perspective. Maintaining a high level of consistency across items is one important consideration for the validity of the questionnaires as well as the trend information being collected.

Response Time /Burden 

### Item selection was guided by available response time data from the pilot assessment. All timing estimates were based on the 90 th percentile of the response time based on available pilot data. The rationale for ch o osing this value was that a maximum not

- complete

- rate of 10% is expected when basing timing decisions on the 90 th percentile . Results from previous analyses with NAEP questionnaires showed that a 10% missing rate seemed to be acceptable from a sampling and analysis perspective with unbiased estimates for each subgroup investigated. Further, 10% missing rates for BQ items have been considered acceptable in previous operational NAEP assessments. For example

, in 2011 Grade 8 science some science specific BQ items had 11% (2009: 8 %) or 12% (2009: 13%) missing rate in the teacher’s BQ questionnaire that was used in the NAEP analysis. Our previous operational experience shows that about 90% of the response rate still ensured us a valid sample to represent the whole population. Therefore, using the 90 th percentile of the complete time as the cutoff is effe ctive f rom an analysis perspective. Note that the same rule for timing estimates was used when the TEL pilot questionnaires were assembled. The data used for the item review summarized in5



### this document shows that using the 90 th percentile actually resulted in rather conservative time estimates with missing values rates clearly less than 10%. Even for the items at the very end of the questionnaire, missing value rates only reach ed values of around 5 % for the TEL

p ilot assessment. Overall, missing value rate s were close to zero for most items.

T he number of TEL

- specific items that can be selected for the probe assessment was limited to what will fit into 600 seconds questionnaire time. The recommendations summarized in the following are in line with the criterion of including not more than 600 seconds of material. Wherever feasible

, reducing the number of sub

- items in matrix items was considered as one mean to reduce burden while maintai ning the breadth of the questionnaire.

Because of the strict timing constraints , many of the items that are recommended to be dropped showed satisfactory performance and might serve as an item pool for future

assessments

.

Theoretical C oherence
, R elevance, and C ontent C overage 

### In addition to purely data

- driven evaluation

, a number of theory

- based criteria were applied when making recommendations. First, even a reduced questionnaire needs to include questio ns about each of the three main topic areas as well as the three issues covered in the student questionnaire. Items were selected to create a balance of the number of items or sub

items across the three areas and to cover the most important themes and sub

- themes as defined in the TEL issues paper.

Another important goal was to include items addressing both at

- school and out

- of

- school learning and activities given the importance of out

- of

- school learning experiences for TEL. F ew studies have systematically investigated effects of out

- of

- school learning experiences. The available research suggests that under certain circumstances, technology and engineering education can boost learning and achievement in science and mathematics.

Further, the coverage of all three topic areas for student engagement questions was an important criterion, i.e., the inclusion of questions related to student interest in each of the domains given the importance of interests in particular content areas as predictors of subsequent career choices (e.g., Holland 1997; Lubinski 2000), and a selection of self

- efficacy or self

- concept items for the three areas. Meta

- analyses have shown that self

- efficacy is one of the strongest predictors of student achieveme nt (e.g., Richardson, Abraham, & Bond, 2012). Students’ self

- efficacy perceptions can be considered an important achievement predictor as well as an informative outcome with policy relevance by themselves.

When possible items with clearly quantifiable, behavior

- related response categories were preferred over items with more vague response categories.6

Table 1

- Spiraling Design in the TEL Pilot

J2TXBQ01 J2TXBQ02 J2TXBQ03 J2TXBQ04 J2TXBQ05 J2TXBQ06 J2TXBQ07 J2TXBQ08 J2TXBQ09 J2TX BQ10

VE631435 VE631435 VE631435 VE631435 VE631435 VE631435 VE631435 VE631435 VE631435 VE631435

VE631437 VE631437 VE631437 VE631437 VE631437 VE631437 VE631437 VE631437 VE631437 VE631437

VE011083 VE011083 VE011083 VE011083 VE011083 VE011083 VE011083 VE011083 VE011083 VE011083

VE011103 VE011103 VE011103 VE011103 VE011103 VE011103 VE011103 VE011103 VE011103 VE011103

VF541314 VF541314 VF541314 VF541314 VF541314 VF541314 VF541314 VF541314 VF541314 VF541314

VE011108 VE011108 VE011108 VE011108 VE011108 VE011108 VE011108 VE011108 VE011108 VE011108

VE011109 VE011109 VE011109 VE011109 VE011109 VE011109 VE011109 VE011109 VE011109 VE011109

VE011111 VE011111 VE011111 VE011111 VE011111 VE011111 VE011111 VE011111 VE011111 VE011111

VE011063 VE011063 VE011063 VE011063 VE011063 VE011063 VE011063 VE011063 VE011063 VE011063

VE011064 VE011064 VE011064 VE011064 VE011064 VE011064 VE011064 VE011064 VE011064 VE011064

VE011121 VE011121 VE011121 VE011121 VE011121 VE011121 VE011121 VE011121 VE011121 VE011121

VF541324 VF541324 VF541324 VF541324 VF541324 VF541324 VF541324 VF541324 VF541324 VF541324

VE117468 VE117468 VE117468 VE117468 VE117468 VE117468 VE117468 VE117468 VE117468 VE117468

VE682225 VE639123 VE639025 VE639847 VE639842 VE682225 VE639123 VE639847 VE639025 VE639842

VE682232 VF025108 VF009050 VE638956 VE639172 VE682232 VF025108 VE638956 VF009050 VE639172

VE639166 VE682276 VE682274 VE638983 VE681624 VE639166 VE682276 VE638983 VE682274 VE681624

VE639123 VE639025 VE682217 VF238958 VE682225 VE639025 VE639847 VF238958 VE682217 VE639123

VF025108 VF009050 VE682215 VE639842 VE682232 VF009050 VE638956 VE682225 VE682215 VF025108

VE682276 VE682274 VE639847 VE639172 VE639166 VE682274 VE638983 VE682232 VE639842 VE682276

VE638999 VE682217 VE638956 VE681624 VE638999 VE682217 VF238958 VE639166 VE639172 VE638999

VE639008 VE682215 VE638983 VE638999 VE639008 VE682215 VE638999 VE638999 VE681624 VE639008

VE682315 VE638999 VF238958 VE639008 VE682315 VE638999 VE639008 VE639008 VE638999 VE682315

VE682317 VE639008 VE638999 VE682315 VE682317 VE639008 VE682315 VE682315 VE639008 VE682317

VF009358 VE682315 VE639008 VE682317 VF009358 VE682315 VE682317 VE682317 VE682315 VF009358

VE401773 VE682317 VE682315 VF009358 VE401773 VE682317 VF009358 VF009358 VE682317 VE401773

VE401776 VF009358 VE682317 VE401773 VE401776 VF009358 VE401773 VE401773 VF009358 VE401776

VE401779 VE401773 VF009358 VE401776 VE401779 VE401773 VE401776 VE401776 VE401773 VE401779

VE401776 VE401773 VE401779 VE401776 VE401779 VE401779 VE401776

VE401779 VE401776 VE401779 VE401779

VE401779

Note. Within each booklet, the item sequence follows the order from top to the bottom.

Legend Common (items apply to all the three assessment areas, that is D&S, ICT, and T&S)

Design and Systems Information and Communication Technology Technology and Society

Common Debrief7



### Recommendations for the 2014 TEL Probe A ssessment

– Overview In the following we will summarize our item review in detail for each item, along with the recommendation for the 2014 TEL Probe assessment. Three different cases are distinguished:

1.) An item is recommended to be kept in the questionnaire ; 2.) An item is recommended to be dropped from the questionnaire based on timing constraints
and student burde n; 3.) An item is recommended to be dropped in the questionnaire based on poor item performance

.
Note that, i n case s 1 and 2, an item shows satisfactory performance to be included in the 2014 TEL probe assessment but is recommended to be drop ped only based on timing constraints and student
burden.

For some matrix items we may recommend to keep the item stem and certain sub

- items but to drop specific sub

- items, either based on item performance or for burden reasons.

Table 2 presents an overview of all recommendations. For each item, the number in the booklet, the Accession number, and the area and issue that are addressed are given in the t able. In addition, the two rightmost columns summarize our recommendation and the rationale for recommending dropping complete items or sub

- items.8

Table 2

- Recommendations for 2014 TEL Probe

- Overview

Item # AccNum Area Issue Recommendation Rationale for Recommendation
14 VE682225 Design and

Systems Org. of Instruction Drop Time restrictions/student
burden

15 VE682232 Design and

Systems Student Engagement Drop Time restrictions/student
burden

16 VE639166 Common Student Engagement Keep NA

17 VE639123 ICT Org. of Instruction Keep this item and sub

- items a, c, d, e, and f; Drop sub

- item b (VE639127). Time restrictions/student
burden

18 VF025108 ICT Student Engagement Keep this item and sub

- items a, c, d, e, and f; Drop sub

- item b (VF025110) Time restrictions/student
burden

19 VE682276 Design and

Systems Student Engagement Keep this item and sub

- items b, c, and d; Drop sub

- items a (VE682278) and e (VE682286). Time restrictions/student
burden

20 VE639025 Common Student Engagement Drop Time restrictions/student burden and item performance

21 VF009050 Common Student Engagement Drop Time restrictions/student burden and item performance

22 VE682274 Design and

Systems Availability and Use

of Instructional
Resources Drop Item performance

23 VE682217 ICT Student Engagement Keep this item and sub

- items a, b, and c; Drop sub

- item d (VE682222). Time restrictions/student
burden

24 VE682215 ICT Availability and Use of Instructional
Resources Drop Item performance

25 VE639847 Common Org. of Instruction Drop Time restrictions/student
burden

26 VE638956 Design and

Systems Org. of Instruction Keep this item and sub

- items b, c, d, and e; Drop sub

- items a (VE638957) and f (VE682248). Time restrictions/student
burden and item performance

27 VE638983 Design and

Systems Student Engagement Keep this item and sub

- items c, d, e, and f; Drop sub

- items a (VE638986), b (VE009777), and g (VE682268). Time restrictions/student burden and item performance
for sub

- item g.9

Item # AccNum Area Issue Recommendation Rationale for Recommendation
28 VF238958 ICT Org. of Instruction Drop Time restrictions/student
burden

29 VE639842 Common Org. of Instruction Keep NA

30 VE639172 Common Student Engagement Drop Item performance

31 VE681624 ICT Org. of Instruction Keep NA

32 VE638999 Technology and

Society Org. of Instruction Keep NA

33 VE639008 Technology and

Society Student Engagement Keep NA

34 VE682315 Technology and

Society Availability and Use

of Instructional
Resources Drop Item performance

35 VE682317 Technology and

Society Student Engagement Keep NA

36 VF009358 Common

-- Debrief Student Engagement Keep NA

37 VE401773 Common

-- Debrief Student Engagement Keep NA

38 VE401776 Common

-- Debrief Student Engagement Keep NA

39 VE401779 Common

-- Debrief Student Engagement Keep NA

Note . NA = not applicable10



### Item Review and Recommendations for the 2014 TEL Student Grade 8 Probe

A ssessment

Note: Core NAEP background items are not included in this review

VE682225 14. In school , how often have you learned about or discussed the following? Select one circle in each

ro w.

Never Rarely Sometimes Often

a. The use and purpose of tools, machines, or devices (A) (B) (C) (D) VE682226

b. The care or maintenance of tools, machines, or devices (A) (B) (C) (D) VE682227

c. Designing or creating something to solve a problem (A) (B) (C) (D) VE682228

d. Designing something when there is limited time, money, or materials (A) (B) (C) (D) VE682229

e. Figuring out how to fix something (A) (B) (C) (D) VE682230 f. Finding the right people to work with or get help from to fix something (A) (B) (C) (D) VE682231

Area: Design and Systems

Issue: Organization of Instruction
Booklets: 1, 5, 6, 8

Item Review: This item provides contextual information for the cognitive assessment pertaining to Design and Systems literacy

. It focuses on TEL

- related learning opportunities and experiences at school

. Exactly the same sub

- items as in its outside

- of

- school

- related counterpart ( VE682232 ) are used

. The frequency distribution across response options is satisfactory for all sub

- items with reasonable balance of responses across all response categories

. The missing rate is close to zero (0.23

– 0.29%) across all questionnaire booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 1 3 %

- point s , which indicates that t here is some variation in item performance across booklets that should be investigated more closely

.

Overall, this item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire

. There is no evidence that any response categories need to be collapsed, expanded, or deleted. Item VE638956 , however, covers similar content and should be given hi gher priority as the response options represent more quantifiable, behavior

- related categories than those in item
VE682225

.

Recommendation: Drop this item and all sub

- items from the 2014 TEL Probe Administration based on time restrictions/consideration of student burden.11 VE682232

15. Outside of school , how often have you learned about or discussed the following? Select one circle in each row.

Neve r Rarely Sometimes Often

a. The use and purpose of tools, machines, or devices (A) (B) (C) (D) VE 6 82233

b. The care or m aintenance of tools, machines, or devices (A) (B) (C) (D) VE682234

c. Designing or creating something to solve a problem (A) (B) (C) (D) VE682238

d. Designing something when there is limited time, money, or materials (A) (B) (C) (D) VE682236

e. Figuring out how to fix something (A) (B) (C) (D) VE682237

f. Finding the right people to work with or get help from to fix something (A) (B) (C) (D) VE682235

Area: Design and Systems

Issue: Student Engagement
Booklets : 1, 5, 6, 8

Item Review: This item provides contextual information for the cognitive assessment pertaining to Design and Systems literacy. It focuses on TEL

- related learning opportunities and experiences outside of school. Exactly the same sub

- items as in the previous at

- school item are used. The frequency distribution across response options is satisfactory for five of the six sub

- items with reasonable balance of responses across all response categories

. Sub

- item e has a low response rate ( less than 10% ) for one category

. The missing rate
is close to zero ( 0.31

– 0.39% ) across all questionnaire booklets. The maximum difference in the res ponse proportion across the four booklets in which the item has been administrated is 1 3 %

- points , which might indicate that there is some variation in item performance across booklets

.

Overall, this item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire. There is no evidence that any response categories need to be collapsed, expanded, or deleted. Item VE638983 , however, covers similar con tent and should be given higher priority as the response options represent more quantifiable, behavior

- related anchors than those in item VE682232.

Recommendation: Drop this item and all sub

- items from the 2014 TEL Probe Administration based on time

restr ictions/consideration of student burden.12 VE639166

16. How interested are you in learning about the following areas of technologies? Select one circle in each row.

Not at all interested Not too

interested Somewhat

interested Very

interested

a. Information and communication (for example, computers, Internet, social networking sites) (A) (B) (C) (D) VE639168

b. Transportation (for example, cars, planes, trains, traffic analysis) (A) (B) (C) (D) VE639169

c. Construction (for example, architecture, building a bridge) (A) (B) (C) (D) VE639171

d. Power and energy (for example, dams, power plants, batteries) (A) (B) (C) (D) VE639173

e. Environmental and green technologies (for example, recycling, renewable energy sources such as sunlight and wind) (A) (B) (C) (D) VE639174

f. Agriculture (for example, farming, food chemistry) (A) (B) (C) (D) VE639175

g. Medical technologies (for example, vaccines, drugs, surgical tools, heart monitors, x

- ray machines) (A) (B) (C) (D) VE639176

h. Home and domestic (for example, air conditioning, cleaning, cooking, heating, plumbing, sewing) (A) (B) (C) (D) VF009755

i. Manufacturing (for example, what goes on in factories, developing or improving products) (A) (B) (C) (D) VE639170

Area: Common

Issue: Student Engagement
Booklets: 1, 5, 6, 8

Item Review: This is common item across the three TEL areas and provides information about students’ interest in nine areas of technologies

. The item collects important information regarding whether students would like to learn more in each of the areas of technologies in the future and as such result may inform TEL curriculum design and have important policy implication. The frequency distribution across response options is satisfactory for all sub

- items with reasonable balance of responses across all response categories. The missing rate is less than 1% (0.45

– 0.56%) in the a ggregated response frequency . There is no evidence that any response options need to be collapsed, expanded, or deleted. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 6 %

- point indicat ing that performance is very stable across different booklets for this item.13

This item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire. In direct comparison with the two other matrix items focusing on student engagement ( VE639025, VF009050 ), higher priority should be given to this item because it provides more specific information about multiple areas of technologies, and not only on technology and engineering in general. Moreover, frequency distributio ns for this item are more balanced than those for items VE639025 and VF009050 allowing better differentiation across students.

Recommendation: Keep th is item and all sub

- items in the 2014 TEL Probe Administration

.14 VE639123

17. For school work , how often do you use a computer or other digital technology for the following activities? Select one circle in each row.

Never or almost

never A few times a year Once or twice a month Once or

twice a week Every day

or almost every day

a. Send or receive messages (for example, chat, e

- mail, instant messages, text messages) (A) (B) (C) (D) (E) VE639125

b. View or download digital media (for example, art, books, games, mobile apps, music, pictures, software, videos) (A) (B) (C) (D) (E) VE639127

c. Create, edit, or organize digital media (A) (B) (C) (D) (E) VE639130

d. Send, share, present, or upload digital media (A) (B) (C) (D) (E) VE639131

e. Create a presentation (A) (B) (C) (D) (E) VE639137

f. Create a spreadsheet (a table or grid that displays data into columns and rows and may be used to create charts and graphs) (A) (B) (C) (D) (E) VE639136

Area: Information and Communication Technology

Issue: Organization of Instruction
Booklets : 1, 2, 7, 10

Item Review: This item provides contextual information for the cognitive assessment pertaining to Information and Communication Technology literacy. It focuses on TEL

- related learning opportunities and experiences at school. Exactly the same sub

- items as in its outside

- of

- school

- related counterpart (VF025108) are used. The frequency distribution across response options is satisfactory for four of the s ix sub

- items with reasonable balance of responses across all response categories

. Some response categories for s ub

- items e and f have low response rate s less than 10%. The missing rate is close to zero ( 0.43

– 0.62 %) across all questionnaire booklets. There is no evidence that any response options need to be collapsed, expanded, or deleted. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 9%

- point indicating relatively stable performa nce across different booklets for this item.

T his item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire. The item collects important information regarding frequencies at which students use computer or oth er digital technology while doing ICT related activities

.

A further content review of the TEL cognitive assessment items indicated that sub

- item b of this matrix
item is less strongly linked to the content of any cognitive item compared to the other sub

- items

.15

Despite its satisfactory performance, dropping sub

- item b in order to reduce student burden would be reasonable.

Recommendation: Keep this matrix item and sub

- items a, c, d, e, and f i n the 2014 TEL Probe Administration

. Drop sub

- item b from the 2014 TEL Probe administration based on time restrictions/consideration of student burden.16 VF025108

18. In this question, please think about activities you do that are not related to your school work. How often do you use a computer or other digital technology for the following activities not for school work ? Select one circle in each row.

Never or almost

never A few

times a
year Once or twice a month Once or

twice a week Every day

or almost every day

a. Send or receive messages (for example, chat, e

- mail, instant messages, text messages) (A) (B) (C) (D) (E) VF025109

b. View or download digital media (for example, art, books, games, mobile apps, music, pictures, software, videos) (A) (B) (C) (D) (E) VF025110

c. Create, edit, or organize digital media (A) (B) (C) (D) (E) VF025112

d. Send, share, present, or upload digital media (A) (B) (C) (D) (E) VF025113

e. Create a presentation (A) (B) (C) (D) (E) VF025117 f. Create a spreadsheet (a table or grid that displays data into columns and rows and may be used to create charts and graphs) (A) (B) (C) (D) (E) VF025116

Area: Information and Communication Technology

Issue: Student Engagement
Booklets : 1, 2, 7, 10

Item Review: This item provides contextual information for the cognitive assessment pertaining to Information and Communication Technology (ICT) literacy. It focuses on TEL

- related learning opportunities and experiences outside of school. Exactly the same sub

- items as in its at

- school

- related counterpart ( VE639123 ) are used. Although relative frequencies for response options a or f are less than 10% for sub

items a, b, e, and f , there is sufficient variation across response options to distinguish different levels of student engagement. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for the at

- school related version of this matrix item is important for comparisons of and reporting on the results fo r these items. The missing rate is close to zero (0.43

– 0.51 %) across all questionnaire booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 8 %

- point indicating relatively stable performance across different booklets for this item.

This item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire. The item collects important information regarding the freque ncies at which students use computer or other digital technology engaging in ICT related activities that is not for school work

.17

A further content review of the TEL cognitive assessment items indicated that sub

- item b of this matrix item is less strongly linked to the content of any cognitive item compared to the other sub

- items. Despite its satisfactory performance, dropping sub

- item b in order to reduce student burden

— as for
item VF025110

— would be reasonable.

Recommendation: Keep this matrix item and sub

- items a, c, d, e, and f in the 2014 TEL Probe Administration. Drop sub

- item b from the 2014 TEL Probe administration based on time restrictions/consideration of student burden.18 V E682276

19. Do you think that you would be able to do each of the following? Select one circle in each row.

I definitely can’t I probably

can’t Maybe I probably

can I definitely

can

a. Build a model using a kit (A) (B) (C) (D) (E) VE682278

b. Build a model without using a kit (A) (B) (C) (D) (E) VE682280

c. Use tools or materials to fix something (A) (B) (C) (D) (E) VE682281

d. Take something apart in order to fix it or see how it works (A) (B) (C) (D) (E) VE682284

e. Design a computer
program (A) (B) (C) (D) (E) VE682286

Area: Design and Systems

Issue: Student Engagement
Booklets : 1, 2, 7, 10

Item Review: This item provides contextual information for the cognitive assessment pertaining to Design and Systems (D&S)

. The item collects important information regarding student self

- efficacy in conducting activities that are related to Design and Systems

. Meta

- analyses have shown that s elf

- efficacy is one of the strongest predictors of student achievement (e.g., Richardson et al, 2012) . This item can be considered an important achievement predictor as well as an informative outcome with policy relevance by itself.

Even though relative frequencies for response categories A and B are lower for most sub

- items with frequency d istributions skewed to the right there is sufficient variation across response options to differentiate across the range of levels of student self

- efficacy perceptions. The missing rate is close to zero (0.37

– 0.45%) across all questionnaire booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 8%

- point s indicating reasonably stable performance across different booklets for this item.

This item and all sub

- items show satisfacto ry item statistics to be included in the TEL student questionnaire. In addition to an analysis as stand

- alone items, the sub

- item s should be analyzed as part
of a potential broader student self

- efficacy index based on aggregation of these items with items
VE682217 and VE682317 , which were designed to measure student self

- efficacy in the two other

content areas.

Item content was reviewed in more detail to evaluate whether the number of sub

- items might be reduced considering student burden and timing constraints. This review indicated that sub

- items a and e were less strongly linked to the content of the cognitive items compared to the other sub

- items.
F urther, relative frequencies for sub

- items a and e were less balanced than for the other sub

- items

. D ropping sub

- items a and e in order to reduce student burden would seem reasonable. Retaining three19

s ub

- items for this matrix items would also increase cons istency with its Technology & Society

- related counterpart ( VE682317 ) where only three sub

- items were administered in the pilot.

Recommendation: Keep this item and sub

- items b, c, and d in the 2014 TEL Probe Administration

. Drop sub

- items a and e from the 2014 TEL Probe administration based on time restrictions/consideration of student burden.20 VE639025

20. Technology refers to all the things people make and do to their natural environment in order to get the things they want and need. How much do you disagree or agree with the following statements about technology? Select one circle in each row.

Strongly disagree Disagree Neither disagree

nor agree Agree Strongly

agree

a. Technology is important to society

. (A) (B) (C) (D) (E) VE639028

b. Technology is important to my daily life. (A) (B) (C) (D) (E) VE639043

c. Learning about technology will help me in the future. (A) (B) (C) (D) (E) VE639048

d. Learning about technology will help me do (or get) the job I want. (A) (B) (C) (D) (E) VE639046

e. I enjoy learning about technology. (A) (B) (C) (D) (E) VE639053

f. I enjoy using technology. (A) (B) (C) (D) (E) VF009048
Area: Common

Issue: Student Engagement
Booklets : 2, 3, 6, 9

Item Review: This is a common item across the three TEL areas. It was designed to capture student engagement, specifically interest in technology. The frequency distribution across response options is skewed for all sub

- items

. All six sub

- items have very low response rate s for both options A and B while relative frequencies for the categories D and E are very high

. This might indicate that the sub

- it ems represent highly socially desirable statements. Compared to item VE639166 which also addresses student interest, this item

seems less suited to differentiate across a wide range of levels of interest. The response categories would need to be revised prior to an operational use

. The missing rate is close to zero ( 0.31

– 0.44%) across all questionnaire booklets in the aggregated response frequency

. The maximum difference in the response proportion across the four booklets in which the item has been admi nistrated is 8%

- point s indicating reasonably stable performance across different booklets for this item.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance as well as time restrictions/consideration of student burden.21 VF009050

21. Engineering refers to using skills or knowledge to solve problems that meet people’s wants and needs. How much do you disagree or agree with the following statements about engineering? Select one circle in each row.

Strongly disagree Disagree Neither disagree

nor agree Agree Strongly

agree

a. Engineering is important to society. (A) (B) (C) (D) (E) VF009051

b. Engineering is important to my daily life. (A) (B) (C) (D) (E) VF009052

c. Learning about engineering will help me in the future. (A) (B) (C) (D) (E) VF009053

d. Learning about engineering will help me do (or get) the job I want. (A) (B) (C) (D) (E) VF009054

e. I enjoy learning about engineering. (A) (B) (C) (D) (E) VF009055

f. I enjoy solving problems. (A) (B) (C) (D) (E) VF009056
g. I enjoy fixing things. (A) (B) (C) (D) (E) VF009061 h. I enjoy creating, building, or designing things. (A) (B) (C) (D) (E) VF009064

i. I enjoy figuring out how things work. (A) (B) (C) (D) (E) VF009065

j. I do things that I would describe as engineering. (A) (B) (C) (D) (E) VF009066

Area: Common

Issue: Student Engagement
Booklets : 2, 3, 6, 9

Item Review: This is a common item across the three TEL areas. It was designed to capture student engagement, specifically interest in engineering. The frequency distribution across response options is skewed for all sub

- items

. All ten sub

- items have low response rate less than 10% for both options A and B while relative frequencies for the categories D and E are very high

. This might indicate that the sub

- items represent highly socially desirable sta tements. Compared to item VE639166 which also addresses student interest, this item seems less suited to differentiate across a wide range of levels of interest. The response categories would need to be revised prior to an operational use

. The missing rate is less than 1% (0.51

– 0.66%) across all questionnaire booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 6%

- point s indicating reasonably stable performance across different booklets for this item.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance as well as time restrictions/consideration of student burden.22 V E682274

22. Who taught you most of what you know about building things, fixing things, or how things work?

A. I taught myself

. B. Family members C. Friends D. Teachers E. Someone else
Area: Design and Systems

Issue: Availability and Use of Instructional Resources
Booklets : 2, 3, 6, 9

Item Review: This is an item that provides contextual information for cognitive assessment pertaining to Design and Systems

. The frequency distribution across response options is clearly not satisfactory for the item. Options c, d, and e have low response rate s less th an 10%. The missing rate is close to zero (0.23%) across all booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 4%

- point s indicating reasonably stable performance across differen t booklets for this item.

Overall, this item does not perform well enough to be included in the TEL student questionnaire without making revisions

.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance.23 VE682217

23 . Do you think that you would be able to do each of the following? Select one circle in each row.

I definitely can’t I probably

can’t Maybe I probably

can I definitely

can

a. Publish or maintain a personal website or blog (A) (B) (C) (D) (E) VE682218

b. Create presentations with sound, pictures, or video (A) (B) (C) (D) (E) VE682219

c. Organize information into a chart, graph, or spreadsheet (A) (B) (C) (D) (E) VE682221

d. Compare products using the Internet (A) (B) (C) (D) (E) VE682222

Area: Information and Communication Technology

Issue: Student Engagement
Booklets : 2, 3, 6, 9

Item Review: This is an item that provides contextual information for the cognitive assessment pertaining to Information and Communication Technology (ICT) literacy. The item collects important information regarding student self

- efficacy in conducting activities that are related to ICT . Meta

- analyses have shown that self

- efficacy is one of the strongest predictors of student achievement (e.g., Richardson et al, 2012). This item can be considered an important achievement predictor as well as an informative outcome with policy relevance by itself.

Even though relative frequencies for response categories A and B are lower for most sub

- items with frequency distributions s kewed to the right there is sufficient variation across response options to differentiate across the range of levels of student self

- efficacy perceptions. The missing rate is close to zero (0.37

– 0.44%) across all questionnaire booklets. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 8%

- points indicating reasonably stable performance across different booklets for this item.

This item and all sub

- items show satisfacto ry item statistics to be included in the TEL student questionnaire. In addition to an analysis as stand

- alone items, the sub

- items should be analyzed as part of a potential broader student self

- efficacy index based on aggregation of these items with items
VE682276 and VE682317 , which were designed to measure student self

- efficacy in the two other

content areas.

Item content was reviewed in more detail to evaluate whether the number of sub

- items might be reduced considering student burden and timing constrai nts. This review indicated that sub

- item d was less strongly linked to the content of the cognitive items compared to the other sub

- items. Further, relative frequencies for sub

- item d were less balanced than for the other sub

- items. Dropping sub

- item d in order to reduce student burden would seem reasonable. Retaining three sub

- items for this matrix items would also increase consistency with its Technology & Society

- related counterpart ( VE682317 ) where only three sub

- items were administered in the pilot.24

Re commendation: Keep this item and sub

- items a, b, and c in the 2014 TEL Probe Administration. Drop sub

- item d from the 2014 TEL Probe administration based on time restrictions/consideration of student burden.25 V E 682215

24. Who taught you most of what you know about using computers or other digital technology for collecting or sharing information?

A. I taught myself

. B. Family members
C. Friends
D. Teachers
E. Someone else
Area: Information and Communication Technology

Issue: Availability and Use of Instructional Resources
Booklets : 2, 3, 6, 9

Item Review: This is an item that provides contextual information for cognitive assessment pertaining to Information and Communication Technology (ICT) literacy. The frequency distribution across response options is not satisfactory for th is item. Options c and e have low response rate less than 10%. The missing rate is close

to zero (0.38%) across all booklets

. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 4%

- points indicating reasonably stable performance across different booklets for this item.

Overall, this item does not perform well enough to be included in the TEL student questionnaire without making revisions.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance.26 V E639847

25. Have you ever studied technology or engineering topics in any of the following classes or subjects in school? Select one or more squares.

A. Mathematics
B. Science
C. Social studies or history
D. I have not studied technology or engineering in any of the classes or subjects listed above.
Area: Common

Issue: Organization of Instruction
Booklets : 3, 4, 7, 8

Item Review: This is a common item across the three TEL areas. The frequency distribution across response options is satisfactory for the item. The missing rate is close to zero (0.14%) in the aggregated response frequency

. The maximum difference in the response propor tion across the four booklets in which the item has been administrated is 15%

- point, which might indicate that there is some variation in item performance across booklets.

This item shows satisfactory performance to be included in the TEL student questionnaire.
Item VE639842 , however, covers similar content and should be given higher priority as it addresses technology and engineering classes more explicitly than this item.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on time restrictions/consideration of student burden.27 VE638956

26. In school , how often have you ever done the following activities? Select one circle in each row.

Never Once or

twice Three to

five times More than

five times

a. Used tools or materials to fix or build something (A) (B) (C) (D) VE638957

b. Used different tools, materials, or machines to see which are best for a given purpose (A) (B) (C) (D) VE638959

c. Built or tested a model to see if it solves a problem (A) (B) (C) (D) VE638963

d. Figured out why something is not working in order to fix it (A) (B) (C) (D) VE682247

e. Taken something apart in order to fix it or see how it works (A) (B) (C) (D) VE638965

f. Designed a computer program (A) (B) (C) (D) VE682248
Area: Design and Systems

Issue: Organization of Instruction
Booklets : 3, 4, 7, 8

Item Review:
This item provides contextual information for the cognitive assessment pertaining to Design and Systems

. It focuses on TEL

- related learning opportunities and experiences at school. The same sub

items are administered in its outside

- of

- school counterpart ( VE638983 ). The frequency distribution across response options is satisfactory for five of the six sub

- items w ith a reasonable balance of responses across all response categories . Sub

- item f has low relative frequencies for categories C and D. The response categories seem to work not very well for this sub

- item. The missing rate is close to zero (0.26

– 0.41%) across all booklets. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 8%

- point indicating reasonably stable performance across different booklets for this item.

Overall, this item and sub

- items a

- e show satisfactory item statistics to be included in the TEL student questionnaire. There is no evidence that any response categories for these items need to be collapsed, expanded, or deleted. This matrix item covers similar content as matrix item VE682225 . It should be given priority over item as the response options represent more quantifiable, behavior

- related categories than those in item VE682225

.

Item content was reviewed in more detail to evaluate whether the number of sub

- items might be reduced considering student burden and timing constraints. This review indicated that sub

- items a and f were less strongly linked to the content of the cognitive items compared to the other sub

- items. Dropping sub

- items a and f in order to reduce studen t burden would seem reasonable.

Recommendation: Keep th is item and sub

- items b, c, d, and e in the 2014 TEL Probe Administration

. Drop sub

- item a and f from the 2014 TEL Probe administration based on item performance (sub

- item f) and time restrictions/consideration of student burden.28 VE638 983

27. Outside of school , how often have you ever done the following activities? Select one circle in each

row.

Never Once or twice Three to

five times More than

five times

a. Used tools or materials to fix or build something (A) (B) (C) (D) VE638986

b. Used tools or materials to plan or design something (for example, cake recipe, party) (A) (B) (C) (D) VE009777

c. Used different tools, materials, or machines to see which are best for a given purpose (A) (B) (C) (D) VE638998

d. Built or tested a model to see if it solves a problem (A) (B) (C) (D) VE639038

e. Figured out why something is not working in order to fix it (A) (B) (C) (D) VE682267

f. Taken something apart in order to fix it or see how it works (A) (B) (C) (D) VE639042

g. Designed a computer program (A) (B) (C) (D) VE682268

Area: Design and Systems

Issue: Student Engagement
Booklets : 3, 4, 7, 8

Item Review: This is an item that provides contextual information for the cognitive assessment pertaining to Design and Systems

. It focuses on TEL

- related learning opportunities and experiences outside of school. Sub

items a and c

- g are also administered in its at

- school counterpart ( VE638956 ). The frequency distribution across response options is s atisfactory for six of the seven sub

- items with reasonable balance of responses across all response categories . Sub

- item g has low relative frequencies for categories C and D . The missing rate is less than 1% (0.46

– 0.53 %) across all booklets. The maximu m difference in the response proportion across the four booklets in which the item has been administrated is 8%

- point s indicating reasonably stable performance across different booklets for this item.

Overall, this item and sub

- items a

– f show satisfactor y item statistics to be included in the TEL student questionnaire. There is no evidence that any response categories for these items need to be collapsed, expanded, or deleted. This matrix item covers similar content as matrix item VE682232 . It should be given priority over item as the response options represent more quantifiable, behavior

- related categories than those in item VE682232

.

Item content was reviewed in more detail to evaluate whether the number of sub

- items might be reduced con sidering student burden and timing constraints. This review indicated that sub

- items a , b, and g were less strongly linked to the content of the cognitive items compared to the other sub

- items.29

Moreover, sub

- item b was not included in the at

- school version of this matrix item, making it less valuable for direct comparisons of learning opportunities and behaviors at and outside of school. Dropping sub

- items a, b, and g to reduce student burden would seem reasonable.

Recommendation: Keep this item and sub

- it ems c, d, e, and f in the 2014 TEL Probe Administrat ion

.

Drop sub

- item s a, b, and g from the 2014 TEL Probe administration based on item performance (sub

item g) and time restrictions/consideration of student burden.30 V F238958

28. For school work , how often do you use a computer or other digital technology for the following activities? Select one circle in each row.

Never or almost

never A few times a
year Once or twice a month Once or

twice a week Every day or almost every day

a. Participate in online discussion forums, social networking sites, or virtual communities (A) (B) (C) (D) (E) VE238965

b. Work with others to solve a problem (A) (B) (C) (D) (E) VF238968

c. Get information from experts (people with strong skills or knowledge in a subject) (A) (B) (C) (D) (E) VF238969

d. Maintain a website or blog (A) (B) (C) (D) (E) VF238973

e. Search for information (for example, browse the Internet or check out websites) (A) (B) (C) (D) (E) VF23897 4

f. Play games or run simulations (A) (B) (C) (D) (E) V F238975
Area: Information and Communication Technology

Issue: Organization of Instruction
Booklets : 3, 4, 7, 8

Item Review: This is an item that provides contextual information for cognitive assessment on Information and Communication Technology literacy. It focuses on TEL

- related learning opportunities and experiences at school. In contrast to item VE639123 , this matrix item does not have a outside

- of

- school type counterpart. The frequency distribution across response options is satisfactory for four of the six sub

items with reasonable balance of responses across all response categories. For two sub

- items f requency distributions indicate less variation among students: s ub

- items d and e have low response

rate s less than 10% for some response options

. However, NCES might want to retain the response categories as is to maintain consistency across sub

- items. The missing rate is less than 1% (0.56

– 0.72%) across all booklets. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 8%

- point indicating reasonably stable performance across different book lets for this item.

Overall, this item shows satisfactory performance to be included in the TEL student questionnaire. There is no evidence that any response categories for these items need to be collapsed, expanded, or deleted. This matrix item covers s imilar content as matrix item VE639123 . Compared to VE639123, one issue regarding the current item is that no direct comparison between at school and out

- of

- school experiences can be made. Giving consideration to the timing constraints and reducing student burden, dropping this matrix item including all sub

- items would seem reasonable.

Recommendation: Drop this matrix item including all sub

- items from the 2014 TEL Probe administration based on time restrictions/consideration of student burden.31 V E639842

29. Have you ever taken or are you currently taking any of the following classes or subjects in school? Select one or more squares.

A. Industrial technology (for example, auto mechanics, carpentry)
B. Engineering (for example, robotics, bridge building, rocketry)
C. Any class that involves learning to use, program, or build computers
D. Any other technology

- related class (for example, electronics, sewing, farming)
E. I have not taken any of the classes listed above.
Area: Common

Issue: Organization of Instruction
Booklets : 4, 5, 9, 10

Item Review: This is a common item across the three TEL areas. It provides important contextual information on the organization of instruction, specifically whether students have attended classes specifically targeting at technology and engineering. This is a multiple selection item. Relative selection frequencies are reasonably balanced across all response options. The missing rate is close to zero (0.34%) across all booklets. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 5%

- point s indicating reasonably stable performance across different booklets for this item.

This item shows satisfactory performance to be included in the TEL student questionnaire. In direct comparison to item VE639847 that also addresses organization of instruction, this item should be given higher priority as it addresses technology and engineering classes more explicitly than item VE639847.

Recommendation: Keep this item in the 2014 TEL Probe Administration.32 V E 639172

30. In school or outside of school , how often do the following? Select one circle in each row.

Never or almost

never A few times a
year Once or twice a month Several times a month At least

once a week

a. Participate in clubs, camps, or competitions about technology or engineering (for example, digital art and editing, design, programming, robotics, science) (A) (B) (C) (D) (E) VE639177

b. Go to museums or events to learn about technology or engineering (A) (B) (C) (D) (E) VE639178

c. Edit digital photographs or other graphic images (A) (B) (C) (D) (E) VE639179

d. Create, build, or design things (for example, robots, clothes, science projects, recipes) (A) (B) (C) (D) (E) VE639180

e. Work in a shop or garage with industrial technologies (for example, auto mechanics, machining, metalworking, construction, carpentry) (A) (B) (C) (D) (E) VE639181

f. Work with drafting or design tools (for example, computer aided design [CAD], systems analysis) (A) (B) (C) (D) (E) VE639182

g. Take online classes to learn more about technology or engineering (A) (B) (C) (D) (E) VE639183

h. Watch video or listen to audio to learn more about technology or engineering (video or audio includes online videos, movies, television shows, podcasts, radio programs) (A) (B) (C) (D) (E) VE677642

Area: Common

Issue: Student Engagement
Booklets : 4, 5, 9, 10

Item Review: This is a common item across the three TEL areas. It was developed to provide additional contextual information on student engagement in technology and engineering related activities. The frequency distribution across response options is not satisfactory for the item. Five of the eight sub

- items have low response rate s less than 10 % across several response categories

. The missing rate is less than 1% (0.58

– 0.81%) across all booklets

. T he maximum difference in the response proportion across the four booklets in which the item has been administrated is 12%

- point s , which might indicate that the item did not have a stable performance across the booklets in which the previous items vary.33

Ov erall, this matrix item does not perform well enough to be included in the TEL student questionnaire without making revisions. Given that student engagement is covered by several other items which show satisfactory performance, dropping this matrix items and all sub

- items would be reasonable.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance.34 VE681624

31. In school , how often do you learn about or discuss the following? Select one circle in each row.

Never Rarely Sometimes Often

a. How to judge reliability of sources (for example, how a website might be biased or inaccurate) (A) (B) (C) (D) VE681629

b. How to credit others for their ideas (for example, citing sources, using endnotes and footnotes in reports) (A) (B) (C) (D) VE681632

Area: Information and Communication Technology

Issue: Organization of Instruction
Booklets : 4, 5, 9, 10

Item Review: This is a common item across the three TEL areas. The item collects important information regarding whether students have learned how to judge the reliability of sources and how to credit others for their ideas. Both sub

- items function well. The frequency distribution across response options is satisfactory for both sub

- items with a reasonable balance of responses across all response categories

. There is no evidence that any response options need to be collapsed, expanded, or deleted

. The missing
rate is close to zero (0.34

– 0.38%) across all booklets.

.. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 9%

- points indicating reasonably stable performance across different booklets for thi s item.

This matrix item and both sub

- items show satisfactory performance to be included in the TEL student questionnaire.

Recommendation: Keep this item and both sub

- items in the 2014 TEL Probe Administration.35 VE 638999

32. In school , how often have you learned about or discussed the following? Select one circle in each

row.

Never Rarely Sometimes Often

a. Inventions that change the way people live (A) (B) (C) (D) VE639002 b. Choices people make that affect the
environment (A) (B) (C) (D) VE639004

c. Conditions that influence the use or availability of machines or devices (A) (B) (C) (D) VE639005

d. The ways people work together to solve problems in their community or the world (A) (B) (C) (D) VE682300

Area: Technology and Society

Issue: Organization of Instruction
Booklets : 1

– 10

Item Review: This item provides contextual information for the cognitive assessment pertaining to Technology and Society

. Response on the four sub

- items provide important information regarding students’ opportunities in school to learn how about Technology and Society. Exactly the same four sub

- items as in this item are as well used in its outside

- of

- school

- related counterpart ( VE639008 ). The frequency distribution across response options is very balanced for two of the four sub

- items. Although s ub

- items b and d have low er relative frequencies (<10%) for selected categories , there is still reasonable variation

across response options to make this a useful item in the student questionnaire. In order to maintain consistency in response options across sub

- items, NCES may consider leaving the response categories unchanged. The missing rate is very low (1.55

– 1.75%) across all booklets

.
Although item performance (based on frequencies) is slightly worse for this item than for its outside

of

- school version , it seems important to retain both item s given that no other item in the questionnaire captures students’ learning opportunities regarding Technology and Society at school. Overall, item performance is satisfactory to be included in the TEL student questionnaire, and there is
no evidence that an y response options need to be collapsed, expanded, or deleted. Moreover, the burden for the student associated with this item is low given that this matrix includes only four sub

- items.

Recommendation: Keep this item and all sub

- items in the 2014 TEL Prob e administration.36 VE639008

33. Outside of school , how often have you learned about or discussed the following? Select one circle in each row.

Never Rarely Sometimes Often

a. Inventions that change the way people live (A) (B) (C) (D) VE639012 b. Choices people make that affect the
environment (A) (B) (C) (D) VE639013

c. Conditions that influence the use or availability of machines or devices (A) (B) (C) (D) VE 639014

d. The ways people work together to solve problems in their community or the world (A) (B) (C) (D) VE 682314

Area: Technology and Society

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This item provides contextual information for the cognitive assessment pertaining to Technology and Society

. Response on the four sub

- items provide important information regarding students’ opportunities outside of school to learn about Technology and Society. Exactly the same four sub

- items as in this item are as well used in its at

- school

- related counterpart ( VE638999 ). T his matrix item funct ions very well. T he frequency distribution s across response options are satisfactory for all sub

- items with reasonable variation across response options

. There is no evidence that any response options need to be collapsed, expanded, or deleted. The missing rate is very low (2.18

– 2.39%) across all booklets.

Overall, item performance is satisfactory to be included in the TEL student questionnaire. The burden for the student associated with this item is low given that this matrix includes only four sub

- ite ms.

Recommendation: Keep this item and all sub

- items in the 2014 TEL Probe administration.37 V E 682315

34. Who taught you most of what you know about how technology, people, and the environment are related to each other?

A. I taught myself

. B. Family members
C. Friends
D. Teachers
E. Someone else
Area: Technology and Society

Issue: Availability and Use of Instructional Resources
Booklets : 1

– 10

Item Review: This is an item that provides contextual information for the cognitive assessment pertaining to Technology and Society

. The frequency distribution across response options is not satisfactory for the item. Options c and e have low response rate less than 10%. The missing rate is very low ( 2.15% ) across all booklets. The maximum difference in the response proportion across the ten booklets in which the item has been administrated is 8%

- point s indicating relatively stable performance across different booklets for this item.

Overall, this item does not perform well enough to be included in the TEL student questionnaire without making substantial revisions.

Recommendation: Drop this item from the 2014 TEL Probe Administration based on item performance.38 VE682317

35. Do you think that you would be able to do each of the following? Select one circle in each row.

I definitely can’t I probably

can’t Maybe I probably

can I definitely

can

a. Describe how inventions change society (A) (B) (C) (D) (E) VE682321

b. Compare how different activities affect the

e nvironment (A) (B) (C) (D) (E) VE682323

c. Explain why people have different tools, machines, or devices in different parts of the world (A) (B) (C) (D) (E) VE682324

Area: Technology and Society

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This is an item that provides contextual information for the cognitive assessment pertaining to Technology and Society

. The item collects important information regarding student self

- efficacy in conducting activities that are related to Technology and Soci ety . Meta

- analyses have shown that self

efficacy is one of the strongest predictors of student achievement (e.g., Richardson et al, 2012). This item can be considered an important achievement predictor as well as an informative outcome with policy relevanc e by itself.

Even though relative frequencies for response categories A and B are lower for most sub

- items with frequency distributions skewed to the right there is sufficient variation across response options to differentiate across the range of levels o f student self

- efficacy perceptions. The missing rate is low (2.74

– 3.05%) across all questionnaire booklets. The maximum difference in the response proportion across the four booklets in which the item has been administrated is 11%

- points pointing to som e variation in item performance across booklets.

Overall, t his item and all sub

- items show satisfactory item statistics to be included in the TEL student questionnaire. In addition to an analysis as stand

- alone items, the sub

- items might as well be analy zed as part of a potential broader student self

- efficacy index based on aggregation of these items with
items VE682276 and VE682217, which were designed to measure student self

- efficacy in the two other

content areas. No further reduction of the number of sub

- items seemed feasible given that only three sub

- items were included in the pilot.

Recommendation: Keep this item and all sub

- items in the 2014 TEL Probe Administration.39 VF009358

36. Before today, had you ever taken an interactive computer test similar to the one you just took? Select one circle in each row.

Yes No

a. I had taken an interactive computer test in school. (A) (B) VF009360 b. I had taken an interactive computer test outside of school. (A) (B) VF009361

Area: Common Debrief

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This is a common debrief item asking whether students have taken interactive computer test prior to taking the TEL assessment. The item collects important information regarding prior knowledge and exposure of students to similar assessments.

The relative frequenc ies for sub

- item a are reasonably balanced indicating that there is considerable variation in schools’ use of interact ive computer tests. For sub

- item b, more than 80% of all students indicate that they do not have experiences with interactive computer tests outside of school. Although variation on item b is limited, including this sub

- item seems important for the compari son of at school and

out

- of

- school experiences as well as for future trend analyses. The missing rate is low ( 3.33

– 3.7 9 % ) across all booklets

. The maximum difference in the response proportion across the ten booklets in which the item has been administrated is 12%

- point pointing to some variation in item performance across booklets

.

Overall, this item shows satisfactory performance to be included in the TEL student questionnaire.

Recommendation: Keep this item and both sub

- items in the 2014 TE L Probe Administration.40 VE401773

37. How hard was this test compared to most other tests you have taken this year in school?

A. Easier than other tests
B. About as hard as other tests
C. Harder than other tests
D. Much harder than other tests
Area: Common Debrief

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This is a common debrief item that is typically asked at the end of student questionnaire for every subject. Although response options “C” and “D” have low response rate s , the frequency distribution across all response options can be considered reasonable and satisfactory for this type of questionnaire item The missing rate is low ( 3.71% ) across all questionnaire booklets

. The maximum
difference in the re lative frequencies across the ten booklets in which the item has been administrated

is 8 %

- point s indicating reasonably stable performance across different booklets for this item.

Overall, this item shows satisfactory performance to be included in the TEL student questionnaire.

Recommendation: Keep this item in the 2014 TEL Probe Administration.41 VE401776

38. How hard did you try on this test compared to how hard you tried on most other tests you have taken this year in school?

A. Not as hard as on other tests
B. A bout as hard as on other tests
C. Harder than on other tests
D. Much harder than on other tests
Area: Common Debrief

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This is a common debrief item that is typically asked at the end of all NAEP student questionnaire s

. Although response option “D” has a less than 10% response rate, the frequency distribution across all response options can be considered reasonable and satisfactory for this type of questionnaire item

. The missing rate is low ( 4.33% ) across all booklets. The maximum difference in the response proportion across the ten booklets in which the item has been administrated is 6%

- point s indicating reasonably stable performance across different booklets for this item.

Overall, this item shows satisfactory performance to be included in the TEL student questionnaire.

Recommendation: Keep this item in the 2014 TEL Probe Administration.42 VE401779

39. How important was it to you to do well on this test?

A. Not very important
B. Somewhat important
C. Important
D. Very important
Area: Common Debrief

Issue: Student Engagement
Booklets : 1

– 10

Item Review: This is a common debrief item that is typically asked at the end of all NAEP student questionnaire s

. Although response option “A” has a less than 10% response rate, the frequency distribution across all response options can be considered reasonable and satisfactory for this type of questionnaire item

. The missing rate is low ( 4.78% ) across all booklets. The maximum difference in the response proportion across the ten booklets in which the item has been administrated is 5%

- points indicating reasonably stable performance across different booklets for this item.

Overall, this item shows satisfactory performance to be included in the TEL student questionna ire.

Recommendation: Keep this item in the 2014 TEL Probe Administration.43



### References

Holland, J.L. (1997). Making vocational choices . Odessa: FL: Psychological Assessment
Resources. Lubinski, D. (2000). Scientific and social significance of assessing individual differences: Sinking shafts at a few critical points. Annual Review of Psychology, 51, 405

- 444. Richardson, M., Abraham, C., & Bond, R. (2012). Psychological Correlates of Univ ersity Students Academic Performance: A Systematic Review and Meta

- Analysis, Psychological Bulletin , 138, 353

- 387. WestEd (2010). Technology and Engineering Literacy Framework for the 2014 National Assessment of Educational Progress . Washington, DC: WestEd.

# Technology and Engineering Literacy Reports

2. School report

## Listening

. Learning

. Leading

. 

# NAEP Item Development (ID)
Technology and Engineering Literacy ( TEL )
2014 Probe Survey Questionnaire

Recommendations to NCES

Grade 8 School Questionnaire

Deliverable in response to ID Task 3.1.2

Submitted: March 14 , 20131 

## List

Table of Contents

2014 Technology and Engineering Literacy (TEL) Grade 8 School Questionnaire:

Post

- Pilot Analysis and Recommendations

Background

................................ ................................ ................................ ................................ ............. 2

Criteria for Item Review

................................ ................................ ................................ .......................... 2

Recommen dations for the 2014 TEL Probe Assessment

– Overview

................................ ..................... 4

Item Review and Recommendations for the 2014 TEL School Grade 8 Probe Assessment

................... 6

References

................................ ................................ ................................ ................................ ............ 282 

### 2014 Technology and Engineering Literacy (TEL) Grade 8 School Questionnaires:

Post

- Pilot Analysis and Recommendations

This document provide s a post

- pilot review of the grade 8 school Technology and Engineering Literacy (TEL) questionnaire using data collected in the 2013 pilot administration. The goal of this report is to evaluate the performance of the items and determine whether revisions o r deletions are necessary for the 2014 TEL probe assessment.



### Background

The NAEP TEL assessment measures three core areas of interest: Technology and Society (T&S), Design and Systems (D&S), and Information and Communication Technologies (ICT). Technolo gy and Society addresses the effects that technology has on society and on the natural world. Design and Systems covers the nature of technology, the engineering design process, and basic principles of dealing with everyday technologies. Information and
Co mmunication Technology includes computers and software learning tools, networking systems and protocols, handheld digital devices, and other technologies for accessing, creating, and communicating information and for facilitating creative expression (WestE d, 2010). The TEL Issues Paper identified four broad issues that informed and guided the survey questionnaire development administered in the pilot assessment. These are: availability of school resources; organization of technology and engineering instruct ion; teacher preparation; and student engagement. Each of these issues comprises several sub

- issues. Two of these four issues were covered in both the student and school questionnaires. Teacher preparation was covered in the school questionnaire only and s tudent engagement was covered in the student questionnaire only.



### Criteria for Item Review

Analysis of Frequency Distributions 

### Following the same procedure as for other background questionnaire item reviews, a set of frequency

- based flagging criteria were considered in evaluating whether items were applicable to the targeted population. It is important to keep in mind that flags are indications that a particular item should be thoroughly evaluated. Flags are not absolute criteria for making decisions regar ding the use or quality of items. Instead, the flagging criteria should be viewed collectively, along with other criteria and professional judgment, in recommending keeping, revising, or dropping items from the 2014 TEL Probe survey questionnaires.

For thi s analysis, we examined item response frequencies for response options. We also assessed item non

- response patterns to determine whether problematic items or response3



### options warrant revising items, expanding or collapsing response categories, or dropping an item. The flagging criteria on response patterns and item non

- response (i.e., missing response) rates for reviewing the data are:

 A high percentage of item non

- response (relative to other adjacent items) may indicate that the item content might have be en problematic (e.g., ambiguous, burdensome, overly complex, offensive) or that the format might have caused respondents to overlook the item. Note that, this criterion does not apply to multiple selection multiple

- choice items, because the missing rate fo r “select all that apply” items contains both missing responses and “not apply” responses.

 Low single

- category response rates (e.g., <10%) may indicate that a category does not apply to this population and possibly that different categories may be more
in formative.

 High single

- category response rates (e.g., >80%) may indicate that almost all respondents in the population fall into one category and that a limited range of demographics or behavior indicators are collected.

Implications of flags on any of th ese criteria for a given item will be discussed in more detail. Note again that, flagging an item on one of these criteria does not, by itself; warrant the necessity for revising or dropping the item. Whether an item needs to be modified or revised also de pends on whether response categories are unique to an item versus representing one of the standard formats used across many questionnaire items, and whether certain response categories are needed from a theoretical perspective. Maintaining a high level of consistency across items is one important consideration for the validity of the questionnaires as well as the trend information being collected.

Response Time/Burden 

### The length of time it takes a respondent to complete a questionnaire is important. If a

q uestionnaire to too long or “burdensome,” the respondent will not exert their best efforts, especially at the end of the questionnaire. It does not appear that school administrators were overly burdened with the length of this questionnaire. The missing ra te for the TEL School questionnaire ranged from 1%

– 2%.4



### Recommendations for the 2014 TEL Probe Assessment

– Overview In the following we will summarize our item review in detail for each item, along with the recommendation for the 2014 TEL Probe asse ssment.

Table 1 on the following page presents an overview of all recommendations. For each item, the accession number and the area and issue that are addressed are given in the table.5

Table 1

- Recommendations for 2014 TEL School Probe

- Overview

Item # AccNum Area Issue Recommendation
1 VE638378 Common Organization of Instruction Keep

2 VE638432 Common Organization of Instruction Keep

VE638446 Common Organization of Instruction Keep

4 VE638450 Common Organization of Instruction Keep

5 VE638334 Common Organization of Instruction Keep

6 VE681573 Common Organization of Instruction Keep

7 VE638483 Common Organization of Instruction Keep

8 VE638475 Common Availability and Use of Instructional Resources Keep

9 VE675587 Common Availability and Use of Instructional Resources Keep

10 VE638517 Common Availability and Use of Instructional Resources Keep

11 VE638436 Common Availability and Use of Instructional Resources Keep

12 VE675659 Common Availability and Use of Instructional Resources Keep

13 VE638523 Common Availability and Use of Instructional Resources Keep

14 VE638496 Common Teacher Preparation Keep

15 VE638333 Technology and Society Organization of Instruction Keep

16 VE638350 Technology and Society Organization of Instruction Keep

17 VE638372 Design and Systems Organization of Instruction Keep

18 VE638380 Design and Systems Organization of Instruction Keep

19 VE638391 Information and Communication

Technology Organization of Instruction Keep

20 VE638410 Information and Communication

Technology Organization of Instruction Keep6



### Item Review and Recommendations for the 2014 TEL School Grade 8 Probe

A ssessment

Note: The “missing rate” for most items range from 1%

– 2% for this questionnaire. ETS finds this missing rate acceptable and will not address this criteria item

- by

- item below except where indicated.

Area: Common Issue: Organization of Instruction

Item Rev iew:
This is a common item across the three TEL target assessment areas and provides information about students’ previous instruction in six areas of technology. R esponse options are skewed toward the middle of the scale with options “C” and “D” exhibiting low response rates for sub

- items “a

– d” ( 7 %

– 9% )

.
Although relative frequencies for response options “C” and “D” are less than 10% for these four sub

items

, there is sufficient variation across response options to distinguish different levels of instructio n

.7

Moreover, keeping response options the same for all sub

- items is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep the item and sub

- items in the 2014 TEL Probe Administration.8

Area: Common Issue: Organization of Instruction

Item Review:
This is a common item across the three TEL target assessment areas and provides information about
how instru ction i s administered in the three TEL target assessment areas. Response option “C

– After
school” ha d low response rates for all three sub

- items (5%, 9%, and 7%, respectively) , h owever this response option may provide interesting findings for trend going forward. ETS does not recommend removing this response option.

The frequency distribution across the other response options shows a reasonable and expected balance of responses across response categories. Overall, the sub

- items show satisfactory item

stat istics to be included in the TEL school questionnaire.

Recommendation: Keep this item and all sub

- items in the 2014 TEL Probe Administration

.9

Area: Common Issue: Organization of Instruction

Item Review:
This is a common item across the three TEL target assessment areas and provides information about whether the school requires any technology or engineering instruction to students

.
The frequency distribution across both response options is satisfactory with a reasonable bala nce of

response

.

Recommendation: Keep this item and in the 2014 TEL Probe Administration.10

Area: Common Issue: Organization of Instruction

Item Review: This is a common item across the three TEL target assessment areas and provides information about the schools’ instruction in eight areas of technology and engineering . Response option “A” ha d low response rate s for sub

- items “a,” “b,” “e,” and “f”

– (7%

– 9%). In addition, sub

- item “i” had a missing rate of 73%.

Because four other items use the identical set of response options, response option “A” should remain. Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

For sub

- item “i,” the write

- in responses are shown on the following page.11

21 TH INGS FOR STUDENTS

ADMINISTRATIVE DIRECTIVE

GRAPHIC DESIGN ILLUSTRATOR, GOOGLE DOCS, PREP, ETC. ADOBE PHOTO

HABIT
I think society is the driving force of technology use in school.
IB Middle Years Program standards

IB MVP TECHNOLOGY COURSE REQUIREMENTS

IB Program
International Baccalaureate Middle Years Programme Technology
Keyboarding Instruction
Parental technical input
Professional growth
Project Lead the Way
Project Lead the Way (Gateway to Technology Middle School Program)
School developed programming

Student shared information

Teacher Created

TEXT INFORMATIONAL

Workbook

Recommendation: Keep this item and sub

- items in the 2014 TEL Probe Administration.12

Area: Common Issue: Organization of Instruction

Item Review:
This is a common item across the three TEL target assessment areas and provides information about whether the school provides any courses or afterschool programs that cover technology or engineering
concepts.

The frequency distribution across both response options is satisfactory with reasonable balance of

responses and should be included in the 2014 school questionnaire

.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.13

Area: Common Issue: Organization of Instruction

Item Review: The purpose of this constructed

- response item is to identify the most relevant courses that cover technology and engineering concepts. The non

- response rate for sub

- item “a” is 28% and rises to 87%
for sub

- item “e.” Among 467 respondents who were directed to the item, 103 respondents did not provide any write

- in answers to any of the five options , t her efore, the missing rate for this item is 28%.

In the write

- in responses, administrators provided a variety of courses that cover technology or engineering concepts. The item collects important information regarding TEL

- related instruction.

Recommendation: Keep this item in the 2014 TEL Probe Administration

.14

Area: Common Issue: Organization of Instruction

Item Review:
This is a common item across the three TEL target assessment areas and provides information about how often a grade 8 student is assessed about what he or she knows about technology or engineering.

The frequency distribution across the response options is satisfactory with a reasonable balance of responses across all response categories. The items and sub

- items show satisfactory item statistics to be included in the TEL school questionnaire. T here is no evidence that any response categories need to be collapsed, expanded, or deleted.

Overall, the sub

- items show satisfactory item statistics to be includ ed in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.15

Area: Common Issue: Availability and Use of Instructional Resources

Item Review:
This is a common item across the three TEL target assessment areas and provides information about the number of grade 8 students in the school, the number of computers available for educational purposes, the number of computers connected to the Internet, and the number of computers for
students to take home.

Although th e item has low response rates for all sub

- items on specific response options, the options provided clearly cover the full range of all possible responses. The item collects information about school resources regarding computer availability, and could be us ed to measure growth in the future.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.16

Area: Common Issue: Availability and Use of Instructional Resources

Item Review:
This is a common item across the three TEL target assessment areas and provides information about schools providing computers for students to take home.

While only 3% of administrators indicated a response in sub

- item “a,” this is an important sub

- item for understanding trend. T he frequency distribution across the other response options is satisfactory with reasonable balance of responses

.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.17

Area: Common Issue: Availability and Use of Instructional Resources

Item Review:

This is a common item across the three TEL target assessment areas and provides information about students taking advantage of school

- sponsored resources.

The frequency distribution across sub

- items indicates a skew to the left with low responses for sub

items “c,” “d,” and “e” (1%

– 9%). M ost school administrators indicat ed that the school does not provide

resources to students (“A”), (67%

– 76%) or at very low proportions (“B”), (11%

– 20%). Sub

- items “a” and “b” show a satisfactory frequency distribution.

While there is some ev idence to collapse response options “A” and “B” together, these are important measurements for the future study of trend. T his item and all sub

- items sho uld be included in the TEL

s chool questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.18

Area: Common Issue: Availability and Use of Instructional Resources

Item Review:
This is a common item across the three TEL target assessment areas and provides information about
resources available to teachers.

The frequency distribution across response options is satisfactory for all sub

- items with reasonable balance of responses across all response

. Moreover, keeping response options the same for all sub

items is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.19

Area: Common Issue: Availability and Use of Instructional Resources

Item Review: This is a common item across the three TEL target assessment areas and provides information about equipment available for instruction. Response option “A” had low response rates for sub

- items “a

– f” (1%

– 6%) , h owever this response opti on may provide an interesting finding going forward. ETS does not recommend removing this response option.

The frequency distribution across the other response options is satisfactory for all sub

- items with a reasonable and expected balance of responses across response categories. Moreover, keeping response options the same for all sub

- items is important for comparisons of and reporting on the results f or these items.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.20

Area: Common

Issue: Availability and Use of Instructional Resources

Item Review: This is a common item across the three TEL target assessment areas and provides information about whether the school’s capability to provide instruction in technology or engineering concepts is
hindered

.

Sub

- item “d” had low responses for response options “C”

– 9% and “D”

– 4%. Sub

- item “ f ” had a low response for “ D ”

– 5 % , h owever there is sufficient variation across response options to distinguish different levels of problems in the school

. Moreover, keeping response options the same for all sub

items, and consistent with the response options used for other version s of this matrix item is
i mportant for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.21

Area: Common Issue: Teacher Preparation

Item Review:
This is a C ommon item across the three TEL target assessment areas and provides information about teachers’ professional development

.

All sub

- items exhibited low relative frequencies for one or more response o ptions: “a” for D

– F, “b” for B and E, “c” for B , h owever there is sufficient variation across response options to distinguish different levels of professional development . Moreover, keeping response options the same for all sub

- items is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.22

Area: Technology and Society Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Technology and Society (T&S) and discusses the emphasis placed on teaching students various topics. Response option “A” had a low response rate for all sub

- items (1%

– 5%).

Although relative frequencies for response options “A” are less than 10% for the sub

- items, there is sufficient variation across response options to distinguish different levels of teaching eng agement. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on the results for these items.

Overall, the sub

- it ems show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.23

Area: Technology and Society Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Technology and Society (T&S) and discusses various student activities prior to grade 8.

Response option “A” had a low response rate for all sub

- items (2%

– 3%). Although rel ative frequencies for response options “A” are less than 10% for the sub

- items, there is sufficient variation across response options to distinguish different levels of student activities. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questi onnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.24

Area: Design and Systems Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Design and Systems (D&S) and discusses teaching emphasis for various topics.

Two response options had low response rates: “A” had a low response rate for sub

- items “a” and “c” (4%

– 5%) and “D” had low response rates for sub

- items “b,” “d,” and “e” (8%

– 9%).

Alth ough relative frequencies for response options “A” and “ D ” are less than 10% for some sub

- items, there is sufficient variation across response options to distinguish different levels of teacher emphasis. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TE L school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.25

Area: Design and Systems Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Design and Systems (D&S) and discusses the extent students participate in various activities. Response option “D” had a low response rate for six of seven sub

- items (1%

– 8% ).

Although relative frequencies for response options “D” are less than 10% for many sub

- items, there is sufficient variation across response options to distinguish different levels of student participation. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on the results for these items. Overall, the sub

- items show satisfactory item statistics to be included in th e TEL school questionnaire

.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.26

Area: Information and Communication Technology Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Information and Communication Technology (ICT) and discusses teaching emphasis for or various topics.

Two response options had low response rates: “A” had a low response r ate for sub

- items “a

– e” (1%

– 7%) and “B” had a low response rate for sub

- items “c” and “e” (7% and 9%).

Although relative frequencies for response options “A” and “B” are less than 10% for some sub

- items, there is sufficient variation across response opti ons to distinguish different levels of teaching emphasis. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on

t he results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.27

Area: Information and Communication Technology Issue: Organization of Instruction

Item Review: This item provides contextual information for the cognitive assessment pertaining to Information and Communication Technology (ICT) and discusses the extent students’ conduct various activities.

Response option “A” had a low response rate for sub

- items “b” and “d” (5% each).

Although relative frequencies for response options “A” are less than 10% for two sub

- items, there is sufficient variation across response options to distinguish different le vels of student participation. Moreover, keeping response options the same for all sub

- items, and consistent with the response options used for other versions of this matrix item is important for comparisons of and reporting on the results for these items.

Overall, the sub

- items show satisfactory item statistics to be included in the TEL school questionnaire.

Recommendation: Keep this item i n the 2014 TEL Probe Administration.28



### References

WestEd (2010). Technology and Engineering Literacy Framework for the 2014 National Assessment of Educational Progress . Washington, DC: WestEd.

# Technology and Engineering Literacy Reports

3. Final Adjudication Decisions

### 1 

## TEL 2014 Student and School Questionnaires: Final Decisions

The tables below summarize th e final adjudication decisions for the TEL student questionnaire (Table 1) and the TEL school questionnaire (Table 2) that were made after the 2013 pilot. After the initial recommendations reports for the two questionnaires were developed, an additional review and evaluation, conducted by NCES and the Governing Board, informed these final decisions based on policy relevance and conten t coverage of the questionnaire items.

In the table headers, “Type” refers to whether the item described in the row is a discrete item, an item stem of a matrix item, or a sub-item of a matrix item. The “Pilot AccNum” header refers to the item accession number used in the 2013 pilot administration. The “2014 Sequence” header refers to the item sequence that will be us ed in the 2014 administration. The “2014 AccNum”
header refers to the item accession number that will be used in the 2014 administration. The
“Area” header indicates which TEL area the item measures. In the “Area” column, the text
“Common” indicates that the item measures a topic that is comm on to all of the three specific areas of the TEL assessment (i.e., Design and Sy stems, Technology and Society, and Information and Communication Technology). The “Issue” head er indicates which specific issue the item addresses. In the “2014 Sequence” and “2014 AccN um” columns, the text “Dropped” indicates that an item that has been used in the 2013 pilo t administration, but will not be administered in 2014 per the adjudication decision. The rows that have been highlighted in da rk grey are discrete
items or item stems of matrix items. The rows thathave been highlighted in light grey are subitems of matrix items.

Table 1: Final adjudication decisions for the 2014 TEL student questionnaire

Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Discrete VE639842 1 VE639842 Common Organization of Instruction
Discrete VE639847 2 VE639847 Common Organization of Instruction
Item Stem VE681624 3 VE681624 Information and Communication Technology Organization of Instruction

Sub-item VE681629 3a VE681629 Information and Communication Technology Organization of Instruction

Sub-item VE681632 3b VE681632 Information and Communication Technology Organization of Instruction

Item Stem VE639123 4 VE639123 Information and Communication Technology Organization of Instruction

Sub-item VE639125 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VE639127 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VE639130 4a VE639130 Information and Communication Technology Organization of Instruction

### 2 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE639131 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VE639137 4b VE639137 Information and Communication Technology Organization of Instruction

Sub-item VE639136 4c VE639136 Information and Communication Technology Organization of Instruction

Item Stem VF025108 5 VF025108 Information and Communication Technology Student Engagement

Sub-item VF025109 Dropped Dropped Information and Communication Technology Student Engagement

Sub-item VF025110 Dropped Dropped Information and Communication Technology Student Engagement

Sub-item VF025112 5a VF025112 Information and Communication Technology Student Engagement

Sub-item VF025113 Dropped Dropped Information and Communication Technology Student Engagement

Sub-item VF025117 5b VF025117 Information and Communication Technology Student Engagement

Sub-item VF025116 5c VF025116 Information and Communication Technology Student Engagement

Item Stem VE682225 6 VE682225 Design and Systems Organization of Instruction
Sub-item VE682226 Dropped Dropped Design and Systems Organization of Instruction
Sub-item VE682227 Dropped Dropped Design and Systems Organization of Instruction
Sub-item VE682228 6a VE682228 Design and Systems Organization of Instruction
Sub-item VE682229 6b VE682229 Design and Systems Organization of Instruction
Sub-item VE682230 Dropped Dropped Design and Systems Organization of Instruction
Sub-item VE682231 Dropped Dropped Design and Systems Organization of Instruction
Item Stem VE638956 7 VE638956 Design and Systems Organization of Instruction
Sub-item VE638957 Dropped Dropped Design and Systems Organization of Instruction
Sub-item VE638959 7a VE638959 Design and Systems Organization of Instruction
Sub-item VE638963 7b VE638963 Design and Systems Organization of Instruction
Sub-item VE682247 7c VE682247 Design and Systems Organization of Instruction
Sub-item VE638965 7d VE638965 Design and Systems Organization of Instruction
Sub-item VE682248 Dropped Dropped Design and Systems Organization of Instruction
Item Stem VE638983 8 VE638983 Design and Systems Student Engagement
Sub-item VE638986 Dropped Dropped Design and Systems Student Engagement
Sub-item VF009777 8a VF009777 Design and Systems Student Engagement
Sub-item VE638998 8b VE638998 Design and Systems Student Engagement
Sub-item VE639038 8c VE639038 Design and Systems Student Engagement
Sub-item VE682267 8d VE682267 Design and Systems Student Engagement
Sub-item VE639042 8e VE639042 Design and Systems Student Engagement
Sub-item VE682268 Dropped Dropped Design and Systems Student Engagement
Item Stem VE682217 9 VH008232 Design and Systems, Technology and Society, & Information and Communication Technology Student Engagement

### 3 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE682218 Dropped Dropped Information and Communication Technology Student Engagement

Sub-item VE682219 9a VH008238 Information and Communication Technology Student Engagement

Sub-item VE682221 9b VH008240 Information and Communication Technology Student Engagement

Sub-item VE682222 9c VH008241 Information and Communication Technology Student Engagement

Item Stem VE682276 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682278 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682280 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682281 9d VH008243 Design and Systems Student Engagement
Sub-item VE682284 9e VH008244 Design and Systems Student Engagement
Sub-item VE682286 Dropped Dropped Design and Systems Student Engagement
Item Stem VE682317 Dropped Dropped Technology and Society Student Engagement
Sub-item VE682321 9f VH008245 Technology and Society Student Engagement
Sub-item VE682323 9g VH008247 Technology and Society Student Engagement
Sub-item VE682324 9h VH008248 Technology and Society Student Engagement
Item Stem VE638999 10 VE638999 Technology and Society Organization of Instruction
Sub-item VE639002 10a VE639002 Technology and Society Organization of Instruction
Sub-item VE639004 10b VE639004 Technology and Society Organization of Instruction
Sub-item VE639005 10c VE639005 Technology and Society Organization of Instruction
Sub-item VE682300 10d VE682300 Technology and Society Organization of Instruction
Item Stem VE639008 11 VE639008 Technology and Society Student Engagement
Sub-item VE639012 11a VE639012 Technology and Society Student Engagement
Sub-item VE639013 11b VE639013 Technology and Society Student Engagement
Sub-item VE639014 11c VE639014 Technology and Society Student Engagement
Sub-item VE682314 11d VE682314 Technology and Society Student Engagement
Discrete VE682274 12 VE682274 Design and Systems Availability and Use of Instructional Resources
Discrete VE682215 13 VE682215 Information and Communication Technology Availability and Use of Instructional Resources
Discrete VE682315 14 VE682315 Technology and Society Availability and Use of Instructional Resources
Item Stem VF009358 15 VF009358 Common Student Engagement
Sub-item VF009360 15a VF009360 Common Student Engagement
Sub-item VF009361 15b VF009361 Common Student Engagement
Discrete VE401773 16 VE401773 Common Student Engagement
Discrete VE401776 17 VE401776 Common Student Engagement
Discrete VE401779 18 VE401779 Common Student Engagement
Item Stem VE639166 Dropped Dropped Common Student Engagement
Sub-item VE639168 Dropped Dropped Common Student Engagement
Sub-item VE639169 Dropped Dropped Common Student Engagement
Sub-item VE639171 Dropped Dropped Common Student Engagement
Sub-item VE639173 Dropped Dropped Common Student Engagement

### 4 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE639174 Dropped Dropped Common Student Engagement
Sub-item VE639175 Dropped Dropped Common Student Engagement
Sub-item VE639176 Dropped Dropped Common Student Engagement
Sub-item VF009755 Dropped Dropped Common Student Engagement
Sub-item VE639170 Dropped Dropped Common Student Engagement
Item Stem VE639025 Dropped Dropped Common Student Engagement
Sub-item VE639028 Dropped Dropped Common Student Engagement
Sub-item VE639043 Dropped Dropped Common Student Engagement
Sub-item VE639048 Dropped Dropped Common Student Engagement
Sub-item VE639046 Dropped Dropped Common Student Engagement
Sub-item VE639053 Dropped Dropped Common Student Engagement
Sub-item VF009048 Dropped Dropped Common Student Engagement
Item Stem VF009050 Dropped Dropped Common Student Engagement
Sub-item VF009051 Dropped Dropped Common Student Engagement
Sub-item VF009052 Dropped Dropped Common Student Engagement
Sub-item VF009053 Dropped Dropped Common Student Engagement
Sub-item VF009054 Dropped Dropped Common Student Engagement
Sub-item VF009055 Dropped Dropped Common Student Engagement
Sub-item VF009056 Dropped Dropped Common Student Engagement
Sub-item VF009061 Dropped Dropped Common Student Engagement
Sub-item VF009064 Dropped Dropped Common Student Engagement
Sub-item VF009065 Dropped Dropped Common Student Engagement
Sub-item VF009066 Dropped Dropped Common Student Engagement
Item Stem VF238958 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VE238965 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF238968 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF238969 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF238973 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF238974 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF238975 Dropped Dropped Information and Communication Technology Organization of Instruction

Item Stem VE639172 Dropped Dropped Common Student Engagement
Sub-item VE639177 Dropped Dropped Common Student Engagement
Sub-item VE639178 Dropped Dropped Common Student Engagement
Sub-item VE639179 Dropped Dropped Common Student Engagement
Sub-item VE639180 Dropped Dropped Common Student Engagement
Sub-item VE639181 Dropped Dropped Common Student Engagement
Sub-item VE639182 Dropped Dropped Common Student Engagement
Sub-item VE639183 Dropped Dropped Common Student Engagement

### 5 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE677642 Dropped Dropped Common Student Engagement
Item Stem VE682232 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682233 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682234 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682238 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682236 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682237 Dropped Dropped Design and Systems Student Engagement
Sub-item VE682235 Dropped Dropped Design and Systems Student Engagement
Item Stem VF238997 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239009 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239000 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239004 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239007 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239008 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239010 Dropped Dropped Information and Communication Technology Organization of Instruction

Sub-item VF239003 Dropped Dropped Information and Communication Technology Organization of Instruction



### Table 2: Final adjudication decisions for the 2014 TEL school questionnaire

Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Item Stem VE638378 1 VE638378 Common Organization of Instruction

Sub-item VE638386 1a VE638386 Common Organization of Instruction

Sub-item VE638388 1b VE638388 Common Organization of Instruction

Sub-item VE638389 1c VE638389 Common Organization of Instruction

Sub-item VE638390 1d VE638390 Common Organization of Instruction

Sub-item VE638392 1e VE638392 Common Organization of Instruction

Sub-item VE638395 1f VE638395 Common Organization of Instruction

Item Stem VE638432 2 VE638432 Common Organization of Instruction

Sub-item VE638435 2a VE638435 Common Organization of Instruction

Sub-item VE638438 2b VE638438 Common Organization of Instruction

Sub-item VE638442 2c VE638442 Common Organization of Instruction

Discrete VE638446 3 VE638446 Common Organization of Instruction

Item Stem VE638450 4 VE638450 Common Organization of Instruction

### 6 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE638453 4a VE638453 Common Organization of Instruction

Sub-item VE638456 4b VE638456 Common Organization of Instruction

Sub-item VE638457 4c VE638457 Common Organization of Instruction

Sub-item VE638462 4d VE638462 Common Organization of Instruction

Sub-item VE638459 4e VE638459 Common Organization of Instruction

Sub-item VE638464 4f VE638464 Common Organization of Instruction

Sub-item VE638470 4g VE638470 Common Organization of Instruction

Sub-item VE638472 4h VF821977 Common Organization of Instruction

Sub-item VE638467 4i VE638467 Common Organization of Instruction

Discrete VE638334 5 VE638334 Common Organization of Instruction

Discrete VE681573 6 VE681573 Common Organization of Instruction

Item Stem VE638483 7 VE638483 Common Organization of Instruction

Sub-item VE638486 7a VE638486 Common Organization of Instruction

Sub-item VE638487 7b VE638487 Common Organization of Instruction

Sub-item VE638490 7c VE638490 Common Organization of Instruction

Item Stem VE638475 8 VE638475 Common Availability and Use of Instructional Resources Sub-item VE638480 8a VE638480 Common Availability and Use of Instructional Resources Sub-item VE638484 8b VE638484 Common Availability and Use of Instructional Resources Sub-item VE638485 8c VE638485 Common Availability and Use of Instructional Resources Sub-item VE675583 8d VE675583 Common Availability and Use of Instructional Resources Discrete VE675587 9 VE675587 Common Availability and Use of Instructional Resources Item Stem VE638517 10 VE638517 Common Availability and Use of Instructional Resources Sub-item VE638518 10a VE638518 Common Availability and Use of Instructional Resources Sub-item VE638519 10b VE638519 Common Availability and Use of Instructional Resources Sub-item VE638520 10c VE638520 Common Availability and Use of Instructional Resources Sub-item VE638521 10d VE638521 Common Availability and Use of Instructional Resources Sub-item VE638522 10e VE638522 Common Availability and Use of Instructional Resources Item Stem VE638436 11 VE638436 Common Availability and Use of Instructional Resources Sub-item VE638440 11a VE638440 Common Availability and Use of Instructional Resources Sub-item VE638441 11b VE638441 Common Availability and Use of Instructional Resources

### 7 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE638443 11c VE638443 Common Availability and Use of Instructional Resources Sub-item VE638445 11d VE638445 Common Availability and Use of Instructional Resources Sub-item VE638449 11e VE638449 Common Availability and Use of Instructional Resources Sub-item VE638452 11f VE638452 Common Availability and Use of Instructional Resources Sub-item VE638454 11g VE638454 Common Availability and Use of Instructional Resources Sub-item VE675624 11h VE675624 Common Availability and Use of Instructional Resources Item Stem VE675659 12 VE675659 Common Availability and Use of Instructional Resources Sub-item VE677022 Dropped Dropped Common Availability and Use of Instructional Resources Sub-item VE677568 12a VE677568 Common Availability and Use of Instructional Resources Sub-item VE677569 12b VE677569 Common Availability and Use of Instructional Resources Sub-item VE677570 12c VE677570 Common Availability and Use of Instructional Resources Sub-item VE677571 12d VE677571 Common Availability and Use of Instructional Resources Sub-item VE677572 12e VE677572 Common Availability and Use of Instructional Resources Sub-item VE677573 12f VE677573 Common Availability and Use of Instructional Resources Sub-item VE677574 12g VE677574 Common Availability and Use of Instructional Resources Item Stem VE638523 13 VE638523 Common Availability and Use of Instructional Resources Sub-item VE638524 13a VE638524 Common Availability and Use of Instructional Resources Sub-item VE638525 13b VE638525 Common Availability and Use of Instructional Resources Sub-item VE638526 13c VE638526 Common Availability and Use of Instructional Resources Sub-item VE638528 13d VE638528 Common Availability and Use of Instructional Resources Sub-item VE638529 13e VE638529 Common Availability and Use of Instructional Resources Sub-item VE638533 13f VE638533 Common Availability and Use of Instructional Resources Sub-item VE638534 13g VE638534 Common Availability and Use of Instructional Resources Sub-item VE638535 13h VE638535 Common Availability and Use of Instructional Resources Item Stem VE638496 14 VE638496 Common Teacher Preparation

Sub-item VE638497 14a VE638497 Common Teacher Preparation

### 8 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE638498 14b VE638498 Common Teacher Preparation

Sub-item VE638504 14c VE638504 Common Teacher Preparation

Item Stem VE638333 15 VE638333 Technology and Society Organization of Instruction

Sub-item VE638336 15a VE638336 Technology and Society Organization of Instruction

Sub-item VE638338 15b VE638338 Technology and Society Organization of Instruction

Sub-item VE638340 15c VE638340 Technology and Society Organization of Instruction

Sub-item VE677585 15d VE677585 Technology and Society Organization of Instruction

Item Stem VE638350 16 VE638350 Technology and Society Organization of Instruction

Sub-item VE638354 16a VE638354 Technology and Society Organization of Instruction

Sub-item VE638355 16b VE638355 Technology and Society Organization of Instruction

Sub-item VE638356 16c VE638356 Technology and Society Organization of Instruction

Item Stem VE638372 17 VE638372 Design and Systems Organization of Instruction

Sub-item VE638375 17a VE638375 Design and Systems Organization of Instruction

Sub-item VE638376 17b VE638376 Design and Systems Organization of Instruction

Sub-item VE638377 17c VE638377 Design and Systems Organization of Instruction

Sub-item VE639184 17d VE639184 Design and Systems Organization of Instruction

Sub-item VE677599 17e VE677599 Design and Systems Organization of Instruction

Sub-item VE677600 17f VE677600 Design and Systems Organization of Instruction

Item Stem VE638380 18 VE638380 Design and Systems Organization of Instruction

Sub-item VE677603 18a VE677603 Design and Systems Organization of Instruction

Sub-item VE638383 18b VE638383 Design and Systems Organization of Instruction

Sub-item VE638384 18c VE638384 Design and Systems Organization of Instruction

Sub-item VE677604 18d VE677604 Design and Systems Organization of Instruction

Sub-item VE638385 18e VE638385 Design and Systems Organization of Instruction

Sub-item VE677605 18f VE677605 Design and Systems Organization of Instruction

Sub-item VE677606 18g VE677606 Design and Systems Organization of Instruction

Item Stem VE638391 19 VE638391 Information and Communication Technology Organization of Instruction

Sub-item VE638396 19a VE638396 Information and Communication Technology Organization of Instruction

Sub-item VE638399 19b VE638399 Information and Communication Technology Organization of Instruction

Sub-item VE677607 19c VE677607 Information and Communication Technology Organization of Instruction

Sub-item VE677609 19d VE677609 Information and Communication Technology Organization of Instruction

Sub-item VF239167 19e VF239167 Information and Communication Technology Organization of Instruction

Sub-item VE677608 19f VE677608 Information and Communication Technology Organization of Instruction

Item Stem VE638410 20 VE638410 Information and Communication Technology Organization of Instruction

### 9 Type Pilot AccNum 2014 Sequence 2014 AccNum Area Issue Sub-item VE638433 20a VE638433 Information and Communication Technology Organization of Instruction

Sub-item VE638434 20b VE638434 Information and Communication Technology Organization of Instruction

Sub-item VE638428 20c VE638428 Information and Communication Technology Organization of Instruction

Sub-item VE638420 20d VE638420 Information and Communication Technology Organization of Instruction