APPENDIX I

Memorandum on Item and Construct Discussions 

with the Bureau of Justice Statistics (BJS)

Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLSK:2011)

Spring Third-Grade National Data Collection, Fourth-Grade Recruitment, and 

Fifth-Grade Tracking

OMB Clearance Package

# 1850-0750 v.15
MEMORANDUM UNITED STATES DEPARTMENT OF EDUCATION

 Institute of Education Sciences

National Center for Education Statistics

To: Shelley Martinez, OMB

From: Gail Mulligan and Jill McCarroll, NCES

Through: Kashka Kubzdela, NCES

Date: July 19, 2013

Re: Documentation regarding discussions with the Bureau of Justice Statistics (BJS) about the 

peer victimization items proposed for inclusion in the third-grade child questionnaire

I. Background. NCES’s request for approval to conduct the Early Childhood Longitudinal Study, 

Kindergarten Class of 2010-11 (ECLS-K:2011) Pilot Tests of the Third-, Fourth-, and Fifth-Grade Direct 

Child Assessment, Child Questionnaire, and Online School Administrator Questionnaire (OMB# 1850-

0750 v.13) was approved on 1/22/2013 with the following as part of the terms of clearance: 

"While NCES is to be applauded for its proposal to develop and test using ACASI for younger audiences 

and for collecting comprehensive information on "peer victimization," NCES should recognize the 

methodological and subject matter expertise and interests of other statistical agencies and should take 

greater efforts to collaborate with them on these topics. Specifically in this case, NCES should include 

colleagues from the Bureau of Justice Statistics in its discussions of pilot results and planning the full 

scale administration, both on the use of ACASI and on responsible uses of the victimization data given 

the different methods and questions being used to develop them in this study from those in production use 

by BJS for only a slightly older population. NCES should demonstrate in its next clearance package how 

it has consulted in a meaningful way with BJS."

II. Initial Contact with BJS. In accordance with these terms, NCES consulted with Michael Planty and 

Jenna Truman, who work on the National Crime Victimization Survey, School Crime Survey, at BJS. This 

memo documents those consultations. 

NCES staff sent BJS a copy of the Child Questionnaire (CQ) being tested in the 2013 pilot test, 

highlighting the proposed peer victimization items in this questionnaire. The parent interview and teacher 

questionnaire were also provided, since they include comparable questions on children’s peer victimization 

asked of those respondents. BJS staff were asked for any feedback they may have on the wording of the 

items or the ECLS-K:2011 staff’s rationale for including these items in the ECLS-K:2011 child 

questionnaire. 

The following background information was also provided to staff at BJS:

-There was a desire to include a child questionnaire beginning in the third-grade round of the ECLSK:2011, just like there was in the ECLS-K. The CQ includes all the items from the self-description 

questionnaire, which was used in the ECLS-K, as well as items tapping newer constructs that were 

recommended by our content review panel (CRP). One of the constructs we have included items for is 

peer victimization. 

- In the ECLS-K, the CQ was a hard-copy instrument that children completed after the cognitive 
assessment. Assessors entered children’s responses into their laptops at the end of the day. We wanted 

to move to ACASI to give children a greater sense of privacy as well as reduce burden on assessors and 

reduce the potential for data entry error. We’ve only been testing for about 2 weeks now, and the 

children seem to be doing OK with the ACASI. They have headphones and use a stylus pen to enter their 

responses. 

- The purpose of our pilot really is to test the ACASI and the data collection methods, not the items 

themselves. Our schedule is very compressed, so we purposely picked items from established scales for 

which the psychometric properties are known. The one exception is the items on peer victimization. 

More specific background on the peer victimization items:

We are currently in the field (for the second-grade national data collection) asking parents and teachers 

four items tapping peer victimization, which we pilot tested (for parents only) about a year ago. When we 

initially submitted a request for approval to ask these items, we were calling them bullying items. At that 

time, OMB made us aware of the federal effort to develop items tapping bullying that could be asked in 

surveys across the government. At that time, we talked with Kathy Chandler who is working on this issue 

for NCES due to her involvement with the School Crime and Safety Survey. We realized that our items 

were not tapping bullying because they missed a key aspect of bullying (the real or perceived threat of 

repeated occurrences) and decided that we could not effectively tap bullying in items asked of parents 

and teachers. Another key issue was the bullying work group had not yet finalized their items, and we did 

not think it was a good idea to use interim items. Thus, we decided to use the four items on peer 

victimization that are adapted from existing sources

Now that we are asking questions of children, we could ask about bullying, but our CRP strongly 

recommended asking the same questions we ask parents and teachers (reworded slightly to fit the 

context/respondent) so that researchers could compare data across respondents. Before submitting our 

clearance request, we talked to Kathy again to see whether the bullying items had been finalized, and she 

said no. In light of that and the CRP’s recommendation, we decided to stay with the four items we have. 

III. Feedback from BJS and NCES Responses. BJS reviewed the items and provided thoughtful 

feedback on them. This section describes that feedback, as well as NCES’s responses to the comments 

received. 

1. BJS agreed with the general assessment that these items do not comport with the official, or rather working,

definition of bullying and are better conceptualized as items assessing peer victimization. 

2. BJS indicated it is important to consider the purpose of these peer victimization items and how they will be 

presented. They asked whether the items would be used to state victimization in the last year (e.g., X% of students 

were victimized in the previous year) and provided the following supporting information:

o We know from the NCVS and other victimization surveys that respondents telescope events into the 

reference period and that this over-reporting can be as great as 30% or more. The proposed items do 

not attempt to quantify the specific numbers of crimes/incidents just relative amounts which BJS 

indicated probably reduces the potential error. However, how do you assess the validity and reliability 

of the response categories? Have there been studies that demonstrate substantive differences between 

these categories (e.g., rarely versus sometimes; often versus very often)? Maybe there could be some

quantitative guidance given (Never, 0 times; rarely, 1 or 2 times; Sometimes, 3 or 4 times a year; often, 

once or twice a week or every month; very often, on a daily basis or multiple times a week). BJS noted 

that they are not sure about the science behind this type of guidance/cueing.
NCES responded that these items are not intended to be used to obtain an estimate such as the one BJS asked about. 

ECLS staff recognize that producing statistics such as victimization in the last year is the purview of the NCVS 

School Crime Supplement. NCES noted that these items are included to obtain one more piece of information about 

the educational experiences of children in the ECLS-K:2011 that can affect their academic outcomes. 

NCES also noted that we use these response categories for many items in the parent interview. The purpose of the 

peer victimization items really is to get a general sense of children’s experiences with peer victimization, not a 

specific number of instances. Very often it is the case that the items with response options like this end up being 

dichotomized in analyses because of the response distributions we obtain. We expect that we will not see much of 

a difference in outcomes between the children whose parents or teachers say “often” compared to those who say 

“very often.” The differences are likely to be seen between the extremes (e.g., rarely and very often). NCES noted 

that the recommendations for quantitative guidance associated with the categories are good, but there is some 

reluctance to provide guidance like that for some items in the instrument and not all of the items that have the same 

response categories. What would be considered “rarely” or “often” for one behavior or experience might not 

translate to the equivalent frequency for some other behavior or event. If quantitative guidance was provided for 

these items, it might lead parents to use the same frequencies for other items. 

3. BJS noted that the questions state “school year” and use “students” and asked if NCES is interested in peer 

victimization that occurs anywhere or only on school grounds and/or during school hours. BJS noted they have 

spent a lot of time with their definition of school victimization in the NCVS and it includes both location and 

activity. There are victimization incidents that occur on school property during summer days when school is closed, 

events that happen on the bus on the way to and from school, after-school activities, during sporting events on the 

weekends or even at another school. All may introduce confusion or error when answering victimization questions. 

There may be simple ways to modify the items or instructions in the ECLS-K:2011 items to clarify to the child as to 

what should be included. 

NCES responded that we are interested in peer victimization that occurs anywhere, not just on school grounds or 

during school hours. This is why the questions are asked of both parents and teachers. Similar questions are asked 

of both parents and teachers for several of our constructs, because children’s experiences in different contexts can 

be different. For example, we ask both parents and teachers about children’s social skills and behaviors since 

children can behave very differently at home than they do at school. Also, a parent may know little about what 

happens to the child at school, and vice versa for teachers and the home environment. We use the term “students” 

to convey that we are talking about victimization by the child’s peers or school-age students. We are not trying to 

assess abuse by adults. We also do not want parents to report on interaction with siblings. Some help text was 

added to the third-grade parent interview to clarify this.

NCES noted that we do understand that teachers are likely to report only what they know about the child’s 

experiences at school. This is a consideration people using the data will have to keep in mind and is a consideration 

that applies to many of our ECLS items. 

4. BJS asked why the ECLS would ask the child, teacher, and parent the same questions about victimization and 

wondered how a researcher would decide which estimate to use. They indicated that we know that the further one is 

away from the event/incident the less he or she knows about it and the less likely he/she is to report it accurately. 

BJS suspected that parents would have very little information and/or possibly selective information if they only 

know about the severe instances that involved their child because it was reported to them by the principal. BJS also 

provided the following observations: Parents are likely to generalize across many school grades and talk in general 

about their child’s experiences, going beyond the current school year. They may also expand the scope to issues 

outside of school when an incident involves another neighborhood child who also attends the same 

school. Teachers on the other hand are only likely to know the student in that given year. BJS concluded this 

comment by asking how a researcher would negotiate between a high victimization rate given by the child, the 
teacher’s estimate, and probably the lowest estimate by the parent, and wondered whether there is value in making 

comparisons between these sources of information. While they believed there is value, given the various differences 

and potential sources of error, BJS thought it might be hard to sort out why the estimates are different. 

NCES addressed this issue to some degree with the response to the previous comment. NCES noted that another 

issue is that we believe the ECLS-K:2011 can make a unique contribution to the study of peer victimization from a 

methodological standpoint because the study obtains this information from different reporters. We expect parents 

and teachers to be reporting with different knowledge about the child’s experiences and in different contexts. The 

question then becomes which reports are more associated with children’s outcomes? Is it what happens to children 

only in school (from teachers), or is a more general report about children’s experiences from parents more strongly 

related to outcomes? Or are both reports uniquely and strongly predictive? Regarding the data collected from both 

parents and teachers about children’s social skills, NCES had found that in later grades teacher reports are better 

predictors of children’s assessment scores, which led us to drop the social skills items from the parent interview in 

later rounds. This would not have been known if the study had not asked both parents and teachers the same (or 

very similar) questions in earlier rounds. 

5. BJS asked whether these items will be vertically aligned across grades to allow for comparisons. They noted that 

the research with the NCVS demonstrated that they could not interview children under 12 with the same instrument 

as adults due to cognitive differences. They wondered whether there is evidence that children in kindergarten and 

3

rd grade differ substantially in terms of how they view victimization. Along the same lines, they asked what NCES 

had planned to allow for a crosswalk for the estimates from grade to grade by mode differences, given that the CQ 

at third grade would be using ACASI. BJS noted that the literature does show evidence for a substantial increase in 

the reporting of victimization when privacy issues are addressed. Their concern is that the ECLS-K:2011 could 

show an increase in victimization across grades without knowing if and/or how much of the increase was due to 

mode.

NCES responded with an apology for not making it clear that third grade would be the first time these questions 

would be asked of children. The questions could potentially be asked up to three times (third, fourth, and fifth) in 

the ECLS-K:2011 but would always be asked using the same mode. NCES noted that the victimization questions 

were currently being asked of parents and teachers in the spring second-grade collection.

6. BJS had the following specific questions about the items in the parent questionnaire:

o Why are the questions split (yes/no and how often) in the parent questionnaire, but not on the child or

teacher questionnaire?

o Why is the ‘other students told lies or untrue stories’ question missing from the parent questionnaire?

NCES responded that the difference in format is simply an issue of time. The ECLS staff was looking for ways to 

cut the length of the parent interview, which is conducted by telephone. Asking these questions as yes/no and then 

only reading the frequencies if a parent says “yes” to the initial question saves time. NCES did not include the item 

about lies and stories in the second-grade collection because the ECLS staff thought it might be difficult for parents 

to answer and then would be subject to error. (Note that due to strong encouragement from the CRP who wanted 

the same four victimization questions asked of parents, teachers, and children, this item is being included in the 

parent questionnaire for third grade.)

7. BJS had the following specific comments about the items in the teacher questionnaire:

o Questions are not split, similar to student questions.

o Interesting that these items ask teachers about the actions students take as well as the victimization 

against them. Are you interested in asking these “offending” (of sorts) questions to the children? 
NCES responded that the Department of Education is constrained in what it can ask students. NCES cannot ask 

children about these topics without explicit parent permission, as they are seen as reports of self-incriminating 

behavior. 

III. Final summary feedback. BJS felt that NCES’s responses to their questions and comments were 

reasonable. They indicated that they did not see anything that is particularly sensitive or of concern in 

asking the peer victimization items of children, given the context for the questions that NCES described in 

their initial background about the items and in their responses to comments. 

It should also be noted that while the NCVS School Crime Survey is not administered to students younger 

than age 12, the survey asks about more severe and sensitive topics than are covered by the ECLS peer 

victimization items, such as drug use and possession of weapons. Also, the questions asked about similar 

topics as those included in the ECLS are much more extensive. 

As a final note, NCES indicated that they would share the results of the pilot test of the child 

questionnaire to obtain BJS input on anything of concern. However, the pilot test was highly successful so 

there was nothing to consult with BJS about.