# APPENDIX I

Memorandum on Item and Construct Discussions with the Bureau of Justice Statistics (BJS)

**Early Childhood Longitudinal**  **Study , Kindergarten**  **Class of 2010 -**  **11 (ECLS**   ** K:2011)** **Spring Third**  **- Grade National Data Collection, Fourth**  **- Grade Recruitment, and**  **Fifth -**  **Grade Tracking**

**OM B Clearance Package**  **# 1850 -**  **0750 v.15**

# M **E M**  **O RA**  **N DU**  **M UN**  **I T**  **E D**  **S T**  **A T**  **E S**  **D E**  **P A** 

**R T**  **M E**  **N T**  **OF E**  **DU C**  **A**

**T I**  **ON** I n s t i t u t e of E d u c a t i o n S c i e n c es **Nat i**  **on a**  **l Ce**  **n te**  **r f** 

**or E**  **du c** **at i**  **on S**  **tat i**  **st i**  **c s** 


### T **o :**  She l l ey Mar

tine z , O MB **F r** **o m**  **:** G ai l Mul li gan and J i l l M c C arro l l , NC ES Through: Kashka Kubzdela, NCES **D a** **te :**  July 19 , 2013 **R e**  **:** Documentation regarding discussions with the Bureau of Justice Statistics (BJS) about the

peer victimization items proposed for inclusion in the third

- grade child questionnaire **I. Background.** NCES’s request for approval to conduct the Earl y Childhood Longitudinal Study , Kindergarten Class of 2010

- 11 (ECL S

- K:2011) Pilot Tests of the Third

- , Fourth

- , and Fifth

- Grade Direct Child Assessment, Child Questionnaire, and Online School Administrator Que stionnaire (OMB# 1850

- 0750 v.13) was approved on 1/22/2013 with the following as part of the terms of clearance:

" While NCES is to be applauded for its proposal to develop and test using ACASIfor younger audiences and for collecting comprehensive inform ation on "peer victimization," NCES should recognize the methodological and subject matter expertise and interests of other statistical agencies and should take greater efforts to collaborate with them on these topics. Specificall y in this case, NCES should include colleagues from the Bureau of Justice Statistics in its discussions of pilot results and planning the full scale administration, both on the use of ACASI and on responsible uses of the victimization data given the different methods and questions being used to develop them in this study from those in production use by BJS for only

a slightly older population. NCES should demonstrate in its next clearance package how it has consulted in a meaningful way with BJS. " **I I**  **. Initial Contact with BJS**  **.** In accordance with these terms, NCES consulted with Michael Planty and Jenna Truman, who work on the National Crime Victimization Survey , School Crime Survey , at BJS. This memo documents those consultations. NCES staff sent BJS a copy of the Child Question

naire (CQ) being tested in the 2013 pilot test, highlighting the proposed peer victimization items in this questionnaire . The parent interview and teacher questionnaire were also provided, since they include comparable questions on children’s peer victimization asked of those respondents. BJS staff were asked for any feedback they

may have on the wording of the items or the ECL S

- K:2011 staff’s rationale for including the se items in the ECL S

- K:2011 child questionnaire. The following background information was also provided to staff at BJS:

- There was a desire to include a child questionnaire beginning in the third

- grade round of the ECLS

K:2011, just like there was in the ECLS

- K. The CQ includes all the items from the self

- description questionnaire, which was used in the ECLS

- K, as well as items tapping newer constructs that were recommended by our content review panel (CRP). One of the constructs we have included items for is peer victimization.

- In the ECLS

- K, the CQ was a hard

- copy instrument that children completed after the cognitiveassessment. Assessors entered children’s responses into their laptops at the end of the day. We wanted

to move to ACASI to give children a greater sense of privacy as well as reduce burden on assessors and reduce the potential for data entry error. We’ve only been testing for about 2 weeks now, and the children seem to be doing OK with the ACASI. They have headphones and use a stylus pen to enter their responses.

- The purpose of our pilot really is to test the ACASI and the data collection methods, not the items themselves. Our schedule is very compressed, so we purposely picked items from established scales for which the psychometric properties are known. The one exception is the items on peer victimization. More specific background on the peer victimization items: We are currently in the field (for the second

- grade national data collection) asking parents and teachers four items tapping peer victimization, which we pilot tested (for par ents only) about a year ago. When we initially submitted a request for approval to ask these items, we were calling them bullying items. At that time, OMB made us aware of the federal effort to develop items tapping bullying that could be asked in surveys across the government. At that time, we talked with Kathy Chandler who is working on this issue for NCES due to her involvement with the School Crime and Safety Survey. We realized that our items were not tapping bullying because they missed a key aspect o f bullying (the real or perceived threat of repeated occurrences) and decided that we could not effectively tap bullying in items asked of parents and teachers. Another key issue was the bullying work group had not yet finalized their items, and we did not think it was a good idea to use interim items. Thus, we decided to use the four items on peer victimization that are adapted from existing sources Now that we are asking questions of children, we could ask about bullying, but our CRP strongly recommended asking the same questions we ask parents and teachers (reworded slightly to fit the context/respondent) so that researchers could compare data across respondents. Before submitting our clearance request, we talked to Kathy again to see whether the bullying items had been finalized, and she said no. In light of that and the CRP’s recommendation, we decided to stay with the four items we have. 

### II **I . Feedback from BJS and NCES Responses.**  BJS reviewed the items and provided thoughtful feedback on them. This section describes that feedback, as well as NCES’s responses to the comments received. 1. BJS agree d with the general assessment that these items do not comport with the official , or rather working , definition of bullying and are better conceptualized as items assessing peer victimization. 2. BJS indicated it is important to consider the purpose of these peer victimization items and how they will be presented. They asked whether the it ems would be used to state victimization in the last year (e.g., X% of students were victimized in t

he previous year) and provided the following supporting information: o We know from the NCVS and other victimization surveys that respondents telescope events into the reference period and that this over

-reporting can be as great as 30% or more. The proposed items do not attempt to quantify the specific numbers of crimes/inciden ts just relative amounts which BJS indicated probably reduces the potential error. H owever, how do you assess the validity and reliability of the response categories? Have there been studies that demonstrate substantive differences between these categories (e.g., rarely versus sometimes; often versus very often)? Maybe there could be some quantitative guidance given (Never, 0 times; rarely, 1 or 2 times; Sometimes, 3 or 4 times a year; often, once or twice a week or every month; very often, on a daily basis or multiple times a week). BJ S noted that they are not sure about the science behind this type of guidance/cueing.NCES responded that these items are not intended to be used to obtain an estimate such as the one BJ S asked about.

ECLS staff recognize that producing statistics such as victimization in the last year is the purview of the NCVS School Crime Supplement. NCES noted that these items are included to obtain one more piece of information about the educational experiences of children in the ECLS

- K:2011 that can affect their academic outcomes. NCES also noted that we use these resp onse categories for many items in the parent interview. The purpose of the peer victimization items really is to get a general sense of children’s experiences with peer victimization, not a specific number of instances. Very often it is the case that the items with response options like this end up being dichotomized in analyses because of the response distributions we obtain. We expect that we will not see much of a difference in outcomes between the children whose parents or teachers say “ often ” compared to those who say
“ very often. ” The differences are likely to be seen between the extremes (e.g., rarely and very often). NCES noted that the recommendations for quantitative guidance associated with the categories are good, but there is some reluctance to provide guidance like that for some items in the instrument and not all of the items that have the same response categories. What would be considered “rarely” or “ often” for one behavior or experience might not translate to the equivalent frequency for so me other behavior or event. If quantitative guidance was provided for these items, it might lead parents to use the same frequencies for other items. 3. BJS noted that the questions state “scho ol year” and use “students” and asked if NCES is interested in peer victimization that occurs anywhere or only on school grounds and/or during school hours. BJS noted they have spent a lot of time with their definition of school victimization in the NCVS and it includes both location and activity. There are victimiz ation incidents that occur on school property during summer days when school is closed, events that happen on the bus on the way to and from school, after

- school activities, during sporting events on the weekends or even at another school. All may introduc e confusion or error when answering victimization questions. There may be simple ways to modify the items or instructions in the ECLS

- K:2011 items to clarify to the child as to what should be included. NCES responded that we are interested in peer victimization that occurs anywhere, not just on school grounds or during school hours. This is why the questions are asked of both parents and teachers. Similar questions are asked of both parents and teachers for several of our constructs, because children’s experiences in different contexts can be different. For example, we ask both parents and teachers about children’s social skills and behaviors since children can behave very differently at home than they do at school. Also, a parent may know little about what happens to the child at school, and vice versa for teachers and the home environment. We use the term “students” to convey that we are talking about victimization by the child’s peers or school

- age students. We are not trying to asse ss abuse by adults. We also do not want parents to report on interaction with siblings. S ome help text was added to the third

- grade parent interview to clarify this. NCES noted that w e do understand that teachers are likely to report only what they know ab out the child’s experiences at school. This is a consideration people using the data will have to keep in mind and is a consideration that applies to many of our ECLS items. 4. BJS asked why the ECLS would ask the child, te acher, and parent the same questions about victimization and wondered how a researcher would decide which estimate to use. They indicated that we know that the further one is away from the event/incident the less he or she know s about it and the less likely he/she is to report it accurately

. BJS suspected that parents would have very little information and/or possibly selective information if they only know about the severe instances that involved their child because it was reported to them by the principal

. BJ S also provided the following observations: Parents are likely to generalize across many school grades and talk in general about their child’s experiences, going beyond the current school year. They may also expand the scope to issues outside of school when an incident involves another neighborhood child who also attends the same school. Teachers on the other hand are only likely to know the student in that given year. BJ S concluded this comment by asking how a researcher would negotiate between a high victimization rate given by the child, thetea cher’s estimate, and probably th e lowest estimate by the parent, and wondered whether there is value in making

comparisons betwe en these sources of information. While they believed there is value, given the various differences and potential sources of erro r, BJS thought it might be hard to sort out why the estimates are different.
NCES addressed this issue to some degree with the response to the previous comment

. NCES noted that another issue is that we believe the ECLS

- K:2011 can make a unique contribution to the study of peer victimization from a methodological standpoint because the study obtains this information from different reporters. We expect parents and teachers to be reporting with different knowledge about the child’s experiences and in different contexts. The question then becomes which reports are more associated with children’s outcomes? Is it what happens to children only in school (from teachers), or is a more general report about children’s experiences from parents more strongly related t o outcomes ? Or are both reports uniquely and strongly predictive? Regarding the data collected from both parents and teachers about children’s social skills , NCES had found that in later grades teacher reports are better predictors of children’s assessment scores, which led us to drop the social skills items from the parent interview in later rounds. This would not have been known if the study had no t asked both parents and teachers the same (or very similar) questions in earlier rounds. 5. BJS asked whether these items will be vertically aligned across grades to allow for comparisons. They noted that the research with the NCVS demonstrated that they could not interview children under 12 with the same instr ument as adults due to cognitive differences. They wondered whether there i s evidence that children in kindergarten and 3 rd grade differ substantial ly in terms of how they view victimization

. Along the same lines, they asked what NCES had planned to allow for a crosswalk for the estimates from grade to grade by mode differences , given that the CQ at third grade would be us ing ACASI

. BJS noted that the literature does show evidence for a substantial increase in the reporting of victimization when privacy issues are addressed. The ir concern is that the ECLS

- K:2011 could show an increase in victimization across grades without knowing if and/or how much of the increase was due to mode. NCES responded with an apology for not making it clear that third grade would be the first time these questions would be asked of children. The questions could potentially be asked up to three times (third, fourth, and fifth) in the ECLS

- K:2011 but would always be asked using the same mode. NCES noted that the victimization questions were currently being asked of parents and teachers in the spring second

- grade collection. 6. BJS had the following specific questions about the items in the parent questionnaire:

o Why are the questions split (yes/no and how often) in the parent questionnaire , but not on the child or teacher questionnaire? o Why is the ‘other students told lies or untrue stories’ question missing from the parent questionnaire ? NCES responded that the difference in format is simply an issue of time. The ECLS staff was looking for ways to cut the length of the parent interview, which is conducted by telephone. Asking these questions as yes/no and then only reading the frequencies if a parent says “yes ” to the initial question saves time. NCES did no t include the item about lies and stories in the second

- grade collection because the ECLS staff thought it might be difficult for parents to answer and then would be subject to error. (Note that due to strong encouragement from the CRP who wanted the same four victimization questions asked of parents, teachers, and children, this item is being included in the parent questionnaire for third grade.) 7 . BJS had the following specific comments about the items in the teacher questionnaire:

o Questions are not split, similar to student questions. o Interesting that these items ask teachers about the actions students take as well as the victimization against them. Are you interested in asking these “offending” (of sorts) questions to the children?NCES responded that the Department of Education is constrained in what it c an ask students. NCES cannot ask

children about these topics without explicit parent permission , as they are seen as reports of self

- incriminating behavior. 

### III. Final summary feedback. BJS felt that NCES’s responses to their questions and comments were reasonable. They

indicated that they

did not see any

thing that is particularly

sensitive or of concern in asking the peer victimization items of children, given the context for the questions that NCES described in their initial background about the items and in their responses to comments. It should also be noted that while the NCVS School Crime Survey is not administered to students younger than age 12, the survey

asks about more severe and sensitive topics than are covered by the ECLS peer victimization items, such as drug use and possession of weapons

. Also, the questions asked about similar topics as those included in the ECL S are much more extensive. As a final note, NCES indicated that they would share the results of the pilot test of the child questionnaire to obtain BJS input on any thing of concern. However, the pilot test was highl y successful so there was nothing to consult with BJS about

.